Cas isn’t allowed to drive the Impala because he crashed it.
Like, actually crashed it.
Sam once nicked the paint job when he opened the door and it hit the wall. Dean didn’t speak to him for three days- and that was after cussing him out in the parking lot of a gas'n'gulp so that everyone in a three-mile-radius could hear.
Cas crashed the car.