clearly the answer of where cas was after everyone got back to the bunker is that he and dean fucked once they got home and cas was really loud and he was too embarrassed to come out during dinner (or dean was too embarrassed for him and cas was like whatever i’m going to lounge naked in your bed go eat your molecules).
anyway then later dean snuck away after cas fell asleep after round 2 to go look at pictures, drink beer, and be emo on the kitchen floor. why else would he be in the kitchen? he could bring the beer to his room and look through his mementos in a less conspicuous and more comfortable place… unless someone was already occupying his bed and he didn’t want to wake him up with his pity party.
Q.E.D. cas was sleeping in dean’s bed and he was too comfy and happy to move. dean brought him back a slice of pie. all was well.