I was rewatching the first episode of Haikyuu and I realized something.
The first two screencaps above are the building where Hinata and Kageyama had their Junior High tournament in the first episode. The next two screencaps are where Kageyama went in season 2 episode 6 and ended up running into Oikawa. There was never a shot in this scene of the whole building, but you can see upon comparison that the stair railing and the columns are the same as the tournament building. Where Kageyama and Oikawa met was the same place where Hinata had his first and last Jr High tournament.
Conclusion: After their fight, Kageyama returned to the place where he met Hinata for the first time.