Here’s an idea
What if Isaac grew up in a museum? Like, Addams family style?
We know Isaac has a very broad, disjointed knowledge of very specific trivia and historical stories. Imagine a little kid growing up playing pretend around the exhibits, learning to read off the plaques and making up stories about how the princes in the Story of the Four Kingdoms faced off with Billy the Kid (and regaling an impressed equally smol Miria with his knowledge!)
We also know he grew up in a house reminiscent of a mansion on Millionaire Row. Think of a museum really fast- lots of marble, broad staircases and high ceilings. Sounds pretty mansion-y to me!
(I haven’t read the novels past the anime, so it might not hold up but it’s a good theory so far)