Milah was probably never there even before she left with Killian. She didn’t want to be anywhere near him after he deserted. I like to think Rumple was the one who taught Bae how to walk and talk. Rumple laughing and smiling as Bae takes his first steps and happily congratulating him when his first words are “papa!”
He may be the town coward, he may have a wife who hates him but when he’s with Bae that all fades away. Bae loves him. He loves this crippled, ugly, town coward no mater what. He smiles and babbles when he sees him and he waddles over to him when he comes home.
Imagine Rumple being jolted awake when baby Baelfire leaps into his bed, terrified by the thunder from the storm. It nearly gives Rumple a heart attack but once he realizes what spooked his son he begins to calm down. He can do this. He can comfort his son. He hugs him close and tries to explain that the thunder can’t hurt him (the lighting technically could but he keeps that to himself) and what finally calms Baelfire down is “I’ll always keep you safe, Bae. You don’t have to be scared. Papa’s here.” And soon he’s falling asleep, his tiny body no longer shaking like a leaf. Rumple’s own words haunt him for the rest of his life but never more than after he watches his son die in front of him.