Reordering Video Frames based on Audio Frequency Spectrum Similarity
Experiment by @mario-klingemann processes short video scene and rearranges frames in order of an audio spectrum, often placing very similar short sounds together one after the other - very surreal and possibly epilepsy-inducing:
The audio frequency spectrum of a video’s frames are clustered by their
similariy using a t-sne embedding, Then the new order of frames is
calculated using a nearest-neighbor traversal so frames that sound
similar are following each other.
Another experiment takes a music video and orders the parts of similar sounds by their pitch, using ‘Never Gonna Give You Up’:
After calculating a Constant-Q chromagram with librosa the frames of the clip are regrouped by their pitch classes.
“Hey guys, Phil here. Today’s the day we get her and Dan is freaking out. He’s been up since 3am arranging and rearranging her room, to make sure it’s absolutely perfect. I mean, I knew he was a perfectionist, but I didn’t know that he’d have enough dedication to get up at 3am…”
“I can hear you, you know!” Dan shouts. Phil follows Dan’s voice to the open door through which a frantic Dan can be seen, red in the face, rushing around. “It needs to be perfect for her! I refuse to let our child sleep in a non-aesthetic bedroom!”
Phil shows the room to the camera. The walls are painted a soft pastel pink and adorned with animal paintings in small, artistically placed, picture frames and clusters of decorative butterflies. Above the cosy silver crib with soft toys pouring out of it, are matching silver letters, spelling out the name Delilah. Next to the crib lies a very adult looking cream-white sofa, and a yellow dresser topped with cute little trinkets and bunnies made of china.
“I’d like to say that decorating the room was a joint effort, but really, it was all Dan. He may or may not have refused to let me help. Oh well!” Phil says lightheartedly, then laughs. “I still can’t believe that today’s the day!”
“Me neeiitheeerrrrrr!” Dan rushes up to Phil and embraces him, squeezing him tightly.
Phil turns to the camera and says “We’ll see you there!”
Phil is talking in a hushed voice. “Ok, so we just got here, and Dan’s filling out all the last minute paperwork. When he’s done, we’ll finally be able to take her home! I can’t believe it!”
A baby wrapped in blankets is lowered into Dan’s shaking arms. He tears up. He wraps his arms around the bundle and brings it close to his chest. Pride, joy and an overwhelming need to protect the little girl in his arms seem to fill him up. He rocks the bundle gently, stroking her head and whispering inaudibly to her. She giggles, and Dan practically melts. Tears fall down his cheeks and drip onto the baby’s forehead. “I love you” he whispers.
“We’re home! With her!” The fathers walk through the door of their daughter’s bedroom.
“Shhh. She’s sleeping,” Dan scorns, then softens his gaze as he returns it to his newly adopted daughter who is lying in a baby basket.
“Oops sorry.” Phil continues in a hushed voice. “We are now spending most of our time in this room. I guess it’s good that Dan decorated it so aesthetically. Can I hold her now?”
“No!” Dan whisper-shouts.
Delilah is sleeping in the crib, squished within the stuffed animals, and both Dan and Phil sit on the couch next to the crib, Dan in the position closer to the crib.
“We’re fathers now. How do you feel.”
“Because we’ll try to be the best parents we can be, but what if that’s not good enough?’ He looks at Delilah sleeping peacefully in her crib. “What if she grows up to hate us? What if-”
Phil grabs Dan’s shoulder. “Listen to me Dan. Being a good parent means providing safety, unconditional love, and unwavering loyalty. You’ve already ticked all those boxes. You will be the greatest dad like no one ever was.”
Dan slowly turns his head away from his daughter and towards Phil. “Did you just…?” Phil nods and smirks. Dan pulls him into a long, lingering hug. “Thank you. But it won’t just be me. We’re in this together. You and I. We’ll be the best dads she could ever want.” Phil smiles.
“Yes, we will” Phil turns to the camera and waves. “Bye!”