Reordering Video Frames based on Audio Frequency Spectrum Similarity
Experiment by @mario-klingemann processes short video scene and rearranges frames in order of an audio spectrum, often placing very similar short sounds together one after the other - very surreal and possibly epilepsy-inducing:
The audio frequency spectrum of a video’s frames are clustered by their
similariy using a t-sne embedding, Then the new order of frames is
calculated using a nearest-neighbor traversal so frames that sound
similar are following each other.
Another experiment takes a music video and orders the parts of similar sounds by their pitch, using ‘Never Gonna Give You Up’:
After calculating a Constant-Q chromagram with librosa the frames of the clip are regrouped by their pitch classes.
Request from InkPoisonin! Gosh I feel a little self conscious ‘bout my singing. I mean, I’m getting much much better with getting comfy singing to a crowd, but I guess this brings whole new level of nervousness when it’s a character I’m playing as. I can’t fail you guys! I’ll always try my best! But yeah I really love this song. I figured Sans would like this genre maybe. Enjoy!