Music Reactive Video Generation

While generating music-reactive videos is an impor-tant task in real-world applications, it has been a challenging generation task. We argue that this challenge arises from three factors: 1) emotional feelings about music exist in a too-small region of the chromagram to transform it into a video, 2) evaluating the music-reactive generation task is fundamentally a subjective matter of individual humans, and thus, 3) applying metrics for quantitative measurement used in existing generative models is tricky. For these reasons, this paper introduces an efficient music-reactive video generation model and a metric for accurately evaluating generated music-reactive videos. For our music-reactive generation model, we present the highlighted chromagram approach to effectively represent musical struc- ture information (e.g., melody, beat, rhythm) in videos. We also introduce a quantitative measurement metric, Beat Co- occurrence, which is strongly correlated with human survey results. Our experiments demonstrate that our music-reactive generation model performs favorably in human surveys and the proposed Beat Co-occurrence distance metric.