Video Generation with Learned Priors

visual prediction task

Given \(n\) frames of video \(x_{1}, …, x_{n}\), predict \(T\) subsequent frames \(x_{n+1}… x_{n+T}\).

Action Conditioned Prediction Network

Visual prediction task. Two inputs:

1. taken action
2. image

Predict the next \(t\) frame. Challenge! What is the action? Predicting the action is not super easy.

RAFI

Conditional Flow Matching with Video Generation.

https://arxiv.org/pdf/2406.14436