VideoGen-RewardBench: Evaluating Reward Models for Video Generation

Overview

The prompt-video pairs are sourced from VideoGen-Eval, a dataset featuring a diverse range of prompts and videos generated by state-of-the-art video diffusion models (VDMs). Our benchmark comprises 26.5k video pairs, each annotated with a corresponding preference label.

We report two accuracy metrics: ties-included accuracy (w/ Ties) and ties-excluded accuracy (w/o Ties).

• For ties-excluded accuracy, we exclude all data labeled as ”ties” and use only data labeled as ”A wins” or ”B wins” for calculation. We compute the rewards for each prompt-video pair, convert the relative reward relationships into binary labels, and calculate classification accuracy.
• For ties-included accuracy, we adopt Algorithm 1 proposed by Ties Matter. This method traverses all possible tie thresholds, calculates three-class accuracy for each threshold, and selects the highest accuracy as the final metric. See calc_accuracy for the implementation of ties-included accuracy.

We include multiple types of reward models in this evaluation:

1. Sequence Classifiers (Seq. Classifier): A model that takes in a prompt and a video and outputs a score.
2. Custom Classifiers: Research models with different architectures and training objectives.
3. Random: Random choice baseline.
4. Generative: Prompting fine-tuned models to choose between two answers.

Note: Models with (*) after the Model are independently submitted model scores which have not been verified by the VideoGen-RewardBench team.

Acknowledgments

Our leaderboard is built on RewardBench. The prompt-video pairs are sourced from VideoGen-Eval. We sincerely thank all the contributors!

How to Submit Your Results on VideoGen-RewardBench

Please follow the steps below to submit your reward model's results:

Step 1: Create an Issue

Open an issue in the VideoAlign GitHub repository.

Step 2: Calculate Accuracy Metrics

Use our provided scripts to compute your model's accuracy:

• Ties-Included Accuracy (w/ Ties): Use calc_accuracy_with_ties
• Ties-Excluded Accuracy (w/o Ties): Use calc_accuracy_without_ties

Step 3: Provide Your Results in the Issue

Within the issue, include your reward model's results in JSON format. For example:

{
    "with_tie": {
        "overall": 61.26,
        "vq": 59.68,
        "mq": 66.03,
        "ta": 53.80
    },
    "without_tie": {
        "overall": 73.59,
        "vq": 75.66,
        "mq": 74.70,
        "ta": 72.20
    },
    "model": "VideoReward",
    "model_link": "https://huggingface.co/KwaiVGI/VideoReward",
    "model_type": "Seq. Classifiers"
}

Additionally, please include any relevant information about your model (e.g., a brief description, methodology, etc.).

Step 4: Review and Leaderboard Update

We will review your issue promptly and update the leaderboard accordingly.

Random Dataset Sample Viewer

{sampled data loads here}