PexelsCustom-1M

A Comprehensive Ecosystem for Open-Domain Customized Video Generation (ICASSP 2026)

The first large-scale, publicly available dataset for customized video generation (CVG), providing 1,036,431 curated (identity, text, video) triplets across 8,373 identity categories from ~320K Pexels HD videos.

• Code: https://github.com/carpedkm/CustoMDiT

Dataset Contents

CSV Schema

Annotation JSON Schema

Each JSON file contains:

Setup Guide

Follow these steps to prepare the dataset for training with CustomDiT. The goal is to create the following directory structure:

data/
├── videos/
│   └── pexels/videos-popular/{VIDEO_ID}.mp4
├── metadata/
│   ├── train.csv
│   └── val.csv
├── annotations/
│   └── pexels/videos-popular/{VIDEO_ID}/{VIDEO_ID}.json
└── reference_images/
    └── pexels/videos-popular/{VIDEO_ID}/original.png

Step 1: Download this dataset

# Install huggingface_hub if needed
pip install huggingface_hub

# Download all files
huggingface-cli download carpedkm/CustoMDiT --repo-type dataset --local-dir ./hf_download

Step 2: Set up directory structure

# Create data root
mkdir -p data/{videos,metadata,annotations,reference_images}

# Copy metadata CSVs
cp hf_download/metadata/train.csv data/metadata/
cp hf_download/metadata/val.csv data/metadata/

# Unpack annotations (preserves directory structure)
tar xzf hf_download/annotations.tar.gz -C data/annotations/

Step 3: Download videos from Pexels

Videos are hosted on Pexels and must be downloaded separately. Each videoid in the CSV maps to a Pexels URL:

• videoid = pexels/videos-popular/8500526
• URL = https://www.pexels.com/video/8500526/

Videos should be saved as MP4 files matching the videoid path:

data/videos/pexels/videos-popular/8500526.mp4

Note: We cannot provide a download script due to Pexels terms of service. You may use the Pexels API to download videos programmatically. Ensure you download the HD (1920x1080) version for best results.

Step 4: Extract reference keyframes

Use the provided script to extract the reference frame from each video:

# Install decord if needed
pip install decord

# Extract keyframes
python hf_download/extract_frames.py \
    --video_dir data/videos \
    --annotation_dir data/annotations \
    --output_dir data/reference_images \
    --metadata_csv data/metadata/train.csv

This reads the keyframe_index from each annotation JSON, extracts that frame from the video, and saves it as original.png.

Step 5: Train CustomDiT

Clone the code repository and update the training config:

git clone https://github.com/carpedkm/CustoMDiT.git
cd CustoMDiT

Edit customdit/configs/train.yaml to point to your data:

data_config:
  data_dir: /path/to/data/videos
  metadata_path: /path/to/data/metadata/train.csv
  cond_image_dir: /path/to/data/reference_images
  cond_json_dir: /path/to/data/annotations

Then start training:

conda activate customdit
pip install -r customdit/requirements.txt
export PYTHONPATH=$PWD:$PYTHONPATH
cd customdit
accelerate launch --config_file configs/accelerate_single_node.yaml train.py --config configs/train.yaml

See the code repository for full training and inference instructions.

Citation

@inproceedings{zhang2026pexelscustom,
  title={A Comprehensive Ecosystem for Open-Domain Customized Video Generation},
  author={Zhang, Jingxu and Hong, Yuqian and Kim, Daneul and Qiu, Kai and Dai, Qi and Bao, Jianmin and Yang, Yifan and Sun, Xiaoyan and Luo, Chong},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2026},
  organization={IEEE}
}