CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation

Yuheng Chen1, Teng Hu1, Yuji Wang1, Qingdong He2, Zhucun Xue3, Qianyu Zhou4, Jason Li5, Lizhuang Ma1, Jiangning Zhang3, Dacheng Tao5
1Shanghai Jiao Tong University 2University of Electronic Science and Technology of China 3Zhejiang University 4The University of Tokyo 5Nanyang Technological University
Paper arXiv Code Dataset

📢 The dataset is large in scale, exceeding 80 TB. To save time before training, we did not apply hard cropping to the raw videos; instead, we used slice-based soft cropping guided by metadata. We are now performing hard cropping, which is a purely CPU-bound and highly CPU-intensive task, so the processing time is expected to be substantial. Meanwhile, the authors are discussing the open-source release approach for the dataset, and we plan to adopt a gated access mechanism. If you have any questions, feel free to reach out at fengjianliuli627@gmail.com 😊. Thanks for your attention 🙏!

CineDance-1M overview showing long-form multi-shot audio-video sequences and narrative structures
CineDance-1M features 1M long-form (92.8s) and multi-shot (24.2 shots) audio-video sequences, paired with hierarchical structured captions for both modalities. Compared with typical T2V datasets, it covers richer narrative structures for cinematic, narrative-driven joint generation.

Abstract

The fidelity and structural diversity of training datasets fundamentally determine the capabilities of video generation models. While commercial systems show remarkable ability to generate cinematic narratives, the progress of open-source models remains limited by the scarcity of high-quality training data.

To bridge this gap, we introduce CineDance-1M, a large-scale, open research Text-to-Audio-Video (T2AV) dataset designed specifically for multi-shot, long-form joint audio-video generation. Averaging 92.8 seconds and 24.2 continuous shots per video, it provides configurable, structured annotations for both audio and video modalities. This quality is achieved through a rigorous three-stage curation pipeline: (i) diverse sourcing and comprehensive cleansing, (ii) film-theory-inspired narrative parsing, and (iii) hierarchical dual-modal captioning.

For comprehensive assessment, we propose CineBench, featuring a diverse prompt suite and a six-dimensional, human-aligned metric system tailored for complex narrative audio-video evaluation. Furthermore, we adapt LTX-2.3 into CineDance, which demonstrates strong single-modality quality, precise audio-video alignment, and robust subject and environment consistency.

CineDance-1M

CineDance-1M targets the missing training unit for modern cinematic generation: not isolated short clips, but long audio-video sequences with consistent shot structure, aligned sound, and reusable annotations. The curation pipeline consists of three stages: data preparation and quality assessment, bottom-up multi-shot narrative parsing, and configurable structured dual-modal annotation.

1M Curated sequences
92.8s Average duration
24.2 Average shots
1080p Minimum resolution
CineDance-1M curation pipeline
Curation pipeline. CineDance-1M combines source preparation, quality assessment, multi-shot narrative parsing, and hierarchical audio-video annotation to produce structured long-form T2AV examples.

Dataset Comparison

CineDance-1M is the only 1M-scale dataset in this comparison with 1080p resolution, long-form multi-shot sequences, all-native audio, and structured audio-video annotations.

Dataset Res. Avg. dur. Avg. shots Shot caps. Audio Audio ann. Cap. len. Total dur. Clips Year
HowTo100M240p3.6s1NoneNoneNone4134.5Khr136M2019
HD-VILA-100M720p13.4s1NoneNoneNone32.5371.5Khr103M2022
Koala-36M720p13.6s1NoneNoneNone202.3137Khr36M2024
VIDGEN-1M720p10.6s1NonePartialNone89.32.9Khr1M2024
MiraData720p72.1s7.15NoneNoneNone3196.6Khr330K2024
LVD-2M720p20.2s1.86NoneNoneNone88.814.6Khr2.1M2024
OpenHumanVid720p4.6s1NoneAllNone99.712Khr16M2025
OpenS2V-5M720p5.6s1NonePartialNone312.065.8Khr3.75M2025
UltraVideo4K/8K5.3s1.17NoneNoNone824.362hr42K2025
OpenVid-1M720p7.2s1NonePartialNone126.52.1Khr1M2025
CineTrans720p10.7s2.532NoneNone250.78752hr252K2025
SpeakerVid-5M1080p8.3s1.27NoneAllASR20.6911.6Khr5.07M2025
CineDance-1M1080p92.8s24.222AllStructured6496.326.3Khr1M2026

Dataset Quality And Diversity

Statistical overview of the CineDance-1M dataset
CineDance-1M is analyzed across taxonomy, quality, duration, annotation density, narrative structure, and semantic vocabulary. These statistics support filtering, benchmark construction, and controllable long-form generation research.

CineBench

CineBench evaluates whether a model can synthesize temporally ordered multi-shot sequences from structured conditions. It contains 1,000 test cases stratified by theme/style, duration and shot count, and generation difficulty. The benchmark covers 10s with 2-3 shots, 30s with 4-9 shots, and forward-looking 60s with 10-20 shots.

Comprehensive statistical overview of CineBench
CineBench. The benchmark combines diverse taxonomic flow, quality distributions, and semantic vocabulary for long-form multi-shot audio-video evaluation.
Video Qualityfidelity, imaging quality, motion smoothness
Audio Qualityspeech clarity, background sound, acoustic artifacts
AV Synclip synchronization and semantic audio-video alignment
Prompt Alignmentcharacters, scenes, events, dialogue, sound descriptions
Narrative Continuityidentity, scene, object persistence, ordered events
Shot Structure Responseshot count, transitions, and temporal shot layout

CineDance Model

We adapt LTX-2.3 into CineDance as a robust open baseline for multi-shot long-form audio-video generation. The model uses the native joint audio-video backbone while learning shot-transition capability, subject and environment consistency, and structured prompt response from CineDance-1M.

The training strategy uses visual-temporal reference scaffolds as optimization aids, then progressively removes them so the model can retain multi-shot organization under reduced inference conditions.

BibTeX

@misc{chen2026cinedancenextgenerationmultishotlongform,
      title={CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation}, 
      author={Yuheng Chen and Teng Hu and Yuji Wang and Qingdong He and Zhucun Xue and Qianyu Zhou and Jason Li and Lizhuang Ma and Jiangning Zhang and Dacheng Tao},
      year={2026},
      eprint={2606.09639},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.09639}, 
}