FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

           

Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia*, and Ziwei Liu*
(* corresponding author)

🔆 Introduction

🤗🤗🤗 FreeTraj is a tuning-free method for trajectory-controllable video generation based on pre-trained video diffusion models.

Showcases (320x512)

"A chihuahua in an astronaut suit floating in the universe, cinematic lighting, glow effect."	"A swan floating gracefully on a lake."	"A corgi running on the grassland on the grassland."

"A barrel floating in a river."	"A dog running across the garden, photorealistic, 4k."	"A helicopter hovering above a cityscape."

📝 Changelog

• [2024.07.04]: 🔥🔥 Release the FreeTraj, trajectory controllable video generation!
• [2024.07.09]: 🔥🔥 Release a user-friendly interface.
• TODO: 1. a powerful mode for better control.

🧰 Models

Model	Resolution	Checkpoint	Description
VideoCrafter2 (Text2Video)	320x512	Hugging Face

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n freetraj python=3.8.5
conda activate freetraj
pip install -r requirements.txt

🤗 Quick start with Gradio

  gradio app/app.py

💫 Inference with Command

1. Demo

1. Download pretrained T2V models via Hugging Face, and put the model.ckpt in checkpoints/base_512_v2/model.ckpt.
2. Input the following commands in terminal.

  sh scripts/run_text2video_freetraj_512.sh

2. Plan new trajectory

1. Write new trajectory files, the format should be frame index, h start, h end, w start, w end. In the current version, the bbox size should be the same. Please refer to prompts/freetraj/traj_l.txt.
2. Modify scripts/run_text2video_freetraj_512.sh and set $traj_file.
3. Slightly increase $ddim_edit to enhance the control ability, but may reduce the video quality.

😉 Citation

@misc{qiu2024freetraj,
      title={FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models}, 
      author={Haonan Qiu and Zhaoxi Chen and Zhouxia Wang and Yingqing He and Menghan Xia and Ziwei Liu},
      year={2024},
      eprint={2406.16863},
      archivePrefix={arXiv}
}

📢 Disclaimer

We develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes. The success rate is not guaranteed due to the variety of generative video prior.

---