VEGGIE: Instructional Editing and Reasoning of Video Concepts with Grounded Generation

Shoubin Yu*, Difan Liu*, Ziqiao Ma*, Yicong Hong, Yang Zhou, Hao Tan, Joyce Chai, Mohit Bansal

Code | Project Page

• We propose VEGGIE, a unified and versatile video generative model that handles various tasks for both video concept grounding and editing according to user instructions.

RACCooN: Remove, Add, and Change Video Content with Auto-Generated Narratives

Jaehong Yoon*, Shoubin Yu*, Mohit Bansal

Code | Project Page

• We present RACCooN, a versatile and user-friendly video-to-paragraph-to-video framework, enables users to remove, add, or change video content via updating auto-generated narratives.

Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, Chen Chen

Code | Project Page

• We present GroundMoRe, a new benchmark for novel Motion-Grounded Video Reasoning, designed to assess multimodal models’ reasoning and perception capabilities for motion understanding.

VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos

Ziyang Wang*, Shoubin Yu*, Elias Stengel-Eskin*, Jaehong Yoon, Feng Cheng, Gedas Bertasius, Mohit Bansal

Code | Project Page

• We present VideoTree, an adaptive tree-based video presentation/prompting with simple visual clustering for long video reasoning with LLM.

SAFREE: Train-free And Adaptive Guard For Safe Text-to-Image And Video Generation

Jaehong Yoon*, Shoubin Yu*, Vaidehi Patil, Huaxiu Yao, Mohit Bansal

Code | Project Page

• We propose SAFREE, a concept guard that can zero transfer to any visual diffusion models for safe generation.

CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion

Shoubin Yu*, Jaehong Yoon*, Mohit Bansal

Code | Project Page

• We present CREMA, an efficient & modular modality-fusion framework for injecting any new modality into video reasoning.

Bootstrapping Language-guided Navigation Learning with Self-refining Data Flywheel

Zun Wang, Jialu Li, Yicong Hong, Songze Li, Kunchang Li, Shoubin Yu, Yi Wang, Yu Qiao, Yali Wang, Mohit Bansal, Limin Wang

Code

• We present a Self-Refining Data Flywheel strategy for VLN, surpassing/approaching human performance on several benchmarks.

A Simple LLM Framework for Long-Range Video Question-Answering

Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, Gedas Bertasius

Code

• We present LLoVi, a simple yet effective framework with LLM for long-range video question-answering.

Zero-Shot Controllable Image-to-Video Animation via Motion Decomposition

Shoubin Yu, Jacob Zhiyuan Fang, Skyler Zheng, Gunnar Sigurdsson, Vicente Ordonez, Robinson Piramuthu, Mohit Bansal

Code | Homepage

• We present IVA-0, a Image-to-Video animator, enables precise control from users through in-place and out-of-place motion decomposition.

Self-Chained Image-Language Model for Video Localization and Question Answering

Shoubin Yu, Jaemin Cho, Prateek Yadav, Mohit Bansal

Code | Demo | Talk

• We propose SeViLA, which self-chained BLIP-2 for 2-stage video question-answering (localization + QA) & refine localization with QA feedback.

Regularity Learning via Explicit Distribution Modeling for Skeletal Video Anomaly Detection

Shoubin Yu, Zhongyin Zhao, Haoshu Fang, Andong Deng, Haisheng Su, Dongliang Wang, Weihao Gan, Cewu Lu, Wei Wu

Code

• We propose MoPRL, a transformer-based model incorporated with skeletal motion prior for efficient video anomaly detection.

STAR: A Benchmark for Situated Reasoning in Real-World Videos

Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B. Tenenbaum, Chuang Gan

Code | Project Page

• We propose STAR, a benchmark for neural-symbolic video reasoning in real-world scenes.

• 2022.09 - Present
• The University of North Carolina at Chapel Hill
• Computer Science, Ph.D.

• 2017.09 - 2022.06
• Shanghai Jiao Tong University
• Information Security, B.Eng.

• 2024.05 - 2024.11, Research Scientist Intern
• work with Difan Liu, Yicong Hong, Yang Zhou, Hao Tan

• 2023.05 - 2023.11, Research Scientist Intern
• work with Jocob Zhiyuan Fang, Robinson Piramuthu