From Perception to Reasoning: Towards Adaptive Video Intelligence

Dissertation

Every day, cameras capture an enormous amount of video, from sports broadcasts and movies to security footage, hospital monitoring, traffic cameras, and personal recordings. Far more video is recorded than any person could ever watch, so we increasingly rely on AI systems to watch it for us: to find the moment a game was won, alert a caregiver when a patient falls, help a driver react to the road, or answer questions about what happened in a recording. Large multimodal models that combine vision and language have made this possible, but they still fall short of how people understand video. A person can follow an hour-long story and remember who did what, yet these models can only take in a limited amount of video at once and lose important details when longer videos are cut down to fit. A person watching a live event knows when they have seen enough to answer a question, while these models either guess too early or respond too late, which matters when a timely answer can prevent an accident or catch a critical event. And a person can often recognize someone they know from the way they move, even without seeing their face, whereas current systems rely on faces, clothing, or walking patterns that fail when a face is hidden, clothing changes, or someone is simply going about their day. Thus, closing these gaps is essential for building video systems that people can trust in the real world.

This dissertation addresses these gaps across three questions: what is happening, when to answer, and who is involved. We first present a method that processes long videos piece by piece, focusing on details relevant to the task while remembering both recent and earlier events, so it can handle videos of any length at a fixed cost. We then introduce a way for models watching live video to judge when they have seen enough evidence to answer, along with a new score and benchmark that reward answers that are both correct and on time. Finally, we introduce the task of recognizing people from how they perform daily activities, and present methods that separate a person’s identifying traits from their appearance and movement, making identification more reliable in real-world settings. Together, these contributions move video models from passively watching toward actively reasoning about what they see, when to speak, and whom to recognize.