Imagine you need to find a specific moment in a two-hour recording of a meeting. You could watch the entire two hours at 1x speed. Or you could open the transcript, find the right spot, and watch just that part. Until recently, AI operated according to the first scenario. On September 1, Google released a feature that taught Gemini the second one.

How it used to work

When you fed a video to the model, it analyzed it statically: it sliced it into frames at a fixed rate—by default, one frame per second—and processed them all in their entirety. It also transcribed the audio track separately.

A two-hour recording consists of 7,200 frames. Each frame takes up tokens. Each token costs money. And it doesn’t matter that 7,100 of those frames are a static presentation where nothing happens.

Another problem: if an event lasted less than a second, it could simply fall between frames. The model wouldn’t see it.

What Has Changed

The new feature is called “agent-based video understanding.”

Instead of passively absorbing everything indiscriminately, the model decides on its own strategy: what to watch, at what speed, and through which channel. It can refer to the transcript, the video frames, or the audio—depending on what’s needed to answer a specific query. It can fast-forward, rewind, or replay a segment at a higher frame rate if something is happening quickly there. Essentially, the model has gained the ability to interact with video the way a human does: not by watching everything, but by searching for specific information.

The Numbers

Google reports the following metrics compared to static processing:

  1. up to 88% fewer tokens
  2. up to 66% lower analysis cost
  3. up to 7% higher accuracy

Interestingly, accuracy increased while the volume of processed data decreased. The logic here is that the context window isn’t cluttered with thousands of meaningless frames, so the model doesn’t get lost in the noise.

A nuance that isn’t mentioned

On short videos, agent mode may run slower. For videos up to five minutes long, the time to the first token increases: the model first needs to think, decide on a navigation strategy, and make several calls to tools before it can begin responding. In other words, this isn’t a “one-size-fits-all improvement.” It’s a tool for a specific scenario: long-form content and a query about a specific moment. For a 30-second clip, the old static processing remains the better choice. Google’s documentation recommends starting with agent mode if you’re optimizing response quality or token usage—but specifically for long-form content.

The second nuance is that token counting has become more complex. The model’s deliberations during navigation are counted as “thought tokens,” while on-demand images, audio, and transcripts are counted as “tool use tokens.” If you’re managing your budget, this is worth keeping in mind: the cost structure is different than before.

What’s Available and Who Can Use It

The feature works on three models: Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. Access is available via the Gemini API in Google AI Studio and the Gemini Enterprise Agent Platform. There’s no separate fee for the feature; standard token rates apply. Both uploaded files and YouTube videos accessed via a link are supported. Not yet for regular users. In the Gemini app, the feature is promised “soon,” and on YouTube, via the “Ask YouTube” feature on the watch page, “in the coming months.” Google has not specified any exact dates or countries.

Why is this useful in practice?

Google lists four main scenarios: searching for moments shorter than a second, including state changes that would be lost at one frame per second; searching through hours-long videos with complex queries; detecting anomalies by sampling suspicious segments; and working with long-form content in general. The most obvious use case is something everyone who works remotely faces: a two-hour meeting that no one wants to watch again, or a webinar recording. A lecture where you need just three minutes from the middle. Here, it’s important to consider a technical limitation: on the Gemini Enterprise platform, the maximum video length with audio is about 45 minutes, and without audio, about an hour. So, you won’t be able to “parse a two-hour call” with a single query; you’ll have to split it into parts.

The second category includes anything with a large volume of homogeneous video: video surveillance, quality control in manufacturing, and content moderation. In these cases, processing time savings—often several times over—aren’t just a convenience; they’re the difference between a project “paying off” and “not paying off.”

This isn’t groundbreaking news about a new model—it’s infrastructure optimization. But it’s precisely these kinds of optimizations that most often change how technology is actually used.

When analyzing an hour of video was expensive, it was done selectively. When the cost drops by a factor of three, scenarios emerge that previously didn’t make economic sense.


Sources: Official Google Blog (blog.google), Gemini API documentation (ai.google.dev/gemini-api/docs/video-understanding).