Gemini
Also known as: Google Gemini
A family of multimodal models from Google that can read video, audio and text together, used to judge what is happening inside footage.
Updated
What it means
Gemini is a family of models rather than one product, and the trait that matters for editing is that it takes video and audio directly instead of requiring frames to be described first.
Asked what happens in a clip, it can answer in a sentence. That collapses several specialised detectors into one call, which changes how an analysis pipeline is put together rather than merely how fast it runs.
The generality is the appeal and also the caveat. A purpose built detector beats it on a narrow task such as counting faces, while it wins on the questions nobody wrote a detector for, like whether a moment reads as a peak.
How it works in practice
In a highlight pipeline it answers judgement questions rather than measurement ones. Motion and brightness are better measured arithmetically, so the model is asked which stretches are worth keeping and why.
Those answers then feed Shot scoring, which combines them with the measured signals instead of trusting either on its own. Neither source is reliable enough alone.
Input is prepared rather than dumped. Frames chosen by Frame sampling cover a clip at a fraction of the data, which keeps each answer affordable and fast enough to run across a whole upload.
What to watch out for
It will answer confidently about things that are not in the footage, so output needs sanity checks against measurable signals. An asserted jump with no corresponding motion spike is a Hallucination, not a discovery.
Behaviour also shifts between model versions. A prompt tuned on one release can score footage differently on the next, so a pipeline depending on specific phrasing needs pinning and periodic re-checking.
Long material still has to be divided up, because even a generous Context window does not hold hours of frames, and anything compared across chunk boundaries is compared indirectly.
How RawClip handles it
Footage is analysed by a Gemini model, which is why the system can tell a genuine peak from mere movement. Measurable things are still measured directly, and the model is asked only the questions arithmetic cannot answer.