Multimodal model
Also known as: multimodal AI, vision language model
A model that takes more than one kind of input at once, such as frames and sound together, and reasons across them in a single answer.
Updated
What it means
A multimodal model accepts several kinds of input in the same request. For video that usually means frames plus audio plus a question in words.
The point is that the answer accounts for all three at once rather than merging three separate verdicts afterwards, which is what a chain of single purpose detectors has to do.
The difference shows up on moments that only make sense in combination. A shout over a still frame and a jump with no sound are different events, and a Large language model given both channels can tell them apart in one step.
How it works in practice
A highlight pipeline hands it a sampled set of frames, the matching audio and a narrow question. Asking which stretches are strongest gets a usable ranking; asking for a description of the session gets prose nobody can act on.
It replaces several narrow passes at once, which is why one call can stand in for separate Face detection, speech and scene checks when the question is about judgement rather than counting.
Pairing matters mechanically. If frames and audio drift out of alignment the model reasons about a mismatch, so Audio sync has to be right before anything is sent.
What to watch out for
Channels are not weighted the way an editor would expect. Loud audio can dominate a verdict about a visually dull stretch, so a crowd cheering off camera gets ranked above the moment that earned the cheer.
Breadth also means less accuracy on narrow tasks. Counting people or measuring sharpness is better left to purpose built checks such as Blur detection, with the general model reserved for what those cannot answer.
Input limits bite sooner than on text. Frames are expensive, so a long clip has to be sampled hard, and two clips sampled at different rates are not really being compared on equal terms.
How RawClip handles it
Picture and sound are analysed together rather than separately, so a moment that only reads as significant because of what is audible is weighed alongside one that is obvious on screen, in the same pass.