Vision-language model
Also known as: VLM, image language model
A model trained on pictures and text together, so it can be asked in words what is happening in a frame and answer in words.
Updated
What it means
A vision-language model connects two kinds of input that used to need separate systems. Frames and a question go in together, and the answer is ordinary language rather than a list of boxes.
That is what makes open ended questions possible. There is no detector for whether a frame looks like the peak of a run, but a model that has seen pictures described in words can be asked.
It is the component underneath most video analysis that behaves like understanding rather than measurement, and a Multimodal model is the same idea extended to audio as well.
How it works in practice
In a pipeline it is given sampled frames and a narrow question. Asked to rank five candidate stretches it produces something usable; asked to describe a session it produces prose nothing downstream can consume.
It is poor value for anything countable. Measuring sharpness or counting faces is better done by Blur detection and a dedicated detector, with the model reserved for judgement.
Frames chosen deliberately beat frames taken at intervals, because a blurred or empty frame wastes one of the few looks the model gets at that clip, which is why Automatic keyframe selection sits in front of it.
What to watch out for
It describes what is plausible as readily as what is present, which is the standard route to a Hallucination, and the answer carries no signal that it has happened.
Action footage sits outside most training data. Fisheye distortion, spray, extreme angles and near darkness all degrade accuracy exactly where the interesting footage is.
Answers also shift with model version. Wording that produced a sensible ranking on one release can rank differently on the next, so behaviour has to be re-checked rather than assumed stable, and the instruction pinned to the release it was tuned against.
How RawClip handles it
A model that reads frames and sound together is asked which stretches of an upload are worth keeping, and every claim it makes is checked against measured motion and audio before it can influence an edit point. Judgement comes from the model, facts from the footage.