Large language model
Also known as: LLM, language model
A model trained to predict text that can also be given frames and sound, which is how it ends up making editing judgements.
Updated
What it means
A large language model learns to continue text, and that turns out to generalise further than the description suggests. Given a question it produces an answer; given frames alongside the question it answers about those frames.
That is the route by which a text model ends up reasoning about footage, and why the same family of models powers both captioning and moment selection in a Multimodal model setup.
The relevant property for editing is that it handles open questions. There is no detector for whether a stretch reads as the climax of a Story arc, but that can be asked in words and answered usefully.
How it works in practice
It is used for judgement, not measurement. Brightness, motion and tempo are better computed arithmetically, so the model is handed the questions that resist arithmetic and its answers feed Shot scoring.
Asking for structured output keeps it usable. A ranked list of stretches with reasons can be parsed and checked; a paragraph of prose about the session cannot be acted on by anything downstream.
Narrow questions also behave better than broad ones. Five candidates to rank produces a steadier answer than an hour of footage to summarise, and the ranking is what the cut actually needs.
What to watch out for
Fluency is not accuracy, and the failure mode is a confident invented detail rather than an obvious error. That makes Hallucination the first thing to design against when model output can influence a cut.
Output also drifts with version and phrasing. A prompt that ranked footage sensibly last month can rank it differently after an update, so anything depending on exact wording needs pinning and re-checking.
Reasons are not explanations. Asked why it picked a moment, the model will supply a plausible justification regardless of what actually drove the choice, so the reason cannot be used as evidence.
How RawClip handles it
A multimodal model is asked which stretches of an upload are worth keeping and why, and its reasons are checked against measured motion and sound, so judgement comes from the model and facts come from the footage.