Machine learning in video
Also known as: video machine learning, learned video processing
The broad use of trained models on footage, from stabilisation and noise reduction through to deciding which moments are worth keeping.
Updated
What it means
Machine learning in video is the umbrella rather than any one feature. It covers the quiet uses that already live inside cameras and editors, such as Denoise and upscaling, and the louder ones that choose what a viewer sees.
The common thread is that behaviour was learned from examples instead of written as rules. Nobody specified what a jump looks like; a model was shown many and inferred the rest.
The split worth keeping in mind is between fixing and deciding. Fixing a frame is reversible and easy to judge, while deciding what to discard is neither, and it carries the taste of whoever supplied the examples.
How it works in practice
Most pipelines layer cheap learned checks under expensive ones. Sharpness and exposure are settled first, then a smaller surviving set goes to something that reasons about content, which is the ordering Frame sampling serves.
Hybrid designs work better than purely learned ones. Measuring what can be measured and learning only what cannot keeps results explainable when something goes wrong, which matters when the output is shown to a customer.
Evaluation has to be on real footage rather than benchmarks. A model that scores well on tidy clips can fail on a fisheye frame of spray, and only the second case tells you anything useful.
What to watch out for
Training data decides behaviour on edge cases, and action footage is mostly edge cases. Fisheye frames, powder, near darkness and extreme motion are underrepresented almost everywhere.
Accuracy therefore drops exactly where the footage is most interesting, which is the opposite of what anybody wants and is invisible in aggregate accuracy numbers.
Learned processing also fabricates detail when pushed. AI upscaling and aggressive sharpening invent plausible texture, which looks fine in isolation and wrong beside an untouched shot in the same Final cut. The invented texture is also unstable between frames, so it shimmers in movement even when each still frame looks convincing on its own.
How RawClip handles it
Learned models choose which moments survive and arithmetic handles what it can measure, so the parts of the decision that are matters of fact stay checkable while only the matters of judgement are left to a model.