GPU inference
Also known as: accelerated inference, graphics card inference
Running a trained model on a graphics processor rather than a general purpose one, which is what makes video analysis practical at all.
Updated
What it means
GPU inference is the practical answer to a volume problem. A model asked about thousands of frames is doing the same arithmetic over and over, which is exactly the shape of work a graphics processor handles in parallel.
The same job on a general purpose processor is not wrong, merely slow enough to change what is possible. An analysis that takes an afternoon stops being a feature and becomes an obstacle.
The distinction matters only for throughput. The model and its answers are identical either way; what changes is how many hours of footage can be looked at before anybody gives up waiting, which is why Inference is usually discussed in terms of hardware.
How it works in practice
Work is fed in groups rather than one frame at a time, because a half empty processor finishes no faster than a full one. That is why analysis is naturally a Batch processing job.
Memory is the binding limit more often than raw speed. A long clip at full resolution does not fit, so frames are downscaled and sampled before they are ever sent across.
Sizing follows the slowest stage rather than the model. If decoding cannot keep the card fed, a faster card changes nothing, and the money goes on the wrong part of the machine.
What to watch out for
Moving data takes longer than people expect. Decoding video and copying frames across can dominate a run, so a pipeline that looks model bound is often really busy with Transcoding and shuffling bytes.
Capacity caps concurrency in a way that shows up as waiting rather than as an error. Two large jobs on one card do not run twice as fast; they queue, and the visible symptom is a jump in Model latency.
Hardware also drifts out from under a pipeline. Driver and library versions change numerical behaviour slightly, so results are reproducible in practice but rarely bit for bit identical across upgrades.
How RawClip handles it
Analysis runs on dedicated hardware rather than shared general purpose machines, which is what allows a multi hour upload to be examined closely instead of skimmed. The hardware changes the throughput, never the judgement.