Model latency
Also known as: inference latency, response time
How long a model takes to answer one request, which decides whether analysing a long upload is practical or merely theoretical.
Updated
What it means
Model latency is the delay between asking and being answered. On a single question it is barely noticeable, and nobody would design around it.
Across a pipeline that asks thousands of questions about one upload it becomes the entire wait, which is why Inference latency is treated as a design constraint rather than a detail of the hardware.
It is distinct from throughput. A system can be slow per answer yet fast overall by handling many answers at once, and most video analysis is deliberately built to trade the first for the second.
How it works in practice
Grouping requests is the standard remedy, since hardware sitting half idle finishes no sooner than hardware kept busy. That is the reasoning behind Batch processing and behind queueing an order rather than streaming it.
Reducing how much is sent helps more than tuning anything. Fewer and smaller frames mean shorter answers, which follows directly from how Input token counts accumulate on video.
Measuring the whole chain matters more than measuring the model. Decode, resize and transfer all sit inside the same wait and often dominate it, so a faster model can leave the total time essentially unchanged while looking like an improvement on paper.
What to watch out for
Latency is not constant, so a design tuned on a quiet system falls over on a busy one. Two large jobs sharing hardware queue rather than accelerate, and the symptom is waiting rather than any error message.
Chasing it can also degrade results. Cutting the sample rate until answers arrive quickly produces a pipeline that is responsive and blind, having looked at too little footage to rank it sensibly.
Averages hide the cases that matter. A mean response time looks fine while the slowest requests, usually the longest clips, are the ones holding up the whole Smart trimming stage.
How RawClip handles it
Analysis is queued rather than interactive, which means the wait is governed by total throughput instead of the response time of any single request, and the sampling rate is never cut just to make a job feel faster.