Prompt engineering
Also known as: prompt design, prompt tuning
The practical work of wording a model instruction so it answers the intended question reliably, on footage it has never seen.
Updated
What it means
Prompt engineering is iteration on the instruction rather than on the model. The model stays fixed; what changes is the question, the stated rules and the shape of the answer demanded.
On video it has an unusual property: the same Prompt can behave well on bright bike footage and badly on a dim dive, so tuning has to happen across the range of material, not on one favourite clip.
The honest description is empirical. There is little theory to apply, so progress comes from running the same set of clips through variations and keeping what ranks them the way a person would.
How it works in practice
A fixed evaluation set is the whole discipline. A few dozen clips with known correct answers, spanning everything from a bright Park lap to a dim dive, turn tuning from an argument into a measurement, and without one every change feels like an improvement.
Demanding structured output early saves repeated work later, because a parseable answer can be scored automatically against the known result instead of read by hand. That turns a tuning round into a few seconds of arithmetic.
Giving the model an explicit escape helps accuracy. Allowing it to answer that nothing here stands out stops it inventing a peak in footage that genuinely has none, which is the cheapest guard against a Hallucination there is.
What to watch out for
Overfitting to the evaluation set is the classic failure. Wording tuned until it is perfect on forty clips can be worse than the first draft on the forty first.
Tuning also drifts out of date silently. A model update changes behaviour without changing the instruction, so results degrade with nothing in the repository having been touched and nothing in the history to explain it.
Prompts can also paper over a missing measurement. Asking a model how shaky a clip is works less well than measuring it, which is what Shake detection exists to do.
How RawClip handles it
The instructions behind the analysis are tuned against a fixed set of real uploads rather than by eye, and the model is allowed to say that a stretch is unremarkable instead of being pushed to find something.