Object detection
Also known as: object recognition, bounding box detection
Finding what is in a frame and where, returning a labelled box around each thing rather than one label for the whole picture.
Updated
What it means
Object detection reports both identity and position. A frame comes back as a list: a person here, a bicycle there, a board in the corner, each inside a box with a certainty attached.
That is more than classification, which gives a single label for the whole frame, and less than segmentation, which traces exact outlines instead of rectangles.
Position is what makes it useful for editing. Knowing a rider is in the frame is mildly interesting; knowing the rider occupies the left third is what lets a system judge framing and feed Auto-reframe.
How it works in practice
In a highlight pipeline it answers presence and prominence. A clip where a person fills a quarter of the frame is weighted above one where somebody is a distant speck on a slope.
Boxes across successive frames are also the raw material for a path, which is what Object tracking stitches together from otherwise independent detections.
Detections come with a certainty, so a working threshold decides what counts. Anything below it is treated as absent, which is what a Confidence score is for at this stage, and the threshold is usually set per category rather than once for everything.
What to watch out for
Action footage breaks the usual assumptions. Detectors are trained largely on upright, unobstructed, slow moving subjects, and a fisheye frame of a rider mid rotation is none of those.
Recall therefore drops exactly where the action is, and the drop is invisible in any overall accuracy figure measured on ordinary video.
Labels are also coarser than they look. A snowboard, a wakeboard and a skateboard may all come back as one category, so anything built on the label alone misreads the sport that Activity classification would have named correctly. The box is reliable even when the label is not, which is why position is the part worth trusting.
How RawClip handles it
Clips are weighed partly on whether a subject is present and how much of the frame it occupies, so an empty slope ranks below the same slope with a rider on it, without anybody tagging the footage by hand.