Saliency detection
Also known as: visual saliency, attention prediction
Predicting where a viewer would look inside a frame, used to judge framing and to decide which part of the picture a crop must never cut away.
Updated
What it means
Saliency detection produces a heat map of likely attention rather than a list of objects. It answers where the eye goes, which is not the same question as what is present.
A rider against empty snow is highly salient without any label being involved, and a cluttered frame can contain a dozen results from Object detection with no clear focus at all.
For editing it is most useful as a framing check. A shot whose attention sits hard against one edge is composed badly or has been cropped badly, and either way it reads as uncomfortable.
How it works in practice
The common use is guarding a crop. Reshaping a wide clip for a taller format must not cut the salient region away, which is the constraint Auto-reframe is really working under.
It also helps rank otherwise equal clips. Between two shots of the same moment, the one whose attention lands near a Rule of thirds intersection usually reads better.
Saliency is cheap enough to run on sampled frames, so it fits alongside the other per frame checks rather than needing a pass of its own, which is why it usually sits next to Blur detection in the same stage.
What to watch out for
Models are trained mostly on still images viewed calmly, which is nothing like a fast moving fisheye frame, so predictions on action footage are weaker than the tidy heat maps suggest.
Motion dominates real attention in a way a frame based map cannot capture. A viewer follows the moving subject even when a brighter static object would win on any single frame.
Treating the map as truth also overrides intent. A deliberately off centre composition reads as an error, and correcting it undoes the framing the Camera flyer chose on purpose. Leading the subject into empty frame is a choice, not a mistake.
How RawClip handles it
Framing is judged partly on where attention falls in the frame, so a clip with the subject jammed against an edge ranks below one that is composed with room around it, without anybody reviewing compositions by hand.