Real-Time Hand-Sign Recognition Backend

A Python backend that turns a webcam stream into a stable numeric prediction. MediaPipe's Hand Landmarker locates 21 landmarks on a single hand, those coordinates become a 63-value feature vector, a scikit-learn model classifies the sign, and a mode-based smoothing window converts noisy per-frame output into a prediction that is steady enough to act on. Both the raw and the stabilised prediction are exposed, so the consumer can choose responsiveness or reliability.

Per-frame classification of a live video stream is unusable straight out of the model. A hand in motion, a moment of partial occlusion, or a single blurred frame produces a prediction that flickers between neighbouring classes several times a second.

The recognition problem was largely solved by the landmark model. The engineering problem was turning a sequence of individually plausible guesses into one answer a consumer can act on.

The Hand Landmarker returns 21 points for a detected hand, each with three coordinates. Flattened, that is a fixed 63-value vector per frame — small enough for a classical model to classify in well under the frame budget, and stable enough in structure that no per-frame reshaping is needed.

Using landmarks rather than raw pixels is what makes the model this cheap. The expensive perception work is done by the landmarker; the classifier only has to separate ten shapes in a low-dimensional space.

The raw prediction is the classifier's opinion about the current frame. It is immediate and it flickers. The stable prediction is the mode of the last ten raw predictions: a single frame of noise cannot move it, but a genuine change of sign moves it once the majority of the window agrees.

Both values are returned. A consumer that wants responsiveness — a live debug overlay, for example — can read the raw value; a consumer that will take an action reads the stable one. Choosing for the consumer would have been the wrong call, because the right trade-off depends on what the prediction is used for.

Each stage — capture, detection, feature extraction, classification, smoothing, API — is a separate module with a narrow interface. The practical benefit showed up during tuning: the smoothing window could be changed, and the classifier swapped, without touching capture or detection.

The current version feeds landmark coordinates to the classifier without first translating them relative to the wrist. The model therefore sees a sign made at the left of the frame as different from the same sign made at the right, and has to spend capacity learning that they are equivalent.

This is a real weakness, not a stylistic one: it makes accuracy depend on where the hand sits in frame and on distance from the camera. The fix is to translate all landmarks into a wrist-anchored frame of reference and scale by hand size before classification, which should improve generalisation without changing the pipeline's shape.