referee-hands-s

A 1.5M-parameter MobileNetV3-small classifier for hand/gripper visibility in robot-training footage. 96px inputs, sized for microcontroller-class deployment.

The published 97.8% result is from a 45-clip, in-domain development sheet that was repeatedly consulted during development and has not yet passed the declared independent two-grader gate. It is reproducible development evidence, not a frozen external benchmark.

Benchmark

97.8% clip accuracy on referee-lab/hands-visible (45-clip provisional development sheet, robot + wildlife domains) — matching gemma3:27b on that sheet at 1/18,000th the parameters.

Model	Params	Accuracy
gemma3:27b	27B	97.8%
referee-hands-s	1.5M	97.8%
referee-hands-s, cross-video split	1.5M	95.6%
qwen2.5vl:32b	32B	74.4%
constant-answer floor	—	68.9%

The cross-video row keeps every training frame at least 60 seconds away from any test clip on its source stream. Under that separation the model scores 100% on real footage (41/41); its two misses are synthetic drawn-hand control clips outside the training distribution.

Intended use and scope

Version 1 targets fixed-camera and wrist-camera robot workspace footage — the domain it was trained on (DROID, ALOHA-sim, KABR negatives). Human egocentric and general webcam scenes are outside the v1 training distribution; a mixed-domain v2 is in development. Use for capture-pipeline gating in the supported domain.

Usage

import torch
from torchvision import models, transforms
from safetensors.torch import load_file

net = models.mobilenet_v3_small()
net.classifier[3] = torch.nn.Linear(net.classifier[3].in_features, 1)
net.load_state_dict(load_file("model.safetensors"))
net.eval()
tf = transforms.Compose([transforms.Resize((96, 96)), transforms.ToTensor()])
p = torch.sigmoid(net(tf(img).unsqueeze(0)).squeeze())   # P(hand/gripper visible)

Clip-level verdict: sample ~12 frames, yes if max(p) > 0.5.

Training: distilled from gemma3:27b pseudo-labels on CC BY / CC0 / MIT media. Benchmark, gates, and tooling: https://github.com/publu/referee-lab