How SafeFrame decides

The problem

Accept a video by upload or URL, decide whether it contains inappropriate visual content, and return a verdict with a confidence. The governing constraint is that all of it runs in the browser: no inference API, no server-side processing, no telemetry. The production target is a mobile browser on hardware that may be several years old.

That constraint is the product, not an obstacle. A kids-safety browser cannot ship children's screen content to a server — it fails on privacy, on legal exposure, on offline reliability, on latency, and on per-user cost. So the architecture treats “no backend” as a property to prove rather than a limitation to route around. There is no database and no server component anywhere in this system. The only things persisted are model weights in the Cache API and preferences in localStorage, both on the user's own machine.

Why image classifiers, not a vision LLM

Deciding whether a frame contains nudity is an image-classification problem, and it has purpose-built open models measured in megabytes. A multimodal LLM can also answer the question, but the smallest usable vision-language models are hundreds of megabytes to gigabytes, take seconds per frame on a phone, and drain a battery doing it. Against the constraints that actually matter here — compute, memory, battery, device variability — they lose on every one.

There is a second, quieter reason. A classifier returns a calibrated probability per class, which is what makes smoothing, hysteresis, thresholds and a tunable precision/recall posture possible. A generative model returns text you then have to parse and cannot easily calibrate. For a safety decision that a parent may need to adjust, the numeric output is the more honest interface.

The pipeline

Six stages, each chosen so the expensive work happens as rarely and as cheaply as possible.

1. 1IngestFile or URL — object URL, never uploaded
2. 2SampleSeek-driven, adaptive fps, densify on hits
3. 3PreprocessDownscale to 224×224 RGB on a canvas
4. 4ClassifyTier 1 always · Tier 2 on borderline
5. 5AggregateEWMA → hysteresis → segments
6. 6RespondVerdict JSON + blur / pause / replace

The two-tier cascade

Every sampled frame is scored by the cheap model. That score routes the frame into one of three bands, and only the middle band costs anything more.

Fallback chains

Nothing in this system has a single point of failure. Four independent chains degrade rather than break, and all four are in the shipped code rather than in a plan for later.

Scoring and the decision layer

A per-frame score is combined from the classifier's classes as p = clamp(porn + hentai + sexy × 0.4). The ×0.4 weight is a deliberate policy choice, not a tuning artefact: it makes it arithmetically impossible for swimwear alone to flag a video at default thresholds, which is the single most common false positive in this domain.

Frame scores then feed two independent decision paths. An EWMA-smoothed series with hysteresis draws segment boundaries — it ignores single-frame spikes, which is exactly what you want for deciding where content starts and stops. But smoothing also damps short events, so a second pass looks at the raw series and treats a sustained run of over-threshold frames as evidence in its own right. Either path can flag a video; both produce segments, so the verdict and the timeline can never disagree on screen.

Device tiering

The same build serves a low-end phone and a high-end desktop. It does that by deciding, before it downloads anything large, what this particular device can actually sustain.

The models

Two open models, chosen for complementary reasons: one small enough to run on everything, one accurate enough to be worth asking when the first is unsure.

Offline behaviour and caching

Model weights are the only meaningful bytes this app moves, so where and when they move is a design decision rather than an implementation detail.

1. 1First scan

   The app shell arrives on the first visit; the detection runtime and the 4 MB screener follow as same-origin chunks when a scan starts. They are part of the app's own build, not a model-hub download, so nothing third-party sits on the cold-start path.

2. 2First borderline frame

   Only now is the ViT requested, and only on mid/high-tier devices. A video that is obviously safe or obviously explicit never triggers it, so most scans cost zero extra bytes.

3. 3Every visit after

   Weights are served from the Cache API and the shell from the service worker. Detection then runs with the radio off — which is the honest test of whether inference is really local.

Alternatives considered

The cascade is the load-bearing decision, so here is what it was chosen over and why.

Responding to a detection

Detection is only half the problem. The recommended default for a child audience is a composite response: blur the frame within one animation tick, pause playback, and cover the stage with a calm replacement card that offers an adult-gated way to continue. Blur alone leaves audio playing. Pause alone leaves the offending frame on screen. Hiding the player entirely reads as breakage to a child. The combination is what avoids all three failures.

Path to production on mobile

The pipeline ports as-is; what changes underneath it is the runtime. Inference moves to ONNX Runtime Mobile, Core ML or NNAPI for a large throughput and battery improvement — the Expo app in this repo already demonstrates that path with on-device ONNX inference and no network access at all.

Beyond the runtime: distil the verifier toward MobileViT or EfficientNet-Lite scale so mid-tier devices can afford it, add heads for violence and gore, and curate a labelled evaluation set so thresholds can be set per age band from measured precision/recall rather than judgement. And in a kids browser, the detector belongs in the engine rather than in a page: hooking the render pipeline removes the CORS problem entirely and scores every media surface uniformly.

Known limitations

• URLs whose host sends no CORS headers cannot be analyzed client-side. A proxy would fix it and would also break the central promise, so the app explains the situation instead. File upload always works, and an in-engine deployment does not have the problem at all.
• DRM and encrypted-media streams are out of reach for any client-side approach — the decoder deliberately yields no readable frames.
• Scope is sexual content. Violence, gore and drug imagery are not detected; that is an additional classifier head on the same pipeline, not a redesign.
• Sampling can miss a sub-second flash between samples. Densification narrows the window but does not close it; cheap scene-change detection is the production mitigation.
• Skin-heavy but safe content — beaches, swimwear, medical footage — remains the hardest negative. The ×0.4 weighting and the verifier both push against it, and a real product needs curated thresholds per age band.
• The live guard is Tier-1 only, by design: it favours latency over accuracy, so it is more prone to false positives than a pre-scan. A pre-scan is the higher-assurance path.
• Audio is not analyzed at all.
• Sustained processing on older hardware throttles thermally, which is why the low tier samples at half rate rather than trying to keep up.