probe-factory · private beta · design partners open

Beat Llama Guard on quality. For 1/122 the cost.

probe-factory reads your self-hosted model's own internals to flag misuse, with higher AUROC and fewer false positives than Llama Guard 3 8B, at one matmul on activations you already computed. Because it runs during prefill, it decides before the first output token exists. An output guard can't.

Become a design partner → or read the full results memo

1/122 the cost

Beats Llama Guard 3 8B on detection quality and false positives, at 122× lower cost and 135× lower latency. It's one matmul on activations you already computed.

Token 0 decides at prefill

Earlier than any output guard can manage, since it has to generate the harmful text before it can judge it. Median 87× faster; 94% of harmful prompts caught before a single token ships.

0.978 AUROC · beats the guard

Higher detection quality and a lower false-positive rate than Llama Guard 3 8B (0.975). It isn't the cheap option that gives something up. It's the better one that also happens to cost 1/122 as much.

What it is

A white-box safety sidecar for open-weight models you run yourself.

If you run your own model, whether for cost, latency, data residency, or because you fine-tuned it, safety is your job, and the tools you're handed are text-level guard models: a second forward pass that reads the prompt or the response. probe-factory instead reads the activations your model already produced. Intent shows up there before it shows up in the output, so a linear probe catches it earlier, cheaper, and, on our data, more accurately than the guard model everyone already knows.

The approach is proven: Google DeepMind ships production probes for Gemini; Goodfire deployed probes at Rakuten at 10–500× lower cost than an LLM judge. What no one has built is the part that keeps a probe valid once it's deployed against weights that never stop changing.

Where it runs — and where it can't

One requirement: you control the model's forward pass.

A probe reads the model's own activations, so it works wherever you — or your provider — control the process the model runs in, and nowhere you don't. Plainly:

Works

Own or rented GPUs — you run the serving stack.

On-prem, or a box you rent on RunPod, Lambda, CoreWeave, or an AWS/GCP GPU VM. Renting the GPU is the same as owning it: you control the process, so the probe taps it inline.

Works

Bring-your-own-container serverless.

Baseten, Modal, Replicate (custom images), RunPod serverless. You ship the container that runs the model, so the probe ships inside it — no provider buy-in needed.

Needs the provider

Fixed managed endpoints.

Fireworks, Together's shared endpoints, AWS Bedrock, Vertex. You get text in and text out; only the provider can reach activations. Here we're a feature they integrate and resell — a partnership, not a self-serve install.

Can't

Closed model APIs.

OpenAI, Anthropic, Gemini. No access to internals, ever — which is exactly why we build for open weights.

"Self-hosted" undersells it: the market isn't just companies with their own racks — it's anyone who controls an open-weight forward pass, which includes the large and growing crowd renting GPUs to serve their own fine-tuned models.

What you get

A calibrated probe, kept valid as your model evolves.

A probe is calibrated to a specific set of weights, so as you fine-tune and upgrade, we keep yours current. You never operate a stale one, and you never touch the machinery. It ships as a managed lifecycle, not a library you maintain.

Certify

Fit and calibrate a probe against your actual weights.

Tuned to your model and your traffic, and benchmarked head-to-head against the guard models you'd otherwise run.

Keep current

Re-calibrated on every fine-tune, merge, and version bump.

Included and automatic. You ship model changes on your schedule; the safety layer stays in lockstep without you managing it.

Harden

New obfuscations closed as they emerge.

Character-level tricks and fresh jailbreaks get folded into your probe, with no regression on clean traffic.

Evidence

Audit log and evidence export.

Every verdict, threshold, and re-certification, exportable. The compliance artefact that turns a security tool into a budgeted line item.

Straight about the limits

Our headline table trains the probe and runs the guards zero-shot, so it isn't an apples-to-apples generalization claim, and we say so wherever we show it. We haven't yet tested the guards under attack, cross-model transfer, or a real buried-in-long-context case. If your model catches us overclaiming, that's the conversation we want. The full memo lists every gap as plainly as every win.

Who we're looking for

Two boxes: you run the model yourself, and you fine-tune.

Regulated & privacy-bound

Fintech, health, legal, gov. You run it yourself because sensitive traffic can't leave, and compliance already owns a budget for evidence.

AI-native at scale

Companions, agents, coaches, high-volume assistants. You fine-tune constantly and feel every extra forward pass in the GPU bill.

Inference & serving providers

RunPod, Baseten, Fireworks, Together. A white-box safety add-on you could resell to the open-weight workloads you already host.

Platform / ML-infra teams

You operate open weights internally and you're the one who finds out what breaks on a version bump. Usually the hard way.

For investors Why the maintenance is the moat

The failure case is the better business.

A linear probe is ~200 lines and DeepMind published the recipe, so any competent ML team builds one in a week. What they cannot cheaply do is keep it valid. We measured where it goes stale, and each failure a customer can't self-detect is recurring, ops-shaped revenue.

Fine-tuning silently breaks it, and we measured it. A benign, general-purpose LoRA cut recall from 95% to 42% with no error and no alert; refit restores it to 0.97. The customer cannot see this happen, which is why re-certification recurs.
Quantization does not break it. By contrast, int8/int4 preserve the probe almost exactly, so we refuse to sell re-cert for it. Being honest about what doesn't need us is what makes the rest credible.
Obfuscation is a live hole with a paid fix. Leetspeak drops the base probe to chance; adversarial re-cert recovers most of it, and new encodings keep arriving.
The open question we're not hiding: whether re-certification reads to buyers as a valued service or a tax on a fragile approach. Customer discovery is settling that now, and we won't let the moat framing paper over it.

Full numbers: the internal results memo, reproducible for under $1.

Running an open-weight model in production? Let's benchmark it.

We'll fit a probe to your weights and put it head-to-head with whatever you run today (cost, latency, false positives) on a non-prod model, in about an afternoon. Worst case, you walk away with a free read on your current safety stack.

Start a pilot conversation →