Every number we measured

Accuracy: what 27.4% is made of

We scored the shipped pipeline against human transcripts from the AMI meeting corpus: real recorded meetings, annotated by people. The mean word error rate was 27.4%.

Quoted alone that number misleads, and it misleads against us. Word error rate adds up three quite different failures, and ours are wildly unequal.

Deletions are the majority of the total. A deletion is a word the model never produced, and in four-way meeting audio that usually means one thing: somebody else was talking at the same moment. Whisper writes a single line of text. When two people speak at once it can't write both, so every overlapped word counts against it. In two of these meetings, around 29% of the reference words overlap another speaker.

We stopped asserting that and measured it. Using the AMI annotations we tagged every reference word with whether another speaker was talking over it, then asked which words the model missed. Overlapped words are missed 38.1% of the time; words spoken in the clear, 8.9%. So of the 27.4 points, 11.4 are errors on overlapping speech that no single-line transcript can avoid, and 16.0 are ours. We publish both because only the second number can be improved, and quoting the first alongside it is how a tool flatters itself.

Mishearings are 7.8 points. That's the number most people picture when they hear "word error rate", and it's the one that predicts how much correcting you'll do on clean audio.

Work the components back and about 87% of the words Ratproof puts on the page match the human transcript. You can check that from the table rather than taking it from us: correct words are 100 minus substitutions minus deletions, and words emitted are that figure plus substitutions plus insertions.

Two qualifications, one each way

In our favour: AMI is brutal by design, which is why researchers use it. Cleaner audio with one speaker at a time does better, and microphone placement matters more than people expect. The same meeting, same pipeline, scored 28.5% on a headset and 37.1% on a far-field room array: 8.6 points, from nothing but where the mic sat. Invented words more than double across that gap, 4.7 points to 8.6.

Against us: we used to stop at that paragraph, which was too comfortable. When we measured the overlap properly, the meeting with the least overlap (12.4%) scored worse overall than the one with the most, because its insertion rate was 4.5 times higher. Insertions are invented words and no scoring artefact explains them away. Deletions are partly an artefact of how the reference is built. Insertions are entirely real.

It sometimes invents text, and tells you where

Given silence or noise, Whisper will occasionally produce a fluent, grammatical, completely fabricated sentence. That's how the model generates text, not a bug awaiting a patch, and it's equally true of the Whisper running inside products that never mention it. The question isn't whether a tool hallucinates. It's whether the tool admits it in the file it hands you.

• Silence never reaches the model. A voice activity detector runs first, so the model only sees stretches where somebody is speaking. On a meeting whose first real word lands at 55 seconds, this replaced an invented line at 15 seconds with a real one at 22.7.
• Repetition loops get flagged, never edited. Whisper can get stuck repeating a phrase. In one test meeting it emitted the same word 218 times in a row, which by itself accounts for nearly that meeting's whole invented-word rate. Two checks look for it: text that compresses abnormally well, and any eight-word run that recurs within sixty words. Findings appear above the transcript with a link into the audio, and as a header block in the text, Markdown, VTT and JSON exports. The transcript body is never altered. You get told, not edited.
• What the checks miss. The compression check flagged eight passages across three meetings with no false positives, but its recall is poor: on seven windows confirmed to contain fabricated repeats, it fired on none. The eight-word rule catches those, at a cost: run against human transcripts, where any flag is wrong by definition, it produced two false positives in 7,205 words — roughly one every 24 minutes of speech. That's still the same three meetings, and a deposition or a speaker with a verbal tic is a population we haven't tested.
• The residual is real. Between the point the detector first hears a voice and the point the human annotator recorded the first word, there's a window of plausible but unannotated text. Nothing flags it. The detector hears speech there and the text doesn't repeat, so neither check has anything to fire on. We know the window exists. We haven't closed it.

A feature we deleted for being wrong twice

An earlier version marked text sitting mostly outside detected speech, on the theory it was probably invented. Measured on a 187-segment meeting it fired twice, and both passages turned out to contain real speech. It wasn't detecting invention. It was detecting low speech density, and labelling honest transcription as suspect.

We removed it rather than keep it for the reassurance it offered, then removed the leftover fields from the JSON export too, because a machine-readable file was still carrying "uncertain": true without any of the interface's context. A flag pointing at the wrong thing is worse than no flag.

The hardware bill, exactly

• About 2 GB of memory, or about 3 GB using your graphics card. Measured across the whole browser on a 57-minute recording: 2,214 MB on the processor, 3,229 MB on the graphics card — which is the faster path, and the default where your hardware supports it. That extra gigabyte is the model sitting in your graphics card rather than in the page. The cost does not grow with length: the 57-minute recording used slightly less than a 28.6-minute one measured earlier, so it is the model that sets the floor, not your audio. An earlier version of this page said length adds roughly 11 MB per minute; measuring a file twice as long found no such increase. There's no useful length limit for us to impose, and phones generally won't manage it. The app warns you when your browser reports a low-memory device, but that reading only exists in Chrome and Edge and is coarse even there, so no warning isn't a clearance. If the tab does run out, the browser kills it and the page reloads showing nothing, so the app leaves a marker beforehand and explains what happened on the next load.
• 11 to 50 minutes per hour of audio. That range isn't hedging, it's the measured spread, and it's bimodal rather than even. We traced it to the power and turbo state of the processor: a warm laptop, or battery power, or a busy machine, lands in the slow mode and stays there for the session. We can't detect that in advance, so we won't pretend to a better estimate.
• About 235 MB on the first visit. The model, runtime and voice detector, downloaded once and cached. A second visit measured 0.0 MB over the wire. Roughly 40 seconds on 50 Mbps, about three minutes on 10 Mbps. On a cold profile time to first text was 14.6 seconds, against a 10-second target.

Smaller quantized variants could cut that download to around 110 MB. We haven't shipped them, because a smaller download that transcribes worse isn't an improvement and we haven't re-run the accuracy measurements on them.

The summary is generated text, and not everyone gets it

Two different things sit above your transcript, and the difference matters more than either feature does. Key passages are quotes: lines lifted out of your own transcript, unedited, with the time they were said. The summary is written by a language model, which means it produces sentences nobody in your recording said. That is what summarising is. We do not blur the two — they sit in separate boxes, they are labelled differently in every file you export, and the summary carries a line saying software wrote it.

Every summary line is matched back to the passage of your transcript it most resembles, and that match is arithmetic we do, not something we ask the model for — a model willing to invent a fact is equally willing to invent the timestamp it came from. When nothing in your recording supports a line, it is marked no match rather than given a plausible-looking time. Words the model used that appear nowhere in your transcript count against the match rather than being ignored.

The model belongs to your browser, not to us. Chrome and Edge ship one; Firefox and Safari do not. It also wants about 22 GB of free disk and either a decent graphics card or 16 GB of memory. On the machines that fail those tests — which is most of them, and disproportionately the locked-down work laptops this tool is for — the button is not offered and the key passages are what you get. We would rather say that than promise a feature most visitors cannot run.

What we have not measured yet: how long a summary takes on a machine where the model actually runs, and how often the "no match" check refuses a line it should have kept. When we have those numbers they will be on this page.

Three things that aren't there yet

No speaker labels. You get a timestamped transcript with no indication of who's talking. Speaker separation is built and measured but not in the product. If you need "Speaker 1 / Speaker 2" today, this isn't your tool yet.

Not fully offline. The model weights are cached and aren't fetched again, but the page still has to load and there's no service worker. Nothing you transcribe leaves your machine, which is the claim we make. "Works with the network unplugged" is a different claim and we're not entitled to it.

English only. The model underneath is multilingual, but we hand it a fixed English language token on every recording, so a file in another language is decoded as though it were English and comes back wrong. That started as an accident rather than a decision: the library we transcribe through has language detection stubbed out and quietly defaults to English, announced only in a console message. It's now stated in the code and shown in the app beside the model name. Real language support needs a picker and a fresh accuracy run, and neither exists yet.

What it isn't for

Ratproof transcribes a recording you already have. It doesn't join meetings, doesn't record for you, and takes no position on whether the people on your recording agreed to be recorded. That part is yours, and it's where most of this industry's privacy trouble has come from.

We make no compliance claims either. Nothing is certified against HIPAA, GDPR or any other regime. A tool that keeps audio on your machine is a helpful fact for that assessment, not a substitute for one.

Why we publish all this

Partly because you can't choose well without it. Mostly because the alternative is what everyone else does, and it's the weaker position: a page of adjectives, no benchmark, no error rate, and a model that hallucinates exactly as much as ours without ever saying so.

Every number above was measured on the build you can open right now. Not estimated, not borrowed from a model card. If it changes, this page changes.