# Reading silent speech from a webcam: what held up when I checked Elliott Murray · November 2026 · [Code](https://github.com/devoided/silent-speech-eval) ## Summary I dictate most of what I write, using a local macOS dictation app I built for my own use. In November 2026 I tried to make that work without speaking: mouthing words at a webcam, with no sound, and having the computer type them. I fine-tuned an open lip-reading model on twelve minutes of video of my own mouth, then had Claude pick the most likely sentence from the model's ten best guesses. On 15 test sentences, that cut the share of mouthed words read wrong from 53%, for the published model on its own, to about 10%, with 9 or 10 of the 15 sentences exactly right. The goal wasn't a new lip-reading method — lipflow, the open project I built on, already uses the same pipeline — and 15 sentences from one speaker in one sitting is a small test. The useful part was what happened when I checked the numbers: 1. An evaluation run inside an AI coding agent reported 13 of 15 sentences right. It had seen the test items. Re-run blind, it was 9 or 10 at best. 2. The model's built-in language model had memorised the standard test sentences. 3. Two training runs with identical settings disagreed by more than the differences I was trying to measure. 4. Fitting a model to one person didn't carry over to other people. I built and ran the experiments on Modal using Claude Code, keeping a lab notebook of every run, including the ones that failed. ## Setup The first step, before recording anything, was a best-case estimate for each possible sensor. It used an open physical model of the mouth (CHIMERA) and the text of 5,936 of my past dictations. A perfect sensor of jaw movement, which is roughly what earbud motion sensors could give, would recover at most 67–90% of words. A camera that sees the lips and jaw could reach 95–97%. That pointed at the camera. There was one recording session: 30 sentences, each said four ways (aloud, whispered, as faint a whisper as possible, and mouthed with no sound), plus ten command words. That came to 240 takes and about twelve minutes of video. Fifteen of the sentences were Harvard sentences ("The birch canoe slid on the smooth planks"), a standard test set. The other fifteen were written to sound like my everyday dictation ("Add a button that copies the link to the clipboard"). The lip reader was Auto-AVSR's published model. It was fine-tuned on crops of the mouth with lipflow's training code, on rented GPUs. Thirty sentences leave nothing spare for a separate test set. So every recipe was trained five times, each time holding back a different six sentences in all four manners and testing only on those. That way every sentence was scored by a model that had never seen it.
A real test sentence, start to finish, with wrong words in red. Claude saw neither the video nor the answer, only the ten guesses.
## Results Words wrong on the 15 sentences in my own style, mouthed with no sound: | | Lip reader alone | With Claude | |---|---:|---:| | Published model, not fitted to me | 53% | 32% (5 of 15 exact) | | lipflow's own recipe (24 mouthed sentences) | 30% | 19–⁠20% (7 of 15) | | Fitted on all 12 minutes, four runs averaged | 24% | 9–⁠11% (9–⁠10 of 15) | "With Claude" means Claude Fable 5.2 choosing from the lip reader's ten best guesses, through the API. It got one sentence per request, with only the ten guesses and a short note on how lip readers go wrong. Ranges are repeats of the same request. Claude Opus 5.5 did a little worse (13–14% of words wrong), and longer thinking didn't reliably help either model. Two caveats. The test has only 160 words, so one sentence read completely wrong moves the figure by about six points. And the lipflow row is a single training run. As section 3 shows, single runs vary by more than the gap between the last two rows. ## What didn't hold up ### 1. A 13 out of 15 from a leaky evaluation The first time Claude read the guesses, it was inside a Claude Code session. The agent (Claude Opus 5.5) was given a blind file: the lip reader's guesses for all fifteen test sentences, shuffled, under made-up IDs, with the answer key kept elsewhere. It worked out each sentence and got 13 of 15 exactly right (7% of words wrong). The file was blind, but the setup wasn't. The session saw all fifteen items at once, so it could use one to inform another, which never happens in real use. And its instructions included three examples of words the lip reader had confused, two of them taken from sentences in the test. So the reading was redone through the API. Each sentence got its own request with nothing else in it, and the examples in the instructions were invented look-alikes ("bat" for "mat"). Measured this way, on the same lip reader's guesses, Opus 5.5 got 7 or 8 sentences right instead of 13. Claude Fable 5.2, the best reader I tried, got 9 or 10 in four repeats.
The same lip reader's guesses, scored two ways. Leaky: one session saw all fifteen test sentences at once, and two of the three examples in its instructions came from the test. Blind: one sentence per API request, with invented examples. Pale dots were right in some repeats but not others; in both blind rows, that's the sentence in Figure 1.
Nothing dishonest happened; the setup just let the number drift upward. The lesson I took is that a figure is only as good as the way it was made, and it should be measured the way it will be used. ### 2. Test sentences the model had memorised The lip reader includes a language model that steers it towards likely sentences. Its perplexity on the 15 Harvard sentences was 4.9; on my 15 sentences it was 83.5. In plain terms, it was about 17 times less surprised by the Harvard sentences. They are among the most reprinted sentences in English, so it had very likely seen them in training. Any score that mixed the two halves flattered the model, and more so once a second language model (Claude) was choosing between guesses. From then on, only my own sentences counted towards headline figures. ### 3. One training run is a noisy measurement The same recipe was trained twice with identical settings, down to the random seed. It got 24.9% and then 30.2% of mouthed words wrong across all 30 sentences. On my 15 sentences it was 26.2% and then 40.0%. Training on a GPU isn't exactly repeatable, and with this little data the small differences compound. Most of the differences between the recipes I was comparing were smaller than that. So the recipe was trained from four different random seeds, and the four models' weights were also averaged into one (sometimes called a "model soup"). The four runs on their own scored 23.1–30.2%. The averaged model scored 21.4%, better than any single run, and costs nothing extra to use.
Trained twice with identical settings, the same recipe landed 5.3 points apart. The bar spans four more runs from other seeds; averaging their weights into one model beat every one of them.
### 4. Fitting a model to one person made it worse for others Twelve minutes of video can't say much about how this scales. So I used SoniSpeech, a public research dataset from Cornell with 26.7 hours of one person mouthing and speaking 9,000 sentences. On 250 held-out mouthed sentences, fitting the model on 2.4 hours of their mouthing cut the words wrong from 74% to 20%, and 12 hours cut them to 12%. With Claude choosing from the top 10 guesses, the 12-hour model got 6% wrong, and 62 of the first 100 sentences exactly right. Then the same models were tested on people they hadn't seen: - On 42 speakers from TaL80, an Edinburgh dataset, the published model got 84% of mouthed words wrong. The model fitted to the Cornell speaker got 93%, and the one fitted to me got 81%. (TaL80 films the lips from a helmet camera, so treat these as relative.) - On my mouthing (all 30 sentences), models fitted to the Cornell speaker got 48–70% wrong, against 50% for the published model. - My own fitted model got 91% of the Cornell speaker's mouthed words wrong, against 74% for the published one.
Fitting helped a lot on the person a model was fitted to. On anyone else it helped little or made things worse. The Cornell-fitted model is the 2.4-hour one; my figures here are across all 30 sentences, and on my mouthing the bar spans several models fitted to the Cornell speaker.
Nothing learned from one person clearly carried over to anyone else. There was also a quieter effect inside the Cornell data. Nine minutes of mouthing from recording sessions close to the test sessions gave 42% wrong, against 54% for nine minutes from earlier sessions. The likely cause is that recording conditions drift over time. It's a reminder that "held out" doesn't mean "independent". ### 5. A check that did hold up Before the lip reading, the local dictation app needed its own small model for cleaning up transcripts (0.6B parameters, fine-tuned on 7,615 examples). On 135 held-out dictations, it left 3.8% of words wrong against 5.9% for the model it replaced. But those dictations came from the same month as the training data, and the number of training passes had been chosen by scoring on them. So I asked whether it was overfitted. The answer was a test it couldn't have been tuned on: 120 recordings from eleven months earlier. It held up, at 3.2% against 4.3%. It is still one speaker. ## Ideas the data killed - **A small local model trained on simulated mistakes.** I'd hoped a small model could replace Claude for choosing between guesses. It was trained on 8,090 of my dictated sentences, corrupted with simulated lip-reading errors. It made things worse: 32–37% of words wrong, against 26% for the lip reader's own first guess. The simulated errors weren't the errors the real model makes. Small open models used as they come (Qwen3-8B, Qwen3.5-4B and 9B) didn't beat the first guess either (28–31%). - **Teaching the model's language model my phrasing.** Fine-tuning it on 13,446 of my dictated sentences made it less surprised by my test sentences (perplexity 83.5 down to 50). But the more weight it was given, the worse the reading got: 40%, 43%, 46%, then 48% of words wrong. Where the picture is ambiguous, a stronger expectation writes fluent sentences that weren't said. - **Reading points on the face instead of pixels.** Pixels carry skin, beard, lighting and camera along with the movement. The hope was that a model reading only facial landmarks would carry over between people. Trained on the same 12 hours, it got 60% wrong on its own speaker (pixels: 12%) and 104% on me, writing fluent sentences that had nothing to do with what I said. The test was weaker than it looked, though. The landmark model had only ever seen one person, while the pixel model had been pre-trained on thousands. And landmarks can't see the teeth or tongue. - **Two ideas dropped on existing results, before spending GPU time.** One was treating the lips as a 13-key keypad and letting a language model do the rest. Only 2–6% of the model's wrong words looked the same on the lips as the right ones, so the errors were in the seeing. The other was adding the faint whisper's audio to the lip guesses: 14 of 15 sentences right with it, against 13–14 without. At this size, that's no measurable difference. ## Limitations - One speaker, one sitting, and 15 test sentences in my own style, read from a screen rather than made up on the spot. Five improvised sentences mouthed the next day all came out right. Nothing was recorded to check them against, so that's encouragement, not measurement. - The model weights and datasets used here are licensed for non-commercial research only. - This isn't a new method; the contribution is the measurement and the failure modes. ## Takeaway Working with Claude Code on Modal made it fast to build the capture tooling, iterate through GPU runs, and keep a complete lab notebook. That speed is also what made the first finding stand out: when an AI agent runs an evaluation in good faith and produces a number better than the truth, the fix is a stricter harness rather than a smarter model.