MS·

Project / LIP-TRAC

← Projects Research / Visual speech recognition

LIP-TRAC

Lipreading through a Temporal Recurrent and Convolutional network

The assembled LIP-TRAC prototype hanging from a clip: a red 3D-printed case around a Raspberry Pi 5 with its status LED lit, a camera module at the top, and a black LIP-TRAC nameplate.
The assembled prototype used in the field trials

LIP-TRAC reads sentences from silent video of a speaker’s mouth. Three 3D convolutions and two bidirectional GRUs, trained with CTC loss on the BBC LRS2 dataset, transcribe each clip on a Raspberry Pi 5 inside a 3D-printed case.

Word error rate
32.7%

LRS2, 456 held-out clips

Character error rate
14.0%

Same split

Inference
6.3s

Mean per clip, Raspberry Pi 5

RTPS
0.1068

Highest of 7 models compared

Read the paper PDF, 21 pp. ↗ Poster PDF ↗

Model input, one frame 40 × 120 × 1 · uint8

Hover a cell row ·· col ··· 0x·· value ··· 4,800 bytes. Values recovered from the paper’s Fig. 3, a crop of a StyleGAN2-generated face. actual size

Why read lips at all

Speech recognition that listens breaks down where people with hearing loss most need help: loud rooms, several voices at once, and conversations with no audible speech, whether that is a library or a speaker with aphonia. Video of the mouth carries none of that noise.

The hard parts are accuracy and cost. Unaided human lipreaders recognise roughly 30% of words. The most accurate lipreading models are large networks built for benchmark scores, and many of those papers don’t report how long inference takes. A hearing aid averages over $1,000 and does little for aphonia or aphasia.

So LIP-TRAC was given three targets before training began: work across speakers, reach under 35% word error and 20% character error, and return text within 10 seconds of the speaker finishing, on hardware a person can actually buy.

1 in 5
people worldwide live with hearing loss WHO, 2024
700M+
projected to have disabling hearing loss by 2050 WHO, 2024
~30%
of words recognised by human lipreaders Assael et al., 2016
$1,000+
average price of a hearing aid paper, §I

From a camera frame to 4,800 bytes

Five steps turn every clip into network input, identically in training and on the device; a sixth prepares the transcripts for training. The goal is to hand the network only the part of the image that moves when someone speaks.

  1. 01

    Capture

    A 160 × 160 RGB stream, cut into clips shorter than 150 frames.

    160 px t < 150 frames / clip
  2. 02

    Find the face

    OpenCV’s frontal-face Haar cascade runs on every frame. When a frame has no detection, the last good box is reused, so one missed frame doesn’t break the clip.

    haarcascade_frontalface_default.xml

    Haar-like features, boosted and chained into a cascade (Viola–Jones)
  3. 03

    Crop the mouth

    The crop is relative to the face box: from 65% of its height down to the bottom edge, and from 5% to 95% of its width. Because it scales with the box, the same rule works whether the speaker is close to the camera or far from it.

    t−1 box, reused on a miss w h 0.65 h 0.05 w 0.95 w mouth crop
  4. 04

    Shrink

    Resize to 40 × 120 and drop to grayscale. Lip shape is carried mostly by luminance, and a single 8-bit channel at this size is 4,800 bytes per frame, 16× less than the 76,800-byte source.

    160 × 160 × 3 76,800 B 16× 40 × 120 × 1 4,800 B
    Both drawn at the same scale.
  5. 05

    Normalise per clip

    Subtract the clip’s average frame, then divide by the standard deviation of every pixel in the clip. What reaches the network is how far each pixel strays from that clip’s average, which is the part that changes when someone talks. Static lighting and skin tone mostly drop out.

    xt, one frame
    μ, clip average
    deviation
    Frame and clip average recovered from the poster (panel 3). The difference is computed in your browser, after putting both on a common scale and lightly smoothing. Brighter and darker than average.
  6. 06

    Encode the transcript

    Labels use upper-case A–Z and space. Numerals are written out (“5” becomes “FIVE”), and every transcript is padded to 145 characters. CTC adds one more symbol, the blank.

    1. A
    2. B
    3. C
    4. D
    5. E
    6. F
    7. G
    8. H
    9. I
    10. J
    11. K
    12. L
    13. M
    14. N
    15. O
    16. P
    17. Q
    18. R
    19. S
    20. T
    21. U
    22. V
    23. W
    24. X
    25. Y
    26. Z
    27. ␣
    28. ·
    26 letters, space, and the CTC blank (·)
Twelve crops the pipeline produced from a range of LRS2 speakers (poster, panel 1). Facial hair, teeth, turned heads, blur and uneven lighting all have to map onto the same 26 letters.

Twelve layers

The network follows the spatiotemporal-CNN, recurrent, CTC design that LipNet introduced for sentence-level lipreading, kept small enough to run on a Raspberry Pi 5.

  • 3 × 3 × 3 kernels span three neighbouring frames, so even the first layer sees motion, not only shape.
  • Pooling is (1, 2, 2). Height and width halve at each stage; time never does, so every input frame keeps its own output step for CTC.
  • GRU over LSTM. Two gates instead of three. GRUs were chosen after testing both, for faster inference.
  • Bidirectional. Each clip is read forwards and backwards, so every frame sees context from both sides.
  • Dropout 0.5 after each recurrent layer.
#LayerWhat it doesParams
inInputT × 40 × 120 × 1 normalised mouth crops0
1Conv3D 3×3×3, 32, ReLU32 feature maps per frame896
2MaxPool3D 1×2×2H, W halved · T kept0
3Conv3D 3×3×3, 64, ReLU64 feature maps55,360
4MaxPool3D 1×2×2H, W halved · T kept0
5Conv3D 3×3×3, 96, ReLU96 feature maps165,984
6MaxPool3D 1×2×2H, W halved · T kept0
7TimeDistributed Flattenone feature vector per frame0
8Bidirectional GRUforward ⊕ backward over T–
9Dropout 0.5regularisation0
10Bidirectional GRUforward ⊕ backward over T–
11Dropout 0.5regularisation0
12Dense softmaxcharacter + blank distribution per frame14,877
Layer order, kernel sizes, pooling and dropout from the paper (§III, Fig. 5). Parameter counts are from the paper’s Fig. 5 (recurrent layers not broken out here). Convolution widths follow from them, e.g. 896 = 32 × (3·3·3 + 1).

Training without frame-level labels

LRS2 gives one transcript per clip, not a letter per frame. Connectionist Temporal Classification removes the need for that alignment. At every frame the network outputs a character or a blank, and the loss adds up the probability of every frame-by-frame path that collapses to the right transcript. Decoding is two rules: merge repeated symbols, then delete blanks.

path
· B B B K ·
merge repeats
· B K ·
drop blanks
BK

LIP-TRAC’s own output after 5 epochs of training (paper, Fig. 8). It has found the B and the K but not the vowels.

Why word error is higher than character error

A doubled letter survives decoding only if the network emits a blank between the two copies. LIP-TRAC’s most frequent mistake was missing that blank, so HELLO came out as HELO. That is one wrong character, and a whole wrong word, which is the main reason its WER (32.7%) sits so far above its CER (14%). In practice HELO is still readable.

P(y | X) sums over every alignment that collapses to transcript y. Over training, the loss fell from about 210 to about 27.

Trained on one consumer GPU

Framework
TensorFlow, Keras
Data
BBC LRS2, 683 training and 456 test clips, shuffled before the split
Hardware
NVIDIA RTX 3060 Ti, 4,864 CUDA cores, 8 GB. First trials ran on an Intel Core i5-12600K CPU.
Optimizer
Adam
Learning rate
3.0 × 10−6 initial, exponential decay
Batch size
1. Clips vary in length and 3D convolutions are memory-hungry on 8 GB.
Epochs
Up to 300, with periodic checkpoints
Iterations
14 model versions
10⁻⁸10⁻⁷10⁻⁶10⁻⁵10⁻⁴10⁻³10⁻² 3.0 × 10⁻⁶ start ← decays
The starting rate came from sweeping loss and accuracy against log learning rate across optimizers and schedulers, then taking the minimum.
CTC loss
≈210 → ≈27
Word accuracy
≈0% → 67.3%
Character accuracy
rising from the start → 86.0%

Word accuracy stayed near zero for roughly the first 75 epochs while character accuracy was already climbing. A word only counts once every letter in it is right.

Epoch 92/300
1/1 [==============================] - 0s 93ms/step
Original:   WHATEVER YOUR REQUIREMENTS
Prediction: WHATEVER YOUR REQUIREMENTS
Word Error Rate:  0.0%
One sample from the training log (poster, panel 5). A single clip is a best case; the averages under Results are the real measure.

Results

CriterionTargetMeasured
Word accuracyWER < 35%32.7% WER on 456 held-out LRS2 clipsmet
Character accuracyCER < 20%14.0% CERmet
Latency< 10 s after speech ends6.3 s mean per clip on a Raspberry Pi 5met
SpeakersNot tied to one speaker65.8% word accuracy on new speakers and sentencesmet

Real-Time Performance Score

Accuracy-only leaderboards hide latency. RTPS, introduced in the paper, divides word accuracy by inference time:

RTPS = 1 − WERt = 0.6736.3 s = 0.1068

On the chart it is the slope of a line from the origin. Anything steeper is better. Move across the chart to read off the score for any accuracy and latency.

In the paper’s comparison with six published LRS2 lipreading models (2017–2023), LIP-TRAC had the highest RTPS.

Word accuracy, alongside published models

  1. Ma et al., 202372.1%
  2. Prajwal et al., 202271.1%
  3. LIP-TRAC67.3%
  4. Ma et al., 202267.1%
  5. Yu et al., 202051.1%
  6. Ren et al., 202150.8%
  7. Chung et al., 201729.6%
  8. Human lipreaders~30%
1 − WER, as compiled in the paper (Fig. 10). Other models’ figures are their published LRS2 results; LIP-TRAC’s is on its own 456-clip held-out split, so read the comparison as indicative.

Field trials on the device

New sentences, spoken by people outside the training set, run on the Pi 5 in its printed enclosure. The averages landed within about 5% of the test-set results.

PhraseWord acc.Char. acc.Inference
161.2%89.1%7.1 s
283.5%94.0%6.6 s
359.7%67.8%6.4 s
467.8%92.3%6.3 s
556.6%74.5%5.8 s
Mean65.8%83.5%6.4 s
CAD render of the LIP-TRAC handheld: a red 3D-printed shell holding a Raspberry Pi 5, a camera module at the top, and a LIP-TRAC nameplate across the middle.
  1. Camera module
  2. Raspberry Pi 5
  3. 3D-printed enclosure

The device

The enclosure was modelled in CAD around a Raspberry Pi 5 and its camera module, then 3D-printed. The assembled unit, pictured at the top of this page, is the hardware the field trials ran on. Face detection, cropping, normalisation and inference all happen on the Pi, from video alone.

Compute
Raspberry Pi 5
Input
Camera, 160 × 160 RGB
Latency
6.3 s mean per clip
Enclosure
3D-printed, CAD-modelled

Limits, and what comes next

What it can’t do yet

  • It has only seen British English broadcast speech (LRS2). Other languages, accents, very low light and extreme head poses may not hold up.
  • It is visual-only, so it can’t use clean audio even when some is available.
  • CTC decoding drops doubled letters when the blank between them is missed.
  • 6.3 seconds clears the paper’s real-time bar but is still slower than conversation.
  • The Pi’s compute budget is fixed, so accuracy gains have to come without giving up efficiency.

Next

  • Add a microphone and use audio to validate the visual transcript.
  • Predict words from a dictionary instead of single characters.
  • Add an n-gram language model for sentence context.
  • Use attention and the whole face, not just the mouth.
  • Other languages, a multilingual model, or the IPA, once suitable data exists.

Documents and references

  1. Chung, Senior, Vinyals & Zisserman. Lip reading sentences in the wild. CVPR 2017. doi:10.1109/CVPR.2017.367
  2. Graves, Fernández, Gomez & Schmidhuber. Connectionist Temporal Classification. ICML 2006. cs.toronto.edu
  3. Assael, Shillingford, Whiteson & de Freitas. LipNet: End-to-End Sentence-level Lipreading. 2016. arXiv:1611.01599
  4. Lip Reading Sentences 2 (LRS2) dataset. Visual Geometry Group, University of Oxford. robots.ox.ac.uk
  5. World Health Organization. Deafness and hearing loss. 2024. who.int