Computer vision · Accessible learning
SignSpeak.
Webcam sign recognition, spoken output, and a 3D viewer for studying 497 recorded signs.
- Status
- In development
- Context
- Team Quantum-X · Data Odyssey finalist
- Technology
- Python · TensorFlow · MediaPipe · JavaScript · Three.js · Blender

Problem
Learning a sign involves both understanding its movement and getting useful feedback when practicing it. A static word list cannot provide that interaction.
Solution
SignSpeak combines webcam-based recognition and spoken output with an interactive 3D viewer. Learners can rotate the hand, slow playback, step through recorded frames, inspect landmarks, build sign sequences, and take practice quizzes. The Python pipeline collects and audits samples, trains an LSTM classifier, and evaluates recognition.
Architecture
- 01Webcam input
- 02MediaPipe landmarks
- 03Recognition or pose playback
- 04Text + speech / 3D hand
My contribution
- Expanded the vocabulary to 497 recorded signs and trained the two-layer LSTM classifier on the expanded landmark dataset.
- Rebuilt the hand in Blender and mapped its 20-bone rig to 21 MediaPipe landmarks per hand.
- Connected landmark playback to Three.js and added frame controls, a skeleton overlay, and alignment diagnostics.
Engineering challenge
The viewer fits every rig segment to its recorded endpoints, including segment length. I checked 1,888 sampled hand poses across the 497-sign vocabulary. The alignment check measures rig endpoints against the recordings.



Results & demo
Selected for the Data Odyssey 2026 final round with Team Quantum-X.
The public demo shows an earlier viewer. The cover, two-hand playback, and topology screenshots show the newer local Blender build.
Try the public demo (opens in a new tab)SignSpeak Junior AI Mobile
I built this companion Android app to help children build sentences from picture cards, get next-word suggestions from an on-device AI helper, and hear their sentences spoken aloud.

What I’d improve
Capture wrist movement and relative hand placement, then review sign playback with fluent signers. Evaluate recognition on clips from signers held out of training.
The recordings normalize each hand independently, so two-hand playback uses display spacing. The current public browser demo uses a landmark matcher. Deploying the trained Python classifier in the browser requires a TensorFlow.js conversion.