All projects

Computer vision · Accessible learning

SignSpeak.

Webcam sign recognition, spoken output, and a 3D viewer for studying 497 recorded signs.

Status
In development
Context
Team Quantum-X · Data Odyssey finalist
Technology
Python · TensorFlow · MediaPipe · JavaScript · Three.js · Blender
SignSpeak 3D viewer showing the recorded hello sign and its playback controls
Latest local Blender viewer · SignSpeak

Problem

Learning a sign involves both understanding its movement and getting useful feedback when practicing it. A static word list cannot provide that interaction.

Solution

SignSpeak combines webcam-based recognition and spoken output with an interactive 3D viewer. Learners can rotate the hand, slow playback, step through recorded frames, inspect landmarks, build sign sequences, and take practice quizzes. The Python pipeline collects and audits samples, trains an LSTM classifier, and evaluates recognition.

Architecture

  1. 01Webcam input
  2. 02MediaPipe landmarks
  3. 03Recognition or pose playback
  4. 04Text + speech / 3D hand
Simplified data flow through SignSpeak.

My contribution

  • Expanded the vocabulary to 497 recorded signs and trained the two-layer LSTM classifier on the expanded landmark dataset.
  • Rebuilt the hand in Blender and mapped its 20-bone rig to 21 MediaPipe landmarks per hand.
  • Connected landmark playback to Three.js and added frame controls, a skeleton overlay, and alignment diagnostics.

Engineering challenge

The viewer fits every rig segment to its recorded endpoints, including segment length. I checked 1,888 sampled hand poses across the 497-sign vocabulary. The alignment check measures rig endpoints against the recordings.

Two articulated hands playing the recorded school sign
Two-hand playback for “school”
Wireframe hand with the landmark skeleton and endpoint alignment readout
Blender surface topology and landmark alignment
Webcam recognition tab with camera off, model status, and sentence controls
Recognition and speech interface

Results & demo

Selected for the Data Odyssey 2026 final round with Team Quantum-X.

The public demo shows an earlier viewer. The cover, two-hand playback, and topology screenshots show the newer local Blender build.

Try the public demo (opens in a new tab)

SignSpeak Junior AI Mobile

I built this companion Android app to help children build sentences from picture cards, get next-word suggestions from an on-device AI helper, and hear their sentences spoken aloud.

Android sentence builder and on-device word suggestions for “I want water”
Android sentence builder and on-device word suggestions for “I want water”

What I’d improve

Capture wrist movement and relative hand placement, then review sign playback with fluent signers. Evaluate recognition on clips from signers held out of training.

The recordings normalize each hand independently, so two-hand playback uses display spacing. The current public browser demo uses a landmark matcher. Deploying the trained Python classifier in the browser requires a TensorFlow.js conversion.

Project references

Next projectQuantumX AEGIS

Find a page