Pitch → lightness
higher = lighter
Higher pitch → lighter / brighter color
TL;DR
Synthetic Synesthesia is a real-time web app that simulates chromesthesia — the form of synesthesia where sound is seen as color. It runs in both directions: aim it at sound and it paints; aim it at color and it plays. Every mapping rule is drawn from peer-reviewed research, not aesthetic guesswork.
The Sound → Color direction reads your microphone with a live FFT, extracts pitch class, octave, loudness, and timbre, and maps each to hue, lightness, saturation, and shape on a glowing canvas. The Color → Sound direction points the camera at anything, pulls the dominant colors with k-means clustering, and reverse-maps them through the same model into a chord that shifts as you pan. One science engine, run forwards and backwards, so the simulation stays internally consistent.
Synthetic Synesthesia runs entirely in your browser and asks for microphone and camera access. Nothing is recorded or uploaded — all analysis happens locally on your device.
THE PROBLEM
Think about this:
you hear a violin, or the letter A, or a single piano note.
For about one in twenty-five people, it has a color,
and the same sound calls up the same color every time.
The note might flash red; a higher octave, lighter.
For the other 96%, this entire experience is invisible.
Chromesthesia is not random. Across studies, some sound–color links are strikingly systematic: higher pitch reads as lighter and brighter, louder sounds as more saturated, sharper timbres as more vivid, and pitch classes tend to fall along a rainbow from red to violet. These are real, measurable regularities in how brains bind hearing to sight.
But almost every audio visualizer you've ever seen ignores this. It maps color to sound arbitrarily — whatever looks good — which means it scientifically misrepresents a genuine neurological phenomenon and reduces it to decoration.
The consensus from the literature
Loudness → saturation
louder = richer
Louder / more energy → more saturated color
Timbre → brightness
sharper = brighter
Sharper timbre → brighter, higher-value color
Pitch class → hue
C ≈ red
Pitch class → hue along a red-to-violet rainbow
There is no single universal "sound → color code" — exact palettes are deeply individual. But the tendencies are robust and probabilistic, which means they can be modeled. So the goal became: build a simulator where every rule is defensible, and let anyone step inside a perception they may never have had.
THE SOLUTION
I built a single, reversible science engine and wrapped it in two live, sensor-driven interfaces.
The pipeline
-
01
Grounding in research
- Surveyed the peer-reviewed chromesthesia literature (15+ studies across synesthetes and non-synesthetes).
- Extracted the consensus mappings: pitch class → hue, octave → lightness, loudness → saturation, spectral centroid → brightness, timbre → shape.
- Encoded each rule as a deterministic function in a custom chromesthesia.js module, with exponential smoothing to mirror how synesthetes report stable, consistent associations.
-
02
Sound → Color
- Live microphone analysis via the Web Audio API: a 4096-sample FFT, spectral-centroid calculation, and RMS loudness.
- Pitch detected and converted to a MIDI note to recover pitch class (hue) and octave (lightness).
- Rendered on an HTML5 Canvas with device-pixel-ratio-aware drawing, trail effects, and shape generation driven by timbre.
-
03
Color → Sound
- Camera frames sampled at ~4 FPS; the 3–4 dominant colors extracted with custom k-means clustering (8 iterations).
- Each color is reverse-mapped back through the same model to recover musical notes.
- Played as a continuous chord with Tone.js — four independent sine synths with per-voice frequency ramping so the harmony glides rather than jumps as the camera pans.
-
04
Bidirectionality as the design
- Both interfaces call the same engine, so the two directions are inverses of each other rather than two separate demos.
- A given note produces a color that, fed back through the camera path, resolves toward that same note — the simulation is self-consistent by construction.
THE MAPPING
This is the engine, made playable. Pick a pitch class to set the hue, then move octave, loudness, and timbre to hear — in color — how the model translates a sound. Every rule below is the one running live in the app.
sound → color · mapping key
grounded in Chiou 2013 · Sun 2018 · Anikin & Johansson 2018 · Orlandatou 2015
note C4
hsl(0°, 70%, 55%)
hsl(0°, 70%, 55%)
round · soft timbre
Bouba/kiki in action: soft timbres round the shape off; sharp timbres pull it toward points. The Color → Sound path runs this key in reverse — a swatch of color is read back into the nearest note.
THE RESULT
- Shipped and public — a live, self-contained web app deployed on Vercel with GitHub CI/CD, runnable in any modern browser with no install.
- Fully bidirectional — sound → color and color → sound driven by one shared, research-backed model, rather than two unrelated visualizers.
- Grounded, not arbitrary — every mapping traces to published chromesthesia findings; the science engine is a discrete, auditable module.
- An accessibility bridge — a scientifically grounded visual representation of sound for deaf and hard-of-hearing users, and sonification of the visual environment for blind and low-vision users.
Scope, honestly: this is a fast, focused proof-of-concept built in a short window with an AI pair-programmer — its value is in the rigor of the mapping and the bidirectional design, not in production polish or a user study.
WHAT THIS PROJECT DEMONSTRATES
-
Research-grounded systems design
Translating a body of peer-reviewed cognitive-science findings into a small set of deterministic, defensible mapping functions — and making the reasoning legible rather than hiding it behind "it looks nice."
-
Real-time DSP & machine learning
A browser audio pipeline (4096-sample FFT, spectral centroid, RMS, MIDI pitch detection) alongside k-means color clustering on live camera frames and Tone.js synthesis with per-voice ramping.
-
Reversible, bidirectional modeling
One model expressed as its own inverse, so a single engine drives both directions and the two experiences stay internally consistent.
-
Accessibility & science communication
Making a phenomenon that ~4% of people have — and 96% cannot access — experienceable by anyone with a browser, with genuine affordances for deaf/HoH and blind/low-vision users.