Lesson 1 of 9 · about 17 minutes
How Synthetic Speech Works
You'll be able to
- Name the stages a text-to-speech system uses to turn writing into a waveform
- Explain what a voice embedding is and why a few seconds of audio can be enough to imitate someone
- Describe prosody and predict where a synthetic read will put the wrong stress
- Choose between text to speech and speech to speech for a given job
A text-to-speech system has to answer two questions for every sentence you give it. Who is speaking, and how do they say it? The words settle neither one. "The board voted four to one" can sound bored, alarmed, smug or relieved, in a baritone or a soprano, and the text gives the machine no clue which you meant. Everything in this lesson is about how current models fill those gaps, and where their guesses go wrong.
The module on how AI works covered the basic idea that modern TTS predicts audio tokens. Here we go one layer down, far enough that you can predict failures and fix them in the script instead of regenerating twenty times.
The pipeline, stage by stage
1. Normalization
Before any sound is made, symbols become words. "$42.50" has to become "forty-two dollars and fifty cents." "Dr." has to become "doctor" or "drive." "1/2" might be one half, January second or February first, depending on country. ElevenLabs' own documentation gives a blunt example: its larger models read "$1,000,000" correctly, while the fast Flash v2.5 model may say "one thousand thousand dollars." That error happens before the voice is involved, so no voice setting can fix it. The script can. Lesson 4 covers how.
2. The acoustic model
This is the part that decides what the speech should sound like, moment by moment. Two designs matter for understanding today's tools.
The older one, made famous by Google's Tacotron 2 paper in 2017, predicts a mel spectrogram: a grid with time across and frequency up, where brightness shows energy. A second network, the vocoder, turns that picture into a waveform. Tacotron 2 used a modified WaveNet as its vocoder and scored a mean opinion score of 4.53 against 4.58 for professional recordings, on a 5-point scale. Later vocoders such as HiFi-GAN made the second step fast enough to run in real time on ordinary hardware.
The newer design treats speech like language. A neural audio codec such as SoundStream (2021) learns to squeeze audio into a stream of discrete codes and rebuild it. Once speech is a sequence of codes, a transformer can predict the next code the way a chatbot predicts the next word. Microsoft's VALL-E paper (2023) did exactly that, trained on 60,000 hours of English speech, and reported convincing imitation of a new speaker from a 3-second sample. Many commercial systems now work along these lines, though few publish their exact architecture.
| Spectrogram plus vocoder | Codec language model | |
|---|---|---|
| Example papers | Tacotron 2 (2017), HiFi-GAN (2020) | SoundStream codec (2021), VALL-E (2023) |
| What the model predicts | A mel spectrogram | Discrete audio codes, one after another |
| How a new voice gets in | A speaker embedding fed in alongside the text | A few seconds of the target voice used as a prompt |
| Typical failure | Flat, slightly buzzy reads; robotic on long passages | Very natural, but can skip, repeat or invent words |
| What it means for you | Consistent and predictable | Listen to every take; transcribe and compare |
Worked example: why codes make audio manageable
Raw CD-quality mono audio is 44,100 numbers per second. A 30-second read is 44,100 x 30 = 1,323,000 numbers. Predicting those one at a time is slow and hard.
Suppose a codec instead describes each second with 75 frames of 8 codes each. That is 75 x 8 = 600 codes per second, or 600 x 30 = 18,000 codes for the same read. The model now predicts 18,000 symbols instead of 1.3 million, about 73 times fewer (1,323,000 / 18,000 = 73.5). The numbers here are illustrative. The ratio is the point: compression is what lets a l 30 minutes for its professional clone, which fine-tunes a model on that one voice. Research systems have shown recognizable results from seconds. Quality still rises with clean, varied source audio, and the professional route captures habits an instant clone misses, like how a person laughs or trails off.
How they say it: prosody
Prosody is the melody and rhythm of speech: pitch rising and falling, syllables stretched or clipped, pauses, loudness. Text carries almost none of it. A comma hints at a pause. A question mark hints at a rise. Stress, the thing that changes meaning most, is usually invisible.
| Stressed word | What a listener hears |
|---|---|
| The BOARD didn't approve the bond. | Somebody else did, maybe the voters. |
| The board DIDN'T approve the bond. | Flat denial of a rumor. |
| The board didn't APPROVE the bond. | They did something else with it: tabled it, amended it. |
| The board didn't approve the BOND. | They approved something, just not the bond. |
A model picks the most statistically likely stress for a sentence like that, which is usually the last content word. Most of the time that is fine. In announcements about public decisions, where the surprising word is often in the middle, it is wrong often enough to matter. You steer it with sentence order (put the key word last), with punctuation, with capitals on models that honor them, and on expressive models with audio tags. ElevenLabs' prompting guide, which it says applies to Eleven v3 as well as the newer v4, says capitalizing a word increases emphasis and ellipses add pauses and weight.
Stability settings are a prosody dial in disguise. Low stability lets the model wander more in pitch and timing, which sounds lively on a short line and drunk on a long one. High stability narrows the range, which sounds steady and, past a point, bored.
Speech to speech: borrowing a performance
Voice conversion skips the hardest guess. You record the line yourself, with the stress and timing you want, and the model replaces only the voice. In Voice Studio's Speech2Speech tool you upload an MP3, WAV, M4A, OGG or FLAC recording, pick a target voice, choose the Multilingual STS v2 or English STS v2 model, and can tick "Denoise input" to clean the source first. Stability, similarity and style sliders work as they do in Text2Speech.
Use it when prosody matters more than convenience: a line with sarcasm, a name you need said a particular way, a character whose rhythm you can perform but whose voice you cannot. The catch is that every flaw in your delivery comes along too. Mumble, and the new voice mumbles.
Common mix-ups
- Claim: Cloning a voice takes hours of studio audio. What the evidence shows: Published research imitated speakers from 3 seconds, and commercial instant clones ask for a minute or two. Hours buy polish, not possibility.
- Claim: The AI reads exactly what you typed. What the evidence shows: Normalization can misread numbers, and codec-style models occasionally skip or repeat words. Check every take against the script.
- Claim: A synthetic voice is a recording of the original person. What the evidence shows: It is new audio generated to land near that person's embedding. Nothing in it was ever said by them, which is the whole legal and ethical problem with unauthorized clones.
Reporting angle
Voice-clone scams, where a caller sounds like a grandchild or a boss asking for money, are now a police-blotter staple. When one surfaces locally:
- Request the incident report and ask whether the department logs suspected voice-clone calls as their own category, and how many it has logged this year.
- Ask where the scammer could have found the victim's family member's voice. Public video with a minute of clean speech is enough, as this lesson shows.
- Ask the bank or credit union whether it still uses voice recognition to verify customers by phone.
- Never publish a "recording of the scam" without saying how you know it is the real call and not a re-creation.
Research angle
- WaveNet paper, the 2016 model that made neural speech sound natural.
- Tacotron 2 paper: text to mel spectrogram to waveform.
- VALL-E paper on zero-shot voice cloning from a short sample.
- ElevenLabs docs on Instant Voice Cloning and its rules.
- ElevenLabs docs on Professional Voice Cloning and its verification step.
- ElevenLabs prompting controls: pauses, pronunciation and normalization.
Check yourself
1. In the Tacotron 2 design described by Google researchers in 2017, what does the first network produce from the text?
Tacotron 2 maps characters to a mel spectrogram, a picture of energy by frequency over time. A modified WaveNet vocoder then builds the actual waveform from it.
2. Microsoft's 2023 VALL-E paper reported imitating an unseen speaker from how much enrolled audio?
The abstract says VALL-E can synthesize personalized speech from a 3-second enrolled recording, after pre-training on 60,000 hours of English speech.
3. Two voice embeddings point in nearly the same direction. What does that usually mean?
Speaker encoders are trained so clips from one person land close together, whatever words are spoken. Closeness is measured by angle, often as cosine similarity.
4. Which of these is part of prosody?
Prosody is the music of speech: pitch movement, timing, loudness and pauses. Stress placement is the part listeners notice most.
5. You like your own pacing and emphasis on a read but want a different voice for a character. Which Voice Studio tool fits?
Speech2Speech converts a recording into another voice while keeping its timing and delivery, so your performance carries over.
6. Why can a text-to-speech system read '$1,000,000' wrong even when the voice sounds perfect?
ElevenLabs' docs say larger models read '$1,000,000' correctly while Flash v2.5 may say 'one thousand thousand dollars.' That is a normalization failure, upstream of the voice.
Terms in this lesson
- Mel spectrogram
- A picture of sound showing energy at each frequency over time, with frequencies spaced the way human hearing spaces them.
- Vocoder
- The part of a speech system that turns an intermediate representation, such as a spectrogram or codec tokens, into an audible waveform.
- Neural audio codec
- A trained network that squeezes audio into a short stream of discrete codes and rebuilds audio from them. Modern speech and music models predict these codes.
- Voice embedding
- A fixed-length list of numbers that summarizes how a speaker sounds, regardless of what they are saying.
- Prosody
- The rhythm, stress, pitch movement and pausing of speech.
- Voice conversion
- Changing the voice in a recording to a different voice while keeping the words, timing and delivery. Voice Studio calls it Speech2Speech.
Sources
- van den Oord et al., WaveNet: A Generative Model for Raw Audio (arXiv 1609.03499) checked 2026-10
- Shen et al., Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2, arXiv 1712.05884) checked 2026-10
- Wan et al., Generalized End-to-End Loss for Speaker Verification (arXiv 1710.10467) checked 2026-10
- Jia et al., Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (arXiv 1806.04558) checked 2026-10
- Kong et al., HiFi-GAN (arXiv 2010.05646) checked 2026-10
- Zeghidour et al., SoundStream: An End-to-End Neural Audio Codec (arXiv 2107.03312) checked 2026-10
- Wang et al., Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E, arXiv 2301.02111) checked 2026-10
- ElevenLabs docs: Prompting controls (normalization, pauses, pronunciation) checked 2026-10
- ElevenLabs docs: Instant Voice Cloning checked 2026-10
- ElevenLabs docs: Professional Voice Cloning checked 2026-10
- Voice Studio: Speech2Speech (Journal Tools) checked 2026-10