How Image, Audio and Video Generators Work

Type "a red barn at dusk with a thunderstorm rolling in" and twenty seconds later you have a picture of a barn nobody built, under a storm that never happened. The machine didn't search for a photo. It didn't paste pieces together. It started with static and carved a picture out of it.

Diffusion: sculpting from noise

Most image generators today use diffusion. The modern recipe traces to a 2020 paper by Jonathan Ho, Ajay Jain and Pieter Abbeel at UC Berkeley. Training works backward from what you'd expect.

1. Take a real image from the training set, say a photo of a dog on a porch.
2. Add a little random noise, like TV static. Then a little more. Keep going over hundreds of steps until it's pure static.
3. Train a neural network to look at any noisy version and predict the noise that was added. Subtract the predicted noise and you get a slightly cleaner image.

Do this with hundreds of millions of images and the network gets very good at cleaning up noise. Then comes the trick. Hand it pure random static, the kind with no image in it at all, and ask it to clean up. It "finds" an image that could have been under the noise. Repeat 20 to 50 times and a sharp, brand new picture appears.

That's why the same prompt gives a different picture every time: each run starts from different random static. Many tools let you fix the "seed" number to get the same starting noise, and so a near-identical image, again.

Working small: latent space

Denoising millions of pixels 50 times is slow. A 2021 paper by Robin Rombach and colleagues, the basis of Stable Diffusion, moved the work into a compressed version of the image called latent space. A separate network squeezes images down and expands them back. Stable Diffusion's design shrinks each side by 8 and keeps 4 channels, so a 1024 by 1024 color image (1024 x 1024 x 3 = 3,145,728 numbers) becomes a 128 by 128 grid with 4 channels (65,536 numbers). That's 48 times less to process. The diffusion happens in that small space, and only the final result is expanded into pixels. Most image and video generators now work this way.

Steering with words: the text encoder

Denoising alone would give you a random picture. To get your barn, the generator needs to understand the prompt. That's the text encoder's job. It turns your words into a list of numbers, and at every denoising step the image network uses those numbers (through the same attention idea from lesson 5) to push the picture toward the description.

OpenAI's CLIP, published in 2021, showed how to build such an encoder. It trained on 400 million images paired with their captions from the internet, learning to put an image and its caption close together in number space. After that training, "golden retriever on a porch" lands near pictures of golden retrievers on porches. Newer generators often use a full language model as the encoder, which is one reason they follow long, detailed prompts better than 2022 tools did.

This explains a lot about prompting. The encoder learned from captions people wrote online. Words that commonly appeared next to good photos ("golden hour," "35mm," "shallow depth of field") pull images toward that look. Negations are weak: "no people" still contains "people," and older models would sometimes add a crowd.

Why the hands look wrong

Generators learn what things usually look like. They never learn the rules of how things are built. They have seen millions of hands at every angle, often partly hidden, so the "average hand" is fuzzy about finger count. Text is worse: "FEED STORE" needs exact letters in exact order, and early models learned only that signs have letter-ish squiggles. Counting ("exactly seven ducks"), clocks, reflections and left-versus-right also trip them up. The 2025 and 2026 generation is far better at all of this. Still, zoom in and check every letter, finger and reflection before you trust or share one.

Video: diffusion that moves

A video generator runs diffusion over a block of frames at once, so the model learns how things change over time as well as how they look. That's much harder. Each second at 24 frames per second is 24 images that must agree with each other. Typical failures: objects that melt into each other, people who gain a finger mid-wave, physics that almost works. As of October 2026, Google's developer documentation lists Veo 3.1, which generates video with matching audio, and other companies such as OpenAI and xAI offer their own video models. Clips usually run seconds, and longer pieces are stitched from many generations.

Audio: speech, voices and music

Text-to-speech used to mean stitching recorded syllables together, which is why old GPS voices sounded choppy. Modern TTS uses neural networks trained on thousands of hours of recorded speech with transcripts. Many work much like a language model: they turn audio into a sequence of tokens and predict the next audio token, then a decoder turns those tokens back into a sound wave. That's why they can add breaths, pauses and emphasis that weren't in the text.

Voice cloning conditions that model on recordings of one specific person. As of October 2026, ElevenLabs recommends about 1 to 2 minutes of clean audio for its instant cloning, and at least 30 minutes (two hours or more for best results) for its higher-quality professional cloning. Before saving an instant clone, its users must confirm they have the right and consent to clone that voice. A clean, quiet recording matters more than length; ElevenLabs says going past about 3 minutes for instant cloning adds little.

Music generators combine these ideas: a text encoder for the style prompt, and a diffusion or token-prediction model producing audio. The same weaknesses show up. They're good at the average sound of a genre and weaker at the particular choices that make a song memorable.

Medium	Common method	Steered by	Typical tells
Still images	Latent diffusion	Text encoder, reference images	Garbled text, extra fingers, too-smooth skin, impossible lighting
Video	Diffusion over many frames	Text, a starting image	Morphing objects, physics glitches, short clip lengths
Speech	Audio-token prediction plus a decoder	Text, voice sample	Odd stress on names, flat emotion over long reads, breath in wrong places
Music	Diffusion or audio-token prediction	Style prompt, lyrics	Smeared lyrics, generic structure, muddy mix in dense parts

Marking what's synthetic

Several companies now hide watermarks in generated media. Google's documentation says all images from its Gemini image models carry SynthID, an invisible watermark software can detect. Industry groups also push content credentials, metadata that records how a file was made. Neither is foolproof. Metadata gets stripped when files are re-uploaded or screenshotted, and not every tool adds a watermark. The absence of a mark proves nothing.