What AI Runs On: Chips, Compute and Cost

Every chatbot answer ends as electricity moving through a chip in a building somewhere. Most talk about AI skips that part and goes straight to what the software can do. The physical layer is where the money, the power lines, the trade fights and the local zoning hearings live, so it is where residents and local officials have the most say.

Why AI needs special chips

A large language model is a few hundred billion numbers, called weights, arranged in grids. Answering a prompt means multiplying your words, turned into numbers, through those grids layer after layer. That is matrix multiplication, repeated an absurd number of times. A normal computer processor (a CPU) is a brilliant generalist that does a few things at a time very fast. AI wants the opposite: millions of simple multiplications at once.

A GPU (graphics processing unit) was built to color millions of screen pixels at the same moment, which turns out to be the same kind of parallel math. Nvidia's H100, the chip most AI buildouts were planned around from 2023 on, draws up to 700 watts by itself and carries 80 GB of very fast memory, according to Nvidia's spec sheet. Newer Blackwell parts draw more.

A TPU (tensor processing unit) is Google's own chip. Google describes it as an application-specific integrated circuit whose core is a 128 by 128 grid of multipliers called a systolic array. Numbers flow through the grid like a bucket brigade. It is superb at matrix-heavy work and poor at anything branchy or custom. Amazon, Microsoft and Meta have their own custom chips too, mostly to depend less on one supplier.

Chip type	Good at	Who makes it	Where you meet it
CPU	General tasks, logic, running the operating system	Intel, AMD, Arm designs	Every server, including AI servers, as the "host"
GPU	Massively parallel math, flexible programming	Nvidia (dominant), AMD	Most AI training and inference worldwide
TPU and other custom chips	Matrix math for specific model types	Google (TPU), Amazon, Microsoft, Meta	Inside one company's cloud

Nvidia's position

Nvidia sells the chips, the networking that ties thousands of them together, and the CUDA software most AI code is written for. That software lock-in matters as much as the silicon. The scale shows up in its filings: for the quarter ending July 26, 2026, Nvidia reported $96.2 billion in revenue, $89.0 billion of it from data center products. Nvidia's outlook for the next quarter assumed no data center compute revenue from China at all. When a single supplier's quarterly data center sales rival the annual budget of a mid-sized state, supply problems and price swings in chips show up in everybody's AI plans, including a county that wants to buy an AI tool.

Training versus inference

This is the distinction that clears up half of the confusing AI numbers you will read.

A training run is the one-time job of building a model. Thousands to hundreds of thousands of chips run for weeks or months, adjusting the weights over and over until the model predicts text well. It is expensive, concentrated and done by a handful of companies.

Inference is using the finished model. Every prompt, every autocomplete suggestion, every AI summary at the top of a search page. Each one is cheap. There are billions of them a day, and the count keeps rising, so over a model's life inference can add up to more total computing than training did.

How big and how fast it is growing

Epoch AI, a research group that tracks this, estimates that the computing used to train frontier language models has grown about 5 times per year since 2020, and that the cost of those training runs has risen about 3.5 times per year. Its earlier paper on training costs found growth of about 2.4 times per year from 2016 and projected the largest single runs could pass $1 billion by 2027.

Where does the money go? For models like GPT-4 and Gemini Ultra, Epoch estimated hardware at 47 to 67 percent of development cost, research staff at 29 to 49 percent, and energy at just 2 to 6 percent. Energy is a small slice of the company's cost and a large item for the local utility. Both are true, and that gap explains a lot of the friction at data center hearings.

Meanwhile, using a model of a given quality keeps getting cheaper. Stanford's AI Index reports that the price of getting GPT-3.5 level performance fell from $20 to $0.07 per million tokens between November 2022 and October 2024, roughly a 280-fold drop. Cheaper answers usually mean more answers, which is why per-unit savings have not shrunk total demand for chips or power.

Why AI needs data centers

Those chips have to sit somewhere with huge power feeds, cooling that can carry away hundreds of kilowatts per rack, fiber connections, and security. That place is a data center. Lawrence Berkeley National Laboratory's December 2024 report found US data centers used about 176 terawatt-hours in 2023, 4.4 percent of US electricity, and projected 325 to 580 TWh by 2028. The report names growth in GPU-accelerated AI servers as the main driver. The wide range comes from honest uncertainty about chip shipments, how hard the chips are run and which cooling designs win.

This course covers the buildings themselves in two other modules: "Data Centers: Types, Purposes and How They Work" explains the facility types and how they are built, and "Data Centers in Your Community: Power, Water, Land, Money" covers tax deals, water and the grid. "How the Power Grid Works" explains where those megawatts come from.

Chips as geopolitics: export controls

Because advanced AI chips are useful for military and intelligence work, the US Commerce Department's Bureau of Industry and Security (BIS) restricts who can buy them. The rules have swung back and forth, so date every claim you repeat.

Date	What happened
Oct 7, 2022	BIS restricts advanced AI chips and chipmaking tools for China
Oct 17, 2023	Rules tightened; Nvidia's China-specific A800 and H800 now covered
Jan 13, 2025	"AI Diffusion" rule sets country-by-country chip limits
Apr 9, 2025	License now required for Nvidia's H20, a chip designed to fit earlier limits
May 13, 2025	AI Diffusion rule rescinded before it took effect
Aug 2025	Some H20 licenses granted; reports of a 15 percent revenue share to the US government
Dec 8, 2025	President allows H200 sales to approved Chinese buyers with 25 percent of revenue to the US
Jan 13, 2026	BIS moves H200 and AMD MI325X to case-by-case review with conditions
Jul to Aug 2026	BIS Under Secretary Jeffrey Kessler tells Congress (July 14) that H200 chips have shipped to China in "very few" numbers; Nvidia's August quarterly filing reports a $0.4 billion H200 charge and only "a fraction of the allowed shipments"

Washington and Beijing now both push on the same chips, with Washington deciding what may leave, and Beijing steering its companies toward domestic chips such as Huawei's Ascend. Analysts disagree about whether selling China older top-end chips slows its own chip industry or feeds its AI labs. The Institute for AI Policy and Strategy, for one, argues the 2026 licensing rules are tough on paper and weak in enforcement. Several 2026 shipment figures rest on anonymous sourcing, so attribute them.