Tokens Are the New Cloud Bill: The Real Economics of LLM Inference

Stanford's AI Index put a number on what API users had been feeling: querying a model at GPT-3.5's capability level cost about 280x less in November 2024 than it did in November 2022 — from $20 down to roughly 7 cents per million tokens. Epoch AI's broader analysis found prices for a fixed capability level falling anywhere from 9x to 900x per year depending on the benchmark. The mechanisms compound: better hardware (each accelerator generation delivering multiples of inference throughput), algorithmic efficiency (quantization, speculative decoding, mixture-of-experts serving), distillation (small models inheriting big-model capability), and brutal competition.

The competition deserves its own line. In January 2025, DeepSeek released R1 — a frontier-class reasoning model, open-weights, trained for a claimed ~$5.6M in final-run compute — priced at a small fraction of Western frontier rates. Nvidia lost ~$590 billion in market value in a single trading day, the largest one-day loss in stock market history, as investors briefly repriced what intelligence should cost. Whatever one thinks of the training-cost accounting, the effect was durable: open-weight models now set a price floor barely above raw GPU economics, and every closed provider prices against it.

The collapse has not stopped. As of July 2026, frontier-tier list prices sit around $5 per million input tokens and $25–30 per million output (GPT-5.5 at $5/$30, Claude Opus 4.8 at $5/$25), mid-tiers like Gemini 3.1 Pro at $2/$12 — while capable small models such as Gemini 3.1 Flash-Lite go for $0.10/$0.40. That is a ~60x spread between frontier and budget tiers you can exploit in a single afternoon of routing work.

The result is a strange market where the product gets better and cheaper simultaneously and permanently — a token of frontier capability today costs less than a token of mid-tier capability did eighteen months ago. If that were the whole story, AI budgets would be rounding errors by now. It is not the whole story.

The 19th-century economist William Stanley Jevons observed that as steam engines became more coal-efficient, Britain burned more coal, not less — efficiency made steam power economical for uses where it previously wasn't. LLM tokens are having the most extreme Jevons moment ever recorded, for three stacking reasons:

• Cheaper tokens unlocked token-hungry patterns. At 2022 prices, nobody would run a model in a loop. At 2026 prices, everybody does: a single agentic coding task routinely consumes 100k–1M+ tokens as the loop re-reads files, calls tools, and re-sends its growing context every iteration. Agents turned tokens from a per-request cost into a per-task cost with a fat tail.
• Reasoning models spend tokens as compute. Post-2024 models think before answering — and the thinking is billed as output tokens. A hard question can burn 10,000 invisible reasoning tokens to produce a 200-token answer. Output prices being 3–5x input prices makes this the most expensive kind of token precisely as models learned to emit far more of them.
• Usage went vertical. Token consumption across the industry grew orders of magnitude faster than prices fell — Google alone reported monthly token processing growing ~50x year-over-year into 2025. Price-per-token down 90%, tokens-per-user up 100x is a bill that grows 10x.

The practical consequence: the unit that matters is no longer the token but the task. "Input is $3 per million" tells you almost nothing about what resolving one support ticket or one coding task costs; loop length, context growth, and reasoning budgets dominate. Teams that track cost-per-completed-task discover 100x differences between naive and engineered implementations of the same feature.

Four structural facts explain most real-world LLM bills, and all four are levers:

• Input dwarfs output in agents. Because the loop re-sends conversation history every iteration, input tokens typically account for 80–95% of an agentic workload's bill even though output tokens cost 3–5x more per token. Context is quadratic-ish: a 20-turn task re-transmits early turns twenty times.
• Prompt caching is the biggest single discount. Providers charge 50–90% less for input tokens they've recently processed (cache reads are 90% off at Anthropic; 50–75% off elsewhere). For agents — where the system prompt, tool definitions, and history prefix repeat every iteration — caching alone routinely cuts bills 50–80%. It's opt-in structure, not magic: stable prefixes cache; prompts that interleave volatile content (timestamps, request IDs) into the prefix don't.
• Batch APIs halve anything async. Both major providers offer ~50% off for jobs that can tolerate up-to-24-hour turnaround — evals, backfills, classification pipelines, embedding runs. Anything without a user waiting should be batched by default.
• Model routing is a 10–60x lever. The price spread between a frontier model and a fast small model from the same provider is one to two orders of magnitude — at July 2026 list prices, GPT-5.6's own tiers span $5/$30 (Sol) down to $1/$6 (Luna), and Gemini 3.1 runs from $2/$12 (Pro) to $0.10/$0.40 (Flash-Lite). Most production request mixes are majority-easy: classify, extract, reformat, answer-from-context. Cascades — try cheap, escalate on low confidence — and per-route model choice are where the largest savings live, and the small models of 2026 are better than the frontier models of 2024.

A worked example, agentic support bot, 10,000 tasks/month: naive implementation (frontier model everywhere, no caching, full history) ≈ 40M input + 2M output tokens per 1,000 tasks. Engineered implementation (cached prefix, compaction at 50% of context, cheap model for triage and summarization, frontier only for resolution) commonly lands at a fifth to a tenth of the cost with equal task success — the difference between a $6,000/month and a $700/month line item. That engineering is now a permanent part of shipping AI features, the way query optimization is part of shipping databases.

The consumption explosion collided with subscription pricing in mid-2025, and the wreckage taught the industry its unit economics in public:

• Cursor — the fastest-growing dev tool in history — replaced its $20/month "500 fast requests" plan with an opaque usage-based scheme in June 2025, applied it to existing subscribers, and faced a community revolt severe enough to force a public apology and refunds. The underlying math was simple: agentic power users on frontier models were consuming hundreds of dollars of inference against a $20 subscription.
• Anthropic added weekly caps to Claude Code's flat plans after discovering exactly what unlimited plus agents produces: a handful of users running the tool continuously, some reportedly consuming tens of thousands of dollars in inference on $200/month subscriptions.
• Replit, Lovable, and the vibe-coding wave all converged on the same destination from different starts: credits, effort-based pricing, or metered agent minutes. Nobody who survived kept true unlimited.

The structural lesson generalizes beyond dev tools. Flat-rate pricing works when marginal cost per user is near zero and usage follows a tight distribution — SaaS conditions. Agentic AI violates both: marginal cost is real (the provider pays per token) and usage follows a power law where the top 1% of users can consume 100x the median. Under those conditions a flat price is an uncapped short position against your heaviest users. The pricing models that survived 2025 all share one property: the price the customer pays scales, at least loosely, with the tokens the vendor burns. Anyone building an AI product should read their own pricing page against that test before their power users do it for them.

The durable playbook, condensed from the teams whose AI features have gross margins instead of gross surprises:

• 1. Instrument cost-per-task from day one. Tag every API call with a task ID; report spend per completed outcome, not per month. The power-law user or runaway loop shows up in this metric weeks before it shows up in the invoice.
• 2. Structure prompts for the cache. Stable system prompt and tool definitions first, volatile content last. Verify cache-hit rates in production — a 90% discount you're not actually getting is the most common silent leak.
• 3. Route by difficulty, escalate on failure. Small-model-first cascades with confidence thresholds. Re-benchmark quarterly: the cheap tier improves fast enough that routing tables rot in months.
• 4. Budget every loop. Max iterations, max spend per task, context compaction thresholds, and a defined out-of-budget behavior (summarize and escalate — never silently retry). An unbudgeted agent is an unbounded liability wearing a product feature's clothes.
• 5. Batch everything without a human waiting. The 50% discount for patience is the easiest money in the stack.
• 6. Price your product like your costs are real — because they are. Meter, cap, or credit-gate anything agentic. The 2025 repricings weren't greed; they were arithmetic arriving late.

Zoom out and the shape of the era is clear: capability per dollar improves relentlessly, consumption per product improves faster, and the margin lives in the gap. The gap is now visible at industry scale — mid-2026 saw Anthropic report $30B+ in annualized revenue (passing OpenAI's ~$25–33B, though the two dispute each other's accounting), Nvidia post an $81.6B quarter with data-center revenue up 92% year over year, and the four big hyperscalers guide roughly $725B of 2026 capex, up ~77% in a year, to serve demand that keeps arriving. Cloud computing spent fifteen years teaching companies FinOps. Inference is compressing the same lesson into about three — and the teams that treat tokens as a first-class engineering budget, with the same rigor as latency or memory, are the ones whose AI features survive contact with their own success.