AI Has Done It Again…

Every time you ask ChatGPT: "Help me write an email." or ask Claude: "How can I be rich like Elon Musk." or tell Gemini: "Summarize this document." ...the AI doesn't read your sentence word by word. Instead, it breaks everything into tiny pieces called tokens.

Think of tokens as a small chunk of text, smaller than a word and bigger than a letter. A useful rule of thumb is that one token is roughly four characters, so a 1,000-word piece of text is roughly 1,300–1,400 tokens. Every single one of those tokens must be processed by expensive AI hardware.

Here's where it gets interesting... Reading your prompt is relatively cheap. The AI can process much of your input simultaneously. But generating a response is different.

Imagine trying to recite the alphabet. You can't jump from A straight to Z. You must read it as it comes... AI works in almost the same way. It predicts one token at a time, using every previous token as context before generating the next one. A 500-token answer therefore requires roughly 500 sequential prediction steps.

So... whenever you prompt "write me a break-up text", the prompt is first broken down into tokens (small units of text), processed by large-scale computing infrastructure, which runs the model's calculations. This consumes computational resources, and the AI company incurs costs for that usage.

Across virtually every major AI provider, generating text costs several times more than simply reading your prompt.

For example:

- Claude Sonnet 4.5 charges about $3 per million input tokens versus $15 per million output tokens. - GPT-5.4 charges roughly $2.50 per million input tokens but $15 per million output tokens — about a sixfold difference.

There are two very different types of AI spending.

1. Training: This is the cost of building the model. Think of it as constructing Dangote Refinery. It costs an enormous amount upfront. But once construction finishes... that expense largely stops. 2. Inference: Inference is the actual running of the model to answer users' questions and generate tokens, every time someone uses it. This is like running the refinery every day — Fuel, Electricity, Maintenance. Everything required to keep operations going.

A few years ago, everyone talked about training costs. Today, inference has stolen the spotlight. Industry estimates show that roughly two-thirds of AI computing demand across major providers comes from serving real users rather than training new models, with many analysts estimating above 60% of ongoing AI infrastructure spending is now driven by inference. That means AI companies are increasingly spending money running existing models for users.

Here's the interesting part: The cost of generating one token keeps falling. The Stanford Institute of Human-Centered AI shows that the cost of running/querying a GPT 3.5 level model fell from $20 to $0.07 per million token — a 99.65% drop — between 2022 and 2024, thanks to better chips, improved software and more efficiency.

So, shouldn't AI become cheaper? Not necessarily — because people keep using much more of it. Imagine tomatoes become 80% cheaper, but everyone suddenly starts cooking five pots of stew every day. Your total grocery bill may still go up. That's exactly what's happening with AI.

Overall inference spending increased significantly not because tokens became more expensive, but because the number of tokens being generated exploded much faster than costs declined, following the rising and wide adoption of AI. This explains the massive capital expenditure we've been seeing. The goal isn't just to build smarter AI — it's to build enough capacity to serve billions of AI requests every day.

Collectively, the four largest U.S. technology companies (Amazon, Meta, Microsoft, Alphabet) are expected to spend well over $700 billion on AI infrastructure in 2026, a sharp increase from the previous year. A large share of that money goes into: high-performance GPUs, AI servers, networking equipment, data centres and power infrastructure. These aren't primarily for training the next groundbreaking model. They're increasingly for serving today's users fast enough that nobody must wait thirty seconds for AI to answer, "Who is the GOAT."

This is where the debate gets interesting. Nobody doubts AI demand is real. The concern is cost per user growing as fast as the user base itself.

Remember, every single query — yours, mine, everybody's — runs through that expensive, sequential, token-by-token generation process we just walked through. Unlike training, which is a one-time bill, inference doesn't stop. It scales with usage. More adoption doesn't dilute the cost — rather, it multiplies it, because every new user means millions more tokens being generated, every single day, forever.

That's the uncomfortable math sitting underneath the AI trade right now. If companies spend hundreds of billions today building capacity to serve all those tokens... how many years will it take before that spending is paid back?

And if inference costs keep climbing in step with adoption rather than shrinking fast enough to outrun it, profit margins could come under pressure even as usage keeps surging. In other words, the very success AI companies are chasing — more people using the product — is also what keeps driving their biggest expense higher.