The token gap

Last October we were in the audience at OpenAI’s DevDay. At one point Sam Altman stepped out in front of a wall of names: the developers who had processed the most tokens through the API. Ten billion tokens over a lifetime of building earned your name on that wall and a public tribute from Sam Altman himself.

Eight months later, this June, we watched a single user on Umans AI run nearly eleven billion tokens in a single day.

That is how fast demand for tokens is moving. The rest of this article is about what that means at market scale: how fast demand grows, why compute supply cannot keep up, and why closing that gap is the work we chose at Umans AI.

1. Token demand grows faster than compute supply

Eleven billion tokens in a day is where the whole market is heading, just a few months early. Demand for tokens grows about an order of magnitude every year. Measured growth rates range from 7x to 30x per year depending on the vantage point, from Google’s own disclosures to OpenRouter’s traffic, and the best market-wide estimate sits around 10x. The world’s AI compute supply, counting both more chips and better chips, grows 3.4x per year on Epoch AI’s measurements.

Run the two rates forward from today’s serving base and they cross. Somewhere between late 2026 and mid 2027, on central estimates, demand passes what the installed base can serve at today’s efficiency, and the gap widens from there.

Those two rates do not close on their own. For demand to remain servable, tokens delivered per chip must improve 2x to 9x every year, on top of all hardware progress. That number is not a forecast; it is what dividing the two growth rates gives.

2. Why hardware cannot catch up alone

The supply curve is not slow for lack of money or ambition. It is slow because it runs through a chain of physical choke points, and the chain narrows toward single companies.

One company on earth builds the lithography machines every advanced chip depends on, and it ships a few dozen per year. One foundry fabricates over 90% of leading-edge chips, and its challengers trail on the one metric that decides everything: yield. Behind fabrication sits a second, independently constrained step, advanced packaging, sold out industry-wide for years. A new fab costs tens of billions and takes years to qualify. A gigawatt data-center site takes about two years to energize, behind lengthening grid queues.

This chain already runs at maximum effort, under the industrial policy of every major power. 3.4x per year is what physical manufacturing delivers at full capacity. The remaining 2x to 9x must therefore come from the only layer that ships in weeks instead of years: the engineering that turns GPU time into delivered tokens. The headroom there is real. NVIDIA’s own AI factory numbers show software improvements alone cutting token cost 5x in two months on the same hardware.

3. Agentic work is where the economic value is

In a market in tension, every GPU-hour is an allocation decision. Compute that serves low-value tokens is compute taken away from high-value ones, so the scarcer compute gets, the more it matters where it goes.

Where it is worth going is where AI does actual work. Agentic tokens write the code, run the analysis, complete the task: they are what makes automating high-value work possible, and that is what the whole chain exists to produce. A cheap token that fails the task is not cheap; it is scarce compute wasted. A more expensive token that finishes the job is the better allocation.

So the problem worth solving is precise: make the tokens that do valuable work abundant and cheap to produce. That is efficient agentic inference, and it is why we focus there.

4. Where we sit: efficient agentic inference

Focus is structural. Pure-play agentic inference means every engineer-hour compounds into a single asset: the delivered-token frontier, how many tokens a GPU produces at each level of per-user speed. And agentic inference is its own discipline. Look back at the usage screenshot that opened this article: 99% of those eleven billion tokens were served from cache. That is what agentic work looks like from the serving side. An agent rereads a large, growing workspace to emit a few hundred tokens, so serving it well is a prefill and cache problem, a different problem from chat, and the one our stack is built for.

The same focus shapes the lineup. We serve a short list of frontier models, only models capable of valuable work, and change it carefully. When a high-value model lacks a capability worth having, we add it ourselves, as we did with vision on DeepSeek V4 Flash.

Efficiency compounds. Serving trillions of agentic tokens a month, we improve the stack continuously, and the result is public: the same GPUs deliver more tokens every week on our metrics. Every model is validated in Labs with our community before it earns production traffic.

The results are public too. We have served DeepSeek V4 Flash faster than anyone since the day the weights were released, the model maker included: their published frontier ends near 230 tokens per second per user, and provider tracking tops out well below 300, while our production median runs at 367. For agentic serving, our delivered-token frontier is the highest documented anywhere; Tokenomics, behind the scenes walks through the measurements. And it runs as a service, not a benchmark: redundancy and cache-aware failover are priced into our token economics, because only tokens delivered under the promised service level count.

5. Efficiency is what democratization runs on

On the current trajectory, demand outgrows what hardware alone can serve; the figures in section 1 show it. In that world, serving efficiency is what keeps valuable tokens affordable, and automated work accessible beyond the few who can outbid everyone else for compute. Efficiency is what democratization runs on.

The wall at DevDay celebrated ten billion tokens as a lifetime of work. A few months later that is one motivated user’s Monday, and the curve does not stop there. Keeping that curve affordable to ride is the problem we chose.