Long read: Why Token Optimization Is a Gift to the Hyperscalers

Rihard Jarc is a former CEO and founder of an AI startup and now a tech investor. In this two part essay, he talks about the recent development where many enterprises have blown up their AI budgets (often referred to as token maxxing) without much to show in terms of benefit and what that means for the AI trade. In the first part of the essay, he argues “that the AI market will bifurcate: the frontier model goes to the small slice of work where intelligence is unbounded in economic value (drug discovery, novel math, the hardest agentic reasoning), and the middle of the economy — classification, extraction, summarization, routine code, support — runs on the cheapest model that clears the quality bar.”

In the sequel, he argues why this shift from token maxxing to token optimisation is positive for hyperscalers – Amazon (AWS), Microsoft (Azure) and Google (Cloud) using an analogy:

“Think about a highway toll road. There are two businesses operating on it. The first is the company that manufactures the cars — they make the actual machine that does the work of getting you somewhere, and there’s a real margin in a car. The second is the company that owns the tollbooth. The tollbooth owner doesn’t care what car you drive. Ferrari, Toyota, or a 12-year-old used Honda. Every one of them pays the same toll to cross the bridge. Now imagine a world where, suddenly, everyone realizes they were commuting to work in a Ferrari for no reason, and they all downgrade to the cheaper Honda to save money. The carmaker’s revenue per vehicle collapses, but people don’t drive less because they switched to a cheaper car. They drive more, because now it’s cheap enough to justify trips they’d never have taken before. And every one of those extra trips still crosses the bridge. The tollbooth’s revenue goes up.

In the AI stack, the AI labs are the carmakers. The hyperscalers are the tollbooths. And in the next months, we are entering a period where most of the economy is trading down from the Ferrari to the Honda, while simultaneously driving 10x more miles.”

This framing suggests the value capture in the AI stack resides with the hyperscalers. But will token optimisation not make the massive investments of hyperscalers in compute infrastructure redundant? The author argues otherwise.

“…This is token optimization, and it doesn’t reduce total token consumption — it accelerates it. The moment inference gets cheap enough, you stop rationing it. You run the agent in a loop. You let it read the whole codebase. You re-run it five times and vote on the answer. A single coding-agent session now chews through millions of tokens of context where a chatbot query used a few thousand.

You can see this in the hard numbers the hyperscalers themselves disclose. Microsoft said it processed over 100 trillion tokens in a single quarter in 2025, up 5x year-over-year, with a record 50 trillion in one month alone. By its fiscal Q3 2026 call, Microsoft said over 300 customers were on track to process more than a trillion tokens each on Foundry this year — and that this was accelerating 30% quarter-over-quarter. Google went from 480 trillion tokens per month at I/O in May 2025, to 980 trillion by July, to 1.3 quadrillion by October — and in its Q1 2026 filing disclosed that its first-party models alone were processing more than 16 billion tokens per minute via direct API, up 60% in a single quarter.

At the same time, the price per unit of capability is falling substantially— the Stanford HAI AI Index found inference cost for GPT-3.5-level performance fell more than 280-fold in two years, and a16z pegs the decline at roughly 10x per year for any fixed capability level. And yet, total tokens processed are growing several-fold per year.”

He argues that whilst a leading frontier model might still retain its margins, for most others, margins go to zero. “…because the model is open-weight and nobody is charging a brand premium for it. But the token still has to run on somebody’s GPUs, inside somebody’s data center, behind somebody’s managed API with its security, compliance, logging, and SLAs. And that somebody are the hyperscalers. They still charge their full infrastructure margins on the token. AWS has historically run a roughly 35–38% operating margin. Google Cloud, which lost money for years, posted operating margins above 33% in Q1 2026 and is still climbing. That margin doesn’t care whether the token came from a $50/million frontier model or a $1/million open-weight model. To return to the analogy that from before, the tollbooth charges the same toll regardless of the car.

So here is the structural beauty of it for the hyperscalers, and the structural danger for the labs:
Per-token economics compress, but the compression lands almost entirely on the model layer, not the infrastructure layer. The AI lab’s margins on simple workloads get squeezed. The hyperscaler’s infrastructure margin is sticky.
Total token volume explodes, and almost every single token crosses the hyperscaler’s tollbooth. More usage, on a partly-depreciated, increasingly efficient installed base, means absolute infrastructure revenue and gross profit dollars go up even as the price of any individual token falls.
This is Jevons’ paradox pointed directly at the cloud P&L. The hyperscalers squeeze more revenue out of the infrastructure they already own, and they don’t need the model-provider margin to do it. Outside of Google, they were never in that business in the first place.”

Amazon, Microsoft, and Alphabet Inc. (Google) are part of holdings within the Marcellus Global Compounders Portfolio, a strategy offered by the IFSC branch of Marcellus Investment Managers Private Limited and regulated by the IFSCA. Accordingly, Marcellus, its employees, their immediate relatives, and clients may maintain interests or positions in these securities. Any reference to these companies is intended strictly for informational and educational purposes within the context of this discussion and should not be construed as investment advice.

If you want to read our other published material, please visit https://marcellus.in/blog/

Note: The above material is neither investment research, nor financial advice. Marcellus does not seek payment for or business from this publication in any shape or form. The information provided is intended for educational purposes only. Marcellus Investment Managers is regulated by the Securities and Exchange Board of India (SEBI) and is also an FME (Non-Retail) with the International Financial Services Centres Authority (IFSCA) as a provider of Portfolio Management Services. Additionally, Marcellus is also registered with US Securities and Exchange Commission (“US SEC”) as an Investment Advisor.