Running a trained model for the people who use it. Training builds a model, and inference runs it. Chapter 1.
From The AI Compute Race. Read the book
Where the book uses it
- Chapter 1 — The Stakes
What the race is for: the belief that a lead in AI can compound into a lasting advantage, how long such leads have actually lasted, and what a trillion dollars a year is being bet against.
Where the record uses it
- Person Dally, Bill
Sources that use it
- Groq, “Groq and Nvidia Enter Non-Exclusive Inference Technology Licensing Agreement to Accelerate AI Inference at Global Scale,” 24 December 2025 (“Jonathan Ross, Groq’s Founder, Sunny Madra, Groq’s President, and other members of the Groq team will join Nvidia”; “GroqCloud will continue to operate without interruption”; no price disclosed); the $20 billion: CNBC, 24 December 2025 (“for $20 billion in cash, according to Alex Davis, CEO of Disruptive, which led the startup’s latest financing round in September” — the investor’s figure, not NVIDIA’s; Jensen Huang: “we are not acquiring Groq as a company”; Groq founder and CEO Jonathan Ross along with Sunny Madra, the company’s president, and other senior leaders “will join Nvidia to help advance and scale the licensed technology”). In the Sources
- Groq, “Groq Closes $350 million Series A, Building the World’s Leading AI Inference Cloud,” 17 August 2026 (a $350 million Series A at a $3.5 billion valuation, led by Disruptive “with planned participation from NVIDIA”; “This latest round, together with $650 million raised in June 2026, brings recent funding in the company to $1 billion”; “Groq is an NVIDIA Cloud Partner (NCP), certified to design, deploy, and operate NVIDIA accelerated computing”; “Groq today operates 13 data centers across North America, Europe, the Middle East, and Asia Pacific”). In the Sources
- Groq, “Groq Raises $650M to Scale Its AI Inference Cloud Business,” 22 June 2026 (“$650 million in new growth capital to accelerate the expansion of its AI inference cloud”; “The round was led by Disruptive and Infinitum, with participation from investors who elected to reinvest in the company”). In the Sources
- Moonshot AI, “Kimi K3 Tech Blog: Open Frontier Intelligence,” Kimi blog, July 2026 (“Pricing is $0.30/MTok for cache-hit input, $3.00/MTok for cache-miss input, and $15.00/MTok for output”, that is, $3 and $15 per million tokens of input and output; the post prints no date of its own, gives its benchmark scores as “current as of July 16, 2026” and promises the full weights “by July 27, 2026”; Moonshot’s API price card, “Model Inference Pricing Explanation,” Kimi API Platform, listed the same $0.30, $3.00 and $15.00 for kimi-k3 on 20 September 2026, with cache-write prices of $3.00 and $6.00 that the post does not give; read live 20 September 2026). In the Sources
- Andrew Feldman, co-founder and chief executive of Cerebras, interviewed by Matt Turck, “Cerebras CEO: Why GPUs Can’t Do Fast Inference,” The MAD Podcast with Matt Turck, FirstMark, YouTube, 23 July 2026 (a seller of the wafer-scale alternative to the GPU, speaking of his own product and of his competitors’ supply, with no independent benchmark in the interview; of the memory made by three companies, “they’re sold out. That’s the number one problem. We don’t use it” 19:41–19:46; of TSMC’s CoWoS packaging, “That process is sold out, is highly constrained. We don’t use it” 20:38–20:45; “we don’t use the 3-nanometer technology” 21:00, his chips being at 5 nanometers; of the push for more CPUs, “is being driven by AI on machines like ours and GPUs doing agentic work” 24:14–24:22; of datacenter availability, “is a limiting factor for everybody” 55:58; of the deal with OpenAI, “because data center capacity is a limiting constraint” 56:16–56:20, measured in megawatts; “It takes you eight keystrokes to move from a GPU to us in the cloud” 1:02:20–1:02:26, after his sentence that inference has no moat; of TSMC at 1:04:04, close collaboration, an investor in the company and an exceptional partner, paraphrased; of diversifying away from the foundry at 1:08:26, very hard, since part of designing a chip is designing for the foundry’s rules, paraphrased; the quotations read against two transcriptions of the audio held by the set, which carry three words the automatic caption drops, the word moat taken on the term and the speaker’s use of it elsewhere in the interview where the caption and both transcriptions render it otherwise, the 3-nanometer clause held on the caption and the sentences either side of it, the foundry’s name at 1:04:04 the host’s, TSMC on both transcriptions of 3 October 2026, and at 1:08:26 not heard cleanly and so not quoted). In the Sources
And 8 more: search the Sources.
Related terms
From the glossary of The AI Compute Race.
How to cite this page
Dryburgh, Lee S. “Inference.” In The AI Compute Race. DRYBURGH. https://dryburgh.com/vol1/glossary/inference (accessed 2026-11-01).
BibTeX
@misc{dryburgh-inference-2026,
author = {Dryburgh, Lee S.},
title = {{Inference}},
howpublished = {In \emph{The AI Compute Race}, DRYBURGH},
year = {2026},
url = {https://dryburgh.com/vol1/glossary/inference},
note = {Accessed 2026-11-01}
}