Transformer

Where the book uses it

• Chapter 2 — The Discovery That Changed Everything

  How a formula about compute and capability became a bill: the scaling laws, the first supercomputers built to test them, and why the money came before the paper.

Where the record uses it

• Scorecard indicator Generator step-up transformer lead time
• Dated test Chapter 6, the transformer, due 31 December 2028
• Dated test Chapter 12, claim 2, the relay, due 31 December 2028
• Dated test Chapter 14, the movement sideways, toward commodity, due 31 August 2029
• Dated test Chapter 6, the transformer, due 31 December 2029

Sources that use it

• Dwarkesh Patel with Gavin Leech, The Scaling Era: An Oral History of AI, 2019–2025 (Stripe Press, 2025) (curated interviews from the Dwarkesh Podcast, 2019–2025, the book’s own note on its text being “The interview excerpts have been edited for clarity and flow”, so each quotation here is the editors’ text, and where the recording differs the chapter cites the podcast’s own row; Amodei’s unpublished 2017 OpenAI memo “The Big Blob of Compute”; Zuckerberg, 2024, as the editors print him: “I think it’s worth investing $100 billion plus in building the infrastructure”, where the podcast’s transcript and the recording have the $10Bs or $100B+ and “and assuming that if it keeps going, you’re going to get some really amazing things… I don’t think anyone in the industry can really tell you for sure that it will continue scaling at that rate” (chapter 5, the ellipsis for “that make amazing products”); Zuckerberg, 2024, energy the coming bottleneck: “No one has built a 1 GW data center yet”, the recording’s a single gigawatt, and of new power plants and transmission lines, “You’re talking many years of lead time”; Dylan Patel, 2024, in the editors’ text and found on no published recording or transcript of the podcast: “In 1990, ExxonMobil was not laying fiber. In 2025, Microsoft is building data centers”; “prior bubbles didn’t have the most profitable companies that humanity has ever created investing in them. They were debt financed instead. AI is not debt financed yet”, the editors’ wording of the recording’s they were debt financed. This is not debt financed yet; Dylan Patel, 2024, in the editors’ text and found on no published recording or transcript of the podcast: “If human capital is infinite, which is what AGI is, then theoretically the returns of AGI are infinite. If I’m Mark Zuckerberg or Satya Nadella, I now have potentially infinite returns — if I get there first” (chapter 6, the book’s closed dash spaced); Aschenbrenner, 2024: a 10 GW cluster “a couple hundred billion”, the recording’s maybe before it dropped by the editors, the payoff test “you have to get to, like, $100 billion a year in revenue”, the podcast transcript’s form of the recording’s eventually you got to get to like 100 billion a year; Aschenbrenner, 2024: “Let’s estimate the GPT-4 cluster at around $500 million”, and of the $100 million figure, “That’s just the rental price. If you’re building the biggest cluster, you have to build and pay for the whole cluster”, the second sentence the podcast transcript’s form of the recording’s you got to build the whole cluster, you got to pay for the whole cluster; Aschenbrenner, 2024, on buying an aluminum smelter’s gigawatt: “a lot of these power contracts are locked in long term. Also, people don’t like it”, the recording’s people don’t like things like this; Aschenbrenner, 2024: “Would you do the Manhattan Project in the UAE?”, “Once the cluster is there, it’s much easier for them to exfiltrate the weights”; Shulman: a percentage point of search share “a billion-dollar matter”, $50–70 trillion of annual wages as the prize, and “the chance of advanced AI happening is relatively concentrated in the next 10 years [between 2024 and 2034] compared to the rest of the century, because we can’t keep going with our current rapid redirection of resources into AI. That’s a one-time thing”; Amodei: datacenter networks “cost as much as aircraft carriers”, a forecast on the recording (we’re anyway heading to a world where), which the editors set as already; Legg’s 50 percent by 2028; Sholto Douglas, March 2024, as the editors print him: “That is exactly the problem. But the key issue is that your base task solve rate is 90 percent. If it was 99 percent, it wouldn’t be a problem”; of the memo, “an unpublished memo he wrote at OpenAI in 2017—before scaling laws, before the Transformer, and before LLMs”, the editors’ order, where the recording they draw on has Amodei saying of the transformer paper It might have been just before. It might have been just after; chapter 5, “Inputs”, the same exchange as the book edits it: “You can put the clusters in the US and in allied democracies. Once you put them in authoritarian dictatorships, you create this irreversible security risk”; “Another thing: They can just seize the compute”; “Suppose we put 25 percent of the compute capacity in these Middle Eastern dictatorships. Say they seize that. Now the US has a 3-to-1 compute ratio.”). In the Sources
• Cleveland-Cliffs, “Cleveland-Cliffs Announces Its New State-of-the-Art Electrical Transformer Production Plant in Weirton, West Virginia,” 22 July 2024 (“American-made GOES, exclusively produced in the United States by Cleveland-Cliffs, at its Butler Works steel mill in Butler, Pennsylvania”). In the Sources
• Google Research, “Accelerating Deep Learning Research with the Tensor2Tensor Library,” 19 June 2017 (a Transformer implementation and reproduction recipes released seven days after the paper’s preprint — the first organizational order the paper produced, and it was software, not hardware). In the Sources
• OpenAI, “Improving Language Understanding with Unsupervised Learning,” blog post, 11 June 2018 (“Our approach requires an expensive pre-training step - 1 month on 8 GPUs” and “we used a 37-layer (12 block) Transformer architecture”; read in the Internet Archive’s copy of the post at its original blog address). In the Sources
• US Department of Energy, “Large Power Transformer Resilience,” Report to Congress, signed 10 July 2024 (§II.1: “36-month lead times being commonly quoted and maximum lead times reaching as much as 60 months”; p. 11: domestic producers indicating lead times “up to and exceeding 36 months”). In the Sources

And 5 more: search the Sources.

Related terms

• Generator step-up transformer
• API
• Docket
• Floating-point operation
• Loss
• Power law
• Pre-training
• Reasoning model
• Reinforcement learning
• Scaling laws