---
title: "Pre-training: what it means · The AI Compute Race · DRYBURGH"
source: https://dryburgh.com/vol1/glossary/pre-training
description: "Pre-training: The field's word for the training run itself, in which a model learns from its data, such as human text."
---

# Pre-training

What does “Pre-training” mean?

Pre-trainingThe field’s word for the training run itself, in which a model learns from its data, such as human text. Post-training is the work done on the model after it, such as reinforcement learning. *Chapter 2.*

From *The AI Compute Race*. [Read the book](https://dryburgh.com/vol1#buy)

## Where the book uses it

- [Chapter 2 — The Discovery That Changed Everything](https://dryburgh.com/vol1/notes/chapter-2)How a formula about compute and capability became a bill: the scaling laws, the first supercomputers built to test them, and why the money came before the paper.

## Sources that use it

- Jan Kocoń and 98 co-authors of the PLLuM consortium (Wrocław University of Science and Technology, NASK, OPI, IPI PAN, IS PAN and the University of Łódź), “PLLuM: A Family of Polish Large Language Models,” arXiv 2511.03823, version 1 of 5 November 2025 (“The project received funding from the Ministry of Digital Affairs in 2023, and full-scale development began in early 2024”; “Pre-training experiments were conducted on the WCSS LEM cluster, which features NVIDIA 304 H100 GPUs, as well as the Cyfronet Helios cluster, which features 440 NVIDIA GH200 GPUs”; Table 19, every publicly released model “Based On” Llama 3.1, Mistral-Nemo-Base-2407 or Mixtral-8x7B-v0.1). [In the Sources](https://dryburgh.com/vol1/sources/research#pllum-paper-nov25)

## Related terms

- [Reinforcement learning](https://dryburgh.com/vol1/glossary/reinforcement-learning)
- [Post-training](https://dryburgh.com/vol1/glossary/post-training)
- [Training](https://dryburgh.com/vol1/glossary/training)
- [API](https://dryburgh.com/vol1/glossary/api)
- [Docket](https://dryburgh.com/vol1/glossary/docket)
- [Floating-point operation](https://dryburgh.com/vol1/glossary/floating-point-operation)
- [Loss](https://dryburgh.com/vol1/glossary/loss)
- [Power law](https://dryburgh.com/vol1/glossary/power-law)
- [Reasoning model](https://dryburgh.com/vol1/glossary/reasoning-model)
- [Scaling laws](https://dryburgh.com/vol1/glossary/scaling-laws)
- [Transformer](https://dryburgh.com/vol1/glossary/transformer)

From the glossary of The AI Compute Race.

- [Campuses](https://dryburgh.com/vol1/ledger)
- [Tests](https://dryburgh.com/vol1/tests)
- [Scorecard](https://dryburgh.com/vol1/scorecard)
- [Data](https://dryburgh.com/vol1/data/releases)
- [Chapter by chapter](https://dryburgh.com/vol1/companion)
- [The record at a glance](https://dryburgh.com/vol1/observatory)
- [The machine](https://dryburgh.com/vol1/machine)
- [One campus, four futures](https://dryburgh.com/vol1/hyperion)
- [Glossary](https://dryburgh.com/vol1/glossary)
- [People](https://dryburgh.com/vol1/people)
- [Sources](https://dryburgh.com/vol1/sources)
- [Notes](https://dryburgh.com/vol1/notes)
- [Where the book was wrong](https://dryburgh.com/vol1/corrections)
- [Where its sources were wrong](https://dryburgh.com/corrections?book=vol1)
- [The Paper](https://dryburgh.com/vol1/paper)
- [Consent](https://dryburgh.com/vol1/consent)
- [Who builds and pays](https://dryburgh.com/vol1/parties)

*The AI Compute Race: Five Races, One Machine* is a book by Lee S. Dryburgh, the first volume of THE COMPUTE ECONOMY, published by Dryburgh, an imprint of Contestra OÜ, Tallinn, Estonia; its record closes on 8 November 2026.

[About the book](https://dryburgh.com/vol1)[Tell me when it is out](https://dryburgh.com/vol1#notify)