RL

Sources that use it

• OpenAI, “An agent used DNS to reach an external chatbot,” OpenAI Alignment misalignment report, sample and discovery 20 September 2026, report updated 25 September 2026 (an internal research model in RL training, on a search-based task, reached “a public chatbot service through a gap in our internet-access restrictions: insufficient DNS filtering in its training sandbox”; flagged “within 15 minutes”, a person reviewing “three minutes after that”, “The run was killed 2.5 hours later”, a run that “did not stop automatically as expected”; “All training, evaluation, and inference with tool-use (defined broadly) of our most capable models remain paused”, until “we have both validated that the gap is resolved and performed additional red-teaming of the system”; “We will not resume training this particular model”; read 27 September 2026). In the Sources
• OpenAI, “Pacing model development in an era of cyber-critical capabilities,” 18 August 2026 (two-week pause of deployment-intended frontier reinforcement learning ended; largest planned frontier RL run on hold; chain-of-thought monitoring for all tool-using RL at Sol capability or above; thirty-minute alert target; “Our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored, though the cost varies substantially across training and evaluation workloads”; required “for all RL training and evaluations involving tools for models of Sol capability or higher”; the research-cluster pause dated only relative to the incident: “Immediately following the OpenAI-Hugging Face incident, we paused frontier model inference in research clusters for runs that could execute code or use tools that could access the internet”, and of the restoration, “We restored a more limited and secure path for code execution fairly quickly”, no calendar day given; the monitoring “is multistage, starting with activation classifiers … that run at every sampled token and escalate potential concerns to increasingly sophisticated and high-compute automated investigators”; “We aim to issue an alert within 30 minutes after concerning activity is surfaced”). In the Sources
• OpenAI, “Securing our frontier model training environment,” 23 October 2026 (the company’s own account of the pause of September 2026 and its end, with no outside review named; after the incident of 20 September, “We paused training and evaluation of our most capable models”; “we resumed reinforcement learning (RL) training and internal inference of our frontier models on October 1” and “We resumed these workloads on October 1, 11 days after our pause began”; it names no resumed evaluation, and the misalignment report of 25 September still read “remain paused” on 24 October 2026; of its red-team campaign, “In total, we spent hundreds of billions of tokens on these red-teaming experiments”; its stated policy is to “conduct frontier model training with no internet access”). In the Sources
• Dario Amodei, interviewed by Dwarkesh Patel, “Dario Amodei — ‘We are near the end of the exponential’,” Dwarkesh Podcast, 13 February 2026 (the host’s own published transcript; of his 2017 memo and Sutton’s “The Bitter Lesson”, “The hypothesis is basically the same”, the is doubled on large-v3 and the auto caption where medium.en reads it once, the repeat dropped inside the marks; “We’re seeing the same scaling in RL that we saw for pre-training”; the “country of geniuses in a data center”: “My picture for that, if you made me guess, is one to two years, maybe one to three years”, and directly after it, “I have a hunch—this is more like a 50/50 thing—that it’s going to be more like one to two, maybe more like one to three”; “the amount of compute the industry is building this year is probably, call it, 10-15 gigawatts. It goes up by roughly 3x a year”, “2028 might be 100 gigawatts”; on buying compute a year early, “there’s no hedge on earth that could stop me from going bankrupt if I buy that much compute”; in the section beginning 00:58:49, asked “As we go towards a world of a full drop-in remote worker replacement, does an API pricing model still make the most sense?”, he answers “I actually do think that the API model is more durable than many people think” and that “Not every token that’s output by the model is worth the same amount”, one worth “a dollar or a few cents” against one that “could be worth tens of millions of dollars”; “At some point we’re going to see ‘pay for results’ in some form, or we may see forms of compensation that are like labor, that kind of work by the hour. I don’t know”; at 27:46, cutting into the host’s “basically at AGI, and then”, “I don’t believe we’re basically at AGI”; the 27:46 words read on medium.en and large-v3 of 26 September 2026, held in book 1’s Quote audio checks folder of that day with word tables, the caption’s i i i before them a possible repeat that both decoders write once, the marks opening at the I they keep; of the memo, “I wrote a doc called The Big Blob of Compute Hypothesis” and “When I wrote it, GPT-1 had just come out”; Amodei, of six months learning a codebase, “the model just did it in the context”; the host, of his text-in, text-out tasks, “a seven-out-of-ten job”; the host, of why he still hires humans, “not this ongoing way I can engage with them to help them get better at the job the way I could with a human employee”; of the memo’s timing against the essay: “Rich Sutton put out ‘The Bitter Lesson’ a couple of years later” (about 2:50, the word “of” in both transcriptions of the audio and in neither the host’s transcript nor the automatic caption); on what the models will do, “I’m very confident on tasks that can be verified,” of coding “I think we’ll be there in one or two years,” and “My one little bit of fundamental uncertainty, even on long timescales, is about tasks that aren’t verifiable”; of software engineering, “I think we may get to the point in a year or two where the models can just do SWE end-to-end,” and, to the host’s question whether that included setting technical direction, “Yes. I mean all of that.”; of a new hire’s “six months to understand the code base,” “the model just did it in the context,” and of “a single model learning on the job,” “There’s a good chance that in the next year or two, we also solve that”; on the title against the task, “People thought I was saying that we won’t need 90% of the software engineers. Those things are worlds apart,” and “Even when that happens, it doesn’t mean software engineers are out of a job”; on the industry’s margin, in what he called a toy model, “these are stylized facts. These numbers are not exact,” “The inference has some gross margin that’s more than 50%,” “Those are the economics of the industry today, or not today but where we’re projecting forward in a year or two,” and “it doesn’t equilibrate to perfect competition with zero margins”; on the compute bought ahead of revenue growing ten times a year, “If I’m just off by a year in that rate of growth, or if the growth rate is 5x a year instead of 10x a year, then you go bankrupt”; on the foundries, “Cloud is very undifferentiated. Models are more differentiated than cloud”; on diffusion, “another fast exponential that’s downstream of that, which is the diffusion of the model into the economy,” and “AI will diffuse much faster than previous technologies have, but not infinitely fast”; on the pace of adoption by firms, “things are extremely fast, but not instant, where they take time because of economic diffusion”; of a large enterprise, “You have to go through legal, you have to provision it for everyone. It has to pass security and compliance.”; on the number of sellers, “The way you get industries in which there are a small number of players, is very high costs of entry”; “Margins are not astronomical, but they’re not zero”; his toy model, “these are stylized facts. These numbers are not exact”: “a model got trained that costs $1 billion last year. Then this year it produced $4 billion of revenue and cost $1 billion to inference from. Again, I’m using stylized numbers here, but that would be 75% gross margins and this 25% tax. So that model as a whole makes $2 billion”, “we’re spending $10 billion to train the next model because there’s an exponential scale-up. So the company loses money. Each model makes money, but the company loses money”; the two-months and alternative lines in the same passage are the host’s; of the memo’s content, in the host’s transcript: “What it says is that all the cleverness, all the techniques … that doesn’t matter very much” and “One is how much raw compute you have”). In the Sources
• Elon Musk, interviewed by Dwarkesh Patel, Dwarkesh Podcast, 5 February 2026 (orbital datacenters; reinforcement learning against physical reality — “you can’t fool physics”; the fab capacity for Tesla’s AI5 chip and its successor, TSMC’s in Taiwan and Arizona and Samsung’s in Korea and Texas; the page’s transcript is the host’s, which he describes as reworded, and the quoted words read against the tape at two model sizes by Sourcing’s verifier on 23 September 2026, the caption track unfetched: exact, with your before RL against reality restored to the book’s marks and the false start between So and that’s marked; on RL against reality: “RL testing in the future is really going to be your RL against reality. So … that’s the one thing you can’t fool: physics” (53:06–53:17)). In the Sources

And 4 more: search the Sources.

Related terms

• Reinforcement learning