Scaling Laws for LLMs: Kaplan vs. Chinchilla

Two answers to the same question, and why the 0.23 gap between them turned out to be methodology

LLM

Scaling Laws

Research

Reproducibility

A practitioner’s walkthrough of the two scaling laws that shaped LLM pre-training, why their compute-allocation exponents disagreed, what the 2024 replications found, and what to actually do with any of it. Every number is either printed in a cited paper or computed from published constants.

Published

September 21, 2026

The short version

Kaplan et al. (2020) found that compute-optimal training puts almost everything into parameters: $N \propto C^{0.73}$, $D \propto C^{0.27}$. Their advice was to train very large models and stop “significantly before convergence.” (Kaplan et al. 2020)
Hoffmann et al. (2022), the Chinchilla paper, found the two should scale equally: $a \approx b \approx 0.5$, i.e. “for every doubling of model size the number of training tokens should also be doubled.” A 70B model on 1.4T tokens beat the 280B Gopher trained on the same FLOPs. (Hoffmann et al. 2022)
Kaplan’s exponent was a methodology artifact, and it has since been reproduced and decomposed. Porian et al. reproduce Kaplan’s raw fit (0.835) and walk it down to 0.497 by counting the output layer’s FLOPs, scaling warmup with model size, and tuning the optimizer per scale. Notably, the cosine-schedule explanation Chinchilla itself offered accounts for only a small part of the gap. (Porian et al. 2024)
Chinchilla’s own Approach 3 is also off. Epoch AI’s replication shows its parametric fit disagrees with the paper’s other two methods, and that its reported confidence intervals would need about 600,000 training runs to be that narrow. The cause was an optimizer that stopped before converging. (Besiroglu et al. 2024) And Chinchilla-optimal is the wrong objective if you plan to serve the model: it minimizes loss per training FLOP and ignores inference entirely.

Between 2020 and 2022 almost every frontier lab trained on roughly 300 billion tokens and competed on parameter count. Given the scaling law they had, that was the right call: if optimal model size grows like $C^{0.73}$ and optimal token count like $C^{0.27}$, a 10× compute increase buys 5.4× the parameters and only 1.9× the data.

Chinchilla showed that the data axis had been starved. Here is what the 2020–2022 generation looked like in tokens per parameter.

Figure 1: Training tokens per parameter. Every large model of the Kaplan era sits between 0.5 and 1.7; Chinchilla sits at 20, on the same training budget as Gopher. Source: Hoffmann et al. (2022), Table 1. The dashed rule marks ≈20 tokens/parameter, the ratio implied by the paper’s Approach 1 projections (Table 3: a 1B model → 20.2B tokens) and the ratio Chinchilla was trained at.

What each paper actually fit

The two papers are not two measurements of the same quantity. They fit different equations to different data, and the difference in form matters as much as the difference in exponent.

Kaplan et al. 2020, Eq. 1.1–1.3

\[ L(N) = \left(\tfrac{N_c}{N}\right)^{\alpha_N}\!,\quad L(D) = \left(\tfrac{D_c}{D}\right)^{\alpha_D}\!,\quad L(C_{\min}) = \left(\tfrac{C_c}{C_{\min}}\right)^{\alpha_C} \]

$\alpha_N = 0.076,\ N_c = 8.8\!\times\!10^{13}$ $\alpha_D = 0.095,\ D_c = 5.4\!\times\!10^{13}$ $\alpha_C^{\min} = 0.050,\ C_c^{\min} = 3.1\!\times\!10^{8}$ PF-days

Pure power laws, no offset. Loss goes to zero as $N \to \infty$. $N$ counts non-embedding parameters, “excluding all vocabulary and positional embeddings.” Models spanned 768 to 1.5B non-embedding params; data 22M to 23B tokens.

Hoffmann et al. 2022, Eq. 2 / Eq. 10

\[ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \]

$E = 1.6934,\ A = 406.4,\ B = 410.7$ $\alpha = 0.3392,\ \beta = 0.2849$ (TeX-source precision) $a = \beta/(\alpha{+}\beta) = 0.46$

An irreducible-loss term. $E$ is the entropy of natural text you can never fit away. $N$ is total parameters, embeddings included. Models ran up to 16B params, budgets $6\!\times\!10^{18} \to 3\!\times\!10^{21}$ FLOPs.

Three structural differences matter here. Kaplan’s form has no offset $E$, so the fit has to bend its exponents to absorb the flattening it sees at scale. Kaplan excludes embedding parameters and Chinchilla includes them; at 768 non-embedding parameters with a 50,257-token vocabulary, the embedding matrix is effectively the whole model. And both use $C \approx 6ND$, but Kaplan’s $N$ leaves out the output layer, an under-count Porian et al. put at “roughly 10% at larger models to roughly 90% at smaller models.”

Put the two allocation rules on the same axes and the practical consequence is immediate.

The two allocation laws, as published

def kaplan_N(C):
    """Kaplan Fig. 14 / Table 6: N = 1.3e9 * (C_min / 1 PF-day)^0.73, non-embedding
    params. 'Adjusted' for batch size, as Porian et al. (2024) call it."""
    return 1.3e9 * (C / 8.64e19) ** 0.73

def chinchilla_N(C):
    """Chinchilla Approach 1, anchored on Hoffmann et al. Table 3
    (67B parameters at the 5.76e23 FLOP Gopher budget)."""
    return 6.7e10 * (C / GOPHER_BUDGET) ** 0.50

Figure 2: Compute-optimal model size under each law, extrapolated across the budgets labs actually spent. Model markers use each paper’s published parameter and token counts; compute for GPT-3, Jurassic-1, LaMDA and MT-NLG is 6ND (our arithmetic), while Gopher and Chinchilla use the 5.76e23 FLOPs printed in Hoffmann et al. Fig. 2. Kaplan’s curve is in non-embedding parameters, so the gap to Chinchilla is slightly overstated at small budgets.

Three approaches, and one that doesn’t agree

Chinchilla’s credibility rests on measuring the same quantity three independent ways. Two agree tightly. The third, the one everybody quotes because it gives you a closed-form equation, is the odd one out.

Approach	$a$ ($N^* \propto C^a$)	interval	$b$ ($D^* \propto C^b$)	interval
1. Minimum over training curves	0.50	(0.488, 0.502)	0.50	(0.501, 0.512)
2. IsoFLOP profiles	0.49	(0.462, 0.534)	0.51	(0.483, 0.529)
3. Parametric loss modelling	0.46	(0.454, 0.455)	0.54	(0.542, 0.543)
Kaplan et al. (2020)	0.73	—	0.27	—

Table 1: Hoffmann et al. (2022), Table 2. 10th/90th percentiles from bootstrap.

Approach 1 takes the minimum over training curves: train a fixed set of model sizes, vary the token count, and read off the envelope. Approach 2 runs IsoFLOP profiles: nine fixed FLOP budgets from $6\times10^{18}$ to $3\times10^{21}$, sweeping model size within each and finding the parabola’s minimum. Approach 3 fits the parametric loss and derives the frontier analytically.

Approach 3’s $a = 0.46$ looks close enough to 0.5 to ignore. It is not, for two reasons: the fitted constants already put the ratio near 40 tokens per parameter at $10^{22}$ FLOPs, and the 0.08 gap between $b$ and $a$ makes it climb from there. You can check both from the paper’s own printed constants.

# Hoffmann et al. (2022), Eq. 10 constants. Printed as 0.34 / 0.28 in the body;
# the extra digits come from the arXiv TeX source, recovered by Besiroglu et al.
HOFFMANN = dict(E=1.6934, A=406.4,  B=410.7,   alpha=0.3392, beta=0.2849)
EPOCH    = dict(E=1.8172, A=482.01, B=2085.43, alpha=0.3478, beta=0.3658)

def loss(N, D, E, A, B, alpha, beta):
    """Predicted cross-entropy for N parameters trained on D tokens."""
    return E + A / N**alpha + B / D**beta

def frontier(C, E, A, B, alpha, beta):
    """Closed-form compute-optimal (N*, D*) for budget C. Hoffmann Eq. 4."""
    a = beta / (alpha + beta)
    b = alpha / (alpha + beta)
    G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
    return G * (C / 6) ** a, (C / 6) ** b / G

N_star, D_star = frontier(GOPHER_BUDGET, **HOFFMANN)
print(f"N* = {N_star/1e9:.1f}B   D* = {D_star/1e12:.2f}T   D/N = {D_star/N_star:.0f}")

N* = 40.3B   D* = 2.38T   D/N = 59

That 40.3B is a genuine check on the arithmetic: Hoffmann et al.’s Figure 4 caption reads “We project the optimal model size given the Gopher FLOP budget to be 40B parameters.” The constants reproduce the paper’s own figure.

But look at the ratio the same code prints. Approach 3 says 59 tokens per parameter at the Gopher budget, and the number keeps climbing with compute. Approach 1, on the same budget, says 67B parameters on 1.5T tokens: 22 tokens per parameter. The paper trained Chinchilla at 20. Its own headline model does not follow its own headline equation.

The replication

Besiroglu et al. digitised 240 points from Hoffmann’s Figure 4 (they converted the figure to SVG and mapped each point’s fill colour back to a loss value) and refit the law. Their result is $L(N,D) = 1.8172 + 482.01\,N^{-0.3478} + 2085.43\,D^{-0.3658}$, giving $a = 0.5126$ (SE 0.02). That agrees with Approaches 1 and 2, and with how Chinchilla was actually trained. They also note the published intervals on $a$ (0.454–0.455) are implausibly tight: “intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500.” (Besiroglu et al. 2024)

The cause, confirmed by one of Hoffmann’s lead authors: the Huber loss was averaged rather than summed over points, so L-BFGS hit its stopping tolerance early, both in the main fit and in every bootstrap.

Swap the Epoch constants into the same function and the ratio stabilises where practitioners expect it:

rows = []
for C_budget in [1e22, 1e23, GOPHER_BUDGET, 1e25]:
    nh, dh = frontier(C_budget, **HOFFMANN)
    ne, de = frontier(C_budget, **EPOCH)
    rows.append({
        "Compute (FLOPs)":  f"{C_budget:.1e}",
        "Hoffmann A3 N*": human(nh), "Hoffmann A3 tok/param": f"{dh/nh:.0f}",
        "Epoch N*":       human(ne), "Epoch tok/param":       f"{de/ne:.0f}",
    })
pd.DataFrame(rows)

	Compute (FLOPs)	Hoffmann A3 N*	Hoffmann A3 tok/param	Epoch N*	Epoch tok/param
0	1.0e+22	6.3B	42	9.0B	20
1	1.0e+23	18B	51	29B	19
2	5.8e+23	40B	59	72B	18
3	1.0e+25	148B	76	312B	17

Table 2: Compute-optimal allocation under Hoffmann’s Approach 3 versus Epoch AI’s refit.

The Epoch refit stays between 17 and 20 tokens per parameter across three orders of magnitude, drifting slowly down rather than up. That stability is what makes “20 tokens per parameter” a usable rule of thumb rather than a coincidence at one budget. Note that the rule comes from Approaches 1 and 2 and the replication, not from the equation most people copy.

Figure 3: IsoFLOP curves drawn straight from Hoffmann et al.’s published loss function, with tokens set by D = C/6N. No refitting. At the Gopher budget the minimum lands at about 40B, matching the paper’s stated 40B projection. These are the model’s predictions, not measured runs. The paper’s *measured* IsoFLOP minima (Approach 2) sit closer to 63B at that budget.

Where 0.73 actually came from

Chinchilla’s own explanation for the discrepancy was the learning-rate schedule: Kaplan used one cosine decay sized for about 131B tokens across all runs, so a model evaluated at an intermediate checkpoint looks worse than a model whose schedule was sized to stop there. That makes short runs look worse than they are, and pushes the fit toward bigger models on less data.

It is a good hypothesis. It is also mostly wrong, and we know this because Porian et al. ran over 900 training runs (16 model sizes from 5M to 901M parameters on OpenWebText2 and RefinedWeb), reproducing Kaplan’s setup and then removing one confounder at a time.

One wrinkle before the numbers. Kaplan’s raw fit was actually $N \propto C^{0.88}$; the famous 0.73 is their adjusted law, after correcting for using too large a batch at small scale. Porian et al.’s baseline (0.835 on RefinedWeb) reproduces the raw 0.88 setup, and their per-scale tuning step does roughly the job Kaplan’s batch adjustment did. They check this directly: with tuned hyperparameters but the head-FLOP and warmup issues put back, they get 0.717, essentially Kaplan’s 0.73 (their Appendix H). So the gap from 0.73 to 0.5 is mostly the two fixes below.

Figure 4: Walking Kaplan’s exponent to Chinchilla’s, one fix at a time. Each point is a re-measurement of a after one more correction; the last step also switches to a constant learning rate. Source: Porian et al. (2024), Table 1, RefinedWeb column, with 95% confidence intervals as printed. Their OpenWebText2 column follows the same path (0.864 → 0.699 → 0.603 → 0.574 → 0.518).

1. Count the output layer’s FLOPs (−0.13)

Kaplan excluded embeddings from $N$ and, probably because the output head shared weights with the input embedding, its per-token FLOPs were left out of the compute count too. The resulting under-approximation of compute runs from “roughly 10% at larger models to roughly 90% at smaller models,” a bias that scales with the very axis you are fitting against. Counting it correctly moves $a$ from 0.835 to 0.706.

Pearce & Song come at a closely related issue analytically: model total parameters as $N_T = N_E + \omega N_E^{1/3}$ (fitting $\omega = 47{,}491$ to Chinchilla’s own configurations) and the non-embedding relationship stops being a power law at all. It has a small-scale regime, a large-scale regime and a transition between them, with the crossover where embeddings are half the model, around $10^7$ non-embedding parameters. Kaplan’s entire sweep, 768 to 1.5B, straddles that. Taking the Chinchilla (or Epoch) loss surface and measuring it in Kaplan’s non-embedding units, they get a local exponent of 0.74 (or 0.78). Their decomposition puts more weight on the embedding convention than Porian’s does, partly because a fitted loss surface has no warmup problem to find. (Pearce and Song 2024)

2. Stop warming up for 1.57B tokens (−0.10)

This is the subtle one. Kaplan used a fixed 3,000-step warmup at a $2^{19}$-token batch, about 1.57 billion warmup tokens, identical for every model. For a small model at a small budget, the compute-optimal token count is at or below the warmup length. Those runs never leave warmup, so they look terrible, so small models look worse than they are, so the fit tilts toward large models. Setting warmup tokens proportional to $N$ moves $a$ to 0.602.

3. Tune the optimizer per scale (−0.07)

Sweeping learning rate, batch size and AdamW’s $\beta_2$ per model size lands at $a = 0.497$, 95% CI (0.49, 0.50). Along the way they fit $\mathrm{BS} = 0.00037\,N^{0.703}$ and $\mathrm{LR} = 3.7\,N^{-0.36}$, and find the usual $\beta_2 = 0.95$ is too low at small batch sizes. This last step also switches to a constant learning rate. Their summary: the exponent matches 0.5 “to within 0.6%” and the predicted model size at Chinchilla compute lands “within 15% of Chinchilla’s size.”

And the cosine schedule?

Porian et al. tested it in isolation (§3.4) and it moves $a$ from 0.60 to 0.57, and “most of the gap remains.” Their reasoning: with decay targeted at about 131B tokens, Kaplan’s compute-constrained runs “see virtually no learning rate decay,” less than 1.5% of the decay. Their final 0.497 is achieved with a constant learning rate. Pearce & Song find the same thing independently: switching from Kaplan’s schedule to Chinchilla’s raised their coefficient from 0.46 to 0.49 rather than lowering it.

So the field’s canonical story, that Kaplan got it wrong because of the learning-rate schedule, is itself wrong. The dominant factors are an accounting error in the compute axis and a warmup length that poisoned the small-model end of the fit.

What each law tells you to build

Four laws, one budget.

Show the allocation code

def allocate(C):
    kN = kaplan_N(C);     kD = C / (6 * kN)
    cN = chinchilla_N(C); cD = C / (6 * cN)
    hN, hD = frontier(C, **HOFFMANN)
    eN, eD = frontier(C, **EPOCH)
    return [("Kaplan 2020 (adjusted)", kN, kD),
            ("Chinchilla · Approach 1", cN, cD),
            ("Chinchilla · Approach 3", hN, hD),
            ("Epoch AI refit (2024)",   eN, eD)]

BUDGETS = {"Chinchilla / Gopher": GOPHER_BUDGET, "1e24": 1e24, "Llama 3 405B": 3.8e25}

def ratio(r):
    return f"{r:.2f}" if r < 1 else (f"{r:.1f}" if r < 10 else f"{r:,.0f}")

out = []
for name, C_budget in BUDGETS.items():
    for law, n, d in allocate(C_budget):
        out.append({"Budget": name, "Law": law, "N*": human(n),
                    "D*": human(d), "tok/param": ratio(d / n)})
pd.DataFrame(out).set_index(["Budget", "Law"])

		N*	D*	tok/param
Budget	Law
Chinchilla / Gopher	Kaplan 2020 (adjusted)	804B	119B	0.15
	Chinchilla · Approach 1	67B	1.4T	21
	Chinchilla · Approach 3	40B	2.4T	59
	Epoch AI refit (2024)	72B	1.3T	18
1e24	Kaplan 2020 (adjusted)	1.2T	139B	0.12
	Chinchilla · Approach 1	88B	1.9T	21
	Chinchilla · Approach 3	52B	3.2T	62
	Epoch AI refit (2024)	96B	1.7T	18
Llama 3 405B	Kaplan 2020 (adjusted)	17T	370B	0.02
	Chinchilla · Approach 1	544B	12T	21
	Chinchilla · Approach 3	273B	23T	85
	Epoch AI refit (2024)	619B	10T	17

Table 3: Each row applies one published law to the same training budget. Kaplan uses the adjusted law with D = C/6N; Chinchilla A1 uses the Table 3 anchor; A3 and the Epoch refit use the closed form. Extrapolating any of these far beyond its fitting range is exactly the move this whole story is a cautionary tale about.

Chinchilla-optimal is not deployment-optimal

Here is the part that matters most for anyone shipping a model, and it is a limitation both papers share rather than a disagreement between them. Both optimise loss given a training budget. Neither includes a single inference FLOP. If you train once and serve billions of tokens, you are optimising the wrong objective.

Add inference to the objective

Sardana et al. invert the problem: fix a quality target, minimise total compute. Training costs $6N$ per token, inference costs $2N$ per token, so the objective becomes (Sardana et al. 2024)

\[ N^*,\, D^*_{\text{tr}} \;=\; \operatorname*{argmin}_{N,\, D_{\text{tr}} \;\mid\; L(N, D_{\text{tr}}) = \ell} \;\; 6N D_{\text{tr}} + 2N D_{\text{inf}} \]

where $D_{\text{inf}}$ is total lifetime inference tokens, input plus output. With $D_{\text{inf}} > 0$ there is no closed form; they solve it by root-finding. It is short enough to reproduce:

from scipy.optimize import brentq, minimize_scalar

# Sardana et al. Appendix A: Chinchilla's A, B, E with alpha = 0.336, beta = 0.283,
# taken from De Vries (2023) to better match Hoffmann's Table A3 (not the
# TeX-source values above).
SARDANA = dict(E=1.69, A=406.4, B=410.7, alpha=0.336, beta=0.283)

def tokens_for_loss(N, target, E, A, B, alpha, beta):
    """Tokens needed for N parameters to reach a given loss."""
    residual = target - E - A / N**alpha
    return np.inf if residual <= 0 else (B / residual) ** (1 / beta)

def total_flops(N, target, D_inf, p):
    return 6 * N * tokens_for_loss(N, target, **p) + 2 * N * D_inf

def inference_aware(N_chinchilla, D_inf, p=SARDANA):
    """Cheapest (N, D) reaching the quality of a Chinchilla-optimal model
    of size N_chinchilla, once D_inf inference tokens are paid for."""
    budget = 10 ** brentq(lambda lc: frontier(10**lc, **p)[0] - N_chinchilla, 18, 30)
    N0, D0 = frontier(budget, **p)
    target = loss(N0, D0, **p)
    floor = (p["A"] / (target - p["E"])) ** (1 / p["alpha"])   # N below this can't reach it
    best = minimize_scalar(lambda x: total_flops(np.exp(x), target, D_inf, p),
                           bounds=(np.log(floor * 1.001), np.log(N_chinchilla)),
                           method="bounded")
    N1 = np.exp(best.x)
    saved = 1 - total_flops(N1, target, D_inf, p) / total_flops(N0, target, D_inf, p)
    return N0, D0, N1, tokens_for_loss(N1, target, **p), saved

N0, D0, N1, D1, saved = inference_aware(30e9, D_inf=1e13)
print(f"Chinchilla-optimal : {N0/1e9:5.1f}B on {D0/1e12:.2f}T tokens")
print(f"Inference-aware    : {N1/1e9:5.1f}B on {D1/1e12:.2f}T tokens "
      f"({D1/D0:.2f}x the data)")
print(f"Total FLOPs saved  : {saved:.0%}")

Chinchilla-optimal :  30.0B on 1.56T tokens
Inference-aware    :  13.6B on 4.43T tokens (2.84x the data)
Total FLOPs saved  : 28%

That reproduces the paper’s stated result exactly: a developer expecting a 30B-Chinchilla-quality model to see $10^{13}$ inference tokens “can reduce their total FLOPs by 28% by training a 13.6B model on 2.84× the data.” In dollar terms, with realistic MFU asymmetry between training and autoregressive generation, the same quality target at 17.5B requests is reached ~58% cheaper by an 8.58B model trained on 12.1T tokens.

The caveat, from the same paper

Fitting scaling-law coefficients only on typical token/parameter ratios “overestimates the impact of additional tokens at these extreme ranges.” Refitting with their extreme-ratio data (47 models, 150M–6B, up to 10,000 tokens/param) gives $\alpha = 0.18$, $\beta = 0.24$ against Chinchilla’s 0.34 / 0.28. Extrapolating Chinchilla’s constants out to 1,000+ tokens/param will flatter your plan.

What if you run out of data?

Muennighoff et al. ran more than 400 models, 10M to 9B parameters, for up to 1,500 epochs, and their finding is unusually actionable: up to roughly 4 epochs of repeated data is nearly free. Their 8.7B model at four epochs on 44B unique tokens ended “with only 0.5% higher validation loss than the single-epoch model.” Returns then decay fast, with $R_D^* \approx 15$ repetitions as the point where repeated tokens have lost about 63% of their value, and runs at 44 epochs diverge outright. They model it by replacing $D$ with effective data (Muennighoff et al. 2023)

\[ D' = U_D + U_D \, R_D^* \left(1 - e^{-R_D / R_D^*}\right) \]

which collapses to Chinchilla exactly when there is no repetition. Under data constraint they also find you should scale epochs faster than parameters, the opposite tilt to Chinchilla’s equal scaling.

What frontier labs actually did

The folk version is that Meta over-trained Llama 3. Only partly. Meta’s Llama 3 paper reports IsoFLOPs experiments from $6\times10^{18}$ to $10^{22}$ FLOPs, extrapolating to a $3.8\times10^{25}$ FLOP budget and suggesting “training a 402B parameter model on 16.55T tokens.” They trained 405B on 15.6T, approximately compute-optimal, not over-trained. The over-training was deliberate and confined to the small models: “we also train our smaller models for much longer than is compute-optimal… The resulting models perform better than compute-optimal models at the same inference budget.” (Llama Team, AI @ Meta 2024)

Note the ratio: 15.6T tokens on 405B parameters is about 39 tokens per parameter, and Meta’s own fit (402B on 16.55T) says 41. That is roughly double Chinchilla’s 20, from IsoFLOP curves on their own data mix. The planner table above, which puts Chinchilla-optimal at 544B on 12T for the same budget, shows how much the “optimal” ratio moves with the data and stack.

That is how Meta split the problem, and it is how I would split it. Chinchilla-optimal is the right target for the flagship, where training cost dominates; heavy over-training is the right target for the models you actually serve.

The practitioner’s checklist

Never use Kaplan’s allocation exponents. 0.73/0.27 is a measurement artifact with a known, reproduced cause. The loss-vs-scale phenomenology (smooth power laws, predictable returns) survives intact.
Use ~20 tokens/parameter as a starting anchor, not a constant. It comes from Approaches 1 and 2 plus the Epoch refit, not from Chinchilla’s published Eq. 10, which drifts to 59+ at frontier budgets. Meta’s own IsoFLOP fit for Llama 3 landed near 40.
If you will serve the model, over-train it. Compute the inference-inclusive objective before sizing. At $10^{12}$–$10^{13}$ lifetime tokens the optimum moves substantially smaller-and-longer.
Fit your own law on your own data and stack. Every exponent here is entangled with tokenizer, data mix, optimizer tuning and architecture. Chinchilla’s own three approaches disagreed on their own data.
Sweep hyperparameters per scale, and warm up proportionally to model size. Kaplan’s discrepancy traces back to holding warmup and optimizer settings fixed across scales that should not have been.
Check that your scaling-law fit actually converged. Chinchilla’s Approach 3 constants came from an L-BFGS run that stopped early because the Huber loss was averaged instead of summed, which is also why its bootstrap intervals came out absurdly narrow.
Include the output layer in your FLOP accounting. The single largest term in the reconciliation was an accounting choice.

The meta-lesson

Kaplan’s paper was careful, influential, and wrong about one thing, and the field spent two years and a lot of compute acting on it. Chinchilla corrected it, and had a serious flaw of its own in the very equation everyone copied. Both were caught only when independent groups re-derived the numbers from scratch, which is a good argument for re-fitting before you trust anyone’s constants, including the ones in this note.

Sources

Figures Figure 1 and Figure 4 plot values printed in the cited papers. Figure 2, Figure 3 and every code output above are computed from published constants and are reproducible by rendering this document. Where a number is our arithmetic rather than a paper’s, the caption says so.

Besiroglu, Tamay, Ege Erdil, Matthew Barnett, and Josh You. 2024. “Chinchilla Scaling: A Replication Attempt.” arXiv Preprint arXiv:2404.10102. https://arxiv.org/abs/2404.10102.

Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, et al. 2022. “Training Compute-Optimal Large Language Models.” arXiv Preprint arXiv:2203.15556. https://arxiv.org/abs/2203.15556.

Kaplan, Jared, Sam McCandlish, Tom Henighan, et al. 2020. “Scaling Laws for Neural Language Models.” arXiv Preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361.

Llama Team, AI @ Meta. 2024. “The Llama 3 Herd of Models.” arXiv Preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783.

Muennighoff, Niklas, Alexander Rush, Boaz Barak, et al. 2023. “Scaling Data-Constrained Language Models.” Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2305.16264.

Pearce, Tim, and Jinyeop Song. 2024. “Reconciling Kaplan and Chinchilla Scaling Laws.” arXiv Preprint arXiv:2406.12907. https://arxiv.org/abs/2406.12907.

Porian, Tomer, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. 2024. “Resolving Discrepancies in Compute-Optimal Scaling of Language Models.” arXiv Preprint arXiv:2406.19146. https://arxiv.org/abs/2406.19146.

Sardana, Nikhil, Jacob Portes, Sasha Doubov, and Jonathan Frankle. 2024. “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws.” arXiv Preprint arXiv:2401.00448. https://arxiv.org/abs/2401.00448.

--- title: "Scaling Laws for LLMs: Kaplan vs. Chinchilla" subtitle: "Two answers to the same question, and why the 0.23 gap between them turned out to be methodology" description: > A practitioner's walkthrough of the two scaling laws that shaped LLM pre-training, why their compute-allocation exponents disagreed, what the 2024 replications found, and what to actually do with any of it. Every number is either printed in a cited paper or computed from published constants. date: "2026-09-21" categories: - LLM - Scaling Laws - Research - Reproducibility bibliography: references.bib include-in-header: text: | <style> /* Reading note: keep only two heading levels visible - the page title (from the title block above) and the H2 sections below. Quarto/Pandoc always emits a body-level H1 matching the title when content starts at H2; hide that duplicate here. */ main#quarto-document-content > section.level1 > h1:first-child { display: none; } </style> format: html: toc: true toc-depth: 2 toc-title: "Contents" code-tools: true code-overflow: wrap fig-width: 9 fig-height: 4.6 fig-format: svg fig-align: center tbl-cap-location: bottom execute: echo: true warning: false message: false freeze: auto # inside a Quarto website, re-executes only when this file changes jupyter: python3 --- ```{python} #| label: setup #| echo: false import numpy as np import pandas as pd import matplotlib as mpl import matplotlib.pyplot as plt from matplotlib.ticker import FuncFormatter # ---------------------------------------------------------------- style # Figures carry their own light surface so they stay legible whatever # theme the surrounding site uses. PAL = dict( kaplan="#eb6834", chin="#2a78d6", third="#1baf7a", surface="#fcfcfb", ink="#101318", ink2="#4b5159", muted="#848b95", rule="#e0e3e8", rule_strong="#c3c9d2", ) mpl.rcParams.update({ "figure.facecolor": PAL["surface"], "axes.facecolor": PAL["surface"], "savefig.facecolor": PAL["surface"], "savefig.bbox": "tight", "font.family": "sans-serif", "font.size": 10.5, "text.color": PAL["ink"], "axes.labelcolor": PAL["ink2"], "xtick.color": PAL["muted"], "ytick.color": PAL["muted"], "axes.edgecolor": PAL["rule_strong"], "axes.linewidth": 0.8, "xtick.major.size": 3, "ytick.major.size": 3, "xtick.minor.size": 1.8, "ytick.minor.size": 1.8, "grid.color": PAL["rule"], "grid.linewidth": 0.8, "axes.grid": True, "axes.axisbelow": True, "legend.frameon": False, "axes.spines.top": False, "axes.spines.right": False, }) def style(ax, xlabel=None, ylabel=None, title=None): if xlabel: ax.set_xlabel(xlabel, fontsize=9, labelpad=8) if ylabel: ax.set_ylabel(ylabel, fontsize=9, labelpad=8) if title: ax.set_title(title, fontsize=11.5, color=PAL["ink"], loc="left", pad=12, fontweight="semibold") return ax def human(n, _=None): """1.4e12 -> '1.4T'. Used for axis ticks and tables alike.""" n = float(n) for div, suf in ((1e12, "T"), (1e9, "B"), (1e6, "M")): if abs(n) >= div: v = n / div return f"{v:.1f}{suf}" if v < 10 else f"{v:.0f}{suf}" return f"{n:.0f}" HUMAN = FuncFormatter(human) # ---------------------------------------------------------------- data # Hoffmann et al. (2022), Table 1: parameter counts and training tokens as printed. MODELS = pd.DataFrame([ ("MT-NLG 530B", 530e9, 270e9, 6*530e9*270e9), ("Gopher", 280e9, 300e9, 5.76e23), # FLOPs as printed, Fig. 2 caption ("Jurassic-1", 178e9, 300e9, 6*178e9*300e9), ("GPT-3", 175e9, 300e9, 6*175e9*300e9), ("LaMDA", 137e9, 168e9, 6*137e9*168e9), ("Chinchilla", 70e9, 1.4e12, 5.76e23), ], columns=["model", "params", "tokens", "flops"]) MODELS["tok_per_param"] = MODELS.tokens / MODELS.params # Porian et al. (2024), Table 1, RefinedWeb column, 95% CIs as printed. WATERFALL = pd.DataFrame([ ("Reproducing Kaplan et al.", 0.835, 0.82, 0.85, "Baseline: Kaplan's setup reproduced"), ("+ count last-layer FLOPs", 0.706, 0.69, 0.72, "Compute axis now includes the output head"), ("+ warmup proportional to $N$", 0.602, 0.59, 0.62, "Replaces a fixed 1.57B-token warmup"), ("+ matched cosine decay", 0.571, 0.56, 0.59, "Chinchilla's own proposed explanation"), ("+ per-scale optimizer tuning", 0.497, 0.49, 0.50, "LR, batch size and AdamW $\\beta_2$ swept per size"), ], columns=["step", "a", "lo", "hi", "what"]) GOPHER_BUDGET = 5.76e23 # FLOPs, Hoffmann et al. (2022) Fig. 2 caption ``` ::: {.callout-important icon=false} ## The short version 1. **Kaplan et al. (2020)** found that compute-optimal training puts almost everything into parameters: $N \propto C^{0.73}$, $D \propto C^{0.27}$. Their advice was to train very large models and stop "significantly before convergence." [@kaplan2020] 2. **Hoffmann et al. (2022)**, the Chinchilla paper, found the two should scale *equally*: $a \approx b \approx 0.5$, i.e. "for every doubling of model size the number of training tokens should also be doubled." A 70B model on 1.4T tokens beat the 280B Gopher trained on the same FLOPs. [@hoffmann2022] 3. **Kaplan's exponent was a methodology artifact**, and it has since been reproduced and decomposed. Porian et al. reproduce Kaplan's raw fit (0.835) and walk it down to 0.497 by counting the output layer's FLOPs, scaling warmup with model size, and tuning the optimizer per scale. Notably, the cosine-schedule explanation Chinchilla itself offered accounts for only a small part of the gap. [@porian2024] 4. **Chinchilla's own Approach 3 is also off.** Epoch AI's replication shows its parametric fit disagrees with the paper's other two methods, and that its reported confidence intervals would need about 600,000 training runs to be that narrow. The cause was an optimizer that stopped before converging. [@besiroglu2024] *And* Chinchilla-optimal is the wrong objective if you plan to serve the model: it minimizes loss per *training* FLOP and ignores inference entirely. ::: Between 2020 and 2022 almost every frontier lab trained on roughly 300 billion tokens and competed on parameter count. Given the scaling law they had, that was the right call: if optimal model size grows like $C^{0.73}$ and optimal token count like $C^{0.27}$, a 10× compute increase buys 5.4× the parameters and only 1.9× the data. Chinchilla showed that the data axis had been starved. Here is what the 2020–2022 generation looked like in tokens per parameter. ```{python} #| label: fig-ratio #| echo: false #| fig-cap: > #| Training tokens per parameter. Every large model of the Kaplan era sits #| between 0.5 and 1.7; Chinchilla sits at 20, on the same training budget as Gopher. #| Source: Hoffmann et al. (2022), Table 1. The dashed rule marks ≈20 #| tokens/parameter, the ratio implied by the paper's Approach 1 projections #| (Table 3: a 1B model → 20.2B tokens) and the ratio Chinchilla was trained at. d = MODELS.sort_values("tok_per_param") fig, ax = plt.subplots(figsize=(9, 4.2)) colors = [PAL["chin"] if m == "Chinchilla" else PAL["kaplan"] for m in d.model] ax.barh(d.model, d.tok_per_param, height=0.46, color=colors, alpha=1.0, zorder=3) for y, (v, m) in enumerate(zip(d.tok_per_param, d.model)): ax.text(v + 0.35, y, f"{v:.2f}", va="center", fontsize=10, fontweight="semibold", color=PAL["ink"] if m == "Chinchilla" else PAL["ink2"]) ax.axvline(20, color=PAL["chin"], lw=1.4, ls=(0, (4, 3)), alpha=.7, zorder=2) ax.text(20, len(d) - 0.35, "Chinchilla-optimal ≈ 20 ", ha="right", va="bottom", fontsize=9.5, color=PAL["chin"], fontweight="semibold") ax.set_xlim(0, 23) ax.grid(axis="y", visible=False) ax.tick_params(axis="y", length=0, labelcolor=PAL["ink"], labelsize=10.5) style(ax, xlabel="TRAINING TOKENS PER PARAMETER") plt.show() ``` ## What each paper actually fit The two papers are not two measurements of the same quantity. They fit different equations to different data, and the difference in form matters as much as the difference in exponent. ::: {layout-ncol=2} **Kaplan et al. 2020**, Eq. 1.1–1.3 $$ L(N) = \left(\tfrac{N_c}{N}\right)^{\alpha_N}\!,\quad L(D) = \left(\tfrac{D_c}{D}\right)^{\alpha_D}\!,\quad L(C_{\min}) = \left(\tfrac{C_c}{C_{\min}}\right)^{\alpha_C} $$ $\alpha_N = 0.076,\ N_c = 8.8\!\times\!10^{13}$ $\alpha_D = 0.095,\ D_c = 5.4\!\times\!10^{13}$ $\alpha_C^{\min} = 0.050,\ C_c^{\min} = 3.1\!\times\!10^{8}$ PF-days *Pure power laws, no offset.* Loss goes to zero as $N \to \infty$. $N$ counts **non-embedding** parameters, "excluding all vocabulary and positional embeddings." Models spanned 768 to 1.5B non-embedding params; data 22M to 23B tokens. **Hoffmann et al. 2022**, Eq. 2 / Eq. 10 $$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} $$ $E = 1.6934,\ A = 406.4,\ B = 410.7$ $\alpha = 0.3392,\ \beta = 0.2849$ (TeX-source precision) $a = \beta/(\alpha{+}\beta) = 0.46$ *An irreducible-loss term.* $E$ is the entropy of natural text you can never fit away. $N$ is **total** parameters, embeddings included. Models ran up to 16B params, budgets $6\!\times\!10^{18} \to 3\!\times\!10^{21}$ FLOPs. ::: Three structural differences matter here. Kaplan's form has no offset $E$, so the fit has to bend its exponents to absorb the flattening it sees at scale. Kaplan excludes embedding parameters and Chinchilla includes them; at 768 non-embedding parameters with a 50,257-token vocabulary, the embedding matrix is effectively the whole model. And both use $C \approx 6ND$, but Kaplan's $N$ leaves out the output layer, an under-count Porian et al. put at "roughly 10% at larger models to roughly 90% at smaller models." Put the two allocation rules on the same axes and the practical consequence is immediate. ```{python} #| label: laws #| code-fold: true #| code-summary: "The two allocation laws, as published" def kaplan_N(C): """Kaplan Fig. 14 / Table 6: N = 1.3e9 * (C_min / 1 PF-day)^0.73, non-embedding params. 'Adjusted' for batch size, as Porian et al. (2024) call it.""" return 1.3e9 * (C / 8.64e19) ** 0.73 def chinchilla_N(C): """Chinchilla Approach 1, anchored on Hoffmann et al. Table 3 (67B parameters at the 5.76e23 FLOP Gopher budget).""" return 6.7e10 * (C / GOPHER_BUDGET) ** 0.50 ``` ```{python} #| label: fig-frontier #| echo: false #| fig-cap: > #| Compute-optimal model size under each law, extrapolated across the budgets #| labs actually spent. Model markers use each paper's published parameter and #| token counts; compute for GPT-3, Jurassic-1, LaMDA and MT-NLG is 6ND (our #| arithmetic), while Gopher and Chinchilla use the 5.76e23 FLOPs printed in #| Hoffmann et al. Fig. 2. Kaplan's curve is in non-embedding parameters, so #| the gap to Chinchilla is slightly overstated at small budgets. C = np.logspace(19, 26, 300) fig, ax = plt.subplots(figsize=(9, 5.2)) ax.plot(C, kaplan_N(C), color=PAL["kaplan"], lw=2.2, zorder=4) ax.plot(C, chinchilla_N(C), color=PAL["chin"], lw=2.2, zorder=4) ax.annotate(r"Kaplan $N \propto C^{0.73}$", (3e22, kaplan_N(3e22)), xytext=(-8, 10), textcoords="offset points", ha="right", color=PAL["kaplan"], fontsize=11, fontweight="semibold") ax.annotate(r"Chinchilla $N \propto C^{0.50}$", (1.3e24, chinchilla_N(1.3e24)), xytext=(8, -20), textcoords="offset points", ha="left", color=PAL["chin"], fontsize=11, fontweight="semibold") # the divergence at the Gopher budget gk, gc = kaplan_N(GOPHER_BUDGET), chinchilla_N(GOPHER_BUDGET) ax.plot([GOPHER_BUDGET] * 2, [gc, gk], color=PAL["ink2"], lw=1, ls=(0, (2, 2)), zorder=3) ax.annotate(f"Kaplan would build {human(gk)} here, {gk/gc:.0f}× Chinchilla", (GOPHER_BUDGET, gk), xytext=(-10, 8), textcoords="offset points", ha="right", fontsize=9.5, color=PAL["ink2"], fontweight="semibold") OFF = {"MT-NLG 530B": (11, 0, "left"), "Gopher": (11, 0, "left"), "Jurassic-1": (-7, 9, "right"), "GPT-3": (0, -15, "center"), "LaMDA": (-11, 0, "right"), "Chinchilla": (0, -16, "center")} for r in MODELS.itertuples(): c = PAL["chin"] if r.model == "Chinchilla" else PAL["ink2"] ax.scatter(r.flops, r.params, s=48, color=c, zorder=6, edgecolor=PAL["surface"], linewidth=2) dx, dy, ha = OFF[r.model] ax.annotate(r.model, (r.flops, r.params), xytext=(dx, dy), textcoords="offset points", ha=ha, va="center", fontsize=9.5, color=c, fontweight="semibold", zorder=7) ax.set(xscale="log", yscale="log", xlim=(1e19, 1e26), ylim=(1e8, 3e12)) ax.yaxis.set_major_formatter(HUMAN) style(ax, xlabel="TRAINING COMPUTE $C$ (FLOPs)", ylabel="COMPUTE-OPTIMAL PARAMETERS $N^*$") plt.show() ``` ## Three approaches, and one that doesn't agree Chinchilla's credibility rests on measuring the same quantity three independent ways. Two agree tightly. The third, the one everybody quotes because it gives you a closed-form equation, is the odd one out. | Approach | $a$ ($N^* \propto C^a$) | interval | $b$ ($D^* \propto C^b$) | interval | |:---|---:|:---:|---:|:---:| | 1. Minimum over training curves | 0.50 | (0.488, 0.502) | 0.50 | (0.501, 0.512) | | 2. IsoFLOP profiles | 0.49 | (0.462, 0.534) | 0.51 | (0.483, 0.529) | | 3. Parametric loss modelling | 0.46 | (0.454, 0.455) | 0.54 | (0.542, 0.543) | | Kaplan et al. (2020) | 0.73 | — | 0.27 | — | : Hoffmann et al. (2022), Table 2. 10th/90th percentiles from bootstrap. {#tbl-approaches tbl-colwidths="[40,12,18,12,18]"} **Approach 1** takes the minimum over training curves: train a fixed set of model sizes, vary the token count, and read off the envelope. **Approach 2** runs IsoFLOP profiles: nine fixed FLOP budgets from $6\times10^{18}$ to $3\times10^{21}$, sweeping model size within each and finding the parabola's minimum. **Approach 3** fits the parametric loss and derives the frontier analytically. Approach 3's $a = 0.46$ looks close enough to 0.5 to ignore. It is not, for two reasons: the fitted constants already put the ratio near 40 tokens per parameter at $10^{22}$ FLOPs, and the 0.08 gap between $b$ and $a$ makes it climb from there. You can check both from the paper's own printed constants. ```{python} #| label: repro-a3 # Hoffmann et al. (2022), Eq. 10 constants. Printed as 0.34 / 0.28 in the body; # the extra digits come from the arXiv TeX source, recovered by Besiroglu et al. HOFFMANN = dict(E=1.6934, A=406.4, B=410.7, alpha=0.3392, beta=0.2849) EPOCH = dict(E=1.8172, A=482.01, B=2085.43, alpha=0.3478, beta=0.3658) def loss(N, D, E, A, B, alpha, beta): """Predicted cross-entropy for N parameters trained on D tokens.""" return E + A / N**alpha + B / D**beta def frontier(C, E, A, B, alpha, beta): """Closed-form compute-optimal (N*, D*) for budget C. Hoffmann Eq. 4.""" a = beta / (alpha + beta) b = alpha / (alpha + beta) G = (alpha * A / (beta * B)) ** (1 / (alpha + beta)) return G * (C / 6) ** a, (C / 6) ** b / G N_star, D_star = frontier(GOPHER_BUDGET, **HOFFMANN) print(f"N* = {N_star/1e9:.1f}B D* = {D_star/1e12:.2f}T D/N = {D_star/N_star:.0f}") ``` That 40.3B is a genuine check on the arithmetic: Hoffmann et al.'s Figure 4 caption reads *"We project the optimal model size given the Gopher FLOP budget to be 40B parameters."* The constants reproduce the paper's own figure. But look at the ratio the same code prints. Approach 3 says **59 tokens per parameter** at the Gopher budget, and the number keeps climbing with compute. Approach 1, on the same budget, says 67B parameters on 1.5T tokens: **22 tokens per parameter**. The paper trained Chinchilla at 20. Its own headline model does not follow its own headline equation. ::: {.callout-warning icon=false} ## The replication Besiroglu et al. digitised 240 points from Hoffmann's Figure 4 (they converted the figure to SVG and mapped each point's fill colour back to a loss value) and refit the law. Their result is $L(N,D) = 1.8172 + 482.01\,N^{-0.3478} + 2085.43\,D^{-0.3658}$, giving $a = 0.5126$ (SE 0.02). That agrees with Approaches 1 and 2, and with how Chinchilla was actually trained. They also note the published intervals on $a$ (0.454–0.455) are implausibly tight: *"intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500."* [@besiroglu2024] The cause, confirmed by one of Hoffmann's lead authors: the Huber loss was averaged rather than summed over points, so L-BFGS hit its stopping tolerance early, both in the main fit and in every bootstrap. ::: Swap the Epoch constants into the same function and the ratio stabilises where practitioners expect it: ```{python} #| label: tbl-compare #| tbl-cap: "Compute-optimal allocation under Hoffmann's Approach 3 versus Epoch AI's refit." rows = [] for C_budget in [1e22, 1e23, GOPHER_BUDGET, 1e25]: nh, dh = frontier(C_budget, **HOFFMANN) ne, de = frontier(C_budget, **EPOCH) rows.append({ "Compute (FLOPs)": f"{C_budget:.1e}", "Hoffmann A3 N*": human(nh), "Hoffmann A3 tok/param": f"{dh/nh:.0f}", "Epoch N*": human(ne), "Epoch tok/param": f"{de/ne:.0f}", }) pd.DataFrame(rows) ``` The Epoch refit stays between 17 and 20 tokens per parameter across three orders of magnitude, drifting slowly down rather than up. That stability is what makes "20 tokens per parameter" a usable rule of thumb rather than a coincidence at one budget. Note that the rule comes from Approaches 1 and 2 and the replication, *not* from the equation most people copy. ```{python} #| label: fig-isoflop #| echo: false #| fig-cap: > #| IsoFLOP curves drawn straight from Hoffmann et al.'s published loss function, #| with tokens set by D = C/6N. No refitting. At the Gopher budget the minimum #| lands at about 40B, matching the paper's stated 40B projection. These are the #| model's predictions, not measured runs. The paper's *measured* IsoFLOP #| minima (Approach 2) sit closer to 63B at that budget. fig, ax = plt.subplots(figsize=(9, 4.8)) N = np.logspace(8.4, 12, 400) for C_budget, col in [(1e21, PAL["third"]), (1e23, PAL["kaplan"]), (GOPHER_BUDGET, PAL["chin"])]: D = C_budget / (6 * N) L = loss(N, D, **HOFFMANN) ax.plot(N, L, color=col, lw=2.2, zorder=4, label=f"C = {C_budget:.1e} FLOPs") i = int(np.argmin(L)) ax.scatter(N[i], L[i], s=48, color=col, zorder=6, edgecolor=PAL["surface"], linewidth=2) ax.vlines(N[i], L[i], 1.85, color=col, lw=1, ls=(0, (2, 2)), alpha=.55, zorder=3) ax.annotate(human(N[i]), (N[i], L[i]), xytext=(0, 11), textcoords="offset points", ha="center", fontsize=10, color=col, fontweight="semibold") ax.set(xscale="log", xlim=(3e8, 1e12), ylim=(1.85, 2.75)) ax.xaxis.set_major_formatter(HUMAN) ax.legend(loc="upper right", fontsize=9.5, labelcolor=PAL["ink2"]) style(ax, xlabel="MODEL PARAMETERS $N$", ylabel="PREDICTED LOSS (NATS)") plt.show() ``` ## Where 0.73 actually came from Chinchilla's own explanation for the discrepancy was the learning-rate schedule: Kaplan used one cosine decay sized for about 131B tokens across all runs, so a model evaluated at an intermediate checkpoint looks worse than a model whose schedule was sized to stop there. That makes short runs look worse than they are, and pushes the fit toward bigger models on less data. It is a good hypothesis. It is also mostly wrong, and we know this because Porian et al. ran over 900 training runs (16 model sizes from 5M to 901M parameters on OpenWebText2 and RefinedWeb), reproducing Kaplan's setup and then removing one confounder at a time. One wrinkle before the numbers. Kaplan's raw fit was actually $N \propto C^{0.88}$; the famous 0.73 is their *adjusted* law, after correcting for using too large a batch at small scale. Porian et al.'s baseline (0.835 on RefinedWeb) reproduces the raw 0.88 setup, and their per-scale tuning step does roughly the job Kaplan's batch adjustment did. They check this directly: with tuned hyperparameters but the head-FLOP and warmup issues put back, they get 0.717, essentially Kaplan's 0.73 (their Appendix H). So the gap from 0.73 to 0.5 is mostly the two fixes below. ```{python} #| label: fig-waterfall #| echo: false #| fig-cap: > #| Walking Kaplan's exponent to Chinchilla's, one fix at a time. Each point is a #| re-measurement of *a* after one more correction; the last step also switches #| to a constant learning rate. Source: Porian et al. (2024), Table 1, RefinedWeb column, with #| 95% confidence intervals as printed. Their OpenWebText2 column follows the #| same path (0.864 → 0.699 → 0.603 → 0.574 → 0.518). fig, ax = plt.subplots(figsize=(9, 4.6)) ys = np.arange(len(WATERFALL))[::-1] for y, r in zip(ys, WATERFALL.itertuples()): col = PAL["chin"] if r.Index == len(WATERFALL) - 1 else PAL["kaplan"] if r.Index > 0: prev = WATERFALL.a[r.Index - 1] ax.barh(y, prev - r.a, left=r.a, height=0.20, color=col, alpha=.22, zorder=2) ax.plot([prev, prev], [y, y + 1], color=PAL["rule_strong"], lw=0.9, ls=(0, (2, 3)), zorder=2) ax.annotate(f"{r.a - prev:+.3f}", ((prev + r.a) / 2, y + 0.22), ha="center", fontsize=9, color=col, fontweight="semibold") ax.hlines(y, r.lo, r.hi, color=col, lw=2, zorder=4) ax.scatter(r.a, y, s=54, color=col, zorder=5) ax.annotate(f"{r.a:.3f}", (0.905, y), ha="left", va="center", fontsize=10.5, color=PAL["ink"], fontweight="semibold") ax.axvline(0.5, color=PAL["chin"], lw=1.4, ls=(0, (4, 3)), alpha=.75, zorder=3) ax.annotate("Chinchilla 0.50", (0.5, len(WATERFALL) - 0.45), ha="center", va="bottom", fontsize=9.5, color=PAL["chin"], fontweight="semibold") ax.set_yticks(ys, [f"{s}\n{w}" for s, w in zip(WATERFALL.step, WATERFALL.what)]) ax.set(xlim=(0.4, 0.93), ylim=(-0.75, len(WATERFALL) - 0.15)) ax.tick_params(axis="y", length=0, labelcolor=PAL["ink"], labelsize=9.5) ax.grid(axis="y", visible=False) style(ax, xlabel="FITTED EXPONENT $a$ IN $N^* \\propto C^{a}$") plt.show() ``` ### 1. Count the output layer's FLOPs (−0.13) Kaplan excluded embeddings from $N$ and, probably because the output head shared weights with the input embedding, its per-token FLOPs were left out of the compute count too. The resulting under-approximation of compute runs from "roughly 10% at larger models to roughly 90% at smaller models," a bias that scales with the very axis you are fitting against. Counting it correctly moves $a$ from 0.835 to 0.706. Pearce & Song come at a closely related issue analytically: model total parameters as $N_T = N_E + \omega N_E^{1/3}$ (fitting $\omega = 47{,}491$ to Chinchilla's own configurations) and the non-embedding relationship stops being a power law at all. It has a small-scale regime, a large-scale regime and a transition between them, with the crossover where embeddings are half the model, around $10^7$ non-embedding parameters. Kaplan's entire sweep, 768 to 1.5B, straddles that. Taking the Chinchilla (or Epoch) loss surface and measuring it in Kaplan's non-embedding units, they get a local exponent of 0.74 (or 0.78). Their decomposition puts more weight on the embedding convention than Porian's does, partly because a fitted loss surface has no warmup problem to find. [@pearce2024] ### 2. Stop warming up for 1.57B tokens (−0.10) This is the subtle one. Kaplan used a fixed 3,000-step warmup at a $2^{19}$-token batch, about **1.57 billion warmup tokens**, identical for every model. For a small model at a small budget, the compute-optimal token count is *at or below the warmup length*. Those runs never leave warmup, so they look terrible, so small models look worse than they are, so the fit tilts toward large models. Setting warmup tokens proportional to $N$ moves $a$ to 0.602. ### 3. Tune the optimizer per scale (−0.07) Sweeping learning rate, batch size and AdamW's $\beta_2$ per model size lands at **$a = 0.497$, 95% CI (0.49, 0.50)**. Along the way they fit $\mathrm{BS} = 0.00037\,N^{0.703}$ and $\mathrm{LR} = 3.7\,N^{-0.36}$, and find the usual $\beta_2 = 0.95$ is too low at small batch sizes. This last step also switches to a constant learning rate. Their summary: the exponent matches 0.5 "to within 0.6%" and the predicted model size at Chinchilla compute lands "within 15% of Chinchilla's size." ::: {.callout-note icon=false} ## And the cosine schedule? Porian et al. tested it in isolation (§3.4) and it moves $a$ from 0.60 to 0.57, and "most of the gap remains." Their reasoning: with decay targeted at about 131B tokens, Kaplan's compute-constrained runs "see virtually no learning rate decay," less than 1.5% of the decay. Their final 0.497 is achieved with a *constant* learning rate. Pearce & Song find the same thing independently: switching from Kaplan's schedule to Chinchilla's *raised* their coefficient from 0.46 to 0.49 rather than lowering it. ::: So the field's canonical story, that Kaplan got it wrong because of the learning-rate schedule, is itself wrong. The dominant factors are an accounting error in the compute axis and a warmup length that poisoned the small-model end of the fit. ## What each law tells you to build Four laws, one budget. ```{python} #| label: tbl-planner #| code-fold: true #| code-summary: "Show the allocation code" #| tbl-cap: "Each row applies one published law to the same training budget. Kaplan uses the adjusted law with D = C/6N; Chinchilla A1 uses the Table 3 anchor; A3 and the Epoch refit use the closed form. Extrapolating any of these far beyond its fitting range is exactly the move this whole story is a cautionary tale about." def allocate(C): kN = kaplan_N(C); kD = C / (6 * kN) cN = chinchilla_N(C); cD = C / (6 * cN) hN, hD = frontier(C, **HOFFMANN) eN, eD = frontier(C, **EPOCH) return [("Kaplan 2020 (adjusted)", kN, kD), ("Chinchilla · Approach 1", cN, cD), ("Chinchilla · Approach 3", hN, hD), ("Epoch AI refit (2024)", eN, eD)] BUDGETS = {"Chinchilla / Gopher": GOPHER_BUDGET, "1e24": 1e24, "Llama 3 405B": 3.8e25} def ratio(r): return f"{r:.2f}" if r < 1 else (f"{r:.1f}" if r < 10 else f"{r:,.0f}") out = [] for name, C_budget in BUDGETS.items(): for law, n, d in allocate(C_budget): out.append({"Budget": name, "Law": law, "N*": human(n), "D*": human(d), "tok/param": ratio(d / n)}) pd.DataFrame(out).set_index(["Budget", "Law"]) ``` ## Chinchilla-optimal is not deployment-optimal Here is the part that matters most for anyone shipping a model, and it is a limitation both papers share rather than a disagreement between them. Both optimise *loss given a training budget*. Neither includes a single inference FLOP. If you train once and serve billions of tokens, you are optimising the wrong objective. ### Add inference to the objective Sardana et al. invert the problem: fix a quality target, minimise total compute. Training costs $6N$ per token, inference costs $2N$ per token, so the objective becomes [@sardana2024] $$ N^*,\, D^*_{\text{tr}} \;=\; \operatorname*{argmin}_{N,\, D_{\text{tr}} \;\mid\; L(N, D_{\text{tr}}) = \ell} \;\; 6N D_{\text{tr}} + 2N D_{\text{inf}} $$ where $D_{\text{inf}}$ is total lifetime inference tokens, input plus output. With $D_{\text{inf}} > 0$ there is no closed form; they solve it by root-finding. It is short enough to reproduce: ```{python} #| label: inference-aware from scipy.optimize import brentq, minimize_scalar # Sardana et al. Appendix A: Chinchilla's A, B, E with alpha = 0.336, beta = 0.283, # taken from De Vries (2023) to better match Hoffmann's Table A3 (not the # TeX-source values above). SARDANA = dict(E=1.69, A=406.4, B=410.7, alpha=0.336, beta=0.283) def tokens_for_loss(N, target, E, A, B, alpha, beta): """Tokens needed for N parameters to reach a given loss.""" residual = target - E - A / N**alpha return np.inf if residual <= 0 else (B / residual) ** (1 / beta) def total_flops(N, target, D_inf, p): return 6 * N * tokens_for_loss(N, target, **p) + 2 * N * D_inf def inference_aware(N_chinchilla, D_inf, p=SARDANA): """Cheapest (N, D) reaching the quality of a Chinchilla-optimal model of size N_chinchilla, once D_inf inference tokens are paid for.""" budget = 10 ** brentq(lambda lc: frontier(10**lc, **p)[0] - N_chinchilla, 18, 30) N0, D0 = frontier(budget, **p) target = loss(N0, D0, **p) floor = (p["A"] / (target - p["E"])) ** (1 / p["alpha"]) # N below this can't reach it best = minimize_scalar(lambda x: total_flops(np.exp(x), target, D_inf, p), bounds=(np.log(floor * 1.001), np.log(N_chinchilla)), method="bounded") N1 = np.exp(best.x) saved = 1 - total_flops(N1, target, D_inf, p) / total_flops(N0, target, D_inf, p) return N0, D0, N1, tokens_for_loss(N1, target, **p), saved N0, D0, N1, D1, saved = inference_aware(30e9, D_inf=1e13) print(f"Chinchilla-optimal : {N0/1e9:5.1f}B on {D0/1e12:.2f}T tokens") print(f"Inference-aware : {N1/1e9:5.1f}B on {D1/1e12:.2f}T tokens " f"({D1/D0:.2f}x the data)") print(f"Total FLOPs saved : {saved:.0%}") ``` That reproduces the paper's stated result exactly: a developer expecting a 30B-Chinchilla-quality model to see $10^{13}$ inference tokens "can reduce their total FLOPs by 28% by training a 13.6B model on 2.84× the data." In dollar terms, with realistic MFU asymmetry between training and autoregressive generation, the same quality target at 17.5B requests is reached ~58% cheaper by an 8.58B model trained on 12.1T tokens. ::: {.callout-caution icon=false} ## The caveat, from the same paper Fitting scaling-law coefficients only on typical token/parameter ratios "overestimates the impact of additional tokens at these extreme ranges." Refitting with their extreme-ratio data (47 models, 150M–6B, up to 10,000 tokens/param) gives $\alpha = 0.18$, $\beta = 0.24$ against Chinchilla's 0.34 / 0.28. Extrapolating Chinchilla's constants out to 1,000+ tokens/param will flatter your plan. ::: ### What if you run out of data? Muennighoff et al. ran more than 400 models, 10M to 9B parameters, for up to 1,500 epochs, and their finding is unusually actionable: **up to roughly 4 epochs of repeated data is nearly free**. Their 8.7B model at four epochs on 44B unique tokens ended "with only 0.5% higher validation loss than the single-epoch model." Returns then decay fast, with $R_D^* \approx 15$ repetitions as the point where repeated tokens have lost about 63% of their value, and runs at 44 epochs diverge outright. They model it by replacing $D$ with effective data [@muennighoff2023] $$ D' = U_D + U_D \, R_D^* \left(1 - e^{-R_D / R_D^*}\right) $$ which collapses to Chinchilla exactly when there is no repetition. Under data constraint they also find you should scale epochs faster than parameters, the opposite tilt to Chinchilla's equal scaling. ### What frontier labs actually did The folk version is that Meta over-trained Llama 3. Only partly. Meta's Llama 3 paper reports IsoFLOPs experiments from $6\times10^{18}$ to $10^{22}$ FLOPs, extrapolating to a $3.8\times10^{25}$ FLOP budget and suggesting "training a 402B parameter model on 16.55T tokens." They trained 405B on 15.6T, *approximately compute-optimal*, not over-trained. The over-training was deliberate and confined to the small models: "we also train our smaller models for much longer than is compute-optimal… The resulting models perform better than compute-optimal models at the same inference budget." [@llama3] Note the ratio: 15.6T tokens on 405B parameters is about 39 tokens per parameter, and Meta's own fit (402B on 16.55T) says 41. That is roughly double Chinchilla's 20, from IsoFLOP curves on their own data mix. The planner table above, which puts Chinchilla-optimal at 544B on 12T for the same budget, shows how much the "optimal" ratio moves with the data and stack. That is how Meta split the problem, and it is how I would split it. Chinchilla-optimal is the right target for the flagship, where training cost dominates; heavy over-training is the right target for the models you actually serve. ## The practitioner's checklist - Never use Kaplan's allocation exponents. 0.73/0.27 is a measurement artifact with a known, reproduced cause. The loss-vs-scale *phenomenology* (smooth power laws, predictable returns) survives intact. - Use ~20 tokens/parameter as a starting anchor, not a constant. It comes from Approaches 1 and 2 plus the Epoch refit, not from Chinchilla's published Eq. 10, which drifts to 59+ at frontier budgets. Meta's own IsoFLOP fit for Llama 3 landed near 40. - If you will serve the model, over-train it. Compute the inference-inclusive objective before sizing. At $10^{12}$–$10^{13}$ lifetime tokens the optimum moves substantially smaller-and-longer. - Fit your own law on your own data and stack. Every exponent here is entangled with tokenizer, data mix, optimizer tuning and architecture. Chinchilla's own three approaches disagreed on their own data. - Sweep hyperparameters per scale, and warm up proportionally to model size. Kaplan's discrepancy traces back to holding warmup and optimizer settings fixed across scales that should not have been. - Check that your scaling-law fit actually converged. Chinchilla's Approach 3 constants came from an L-BFGS run that stopped early because the Huber loss was averaged instead of summed, which is also why its bootstrap intervals came out absurdly narrow. - Include the output layer in your FLOP accounting. The single largest term in the reconciliation was an accounting choice. ::: {.callout-tip icon=false} ## The meta-lesson Kaplan's paper was careful, influential, and wrong about one thing, and the field spent two years and a lot of compute acting on it. Chinchilla corrected it, and had a serious flaw of its own in the very equation everyone copied. Both were caught only when independent groups re-derived the numbers from scratch, which is a good argument for re-fitting before you trust anyone's constants, including the ones in this note. ::: ## Sources {.unnumbered} Figures @fig-ratio and @fig-waterfall plot values printed in the cited papers. @fig-frontier, @fig-isoflop and every code output above are computed from published constants and are reproducible by rendering this document. Where a number is our arithmetic rather than a paper's, the caption says so. ::: {#refs} :::