Neural scaling law

In most cases, the size of the model is simply the number of parameters. However, one complication arises with the use of sparse models, such as mixture-of-expert models.[3] In sparse models, during every inference, only a fraction of the parameters are used. In comparison, most other kinds of neural networks, such as Transformer networks, always use all their parameters during every inference.

The size of the training dataset is usually quantified by the number of data points it contains. Larger training datasets are typically preferred as they provide a richer and more diverse source of information for the model to learn from. This in turn can lead to improved generalization performance when the model is applied to unseen data.[4] However, increasing the size of the training dataset also increases the computational resources and time required for model training.

With the "pretrain, then finetune" method used in most large language models, there are two kinds of training dataset: the pretraining dataset and the finetuning dataset. Their sizes would have different effects on model performance. Generally, the finetuning dataset is less than 1% the size of pretraining dataset.[5]

In some cases, a small amount of high quality data suffices for finetuning, and more data does not improve performance.[5]

The cost of training is typically measured in terms of time (how long it takes to train the model) and computational resources (how much processing power and memory are required to train the model). It's important to note that the cost of training can be significantly reduced with efficient training algorithms, optimized software libraries, and parallel computing on specialized hardware like GPUs or TPUs.

The cost of training a neural model is a function of several factors including the size of the model, the size of the training dataset, the complexity of the training algorithm, and the computational resources available.[4] In particular, doubling the training dataset does not necessarily double the cost of training, because one may train the model for several times over the same dataset (each being an "epoch").

The performance of a neural model is evaluated based on its ability to accurately predict the output given the input data. Common metrics for evaluating model performance include:[4]

• accuracy, precision, recall, and F1 score for classification tasks;
• mean squared error (MSE) or mean absolute error (MAE) for regression tasks;
• negative log-likelihood per token (logarithm of perplexity) for language modeling.
• Elo rating in a competition against other models, such as gameplay[6] or preference by a human judge[7]

Performance can be improved by using more data, larger models, different training algorithms, regularizing the model to prevent overfitting, and early stopping using a validation set.

A 2020 analysis [8] studied statistical relations between ${\displaystyle C,N,D,L}$ over a wide range of values and found similar scaling laws, over the range of ${\displaystyle N\in [10^{3},10^{9}]}$, ${\displaystyle C\in [10^{12},10^{21}]}$, and over multiple modalities (text, video, image, text to image, etc.).[8]

In particular, the scaling laws it found are (Table 1 of [8]):

• For each modality, they fixed one of the two ${\displaystyle C,N}$, and varying the other one (${\displaystyle D}$ is varied along using ${\displaystyle D=C/6N}$), the achievable test loss satisfies
  $${\displaystyle L=L_{0}+\left({\frac {x_{0}}{x}}\right)^{\alpha }}$$

  
  where ${\displaystyle x}$ is the varied variable, and ${\displaystyle L_{0},x_{0},\alpha }$ are parameters to be found by statistical fitting. The parameter ${\displaystyle \alpha }$ is the most important one.
  • When ${\displaystyle N}$ is the varied variable, ${\displaystyle \alpha }$ ranges from ${\displaystyle 0.037}$ to ${\displaystyle 0.24}$ depending on the model modality. This corresponds to the ${\displaystyle \alpha =0.34}$ from the Chinchilla scaling paper.
  • When ${\displaystyle C}$ is the varied variable, ${\displaystyle \alpha }$ ranges from ${\displaystyle 0.048}$ to ${\displaystyle 0.19}$ depending on the model modality. This corresponds to the ${\displaystyle \beta =0.28}$ from the Chinchilla scaling paper.
• Given fixed computing budget, optimal model parameter count is consistently around
  $${\displaystyle N_{opt}(C)=\left({\frac {C}{5\times 10^{-12}{\text{petaFLOP-day}}}}\right)^{0.7}=9.0\times 10^{-7}C^{0.7}}$$

  
  The parameter ${\displaystyle 9.0\times 10^{-7}}$ varies by a factor of up to 10 for different modalities. The exponent parameter ${\displaystyle 0.7}$ varies from ${\displaystyle 0.64}$ to ${\displaystyle 0.75}$ for different modalities. This exponent corresponds to the ${\displaystyle \approx 0.5}$ from the Chinchilla scaling paper.
• It's "strongly suggested" (but not statistically checked) that ${\displaystyle D_{opt}(C)\propto N_{opt}(C)^{0.4}\propto C^{0.28}}$. This exponent corresponds to the ${\displaystyle \approx 0.5}$ from the Chinchilla scaling paper.

The scaling law of ${\displaystyle L=L_{0}+(C_{0}/C)^{0.048}}$ was confirmed during the training of GPT-3 (Figure 3.1 [9]).

One particular scaling law ("Chinchilla scaling") states that, for a large language model (LLM) autoregressively trained for one epoch, with a cosine learning rate schedule, we have:[10]

$${\displaystyle {\begin{cases}C=C_{0}ND\\L={\frac {A}{N^{\alpha }}}+{\frac {B}{D^{\beta }}}+L_{0}\end{cases}}}$$

where the variables are

• ${\displaystyle C}$ is the cost of training the model, in FLOPs.
• ${\displaystyle N}$ is the number of parameters in the model.
• ${\displaystyle D}$ is the number of tokens in the training set.
• ${\displaystyle L}$ is the average negative log-likelihood loss per token (nats/token), achieved by the trained LLM on the test dataset.
  • ${\displaystyle L_{0}}$ represents the loss of an ideal generative process on the test data
  • ${\displaystyle {\frac {A}{N^{\alpha }}}}$ captures the fact that a Transformer language model with ${\displaystyle N}$ parameters underperforms the ideal generative process
  • ${\displaystyle {\frac {B}{D^{\beta }}}}$ captures the fact that the model trained on ${\displaystyle D}$ tokens underperforms the ideal generative process

and the statistical parameters are

• ${\displaystyle C_{0}=6}$, meaning that it costs 6 FLOPs per parameter to train on one token. This is estimated by Kaplan et al.[11] Note that training cost is much higher than inference cost, as training entails both forward and backward passes, whereas inference costs 1 to 2 FLOPs per parameter to infer on one token.
• ${\displaystyle \alpha =0.34,\beta =0.28,A=406.4,B=410.7,L_{0}=1.69}$.

The statistical laws were fitted over experimental data with ${\displaystyle N\in [7\times 10^{7},1.6\times 10^{10}],D\in [5\times 10^{9},5\times 10^{11}],C\in [10^{18},10^{24}]}$.

Since there are 4 variables related by 2 equations, imposing 1 additional constraint and 1 additional optimization objective allows us to solve for all four variables. In particular, for any fixed ${\displaystyle C}$, we can uniquely solve for all 4 variables that minimizes ${\displaystyle L}$. This provides us with the optimal ${\displaystyle D_{opt}(C),N_{opt}(C)}$ for any fixed ${\displaystyle C}$:

$${\displaystyle N_{opt}(C)=G\left({\frac {C}{6}}\right)^{a},\quad D_{opt}(C)=G^{-1}\left({\frac {C}{6}}\right)^{b},\quad {\text{ where }}\quad G=\left({\frac {\alpha A}{\beta B}}\right)^{\frac {1}{\alpha +\beta }},\quad a={\frac {\beta }{\alpha +\beta }}{\text{, and }}b={\frac {\alpha }{\alpha +\beta }}{\text{. }}}$$

Plugging in the numerical values, we obtain the "Chinchilla efficient" model size and training dataset size, as well as the test loss achievable:

$${\displaystyle {\begin{cases}N_{opt}(C)=0.6\;C^{0.45}\\D_{opt}(C)=0.3\;C^{0.55}\\L_{opt}(C)=1070\;C^{-0.154}+1.7\end{cases}}}$$

Similarly, we may find the optimal training dataset size and training compute budget for any fixed model parameter size, and so on. There are other estimates for "Chinchilla efficient" model size and training dataset size. The above is based on a statistical model of ${\displaystyle L={\frac {A}{N^{\alpha }}}+{\frac {B}{D^{\beta }}}+L_{0}}$. One can also directly fit a statistical law for ${\displaystyle D_{opt}(C),N_{opt}(C)}$ without going through the detour, for which one obtains:

$${\displaystyle {\begin{cases}N_{opt}(C)=0.1\;C^{0.5}\\D_{opt}(C)=1.7\;C^{0.5}\end{cases}}}$$

or as tabulated:

${\displaystyle N_{opt}(C)}$	${\displaystyle C}$ / FLOP	${\displaystyle C}$ / FLOPs of training Gopher	${\displaystyle D_{opt}(C)}$
400 Million	1.92e+19	1/29968	8.0 Billion
1 Billion	1.21e+20	1/5706	20.2 Billion
10 Billion	1.23e+22	1/2819	205.1 Billion
67 Billion	5.76e+23	1	1.5 Trillion
175 Billion	3.85e+24	6.7	3.7 Trillion
280 Billion	9.90e+24	17.2	5.9 Trillion
520 Billion	3.43e+25	59.5	11.0 Trillion
1 Trillion	1.27e+26	221.3	21.2 Trillion
10 Trillion	1.30e+28	22515.9	216.2 Trillion

In simpler terms, the Chinchilla scaling law for training Transformer language models suggests that when given an increased budget (in FLOPs), to achieve compute-optimal, the number of model parameters (N) and the number of tokens for training the model (D) should scale in approximately equal proportions. This conclusion differs from the previous scaling law for neural language models,[11] which states that N should be scaled faster than D. The discrepancy arises from setting different cycle lengths for cosine learning rate schedulers. In estimating the Chinchilla scaling, the authors set the cycle length to be the same as the training steps, as experimental results indicate that larger cycles overestimate the loss of the models.

A 2022 analysis[12] found that many scaling behaviors of artificial neural networks follow a smoothly broken power law functional form:

${\displaystyle y=a+{\bigg (}bx^{-c_{0}}{\bigg )}\prod _{i=1}^{n}\left(1+\left({\frac {x}{d_{i}}}\right)^{1/f_{i}}\right)^{-c_{i}*f_{i}}}$

in which ${\displaystyle x}$ refers to the quantity being scaled (i.e. ${\displaystyle C}$, ${\displaystyle N}$, ${\displaystyle D}$, number of training steps, number of inference steps, or model input size) and ${\displaystyle y}$ refers to the downstream (or upstream) performance evaluation metric of interest (e.g. prediction error, cross entropy, calibration error, AUROC, BLEU score percentage, F1 score, reward, Elo rating, solve rate, or FID score) in zero-shot, prompted, or fine-tuned settings. The parameters ${\displaystyle a,b,c_{0},c_{1}...c_{n},d_{1}...d_{n},f_{1}...f_{n}}$ are found by statistical fitting.

On a log–log plot, when ${\displaystyle f_{i}}$ is not too large and ${\displaystyle a}$ is subtracted out from the y-axis, this functional form looks like a series of linear segments connected by arcs; the ${\displaystyle n}$ transitions between the segments are called "breaks", hence the name Broken Neural Scaling Laws (BNSL).

The scenarios in which the scaling behaviors of artificial neural networks were found to follow this functional form include large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, out-of-distribution detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, arithmetic, emergent abilities, double descent, supervised learning, unsupervised/self-supervised learning, and reinforcement learning (single agent and multi-agent).

The architectures for which the scaling behaviors of artificial neural networks were found to follow this functional form include ResNets, Transformers, MLPs, MLP-Mixers, Recurrent Neural Networks, Graph Neural Networks, U-Nets, Ensembles (and Non-Ensembles), MoE (Mixture of Experts) (and Non-MoE) Models, and Sparse Pruned (and Non-Sparse Unpruned) Models.

Vision transformers, similar to language transformers, exhibit scaling laws. A 2022 research trained vision transformers, with parameter counts ${\displaystyle N\in [5\times 10^{6},2\times 10^{9}]}$, on image sets of sizes ${\displaystyle D\in [3\times 10^{7},3\times 10^{9}]}$, for computing ${\displaystyle C\in [0.2,10^{4}]}$ (in units of TPUv3-core-days).[13]

After training the model, it is finetuned on ImageNet training set. Let ${\displaystyle L}$ be the error probability of the finetuned model classifying ImageNet test set. They found ${\displaystyle \min _{N,D}L=0.09+{\frac {0.26}{(C+0.01)^{0.35}}}}$.

Ghorbani, Behrooz et al.[14] studied scaling laws for neural machine translation (specifically, English as source, and German as target) in encoder-decoder Transformer models, trained until convergence on the same datasets (thus they did not fit scaling laws for computing cost ${\displaystyle C}$ or dataset size ${\displaystyle D}$). They varied ${\displaystyle N\in [10^{8},3.5\times 10^{9}]}$ They found three results:

• ${\displaystyle L}$ is a scaling law function of ${\displaystyle N_{E},N_{D}}$, where ${\displaystyle N_{E},N_{D}}$ are encoder and decoder parameter count. It is not simply a function of total parameter count ${\displaystyle N=N_{E}+N_{D}}$. The function has form ${\displaystyle L\left(N_{e},N_{d}\right)=\alpha \left({\frac {{\bar {N}}_{e}}{N_{e}}}\right)^{p_{e}}\left({\frac {{\bar {N}}_{d}}{N_{d}}}\right)^{p_{d}}+L_{\infty }}$, where ${\displaystyle \alpha ,p_{e},p_{d},L_{\infty },{\bar {N}}_{e},{\bar {N}}_{d}}$ are fitted parameters. They found that ${\displaystyle N_{d}/N\approx 0.55}$ minimizes loss if ${\displaystyle N}$ is held fixed.
• ${\displaystyle L}$ "saturates" (that is, it reaches ${\displaystyle L_{\infty }}$) for smaller models when the training and testing datasets are "source-natural" than "target-natural". A "source-natural" data point means a pair of English-German sentences, and the model is asked to translate the English sentence into German, and the English sentence is written by a natural English writer, while the German sentence is translated from the English sentence by a machine translator.[15] To construct the two kinds of datasets, the authors collected natural English and German sentences online, then used machine translation to generate their translations.
• As models grow larger, models trained on source-original datasets can achieve low loss but bad BLEU score. In contrast, models trained on target-original datasets achieve low loss and good BLEU score in tandem (Figure 10, 11 [14]).

The authors hypothesize that source-natural datasets have uniform and dull target sentences, and so a model that is trained to predict the target sentences would quickly overfit.

[16] trained Transformers for machine translations with sizes ${\displaystyle N\in [4\times 10^{5},5.6\times 10^{7}]}$ on dataset sizes ${\displaystyle D\in [6\times 10^{5},6\times 10^{9}]}$. They found the Kaplan et al (2020)[11] scaling law applied to machine translation: ${\displaystyle L(N,D)=\left[\left({\frac {N_{C}}{N}}\right)^{\frac {\alpha _{N}}{\alpha _{D}}}+{\frac {D_{C}}{D}}\right]^{\alpha _{D}}}$. They also found the BLEU score scaling as ${\displaystyle BLEU\approx Ce^{-kL}}$.

Hernandez, Danny et al.[17] studied scaling laws for transfer learning in language models. They trained a family of Transformers in three ways:

• pretraining on English, finetuning on Python
• pretraining on an equal mix of English and Python, finetuning on Python
• training on Python

The idea is that pretraining on English should help the model achieve low loss on a test set of Python text. Suppose the model has parameter count ${\displaystyle N}$, and after being finetuned on ${\displaystyle D_{F}}$ Python tokens, it achieves some loss ${\displaystyle L}$. We say that its "transferred token count" is ${\displaystyle D_{T}}$, if another model with the same ${\displaystyle N}$ achieves the same ${\displaystyle L}$ after training on ${\displaystyle D_{F}+D_{T}}$ Python tokens.

They found ${\displaystyle D_{T}=1.9e4\left(D_{F}\right)^{.18}(N)^{.38}}$ for pretraining on English text, and ${\displaystyle D_{T}=2.1e5\left(D_{F}\right)^{.096}(N)^{.38}}$ for pretraining on English and non-Python code.