Generative model

In application to classification, the observable X is frequently a continuous variable, the target Y is generally a discrete variable consisting of a finite set of labels, and the conditional probability ${\displaystyle P(Y\mid X)}$ can also be interpreted as a (non-deterministic) target function ${\displaystyle f\colon X\to Y}$, considering X as inputs and Y as outputs.

Given a finite set of labels, the two definitions of "generative model" are closely related. A model of the conditional distribution ${\displaystyle P(X\mid Y=y)}$ is a model of the distribution of each label, and a model of the joint distribution is equivalent to a model of the distribution of label values ${\displaystyle P(Y)}$, together with the distribution of observations given a label, ${\displaystyle P(X\mid Y)}$; symbolically, ${\displaystyle P(X,Y)=P(X\mid Y)P(Y).}$ Thus, while a model of the joint probability distribution is more informative than a model of the distribution of label (but without their relative frequencies), it is a relatively small step, hence these are not always distinguished.

Given a model of the joint distribution, ${\displaystyle P(X,Y)}$, the distribution of the individual variables can be computed as the marginal distributions ${\displaystyle P(X)=\sum _{y}P(X,Y=y)}$ and ${\displaystyle P(Y)=\int _{x}P(Y,X=x)}$ (considering X as continuous, hence integrating over it, and Y as discrete, hence summing over it), and either conditional distribution can be computed from the definition of conditional probability: ${\displaystyle P(X\mid Y)=P(X,Y)/P(Y)}$ and ${\displaystyle P(Y\mid X)=P(X,Y)/P(X)}$.

Given a model of one conditional probability, and estimated probability distributions for the variables X and Y, denoted ${\displaystyle P(X)}$ and ${\displaystyle P(Y)}$, one can estimate the opposite conditional probability using Bayes' rule:

${\displaystyle P(X\mid Y)P(Y)=P(Y\mid X)P(X).}$

For example, given a generative model for ${\displaystyle P(X\mid Y)}$, one can estimate:

${\displaystyle P(Y\mid X)=P(X\mid Y)P(Y)/P(X),}$

and given a discriminative model for ${\displaystyle P(Y\mid X)}$, one can estimate:

${\displaystyle P(X\mid Y)=P(Y\mid X)P(X)/P(Y).}$

Note that Bayes' rule (computing one conditional probability in terms of the other) and the definition of conditional probability (computing conditional probability in terms of the joint distribution) are frequently conflated as well.

Types of generative models are:

• Gaussian mixture model (and other types of mixture model)
• Hidden Markov model
• Probabilistic context-free grammar
• Bayesian network (e.g. Naive bayes, Autoregressive model)
• Averaged one-dependence estimators
• Latent Dirichlet allocation
• Boltzmann machine (e.g. Restricted Boltzmann machine, Deep belief network)
• Variational autoencoder
• Generative analysis
• Generative adversarial network
• Flow-based generative model
• Energy based model
• Diffusion model

If the observed data are truly sampled from the generative model, then fitting the parameters of the generative model to maximize the data likelihood is a common method. However, since most statistical models are only approximations to the true distribution, if the model's application is to infer about a subset of variables conditional on known values of others, then it can be argued that the approximation makes more assumptions than are necessary to solve the problem at hand. In such cases, it can be more accurate to model the conditional density functions directly using a discriminative model (see below), although application-specific details will ultimately dictate which approach is most suitable in any particular case.

• k-nearest neighbors algorithm
• Logistic regression
• Support Vector Machines
• Decision Tree Learning
• Random Forest
• Maximum-entropy Markov models
• Conditional random fields

Suppose the input data is ${\displaystyle x\in \{1,2\}}$, the set of labels for ${\displaystyle x}$ is ${\displaystyle y\in \{0,1\}}$, and there are the following 4 data points: ${\displaystyle (x,y)=\{(1,0),(1,1),(2,0),(2,0)\}}$

For the above data, estimating the joint probability distribution ${\displaystyle p(x,y)}$ from the empirical measure will be the following:

	${\displaystyle y=0}$	${\displaystyle y=1}$
${\displaystyle x=1}$	${\displaystyle 1/4}$	${\displaystyle 1/4}$
${\displaystyle x=2}$	${\displaystyle 2/4}$	${\displaystyle 0}$

while ${\displaystyle p(y|x)}$ will be following:

	${\displaystyle y=0}$	${\displaystyle y=1}$
${\displaystyle x=1}$	${\displaystyle 1/2}$	${\displaystyle 1/2}$
${\displaystyle x=2}$	${\displaystyle 1}$	${\displaystyle 0}$

Shannon (1948) gives an example in which a table of frequencies of English word pairs is used to generate a sentence beginning with "representing and speedily is an good"; which is not proper English but which will increasingly approximate it as the table is moved from word pairs to word triplets etc.