Censoring (statistics)

One of the earliest attempts to analyse a statistical problem involving censored data was Daniel Bernoulli's 1766 analysis of smallpox morbidity and mortality data to demonstrate the efficacy of vaccination.[2] An early paper to use the Kaplan–Meier estimator for estimating censored costs was Quesenberry et al. (1989),[3] however this approach was found to be invalid by Lin et al.[4] unless all patients accumulated costs with a common deterministic rate function over time, they proposed an alternative estimation technique known as the Lin estimator.[5]

Example of five replicate tests resulting in four failures and one suspended time resulting in censoring.

Reliability testing often consists of conducting a test on an item (under specified conditions) to determine the time it takes for a failure to occur.

• Sometimes a failure is planned and expected but does not occur: operator error, equipment malfunction, test anomaly, etc. The test result was not the desired time-to-failure but can be (and should be) used as a time-to-termination. The use of censored data is unintentional but necessary.
• Sometimes engineers plan a test program so that, after a certain time limit or number of failures, all other tests will be terminated. These suspended times are treated as right-censored data. The use of censored data is intentional.

An analysis of the data from replicate tests includes both the times-to-failure for the items that failed and the time-of-test-termination for those that did not fail.

An earlier model for censored regression, the tobit model, was proposed by James Tobin in 1958.[6]

The likelihood is the probability or probability density of what was observed, viewed as a function of parameters in an assumed model. To incorporate censored data points in the likelihood the censored data points are represented by the probability of the censored data points as a function of the model parameters given a model, i.e. a function of CDF(s) instead of the density or probability mass.

The most general censoring case is interval censoring: ${\displaystyle Pr(a<x\leqslant b)=F(b)-F(a)}$, where ${\displaystyle F(x)}$ is the CDF of the probability distribution, and the two special cases are:

• left censoring: ${\displaystyle Pr(-\infty <x\leqslant b)=F(b)-F(-\infty )=F(b)-0=F(b)=Pr(x\leqslant b)}$

• right censoring: ${\displaystyle Pr(a<x\leqslant \infty )=F(\infty )-F(a)=1-F(a)=1-Pr(x\leqslant a)=Pr(x>a)}$

For continuous probability distributions: ${\displaystyle Pr(a<x\leqslant b)=Pr(a<x<b)}$

Suppose we are interested in survival times, ${\displaystyle T_{1},T_{2},...,T_{n}}$, but we don't observe ${\displaystyle T_{i}}$ for all ${\displaystyle i}$. Instead, we observe

${\displaystyle (U_{i},\delta _{i})}$, with ${\displaystyle U_{i}=T_{i}}$ and ${\displaystyle \delta _{i}=1}$ if ${\displaystyle T_{i}}$ is actually observed, and

${\displaystyle (U_{i},\delta _{i})}$, with ${\displaystyle U_{i}<T_{i}}$ and ${\displaystyle \delta _{i}=0}$ if all we know is that ${\displaystyle T_{i}}$ is longer than ${\displaystyle U_{i}}$.

When ${\displaystyle T_{i}>U_{i},U_{i}}$ is called the censoring time.[7]

If the censoring times are all known constants, then the likelihood is

${\displaystyle L=\prod _{i,\delta _{i}=1}f(u_{i})\prod _{i,\delta _{i}=0}S(u_{i})}$

where ${\displaystyle f(u_{i})}$ = the probability density function evaluated at ${\displaystyle u_{i}}$,

and ${\displaystyle S(u_{i})}$ = the probability that ${\displaystyle T_{i}}$ is greater than ${\displaystyle u_{i}}$, called the survival function.

This can be simplified by defining the hazard function, the instantaneous force of mortality, as

${\displaystyle \lambda (u)=f(u)/S(u)}$

so

${\displaystyle f(u)=\lambda (u)S(u)}$.

Then

${\displaystyle L=\prod _{i}\lambda (u_{i})^{\delta _{i}}S(u_{i})}$.

For the exponential distribution, this becomes even simpler, because the hazard rate, ${\displaystyle \lambda }$, is constant, and ${\displaystyle S(u)=\exp(-\lambda u)}$. Then:

${\displaystyle L(\lambda )=\lambda ^{k}\exp(-\lambda \sum {u_{i}})}$,

where ${\displaystyle k=\sum {\delta _{i}}}$.

From this we easily compute ${\displaystyle {\hat {\lambda }}}$, the maximum likelihood estimate (MLE) of ${\displaystyle \lambda }$, as follows:

${\displaystyle l(\lambda )=\log(L(\lambda ))=k\log(\lambda )-\lambda \sum {u_{i}}}$.

Then

${\displaystyle dl/d\lambda =k/\lambda -\sum {u_{i}}}$.

We set this to 0 and solve for ${\displaystyle \lambda }$ to get:

${\displaystyle {\hat {\lambda }}=k/\sum u_{i}}$.

Equivalently, the mean time to failure is:

${\displaystyle 1/{\hat {\lambda }}=\sum u_{i}/k}$.

This differs from the standard MLE for the exponential distribution in that the any censored observations are considered only in the numerator.