Missing data

Understanding the reasons why data are missing is important for handling the remaining data correctly. If values are missing completely at random, the data sample is likely still representative of the population. But if the values are missing systematically, analysis may be biased. For example, in a study of the relation between IQ and income, if participants with an above-average IQ tend to skip the question ‘What is your salary?’, analyses that do not take into account this missing at random (MAR pattern (see below)) may falsely fail to find a positive association between IQ and salary. Because of these problems, methodologists routinely advise researchers to design studies to minimize the occurrence of missing values.[2] Graphical models can be used to describe the missing data mechanism in detail.[3][4]

The graph shows the probability distributions of the estimations of the expected intensity of depression in the population. The number of cases is 60. Let the true population be a standardised normal distribution and the non-response probability be a logistic function of the intensity of depression. The conclusion is: The more data is missing (MNAR), the more biased are the estimations. We underestimate the intensity of depression in the population.

Missing data reduces the representativeness of the sample and can therefore distort inferences about the population. Generally speaking, there are three main approaches to handle missing data: (1) Imputation—where values are filled in the place of missing data, (2) omission—where samples with invalid data are discarded from further analysis and (3) analysis—by directly applying methods unaffected by the missing values. One systematic review addressing the prevention and handling of missing data for patient-centered outcomes research identified 10 standards as necessary for the prevention and handling of missing data. These include standards for study design, study conduct, analysis, and reporting.[10]

In some practical application, the experimenters can control the level of missingness, and prevent missing values before gathering the data. For example, in computer questionnaires, it is often not possible to skip a question. A question has to be answered, otherwise one cannot continue to the next. So missing values due to the participant are eliminated by this type of questionnaire, though this method may not be permitted by an ethics board overseeing the research. In survey research, it is common to make multiple efforts to contact each individual in the sample, often sending letters to attempt to persuade those who have decided not to participate to change their minds.[11]^{: 161–187} However, such techniques can either help or hurt in terms of reducing the negative inferential effects of missing data, because the kind of people who are willing to be persuaded to participate after initially refusing or not being home are likely to be significantly different from the kinds of people who will still refuse or remain unreachable after additional effort.[11]^{: 188–198}

In situations where missing values are likely to occur, the researcher is often advised on planning to use methods of data analysis methods that are robust to missingness. An analysis is robust when we are confident that mild to moderate violations of the technique's key assumptions will produce little or no bias, or distortion in the conclusions drawn about the population.

Model based techniques, often using graphs, offer additional tools for testing missing data types (MCAR, MAR, MNAR) and for estimating parameters under missing data conditions. For example, a test for refuting MAR/MCAR reads as follows:

For any three variables X,Y, and Z where Z is fully observed and X and Y partially observed, the data should satisfy: ${\displaystyle X\perp \!\!\!\perp R_{y}|(R_{x},Z)}$.

In words, the observed portion of X should be independent on the missingness status of Y, conditional on every value of Z. Failure to satisfy this condition indicates that the problem belongs to the MNAR category.[20]

(Remark: These tests are necessary for variable-based MAR which is a slight variation of event-based MAR.[21][22][23])

When data falls into MNAR category techniques are available for consistently estimating parameters when certain conditions hold in the model.[3] For example, if Y explains the reason for missingness in X and Y itself has missing values, the joint probability distribution of X and Y can still be estimated if the missingness of Y is random. The estimand in this case will be:

${\displaystyle {\begin{aligned}P(X,Y)&=P(X|Y)P(Y)\\&=P(X|Y,R_{x}=0,R_{y}=0)P(Y|R_{y}=0)\end{aligned}}}$

where ${\displaystyle R_{x}=0}$ and ${\displaystyle R_{y}=0}$ denote the observed portions of their respective variables.

Different model structures may yield different estimands and different procedures of estimation whenever consistent estimation is possible. The preceding estimand calls for first estimating ${\displaystyle P(X|Y)}$ from complete data and multiplying it by ${\displaystyle P(Y)}$ estimated from cases in which Y is observed regardless of the status of X. Moreover, in order to obtain a consistent estimate it is crucial that the first term be ${\displaystyle P(X|Y)}$ as opposed to ${\displaystyle P(Y|X)}$.

In many cases model based techniques permit the model structure to undergo refutation tests.[23] Any model which implies the independence between a partially observed variable X and the missingness indicator of another variable Y (i.e. ${\displaystyle R_{y}}$), conditional on ${\displaystyle R_{x}}$ can be submitted to the following refutation test: ${\displaystyle X\perp \!\!\!\perp R_{y}|R_{x}=0}$.

Finally, the estimands that emerge from these techniques are derived in closed form and do not require iterative procedures such as Expectation Maximization that are susceptible to local optima.[24]

A special class of problems appears when the probability of the missingness depends on time. For example, in the trauma databases the probability to lose data about the trauma outcome depends on the day after trauma. In these cases various non-stationary Markov chain models are applied. [25]

• Acock AC (2005), "Working with missing values", Journal of Marriage and Family, 67 (4): 1012–28, doi:10.1111/j.1741-3737.2005.00191.x, archived from the original on 2013-01-05
• Allison, Paul D. (2001), Missing Data, SAGE Publishing
• Bouza-Herrera, Carlos N. (2013), Handling Missing Data in Ranked Set Sampling, Springer
• Enders, Craig K. (2010), Applied Missing Data Analysis, Guilford Press
• Graham, John W. (2012), Missing Data, Springer
• Molenberghs, Geert; Fitzmaurice, Garrett; Kenward, Michael G.; Tsiatis, Anastasios; Verbeke, Geert, eds. (2015), Handbook of Missing Data Methodology, Chapman & Hall
• Raghunathan, Trivellore (2016), Missing Data Analysis in Practice, Chapman & Hall
• Little, Roderick J. A.; Rubin, Donald B. (2002), Statistical Analysis with Missing Data (2nd ed.), Wiley
• Tsiatis, Anastasios A. (2006), Semiparametric Theory and Missing Data, Springer
• Van den Broeck J, Cunningham SA, Eeckels R, Herbst K (2005), "Data cleaning: detecting, diagnosing, and editing data abnormalities", PLOS Medicine, 2 (10): e267, doi:10.1371/journal.pmed.0020267, PMC 1198040, PMID 16138788, S2CID 5667073
• Zarate LE, Nogueira BM, Santos TR, Song MA (2006). "Techniques for Missing Value Recovering in Imbalanced Databases: Application in a marketing database with massive missing data". IEEE International Conference on Systems, Man and Cybernetics, 2006. SMC '06. Vol. 3. pp. 2658–2664. doi:10.1109/ICSMC.2006.385265.