Design effect

The design effect, commonly denoted by ${\displaystyle D_{\mathrm {eff} }}$ (at times with different subscripts), is the ratio of two theoretical variances for estimators of some parameter (${\displaystyle \theta }$):[1][5]

• In the numerator is the actual variance for an estimator of some parameter (${\displaystyle {\hat {\theta }}_{w}}$) in a given sampling design ${\displaystyle p}$;
• In the denominator is the variance assuming the same sample size, but if the sample was obtained using the estimator we would use for a simple random sampling without replacement (${\displaystyle {\hat {\theta }}_{srswor}}$).

So that:

${\displaystyle D_{\mathrm {eff} ,p}({\hat {\theta }})={\frac {\mathrm {var} ({\hat {\theta }}_{w})}{\mathrm {var} ({\hat {\theta }}_{\mathrm {srswor} })}}}$

Put differently, ${\displaystyle D_{\mathrm {eff} }}$ is by how much more the variance had increased (or decreased, in some cases) because our sample was drawn and adjusted to a specific sampling design (e.g.: using weights, or other measures) as it would be if instead the sample was from a simple random sampling (without replacement). There are many ways of calculation ${\displaystyle D_{\mathrm {eff} }}$, depending on the parameter of interest (E.g.: population total, population mean, quantiles, ratio of quantities etc.), the estimator used, and the sampling design (e.g.: clustered sampling, stratified sampling, post-stratification, multi-stage sampling, etc.).

For estimating the population mean, the ${\displaystyle D_{\mathrm {eff} }}$ (for some sampling design p) is:[4]^: 4[2]^: 54

${\displaystyle D_{\mathrm {eff} ,p}={\frac {\mathrm {var} _{p}({\bar {y}}_{p})}{(1-f)S_{y}^{2}/n}}}$

Where ${\displaystyle n}$ is the sample size, f is the fraction of the sample from the population ${\displaystyle (n/N)}$, ${\displaystyle 1-f}$ is the (squared) finite population correction (FPC), and ${\displaystyle S_{y}^{2}=}$ is the unbiassed sample variance.

The estimates of unit variance (or element variance) is when multiplying ${\displaystyle D_{\mathrm {eff} }}$ by the element's variance, so to incorporate all the complexities of the sample design.[1]^: 259

Notice how the definition of ${\displaystyle D_{\mathrm {eff} }}$ is based on parameters of the population that we often do not know (i.e.: the variances of estimators under two different sampling designs). The process of estimating ${\displaystyle D_{\mathrm {eff} }}$ for specific designs will be described in the following section.[6]^: 98

A general formula for the (theoretical) design effect of estimating a total (not the mean), for some design, is given in Cochran 1977.[2]^: 54

A related quantity to ${\displaystyle D_{\mathrm {eff} }}$, proposed by Kish in 1995, is called ${\displaystyle D_{\mathrm {eft} }}$ (Design Effect Factor).[7]^: 56[4] It is defined on the square root of the variance ratios, and also the denominator uses a simple random sample with replacement (srswr), instead of without replacement (srswor):

${\displaystyle D_{\mathrm {eft} }={\sqrt {\frac {\mathrm {var} ({\hat {\theta }}_{w})}{\mathrm {var} ({\hat {\theta }}_{\mathrm {srswr} })}}}}$

In this later definition (proposed in 1995, vs 1965) it was argued that srs "without replacement" (with its positive effect on the variance) should be captured in the definition of the design effect, since it is part of the sampling design. It is also more directly related to the use in inference (since we often use +Z*DE*SE, not +Z*DE*VAR when creating confidence intervals). Also since the finite population correction (FPC) is also harder to compute in some situations. But for many cases when the population is very large, Deft is (almost) the square root of ${\displaystyle D_{\mathrm {eff} }}$ (${\displaystyle D_{\mathrm {eft} }\approx {\sqrt {D_{\mathrm {eff} }}}}$).

The original intention for ${\displaystyle D_{\mathrm {eft} }}$ was to have it "express the effects of sample design beyond the elemental variability ${\displaystyle {\frac {S_{m}^{2}}{m}}}$, removing both the unit of measurement and sample size as nuisance parameters", this is done in order to make the design effect generalizable (relevant for) many statistics and variables within the same survey (and even between surveys).[7]^: 55 However, followup works have shown that the calculation of design effect, for parameters such as a population total or mean, has dependence on the variability of the outcome measure, which limits the original aspiration of Kish for this measure. However, this statement may loosely (i.e.: under some conditions) be true for the weighted mean.[4]^: 5

The effective sample size, also defined by Kish in 1965, is the original sample size divided by the design effect.[1]^{: 162, 259}[8]^{: 190, 192} This quantity reflects what would be the sample size that is needed to achieve the current variance of the estimator (for some parameter) with the existing design, if the sample design (and its relevant parameter estimator) were based on a simple random sample.[9]

Namely:

${\displaystyle n_{\text{eff}}={\frac {n}{D_{eff}}}}$

Put differently, it says how many responses we are left with when using an estimator that correctly adjusts for the design effect of the sampling design. For example, using the weighted mean with inverse probability weighting, instead of the simple mean.

It is also possible to get the effective sample size ratio by taking the inverse of ${\displaystyle D_{\mathrm {eff} }}$ (i.e.: ${\displaystyle {\frac {n_{eff}}{n}}={\frac {1}{D_{eff}}}}$).

When using Kish's design effect for unequal weights, you may use the following simplified formula for "Kish's Effective Sample Size"[10][1]^{: 162, 259}

${\displaystyle n_{\text{eff}}={\frac {n}{D_{\text{eff}}}}={\frac {n}{\frac {\overline {w^{2}}}{{\overline {w}}^{2}}}}={\frac {n}{\frac {{\frac {1}{n}}\sum _{i=1}^{n}w_{i}^{2}}{\left({\frac {1}{n}}\sum _{i=1}^{n}w_{i}\right)^{2}}}}={\frac {n}{\frac {n\sum _{i=1}^{n}w_{i}^{2}}{(\sum _{i=1}^{n}w_{i})^{2}}}}={\frac {(\sum _{i=1}^{n}w_{i})^{2}}{\sum _{i=1}^{n}w_{i}^{2}}}}$

Different sampling designs differ substantially in their impact on estimators (such as the mean) in terms of their bias and variance.

For example, in the cluster sampling case the units may have equal or unequal selection probabilities, irrespective to their intra-class correlation (and their negative effect of increasing the variance of our estimators). In the case of stratified sampling, the probabilities may be equal (EPSEM) or unequal. But regardless, the usage of the prior information on the stratum size in the population, during the sampling stage, could yield statistical efficiency of our estimators. For example: if we know that gender is correlated with our outcome of interest, and also know that the male-female ratio for some population is 50%-50%. Then if we made sure to sample exactly half of each gender, we've thus reduced the variance of the estimators because we've removed the variability caused by unequal proportion of males-females in our sample. Lastly, in case of adjusting to non-coverage, non-response or some stratum split of the population (unavailable during the sampling stage), we may use statistical procedures (E.g.: post-stratification and others). The result of such procedures may lead to estimations of the sampling probabilities that are similar, or very different, than the true sampling probabilities of the units. The quality of these estimators depends on the quality of the auxiliary information and the missing at random assumptions used in creating them. Even when these sampling probability estimators (propensity scores) manage to capture most of the phenomena that has produced them - the impact of the variable selection probabilities on the estimators may be small or large, depending on the data (details in the next section).

Due to the large variety in sampling designs (with or without an effect on unequal selection probabilities), different formulas have been developed to capture the potential design effect, as well as to estimate the correct variance of estimators. Sometimes, these different design effects can be compounded together (as in the case of unequal selection probability and cluster sampling, more details in the following sections). Whether or not to use these formulas, or just assume SRS, depends on to expected amount of bias reduced vs the increase in estimator variance (and in the overhead of methodological and technical complexity).[1]^: 426

There are various ways to sample units so that each unit would have the exact same probability of selection. Such methods are called equal probability sampling (EPSEM) methods. Some of the more basic methods include simple random sample (SRS, either with or without replacement) and systematic sampling for getting a fixed sample size. There is also Bernoulli sampling with a random sample size. More advanced techniques such as stratified sampling and cluster sampling can also be designed to be EPSEM. For example, in cluster sampling we can make sure to sample each cluster with probability that is proportional to its size, and then measure all the units inside the cluster. A more complex method for cluster sampling is to use a two-stage sampling by which we sample clusters at the first stage (as before, proportional to cluster size), and sample from each cluster at the second stage using SRS with a fixed proportion (E.g.: sample half of the cluster).[11]^: 3–8

In their works, Kish and others highlights several known reasons that lead to unequal selection probabilities:[1]^: 425[8]^: 185[7]^: 69[12]^{: 50, 395}[13]^: 306

1. Disproportional sampling due to selection frame or procedure. This happens when a researcher purposefully designs their sample so to over/under sample specific sub-populations or clusters. There are many cases in which this might happen. For example:

   • In stratified sampling when units from some strata are known to have a larger variance than other strata. In such cases, the intention of the researcher may be to use this prior knowledge about the variance between stratum in order to reduce the overall variance of an estimator of some population level parameter of interest (e.g.: the mean). This can be achieved by a strategy known as optimum allocation, in which a strata ${\displaystyle h}$ is over sampled proportional to higher standard deviation and lower sampling cost (i.e.: ${\displaystyle f_{h}\propto {\frac {S_{h}}{\sqrt {C_{h}}}}}$, where ${\displaystyle S_{h}}$ is the standard deviation of the outcome in ${\displaystyle h}$, and ${\displaystyle C_{h}}$ relates to the cost of recruiting one element from ${\displaystyle h}$). An example of an optimum allocation is Neyman's optimal allocation which, when cost is fixed for recruiting each stratum, the sample size is: ${\displaystyle n_{h}=n{\frac {W_{h}S_{Uh}}{\sum _{h}W_{h}S_{Uh}}}}$. Where the summation is over all strata; n is the total sample size; ${\displaystyle n_{h}}$ is the sample size for stratum h; ${\displaystyle W_{h}={\frac {N_{h}}{N}}}$ the relative size of stratum h as compared to the entire population N; and ${\displaystyle S_{Uh}}$ is the standard error of in stratum h. A related concept to optimum design is optimal experimental design.
   • If there is interest in comparing two stratum (E.g.: people from two specific socio-demographic groups, or from two regions, etc.), in which case the smaller group may be over-sampled. This way, the variance of the estimator that compares the two groups is reduced.
   • In cluster sampling there may be clusters of different sizes but the procedure samples from all clusters using SRS, and all elements in the cluster are measured (for example, if the cluster sizes are not known upfront at the stage of sampling).
   • When using a two-stage sampling so that in the first stage the clusters are sampled proportionally to their size (a.k.a.: PPS Probability Proportional to Size), but then at the second stage only a specific fixed number of units (E.g.: one or two) are selected from each cluster - this may happen due to convenience/budget considerations. A similar case is when the first stage attempt to sample using PPS, but the number of elements in each unit are inaccurate (so that some smaller cluster may have a higher-than-it-should chance of being selected. And vis-versa for larger clusters with too-small of a chance to be sampled). In such cases, the larger the errors in the sampling frame in the first stage - the larger will be the needed unequal selection probabilities.[6]^: 109
   • When the frame used for sampling includes duplication of some of the items, thus leading some items to have a larger probability than others to be sampled (E.g.: if the sampling frame was created by merging several lists. Or if recruiting users from several ad channels - in which some of the users are available for recruitment from several of the channels, while others are available to be recruited from only one of the channels). In each of these cases - different units would have different sampling probability, thus making this sampling procedure to not be EPSEM.[11]^: 3–8[8]^: 186
   • When several different samples/frames are combined. For example, if running different ad campaigns for recruiting respondents. Or when combining results from several studies done by different researchers and/or at different times (i.e.: Meta-analysis).[8]^: 188

   When disproportional sampling happens, due to sampling design decisions, the researcher may (sometimes) be able to traceback the decision and accurately calculate the exact inclusion probability. When these selection probabilities are hard to traceback, they may be estimated using some propensity score model combined with information from auxiliary variables (E.g.: age, gender, etc.).

2. Non-coverage.[1]^{: 527, 528} This happens, for example, if people are sampled based on some pre-defined list that doesn't include all the people in the population (E.g.: a phone book or using ads to recruit people to a survey). These missing units are missing due to some failure of creating the sampling frame, as opposed to deliberate exclusion of some people (E.g.: minors, people who cannot vote, etc.). The effect of non-coverage on sampling probability is considered difficult to measure (and adjust for) in various survey situations, unless strong assumptions are made.
3. Non-response. This refers to the failure of obtaining measurements on sampled units that are intended to be measured. Reasons for non-response are varied and depends on the context. A person may be temporarily unavailable, for example if they are not available to pick up the phone when the survey is done. A person may also refuse to answer the survey due to a variety of reasons, E.g.: different tendencies of people from different ethnic/demographic/socio-economic groups to respond in general; insufficient incentive to spend the time or share data; the identity of the institution that is running the survey; inability to respond (E.g.: due to illness, illiteracy, or a language barrier); respondent is not found (E.g.: they've moved an appartement); the response was lost/destroyed during encoding or transmission (i.e.: measurement error). In the context of surveys, these reasons may be related to answering the entire survey or just specific questions.[1]^: 532[8]^: 186
4. statistical adjustments. These may include methods such as post-stratification, raking, or propensity score (estimation) models - used to perform an ad-hoc adjustment of the sample to some known (or estimated) stratum sizes. Such procedures are used to mitigate issues in the sampling ranging from sampling error, under-coverage of the sampling frame to non-response.[14]^: 45[15] For example, if a simple random sample is used, a post-stratification (using some auxiliary information) does not offer an estimator that is uniformly better than just an unweighted estimator. However, it can be viewed as a more "robust" estimator.[16] Alternatively, these methods can be used to make the sample more similar to some target "controls" (i.e.: population of interest), a process also known as "standardization".[8]^: 187 In such cases, these adjustments help with providing unbiased estimators (often with the cost of increased variance, as seen in the following sections). If the original sample is a nonprobability sampling, then post-stratification adjustments are just similar to an ad-hoc quota sampling.[8]^{: 188, 189}

When the sampling design is fully known (leading to some ${\displaystyle p_{h}}$ probability of selection for some element from strata h), and the non-response is measurable (i.e.: we know that only ${\displaystyle r_{h}}$ observations answered in strata h), then an exactly known inverse probability weight can be calculated for each element i from strata h using:${\displaystyle w_{i}={\frac {1}{p_{h}r_{h}}}}$.[8]^: 186 Sometimes a statistical adjustment, such as post-stratification or raking, is used for estimating the selection probability. E.g.: when comparing the sample we have with same target population, also known as matching to controls. The estimation process may be focused only on adjusting the existing population to an alternative population (for example, if trying to extrapolate from a panel drawn from several regions to an entire country). In such a case, the adjust might be focused on some calibration factor ${\displaystyle c_{i}}$ and the weights be calculated as ${\displaystyle w_{i}={\frac {c_{i}}{p_{h}r_{h}}}}$.[8]^: 187 However, in other cases, both the under-coverage and non-response are all modeled in one go as part of the statistical adjustment, which leads to an estimation of the overall sampling probability (let's say ${\displaystyle p_{i}'}$). In such a case, the weights are simply: ${\displaystyle w_{i}={\frac {1}{p_{i}'}}}$. Notice that when statistical adjustments are used, ${\displaystyle w_{i}}$ is often estimated based on some model. The formulation in the following sections assume this ${\displaystyle w_{i}}$ is known, which is not true for statistical adjustments (since we only have ${\displaystyle {\widehat {w}}_{i}}$). However, if it is assumed that the estimation error of ${\displaystyle {\widehat {w}}_{i}}$ is very small then the following sections can be used as if it was known. Having this assumption be true depends on the size of the sample used for modeling, and is worth keeping in mind during analysis.

When the selection probabilities may be different, the sample size is random, and the pairwise selection probabilities are independent, we call this Poisson sampling.[17]

When adjusting for unequal probability selection through "individual case weights" (E.g.: inverse probability weighting), we get various types of estimators for quantities of interest. Estimators such as Horvitz–Thompson estimator yield unbiased estimators (if the selection probabilities are indeed known, or approximately known), for total and the mean of the population. Deville and Särndal (1992) coined the term "calibration estimator" for estimators using weights such that they satisfy some condition, such as having the sum of weights equal the population size. And more generally, that the weighted sum of weights is equal some quantity of an auxiliary variable: ${\displaystyle \sum w_{i}x_{i}=X}$ (e.g.: that the sum of weighted ages of the respondents is equal to the population size in each age bucket).[18][15]^: 132[19]^: 1

The two primary ways to argue about the properties of calibration estimators are:[15]^{: 133–134}[20]

1. randomization based (or, sampling design based) - in these cases, the weights (${\displaystyle w_{i}}$) and values of the outcome of interest ${\displaystyle y_{i}}$ that are measured in the sample are all treated as known. In this framework, there is variability in the (known) values of the outcome (Y). However, the only randomness comes from which of the elements in the population were picked into the sample (often denoted as ${\displaystyle I_{i}}$, getting 1 if element ${\displaystyle i}$ is in the sample and 0 if it is not). For a simple random sample, each ${\displaystyle I_{i}}$ will be an i.i.d bernoulli distribution with some parameter ${\displaystyle p}$. For general EPSEM (equal probability sampling) ${\displaystyle I_{i}}$ will still be bernoulli with some parameter ${\displaystyle p}$, but they will no longer be independent random variables. For something like post stratification, the number of elements at each strata can be modeled as a multinomial distribution with different ${\displaystyle p_{h}}$ inclusion probabilities for each element belonging to some strata ${\displaystyle h}$. In these cases the sample size itself can be a random variable.
2. model based - in these cases the sample is fixed, the weights are fixed, but the outcome of interest is treated as a random variable. For example, in the case of post-stratification, the outcome can be modeled as some linear regression function where the independent variables are indicator variables mapping each observation to its relevant strata, and the variability comes with the error term.

As we will see later, some proofs in the literature rely on the randomization-based framework, while others focus on the model-based perspective. When moving from the mean to the weighted mean, more complexity is added. For example, in the context of survey methodology oftentimes the population size itself is considered an unknown quantity that is estimated. So in the calculation of the weighted mean is in fact based on a ratio estimator, with an estimator of the total at the numerator and an estimator of the population size in the denominator (making the variance calculation to be more complex).[21]

There are many types (and subtypes) of weights, with different ways to use and interpret them. With some weights their absolute value has some important meaning, while with other weights the important part is the relative values of the weights to each other. This section presents some of the more common types of weights so that they can be referenced in followup sections.

• Frequency weights are a basic type of weighting, presented in introduction to statistics courses. With these, each weight is an integer number that indicates the absolute frequency of an item in the sample. These are also sometimes termed repeat (or occurrence) weights. The specific value has an absolute meaning that is lost if the weights are transformed (e.g.: scaling). For example: if we have the numbers 10 and 20 with the frequency weights values of 2 and 3, then when "spreading" our data it is: 10,10, 20, 20, 20 (with weights of 1 to each of these items). Frequency weights includes the amount of information contained in a dataset, and thus allows things like creating unbiased weighted variance estimation using Bessel's correction. Notice that such weights are often random variables, since the specific number of items we will see from each value in the dataset is random.
• inverse-variance weighting is when each element is assigned a weight that is the inverse of its (known) variance.[22][8]^: 187 When all elements have the same expectancy, using such weights for calculating weighted average has the least variance among all weighted averages. In the common formulation, these weights are known and not random (this seems related to reliability weights).
• Normalized (convex) weights is a set of weights that form a convex combination. I.e.: each weight is a number between 0 and 1, and the sum of all weights is equal to 1. Any set of (non negative) weights can be turned into normalized weights by dividing each weight with the sum of all weights, making these weights normalized to sum to 1.

A related form are weights normalized to sum to sample size (n). These (non-negative) weights sum to the sample size (n), and their mean is 1. Any set of weights can be normalized to sample size by dividing each weight with the average of all weights. These weights have a nice relative interpretation where elements with weight larger than 1 are more "important" (in terms of their relative influence on, say, the weighted mean) then the average observation, while weights smaller than 1 are less "important" than the average observation.

• Inverse probability weighting is when each element is given a weight that is (proportional) to the inverse probability of selecting that element. E.g., by using ${\displaystyle w_{i}={\frac {1}{p_{i}}}}$.[8]^: 185 With inverse probability weights, we learn how many items each element "represents" in the target population. Hence, the sum of such weights returns the size of the target population of interest. Inverse probability weights can be normalized to sum to 1 or normalized to sum to the sample size (n), and many of the calculations from the following sections will yield the same results.

When a sample is EPSEM then all the probabilities are equal and the inverse of the selection probability yield weights that are all equal to one another (they are all equal to ${\displaystyle {\frac {N}{n}}={\frac {1}{f}}}$, where ${\displaystyle n}$ is the sample size and ${\displaystyle N}$ is the population size). Such a sample is called a self weighting sample.[8]^: 193

There are also indirect ways of applying "weighted" adjustments. For example, the existing cases may be duplicated to impute missing observations (e.g.: from non-response), with variance estimated using methods such as multiple imputation. A complementary dealing of data is to remove (give weight of 0) to some cases. For example, when wanting to reduce the influence of over-sampled groups that are less essential for some analysis. Both cases are similar in nature to inverse probability weighting but the application in practice gives more/less rows of data (making the input potentially simpler to use in some software implementation), instead of applying an extra column of weights. Nevertheless, the consequences of such implementations are similar to just using weights. So while in the case of removing observations the data can easily be handled by common software implementations, the case of adding rows requires special adjustments for the uncertainty estimations. Not doing so may lead to erroneous conclusions(i.e.: there is no free lunch when using alternative representation of the underlying issues).[8]^{: 189, 190}

The term "Haphazard weights", coined by Kish, is used to refer to weights that correspond to unequal selection probabilities, but ones that are not related to the expectancy or variance of the selected elements.[8]^{: 190, 191}

When taking an unrestricted sample of ${\displaystyle n}$ elements, we can then randomly split these elements into ${\displaystyle H}$ disjoint stratum, each of them containing some size of ${\displaystyle n_{h}}$ elements so that ${\displaystyle \sum \limits _{h=1}^{H}n_{h}=n}$. All elements in each strata ${\displaystyle h}$ has some (known) non-negative weight assigned to them (${\displaystyle w_{h}}$). The weight ${\displaystyle w_{h}}$ can be produced by the inverse of some unequal selection probability for elements in each strata ${\displaystyle h}$ (i.e.: inverse probability weighting following something like post-stratification). In this setting, Kish's design effect, for the increase in variance of the sample weighted mean due to this design (reflected in the weights), versus SRS of some outcome variable y (when there is no correlation between the weights and the outcome, i.e.: haphazard weights) is:[1]^: 427[8]^: 191(4.2)

${\displaystyle D_{eff}={\frac {n\sum \limits _{h=1}^{H}(n_{h}w_{h}^{2})}{(\sum \limits _{h=1}^{H}n_{h}w_{h})^{2}}}}$

By treating each item from coming from its own stratum ${\displaystyle \forall h:n_{h}=1}$, Kish (in 1992) simplified the above formula to the (well known) following version:[8]^: 191(4.3)[23]^: 318[4]^: 8

${\displaystyle D_{eff}={\frac {n\sum _{i=1}^{n}w_{i}^{2}}{(\sum _{i=1}^{n}w_{i})^{2}}}={\frac {{\frac {1}{n}}\sum _{i=1}^{n}w_{i}^{2}}{\left({\frac {1}{n}}\sum _{i=1}^{n}w_{i}\right)^{2}}}={\frac {\overline {w^{2}}}{{\overline {w}}^{2}}}}$

This version of the formula is valid when one stratum had several observations taken from it (i.e.: each having the same weight), or when there are just many stratum were each one had one observation taken from it, but several of them had the same probability of selection. While the interpretation is slightly different, the calculation of the two scenarios comes out to be the same.

Notice that Kish's definition of the design effect is closely tied to the coefficient of variation (also termed relative variance, relvariance or relvar for short) of the weights (when using uncorrected (population level) sample standard deviation for estimation). This has several notations in the literature:[8]^: 191[12]^: 396

${\displaystyle D_{eff}=1+L=1+{C_{V}}^{2}=1+relvar(w)=1+{\frac {V(w)}{{\bar {w}}^{2}}}}$.

Where ${\displaystyle V(w)={\frac {\sum (w_{i}-{\bar {w}})^{2}}{n}}}$ is the population variance of ${\displaystyle w}$, and ${\displaystyle {\bar {w}}={\frac {\sum w_{i}}{n}}}$ is the mean. When the weights are normalized to sample size (so that their sum is equal to n and their mean is equal to 1), then ${\displaystyle {C_{V}}^{2}=V(w)}$ and the formula reduces to ${\displaystyle D_{eff}=1+V(w)}$. While it is true we assume the weights are fixed, we can think of their variance as the variance of an empirical distribution defined by sampling (with equal probability) one weight from our set of weights (similar to how we would think about the correlation of x and y in a simple linear regression).

The above formula gives the increase in the variance of the weighted mean based on "haphazard" weights, which reflects when y are observations selected using unequal selection probabilities (with no within-cluster correlation, and no relationship to the expectancy or variance of the outcome measurement);[8]^{: 190, 191} and y' are the observations we would have had if we got them from simple random sample, then:

${\displaystyle D_{eff(kish)}={\frac {var\left({\bar {y}}_{w}\right)}{var\left({\bar {y}}'\right)}}={\frac {var\left({\frac {\sum \limits _{i=1}^{n}w_{i}y_{i}}{\sum \limits _{i=1}^{n}w_{i}}}\right)}{var\left({\frac {\sum \limits _{i=1}^{n}y_{i}'}{n}}\right)}}}$

From a model based perspective,[24] this formula holds when all n observations (${\displaystyle y_{1},...,y_{n}}$) are (at least approximately) uncorrelated (${\displaystyle \forall (i\neq j):cor(y_{i},y_{j})=0}$), with the same variance (${\displaystyle \sigma ^{2}}$) in the response variable of interest (y). It also assumes the weights themselves are not a random variable but rather some known constants (E.g.: the inverse of probability of selection, for some pre-determined and known sampling design).

The conditions on y are trivially held if the y observations are i.i.d with the same expectation and variance. In such case we have ${\displaystyle y=y'}$, and we can estimate ${\displaystyle var\left({\bar {y}}_{w}\right)}$ by using ${\displaystyle {\overline {var\left({\bar {y}}_{w}\right)}}={\overline {var\left({\bar {y}}\right)}}\times D_{eff}}$.[8][25] If the y's are not all with the same expectations then we cannot use the estimated variance for calculation, since that estimation assumes that all ${\displaystyle y_{i}}$s have the same expectation. Specifically, if there is a correlation between the weights and the outcome variable y, then it means that the expectation of y is not the same for all observations (but rather, dependent on the specific weight value for each observation). In such a case, while the design effect formula might still be correct (if the other conditions are met), it would require a different estimator for the variance of the weighted mean. For example, it might be better to use a weighted variance estimator.

If different ${\displaystyle y_{i}}$s have different variances, then while the weighted variance could capture the correct population-level variance, the Kish's formula for the design effect may no longer be true.

A similar issue happens if there is some correlation structure in the samples (such as when using cluster sampling).

It is worth noting that some sources in the literature give the following alternative definition to Kish's design effect, stating it is: "the ratio of the variance of the weighted survey mean under disproportionate stratified sampling to the variance under proportionate stratified sampling when all stratum unit variances are equal".[23]^: 318[12]^: 396

This definition can be slightly misleading, since it might be interpreted to mean that "proportionate stratified sampling" was achieved via stratified sampling, in which a pre-determined number of units is selected from each stratum. Such selection will yield reduced variance (as compared with simple random sample), since it removes some of the uncertainty in the specific number of elements per stratum. This is different than Kish's original definition which compared the variance of the design to a simple random sample (which would yield approximately probability proportionate to sample, but not exactly - due to the variance in sample sizes in each stratum). Park and Lee (2006) reflects on this by stating that "The rationale behind the above derivation is that the loss in precision of [the weighted mean] due to haphazard unequal weighting can be approximated by the ratio of the variance under disproportionate stratified sampling to that under the proportionate stratified sampling".[4]^: 8 How far are these two definitions differ from each other is not mentioned in the literature. In his book from 1977, Cochran provides a formula for the proportional increase in variance due to deviation from optimum allocation (what, it Kish's formulas, would be called L).[2]^: 116 However, the connection from that formula to Kish's L is not apparent.

Earlier papers would use the term ${\displaystyle Deff}$.[8]^: 192 As more definitions of design effect appeared, Kish's design effect for unequal selection probabilities was denoted ${\displaystyle Deff_{kish}}$ (or ${\displaystyle Deft_{kish}^{2}}$) or simply ${\displaystyle deff_{K}}$ for short.[4]^: 8[12]^: 396[23]^: 318 Kish's design effect is also known as the "Unequal Weighting Effect" (or just UWE), termed by Liu et al. in 2002.[26]^: 2124

The estimator for the total is the "p-expanded with replacement" estimator (a.k.a.: pwr-estimator or Hansen and Hurwitz). It is based on a simple random sample (with replacement, denoted SIR) of m items (${\displaystyle y_{k}}$) from a population of size M. Each item has a probability of ${\displaystyle p_{k}}$ (k from 1 to N) to be drawn in a single draw (${\displaystyle \sum _{U}p_{k}=1}$, i.e.: it's a multinomial distribution). The probability that a specific ${\displaystyle y_{k}}$ will appear in our sample is ${\displaystyle p_{k}}$. The "p-expanded with replacement" value is ${\displaystyle Z_{i}={\frac {y_{k}}{p_{k}}}}$ with the following expectancy: ${\displaystyle E[Z_{i}]=E[I_{i}{\frac {y_{k}}{p_{k}}}]={\frac {y_{k}}{p_{k}}}E[I_{i}]={\frac {y_{k}}{p_{k}}}p_{k}=y_{k}}$. Hence ${\displaystyle {\hat {Y}}_{pwr}={\frac {1}{m}}\sum _{i}^{m}Z_{i}}$, the pwr-estimator, is an unbiased estimator for the sum total of y.[2]^: 51

In 2000, Bruce D. Spencer proposed a formula for estimating the design effect for the variance of estimating the total (not the mean) of some quantity (${\displaystyle {\hat {Y}}}$), when there is correlation between the selection probabilities of the elements and the outcome variable of interest.[27]

In this setup, a sample of size n is drawn (with replacement) from a population of size N. Each item is drawn with probability ${\displaystyle P_{i}}$ (where ${\displaystyle \sum _{i=1}^{N}P_{i}=1}$, i.e.: multinomial distribution). The selection probabilities are used to define the Normalized (convex) weights: ${\displaystyle w_{i}={\frac {1}{nP_{i}}}}$. Notice that for some random set of n items, the sum of weights will be equal to 1 only by expectancy (${\displaystyle E[w_{i}]=1}$) with some variability of the sum around it (i.e.: the sum of elements from a poisson binomial distribution). The relationship between ${\displaystyle y_{i}}$ and ${\displaystyle P_{i}}$ is defined by the following (population) simple linear regression:

${\displaystyle y_{i}=\alpha +\beta P_{i}+\epsilon _{i}}$

Where ${\displaystyle y_{i}}$ is the outcome of element i, which linearly depends on ${\displaystyle P_{i}}$ with the intercept ${\displaystyle \alpha }$ and slope ${\displaystyle \beta }$. The residual from the fitted line is ${\displaystyle \epsilon _{i}=y_{i}-(\alpha +\beta P_{i})}$. We can also define the population variances of the outcome and the residuals as ${\displaystyle \sigma _{y}^{2}}$ and ${\displaystyle \sigma _{\epsilon }^{2}}$. The correlation between ${\displaystyle P_{i}}$ and ${\displaystyle y_{i}}$ is ${\displaystyle \rho _{y,P}}$.

Spencer's (approximate) design effect, for estimating the total of y, is:[27]^: 138[28]^: 4[12]^: 401

${\displaystyle Deff_{Spencer}=(1-{\hat {\rho }}_{y,P}^{2})(1+L)+\left({\frac {\hat {\alpha }}{{\hat {\sigma }}_{y}}}\right)^{2}L}$

Where:

• ${\displaystyle {\hat {\rho }}_{y,P}^{2}}$ estimates ${\displaystyle \rho _{y,P}^{2}}$
• ${\displaystyle {\hat {\alpha }}}$ estimates the slope ${\displaystyle \alpha }$
• ${\displaystyle {\hat {\sigma }}_{y}}$ estimates the population variance ${\displaystyle \sigma _{y}}$, and
• L is the relative variance of the weights, as defined in Kish's formula: : ${\displaystyle L=cv_{w}^{2}=relvar(w)={\frac {V(w)}{{\bar {w}}^{2}}}}$.

This assumes that the regression model fits well so that the probability of selection and the residuals are independent, since it leads to the residuals, and the square residuals, to be uncorrelated with the weights. I.e.: that ${\displaystyle \rho _{\epsilon ,W}=0}$ and also ${\displaystyle \rho _{\epsilon ^{2},W}=0}$.[27]^: 138

When the population size (N) is very large, the formula can be written as:[23]^: 319

${\displaystyle Deff_{Spencer}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+\left({\frac {1}{cv_{Y}^{2}}}\right)^{2}cv_{w}^{2}}$

(since ${\displaystyle \alpha ={\bar {Y}}-\beta \times {\bar {P}}={\bar {Y}}-\beta \times {\frac {1}{N}}\approx {\bar {Y}}}$, where ${\displaystyle cv_{Y}^{2}={\frac {\sigma _{Y}^{2}}{\bar {Y}}}}$)

This approximation assumes that the linear relationship between P and y holds. And also that the correlation of the weights with the errors, and the errors squared, are both zero. I.e.: ${\displaystyle \rho _{w,e}=0}$ and ${\displaystyle \rho _{w,e^{2}}=0}$.[28]^: 4

We notice that if ${\displaystyle {\hat {\rho }}_{y,P}\approx 0}$, then ${\displaystyle {\hat {\alpha }}\approx {\bar {y}}}$ (i.e.: the average of y). In such a case, the formula reduces to

${\displaystyle Deff_{Spencer}=(1+L)+\left({\frac {1}{relvar(y)}}\right)^{2}L}$

Only if the variance of y is much larger than its mean then the right-most term is close to 0 (i.e.: ${\displaystyle relvar(y)={\frac {\sigma _{y}}{\bar {Y}}}\approx 0}$), which reduces Spencer's design effect (for the estimatoed total) to be equal to Kish's design effect (for the ratio means):[28]^: 5 ${\displaystyle Deff_{Spencer}\approx (1+L)=Deff_{Kish}}$. Otherwise, the two formula's will yield different results, which demonstrates the difference between the design effect of the total vs the one of the mean.

In 2001, Park and Lee extended Spencer's formula to the case of the ratio-mean (i.e.: estimating the mean by dividing the estimator of the total with the estimator of the population size). It is:[28]^: 4

${\displaystyle Deff_{Park\&Lee}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+{\frac {{\hat {\rho }}_{y,P}^{2}}{cv_{P}^{2}}}cv_{w}^{2}}$

Where:

• ${\displaystyle cv_{P}^{2}}$ is the (estimated) coefficient of variation of the probabilities of selection.

Park and Lee's formula is exactly equal to Kish's formula when ${\displaystyle {\hat {\rho }}_{y,P}^{2}=0}$. Both formula's relate to the design effect of the mean of y (while Spencer's Deff relates to the estimation of the total).

In general, the Deff for the total (${\displaystyle {\hat {Y}}}$) tends to be less efficient than the Deff for the ratio mean (${\displaystyle {\hat {\bar {Y}}}}$) when ${\displaystyle \rho _{y,P}}$ is small. And in general, ${\displaystyle \rho _{y,P}}$ impacts the efficiency of both design effects.[4]^: 8

For data collected using cluster sampling we assume the following structure:

• ${\displaystyle n_{k}}$ observations in each cluster and K clusters, and with a total of ${\displaystyle n=\sum n_{k}}$ observations.
• The observations have a block correlation matrix in which every pair of observations from the same cluster is correlated with an intra-class correlation of ${\displaystyle \rho }$, while every pair from difference clusters are uncorrelated.[29] I.e., for every pair of observations, ${\displaystyle i}$ and ${\displaystyle j}$, if they belong to the same cluster ${\displaystyle k}$, we get ${\displaystyle cov(y_{i},y_{j})=\rho \sigma ^{2}}$. And two items from two different clusters are not correlated, i.e.: ${\displaystyle cov(y_{i},y_{j})=0}$.
• An element from any cluster is assumed to have the same variance: ${\displaystyle var(y_{i})=\sigma _{h}^{2}=\sigma ^{2}}$.

When clusters are all of the same size ${\displaystyle n^{*}}$, the design effect D_eff, proposed by Kish in 1965 (and later re-visited by others), is given by:[1]^: 162[12]^: 399[4]^: 9[30][31][13]^: 241

${\displaystyle D_{\text{eff}}=1+(n^{*}-1)\rho .}$

It is sometimes also denoted as ${\displaystyle Deff_{C}}$.[26]^: 2124

In various papers, when cluster sizes are not equal, the above formula is also used with ${\displaystyle n^{*}}$ as the average cluster size (it is also sometimes denoted as ${\displaystyle {\bar {b}}}$).[32][24]^: 105 In such cases, Kish's formula (using the average cluster weight) serves as a conservative (upper bound) of the exact design effect.[24]^: 106

Alternative formulas exists for unequal cluster sizes.[1]^: 193 Followup work had discussed the sensitivity of using the average cluster size with various assumptions.[33]

In his paper from 1987, Kish proposed a combined design effect that incorporates both the effects due to weighting that accounts for unequal selection probabilities as well as cluster sampling:[32][24]^: 105[34]^: 4[28]^: 2

${\displaystyle Deff_{Kish}={\frac {n\sum \limits _{h=1}^{H}(n_{h}w_{h}^{2})}{\sum \limits _{h=1}^{H}(n_{h}w_{h})^{2}}}\left(1+(n^{*}-1)\rho \right)=deff_{k}\times deff_{C}}$

With notations similar to above.

This formula received a model based justification, proposed in 1999 by Gabler et al.[24]

In 2000, Liu and Aragon proposed a decomposition of unequal selection probabilities design effect for different strata in stratified sampling.[35] In 2002, Liu et al. extended that work to account for stratified sample were within each strata is a set of unequal selection probability weights. The cluster sampling is either global or per strata.[26] Similar work was done also by Park et al. in 2003.[36]