Recommendations for Improving the PES

Population Estimates Survey (PES)

Warning: All grantees are required to annually complete the Population Estimates Survey (PES).

Objective

The objective of this document is to summarize recommendations for improving CDC’s estimate of vaccine need for the Vaccines for Children Program (VFC) by revising the methods and data sources used to complete the population estimates survey (PES). Although the PES quantifies other population categories (317, state/local), the methodology here included only addresses the VFC population and is therefore only applicable to grantees who have a Title XX1 Medicaid program.

Background

The VFC program was created in 1994 as a federal-state partnership. State- and large urban area-based immunization programs (i.e., immunization grantees) were given the responsibility for program implementation, including recruiting, enrolling, and training immunization providers, conducting provider site visits and employing other strategies to enhance the quality and integrity of the program. Federal responsibilities included interpreting existing statute and regulation, developing program policy, estimating the amount of vaccine needed to fulfill national program needs, and establishing federal contracts for the purchase of pediatric vaccines.

When the VFC program first began in 1994, the data source used to estimate national vaccine need for the VFC program was Biologics Surveillance Data, provided to CDC by the vaccine manufacturers on a regular basis. Although this data source allowed CDC to estimate program need, it had several important limitations, including a lack of awardee- or age-specific data and the inability to account for growth in vaccine uptake, particularly important in estimating the need for newly implemented vaccines.

A special module added to the National Immunization Survey (NIS) on a temporary basis during 2000-2001 included questions about insurance status and immunization benefits, allowing CDC to produce estimates of VFC eligibility at the awardee level. Combining these estimates with state-supplied population estimates provided a more accurate method for estimating vaccine need for the VFC program that addressed important limitations of Biologics Surveillance Data. Starting in FY 2005, CDC began to use this method of combining data from states with NIS data to estimate national vaccine need for the VFC program at both the state and federal levels.

As we complete our second fiscal year using this methodology, greater accuracy is necessary to determine actual vaccine need because the target VFC funding amount derived from VOFA (which uses the PES as a starting point) exceeded actual spending in FY 2006 and FY 2007. In FY 2006, for example, the difference between the VFC target budget (based on VOFA) and total spending was $168 million. In FY 2007, the VFC target budget (based on VOFA) was $2.95 billion dollars, and spending as of the end of the fiscal year is $2.19 billion, a difference of approximately $756 million.

The discrepancy between spending and estimated need has prompted CDC to look closely at the two data sources used to estimate need: NIS estimates of awardee-level VFC-eligibility and the results of the Population Estimates Survey (PES). The NIS insurance module questions were revised and updated in 2006 and the results of the new module are currently being analyzed. We have now implemented a strategy of including the module questions in the survey every year to ensure that data are as current as possible. The methods and data sources used to complete the Population Estimates Survey have been reviewed by a working group consisting of members of the Immunization Services Division since March, 2007, and the results of that review, as well as recommendations for improvements, are the focus of this document.

It is important to note that the purpose of reviewing the methods of the Population Estimates Survey is to improve our annual funding request for VFC vaccine purchase from the Office of Management and Budget. Because VFC is an entitlement program, CDC is committed to ensuring that, when supply permits, vaccine is available in sufficient quantities at VFC provider sites, ensuring that no VFC eligible child is denied vaccine.

If at any time during the year, CDC determines that the annual request for vaccine funding has underestimated the true need for VFC vaccine, CDC is able to request additional VFC vaccine purchase funds. Likewise, for any awardee who determines that its VFC allotment is not sufficient to meet ongoing, legitimate VFC needs, CDC will work with that awardee to ensure that sufficient VFC vaccine purchase funds are available.

Methods

The Population Estimates Working Group (PES Working Group)* reviewed awardee submissions for the PES for the last two years for each VFC eligibility category, including data sources used and additional comments provided by grantees as part of submission of the Population Estimates Survey. Due to a large amount of variability related specifically to the Medicaid category, a brief survey was conducted of immunization grantees to ascertain which data source (national, state-level, or other) they used and why; which variables they chose; and what, if any, adjustments they made to the original data source. The working group also compared the actual 2005 vaccine dose usage and estimated vaccine need in 19 grantees using the 2005 third party distributor order data (as a rough proxy measure of actual usage).

Findings

A review of completed PESs revealed a wide variation in data sources used by immunization grantees to complete each of the VFC eligibility categories within the survey. In general, grantees might use data from different years or different sources that were not the most current. There are two overlaps between the VFC categories: Medicaid and American Indian/Alaska Native (AI/AN), and Uninsured and AI/AN, and though a few states have adjusted the Medicaid populations downwardly, no state fully de-duplicated these categories.

For uninsured children, some of the grantees did not use the most recent Current Population Survey (CPS) data. Previous AI/AN projections were based on 1990 US Census data (updated projections based on 2000 Census data have just recently become available). Uniformly, very little data were provided to support the estimates given for the “delegated authority” category of eligibility.

The Medicaid survey demonstrated a fairly equal split between grantees using state-level data and those using national data provided by the Centers for Medicare and Medicaid Services (CMS). Regardless of data source, there was wide variability regarding which variable grantees selected to estimate the size of the Medicaid eligible population (e.g., ever enrolled, continuously enrolled, monthly average) as well as 8 of 30 respondents (27%) who reported being unsure of which variable they used to estimate the size of their VFC eligible population.

Finally, a comparison of the actual 2005 vaccine dose usage and estimated vaccine need (as estimated by the PES), revealed that VFC population estimates from the PES in the 19 grantees exceeded spending, especially in the young age group (<1), which accounted for the largest expenditure of VFC vaccine funds.

Recommendations for Improving the Methods and Data Sources for the Population Estimates Survey

As part of the process of reviewing the PES data as described above, the working group developed the following six guiding principles:

1. For CDC, the Population Estimates Survey is used to estimate the size of the VFC steady state population at specific ages—it does not quantify the number of children who will be served via catch-up campaigns planned by the awardee.
2. The data used to determine the size of the VFC-eligible population must be as accurate as possible and, to the extent possible, determined in a standardized way across grantees.
   • Among data sources that meet this criteria, data sources should be as current as possible. Where current year data are not available, older data should be adjusted to reflect secular trends (i.e., changes over time) to the extent possible. In addition, efforts will be made to use data that correspond with PES age groups when available.
3. Since there is overlap among VFC eligibility groups, de-duplication is a necessary step in estimating the size of the VFC-eligible population, particularly for large eligibility categories such as Medicaid and Uninsured.
4. The Population Estimates Survey should be constructed in such a way as to reduce the burden associated with completing the survey to the extent possible.
5. Awardees who disagree with pre-populated values and who can provide verifiable data from an alternative source may submit those data to CDC. CDC will review the data and consider making adjustments to the pre-populated fields.

General Recommendations

1. Where possible, CDC will pre-populate data fields for each awardee’s PES using the recommended data sources, variables, and methods described below.
2. Among the five VFC eligibility categories, CDC will pre-populate the PES for four categories [Medicaid, Uninsured, American Indian/Alaska Native (AI/AN), and Underinsured at Federally Qualified Health Centers/Rural Health Centers (FQHC/RHCs)]. States submitting estimates of children served under delegated authority will provide the estimates and supporting data (see below).
3. In deriving estimates for state/city awardee pairs, there is only one eligibility category in which we are not able to obtain awardee level data from the recommended data sources (uninsured children). In order to obtain awardee-level data for these pairs, we will apply a ratio derived from the 2006 NIS Insurance module to the state level data taken from the recommended data source for uninsured children (i.e., Census/CPS data). This ratio consists of NIS data about the number of uninsured children within the city versus the state.
4. For CDC’s island and territory grantees, since CDC is currently unable to identify suitable data sources, these grantees will continue to complete the PES based on local data sources. Data should be from the most current year possible and should include data obtained directly from the local Medicaid agency for the Medicaid eligibility category whenever possible. When completing the PES, island and territory grantees should provide a copy of the actual data used to derive their estimates along with the completed survey to facilitate CDC review.

Recommendations by Eligibility Category

See Table 1 for summary.

Recommendations for the Medicaid Eligibility Category

1. Data source: CMS state-level data for most current year available; same year population estimate and target population projection. Data to be pre-populated by CDC.
2. Variable: Average monthly enrollment
3. De-duplication using CMS data to remove AI/AN children who are also enrolled in Medicaid from this category.

Recommendations for the Uninsured Eligibility Category

1. Data source: Census/CPS state-level data for most current year available and target year population projection. Data to be pre-populated by CDC.
2. Variable: Uninsured for entire year (This variable, while potentially conservative, actually exceeds the same year National Health Interview Survey (NHIS) national point in time estimate.)
3. De-duplication using Census/CPS-supplied data to remove AI/AN children who are also the uninsured category.

Recommendations for the AI/AN Eligibility Category

1. Data source: Census projection for the PES survey year. Data to be pre-populated by CDC.
2. Variable: Children reporting AI/AN as the only racial/ethnic group descriptor or one of multiple racial/ethnic group descriptors
3. De-duplication not necessary for this group, as AI/AN children are being de-duplicated from the other eligibility categories.

Recommendations for the Underinsured at Federally Qualified Health Center/Rural Health Center (FQHC/RHC) Eligibility Category

1. Data source: NIS awardee-level estimates from most recent insurance module (insurance module included in 2006 NIS and to be included annually thereafter). Data to be pre-populated by CDC.
2. Variable: Children whose parents report underinsurance for immunizations who have received at least one vaccine from an FQHC or RHC.
3. Given the small size of this group and the relatively large confidence intervals, de-duplication will not be done.

Recommendations for the Delegated Authority Eligibility Category

1. Data source: Up-to-date provider profiles for all clinics for which a delegated authority agreement exists. Provider profiles should be completed according to the guidelines provided by POB. NOTE: Guidelines are available in the new Vaccines for Children Operations Guide in Module 3 (page 2) and Appendix 2.
2. Variable: Number of underinsured children per clinic
3. De-duplication should be carried out as specified in guidelines provided by POB.

Table 1

Recommendations for PES Survey: Data source, variable, adjustment, de-duplication, and pre-population of data fields, by eligibility category.

Recommendations for Population Estimates Survey

AI /AN= American Indian/Alaska Native
CMS=Centers for Medicare and Medicaid Services
CPS=Current Population Survey (Census Bureau survey)
FQHC/RCH= Federally Qualified Center/Rural Health Center (HRSA designations)
NIS = National Immunization Survey

*PES Working Group members include: Nathan Crawford, Nancy Fasano, Vanda Kelley, Melissa Moore, Victor Negron, Jeanne Santoli and Fangjun Zhou.