Gather Credible Evidence Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs STEP 1: ENGAGE STAKEHOLDERS 1.1 Determine how and to what extent to involve stakeholders in program evaluation STEP 2: DESCRIBE THE PROGRAM 2.1 Understand your program focus and priority areas 2.2 Develop your program goals and measurable (SMART) objectives 2.3 Identify the elements of your program and get familiar with logic models 2.4 Develop logic models to link program activities with outcomes STEP 3: FOCUS THE EVALUATION 3.1 Tailor the evaluation to your program and stakeholders’ needs 3.2 Determine resources and personnel available for your evaluation 3.3 Develop and prioritize evaluation questions . STEP 4: GATHER CREDIBLE EVIDENCE 4.1 Choose appropriate and reliable indicators to answer your evaluation questions 4.2 Determine the data sources and methods to measure indicators 4.3 Establish a clear procedure to collect evaluation information 4.4 Complete an evaluation plan based on program description and evaluation design STEP 5: JUSTIFY CONCLUSIONS 5.1 Analyze the evaluation data 5.2 Determine what the evaluation findings “say” about your program STEP 6: ENSURE USE OF EVALUATION FINDINGS AND SHARE LESSONS LEARNED 6.1 Share with stakeholders the results and lessons learned from the evaluation 6.2 Use evaluation findings to modify, strengthen, and improve your program SUGGESTED CITATION: Salabarría-Peńa, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007. Gather Credible Evidence The information you gather in your evaluation must be reliable and relevant for your program and stakeholders. Gathering credible evidence means that the data you collect answer the evaluation questions you developed. Step 4 will help you gather credible evidence to strengthen your evaluation findings and the recommendations that follow to improve the program component or activity you are evaluating. In Step 4 you will: • Choose appropriate and reliable indicators for your evaluation questions (Tool 4.1). • Determine data sources and methods for indicators (Tool 4.2). • Establish a procedure to collect the information (Tool 4.3). • Complete an evaluation plan based on your program description and evaluation design (Tool 4.4). PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 4 GATHER CREDIBLE EVIDENCE TOOL 4.1: CHOOSE APPROPRIATE AND RELIABLE INDICATORS TO ANSWER YOUR EVALUATION QUESTIONS INTRODUCTION In previous tools, you practiced formulating goals and writing Specific, Measurable, Achievable, Relevant, and Time-bound (SMART) objectives, and developing a logic model to visually describe how your program or program component works. You learned to distinguish between short-term, intermediate, and long- term outcomes, and to develop and prioritize your evaluation questions based on your program and stakeholders’ needs and resources available. Also, you learned to articulate the purpose of your evaluation and to identify its uses and users. The next step in your evaluation is to select the measures (i.e., indicators) you will use to determine the progress your program or activity is making. The flowchart below illustrates where indicators fit into your evaluation planning activities. ENGAGESTAKEHOLDERSTHROUGHOUTEVALUATION UNDERSTANDINGOFPROGRAMFOCUSANDPRIORITYAREASPROGRAMGOALS SMARTOUTCOMEOBJECTIVESSMARTPROCESSOBJECTIVESEVALUATIONPURPOSE,USERS,USESEVALUATIONQUESTIONSEVALUATIONDESIGNPROCESSOUTCOMEINDICATORSLOGICMODELINPUTSACTIVITIESOUTPUTSPROCESSSHORT-TERMOUTCOMESINTERMEDIATEOUTCOMESLONG-TERMOUTCOMESOUTCOME PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 174 GATHER CREDIBLE EVIDENCE 4 LEARNING OBJECTIVES After completing this tool, you should be able to: • Develop indicators that you will use to measure the outcomes and process aspects pertaining to your evaluation questions. WHAT IS AN INDICATOR? An indicator is a specific, observable, and measurable accomplishment or change that shows the progress made toward achieving a specific output or outcome in your logic model. Indicators are markers of accomplishment/progress. Some commonly used indicators include participation rates, attitudes, individual behaviors, community norms, policies, health status, and incidence and prevalence. The indicators you select should answer your evaluation questions and help you determine whether your program objectives have been achieved. WHAT ARE THE KEY ELEMENTS OF AN INDICATOR? The indicators you select must be specific, observable and measurable. This means that they can be seen (e.g., observed behavior), heard (e.g. participant responses), or read (i.e., agency records). • An indicator that is specific provides a clear description of what you want to measure. This is very similar to the “Specific” criterion of SMART objectives discussed in tool 2.2. For example, it is more specific to say “in-school adolescents aged 13- 18 who test positive for Chlamydia,” rather than “youth who have an STD”. With the latter example, what is meant by the terms “youth” and “STD” is unclear. • An indicator that is observable focuses on an action or change. For example, “the proportion of school clinic staff who can identify the risk factors for Chlamydia” will be more difficult to verify than the “proportion of school-clinic staff who can list two risk factors for Chlamydia.” Remember to use verbs that relate to observation of progress, such as “list,” “demonstrate,” “select,” “show,” or “report.” • An indicator that is measurable quantifies change and is usually reported in numerical terms (count, percentage/proportion, or ratio), which is very similar to the “Measurable” criterion of SMART objectives discussed in tool 2.2. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 175 4 GATHER CREDIBLE EVIDENCE Example of an indicator: The proportion of gonorrhea cases among women 14-49 years of age interviewed within 7 days from the date of specimen collection. • Specific: specifies the STD and the target population (gonorrhea cases among women 14-49 years of age) • Observable: indicates the action/change (interviews conducted within 7 days from the date of specimen collection) • Measurable: quantifies change which will be reported as a “proportion of cases” where the numerator will be the total number of gonorrhea cases among women 14-49 years of age interviewed within 7 days from the date of specimen collection and the denominator will be the total number of diagnosed and reported gonorrhea cases among women 14-49 years of age. WHAT IS THE LINK BETWEEN INDICATORS AND PERFORMANCE MEASURES? CDC’s Division of STD Prevention (DSTDP) has developed a set of indicators that are referred to as performance measures, and each project area receiving CDC funds is required to report on the indicators that apply to them. These indicators can be linked to the outputs component of the program logic model(s) that you develop for your overall program (e.g., Comprehensive STD Prevention System-CSPS, Infertility Prevention Program-IPP, and Syphilis Elimination Effort-SEE) and can give you a notion of “where things may stand” and how to improve performance. In Table 1, five of these measures are linked to certain outputs in the generic CSPS program logic model presented in Tool 2.4. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 176 4 GATHER CREDIBLE EVIDENCE Table 1: Examples of CSPS Process Outputs Linked with DSTDP Performance Measures (Indicators) PROCESS OUTPUTS LINKED DSTDP PERFORMANCE MEASURES (INDICATORS) Medical and Lab Services • Female admittees in juvenile detention facilities tested for chlamydia. • Proportion of female admittees to large juvenile detention facilities who were tested for Chlamydia. Partner Services • Syphilis cases’ partners identified. • Proportion of primary and secondary (P&S;) syphilis cases interviewed within 7, 14, and 30 calendar days from the date of specimen collection. • Number of associates and suspects tested, per case of P&S; syphilis. • Number of associates and suspects treated for newly diagnosed syphilis, per case of P&S; syphilis. • Proportion of ‘priority’ gonorrhea cases interviewed within 7, 14, and 30 days from the date of specimen collection. Please note that even though performance measures are indicators, they only partially determine what is going on in a program. Therefore, collecting data on performance measures alone does not constitute evaluation. To have a complete picture of what is happening in your program and answer the ‘whys’ and ‘hows’, you will need to conduct evaluation. Table 2 shows how a performance measure (indicator) is combined with other process indicators to measure outputs in an evaluation of County Z’s Chlamydia (Ct) screening initiative in juvenile detention centers. 1Juvenile detention facilities having booked 500 or more adolescent females annually For the purpose of DSTDP reporting, programs with greater than 500 admitees booked must report on all facilities, those with less than 500 may select one or more facilities to report on. 2Priority population(s) is to be locally determined (e.g., pregnant women, women aged 15-19 years, women of child-bearing age, resistant gonorrhea, MSM, etc.). PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 177 4 GATHER CREDIBLE EVIDENCE Table 2: Examples of Indicators and Use of a CDC Performance Measure for Evaluation of a Ct Screening Initiative PROCESS EVALUATION QUESTION PROCESS OUTPUTS PROCESS INDICATORS As a result of the Ct screening initiative, were adolescent females in juvenile detention center screened, counseled and treated for Ct? Female adolescents screened, counseled and treated for Ct. • Proportion of female admittees to large juvenile detention centers who were tested for Chlamydia (CDC performance measure). • Proportion of female admittees in large juvenile detention centers who received STD prevention counseling. • Proportion of female admittees to large juvenile detention centers who were treated presumptively • Proportion of female admittees to large juvenile detention centers who were treated as a result of a positive Ct test result. HOW DO YOU DEVELOP APPROPRIATE INDICATORS? The following are some steps to guide you in developing indicators. 1. Involve your program’s stakeholders in indicator development. In Tool 1.1 you identified relevant stakeholders to involve in various stages of your evaluation. Bring these individuals together to identify and develop indicators, using the steps outlined below. In this way, they will identify indicators that are meaningful and useful to them. This will help ensure the credibility of, and buy-in for, your evaluation findings. 2. Review your evaluation questions and use your logic model as a template for developing indicators. If you want your evaluation questions (see Tool 3.3) to answer whether your program is functioning as planned, develop process/performance indicators that are logically related to your logic model outputs (see Tools 2.3 and 2.4). If you want your evaluation to answer whether changes in the target population occurred as a result of your program activities, develop outcome indicators linked with your logic model outcomes. The following matrix can serve as a template for developing indicators based on the identified evaluation questions. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 178 GATHER CREDIBLE EVIDENCE 4 PROCESS EVALUATION QUESTIONS PROCESS OUTPUTS PROCESS INDICATORS SHORT-TERM OUTCOME EVALUATION QUESTIONS SHORT-TERM OUTCOMES SHORT-TERM INDICATORS 3. When appropriate, use CDC performance measures as process indicators linked with your logic model outputs. While you are required to report on relevant CDC performance measures for your funded program, these measures can be used as process indicators in your evaluation of a specific program component or activity if applicable. 4. Review your indicators to ensure they are specific, observable, and measurable. Be sure your indicators are focused and clearly written, and that they measure progress related with the program component or activity you are interested in. 5. Include an indicator related with data of previous year/months/weeks, etc. for inputs and outcomes intended to achieve change (e.g., increased adherence of clinical staff to STD guidelines, decreased prevalence of Chlamydia among adolescent females in juvenile detention facilities). It is important to have baseline data when a program wants to measure change. If this is not included, change cannot be quantified/measured. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 179 4 GATHER CREDIBLE EVIDENCE 6. Determine whether the indicators: • Provide useful information that can measure your outcomes or process aspects and answer your evaluation questions. • Are feasible in terms of whether the data are available, can be made available, or can be collected in a timely manner with the resources you have. • Are adequate to capture the measure you need. Depending on how each output/outcome is phrased, you may need to develop more than one indicator to describe completely what is being measured by that output or process aspect and/or outcome. Remember that while multiple indicators are sometimes needed for tracking the implementation and effects of a program, defining too many indicators can detract from the evaluation’s goals. SUMMARY CHECKLIST: Selecting Indicators • Involve your stakeholders in the development of and selection of indicators. • Review your evaluation questions and use your logic model as a template for developing indicators. The indicators should relate to the outputs and the shortterm, intermediate, and long-term outcomes of your logic model. • When appropriate, use CDC performance measures as indicators linked with your logic model outputs. • Review your indicators to ensure they are specific, observable, and measurable. • Include indicators related with data of previous year/months/weeks, etc. for inputs and outcomes intended to achieve change. • Ask yourself if the indicators: • Provide useful information. • Are feasible. • Are adequate in number. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 180 GATHER CREDIBLE EVIDENCE 4 CONCLUSION AND NEXT STEPS Gathering credible evidence is essential for a good evaluation. As you plan and conduct your evaluation, strive to collect information that is relevant and convincing. In this tool you learned how indicators can help you measure progress toward your program goals and objectives, and how to develop them (specific, observable, and measurable). Indicators are used as the starting point for designing data collection and reporting strategies. The next tool (4.2) will help you determine which data sources and methods you will use to gather information for measuring your indicators. ACRONYMS USED IN THIS TOOL CDC – Centers for Disease Control CSPS – Comprehensive STD Prevention System Ct – Chlamydia trachomatis DSTDP – Division of STD Prevention IPP – Infertility Prevention Program MSM – Men who have sex with men SEE – Syphilis Elimination Effort SMART – Specific, Measurable, Achievable, Relevant, and Time-bound STD – Sexually Transmitted Disease KEY TERMS Indicator: A specific, observable, and measurable accomplishment or change that shows whether progress has been made toward achieving a specific program output or outcome. Outcome indicators: These measure whether progress was made toward achieving your short-term, intermediate, or long-term outcomes. Performance measures: A set of indicators developed by CDC’s Division of STD Prevention with input from members of NCSD, state representatives of NCSD member grantees, and seven project areas where the measures were pilot-tested. Each project area receiving CDC funds is required to report on the measures (indicators) that apply to them. Process indicators: Indicators that measure whether progress is made toward achieving implementation fidelity by your program. These indicators measure whether your program is functioning as planned, and relate to the outputs in your program logic model. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 181 4 GATHER CREDIBLE EVIDENCE EXERCISE: DEVELOPING INDICATORS Listed below are the evaluation questions and the logic model outputs and short-term outcomes that you and your stakeholders want to address in a process and outcome evaluation of a syphilis media campaign targeting men who have sex with men (MSM) in City Y. Process evaluation questions: • Were the campaign activities implemented as planned? –Were press releases written and sent to media outlets? –Were posters created and distributed to MSM business venues? –Were educational pamphlets developed and distributed to MSM? Process Outputs: • Press releases written and sent to media outlets • Posters created and distributed to MSM business venues • Educational pamphlets developed and distributed to MSM Short-term Outcome Evaluation Questions: • As a result of the syphilis media campaign, were MSM reached? • As a result of the campaign, did MSM become aware of the syphilis outbreak? • As a result of the campaign, did knowledge of syphilis prevention increase among MSM? Short-term Outcomes: • MSM reached with prevention messages • Awareness of syphilis outbreak among MSM as a result of the campaign • Knowledge of syphilis prevention among MSM as a result of the campaign Review your evaluation questions and corresponding logic model components to develop appropriate indicators. Develop at least one indicator per process output or short-term outcome. Your developed indicators need to be relevant to the STD program goals and objectives of the syphilis elimination campaign. Determine if the developed indicators: PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 182 GATHER CREDIBLE EVIDENCE 4 • Are specific, observable and measurable; • Provide useful information that can measure your process and outcomes aspects and answer your evaluation questions; • Are feasible in terms of whether data is available or can be collected in a timely fashion with the resources you have available; and • Are adequate to capture the measure you need. Put the information provided above in the corresponding columns in the templates below. This will help you identify and develop the appropriate indicators. The completed templates are presented at the end of this tool. [Note: This exercise only includes short-term outcomes since measuring longer term outcomes may require more time and resources than are available in a typical four-year grant period. Tool 4.2 will discuss the sources and methods for collecting indicator information.] PROCESS EVALUATION QUESTIONS PROCESS OUTPUTS PROCESS INDICATORS SHORT-TERM OUTCOME EVALUATION QUESTIONS SHORT-TERM OUTCOMES SHORT-TERM INDICATORS PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 183 4 GATHER CREDIBLE EVIDENCE REFERENCES Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR-11). Retrieved October, 17, 2004, from http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm Centers for Disease Control and Prevention. (2002). Physical activity evaluation handbook. Retrieved October 17, 2004, from http://www.cdc.gov/nccdphp/dnpa/physical/handbook/index.htm MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved October 17, 2004, from http://www.cdc.gov/tobacco/evaluation_manual/ Evaluation.pdf Taylor-Powell, E., Steele, S., & Doughlah, M. (1996, February). Planning a program evaluation. Madison: University of Wisconsin- Extension, Division of Cooperative Extension. Retrieved October 17, 2004, from www.uwex.edu/ces/pdande/evaluation/ evaldocs.html PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 184 GATHER CREDIBLE EVIDENCE ANSWER KEY FOR EXERCISE 4 PROCESS EVALUATION QUESTIONS PROCESS OUTPUTS PROCESS INDICATORS Were the campaign activities implemented as planned? • Were press releases written and sent to media outlets? Press releases written and sent to media outlets # of press releases written # of press releases sent to media outlets # of media outlets receiving press releases • Were posters created and distributed to MSM business venues? Posters created and distributed to MSM business venues # of posters created # of posters distributed to MSM business venues # of MSM business venues accepting posters • Were educational pamphlets developed and distributed to MSM? Educational pamphlets developed and distributed to MSM # of educational pamphlets designed # of educational pamphlets distributed to MSM # of MSM receiving educational pamphlets SHORT-TERM OUTCOME EVALUATION QUESTIONS SHORT-TERM OUTCOMES SHORT-TERM INDICATORS As a result of the syphilis media campaign, were MSM reached? MSM reached with prevention messages % of MSM who recall seeing information (i.e., via brochures, posters, press releases developed for the campaign) on syphilis prevention. As a result of the campaign, did awareness of the syphilis outbreak increase among MSM? Awareness of syphilis outbreak among MSM as a result of the campaign % of MSM who recall the main message(s) of the campaign corresponding to the syphilis outbreak As a result of the campaign, did knowledge of syphilis prevention increase among MSM? Knowledge of syphilis prevention among MSM as a result of the campaign % of MSM who can describe at least two methods of preventing syphilis transmission. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 185 4 GATHER CREDIBLE EVIDENCE Please note that you should always use your logic model as a template for developing the indicators. Remember that if you want your evaluation questions to answer whether your program is functioning as planned, you need to develop process or performance indicators that are logically related to your logic model outputs. If you want your evaluation to answer whether changes in the target population occurred as a result of your program activities, you need to develop outcome indicators linked to your logic model outcomes. Tool 4.2 will assist you in determining which data sources and methods will be used to gather information on indicators. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 186 GATHER CREDIBLE EVIDENCE 4 TOOL 4.2: DETERMINE THE DATA SOURCES AND METHODS TO MEASURE INDICATORS INTRODUCTION In Step 3, you learned how to (1) focus your evaluation, (2) identify the resources and personnel that are available for the evaluation, and (3) develop and prioritize evaluation questions based on your program’s and other stakeholders’ needs and resources. In Step 4, you have begun to examine various processes for gathering credible evidence. Tool 4.1 outlined strategies for choosing appropriate and reliable indicators for your evaluation questions. Tool 4.2 will provide you with strategies for determining data sources and methods for measuring your indicators. The flowchart below provides a description of where data sources and methods fit within your evaluation activities. ENGAGESTAKEHOLDERSTHROUGHOUTEVALUATION UNDERSTANDINGOFPROGRAMFOCUSANDPRIORITYAREASPROGRAMGOALS SMARTOUTCOMEOBJECTIVESSMARTPROCESSOBJECTIVESEVALUATIONPURPOSE,USERS,USESEVALUATIONQUESTIONSEVALUATIONDESIGNDATASOURCESANDMETHODSPROCESSANDOUTCOMEINDICATORSLOGICMODELINPUTSACTIVITIESOUTPUTSPROCESSHORT-TERMOUTCOMESINTERMEDIATEOUTCOMESLONG-TERMOUTCOMESOUTCOME PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 187 4 GATHER CREDIBLE EVIDENCE LEARNING OBJECTIVES Upon completion of this tool, you will be able to: • Determine data sources for each indicator. • Identify data collection methods that are appropriate for indicators and data sources. WHAT DATA SOURCES COULD YOU USE? The first task in this tool is to decide from where/whom you will get the data to measure each indicator and what you should consider when selecting these data sources. When choosing data sources, make sure the data from the chosen source will actually answer your evaluation questions. For example, surveillance data can tell you how many Ct cases you have, but not how many people you screened to get that number of cases. Balance the need for data that can be considered useful against data that is critical with budgetary constraints. You should consider the following questions as you consider your data sources: • What do you need to know? • When do you need the data? • Are the data available from the program and from other sources? • How often do you need to collect the data? • What indicators require collection of new data? • Will the data be compared with data from elsewhere? If so, will it be similar to the other data? • Are the data credible? • How much money do you have to spend and how costly will it be to collect the data from a prospective source? Some of the data sources that you might consider using are presented in Table 1. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 188 GATHER CREDIBLE EVIDENCE Table 1: Advantages and Disadvantages of Possible Data Sources 4 EXAMPLES ADVANTAGES DISADVANTAGES Data Source: Documents • Grant proposals • Administrative records • Registration/enrollment forms • Surveillance reports • Database records • Web pages • Minutes of meetings • Brochures • The data are available and accessible. • You may know how the data were collected if gathered by your program. • The value of the data depends on how accurately and consistently it was recorded. • Existing records may not have the data collected in the format you need. • Existing records may not contain all the data that you need for your evaluation. • Due to privacy considerations, the program may not have permission from clients to use the information in their existing records for this purpose. Data Source: Individuals • STD clients’ or former clients’ knowledge, attitudes, or skills during and after an activity. • Target population(s) perception and/or acceptability of a new modality (e.g., partners therapy) • Changes in STD clients’ behavior • These are data your program directly collects from the target population(s) (primary data). • Reports from individuals and self-report data may be unreliable. However, when a combination of data sources is used (e.g. individuals and data records), the combined set of data can provide useful information. • When obtaining information directly from individuals (e.g., face-to-face interviews) be aware of the possibility of receiving socially desirable responses. For example, a client who has several sexual partners may report in an interview that s/he only has one sexual partner because s/he feels that is the more acceptable response. (continued) PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 189 4 GATHER CREDIBLE EVIDENCE Table 1: Advantages and Disadvantages of Possible Data Sources (continued) EXAMPLES ADVANTAGES DISADVANTAGES Data Source: Observations • Meetings • Client encounters • Data observed by you or by a trained observer on indicators related to behavior, facilities, and environments among others. • The data provide information on verbal/non-verbal behavior and skills. • The data can be used to supplement self-report information. • The value of the data depends on the training and skills of the observer, and the specificity of the instrument used to rate the observations. • Ratings may vary if there is more than one observer. WHAT DATA COLLECTION METHODS COULD YOU USE FOR YOUR INDICATORS ACCORDING TO YOUR RESOURCES? Once you have decided where or from whom you want to obtain the data for each of your indicators, you are ready to decide which data collection methods to use. The data collection methods to be discussed in this tool include: surveys, interviews, focus groups, observation, and document review. Their advantages and disadvantages are discussed in Table 2. You should consider the following questions when choosing a data collection method. Purpose of the evaluation: What method seems most appropriate for the purpose of your evaluation and the evaluation questions that you want answered? Users of the evaluation: Will the method allow you to gather information that can be analyzed and presented in a way that will be seen as credible by your stakeholders? PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 190 GATHER CREDIBLE EVIDENCE 4 Respondents from whom you will collect the data: Where and how can respondents best be reached? What is the culturally and linguistically appropriate method to use? Is conducting a personal interview or a survey more appropriate for certain target populations? Do you need a representative sample, a convenient sample, or the universe? Resources available (time, money, volunteers, travel expenses, supplies): Which method(s) can you afford and manage? What resource allocation is feasible? Consider when results are needed, your own abilities, costs of hiring a consultant, and other resource issues. Degree of intrusiveness: Will the method disrupt the program or be seen as intrusive by the respondents? Also consider issues of confidentiality if the information that you are seeking is sensitive. Type of information: Do you want representative information that applies to all participants (standardized information, such as from a survey that will be comparable across locations)? Or, do you want to examine the range and diversity of experiences, or tell a story about your target population(s) or a program component (e.g., case study)? Advantages and disadvantages of each method: What are the key strengths and weaknesses of each? Consider issues such as time and respondent burden, cost, necessary infrastructure, and access to records. What is most appropriate for your evaluation needs? Data collection methods you might consider using include surveys, interviews (e.g. individual, focus groups), observations, and document reviews. The advantages and disadvantages of these methods are summarized in Table 2. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 191 4 GATHER CREDIBLE EVIDENCE Table 2: Advantages and Disadvantages of Possible Data Sources METHOD ADVANTAGES DISADVANTAGES Surveys • Anonymous completion possible • Can administer to groups of people at the same time • Can be efficient and cost effective • Forced choices may miss certain responses from participants • Wording may bias responses • Impersonal Interviews (individual/ in-depth) • Can build rapport with participant • Can probe to get additional information • Can get breadth or depth of information • Time consuming • Expensive • Interviewing styles and wording may affect responses Focus Groups • Can get common impressions quickly • Can be an efficient way to get breadth and depth of information in a short time frame • Need experienced facilitator • Can be difficult and costly to schedule a group of 6-8 people • Time consuming to analyze responses Observation • Can view program operations as they occur • Difficult to interpret observed behaviors • My influence behaviors of program participants • May be expensive and time consuming to record each individual event. Document Review • Can document historical information about your program • Does not interrupt program or client routine • Information already exists • May be time consuming • Available information may be incomplete • Gathering information is dependent on quality of records kept PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 192 GATHER CREDIBLE EVIDENCE 4 HOW DO YOU IMPLEMENT A MIXED APPROACH? In reviewing which sources or method(s) to use, keep in mind that you can use a mixed approach to answer evaluation questions and measure indicators. This involves using more than one source (e.g., individuals and documents) and/or different methods (e.g., survey and focus groups). With mixed methods, you can obtain different sets of data that have the potential to produce similar results and therefore add credibility to your findings. Mixed methods can also allow you to examine different facets of the same phenomenon and add breadth and depth to your findings. An example of a mixed method approach would be conducting both a focus group and a survey with sexually active men and women aged 15-19 years to understand their sexual history. The survey would provide numeric data on average number of sex partners, frequency of condom use, and other information. The focus group could provide in-depth information on these same variables, and detailed data on “why” members of the target population have multiple sex partners and their views on condom use. Generally, it is best to use some combination of quantitative and qualitative methods to obtain the most comprehensive information to measure your indicators. Be aware that in some instances, the results you obtain from your mixed methods instruments may seem to produce conflicting results. For example, your survey of sexually active men and women aged 15-19 years, may indicate that they do use contraception while your focus group indicates that condoms are not widely used. In this example, the results from the survey would appear to contradict the results from the focus group. In this situation (resources permitting) conduct a focus group or interviews of surveyed participants. This may help you reconcile the original results of the focus group and survey. A valuable source of feedback for decisions on selecting data collection methods is someone with experience using the various methods you are considering. This person may be a staff member of your program or from other programs at the Department of Health, a consultant from a local college or university, your program consultant or evaluation staff from CDC’s Division of STD Prevention (DSTDP). Their input regarding the feasibility and cost of these data collection methods can be invaluable to you and your STD program as you choose the data collection method(s) for your indicators. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 193 4 GATHER CREDIBLE EVIDENCE WHAT IS THE RELATIONSHIPS AMONG EVALUATION QUESTIONS, INDICATORS, AND, DATA SOURCES AND METHODS? Following are two tables that examine the relationship among evaluation questions, indicators, and data collection methods for an evaluation of a professional development workshop on chlamydia screening protocols. The information you obtained in previous evaluation tools will complement the information presented in Tables 3 and 4. These will help you (1) review what process or outcome information should be collected based on the evaluation questions and (2) determine appropriate data sources and collection methods accordingly. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 194 4 GATHER CREDIBLE EVIDENCE Table 3: Example of data collection sources and methods used to measure indicators and to answer process evaluation questions for an evaluation of a professional development workshop on Chlamydia screening protocols. OVERARCHING PROCESS EVALUATION QUESTION: What was the quality of the professional development workshop the STD clinic staff received? •Outofthesixobjectiveslisted, howmanywerecoveredbythefacilitator(s)? •Whatbarriersorfacilitatorstoworkshopimplementationwereevident? •Wasthefacilitatorknowledgeableaboutworkshopcontent? •Wastheinformationpresentedinaculturallyappropriatemanner? •Didparticipantsunderstandthematerialthatwaspresented? •Didparticipantsfeelthatthematerialthatwaspresentedwasculturallyappropriate? •NumberofprofessionalworkshopobjectivesmetduringtheimplementationoftheSTDworkshop•Barriersandsupportsforconductingtheworkshoporforstafftoattend. •Percentofparticipantswhothoughtthefacilitatorwasknowledgeableabouttheworkshopcontent•Percentofparticipantswhofeltthefacilitatorpresentedinformationinaculturallyandlinguisticallyappropriatemanner. •Extenttowhichparticipantsfeltthematerialsprovidedwereunderstandable•Extenttowhichparticipantsfeltthematerialswereculturallyappropriate. EVALUATIONQUESTIONSINDICATORSDocuments(facilitators’trainingimplementationlog) Individuals(facilitatoroftheworkshop,STDprogramstaffwhoorganizedtheevent) DATASOURCEDATACOLLECTIONMETHODSDocumentreview. Interview(withfacilitatorsandprogramstaff) Individuals (workshop participants) Survey (at the end of the workshop) PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 195 4 GATHER CREDIBLE EVIDENCE Table 4: Example of data collection sources and methods used to measure indicators and to answer outcome evaluation questions for an evaluation of a professional development workshop on Chlamydia Screening Protocols. •Whatproportionoftheclinicalstaffwhoattendedtheworkshopapplyprescribedscreeningprotocolsinallclinicalencounters? •PercentofSTDclinicstaffwhoattendedtheworkshopwhofollowtheSTDscreeningprotocolsinpatientencounters. EVALUATIONQUESTIONSINDICATORS•Documents(patientrecords) •Individuals(patients, clinicians) •Observations(onthejobperformanceobservations) DATASOURCEOVERARCHINGOUTCOMEEVALUATIONQUESTION:HowdidtheprofessionaldevelopmentworkshoponSTDscreeningprotocolsimpacttheSTDclinicstaffCtscreeningpractice? DATACOLLECTIONMETHODS•Documentreview•interview(withpatientsandclinicians) •Observation SUMMARY CHECKLIST: Determine the Data Sources and Methods for Indicators Selectpotentialdatasourcesforeachindicator. Considertheadvantagesanddisadvantagesofeachdatasource. Selectyourdatacollectionmethod(s)byconsideringthe: •purposeoftheevaluation•usersoftheevaluation•characteristicsoftherespondentsfromwhomyouwillcollectthedata•resourcesavailable(time,money,volunteers,travelexpenses,supplies) •degreeofintrusiveness•typeofinformationneeded•advantagesanddisadvantagesofeachmethod Linkyourdatasourcesanddatacollectionmethodstotheindicatorsandevaluationquestions. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 196 GATHER CREDIBLE EVIDENCE 4 CONCLUSION AND NEXT STEPS In this tool you have learned the advantages and disadvantages of data sources and data collection methods from which you can gather information on your indicators. You have also learned about the issues and factors to consider as part of the process of selecting this information. Next, based on the data sources and data collection methods you have chosen for your evaluation, you will learn how to effectively collect your data. Tool 4.3 (Establish a Clear Procedure to Collect the Information), will help you establish a clear procedure for the data collection phase of your evaluation design, and address factors that may affect the quality of the information you gather. ACRONYMS USED IN THIS TOOL CDC – Centers for Disease Control Ct – Chlamydia trachomatis DSTDP – Division of STD Prevention JDC – Juvenile Detention Center SMART – Specific, Measurable, Achievable, Relevant, and Time-bound STD – Sexually Transmitted Disease KEY TERMS Focus Group: A qualitative method used to collect data from a group of people (about 6 -11) who meet for 1-2 hours to discuss their insights, ideas, and observations about a particular topic with a trained moderator. Participants are selected because they share certain characteristics (e.g., individuals who have been tested for syphilis, women in detention facilities) relevant to the evaluation. Indicator: A specific, observable, and measurable accomplishment or change that shows whether progress has been made toward achieving a specific program output or outcome. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 197 4 GATHER CREDIBLE EVIDENCE Individual interview: A data collection method which involves dialogue with individuals who are carefully selected for their personal experience and knowledge with the issues at hand. Since these interviews are conducted individually, they are useful when anonymity is an issue or when asking about sensitive topics so participants can feel free to express their ideas. Mixed-method design: A methodological approach where you collect data from more than one source and/or through different methods. The advantages of using mixed methods include: increasing the cross-checks on the evaluation findings, examining different facets of the same phenomenon, and increasing stakeholders’ confidence in the overall evaluation results. An example of mixed methods is using both a focus group and a survey to understand a target population’s reluctance to use condoms. Observation: A data collection method in which you take field notes on the behavior and activities of individuals or describe the environment while observing these in the field. For example, you might take notes on the behavior of gay men in bath houses as part of your data collection procedures, or take notes on how patients are treated by clinic staff, and use such information to further develop or improve your program. Primary data: Data directly obtained by your program (e.g., surveillance, number of sex partners of syphilis cases collected through DIS interviews). Qualitative methods: Data collection methods used to gather narrative data to better understand the experiences of the target population and how a program activity works. Quantitative methods: Data collection methods that are used to collect numerical data. An example is the use of a survey that queries respondents about their sexual history using closed-ended questions in which numbers can be assigned to responses (e.g., number of sexual partners, frequency of condom use). PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 198 GATHER CREDIBLE EVIDENCE 4 Secondary Data: Information your program can use that has been collected by someone else (e.g., national data). This may include epidemiological data, socio-demographics, health risk behaviors and health policies. Stakeholders: Individuals or organizations directly or indirectly affected by your STD program and/or the evaluation results (e.g., STD program staff, family planning staff, representatives of target populations). Surveillance data: Data collected in an ongoing, systematic way regarding agent/hazard, risk factor, exposure, or health event. Surveillance data are essential for the planning, implementation, and evaluation of public health practice. Survey: A method of collecting information that can be self- administered, administered over a telephone, administered using a computer or administered face-to-face. Surveys generally include close-ended questions that are asked to individuals in a specific order and provide multiple choice or discrete responses (e.g., “Have you been tested for syphilis in the last 6 months?”). PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 199 4 GATHER CREDIBLE EVIDENCE EXERCISE: SELECTING DATA SOURCES AND COLLECTION METHODS Listed below in Tables 5 and 6 are the evaluation questions and indicators that you and your stakeholders want to address in a process and outcome evaluation of a Chlamydia screening program in female juvenile detention centers (JDCs) in County Z. Based on the information you have learned in this tool, as well as the information listed in the tables, complete the tables by specifying the data sources and collection methods you would use as well as the rationale for source/method selection. Note: The completed tables are presented at the end of this tool. Table 5: Provide data sources, data collection methods and rationale for their selection. PROCESS EVALUATION QUESTION INDICATORS DATA SOURCE DATA COLLECTION METHODS RATIONALE FOR SOURCE/METHOD SELECTION What were the barriers and facilitators of implementing Ct screening in JDCs? • Extent to which STD program leadership identifies project as a priority. • Percent of JDCs that formally adopt the Ct screening project. • Percent of trained medical providers retained in project. • Percent of trained medical providers who are committed to the objectives of the project. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 200 GATHER CREDIBLE EVIDENCE 4 Table 6: Provide data sources, data collection methods and rationale for their selection. AsaresultoftheCtscreeninginitiative,didmoreJDCsscreenandtreatadolescentfemalesforCt. AsaresultofparticipatingintheCtscreeninginitiative,weremoreadolescentfemalesinJDCsscreenedandtreatedforChlamydia? IncreasednumberofJDCsthatprovideCtscreening, counselingandtreatmentservicesforadolescentfemales. IncreasednumberofadolescentfemalesinJDCswhoarescreenedandtreatedforCt. OUTCOMEEVALUATIONQUESTIONOUTCOMESINDICATORSDATASOURCESRATIONALEFORSOURCE/METHODSELECTIONDATACOLLECTIONMETHODPercentofJDCsthatprovideCtscreening, counseling,andtreatmentservicesforadolescentfemales3monthsbeforeand3monthsaftertheinitiative. IncreasednumberofadolescentfemalesinJDCswhoarescreenedandtreatedforCtPercentoffemalesinJDCswhowerescreenedforCt3monthsbeforeandmonthsaftertheinitiative. PercentofscreenedfemaleswithpositiveCtresultsinJDCswhoweretreated3monthsbeforeand3monthsaftertheinitiative. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 201 4 GATHER CREDIBLE EVIDENCE REFERENCES Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adolescent and School Health. (2003). Handbook for conducting youth risk behavior surveys. Atlanta, GA: Centers for Disease Control and Prevention. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved March 21, 2005, from http://www.cdc.gov/tobacco/evaluation_manual/ Evaluation.pdf Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage. Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage. Porteous, N. L., Sheldrick, B. J., & Stewart, P. J. (1997). Program evaluation tool kit: A blueprint for public health management. Ottawa-Carleton, Ontario, Canada: Ottawa-Carleton Health Department, Public Health Research, Education and Development Program. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage. Sanders, J. A. (1994). The program evaluation standards: How to assess evaluations of educational programs. (2nd ed.). Thousand Oaks, CA: Sage. Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (1994). Handbook of practical program evaluation. San Francisco: Jossey- Bass. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 202 4 GATHER CREDIBLE EVIDENCE Table 5: Process evaluation question, indicators, data source, data collection method and rationale for source/method selection WhatwerethebarriersandfacilitatorsofimplementingCtscreeninginJDCs? •ExtenttowhichSTDprogramleadershipidentifiesprojectasapriority. •PercentofJDCsthatformallyadopttheCtscreeningproject. •Percentoftrainedmedicalprovidersretainedinproject. •Percentoftrainedmedicalproviderswhoarecommittedtotheobjectivesoftheproject. PROCESSEVALUATIONQUESTIONINDICATORSDATASOURCEDATACOLLECTIONMETHODSRATIONALEFORSOURCE/METHODSELECTION•Documents(Programreportsandbudget) •Individuals(Programleadership& Ctscreeningimplementer) •Documents(Programreports) •ObservationatJDCs•Documents(programreports) •Individuals(medicalproviders) •Documentreview•Interviews•Documentreview•Observation•Documentreview•Interviews(withmedicalproviders) Thesemethodswillprovidethemostcomprehensiveandcompletedatainordertoanswertheevaluationquestion. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 203 GATHER CREDIBLE EVIDENCE Table 6: Outcome evaluation question, outcomes, indicators, data sources, data collection method, and rationale for source/method selection. OUTCOMEEVALUATIONQUESTIONOUTCOMESINDICATORSDATASOURCESRATIONALEFORSOURCE/METHODSELECTIONDATACOLLECTIONMETHOD Percent of JDCs As a result of the Increased • Documents • Document Source/method is Ct screening number of JDCs that provide Ct (detainees’ review low-cost; is easy initiative, did that provide Ct screening, medical to complete; more JDCs screen counseling, and screening, records) provides needed and treat counseling and treatment services evidence for adolescent treatment for adolescent stakeholders females for Ct? females 3 months adolescent services for before and 3 • Individuals • Interviews Source/method females. months after the (medical (focus are medium- initiative. staff) groups) cost; help validate record data; provides in-depth information on process and impacts Increased number • Documents • Document Source/method As a result of Increased of adolescent (detainees’ review are low-cost; participating in number of females in JDCs medical easy to abstract the Ct screening adolescent who are screened records, data; highly initiative, were females in JDCs and treated for Ct surveillance credible with more adolescent who are Percent of females reports). stakeholders; females in JDCs screened and in JDCs who were provide required screened and treated for Ct. screened for Ct 3 evidence treated for months before and Chlamydia? 3 months after the initiative. Percent of screened females with positive Ct results in JDCs who were treated 3 months before and 3 months after the initiative. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 204 GATHER CREDIBLE EVIDENCE 4 Tool 4.3: ESTABLISH A CLEAR PROCEDURE TO COLLECT EVALUATION INFORMATION INTRODUCTION You have identified the outputs and outcomes you want to measure, selected indicators, identified data sources and the methods you will use to collect data, and determined the resources you can devote to the evaluation. Now it is time to collect the data to answer your evaluation questions. This evaluation tool will help you establish a clear procedure for the data collection phase of your evaluation design and address factors that may affect the quality of the information you gather. The completeness and quality of the information gathered during data collection will determine whether your evaluation generates practical, accurate, and useful information. The flowchart below illustrates where establishing clear procedures to collect the information fits in with your other evaluation activities. ENGAGESTAKEHOLDERSTHROUGHOUTEVALUATION UNDERSTANDINGOFPROGRAMFOCUSANDPRIORITYAREASPROGRAMGOALS SMARTOUTCOMEOBJECTIVESSMARTPROCESSOBJECTIVESEVALUATIONPURPOSE,USERS,USESEVALUATIONOUTCOMESEVALUATIONDESIGNPROCESSANDOUTCOMEINDICATORSDATACOLLECTIONSOURCESANDMETHODSDATACOLLECTIONPROCEDURESLOGICMOELINPUTSACTIVITIESOUTPUTSPROCESSSHORT-TERMOUTCOMESINTERMEDIATEOUTCOMESLONG-TERMOUTCOMESOUTCOME PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 205 4 GATHER CREDIBLE EVIDENCE LEARNING OBJECTIVES After completing this tool, you will be able to: • Develop data collection procedures. • Establish and implement quality control procedures for data collection and management. WHAT FACTORS DO YOU NEED TO CONSIDER WHEN DEVELOPING DATA COLLECTION PROCEDURES? Data collection involves administering instruments, and gathering and organizing responses before analysis. A number of factors can affect the quality, and thus the credibility, of the information you assemble. Some of these factors include how you design data collection instruments; forecasting when data should be collected; determining who will be responsible for collecting and entering data; and ensuring organizational requirements are met (e.g., informed consent procedures, OMB approval, confidentiality concerns, or departmental approvals). Your data collection procedures should contribute to the reliability and validity of the data collection methods and measures you use. Methods are reliable if they obtain similar or identical results when used repeatedly. For example, if you repeated a blood test three times on the same blood sample, the test would be reliable if it generated the same results each time. Valid methods/measures are those that actually measure what they are supposed to measure. For example, a question that asks sexually active adolescents how often they use a condom is valid if it accurately measures their actual frequency of condom use. It is not valid if instead respondents interpret that they are being asked how frequently they should use condoms. Consider the following factors when developing the data collection procedures. 1. Using existing data systems. You have the option of using existing data systems (e.g., Disease Intervention Specialist (DIS) interviews, surveillance systems, Sexually Transmitted Disease [STD]/Management Information Systems [MIS]). An advantage of this is that the instruments may have been pre-tested to produce valid and reliable data. Be aware that if you use another existing data base to inform aspects of your evaluation, you may need to add specific questions to meet PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 206 GATHER CREDIBLE EVIDENCE 4 your evaluation needs. Although existing data systems may collect data routinely, they may have little flexibility with regard to the questions asked in the instrument. If the existing data systems in your project area can’t answer your evaluation questions, you will need to design you own evaluation instruments. 2. Designing Data Collection Instruments. The design of your data collection instruments is an important factor that influences the quality of your data. If you are designing your own data collection instruments: 1) you may want to obtain advice from individuals who have experience developing evaluation instruments (e.g., STD program or health department staff, CDC evaluation staff); and 2) build in enough time into your evaluation timeline to field-test the instruments and modify them accordingly. When developing questionnaires1, keep them simple, short, focused, and easy for respondents to complete. Use culturally appropriate language, and very precise instructions. Questionnaires generally include close-ended questions with multiple answers. Don’t combine two questions in one. For instance, “What do you think about the Chlamydia counseling you received today and the services you have received at this clinic?” This question asks about multiple issues. The first half has to do with opinions about the Chlamydia counseling, and the other half has to do with opinions about the services received at the clinic. The latter can still be broken-down into different questions by asking about specific services. The problem with this question is that you do not know which of the multiple questions the respondent is answering, which may create confusion and affect the validity of the responses. Also, consider whether the questionnaire will be completed with the aid of a computer, self- administered (paper-pencil), or administered by another person either face-to-face or by telephone. 1 Fowler, F.F. (2002). Survey research methods (3rd ed.) California: Sage Publications. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 207 4 GATHER CREDIBLE EVIDENCE After developing a questionnaire or survey, you will want to pilot test it. Pilot testing will assist you in determining: • How long it takes to complete the questionnaire. • Whether the order of the questions flow well. • Whether ample space is provided for responses. • Whether the directions, as well as the questions are understood. • Whether your questions are both reliable (i.e., obtains very similar or identical results when used repeatedly) and valid (actually measures what is supposed to measure). Another advantage to pilot testing is the opportunity to check the responses you obtain against the major evaluation questions or issues you’re exploring. Are patterns beginning to emerge? Are there identical responses showing up in the “Other (Please Specify)” category? You will want to pilot test your instrument on people who resemble your program participants. You also want to administer the instrument under conditions similar to those you’ll be using in gathering your data. Before pilot testing, ask some of your colleagues to review the questionnaire, particularly those who have interest or experience in evaluation or who may be familiar with the program or audience you’re examining. • Interview guides for use in one-on-one interviews and focus groups should include open-ended questions.2 The guide should allow some flexibility for the interviewer to probe and ask follow- up questions that can add depth to the information to be collected. Develop an interview guide that is clear and easy to follow by the interviewer. Field-test it with the interviewers and among individuals with similar characteristics to the target population(s) and modify accordingly. The effectiveness of interviews depends on the skill of the interviewer. Thus, it is critical to either have experienced interviewers or to train staff on interviewing skills. DIS can be a great resource in this process. 2 Morgan, D.L. and Krueger, R.A. (1998). The focus group kit. Thousand Oaks, CA: Sage Publications. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 208 GATHER CREDIBLE EVIDENCE 4 • When you conduct interviews it is important to document what the interviewees say, their body language, and the surrounding environment (e.g., interview took place in a noisy area and this interfered with the interview) since these factors may affect the quality of the responses. It is recommended, with participants’ permission, to tape the interviews and to take detailed notes, in case technology fails. • Observation is a useful evaluation method to document adherence to a protocol, but it can be very difficult to conduct without using a structured instrument because the lack thereof may affect the focus of the observer. Your instrument can be a checklist that indicates the actions/behaviors that the observer should record. Field-test it with your trained observers. The parameters/criteria identified in your checklist should be clear and unambiguous. • When using Document Review, develop a checklist that includes the information you need to abstract in a logical way. Also train those who will conduct chart abstraction. Keep in mind that for medical records, you may need individuals with clinical background to abstract the information, and you will need to become familiar and follow the confidentiality procedures/protocols of the agency that you will be collecting the data from. • Coding is also an important part of the data collection process. Coding can be initiated as part of the data collection design when developing your evaluation questions (both close-ended and open-ended), or once the instrument is developed. You should note that if coding issues are not thought through early on in the design process, data may not be appropriately coded. Be aware that a code-book should be drafted early in the data collection process. The data coding process is discussed in further detail in Tool 5.1. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 209 4 GATHER CREDIBLE EVIDENCE 3. Determining when you need to collect data. You need to determine when and how frequently to collect the data you need. Will you be collecting data before and/or after a particular intervention (e.g., screening, health education, testing, and media campaign)? Will you be collecting data at one particular time in your program (e.g., quarterly)? Your evaluation design should provide guidance as to when to collect certain data. Let’s assume that you are evaluating a gonorrhea prevention media campaign and one of the outcomes you are measuring relates to increased awareness of the gonorrhea outbreak in your target population. Since you have to compare changes in the target population’s awareness as a result of the media campaign, you may want to collect information from them before and immediately after the intervention. Consider the following questions to determine when and at what intervals you need to collect data: • When will the information be available? For example, if you are using STD surveillance data from your state health department, find out when the information is available and plan accordingly. • When collecting primary data (e.g., focus groups) try to collect the information from your sources when it is the most convenient and the least disruptive to them. • When will you have appropriate resources to collect the data (e.g., availability of staff at particular times, need for technical assistance)? • How long will it take to collect the data? This is particularly important in determining when your data collection efforts will start and end. Be sure to allocate sufficient time to each data collection task (e.g., completing 50 interviews with inmates: arranging interview space to facilitate inmate input, timing the interview during the field-test to determine the range of completion times; allowing for transition of inmates in and out of the interview space). When determining how much time to allocate for interviews, estimate the amount of time it will take a respondent to complete the interview, the number of interviewers available, and the number of interviews they can do in a day/week (given other responsibilities, availability of respondents, etc.). PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 210 GATHER CREDIBLE EVIDENCE 4 4. Determining who will be responsible for collecting the evaluation data you need. First, determine if your program staff have the specific skills/qualities you need (e.g., experience/training in moderating focus groups, speak the language of the target population). In Appendix A you will see suggested skills and qualities of data collectors. Please note that the data collection expertise may be found in most state/local health departments in Assessment Sections, DIS staff (interview skills), and Epidemiology Units. If you lack staff resources with the required skills, then you need a plan for meeting those requirements. Having trained/skilled individuals to collect your data affects the quality of your data. In-service training can address some of the skills you need to build/strengthen among the STD staff (see Appendix B for advice on what information to cover when training data collectors). In other cases, you will need to do outreach and recruitment to identify individuals with the skills you need. Consider contacting your DSTDP program consultant or evaluation staff who can refer you to local resources (see Tool 3.2). Take steps to include training plans or recruitment efforts in your data collection procedures. Along with the skills and qualities of your data collectors, you should consider the types of evaluation data each is collecting. If collecting information involves direct interviews with STD program staff, then someone outside the program should undertake this task to help assure frank responses. If a data collection method involves record reviews (e.g., meeting minutes, patient records), program staff may do this. Keep in mind that if your evaluation design calls for aggregating data across sites (e.g., STD clinics) or if more than one person is collecting the data, it is crucial that data be collected in the same way. This standardization of data collection will allow you to aggregate the data and/or make comparisons. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 211 4 GATHER CREDIBLE EVIDENCE 5. Dealing with administrative/logistical Issues. Your data collection procedures should also address certain administrative or logistical issues that may affect your data collection practices, and thus the quality of your data. • Independent oversight and compliance with existing regulations. Depending on the nature of the evaluation planned, the methods of obtaining data, or the evaluation questions posed, you may need to submit your evaluation proposal and data collection instruments to an institutional review board (IRB) to assure the protection of the rights of human subjects and the use of informed consent, if needed. To do this you should understand the approval process required by the IRB. Policies for the approval of data collection activities vary across states and local jurisdictions. It is important to understand the rules and regulations that apply to your target population. Also, note that if you are planning a data collection that it is federally sponsored and it involves 10 or more respondents, you need to obtain OMB (Office of Management and Budget) approval. For more information on what would required OMB approval and the process please refer to http://intranet.cdc.gov/od/ocso/osrs/ omb/OMBQ&A.pdf; or http://intranet.cdc.gov/od/ ocso/osrs/omb/OMB% 20Paperwork%20Reduction%20Act%20Fact%20Sheet.pdf. • Security and confidentiality of the information being collected. Ensuring the privacy of your program clients in an evaluation should be an important element of your design. You can help maintain confidentiality by stripping any identifiers from the data gathered and making sure it is stored in a secure place. Your data security and confidentiality procedures should extend to data entry and management. For example, decide how the information will be transferred from the data collectors to those responsible for entering data. How will the information be computerized in a way that security and confidentiality of the data are maintained? Your procedures should identify the physical location for storing the data collected. In the case of focus groups, the discussions are often tape recorded and transcribed. Make PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 212 GATHER CREDIBLE EVIDENCE 4 sure that you identify how the tapes will be stored, who will transcribe the tapes, and when will the tapes be destroyed. • Development of a codebook for questions included in the data collection instruments. A codebook is a document detailing instructions on how to code the data so that each data element is coded in a standardized way. In quantitative data, it specifies a brief name and description for each item or question in a data collection instrument. In qualitative data, coding is used to reduce the data by organizing the text (data) into categories/themes. The codes are applied to text segments that match the theme(s) associated with the code. You do not have to wait until the data are collected to develop a codebook. You can start working on this for quantitative data once you finalize the instrument so that you can design the program to be applied in analysis ahead of time. For qualitative data, you can start this coding process by determining main themes based on the questions in the instrument. Then you will modify the code scheme during and after data collection. • Completeness of data collection instruments. Before turning over the completed data collection instruments to those who will be entering, managing and analyzing the data, it is important that you carefully check each instrument to assure its completeness. If certain items or questions have not been completed, your data collectors should try to retrieve this information and/or clarify any responses that may not be clearly written. 6. Monitoring, Reviewing, and Revising Procedures. During data collection, consider what is going well, what is not going as expected, and what needs to be modified for the next data collection. Revise your data collection design and methods as needed, based on your resources (financial and human), how the data collection process is received by clients, and how it is perceived by your stakeholders, especially with respect to the usefulness of the information you are collecting. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 213 4 GATHER CREDIBLE EVIDENCE While data collection procedures are designed to maximize your success with data collection, they do not guarantee that your experience will be trouble-free. Despite all your planning, you may discover that data collection is not going exactly according to plan. This is not unusual, and you can learn from any problems that you encounter. These problems may relate to logistics (e.g. lack of child-care services for women of childbearing age participating in a focus group); personnel (e.g., some staff are not correctly completing client files); or the instruments themselves (e.g., question is being consistently misinterpreted). It is important to identify issues early on and to determine the actual source of the problem in order to correct it. Pilot testing is a way if identifying data collection issues at an early stage and we recommend that data collection instruments be piloted and revisions made before instruments and data collection procedures are used for the actual evaluation. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 214 GATHER CREDIBLE EVIDENCE 4 SUMMARY CHECKLIST: Establishing Clear Procedures for Data Collection Usedatacollectioninstrumentsthatwillcontributetothequalityofdatacollected. •Trytouseexistinginstrumentsthathavebeenpre-testedtoproducevalidandreliabledata. –Ifpossible,discussthestrengthsandweaknessesofexistingdatainstrumentswithindividualswhohaveusedtheinstrumentpreviously. •Whendevelopingnewdatacollectioninstruments: –youmaywanttoobtainadvicefromindividualswhohaveexperiencedevelopingevaluationinstruments; –field-testtheinstrumentsandmodifybasedonfeedback. Determinethetiming/frequency/scheduleofyourdatacollectionactivities. Consider: •Whenwilltheinformationyouneedbeavailable? •Whenwillyouhaveappropriateresourcestocollectthedata? •Howlongwillittaketocollectthedata? Establishwhowillberesponsibleforwhichdatagatheringactivities(e.g.,programstaff,otherstaff,paidconsultants,volunteers). •Whatskillsandqualitiesshouldthedatacollectorshave? •Howwilltheserequirementsbemet?Whatarethetrainingand/orrecruitmentprocedures? •Howwillyouassurethatdatacollectorscollectdatainastandardizedway? Determinewhatadministrativeorlogisticalissuesneedtobeaddressed. •DoyouneedtoobtainOMBapproval? •Doyouneedtoobtainapprovalfromanyboardoragency? •Areinformedconsentproceduresneeded? •Whatrulesandregulationsapplytoyourdatacollectionactivities? •Howwillyouaddresssecurityandconfidentialityrelatedtodatacollection,entry,andstorage? •Howwillyoudevelopyourcodebookforthedatacollectioninstruments? •Howwillyouassurethecompletenessofyourdatacollectioninstruments? Establishproceduresformonitoring,reviewing,andrevisingyourdatacollectioneffortstoidentifyproblemsyouencounter. •Howwillyouaddressproblemsrelatedtopersonnelandotherresources,logistics,andreliabilityandvalidityofinstruments? •Howwillyouaddressproblemsrelatedtohowthedatacollectionprocessisperceivedbystakeholders? PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 215 4 GATHER CREDIBLE EVIDENCE CONCLUSION AND NEXT STEPS Data collection procedures involve administering instruments, gathering responses, and organizing responses before the data can be analyzed. This tool has presented information on various factors and issues to consider when developing your data collection procedures to ensure the quality and accuracy of the information collected. Next, in Tool 4.4 (Complete an Evaluation Plan Based on Program Description and Evaluation Design), you will learn how to develop an evaluation plan that your program can easily follow. ACRONYMS USED IN THIS TOOL Ct – Chlamydia trachomatis DIS – Disease Intervention Specialist MIS – Management Information System STD – Sexually Transmitted Disease OMB – Office of Management and Budget KEY TERMS Code book: A document detailing instructions on how the data for a specific evaluation is coded. It describes each code so that codes are applied to the data in a standardized way. Coding: In quantitative analysis this is the process of arranging the data so that the computer can “read” the code and perform an analysis (e.g., if one of the variables is “sex” you might code this as 1 for “female” and 2 for “male”). In qualitative analysis, coding is used to reduce the data by organizing the text (data) into categories/ themes. The codes are applied to text segments that match the theme(s) associated with the code. Data collection: The process of administering instruments and gathering responses. Reliability: The consistency of a measure or question in obtaining very similar or identical results when used repeatedly. Validity: The extent to which a question actually measures what it is supposed to measure. For example, a question that asks how often an individual uses a condom is valid if it accurately measures the actual level of condom use; it is not valid if instead it measures the extent to which an individual realizes that s/he should wear a condom. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 216 GATHER CREDIBLE EVIDENCE 4 EXERCISE County Z STD program staff and stakeholders have decided to evaluate the Chlamydia (Ct) training that the health department conducted for clinical staff in the juvenile detention center. They have developed a logic model for the training program, specified indicators, and identified the data collection methods they will use based on available resources. They are now identifying the data collection procedures. Based on the information below, complete the template with information about the outcome, indicators, data source and method, and corresponding data collection procedures. In summarizing data collection procedures, specify the schedule for data collection, who is responsible for data collection, the logistics/confidentiality, and data quality control. [Note: The completed template is provided at the end of this tool—Table 1.] OUTCOME: Increased adherence of clinical staff to Ct clinical guidelines. INDICATORS: (1) Percent of trained clinical staff who can correctly apply Ct screening and treatment protocols with patients before the training. (2) Percent of trained clinical staff who can correctly apply Ct screening and treatment protocols with patients 30 days following the training. DATA SOURCE: Trained clinical staff METHOD: Observation (i.e., Ct screening observation instrument) PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 217 4 GATHER CREDIBLE EVIDENCE DATA COLLECTION PROCEDURES: Supervisors who are trained in the use of a Ct screening observation instrument will observe each trained clinical staff person and complete an observation instrument for each person within one month of the training. To test the reliability and validity of the observation instrument, it will be pre-tested with a sample of clinical staff who are conducting Ct screening, but did not attend the training that is being evaluated. To ensure confidentiality of the data, the supervisor will create a unique code number for each completed instrument. (Staff X) will collect all completed observation instruments from supervisors, check for the completeness of the instrument and place them in locked file drawers in the office of the STD Director. Only evaluation staff will have key access to the file drawers. TEMPLATE FOR EXERCISE DATA C O LLECTION P ROCED URES OUTCOME INDICATORS DATA SOURCE DATA COLLECTION METHOD SCHEDULE RESPONSIBILITY LOGISTICS/ CONFIDENTIALITY DATA QUALITY CONTROL PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 218 GATHER CREDIBLE EVIDENCE 4 APPENDIX A Suggested Skills and Qualities of Data Collectors • Organizational skills. • Good memory and an eye for details. These skills are most critical when gathering data through observation and reviewing records. • Familiarity with program activities to be evaluated (e.g., knowledge on social marketing for a condom campaign). • Experience with the data collection methods selected for the evaluation. • Cultural competence and experience in data collection with the populations the program works with. • Communication skills, including being a good listener and communicating concepts clearly. • People skills, such as being respectful (not condescending) and the ability to start and maintain conversations with individuals they do not know. These skills are particularly important when interacting directly with the target population(s) (e.g., interviews and focus groups). • Ability to refrain from expressing one’s own opinions, attitudes, and ideas because this may influence individuals’ responses. • Flexibility in adapting to changes in the way and the number of questions to ask (e.g., you may need to revise your data collection instruments by adding/deleting/rewording questions). • Dependability. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 219 4 GATHER CREDIBLE EVIDENCE APPENDIX B Training Data Collectors More than one person may be involved in your data collection effort. The level and extent of training they need depends on the complexity of the data collection method and the individuals’ skills. Your training should include: • An orientation on the purpose of the evaluation and how the information will be used. • Logistical details involved in data collection, including where the forms/instruments are obtained, what to do with them when completed, any approvals that must be obtained, calendar and timing considerations, and safeguarding confidentiality of information and information sources. • Data collection methods and techniques (e.g., personal interviews, focus groups, review of program /clinical records, and/or observation techniques). For example, if you are conducting interviews, your interviewers should fully understand the questions being asked, when to probe for an expanded response or for clarification, how to carry out the interview, and how to record the data objectively, consistently, and carefully. • Opportunities to practice administering the instrument (e.g., questionnaire, interview guide, observation log). Practice makes perfect and increases the quality of the evaluation data. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 220 GATHER CREDIBLE EVIDENCE 4 REFERENCES Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR-11). Retrieved October 17, 2004, from http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm Centers for Disease Control and Prevention. (n.d.). Oral health infrastructure development tools, Step 2B: Logic models. Retrieved October 17, 2004, from http://www.cdc.gov/oralhealth/ library/pdf/logic_models.pdf Taylor-Powell, E., Steele, S., & Doughlah, M. (1996, February). Planning a program evaluation. Madison: University of Wisconsin- Extension, Division of Cooperative Extension. Retrieved October 17, 2004, from www.uwex.edu/ces/pdande/ evaluation/evaldocs.html W.K. Kellogg Foundation. (1998, January). W.K. Kellogg Foundation evaluation handbook. Retrieved October 17, 2004, from http://www.wkkf.org/Programming/ResourceOverview.aspx? CID=281&ID;=770 PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 221 4 GATHER CREDIBLE EVIDENCE TOOL 4.4: COMPLETE AN EVALUATION PLAN BASED ON PROGRAM DESCRIPTION AND EVALUATION DESIGN INTRODUCTION You learned how to describe your program through well-written goals and SMART objectives (Tool 2.2) and developed logic models (Tool 2.4) — all with input from program staff and stakeholders. You also learned how to select program evaluation questions (Tool 3.3) and identify corresponding indicators for each question (Tool 4.1). Further, you learned how to select appropriate and feasible data collection methods suitable for each source of data, as well as data collection procedures (Tools 4.2 and 4.3). ENGAGESTAKEHOLDERSTHROUGHOUTEVALUATION UNDERSTANDINGOFPROGRAMFOCUSANDPRIORITYAREASPROGRAMGOALS SMARTOUTCOMEOBJECTIVESSMARTPROCESSOBJECTIVESEVALUATIONPURPOSE,USERS,USESEVALUATIONOUTCOMESEVALUATIONDESIGNPROCESSANDOUTCOMEINDICATORSDATACOLLECTIONSOURCESANDMETHODSDATACOLLECTIONPROCEDURESEVALUATONPLANLOGICMODELINPUTSACTIVITIESOUTPUTSPROCESSSHORT-TERMOUTCOMESINTERMEDIATEOUTCOMESLONG-TERMOUTCOMESOUTCOME PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 222 GATHER CREDIBLE EVIDENCE 4 In Tool 4.4, you will learn how to organize all this information into a comprehensive evaluation plan. The flowchart on the previous page provides a description of where an evaluation plan fits in with your past evaluation planning activities. LEARNING OBJECTIVE Upon completion of this tool, you will be able to: • Develop an evaluation plan to guide the planning and implementation of a program evaluation. WHAT IS THE PURPOSE OF AN EVALUATION PLAN? An evaluation plan gives you the opportunity to verify that you have addressed all the elements required to implement and complete a successful evaluation. An evaluation plan also helps program staff to understand stakeholder perspectives on the evaluation process and to clarify stakeholder expectations of program results. WHAT ARE THE COMPONENTS OF AN EVALUATION PLAN? An evaluation plan has two main components: (1) a narrative component and (2) a matrix, both of which enable you to operationalize the evaluation. The narrative component of an evaluation plan includes the following: • Sexually Transmitted Disease (STD) program component or activity to be evaluated • Rationale for evaluating this STD program component or activity • Purpose of the evaluation • Goal(s) and objectives that relate to the program component/activity to be evaluated • Logic model(s) • Individuals and their roles in the evaluation team • Users and uses of evaluation findings • Approach to disseminating evaluation findings to appropriate users • Timeline for completing the evaluation • Evaluation budget PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 223 4 GATHER CREDIBLE EVIDENCE The evaluation matrix of an evaluation plan includes the following: • Evaluation questions • Indicators • Data sources and data collection methods • Data collection procedures – Person(s) responsible – Schedule • Data analysis (see Tool 5.1 for more details) This evaluation plan reflects most of the steps in CDC’s Framework for Program Evaluation in Public Health1, and thus serves as a valuable resource for planning and implementing your evaluation. As you noticed, most of the evaluation plan components have already been addressed and built in the previous tools (1.1, 2.2, 2.4, 3.1, 3.2, 3.3, 4.1, 4.2, 4.3). Tool 4.4 compiles the work done up to this point. HOW DO YOU CONSTRUCT AN EVALUATION PLAN? Following are some suggested steps to help you construct your evaluation plan. 1. Gather all relevant materials. It is likely that you have had several meetings with program staff and stakeholders to discuss elements of the evaluation, taken minutes of these meetings, and developed several documents essential to constructing your evaluation plan. Prior to developing the evaluation plan, make sure you have: • reviewed your Comprehensive STD Prevention Systems (CSPS) grant application • gathered all the documents that describe which STD program component or activity will be evaluated and why • determined the goals and objectives pertaining to the program component or activity to be evaluated • determined who will be involved in the evaluation, who will use the evaluation findings and how they will use them • the final version of the logic model(s) • the list of evaluation questions and corresponding indictors, data sources, data collection methods, ideas on how you may want to analyze the data • available resources and staffing for the evaluation 1 Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999; 48 (no. RR–11) PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 224 GATHER CREDIBLE EVIDENCE 4 2. Organize and sort the relevant materials. Divide the materials into two groups. The first group will include all the information you will need to complete the narrative component of the evaluation plan. The second will include all the information you will need to complete the evaluation matrix. Next, sort the materials in each group to match the sequence of items in the narrative and matrix components. 3. Complete the narrative component of the evaluation plan. The guidelines presented in Table 1 will help you to complete the narrative evaluation plan. 4. Complete the matrix component of the evaluation plan. Table 2 provides you with guidance for completing each column in the evaluation plan matrix. The plan that is presented does not include information on data analysis. Specifics on this can be found in Tool 5.1 (Analyze the Evaluation Data). Table 1: Guidance for completing the evaluation plan narrative OUTCOMEGUIDELINES ListtheSTDprogramactivitytobeevaluated. Liststakeholders(agency)involvedintheevaluation. ListtherationaleforevaluatingthisSTDprogramactivity. Listthepurposeoftheevaluation. Based on your CSPS grant application and per your discussion with the stakeholders, indicate the program activity that you plan to evaluate as well as the purpose and rationale for evaluation of the program activity (refer to Tools 1.1, 2.1, and 2.2). For example: Program activity to be evaluated: Chlamydia (Ct) screening training workshops. Stakeholders involved in the evaluation: Program staff, external evaluator, County STD clinic directors and staff, State STD program staff, professional development organizers and implementers. Rationale for selecting the program activity: Based on the findings from the needs assessment, the program staff and stakeholders decided that clinician training on Ct screening would increase appropriate diagnosis and treatment of Ct infections. It was agreed that screening, treatment, and diagnosis protocols should be consistently implemented, and that since professional development (PD) is the foundation for the implementation of a quality screening initiative, it is important to assess its effectiveness. Purpose of the evaluation: Evaluate the extent to which PD workshops led to clinical staffs’ increased knowledge and skills for screening, counseling, and treatment protocols. continued PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 225 4 GATHER CREDIBLE EVIDENCE Table 1: Guidance for completing the evaluation plan narrative (continued) List the goal(s) and objectives that relate to the program component/ activity to be evaluated. OUTCOME GUIDELINES Your CSPS grant application includes your program goals and objectives. In this section, list only those goals and objectives that are related with the program component or activity you want to evaluate. Make sure that your objectives are “SMART” (i.e. Specific, Measurable, Achievable, Relevant, and Time-bound). Also make sure that they are articulated as process or outcome (short-term, intermediate, or long-term) objectives (refer to Tool 2.2). Example: You may be implementing a Ct screening initiative throughout your project area with one component being PD workshops on Ct screening for clinical staff. Since you have decided to evaluate the effectiveness of the PD workshops in increasing participants’ knowledge and skills, and not other parts of the initiative, include only the objectives related to the PD workshops. Attach appropriate logic models. Make sure you include a logic model of the program component or activity you want to evaluate (refer to Tool 2.4). Keep in mind that the logic model is a work in progress. Highlight the specific components of the logic model you intend to evaluate. List individuals and their roles in the evaluation team. Staff your evaluation with individuals who are well-qualified and available for the job (refer to Tool 3.2). Identifying all members (including stakeholders) and their roles will allow you to proceed smoothly. This will also allow you to assess the need for requesting technical assistance from DSTDP and/or hiring external evaluation consultants. List the users and uses of the evaluation findings. Identify all stakeholders (i.e., implementers, decision makers, participants, and partners) interested in the evaluation findings and how they would intend to use them (refer to Tools 1.1 and 3.1). List the approach to disseminating the evaluation findings to appropriate users. Describe how the evaluation findings will be presented to (method of delivery such as full report, short report, manuscript, presentation, etc.) and shared (mailed report, discussion, etc.) with appropriate users. Attach the timeline for completing the evaluation. The timeline should reflect the entire evaluation timeframe; from discussion of evaluation questions with stakeholders to dissemination of results (refer to Tool 4.3). Attach the evaluation budget. The evaluation budget should include the direct and indirect costs of your evaluation. If you are working with an external contractor, the cost for the contractor can be broken out by consultant’s salary, fringe benefits, and non- personnel costs (refer to Tool 4.3). 5. Update the evaluation plan, as needed. Changes may need to be made in parts of your evaluation plan due to changes in the program component or activity you are evaluating, field conditions of the evaluation or other practical considerations. Update your logic model, staff information, the timeline, or other aspects of the plan, as needed. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 226 Table 2: Evaluation Plan Matrix6 DATA COLLECTION PROCEDURES DATA ANALYSIS1 EVALUATIONQUESTION INDICATORS DATASOURCE(S) DATA COLLECTIONMETHODS PERSONRESPONSIBLE SCHEDULE PROCEDURE TIMELINE PERSONRESPONSIBLE • Develop andprioritize eachevaluation questionwith input fromstakeholders. • Link eachevaluation questionwith the goals andobjectives of yourprogram and thepurpose of yourevaluation. • Verify that thequestions reflectthe key elementsof your programlogic model thatyour evaluationwill address, andthat they can beaddressed usingthe resourcesavailable at hand, i.e. budget, staff, and staff time. • Verify that thefindings generatedfrom the questionswill be useful • For eachevaluationquestion, createan indicator toreflectachievement(e.g. number ofstaff trained, percent of healthclinicsimplementing apolicy, etc.). • If needed, youmay use morethan oneindicator perquestion. • For eachevaluationquestion, indicatewhat source(s) has theinformation toanswer thequestion (i.e., individuals, observations, ordocuments). • A single datasource canprovideinformation onmore than oneevaluationquestion. • For each datasource, identifyand list the bestmethod to gatherdata (e.g. , use oftelephone ratherthan mail surveyfor a certaintarget group). • More than onedata collectionmethod may beused to gatherinformation fromone datasource.) . • For each datacollectionmethod, identifyroles andresponsibilities ofALL individualsinvolved. This willallow you toestimate theworkload of eachindividual, maintain asuitable staffinglevel, anddevelop areasonabletimeline for datacollection. • For each datacollectionmethod, fill inthe datacollectionschedule (e.g., baseline surveydata will becollected bymm/dd/yy; follow-up surveydata will becollected bymm/dd/yy) . 6 This plan needs to incorporate the data analysis component which will be discussed in Tool 5.1. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 4 GATHER CREDIBLE EVIDENCE CONCLUSION AND NEXT STEPS A completed evaluation plan will address most steps from CDC’s program evaluation framework. The plan will include narrative and matrix components to provide a comprehensive picture of the rationale of, and approach to, the evaluation. This Tool has shown you how to consolidate the evaluation information into one plan. Now that you have an evaluation plan, it is time for you to implement your evaluation. In Tool 5.1 (Analyze the Evaluation Data), you will learn how to manage, analyze, and synthesize your evaluation data. SUMMARY CHECKLIST: Analyze the Evaluation Data Gatherrelevantdocumentsfromthevariousmeetingsyouhavehadwithprogramstaffandstakeholders. •CSPSgrantapplication•Meetingminutes•Documentssuchaslogicmodels,evaluationbudgetandevaluationtimeline Sortandorganizerelevantdocumentsforthenarrativeandmatrixcomponentsoftheevaluationplan,toreflecttheflowofthesecomponentsintheevaluationplan. Completethenarrativecomponentoftheevaluationplan.Include: •STDprogramoractivitytobeevaluated•RationaleforevaluatingthisSTDprogramoractivity•Purposeoftheevaluation•Programgoal(s)andobjectivestobeaddressedthroughtheevaluation•Logicmodel(s) •Individualsandtheirrolesintheevaluationteam•Usersandusesofevaluationfindings•Approachtodisseminatingevaluationfindingstoappropriateusers•Timelineforcompletingtheevaluation•Evaluationbudget Completethematrixcomponentoftheevaluationplan.Include: •Evaluationquestions•Indicators•Datasources•Datacollectionprocedures–datacollectionmethods–person(s)responsible–schedule•Dataanalysis Updatetheevaluationplan(e.g.logicmodel,staff,timeline,etc.)ifneeded. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 228 GATHER CREDIBLE EVIDENCE 4 ACRONYMS USED IN THIS TOOL CSPS – Comprehensive STD Prevention Systems Ct – Chlamydia trachomatis JDC – Juvenile Detention Centers STD – Sexually Transmitted Disease PD – Professional Development PDSB – Program Development and Support Branch SMART – Specific, Measurable, Achievable, Relevant, and Time-bound KEY TERMS Evaluation plan: A document that includes what an evaluation consists of (i.e., purpose/uses/users of the evaluation, program goals and objectives related with the evaluation, logic model, evaluations questions and design, data collection sources and methods, and dissemination plan) and the procedures that will help guide the implementation of evaluation activities to be undertaken by your program. Goal: A broad statement related to the purpose of your program that states what your program will accomplish (the desired result). Indicator: A specific, observable, and measurable accomplishment or change that shows whether progress has been made toward achieving a specific program output or outcome. Logic model: A picture of how a program/component/activity is supposed to work. Objectives: Measurable statements that describe the manner in which your program goals will be achieved. Post-only design: A non-experimental design where measures (data collection) are taken from the target population(s) after the activity/intervention. Since this is a non-experimental design, it does not involve comparison/control groups. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 229 4 GATHER CREDIBLE EVIDENCE Pre/post design: A non-experimental design where measures (data collection) are taken from the target population(s) before and after the activity/intervention. Stakeholders: Individuals or organizations directly or indirectly affected by your STD program and/or the evaluation results (e.g., STD program staff, family planning staff, representatives of target populations). Survey: A method of collecting information that can be self- administered, administered over a telephone, administered using a computer or administered face-to-face. Surveys generally include close-ended questions that are asked to individuals in a specific order and provide multiple choice or discrete responses (e.g., “Have you been tested for syphilis in the last 6 months?”). CASE SCENARIO The following is an example of an evaluation plan for assessing a professional development (PD) component of a larger Chlamydia (Ct) screening initiative that the STD Program plans to conduct in juvenile detention centers (JDCs) in County Z. The example provides both the evaluation plan narrative and matrix based on all the steps detailed in this evaluation tool. This evaluation plan is not exhaustive and is provided only to illustrate a sample evaluation plan. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 230 GATHER CREDIBLE EVIDENCE Evaluation Plan Narrative 4 STEPAPPLICATION ListtheSTDprogramactivitytobeevaluated. Liststakeholders(agency)involvedintheevaluation. ListtherationalefortheSTDprogramactivitytobeevaluated. Listthepurposeoftheevaluation. Listthegoal(s)andobjectivesthatrelatetotheprogramcomponent/ activitytobeevaluated. Attachappropriatelogicmodels. • Program activity to be evaluated: PD workshops in Ct screening offered to JDC medical providers in County Z. • Stakeholders involved in the evaluation: Program staff, County and State agency assisting with PD workshops, STD clinical staff participating in the workshop. • Rationale for selecting the program activity: PD activities are the foundation for providing the necessary knowledge and skills to implement appropriate Ct screening, counseling, and treatment protocols. High-quality PD workshops should lead to consistency in implementation of the recommended screening, counseling, and treatment protocols • Purpose of the evaluation: To evaluate the implementation and effectiveness of PD workshops. GOAL: Reduce STD rates among adolescent females in JDCs in County Z. PROCESS OBJECTIVE: • By (month/year), contracted organization (X) will conduct (#) professional development events with (#) medical providers in JDCs in County Z on appropriate Ct screening, counseling, and treatment for female adolescents. SHORT-TERM OUTCOME OBJECTIVES: • By (month/year), the medical staff from County X JDCs who participated in the training will demonstrate, in a pre-and post-training questionnaire, an increase in their knowledge of the Ct screening protocols from X% to Y%. • By (month/year), the medical staff from County X JDCs who participated in the training will demonstrate, in a pre-and post-training questionnaire, an increase in their knowledge of the Ct counseling protocols from X% to Y%. • By (month/year), the medical staff from County X JDCs who participated in the training will demonstrate, in a pre-and post-training questionnaire, an increase in their knowledge of the Ct treatment protocols from X% to Y%. • By (month/year), the medical staff from County Z JDCs who participated in the training will demonstrate, in a performance exercise, an increase in their skills in conducting Ct screening from X% to Y%, as recorded in a skills performance sheet. • By (month/year), the medical staff from County Z JDCs who participated in the training will demonstrate, in a performance exercise, an increase in their skills in conducting Ct counseling from X% to Y%, as recorded in a skills performance sheet. • By (month/year), the medical staff from County Z JDCs who participated in the training will demonstrate, in a performance exercise, an increase in their skills in providing Ct treatment from X% to Y%, as recorded in a skills performance sheet. See Appendix A. continued PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 231 4 GATHER CREDIBLE EVIDENCE Evaluation Plan Narrative (continued) STEP List individuals and roles on the evaluation team. APPLICATION • Project director: Oversees all evaluation activities. • Project manager: Develops timeline; hires and trains; supervises data collection and handling; reviews all components of evaluation and final report; disseminates findings. • College intern: Develops evaluation instruments in consultation with project manager and trainers; trains program staff in data entry; analyzes data. • Contractor (X) training staff: Develops, staffs, and implements training for JDC medical staff; administers pre-and post-training questionnaire and performance exercise; completes skills performance sheet for each training participant; analyzes data; drafts final report. • STD program administrative assistant: Maintains confidential files of completed evaluation tools; conducts data entry of evaluation data. List the users and uses of the evaluation findings. • Implementers (JDC medical staff, JDC administrators, STD program trainers): Determine the effectiveness of the workshop in changing knowledge and skills of participants. Use evaluation findings to improve future workshops. • Decision makers (STD program director and manager, Correction System Director): Ensure the quality of the larger Ct screening initiative in JDCs; demonstrate the value of PD to stakeholders. • Partners (JDCs and community representatives): Encourage in-service PD for JDC staff in the future; ensure that detainees are receiving quality care. List the approach to disseminating the evaluation findings to appropriate users. • Funders: Written report. • Other STD programs: Presentation and Written Report for Program Development and Support Branch (PDSB) Thursday Report, presentation at National STD Prevention Conference • JDC staff and detainees: Report and presentation • Advocacy group: Use of media (radio, newspaper) • Scientific community: Manuscript publication. Attach the timeline for completing the evaluation. See Appendix B. Attach the evaluation budget. See Appendix C. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 232 (continued) Evaluation Plan Matrix DATA COLLECTION PROCEDURES DATA ANALYSIS1 EVALUATIONQUESTION INDICATORS DATASOURCE(S) DATA COLLECTIONMETHODS PERSONRESPONSIBLE SCHEDULE PROCEDURE TIMELINE PERSONRESPONSIBLE 1.How many PDworkshops wereoffered by thecontractedorganization? 2.How many medicalstaff attended eachworkshop? 3.Was the workshopcontent deliveredas planned? 4. Were participantssatisfied with thetraining theyreceived? Process Evaluation Objective: Number of PDworkshopsimplemented by thecontractedorganization. Number of medicalstaff who attendedeach workshop. Number of contenttopics addressedduring training. Percent ofparticipantssatisfied with thetraining. Documents(workshop planninglog to verify if (#) workshops weredelivered in atimely manner) Documents(workshopattendance roster) Documents(trainingimplementation logmaintained byworkshop trainers) Individuals (trainingparticipants) By (month/year) , contracted organization (X) will conduct (# ) professional development events with (# medical providers in juvenile detentioncenters (JDCs) in County Z on appropriate Chlamydia (Ct) screening, counseling, and treatment with female adolescents. Document Review(program managerwill completeworkshop planninglog) Document Review(trainers willprovide workshopattendance rosterand all completedimplementationlogs to the programmanager) Observation(college intern willcomplete achecklist to recordhow well trainersfollow the script) Survey (self- administered post- training survey) Program managerand trainers. All evaluationinstruments will becompleted at thetime of individualtrainings(mm/dd/yy) . Frequency (count) of number ofworkshopsconducted. Count of workshopattendees. Code themes fromprogramimplementation logand observationchecklist for eachsession. Computepercentages. All data will beanalyzed bymm/dd/yy. Tables and graphicswill be developed bymm/dd/yy. All data will beanalyzed bymm/dd/yy. Tables and graphicswill be developedby mm/dd/yy. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS Evaluation Plan Matrix (continued) G A DATA ANALYSIS PERSONRESPONSIBLE Program managerwill work closelywith the collegeintern. Program managerwill work closelywith the collegeintern. TIMELINE All data will beanalyzed bymm/dd/yy. Tables and graphicswill be developedby mm/dd/yy. increase in their skills in conducting Ctfrom X% to Y% , as recorded in skills• By (month/year) , the medical staff from CountyZ JDCs who participated in the training willdemonstrate, in a performance exercise, ancounseling from X% to Y% as recorded in a• By (month/year) , the medical staff from CountyZ JDCs who participated in the training willdemonstrate, in a performance exercise, anincrease in their skills in providing Ct treatmentAll data will beanalyzed bymm/dd/yy. Tables and graphicswill be developedby mm/dd/yy. PROCEDURE Compute changedifference in preand post-trainingscore for eachparticipant. Using post- workshop dataidentify and selectthose with correctresponses for eachprotocol. skills performance sheet. performance sheet. Computepercentage/ proportion ofparticipants whodemonstrate low, medium, and highskills (will use atercile split whereinthe lowest one- third is categorizedas low skill, middleone-third asmedium skill, andtop one-third ashigh skill level) . DATA COLLECTION PROCEDURES SCHEDULE Before (mm/dd/yy) and immediatelyafter PD workshops(mm/dd/yy). from Countyknowledgefrom CountyBy end of trainings(mm/dd/yy) . PERSONRESPONSIBLE Trainer will overseeparticipants’completion ofsurveys. • By (month/year) , the medical staff an increase in their • By (month/year) , the medical staff increase in their skills in conducting CtX JDCs who participated in the training willdemonstrate, in a pre-and post-trainingof the Ct treatment protocols from X% to Y%. Z JDCs who participated in the training willdemonstrate, in a performance exercise, anscreening from X% to Y% , as recorded in aTrainers. DATA COLLECTIONMETHODS Survey (self- administeredquestionnaire to becompleted byparticipants beforeand right after thetraining) questionnaire, skills performance sheet. Observation(trainers will scoreparticipants’ skillsduring exercise andrecord them onskills performancesheets) DATASOURCE(S) Individuals (trainingparticipants) OBJECTIVES: from X% to Y% . Observations (skillsused duringtrainingperformanceexercise) INDICATORS Percent ofparticipants whocan correctlydescribe Ctprotocols forscreening, counseling, andtreatmentof the Ct counseling protocols • By (month/year) , the medical staff from CountyX JDCs who participated in the training willdemonstrate, in a pre-and post-trainingquestionnaire, an increase in their knowledgeof the Ct screening protocols from X% to Y%. • By (month/year) , the medical staff from CountyX JDCs who participated in the training willdemonstrate, in a pre-and post-trainingquestionnaire, an increase in their knowledgePercent ofparticipants whocan demonstrateappropriate skills inconducting Ctscreening, counseling, andtreatment. EVALUATIONQUESTION 1.Did medical staffwho participated inthe trainingincrease theirknowledge ofrelated protocols? OutcomeSHORT-TERM OUTCOME EVALUATION 2. Did medical staffwho participated inthe trainingdemonstrate anincrease in Ctscreening, counseling, andtreatment skills? PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS GATHER CREDIBLE EVIDENCE 4 APPENDIX A1 Evaluation Components Highlighted in the Program Logic Model Logic Model of Chlamydia (Ct) Screening Program for Adolescent Females in County Z Juvenile Detention Centers (JDCs) INPUTS ACTIVITIESOUTPUTSSHORT-TERMOUTCOMES INTERMEDIATE OUTCOMES LONG-TERM OUTCOMES Funds CDC/DSTDP, other federal sources, state sources, local sources, private sources JDC Staff JDC administrators, health care provider staff, administrative staff Technical Direction, Assistance, and Collaboration Local/state health department and laboratories Contracted partners performing staff training Advisory committee Materials Protocols/guide-lines for screening Prevention education material Provideprofessionaldevelopmentinscreeningandtreatment. ProvideJDCsupportfordevelopingandimplementingscreeningprogram. Conductscreening, counseling,andtreatmentforsexuallyactivefemaleadolescentadmittees. JDCclinicalstaffparticipatedinprofessionaldevelopment(PD) eventsonscreeningcounseling,andtreatment. JDCadministrationauthorizedtimeandstafffordevelopmentandimplementationofscreeningprogram. FemaleadolescentscounseledandscreenedJDCProvidersImprovedknowledgeandskillsforscreening, counselingtreatmentservices. JDCsIncreasednumberofprogramsinJDCsthatprovidescreening, counselingandtreatmentservices. AdolescentFemalesIncreasedawarenessofandotherSTDtransmissionandprevention. Increasedintentiontousecondoms. JDCProvidersIncreasedclinicians’adherencetoclinicalpracticeguidelines(screeningandtreatment). AdolescentFemalesDecreasednumberofpartners. Increasedhealthseekingbehavior. JDCs Reduction of prevalence among adolescent females in JDCs. 1 Highlighted boxes indicate program components to be evaluated. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 235 4 GATHER CREDIBLE EVIDENCE APPENDIX B Projected Timeline for Evaluation Activities EVALUATION ACTIVITIES MONTHS 1 2 3 4 5 6 Plan evaluation with program staff and stakeholders. Develop evaluation data collection instruments, and train data collectors. Monitor attendance and participation levels of workshop participants. Train clinical staff and collect pre- and post-training survey from them. Collect post-workshop performance exercise data from participants. Analyze evaluation data Report findings. Revise program, if needed. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 236 4 GATHER CREDIBLE EVIDENCE APPENDIX C Six-Month Evaluation Budget EVALUATION ACTIVITY College intern stipend (instrument development, data analysis, staff training) COST $ 300.00 Contractor $2,000.00 Communications (postage, telephone calls, etc.) $ 30.00 Printing and Duplication $ 30.00 Supplies and Equipment $ 70.00 Indirect Costs $100.00 TOTAL $2,530.00 PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 237 4 GATHER CREDIBLE EVIDENCE REFERENCES Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adolescent and School Health. (2004). Evaluation steps tools. Unpublished document. Stecher, B. M., & Davis, W. A. (1987). How to focus an evaluation. Newbury Park, CA: Sage. PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 238