Engage Stakeholders Engage Stakeholders T he first step in CDC’s framework for program evaluation is to identify and involve the stakeholders of your program in planning and implementing evaluation activities. Your stakeholders — the individuals and organizations that have a vested interest in your program and may be affected by the evaluation results — are essential partners. By engaging a range of stakeholders with different perspectives on your program, you can build both internal and external buy­in and support for the evaluation process. Stakeholder involvement can also make the evaluation process more objective, enhance communication among key parties, and ensure that your data collection is thorough and complete. Tool 1.1 will help you identify, involve, and retain stakeholders for your Sexually Transmitted Disease (STD) program evaluation process. TOOL 1.1: DETERMINE HOW AND TO WHAT EXTENT TO INVOLVE STAKEHOLDERS IN PROGRAM EVALUATION LEARNING OBJECTIVES Upon completion of this tool, you will be able to: 1. Identify key stakeholders who can assist in the program evaluation process. 2. Implement strategies for ensuring broad participation of stakeholders in the evaluation. 3. Implement strategies for retaining broad participation of stakeholders in the evaluation. WHO ARE STAKEHOLDERS? Stakeholders are individuals and organizations that have an interest in or are affected by your evaluation and/or its results. WHY IS STAKEHOLDER INVOLVEMENT IMPORTANT? When you involve stakeholders from the beginning of the evaluation process, you are more likely to: • Reduce stakeholders’ distrust and fear of evaluation. • Increase stakeholders’ awareness of and commitment to the evaluation process. • Increase the chances that stakeholders will support your evaluation efforts, advocate for your program, and adhere to subsequent recommendations. • Increase the chances that the evaluation findings will be used. • Increase the credibility of your evaluation findings. If you do not engage stakeholders, your evaluation runs the risk of missing important elements of the program. In that case, evaluation findings might be ignored, criticized, or resisted because your evaluation did not consider stakeholder concerns or priorities. Stakeholders can make many important contributions to the evaluation process, including: • Providing a reality check on the appropriateness and feasibility of your evaluation questions. • Offering insight on the target populations that may affect program implementation or evaluation. • Providing information on appropriate methods to access target populations. • Reviewing and commenting on the evaluation report. • Helping to disseminate and report the evaluation results. • Providing ongoing feedback and recommendations for improving your program or evaluation activities. HOW CAN YOU IDENTIFY STAKEHOLDERS FOR YOUR PROGRAM EVALUATION? Traditionally, program managers and evaluators alone have handled program evaluation in public health activities. However, a more participatory approach has been taking place lately. For your evaluation to be relevant and successful, other stakeholders need to be involved in the planning and implementation stages, and in the interpretation of evaluation findings and recommendations. By involving stakeholders in the evaluation process, you have the opportunity to understand issues that are important to them and to address their suspicions and fears about the evaluation. We recommend including four types of stakeholders in the evaluation process. They are listed and described below. • Implementers • Decision Makers • Participants • Partners Table 1 lists examples of the various parties in each of the four groups. Every STD program is different, so adapt this list to your own programmatic and evaluation needs. Table 1: Stakeholders to include TYPES OF STAKEHOLDERS DEFINITION EXAMPLES Implementers Those directly involved in the operations of the STD program • STD director • STD program manager • Staff (field operation manager, front­line supervisor, Disease Intervention Specialist (DIS), service providers, health educators, counselors, clinicians, laboratory technicians, clerical staff, among others) • Volunteers and consultants Decision Makers Those in a position to do or decide something about the STD program • STD director • STD program manager • Health department director Participants Those being served or affected by the STD program • Clients/patients • Community members • Community­based organizations (CBOs) • Health service providers Partners Those who actively supports and/or has invested in your program or in the population your program serves. • Funders (federal, state, local) • Academic organizations • Coalition partners • Faith­based community • Professional organizations • Advocacy groups, such as your Human Immunodeficiency Virus (HIV) Community Planning Group (CPG) • Government officials and elected representatives • Representatives of correctional systems (adult, juvenile) • Representatives of the school system Identifying stakeholders is critical and can be complex. Here are some tips. 1. Begin with the categories of stakeholders listed above (i.e., implementers, decision makers, participants, and partners). Think about what different stakeholders want from the evaluation, and how they plan to use the findings. 2. Be sure to consider as your evaluation stakeholders, individuals/organizations that deal with STD prevention, control, and policy; as well as leaders, advocates, and clients from the community. Take special care to promote the inclusion of underserved and/or under­represented persons. Address confidential and sensitive issues when engaging representatives of the program’s target population(s). 3. The number of stakeholders to involve in your effort will depend on the purpose and timeline of your evaluation, and the size and resources of your program. There is no magic formula on how many stakeholders should be included. However, keep in mind that larger groups (more than 9) may take longer to reach decisions than smaller groups, and may make the process more complicated. 4. Be attentive to the level of each stakeholder’s involvement in the evaluation process. This will allow you to identify stakeholders who are not participating in the evaluation. Scheduling a one­on­one meeting with stakeholders who are not participating or doing so at a minimal level will allow you to understand their concerns. As appropriate, discuss these concerns with them individually or with the larger stakeholder group. 5. Look “outside of the box” for additional stakeholders. Even if you are familiar with your program and think you already know who should be involved in your evaluation process, it never hurts to look for previously unidentified stakeholders who can provide valuable insight and feedback. These can include opponents to your program as well as parties that may carry out similar work. 6. If you bring in new stakeholders in the middle of an evaluation, be sure to discuss their inclusion with the existing stakeholders. Similarly, inform your new stakeholders of the current stakeholder representation on your project. The new stakeholders should be debriefed on all evaluation activities prior to their attending a meeting. The exercise on page 15 “Identifying and Determining of Involvement of Stakeholders for Program Evaluation,” provides more guidance on this process. HOW DO YOU INVOLVE STAKEHOLDERS IN THE PROGRAM EVALUATION PROCESS? Here are some suggested steps to help engage stakeholders in the process of planning and implementing your evaluation. 1. Invite the stakeholders you have identified to a meeting, or series of meetings, depending on how much time they have available to spend with you [see Appendix for sample agenda]. At this meeting you want to brief stakeholders on your program as well as the program component or activity you want to evaluate. One major objective of this meeting should be to obtain a clear understanding of stakeholder interests, perceptions, and concerns related to your program and evaluation. The following questions can help you do that. • What do stakeholders perceive as the purpose of your program? What do they think about your program? • What concerns, if any, do they have about the program? • What have they heard about the proposed program evaluation? • What areas do stakeholders think are important to address first in the evaluation? • What do they hope to learn from the evaluation? • What concerns, if any, do they have with the program evaluation? • How available are they to participate in the evaluation process? • What are the political implications of specific stakeholders’ involvement? • What are the program expectations of stakeholders’ involvement in the evaluation? • How can you meet stakeholders’ evaluation and communication needs? A second objective of the meeting is to have all stakeholders clearly identify and agree to their roles and responsibilities related to the evaluation before it begins. In addition, ask them how they want to be involved in the evaluation process given their individual strengths and resources. For example, some stakeholders can make valuable contributions to developing evaluation questions and identifying priority issues to be evaluated, while others can identify data sources or assist in disseminating and reporting results. A final objective for your initial meeting is to assure stakeholders that you will strive to maintain open communications and to address their concerns. To maximize effective communication, let the group know when they will hear from you, including when you will seek their input during the evaluation process. During this meeting, emphasize to the stakeholder group that while their input and recommendations are very valuable, it may not be feasible to implement all of their recommendations. Emphasize that these decisions are driven by the availability of program resources (i.e. program staff, time, and budget priorities). Keep in mind that some stakeholders may not be able to attend all the meetings. In such situations, it is important to request input from those interested through alternative means (e.g., email). 2. Involve stakeholders in key activities throughout the planning and implementation of the evaluation. Some of these activities can include: • Identifying and prioritizing program activities that need to be evaluated. • Overcoming resistance to evaluation and identifying benefits. • Developing evaluation questions that are grounded in the perceptions, experiences, and interests of stakeholders. • Selecting appropriate and acceptable evaluation methods. • Reviewing evaluation findings and making program recommendations. • Disseminating and using evaluation findings for program improvement. How frequently you meet with stakeholders depends on the timeline for the evaluation. Plan regular meetings to discuss the purpose and progress of the evaluation, and keep the stakeholders involved and informed throughout the evaluation process. HOW DO YOU RETAIN STAKEHOLDERS IN THE PROGRAM EVALUATION PROCESS? After all the time and energy you have devoted to identifying and engaging stakeholders, you want to keep them from losing interest and dropping out of the evaluation process. Following are some tips for maintaining stakeholders’ interest and commitment to the program and the evaluation. • Maintain open, honest, and regular communication with the stakeholders by keeping them up to date on issues pertaining to the evaluation and relevant considerations. • Incorporate stakeholders’ opinions and insights into the evaluation process. • Identify during the initial meetings possible conflicts between stakeholders (e.g., competition, rivalry, and existing power structures) and facilitate productive working arrangements. • Identify stakeholders’ barriers to participation, and when possible, address them. • Identify stakeholder expectations from the beginning and take them into account when planning and implementing an evaluation. • Plan before meeting or requesting stakeholder assistance so that everyone’s time can be spent wisely. • Express gratitude and positive reinforcement verbally, and if possible, in more tangible ways (e.g., letter of appreciation, public recognition). • Follow through on what you agree; avoid making promises you cannot keep. • Request volunteers for specific sub­tasks, if needed. HOW DO YOU IDENTIFY, INVOLVE, AND RETAIN STAKEHOLDERS? A CASE SCENARIO The STD program of State X planned and implemented a Chlamydia screening program for adolescent females in Juvenile Detention Centers (JDC). The program staff wanted to conduct an evaluation of this initiative. They recognized that a critical first step in their evaluation efforts was to identify and involve stakeholders in these efforts. They also understood they needed to develop strategies to retain the stakeholders throughout the evaluation. Summarized below are the activities that the program staff conducted to identify, involve, and retain stakeholders in their evaluation. 1. Identify Stakeholders The program staff met to create a list of potential stakeholders. They used the categories of program implementers, decision makers, participants, and partners and related questions to identify a list of stakeholders (see Table 2). Table 2: Stakeholders identified by STD program staff TYPES OF STAKEHOLDERS QUESTIONS USED TO IDENTIFY STAKEHOLDERS STAKEHOLDERS Implementers Who is directly involved in the operations of the Chlamydia screening initiative? • JDC administrators • JDC healthcare provider staff • JDC administrative and detention staff • STD program staff Decision Makers Who is in a position to do or decide something about this initiative? • STD program director • STD program manager • State correctional systems director • Local politicians Participants Who is being served or affected by this initiative? • Adolescent females aged 16­18 years who are admitted to JDCs Partners Who actively support and/or have invested in this initiative? • Medical providers in the community who serve high­risk adolescents • Organizations that train JDC providers in Chlamydia screening/treatment protocols • Health­related community­based organizations that serve high risk adolescent females 2. Involve Stakeholders Next, the STD program communicated with the proposed stakeholders. They invited them to attend a meeting. The meeting first identified the stakeholders’ interests, perceptions, and concerns related to the Chlamydia screening initiative and the evaluation. The STD program reviewed the proposed stages for designing and implementing the evaluation. They asked stakeholders to indicate their prospective involvement in the evaluation process. The information from this meeting is summarized in Table 3. Table 3: Relevant information on stakeholders identified in meeting X IMPLEMENTER STAKEHOLDERS INTERESTS/PERCEPTIONS/ CONCERNS INVOLVEMENT IN THE EVALUATION JDC administrators Maintaining the security and control of JDC detainees; maintaining the health of detainees; cooperating with external state agency initiatives; securing results from the initiative that will reflect positively on JDCs. Input on evaluation design, and interpretation and use of results. JDC healthcare providers Maintaining the health of detainees; securing results from the initiative that will reflect positively on their work; designing an evaluation that will not overly burden medical work of the initiative; being sensitive to juvenile’s rights to be screened. Input on evaluation design, and interpretation and use of results. Data collection on some aspects of the evaluation. JDC administrative staff Being overburdened with work responsibilities from additional initiatives that the JDCs decide to implement. Data collection on some aspects of the evaluation. Receive evaluation results. STD program staff Effective delivery of intervention. Ability to fit this in with their other responsibilities. Input on evaluation design, data collection, and interpretation of findings. DECISION MAKER STAKEHOLDERS INTERESTS/PERCEPTIONS/ CONCERNS INVOLVEMENT IN THE EVALUATION STD program director and STD program manager Reducing STDs among adolescents; gaining the cooperation of JDCs in the initiative; involving family members in the decision making process; implementing a quality screening initiative according to design; developing a strong monitoring and evaluation component. Input on evaluation design, methods, and interpretation and use of results. State correctional systems director Maintaining the security and control of detainees in all correctional facilities; maintaining a positive public opinion of the treatment of detainees; maintaining positive health outcomes of detainees. Input on evaluation design, and interpretation and use of results. (continued) 22 Table 3: Relevant information on stakeholders identified in meeting X (cont.) PARTICIPANT STAKEHOLDERS INTERESTS/PERCEPTIONS/ CONCERNS INVOLVEMENT IN THE EVALUATION Female detainees in JDCs Maintaining health and well­being; gaining access to services that are delivered in a culturally appropriate manner; providing input on service satisfaction; maintaining confidentiality of test results and treatment. Input on evaluation methods to determine client satisfaction. and data interpretation. Community members where detainees reside Maintaining health and well­being of community; assuring that services are delivered in a culturally appropriate manner. Receive evaluation results. PARTNER STAKEHOLDERS INTERESTS/PERCEPTIONS/ CONCERNS INVOLVEMENT IN THE EVALUATION HIV CPG members Maintaining the health of the community; assuring that HIV/STD initiatives reflect comprehensive HIV plan; advocating for affected populations. Input on evaluation design, methods, and interpretation, use, and dissemination of results. Targeted medical providers in high­risk communities Reducing STDs among clients and larger community; assuring a quality STD intervention in the JDCs; following up with detainees’ sexual health after discharge. Initial input on evaluation design, and use and dissemination of results. Organizations that provide STD training to medical providers in JDCs (i.e., Prevention Training Center and STD program staff) Providing quality training to the initiative; cooperating with the funder of training services; securing results from the initiative that will reflect positively on training provided. Input on evaluation design, methods, and interpretation, use, and dissemination of results Health­related CBOs that serve high­risk adolescent females Maintaining health of client base; assuring a quality STD intervention in the JDCs; following up with detainees’ sexual health after discharge, assuring that services are delivered in a culturally appropriate manner. Initial input on evaluation design, and use and dissemination of results. 3. Retain Stakeholders At the end of the stakeholder meetings, the STD Program staff emphasized the importance of stakeholder involvement throughout the evaluation. They also stressed the need for ongoing communication so that stakeholder insights and opinions could be integrated into the evaluation process. With stakeholders, the STD program staff collaboratively developed a communications plan that included the following: • Dissemination of a monthly newsletter that summarizes the progress of the initiative and the evaluation as well as provides state­of­the­art information on STD morbidity and prevention strategies affecting adolescents in juvenile detention facilities. • Scheduling of targeted meetings with stakeholders to: ­Finalize the design of the initiative and identify realistic accomplishments. ­Overcome resistance to evaluation and identify benefits. ­Develop evaluation questions that are grounded in the perceptions, experiences, and interests of stakeholders. ­Select appropriate and acceptable evaluation methods. ­Facilitate confidential and practical data collection. ­Review evaluation findings, and make program recommendations. ­Decide how evaluation findings should be disseminated and used for program improvement. SUMMARY CHECKLIST: Things to consider when involving stakeholders in STD program evaluation CONCLUSION AND NEXT STEPS Stakeholders provide a wealth of information and insight for yourevaluation. As you engage in your evaluation process, identify stakeholders for your evaluation, involve them in the evaluation process, and retain their participation. In Step 2, you will determine the STD needs of your program. Based on the program needs, you will learn how to develop goals and objectives, identify program elements and develop a logic model. ACRONYMS USED IN THIS TOOL CBO – Community­based organization CPG – Community Planning Group DIS – Disease Intervention Specialist HIV – Human Immunodeficiency Virus JDC – Juvenile detention center STD – Sexually transmitted disease KEY TERMS Decision makers: Stakeholders in a position to do or decide something about your STD program/evaluation. Implementers: Stakeholders directly involved in undertaking your STD program activities. Participants: Stakeholders being served or affected by your STD program. Participatory Evaluation: Approach which involves stakeholders in all aspects of the evaluation process (i.e., design, question development, data collection, analysis, reporting, and use of results for decision making). Partners: Stakeholders who actively support and/or who have invested in your STD program. Stakeholders: Individuals or organizations interested in or affected by your STD program and/or the evaluation results (e.g., STD program staff, family planning staff, representatives of target populations). EXERCISE Identifying and Determining Involvement of Stakeholders for Program Evaluation For this exercise choose a program activity that you would like to evaluate. Use the questions that follow to fill in Table 4. • Who are the stakeholders to include in your evaluation? [Put these stakeholders in column 1 of Table 4.] ­Implementers: involved in the day­to­day operations of the activity to be evaluated (e.g., Carol X, DIS) ­Decision makers: have authority to make changes to the activity to be evaluated (e.g., John X, STD Director and/or Program Manager) ­Participants: served by the program/activity (e.g., clients/patients) ­Partners: invested/interested in the program/activity (e.g., Mary X, National Coalition of STD Directors) • What is their interest in the evaluation? (e.g., build their own evaluation capacity, use the evaluation results to improve service delivery, assure that patient’s rights are protected, etc.) [Put this information for each stakeholder in column 2 of Table 4.] • What do you foresee these stakeholders’ involvement in the evaluation? (e.g., planning/development logic model, development of evaluation questions, input on data collection instruments, review interpretation of findings, use findings)? [Put this information for each stakeholder in column 3 of Table 4.] Intervention/Activity to be Evaluated: Table 4: Stakeholder Information IMPLEMENTERSTAKEHOLDERS(specify names of individualsand their organizations) INTERESTS INVOLVEMENT INTHE EVALUATION DECISION MAKERSTAKEHOLDERS(specify names of individualsand their organizations) INTERESTS INVOLVEMENT INTHE EVALUATION (continued) Table 4: Stakeholder Information (continued) PARTICIPANTSTAKEHOLDERS(specify names of individualsand their organizations) INTERESTS INVOLVEMENT INTHE EVALUATION PARTNERSTAKEHOLDERS(specify names of individualsand their organizations) INTERESTS INVOLVEMENT INTHE EVALUATION REFERENCES Allison, M., Kaye, J. (1997). Strategic planning for nonprofit organizations: A practical guide and workbook. New York: John Wiley & Sons. Cantor, J., Kaufman, N., & Klitzner, M. (1984). Four steps to better objectives. Madison, WI: The Wisconsin Clearinghouse. Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR­11). Retrieved [October 20 2004], from http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm Centers for Disease Control and Prevention. (2002). Physical activity evaluation handbook. Retrieved October 20, 2004, from http://www.cdc.gov/nccdphp/dnpa/physical/handbook/index.htm Centers for Disease Control and Prevention. (n.d.). Program operations: Guidelines for STD prevention. Retrieved October 17, 2004, from http://www.cdc.gov/std/program/ Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adolescent and School Health. (2004). Evaluation steps tools. Unpublished manuscript. Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage. MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta: Centers for Disease Control and Prevention. Retrieved October 17, 2004, from http://www.cdc.gov/tobacco/evaluation_manual/ Evaluation.pdf Minkler, M., Wallerstein, N. (Eds.). (2003). Community­based participatory research for health. San Francisco: Jossey­Bass. Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage. APPENDIX [For meeting of approximately 15 stakeholders (including 8 STD program staff)] MEETING TITLE: STAKEHOLDER PARTICIPATION ON THE EVALUATION OF STD INITIATIVE X [DATE] SAMPLE AGENDA SCHEDULED TIME TOPIC 8:30 ­9:00 a.m. Welcome and participant introductions. 9:00 ­9:30 a.m. Overview of how stakeholders were selected, the STD Initiative X, and the proposed steps of an evaluation. 9:30 – 10:30 p.m. Identification of stakeholder interests, perceptions, and issues related to the initiative and the evaluation [small group exercise that feeds into group discussion in plenary]. 10:30 ­10:45 a.m. Break 10:45 ­12:00 p.m. Review of stages of evaluation planning and implementation. Brainstorming on how different stakeholders can be involved in providing input on the evaluation [exercise in plenary with poster board notes indicating roles commitments of stakeholders]. 12:00 ­12:30 p.m. Next Steps: Communication plan for ongoing stakeholder input. 12:30 p.m. Meeting adjourned and lunch