Describe the Program Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs STEP 1: ENGAGE STAKEHOLDERS 1.1 Determine how and to what extent to involve stakeholders in program evaluation . STEP 2: DESCRIBE THE PROGRAM 2.1 Understand your program focus and priority areas 2.2 Develop your program goals and measurable (SMART) objectives 2.3 Identify the elements of your program and get familiar with logic models 2.4 Develop logic models to link program activities with outcomes STEP 3: FOCUS THE EVALUATION 3.1 Tailor the evaluation to your program and stakeholders’ needs 3.2 Determine resources and personnel available for your evaluation 3.3 Develop and prioritize evaluation questions STEP 4: GATHER CREDIBLE EVIDENCE 4.1 Choose appropriate and reliable indicators to answer your evaluation questions 4.2 Determine the data sources and methods to measure indicators 4.3 Establish a clear procedure to collect evaluation information 4.4 Complete an evaluation plan based on program description and evaluation design STEP 5: JUSTIFY CONCLUSIONS 5.1 Analyze the evaluation data 5.2 Determine what the evaluation findings “say” about your program STEP 6: ENSURE USE OF EVALUATION FINDINGS AND SHARE LESSONS LEARNED 6.1 Share with stakeholders the results and lessons learned from the evaluation 6.2 Use evaluation findings to modify, strengthen, and improve your program SUGGESTED CITATION: Salabarría­Peńa, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007. Describe the Program D escribing your program is the second step in CDC’s framework for program evaluation. Note that in this step we describe the program and not the evaluation. A comprehensive program description ensures that program staff, evaluators, and other stakeholders share a clear understanding of what the program entails, and how its goals and objectives will be achieved. This understanding among stakeholders sets the stage for program evaluation and can be helpful in strategic planning, and performance measurement. The tools in Step 2 can help you: (1) identify the needs your program should address, (2) develop clear goals and objectives, (3) describe other program elements, and (4) develop a picture (logic model) of how you believe your program works. • Tool 2.1 provides guidance on how to assess the STD needs of your project area (e.g., city, state, nation, territory). • Tool 2.2 discusses the difference between goals and objectives and provides guidance on how to develop them. • Tool 2.3 describes the components for creating a logic model. • Tool 2.4 provides an opportunity for you to develop logic models and link program activities to outcomes. TOOL 2.1: UNDERSTAND THE YOUR PROGRAM FOCUS AND PRIORITY AREAS INTRODUCTION Describing your program includes explaining the health problems (e.g., syphilis, gonorrhea, Chlamydia) addressed by your program, how these affect different segments of the population, and any trends that may be occurring. This information can be obtained from a needs assessment and Tool 2.1 can help you understand the importance and benefits of identifying program needs and provides guidance on how to conduct a needs assessment. The flowchart below provides a description of the relationship between the activity of identifying the STD needs of your project area and the activity of engaging your stakeholders throughout the evaluation. UNDERSTANDING OF PROGRAM FOCUS AND PRIORITY AREAS LEARNING OBJECTIVES Upon completion of this tool, you will be able to: 1. Explain to other stakeholders the STD–related problem(s) addressed by your program. 2. Use appropriate data sources for justifying the STD needs in your project area (e.g., city, state, nation, territory) WHY IS IT IMPORTANT TO DETERMINE STD­RELATED NEEDS IN YOUR PROJECT AREA? Knowing the STD prevention and control needs of your project area provides an important foundation for understanding the focus of your program. It can also help you to develop realistic and measurable objectives and plan your program activities accordingly. A needs assessment is a starting point for program planning. It can help your program staff identify and measure gaps between “what is” and “what ought to be.” This knowledge can then be used to set program priorities and to develop program interventions and activities that reflect those priorities. Project areas, have undertaken needs assessments for some time since part of your Comprehensive STD Prevention System (CSPS) grant application requires you to provide behavioral and morbidity trends (e.g., number of cases, case rate) for the STDs that affect your project area. You have used that information to determine populations and diseases of high priority in your project area. This tool provides information on the process of conducting a needs assessment and it can also help to plan future needs assessments. If you have already gone through a similar process, discuss the most relevant information with the stakeholders. This will increase their familiarity with the focus and priority areas of the STD program. Next, go to Tool 2.2 where you will review the objectives pertaining to the activity you are interested in evaluating. WHAT ARE THE BENEFITS OF CONDUCTING A NEEDS ASSESSMENT? A needs assessment can benefit your program by helping you to: • Clearly determine both the met and unmet STD prevention and control needs within your area. • Identify target populations’ service needs related to preventing and treating STDs. • Clearly define your program purpose and scope. • Identify appropriate goals, objectives, and corresponding program interventions and activities. • Have a baseline from which to measure program achievements over time. • Get community support for your program. • Provide data to meet funder requirements and to seek additional funding. WHEN SHOULD YOU CONSIDER CONDUCTING A NEEDS ASSESSMENT IN YOUR PROJECT AREA? Consider conducting a needs assessment when: • You are preparing your grant application (e.g., CSPS). • You need to determine the STD­related risk factors among target populations. • Demographic changes occur within your target populations (e.g., increase in immigrant populations) and within the community your program serves. • Shifts occur in the incidence and prevalence of STDs within your community. • You are planning new STD prevention program activities and services. • You need additional support to maintain your program. HOW DO YOU DO A NEEDS ASSESSEMENT IN YOUR PROJECT AREA? A thorough needs assessment will serve as a basis for developing a strategic program and evaluation plan. This section provides information on steps you can take to determine the STD prevention needs in your area. Additional information on conducting needs assessment, including do’s and don’ts of needs assessment can be obtained from the Oxford County Library’s Needs Assessment/ Strategic Planning manual available online1 and additional resources included in the “References” section of this tool. 1. Identify stakeholders, staffing, and resources for the needs assessment. When possible, it is very important to involve key stakeholders in your needs assessment, including members of your target population (see Tool 1.1, Determine How and to What Extent to Involve Stakeholders in Program Evaluation) to determine what is known about the STD needs in your project area, and agree on the purpose of the needs assessment. Stakeholder involvement can help to ensure that needs are appropriately identified and understood and that this information directly informs program planning. They can provide or collect needed data. Most importantly, they can review the information from the needs assessment and help to set program priorities based on this information. Clarify the roles and responsibilities of the key staff within your program who are to plan and carry out components of the needs assessment. Also, map out a timeline and identify needed resources for completing the assessment. 2. Use existing data or collect new data to determine the STD related problems and their magnitude in the target population(s). To determine the STD­related problems of the target population, focus on two things: (1) the incidence and prevalence of STDs among target population(s) and (2) the consequences of STDs in terms of both morbidity and mortality. This means that you need to access and use existing data and/or collect new data. You can collect information directly from your target population(s) to fully understand their STD­related needs. Such information is known as primary data. Examples include STD testing and screening information you obtain from patients directly. Primary data can also be obtained when your program administers surveys, conducts key informant interviews, group interviews, or gathers information in community forums (see Appendix or more information on methods to collect needs assessment data). Example of the collection of primary data: STD program staff from project area X conducted a focus group with 10 individuals representing high school health programs and juvenile detention facilities. The staff queried about their knowledge of current adolescent sexual behaviors and partnering patterns. The participants consistently confirmed that most high school students and detainees had sexual relations before entering high school and that most had multiple partners. You may be able to obtain previously collected local­, state­, or national­level STD­related information (on STDs or your target populations) from other organizations which was not collected specifically for your needs assessment purposes, but may prove useful. Such information is known as secondary data. You may already have a considerable amount of secondary data that you have gathered to write your STD program grant application such as behavioral trends for the STDs that affect your project area. Examples of secondary data include the following: U.S. Census data, Centers for Disease Control and Prevention (CDC) epidemiological profiles, Morbidity and Mortality Weekly Report (MMWR data), national­level risk behavior research data, and data collected by academic institutions. When using secondary data, consider from whom, how, and when data were collected, and their applicability to your area and the community(ies) your program serves. For project areas with limited resources, the use of secondary data may be more feasible and practical than primary data because the latter may be expensive and time consuming. You may want to use the following sources of secondary data: • Local HIV program data • Sexually transmitted disease surveillance from CDC (www.cdc.gov/stats/). • Data from your state or local health department or CDC’s National Center for Health Statistics (NCHS) on mortality, morbidity, and determinants of health status. You can find NCHS data at www.cdc.gov/nchs/hus.htm. • Data from your state or local Department of Education • Substance Abuse and Mental Health Services Administration (SAMHSA) at www.samhsa.gov • Data from your State Department of Corrections • Data from your state or county government agencies on socio­ demographic characteristics including age, gender, ethnicity, income, employment, family size, geographic location, and other social indicators. You can find U.S. Census data at http://quickfacts.census.gov/qfd. Examples of the use of secondary data: While STD rates among teens in project area­X are much lower than national teen rates, project area­X rates have increased significantly since 1998. National data indicated that in project area­X Gonorrhea rates increased 229% from 18.7 per 100,000 in 1998 to 42.8 per 100,000 in 2002. Chlamydia rates almost doubled from 3% in 1998 to 5.7% in 2002. According to the 2003 Youth Risk Behavior Survey (YRBS), 15.9% of middle school students have had sexual intercourse. Nearly 5% had intercourse before age 11, and 5.4% have had sex with three or more people. Of those who had sex, 63.7% used a condom during their last sexual intercourse. You can collect primary and secondary data using a number of different methods. Appendix includes methods used to collect needs assessment data. 3. Identify possible causes of STDs in the target population. Data sources that can determine the health status of your target populations may also be used to better understand the causes and associated factors related to acquiring STDs. Such causes and factors could be intrapersonal such as clients’ lack of knowledge about STDs, inaccurate perceptions of risk for STDs, drug­taking behaviors, and low self­efficacy to negotiate for safer sex practices. Community and social norms (e.g., patterns regarding use of condoms with casual partners) or environmental factors (e.g., availability of free or low­cost condoms) related to risk behaviors and use of health services may play a role in whether the target population engages in safer behaviors. 4. Determine what other programs are available to address STD related problems. The first task is to identify any other current programs – whether from your Department of Health or others — that address STD prevention and control among the target population. Carefully examine and determine how these programs address current needs. Are the programs accessible? Do they adequately reach the target population? 5. Prioritize needs based on available resources and/or resources that can be obtained or developed. Because of limited resources, it is essential that your program staff, along with stakeholders, prioritize program needs based on a discussion of the more pressing STD needs and available resources (i.e., budget, staff, and time). At the end of this discussion, arrive at an agreement about the purpose of your program and your areas of special emphasis. For example, you may decide to implement a health education program to address the STD­related needs of adolescents in the county. Based on the available resources, you may narrow this to implementing the program only in high schools. Thus, the purpose of your program can be to address the STD needs of adolescents, with special emphasis on high school adolescents. If you require more assistance with conducting need assessments refer to the references provided in the tool. SUMMARY CHECKLIST: Sharing Evaluation Findings CONCLUSION AND NEXT STEPS Upon completion of this evaluation tool, you will understand the importance and benefits of determining the STD related need in your project area. You will have implemented the prescribed steps of a needs assessment in your STD program and produced information about the target population(s) and disease priorities you will address. In Tool 2.2, you will learn how to develop goals and measurable objectives. The tool will address the difference between goals and objectives and provides guidance on the development of process and outcome objectives. ACRONYMS USED IN THIS TOOL CBO – Community­based organization CDC – Centers for Disease Control and prevention CPG – Community Planning Group CSPS – Comprehensive STD Prevention System DIS – Disease Intervention Specialist DH – Department of Health HIV – Human Immunodeficiency Virus JDC – Juvenile detention center MSM – Men having sex with men MMWR – Morbidity and Mortality Weekly Report NCHS – National Center for Health Statistics STD – Sexually transmitted disease YRBS – Youth Risk Behavior Survey KEY TERMS Focus group: A qualitative method used to collect data from a group of people (about 6–11) who meet for 1–2 hours to discuss their insights, ideas, and observations about a particular topic with a trained moderator. Participants are selected because they share certain characteristics (e.g., individuals who have been tested for syphilis, women in detention facilities) relevant to the evaluation. Incidence: New cases of a disease in a specific population within a defined time period. Individual interview: A data collection method which involves dialogue with individuals who are carefully selected for their personal experience and knowledge with the issues at hand. Since these interviews are conducted individually, they are useful when anonymity is an issue or when asking about sensitive topics so participants can feel free to express their ideas Morbidity: Sickness or illness. Population at risk: Groups that have a high probability of developing an STD or related condition. Prevalence: Number of cases of a disease in a population at a given point in time. Primary data: Data directly obtained by your STD program (e.g., surveillance, number of sex partners of syphilis cases collected through DIS interviews). Risk factor: A factor that increases a person’s chances of getting a disease or condition (e.g., having multiple sexual partners, lack of access to health care). Secondary data: Information your STD program can use that has been collected by someone else (e.g., national data). This may include epidemiological data, socio­demographics, health risk behaviors, and health policies. CASE STUDY The following is an example of a syphilis­related needs assessment conducted by the STD program in City X. The program decided to examine the syphilis prevention plan in light of new national findings reported in CDC’s MMWR highlighting increasing rates of HIV among young men. The example outlines all the steps used in this tool. The STD program used both primary and secondary data to identify “young men (17­25 years of age) who have sex with men” as the target population. The approaches and methods used in this example are by no means exhaustive and are used only to illustrate a sample needs assessment. Following are the steps that program staff and stakeholders in City X implemented in their needs assessment. 1. Identify stakeholders, staffing, and resources for the needs assessment. The STD program identified the program staff and resources for the needs assessment. Stakeholders invited to be part of the project included the STD program director and manager, medical care providers, prevention service providers and clients from a number of community­based organizations, clinical staff serving populations at high risk of STDs (particularly syphilis), a member of City X HIV Prevention Program, and members of the HIV community planning group (CPG). 2. Use existing data or collect new data to determine the STD related problems and their magnitude in the target population(s). Program staff used primary and secondary data to determine the syphilis­related factors of the target population and their magnitude. For programs with limited resources (staff, budget, and time), the program staff may find it more practical and feasible to begin by using secondary data than by engaging in primary data collection. Primary Data To develop a more comprehensive understanding of syphilis as a public health problem in the community, the STD program, in collaboration with community partners, conducted 75 individual (open­ended) interviews and 5 focus groups, with nine men attending each. Each focus group was composed of men who met one of five distinct characteristics (i.e., white, Latino/Hispanic, African American, out­of­school youth, youth receiving services at the STD clinic). Men were recruited from advertisements in local gay newspapers, through e­mail networks of partner community­based organizations (CBOs), and through Internet chat rooms. Two public health school graduate students, well­trained in interviewing techniques and focus group facilitation, conducted the interviews and focus groups. Each interview lasted approximately 30 minutes. The interviews were conducted over the course of a week, and each participant received a food certificate donated by local businesses. The focus groups took place over a week and lasted 2 hours. Each participant received a movie pass donated by a local movie theatre. The interviews and focus group were designed to assess issues related to syphilis knowledge, attitudes, social norms, and behaviors regarding safer sex behaviors (e.g., condom use; number of sexual partners) among men having sex with men (MSM). In addition, interviews and focus groups explored issues relevant to men’s perceptions about the accessibility and quality of community health services, and factors that would predispose them to use such services. The interviews and focus groups revealed that some young MSM had inaccurate perceptions of their own vulnerability for syphilis given their sexual practices. In addition, many of the young MSM reported lack of skills to negotiate for safer sex, inconsistent condom use with casual partners, and that they felt highly influenced by social norms to use “party” drugs and to have sex with multiple partners. Secondary Data Secondary data were obtained from a recent study conducted by the local school of public health on the overall health of MSM and local epidemiologic profiles. These data showed that among young MSM, 8% were reported to have had syphilis, 10% Chlamydia, and 15% gonorrhea within the past year. 3. Identify possible causes of STDs in the target population. Based on a review of the primary and secondary data, program staff and stakeholders identified possible causes of increased rates of syphilis among MSM. They listed the following risk factors: • Unprotected sexual intercourse. • Multiple sexual partners. • Drug­taking behaviors. • Lack of knowledge about syphilis, symptoms, and treatment options. • Inadequate skills to negotiate for safer sex. 4. Determine what other programs are available to address STD related problems. Based on a review of available information, program staff and stakeholders found limited syphilis prevention programs for young MSM within the community. A local STD coalition consisting of the STD program staff, HIV/AIDS Department of Health (DH) staff and two local CBOs committed resources to address syphilis prevention among the young MSM population. The STD program had adequate resources to screen, test, educate, and provide clinical services to the MSM community, but these services were accessed by small numbers of MSM. One of the CBOs had initiated a community­level syphilis prevention intervention intended to educate the MSM community about syphilis and to change norms about sexual practices. 5.Prioritize needs based on available resources and/or resources that can be obtained or developed. The program staff and stakeholders came together to prioritize needs based on their needs assessment. It was evident to them that broader, more comprehensive STD prevention efforts needed to be undertaken with MSM in City X. They noted that unmet prevention and control needs included: • Availability of syphilis testing at non­traditional venues and times of day. • MSM knowledge about syphilis and other STDs, and HIV. • Community awareness about the link between HIV infection and syphilis. • Venue­based outreach. STD program staff and stakeholders noted that program needs included: • Addressing MSM lack of knowledge about syphilis. • Facilitating MSM acquisition of more realistic self­appraisals of their personal vulnerability for syphilis and other STDs. • Building MSM skills and self­efficacy to communicate and negotiate for safer sex. • Increasing MSM awareness of the syphilis outbreak, symptoms, and treatments. • Promoting community­wide norms that reduce the spread of syphilis. The program staff and stakeholders made a number of conclusions based on the needs assessment. Most importantly, they felt that syphilis screening must change in order to serve MSM more effectively. The data reflected that screening needed to be available in alternative locations such as sex venues and mobile sites. In addition, the times of availability for screening needed to be compatible with the activities and schedules of the population served. Later hours during the weekday and weekend hours should be provided in order to increase access to screening. The STD program staff and stakeholders concluded that increased access to screening increased the opportunity to diagnose and treat infections. Program staff and stakeholders made the following recommendations: 1. Syphilis screening must be pilot­tested at non­traditional venues (bath houses, CBOs, bars, etc.) to determine the effectiveness. In addition, already identified MSM­friendly clinics should have hours that more effectively serve their clientele. 2. A targeted media campaign specifically addressing MSM needs and risks should be implemented for syphilis. Increasing the community’s awareness of the connection between HIV infection and syphilis is also a priority. 3. Program planning and implementation should involve pertinent stakeholders, including: • STD and HIV DPH staff and managers • Staff and managers from four local CBOs serving MSM • Local MSM • Bathhouse owners and managers REFERENCES Academy for Educational Development, Center for Community­Based Health Strategies. (2000, August). Community­based health program tip sheet: The needs assessment – Tools for long term planning. [Brochure]. Washington, DC: Academy for Educational Development. Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR­11). Retrieved February 22, 2005, from http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adolescent and School Health. (2004). Evaluation steps tools. Unpublished manuscript. Last, J. N. (2001). Dictionary of epidemiology (4th ed.). New York: Oxford University Press. MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta: Centers for Disease Control and Prevention. Retrieved October 17, 2004, from http://www.cdc.gov/tobacco/evaluation_manual/ Evaluation.pdf McKenzie, J. F., Jurs, J. L. (1993). Planning, implementing, and evaluating health promotion programs: A primer. New York: Macmillan. Modeste, N. N. (1996). Dictionary of public health promotion and education: Terms and concepts. Thousand Oaks, CA: Sage. State of Wyoming Public Health Department. (2004, April). Maternal and child health needs assessment: Issue brief – Sexually transmitted diseases in adolescents. Retrieved November 5, 2004, from http://wdh.state.wy.us/mchepi/pdf/briefs/STD.pdf Windsor R., Baranowski, T., Clark, N., & Cutter, G. (1999). Evaluation of health promotion, health education, and disease prevention programs (2nd ed.). Mountain View, CA: Mayfield. Witkin, B. R., & Altschuld, J. W. (1995). Planning and conducting needs assessments. Thousand Oaks, CA: Sage. W.K. Kellogg Foundation. (1998, January). W.K. Kellogg Foundation evaluation handbook. Retrieved October 17, 2004, from http://www.wkkf.org/Programming/ResourceOverview.aspx?CID= 281&ID;=770 APPENDIX Methods Frequently Used to Collect Needs Assessment Data METHOD DESCRIPTION OF METHOD SAMPLE ASSESSMENT ISSUES ADDRESSED BY METHOD Focus group Interviews with small, homogenous group (about 6 ­11 individuals) intended to elicit insights, attitudes, and perceptions about a focused topic area. • MSM perceived risks of acquiring an STD if they are already HIV+. • Community members’ opinions of media campaign being designed. Individual interviews Interviews with individuals who are selected because of their relevant personal experience, knowledge, and/or expertise regarding the issues • Identification of best ways to introduce new treatment options for partner referrals. • STD program director and staff • Latino community advocates Community forums and public meetings Structured discussions with community members to elicit insights, opinions, perceptions, and recommendations related to health issues and community services. • Examination and discussion of the barriers to STD screening in community venues. • Discussion of how the local school board views provision of STD prevention education in high school. • Identification of barriers, and solutions to barriers, regarding integrating the local faith community into STD/HIV prevention efforts. Survey Data collection method which usually uses close­ended questions, thus providing quantitative data. It can be self­administered, or administered over the telephone, using a computer, or face­to­face. • Identification of demographic characteristics (e.g., race, ethnicity, sexual orientation) of the clients receiving clinical services. • Identification of service utilization patterns (e.g., number of MSM who received a syphilis screening within the last 6 months). • Identification of patterns and trends in risk behaviors (e.g., frequency of unprotected intercourse among heterosexual women). TOOL 2.2: DEVELOP YOUR PROGRAM GOALS AND MEASURABLE (SMART) OBJECTIVES INTRODUCTION You also need to describe your program goals and objectives. Goals and objectives are essential in program planning and they establish criteria and standards against which you can determine your program performance. STD­related program goals and objectives are based on program needs (see Tool 2.1), Healthy People 2010 objectives, and the Division of STD Prevention (DSTDP) Performance Measures. You can maximize the contribution of your stakeholders on your program goals and objectives by engaging them in their development or by sharing these with them. Understanding that you have already developed your program goals and objectives, you will need to identify the goals and objectives related with the program component or intervention you plan to evaluate. These goals and objectives should meet the criteria offered in this tool. The flowchart below provides a description of where developing program goals and objectives fits in with your past program planning activities. SMART SMART PROCESS OBJECTIVES OUTCOME OBJECTIVES LEARNING OBJECTIVES Upon completion of this tool, you will be able to: 1. Distinguish between a goal and an objective. 2. Develop one or more program goals and related objectives that will provide the basis for determining your program’s performance. WHAT IS A GOAL? A goal is a broad statement about the long­term expectation of what should happen as a result of your program. Well­written goals do the following: 1) establish the overall direction for and focus of your program, 2) define the scope of what you want your program to achieve, and 3) serve as the foundation for developing your program objectives. Constructing a well­written goal involves two steps: Step 1: Specify the STD problem or STD­related health risk factors. Programs funded by DSTDP might address a health problem (e.g., Chlamydia, gonorrhea, syphilis) or health risk factors (e.g., individuals’ low self­perceived risk for STDs; believing that condoms are difficult to use; having multiple sex partners). Step 2: Identify the target population(s) for your program. DSTDP­funded programs target populations and groups that are disproportionately affected by many STD pathogens. These populations and groups include: African Americans, Hispanics/Latinos, American Indian/Alaska Natives, Asian and Pacific Islanders, adolescents/young adults, women, men who have sex with men (MSM), people who have multiple sex partners, and sex workers (people who exchange sex for money, drugs, or other goods). Populations may also be identified by the location where they access care: correctional facilities, family planning clinics, human immunodeficiency virus (HIV) prevention and care clinics, substance abuse centers, and school health clinics, medical care DESCRIBE THE PROGRAM facilities. For some STDs that affect the population in general (e.g., genital herpes, human papillomavirus), you may need to define the target population more broadly. Following are two examples that illustrate a health problem and the related health risk behaviors, target populations, and goals. Example 1 HEALTH PROBLEM: STD officials in County X reported that Chlamydia (Ct) was 4 times higher in female adolescents (aged 16­18 years) screened in juvenile detention facilities than in female adolescents screened in other settings during the year 2002. Among female adolescents entering juvenile detention facilities, the rate for Ct was 16.7%, compared with 4.1% for school attendees. HEALTH RISK FACTORS: Unprotected sex; lack of awareness of how STDs are acquired. TARGET POPULATION: Female adolescents engaging in high­risk behaviors. RELATED GOAL: To reduce the prevalence of Ct among female adolescents in juvenile detention centers in County X. Example 2 HEALTH PROBLEM: STD officials in City Y reported an increased number of syphilis cases among heterosexual males. In 2003, heterosexual males represented 0.9% of syphilis cases in males; in 2004, they made up 30% of male cases. HEALTH RISK FACTORS: Unprotected sex; having multiple partners. TARGET POPULATION: Heterosexual males engaging in high­risk behaviors. RELATED GOAL: To reduce the incidence of primary and secondary syphilis among heterosexual males in City Y. WHAT IS AN OBJECTIVE? Your program objectives are statements describing the results to be achieved, and the manner in which they will be achieved. Objectives are more immediate than goals; they are the mileposts you will pass on the way to achieving your program goal(s). Because objectives detail your program activities, you usually need multiple objectives to address a single goal. Well­written and clearly defined objectives will help you monitor your progress toward achieving your program goals and set targets for accountability. HOW CAN YOU WRITE AN OBJECTIVE THE “SMART” WAY? Your objectives will be appropriate and effective if you follow the SMART technique for writing objectives. The attributes of a SMART objective are: Specific, Measurable, Achievable, Relevant, and Time­bound. [Note: Be aware that there is a variety of terminology used for the five components of SMART. For instance the “A” sometimes stands for appropriate or the “R” sometimes stands for realistic. The bottom line is that the message is the same.] Specific. Making your objectives specific means including the “who,” “what,” and “where” of the objective. “Who” refers to your target population (e.g., Latino adolescents, attendees of STD clinics). “What” refers to the action (e.g., screen, identify). “Where” refers to the location of the action (e.g., STD clinic in City X). Be as specific as possible about the target population (e.g., male and female adolescents between the ages of 15­19 years, instead of “adolescents”). When describing the action, use only one action verb per activity (e.g., develop a workshop instead of develop and implement a workshop). More than one verb means that more than one action must be measured, which will cause problems when it comes to measuring success. For instance, suppose that your program was able to develop the workshop but did not implement it. Did you meet your objective? Because the objective had two actions, your success would be difficult to measure. Also, avoid verbs with vague meanings (e.g., “understand”, “do”) when describing expected results. Instead, use verbs that reflect measurable action, such as “identify” or “list” (see Appendix A or examples of precise and vague verbs). Remember: The greater the specificity, the greater the possibility for measurement. Measurable. Your objectives need to be measurable. Here the focus is on “how much” change is expected. Your objectives should quantify the amount of change you hope to achieve (e.g., Project area X will implement 2 professional development workshops among all STD clinical providers in State X by January 2007.). “2” and “all” represent the “how much” of the objective. Achievable. Your objectives should be realistic given your program resources and planned implementation. For instance, if you read the following: “100% of women in project area X will be screened for Ct” you realize that this is not achievable. Besides the fact that reaching 100% of women is unrealistic, you will be wasting resources because not all women are at risk for Ct. You can use state, county, or local statistics as well as data from similar STD programs to provide context for what is reasonable and to help you ensure that your program objectives are achievable. Relevant. Objectives are relevant when they relate directly to the program’s goals and together represent reasonable programmatic steps that can be implemented within a specific timeframe. For instance, a program goal is “Reduce congenital syphilis in City X”. A relevant objective may be: “By December 2006, increase the percentage of women (from X% to Y%) in City X receiving a test for syphilis at first prenatal visit”. Time­bound. Your objectives should be defined within a timeframe. Here the focus is on “when” the objective will be met. Specifying a timeframe in the objective will help you in both planning and evaluating your program (e.g., at the end of laboratory visits; by January 2007). Example GOAL: Reduce STDs among inmates in correctional institutions in Flowers City. SMART OBJECTIVE: By August 2008, increase the percentage (from X% to Y%) of clinical staff in all correctional facilities in Flowers City that fully adhere to the CDC­STD treatment guidelines. • This objective is specific because it identifies a single action: increase; and focuses on a target group i.e., clinical staff in all correctional facilities in Flowers City. • This objective is measurable because it specifies how much change is expected: increase from X% to Y% of clinical staff in all correctional facilities that fully adhere to the CDC­STD treatment guidelines. • This objective is achievable because it is realistic given the timeframe. • This objective is relevant because it relates to the program goal of reducing the proportion of inmates with STDs in correctional institutions in Flowers City by assuring that infected individuals are treated according to guidelines. • This objective is time­bound because it provides a specified time by which the objective will be achieved: August 2008. WHAT ARE PROCESS AND OUTCOME OBJECTIVES? You can write two types of objectives: process and outcome. When you write a process objective, you describe the activities/services that will be delivered as part of implementing the program. Example of a SMART process objective: By (month/year), STD training staff will provide one professional development workshop for disease intervention specialists (DIS) on protocols for timely interviewing of primary and secondary syphilis cases. DESCRIBE THE PROGRAM When you write an outcome objective, you specify the intended effect of the program in the target population or end result of a program. The outcome objective focuses on what your target population(s) will know or will be able to do at the conclusion of your program/activity. Example of a SMART outcome objective: By (month/year), the proportion of primary and secondary syphilis cases interviewed within 7 days of the date of specimen collection will increase from X% at baseline to Y% in City X. WHAT ARE SHORT­TERM, INTERMEDIATE, AND LONG­TERM OUTCOME OBJECTIVES? You can categorize outcome objectives as short­term, intermediate, or long­term. They should be logically linked to each other and to the process objectives. • Short­term outcome objectives are the initial expected changes in your target population(s) after implementing certain activities or interventions (e.g., changes in knowledge, skills, and attitudes). • Intermediate outcome objectives are those interim results that provide a sense of progress toward reaching the long­term objectives (e.g., changes in behavior, norms, and policy). • Long­term outcome objectives are achieved only after the program has been in place for some time (e.g., changes in mortality, morbidity, quality of life). Example Following is an example of linking a program goal with process and outcome (short­term, intermediate, and long­term) objectives. SAMPLE GOAL: Reduce the prevalence of Ct infection among high­risk female adolescents in County Z. SAMPLE PROCESS OBJECTIVES: 1. By (month/year), STD program staff will disseminate protocols for Ct screening for female adolescents age 15­19 to all family planning clinics in County Z. 2. By (month/year), provide two professional development workshops for all medical staff of County Z family planning clinics on protocols for the Ct screening of female adolescents age 15­19. SAMPLE SHORT­TERM OUTCOME OBJECTIVES: 1. By (month/year), all medical staff who have participated in the Ct training will report in a pre/post questionnaire, an increase in knowledge of the Ct screening protocols from X% to Y%. 2. By (month/year), medical staff who participated in the Ct training will demonstrate, in a performance session, at the end of the training, an improvement in Ct prevention counseling skills from X% to Y%. SAMPLE INTERMEDIATE OUTCOME OBJECTIVES: 1. By (month/year), the percentage of medical staff in family planning clinics in County Z who screen at­risk female adolescents for Ct as a regular part of the health assessment will increase from X% to Y%. 2. By (month/year), the percentage of medical staff who conduct Ct prevention counseling in family planning clinics in County Z will increase from X% to Y%. SAMPLE LONG­TERM OUTCOME OBJECTIVE: 1. The prevalence of Ct among high risk female adolescents within County Z will have been reduced from X% in (year) to Y% in (year). SUMMARY CHECKLIST: Developing Well­Written Goals and Objectives CONCLUSION AND NEXT STEPS After completing this tool, you will have learned to develop program goals that support the purpose of your program. You will be able to use the SMART technique to generate (1) process objectives related to the implementation of your program, and (2) outcome objectives related to the kinds of changes that you expect to make in the target population(s) served by your program. These goals and objectives provide a frame of reference for articulating the other elements of your program and for initially understanding the use of a program logic model, which you will learn more about next in Tool 2.3 (Describe your Program and Get Familiar with Logic Models). ACRONYMS USED IN THE TOOL Ct ­Chlamydia CSPS ­Comprehensive STD Prevention Systems DIS ­Disease Intervention Specialists DSTDP ­Division of STD Prevention, CDC HIV ­Human Immunodeficiency Virus MSM ­Men who have sex with men SMART ­Specific, Measurable, Achievable, Relevant, and Time­bound objectives STD ­Sexually transmitted disease KEY TERMS Goal: A broad statement related to the purpose of your STD program, which states what your program will accomplish (the desired result). Objectives: Measurable statements that describe the manner in which your STD program goals will be achieved. They will help you monitor your progress toward achieving program goals. Outcome Objectives: Measurable statements specifying the intended effect of your program in the target population(s) or end result of your STD program. Process Objectives: Measurable statements describing your STD program activities and the actions involved in their implementation. STD­related risk factors: Specific behaviors, attitudes and/or limited knowledge that put individuals in your project area at risk of STDs SMART: An acronym describing criteria used to write objectives that are Specific, Measurable, Achievable, Relevant, and Time­bound. Note: On the next two pages there are two exercises related to program goals and objectives. We encourage you to complete these exercises in order to apply what you learned from this tool. EXERCISE 1: Matching Goals and Objectives Listed below are goals and objectives for two hypothetical STD program components (A and B). • Improve STD surveillance and data management in City X. • Reduce Ct and gonorrhea rates among women in County X. • Provide staff training on data reporting. • To increase knowledge of CDC requirements for data reporting. • Family planning clinics will develop a computerized case management system that facilitates timely follow­up with clients for treatment appointments. • To increase the proportion of women with positive Ct and gonorrhea tests who are treated. Here are the steps for completing the exercise: • Identify which two statements are the STD program component goals, and write them in the spaces provided (Goal A and Goal B). • Identify which statements are objectives, and match each objective with its respective goal. • Among these objectives, determine which are process objectives and which are outcome objectives. Write them in the appropriate places in the list. • Under each process and outcome objective, rewrite the objectives using the SMART technique. Program Component A Goal A: Process objective A: Revised SMART process objective A: Outcome objective A: Revised SMART outcome objective A: Program Component B Goal B: Process objective B: Revised SMART process objective B: Outcome objective B: Revised SMART outcome objective B: Note: You will find the answers and suggested objectives at the end of this tool. EXERCISE 2: Writing Goals and ‘SMART’ Objectives INSTRUCTIONS • Review the information presented below. • Develop in the table on next page at least one process (operations) and one outcome (effects) objective in the SMART way based on the “Background Information” provided. • Classify your outcome objective(s) into short, intermediate or long term. BACKGROUND INFORMATION This exercise relates to a professional development (PD) component of a larger Chlamydia (Ct) screening initiative that the STD Program plans to conduct in 3 juvenile detention centers (JDCs) in County Z. The goal of the initiative is to reduce the prevalence of Ct among female adolescents (13­17 years of age) in JDCs in County Z. The PD component will include the provision of two Ct training sessions (i.e., a general and a clinical). The general session (for clinicians and non­clinicians) will provide (1) an overview of the epidemiology of Ct, and (2) STD prevention counseling techniques, particularly as they relate to adolescent females. The topics for the clinical session (only for clinicians) will cover Ct screening protocol (STD clinical guidelines), including the quality of specimen collection, diagnostic methods, and treatment options. Training material will be developed accordingly. It is expected that JDC staff (clinicians and non­clinicians) who participate in the general session will increase their knowledge of the epidemiology of Ct and skills in providing STD prevention counseling. It is also expected that JDC clinical staff who participate in the clinical session will increase their skills in screening, diagnosing and treating Ct, according to clinical guidelines (i.e., protocol). In addition, it is expected that after participating in training, the clinical staff will fully apply the Ct screening protocol among eligible detainees, and that the prevalence of Ct among high risk female adolescents within the targeted JDCs will decrease. CRITERIA FOR SMART OBJECTIVES • Specific: includes the “who”, “what”, and “where”. Use only one action verb. • Measurable: focuses on “how much” change is expected. • Achievable: realistic given program resources and planned implementation. • Relevant: relates directly to program/activity goals. • Time­bound: specifies date in which the objective will be achieved. GOAL: To reduce the prevalence of Ct among female adolescents (13­17 years of age) in Juvenile Detention Centers (JDCs) in County Z. PROCESS OBJECTIVES OUTCOME OBJECTIVES SHORT­TERM INTERMEDIATE LONGTERM REFERENCES Academy for Educational Development. (1996). Chapter 4: Goals and objectives. In Planning and evaluating HIV/AIDS prevention programs in state and local health departments. (Document produced for the Centers for Disease Control and Prevention).Washington, DC. Aspen Reference Group. (1997). Community health education and prevention: A guide to program design and evaluation. Rockville, MD: Aspen. Cantor, J., Kaufman, N., & Klitzner, M. (1984). Four steps to better objectives. Madison, WI: The Wisconsin Clearinghouse. Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR­11). Retrieved February 22, 2005 from http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adolescent and School Health. (2004). Evaluation steps tools. Unpublished manuscript. MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta: Centers for Disease Control and Prevention. Retrieved October 17, 2005, from http://www.cdc.gov/tobacco/evaluation_manual/Evaluation.pdf McKenzie, J. F., Jurs, J. L. (1993). Planning, implementing, and evaluating health promotion programs: A primer. New York: Macmillan. Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage. Should you need help or more information regarding this or any other evaluation tool, please contact CDC DSTDP program evaluation staff at (404) 639­8276. POSSIBLE ANSWERS FOR EXERCISE 1 Goal A: Improve STD surveillance and data management in City X. Process objective A: Provide staff training on data reporting. Revised SMART process outcome A: By (month/year), the STD surveillance unit will provide X number of professional development workshops to STD program staff in City X on the importance of data reporting and CDC requirements. Outcome objective A: To increase knowledge of CDC requirements for data reporting. Revised SMART outcome objective A: By (month/year), the percentage of STD surveillance staff (attending the workshops in City X) who can correctly describe the CDC requirements for complete data reporting will increase from X% to Y%. Goal B: Reduce Ct and gonorrhea rates among women in County X. Process objective B: Family planning clinics will develop a computerized case management system that facilitates timely follow­up with clients for treatment appointments. Revised SMART process B: By (month/year), X% of family planning clinics in County X will complete the development of a computerized case management system that facilitates the timely follow­up appointments for treatment of female clients with Ct and gonorrhea. Outcome objective B: To increase the proportion of women with positive Ct and gonorrhea tests who are treated. Revised SMART outcome B: • By (month/year), the STD Program in County X will increase the percentage (from X% to Y%) of women with positive Ct test results who are treated within 14 days of the date of specimen collection. • By (month/year), the STD Program in County X will increase the percentage (from X% to Y%) of women with positive gonorrhea test results who are treated within 14 days of the date of specimen collection. POSSIBLE ANSWERS FOR EXERCISE 2 GOAL: To reduce the prevalence of Ct among female adolescents (13­17 years of age) in Juvenile Detention Centers (JDCs) in County Z. PROCESS OBJECTIVES OUTCOME OBJECTIVES SHORT­TERM INTERMEDIATE LONGTERM • By (month/year), the STD program staff will prepare all training materials (i.e., sessions curriculum, presentation, handouts for participants, performance exercise, pre/post questionnaires) to use with JDC staff. • By (month/year), one training session will be conducted with 15 JDC staff (including 6 clinicians) from 3 JDCs on an overview of epidemiology of Ct. • By (month/year), one training session will be conducted with 6 clinical staff in 3 JDCs on screening protocol (i.e., STD clinical guidelines). • By (month/year), the JDC staff (15) participating in the Ct training will demonstrate increased knowledge (from X% to Y%) in the epidemiology of Ct in a pre/post questionnaire. • By (month/year), all 6 clinical staff who participated in the training will report in a pre/post questionnaire, an increase in knowledge of the Ct screening protocol from X% to Y%. • By (month/year), all training participants will demonstrate, in a performance session at the end of the training, their skills in providing STD prevention counseling. • By (month/year), supervisors of the 6 clinicians in 3 JDCs will report an increase in Ct screening (from X% to Y%) among eligible detainees through clinical audits of staff and chart reviews. • The prevalence of Ct among high risk female adolescents within JDCs participating in the Ct training will be reduced from X% in (year) to Y% in (year). DESCRIBE THE PROGRAM APPENDIX A Examples of Vague and More Precise Verbs APPENDIX B Worksheet for Developing SMART Objectives This worksheet is intended to help you with developing SMART objectives. Begin by stating one or more program goals, from which the SMART objectives will be developed. Program Goal(s): 1. 2. Criteria for goals: 1) Specifies the STD problem or STD­related health risk factors; 2) Identifies the target population(s) for your program. Related SMART Process Objectives: 1. 2. Related SMART Outcome Objectives: 1. 2. Criteria for SMART objectives: • Specific: includes the “who”, “what”, and “where”. Use only one action verb. • Measurable: focuses on “how much” change is expected. • Achievable: realistic given program resources and planned implementation. • Relevant: relates directly to program/activity goals. • Time­bound: specifies date in which the objective will be achieved. TOOL 2.3: IDENTIFY THE ELEMENTS OF YOUR PROGRAM AND GET FAMILIAR WITH LOGIC MODELS INTRODUCTION When you have a clear description of your program, you achieve an important milestone toward designing an evaluation of your program or a program activity. You started to do this when you developed your program goals and objectives. Using this tool, your program staff and stakeholders will be able to describe the program more fully by identifying your program’s core components and activities, and their ability to achieve the desired results of the program. While your program may have been functioning for awhile and many of these elements are already articulated, it is important to engage program staff and stakeholders in a more lengthy discussion about the program. This will allow everyone who is planning the evaluation to better understand how the program will attain the expected results, and identify where the barriers of doing so might be. This tool will allow you to become familiar with logic models which can assist you in strategic planning, program planning, and program evaluation by graphically illustrating how your program is intended to work. This step is extremely fundamental for program evaluation to the point that some people think that this is where the actual evaluation process starts. The flowchart below provides a description of where the logic model fits in with your past planning activities. LEARNING OBJECTIVES Upon completion of this tool, you will be able to: 1. Describe the elements of your program. 2. Identify the components of a logic model. WHAT ARE THE ELEMENTS OF YOUR PROGRAM? The first step in developing a logic model is to consider and identify the elements of your program. Since it may not be feasible to evaluate the entire program, we will refer to a program activity or component you plan to evaluate. The following elements will give you the background information you need to develop a logic model. These include: • Goals and objectives, which are the criteria and standards against which you can determine your program performance (for more information, refer to Tool 2.2.). • Activities, which are those specific actions that your program plans to implement during the funding cycle. • Inputs, which are the resources for conducting program activities (e.g., time, staff, technology, information, money). Two additional things to consider before developing a logic model are the stage of development of the program activity to be evaluated and the context in which it operates. • Stage of development is the level of maturity of your program activity classified in three stages: planning, implementation and maintenance/outcomes achievement. The stage influences the type of evaluation you may want to conduct as will be discussed in Step 3 (tools 3.1). For example: • If you are planning a program activity, the intent of the evaluation is to refine the program plan. This also applies to new activities or components you plan to add to the program. • If you are already implementing the program activity, the evaluation goal may be to see what works and to improve operations. • If you have implemented your program activity for years, your program may be mature enough for you to evaluate its outcomes (effects). • Context refers to the setting and environmental influences in which your STD program activities operates. This can include a broad array of external factors such as STD­related community’s/state’s laws, regulations, political climate, cultural norms, and social and economic conditions. These influences can significantly affect how you implement your program activities, whether you reach specified outcomes, and the design and interpretation of evaluation results. WHAT IS A LOGIC MODEL? A logic model is a picture of how a program works. You may have heard other names for logic models such as theory of change, conceptual map, strategic framework, program hypothesis, causal chain, or roadmap. While logic model terminology may vary, the Division of STD Prevention (DSTDP) uses the terms reflected in Table 1. TABLE 1. Logic Model Terms Used by DSTDP A logic model for your program will identify its main components and show how they relate to one another. See Figure 1 for an illustration of a basic logic model. The left­hand side of the logic model shows the process components (i.e., inputs, activities, and outputs) that correspond to the actions you plan for your STD program. The right­hand side depicts the program outcomes or the effects you hope your STD program will have in your target population(s). The outcomes (i.e., short­term, intermediate, and long­term) are in sequence, according to the time needed to accomplish them. Figure 1: Process and Outcome Components of a Logic Model WHY IS A LOGIC MODEL USEFUL? Logic models can benefit your program planning by (1) building understanding and clarity about your program among program staff and stakeholders, (2) identifying resources needed for your program, (3) identifying the sequencing of activities that should be implemented, and (4) serving as a basis for program evaluation. Another important benefit is that when you involve a broad range of stakeholders in constructing a logic model, they can better understand why it takes time to achieve long­term outcomes (short and intermediate outcomes must be achieved first), and help prioritize the activities that need to be evaluated. Your stakeholders’ involvement in constructing a logic model will promote their commitment to, and shared vision and ownership of the program plan, as well as their initial buy­in on an evaluation. HOW CAN YOU USE YOUR GOALS AND OBJECTIVES TO DEVELOP A LOGIC MODEL? Because you have developed goals and SMART objectives in your CSPS application, you already have the content for the items you will use in the logic model regarding inputs, activities, outputs, short­term outcomes, intermediate outcomes, and long­term outcomes. Essentially, your process objectives contain content that will help you write your inputs, activities and outputs; your short­term objectives have content for your short­term outcomes; your intermediate objectives have content for your intermediate outcomes; and your long­term objectives have content for your long­term outcomes. The following are two examples that illustrate the connection between goals and objectives and elements of logic modeling. Example 1 is of an everyday process like preparing a peanut butter sandwich and Example 2 is related to an STD program campaign. Example 1 Logic Model for Making Peanut Butter Sandwiches GOAL: To eat homemade peanut­butter sandwich for breakfast. PROCESS OBJECTIVE: By (month/year), I will get one loaf of bread, a jar of peanut butter and one jar of jelly of my choice. SHORT­TEM OUTCOME OBJECTIVE: By (month/year), I will prepare for myself one peanut butter and jelly sandwich for breakfast. INTERMEDIATE OUTCOME OBJECTIVE: By (month/year), I will increase the number of sandwiches prepared for my family from 0 to 2 in a week. LONG­TEM OUTCOME OBJECTIVE: By (month/year), I will increase the number of requests I receive from family members for peanut butter and jelly sandwiches for breakfast. Example 2 Logic Model Elements for an STD Program Campaign This is an example of an STD program campaign designed to inform females 20­24 years of age who visit family planning clinics in City X about the risk to their reproductive health from chlamydial (Ct) infections. The goal is to prevent Ct infections among females 20­24 years of age in City X. USE OF A PROCESS OBJECTIVE TO DEVELOP “INPUTS, ACTIVITIES AND OUTPUTS” IN A LOGIC MODEL: Process Objective – By (month/year), STD program staff will distribute 10 posters and 6,000 educational leaflets on chlamydia (Ct) to 10 family planning clinics that are frequented by females 20­24 years of age, in City X. • Inputs: 10 posters; 6,000 educational leaflets, staff time, transportation costs. • Activities: Distribute posters and educational leaflets to family planning clinics that are frequented by females 20­24 years of age. • Outputs: Posters and leaflets distributed to family planning clinics that are frequented by females 20­24 years of age. USE OF A SHORT­TERM OUTCOME OBJECTIVE TO DEVELOP A “SHORT­TERM OUTCOME” IN A LOGIC MODEL: Short­term Outcome Objective ­By (month/year). X% of sexually active females 20­24 years in 10 family planning clinics in City X will report in a questionnaire 3 ways of preventing Ct. Short­term Outcome: Increased knowledge of how to prevent Ct among females 20­24 years of age. USE OF AN INTERMEDIATE OUTCOME OBJECTIVE TO DEVELOP AN “INTERMEDIATE OUTCOME” IN A LOGIC MODEL: Intermediate Objective: By (month/year), X% of sexually active females 20­24 years of age, in 10 family planning clinics in City X will have reported increased use of condoms in the past 6 months by X%. Intermediate Outcome: Increased condom use among females 20­24 years of age. USE OF A LONG­TERM OBJECTIVE TO DEVELOP A “LONG­TERM OUTCOME” IN A LOGIC MODEL: Long­Term Objective: Chlamydia cases among females 20­24 years of age, in family planning clinics in City X will have been reduced from X% in (month/year) to Y% in (month/year). Long­Term Outcome: Reduced prevalence of Chlamydia among females 20­24 years of age who attend family planning clinics. SUMMARY CHECKLIST: Identify program elements of your program and get familiar with logic models CONCLUSION AND NEXT STEPS This tool discussed the importance of articulating the elements of your program and how to use them as a starting point for developing a logic model. In the next tool — 2.4 Develop Logic Models to Link Program Activities with Outcomes – you will use the information on the components of a logic model to actually construct logic models that will help you plan any evaluation activities you intend to undertake. ACRONYMS USED IN THIS TOOL CSPS ­Comprehensive STD Prevention System DIS ­Disease Intervention Specialist DSTDP ­Division of STD Prevention SMART ­Specific, measurable, achievable, relevant, and time­bound objectives STD ­Sexually transmitted disease KEY TERMS Activities: Actual events that take place as part of your STD program (e.g., develop pamphlet, screen patients). Context: The setting and environmental influences in which your STD program operates (e.g., laws, regulations, political climate). Inputs: Your STD program’s resources (e.g., money, staff, materials). Intermediate outcomes: Intended effects of your program in the target population that takes longer than short­term outcomes to occur (e.g., changes in STD­related policy or in behavior of the target population). Logic Model: Also referred to as Global Logic Model, a picture of how your STD program is supposed to work. Long­term outcomes: Intended effects of your program in the target population that may take several years to achieve, such as reduced disease transmission and incidence. Outcomes: The intended effects or changes of in the target population(s) that result from your STD program. Outputs: The direct products of your program activities or services delivered (e.g., pamphlet developed, patients screened). Short­term outcomes: Immediate effects of the program in the target population, such as changes in knowledge, attitudes, skills, awareness, or beliefs of the target population. SMART: An acronym describing criteria used to write objectives that are Specific, Measurable, Achievable, Relevant, and Time­bound. EXERCISE 1: Translating Objectives into Logic Model Components Instructions: Listed below are sample process and outcome objectives that were developed for a program being implemented to reduce Gonorrhea among sexually active male adolescents in state X. Review each of these objectives and translate them into appropriate logic model components (as shown in the example on pages 4 and 5 of this tool). [Please note: You will find the correct answers at the end of this tool.] Process objective: By (month/year), X% of STD program staff in City Z will attend a workshop on techniques to improve partner elicitation and management services among male adolescents (14­18 years of age). Inputs: Activities: Outputs: Short­term outcome objective: By (month/year), X% of STD clinics in City Z will have paired experienced DIS (Disease Intervention Specialists) with new DIS (having less than 6­months experience) to improve partner elicitation and partner management skills. Short­term Outcome: Intermediate outcome objective: By (month/year), X% of DIS in City Z will have increased their rate of eliciting locatable sex partners among male adolescents (14­18 years of age) from Y% to Z%. Intermediate Outcome: Long­term outcome objective: By (month/year), the prevalence of gonorrhea among male adolescents (14­18 years of age) in City Z will be reduced from X% (in month/year) to Y%. Long­Term Outcome: EXERCISE 2: Matching Program Logic Model Components Instructions: Listed below are nine selected items from a logic model depicting an STD screening, treatment, and counseling program. The goal of the program is to reduce Gonorrhea in State X. Match each item with one of the six components of a logic model: Inputs, Activities, Outputs, Short­Term Outcomes, Intermediate Outcomes, and Long­Term Outcomes. Put these items in the correct components in the attached logic model template. [Please note: You will find the correct answers at the end of this tool.] Items in a logic model for an Gonorrhea screening, treatment, and counseling program 1. Screen, treat and counsel patients. 2. Six medical staff, two DIS, and two administrative staff. 3. Increased rates of condom use among patients. 4. Increased awareness of Gonorrhea transmission and prevention among patients. 5. $XX,XXX in health department funds. 6. Increased rate of abstinence until treatment is complete. 7. Patients screened, treated, and counseled. 8. Decreased prevalence of Gonorrhea. Logic Model Template for Exercise 2 SHORT­TERM INTERMEDIATE LONG­TERM INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES REFERENCES Centers for Disease Control and Prevention. (n.d.). Oral health infrastructure development tools, Step 2B: Logic models. Retrieved October 17, 2004, from http://www.cdc.gov/oralhealth/library/pdf/logic_models.pdf Harvard Family Research Project. (1999). Learnings from logic models: An example of a family/school partnership program. Retrieved November 5, 2004, from http://gseweb.harvard.edu/hfrp/pubs/onlinepubs/rrb/learning.html MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta: Centers for Disease Control and Prevention. Retrieved October 17, 2004, from http://www.cdc.gov/tobacco/evaluation_manual/Evaluation.pdf United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA. Retrieved October 17 2004, from http://national.unitedway.org/outcomes/library/pgmomres.cfm W.K. Kellogg Foundation (2001, December). Logic model development guide. Retrieved November 7, 2004, from http://www.wkkf.org/Programming/ResourceOverview.aspx?CID=2 81&ID;=3669 POSSIBLE ANSWERS FOR EXERCISE 1 (The answers provided in the answer key are by no means exhaustive of all potential answers.) Process objective: By (month/year), X% of STD program staff in City Z will attend a workshop on techniques to improve partner elicitation and management services among male adolescents 14­18 years of age. • Inputs: funding for program staff to attend workshop; training schedule; staff time • Activities: program staff sign­up to attend workshop • Outputs: staff signed up to attend workshop; staff attended the workshop Short­term outcome objective: By (month/year), X% of STD clinics in City Z will have paired experienced DIS (Disease Intervention Specialists) with new DIS (having less than 6­months experience) to improve partner elicitation and partner management skills. • Short­term Outcome: Increased instances of DIS working in pairs on partner elicitation and management. Intermediate outcome objective: By (month/year), X% of DIS (Disease Intervention Specialists) in City Z will have increased their rate of eliciting locatable sex partners among male adolescents (14­18 years of age) from Y% to Z%. • Intermediate Outcome: Increased elicitation of locatable sex partners of male adolescents. Long­term outcome objective: By (month/year), the prevalence of gonorrhea among male adolescents (14­18 years of age) in City Z will be reduced from X% (in month/year) to Y%. • Long­Term Outcome: Reduced prevalence of gonorrhea among male adolescents. ANSWERS FOR EXERCISE 2 Logic Model Item 1. Screen, treat, and counsel patients – Activities Logic Model Item 2. Six medical staff, two DIS, and two administrative staff – Inputs Logic Model Item 3. Increased rates of condom use among patients – Intermediate Outcome Logic Model Item 4. Increased awareness of Gonorrhea transmission and prevention among patients – Short­Term Outcome Logic Model Item 5. $XX,XXX in health department funds – Inputs Logic Model Item 6. Increased rate of abstinence until treatment is complete– Intermediate Outcome Logic Model Item 7. Patients screened, treated, and counseled – Outputs Logic Model Item 8. Decreased prevalence of Gonorrhea – Long Term Outcome TOOL 2.4: DEVELOP LOGIC MODELS TO LINK YOUR PROGRAM ACTIVITIES WITH OUTCOMES INTRODUCTION At this point in your evaluation planning process, STD program staff and other stakeholders have a common understanding of the focus and priority areas of the program, formulated goals and objectives, described other program elements, and gotten familiar with what a logic model is. Now is the time for you to construct one or more logic models that will help you plan your evaluation. The flowchart below provides an illustration of where the program logic model fits in with the program planning and program evaluation process. UNDERSTANDING OF PROGRAM FOCUS AND PRIORITY AREAS LEARNING OBJECTIVES After completing this tool, you will be able to: Develop a logic model for your overall program. Develop a logic model for a program component/activity you are interested in evaluating. WHAT ARE THE CHALLENGES AND REWARDS OF THE LOGIC MODEL PROCESS? Developing a logic model may take time because it requires engaging stakeholders in the process, carrying out the actual steps of logic model construction, and adjusting the logic model periodically to reflect changes in programmatic efforts. Nonetheless, program staff and stakeholders can learn a great deal about the STD program or selected program activity, clarify how it is supposed to work, and create a sense of ownership. Also, having a comprehensive description of a program or an activity is an excellent aid in moving programs in a positive direction and deciding what to evaluate. A logic model is a good way to map the expected progress of your program, and focus the evaluation to determine what works, what does not work and how to improve it. WHAT TYPES OF LOGIC MODELS CAN YOU CONSTRUCT AND WHY? You can develop a logic model to illustrate the “big picture” of the entire STD program (i.e., a global logic model) or develop a logic model of a segment of the global logic model (i.e., nested logic model). For instance, Figure 1 shows a global logic model illustrating the general process and outcomes of State X STD Prevention Program. This logic model reflects the epidemiology of STDs in State X and is provided as an example, not a model of a global logic model for any STD program. It displays the connections between inputs, activities, outputs and outcomes. It presents the more immediate outcomes on the way to long­term outcomes, which will take years to attain, so that progress can be tracked and improvements along the way can be made by the STD program. State X STD Prevention Program developed and used that global logic model as a management tool and to monitor the implementation of the program plan. The logic model in Figure 2 shows a nested logic model developed by program staff and relevant stakeholders of State X STD program. Because it was not feasible to evaluate the entire program, the STD program staff decided to focus on the activity pertaining to Chlamydia screening of adolescent females who are admitted to the county’s juvenile detention centers (JDCs). In this logic model, they went into more detail regarding the connections from inputs to outcomes of that particular activity and how the latter is achieved. The logic model helped them determine the progress of the processes implemented, and identify additional actions that needed to take place for this activity to produce changes in the target population(s). Examples on how Figures 1 and 2 were constructed can be found in the Appendix. The Appendix will show you the connection between goals/objectives/ inputs/activities and logic model components. Figure 1: Logic Model for State X Comprehensive STD Prevention Systems (CSPS) Program INPUTS ACTIVITIES OUTPUTS SHORT­TERMOUTCOMES INTERMEDIATEOUTCOMES LONG­TERMOUTCOMES FundsCDC/DSTDP• Other federalsources• State sources• PrivateSourcesAssigned Staff• SHD/LHDCDCguidelines andrecommendationsTechnicalassistance andcollaboration• Federalagencies• State agencies• Local agencies• NGOs &affiliates; • Provide communityand individualbehavior changeinterventions onsyphilis.• Provide medical andlaboratory services.• Provide chlamydiascreening amongsexually activefemale adolescentsand young women.• Ensure syphilispartner services.• Promote leadershipand programmanagement.• Conduct surveillanceand datamanagement.• Provide or ensuretraining andprofessionaldevelopment.• Ensure adocumented STDoutbreak responseplan. Community/ Individual Behavior ChangeInterventions• Community and behavioral interventions onsyphilis implemented among at risk MSM.Medical and Lab Services• Lab/medical facilities and providers reportingtesting results.• Female admittees in juvenile detentionfacilities screened for Chlamydia.Partner Services• Syphilis cases’ partners identified.Leadership and Program Management• Strategic plan in place.• Program operation plan to monitor programactivities.• Appropriate program policies on professionaldevelopment in place.Surveillance and Data Management• Reported cases of P&S; syphilis andChlamydia sent to CDC within 30 to 60 daysfrom the date of specimen collection.Training and Professional Development• Staff training needs regularly assessed.• Training opportunities on syphilis andChlamydia provided and individuals trained.STD Outbreak Response Planning• Plan includes required elements. Increasedknowledge:• Consequences• Safe behaviors• Self assessmentof riskIncreased intentionto use condoms Increased safer sexbehaviors:• AbstinenceMutual monogamy• Fewerconcurrentpartners • ReducedSyphilisincidence • ReducedChlamydiaprevalence Figure 2: Logic Model of Chlamydia (Ct) Screening Program for Adolescent Females in County Z JuvenileDetention Centers (JDCs) INPUTS ACTIVITIES OUTPUTS SHORT­TERMINTERMEDIATELONG­TERMOUTCOMES OUTCOMES OUTCOMES JDCs Reduction of Ctprevalence amongadolescent femalesin JDCs. HOW DO YOU CONSTRUCT A LOGIC MODEL? Logic models play a critical role in evaluation planning, so it is important to take ample time to develop them. Here are some steps to help you do this in a systematic way. 1. Convene program staff and other stakeholders to develop a logic model. • Determine the purpose of developing the logic model (e.g., build consensus on how the program or a program activity works; determine if given your activities the desired results can be achieved; show your funder what you are doing; determine when and what to evaluate; and monitor operations). • Schedule sufficient time for your stakeholders to brainstorm ideas for creating the initial draft of the logic model. • You may want to reserve a room with a large wall or workspace where the group can write down their thoughts, physically display them on the wall, and organize them. • Review Tool 2.3 (Identify Elements of Your STD Program and Get Familiar with Logic Models) and discuss the benefits of logic modeling, and how all of you can use process and outcome objectives to develop the components of the logic model. • On a board or on pieces of paper taped to a wall, write the logic model components (i.e., inputs, activities, outputs, short­term outcomes, intermediate outcomes, and long­term outcomes). The group can organize their thoughts on the board, or on sheets of paper, or on post­it notes under each component. 2. Decide which direction you want to work on the model. If you are working with an existing program with activities already in place and are planning an evaluation of one of the activities, it may make more sense to start with the process components of the logic model (i.e., the inputs, activities, and outputs), which are on the left side of the model, then work toward the long­term outcomes (see Figure 3). This is called the “If­then” approach. Figure 3. Model for Existing Program with Activities in Place IF......THEN IF......THEN IF......THEN IF......THEN IF......THEN That is: If you invest time and money (e.g., inputs) to operate your program, then you can develop a syphilis educational plan (e.g., activities). If there is a plan, then the target population can receive information on syphilis and prevention messages,(e.g., outputs). If the target population receives information on syphilis and its prevention, then its knowledge of syphilis symptoms, transmission and prevention will increase (e.g., short­term outcome). If the target population’s knowledge of syphilis symptoms, transmission and prevention increases, then its health­seeking behavior will increase (e.g., intermediate outcome). Ultimately, if these changes take place in the target population, then the incidence and prevalence of syphilis will be reduced (e.g., long­term outcomes). Remember, when working from left to right, first generate inputs, activities, and outputs; then develop the outcomes. Figure 4 depicts a pictorial example of the “If then” flow. Figure 4: Example of Logic Model Flow on a Syphilis Outbreak Response Project Targeting MSM in City Y INPUTS ACTIVITIES OUTPUTS • Meetings conducted with gay newspaper • 1 full­time, • Press releases written and sent to media outlets 2 part­time staff • Posters created and distributed to businesses • CDC/DSTDP (gay bars and bath houses) funds • Bus billboards developed and posted • Printing • Article written for newsletter and distributed equipment to health providers • Condoms • Training Materials • Meetings conducted with CBOs • HD funds distributed to CBOs supplies • Screening • Training conducted with CBO staff • Educational pamphlets developed and • CBO partners distributed to MSM • Laboratory • CBO meeting with MSM leaders held medication • Appropriate • MSM and partners screened • Condoms distributed to MSM, partners, clinics and gay bars On the other hand, if you are planning a new initiative for which you have determined its end result or goal, it may be easier to work from right to left, and to generate your outcomes and corresponding outputs, activities and inputs (see Figure 5). You would start with your goal(s) and move backward. Figure 5. Model to Plan a New Activity or Initiative For example, if you are planning to conduct syphilis screening in non­traditional venues the logic model will help you plan this initiative by linking outcomes with outputs. If your long­term outcome is reduced incidence of syphilis in a specific population, you will need to ask the question, ‘What intermediate outcome would I expect before that?’. If you decide on the intermediate outcome of increased health­seeking behavior among target population members, you would ask yourself what short­term outcomes you would expect before this. The same questioning would take place to then develop your short­term outcomes, outputs, activities and, finally, your inputs that would be needed to support the new activity. You work from right to left, by first stating your goal and then developing the outcomes before working toward the inputs. Note that when developing new activities some programs have found it beneficial to first work with their stakeholders to develop the backward logic model and then develop their ‘SMART’ objectives. They found that understanding the ‘logic’ of a new activity helped them design realistic and achievable objectives. 3. Identify elements for each logic model component, and link them. • As you brainstorm about the logic model components, write everyone’s ideas on the sheets of paper, or post­it notes, or board under each component (see Figure 6). Reviewing your objectives can help you identify some of the elements (see Tool 2.3). • After you have generated a list for each of the logic model components, place them in chronological order from top to bottom, if possible. For example, certain activities may need to be completed before other activities can occur. • Then, working across, make sure that each activity is linked to corresponding outputs, and outputs are linked to corresponding outcomes. You may need to cut up or re­write your lists to make this work. 4. Periodically revise the logic model, as necessary. As your program or program activity changes, you will also need to change the logic model to include new or revised activities. For example when STD trends change in your project area, your areas of special emphasis and priority activities of your program may change. Therefore, you have to modify your logic model accordingly. In addition, if you are implementing a relatively new activity, you may realize that the related outputs and outcomes are not realistic based on your resources; consequently the logic model should be revised to reflect this. SUMMARY CHECKLIST: Things to consider when you develop logic models for your program and activities. CONCLUSION AND NEXT STEPS Having completed this program evaluation tool, you are aware of the importance of logic models for program planning and evaluation and able to construct one along with your program stakeholders. Now that you have engaged your stakeholders and fully described your program, your next step will be to focus your evaluation and decide which components of your logic model you will evaluate. As you proceed to Step 3, you will begin the process of tailoring your evaluation to the needs of your program and stakeholders, as well as determining what resources and personnel are necessary for your evaluation. ACRONYMS USED IN THIS TOOL CBO – Community­based organization CDC – Centers for Disease Control and Prevention CLIA – Clinical Laboratory Improvement Amendments CPG – Community planning group CSPS – Comprehensive STD Prevention Systems Ct — Chlamydia DSTDP – Division of STD Prevention JDC – Juvenile detention center LHD – Local health department MSM – Men who have sex with other men NGO – Non­governmental organization P&S; syphilis – Primary and secondary syphilis PTC – Prevention training center SHD – State health department STD – Sexually transmitted disease YRBS – Youth Risk Behavior Survey KEY TERMS Activities: Actual events that take place as part of your STD program (e.g., develop pamphlet; screen patients). Global logic model: A type of logic model which depicts how an entire program operates. Goal: A broad statement expressing the purpose of your STD program that states what our program will accomplish (the desired result). Inputs: Your STD program’s resources (e.g., money, staff, materials). Intermediate outcomes: Intended effects of the program in the target population (e.g., changes in STD­related policy or in behavior of the target population). Logic model: A picture of how your STD program is supposed to work. Long­term­outcomes: Intended effects of the program in the target population that may take several years to achieve, such as reduced disease transmission and incidence. Nested logic model: A type of logic model, which depicts a component of a global logic model and describes the component in more detail than the global model. Outcome objectives: Measurable statements specifying the intended effect of your program in the target population(s) or end result of your STD program. Outcomes: The intended effects or changes in the target population(s) that result from your STD program. Outputs: The direct products of program activities or services delivered (e.g., seminar conducted, infected clients treated ). Process Objectives: Measurable statements describing your STD program activities and actions involved in their implementation. Short­term outcomes: Immediate effects of the program in the target population, such as changes in knowledge, attitudes, skills, awareness, or beliefs of the target population. EXERCISE Constructing a Program Component Logic Model Following is a hands­on exercise to construct a logic model for the State X STD Program’s training and professional development component. Everything you need to construct the logic model is provided, including a description of the component and its goals, inputs, activities, and outputs, along with its process objectives and short­term, intermediate, and long­term outcome objectives. Also included are worksheets for developing each component of the logic model and for making linkages. You may find these worksheets helpful in developing your own logic models. Finally, a logic model template for the finalized model is provided [see Figure 6]. [Please note: You will find the actual logic model at the end of this tool, Figure 7.] Description of State X Leadership and Professional Development Program Component Goals • Improve the quality of STD Prevention services in State X. • Reduce the prevalence and incidence of STDs in State X. Inputs • CDC/DSTDP funds. • Staff: STD program training coordinator and syphilis elimination team trainer. • Partner: Area prevention training center (PTC). • Materials: Training curricula and training materials. Activities • The STD training coordinator will conduct a training needs assessment of STD prevention and control staff and share the results with the area PTC. • The STD training coordinator, in collaboration with the STD program manager, will develop a schedule for STD staff to attend pertinent PTC trainings. • STD clinic and disease intervention staff will attend pertinent trainings conducted by the PTC. 100 Process Objectives • By (month/year), the STD training coordinator will conduct a training needs assessment of STD prevention and control staff in which he/she will interview (#) field supervisors about staffs’ skill and knowledge deficiencies. • By (month/year), the STD training coordinator will share the results of the training needs assessment with area PTC. • By (month/year), the STD training coordinator, in collaboration with the STD program manager, will develop a schedule for (#) STD clinical staff and (#) disease intervention staff to attend pertinent PTC trainings. • By (month/year), (#) clinic staff will attend one STD intensive training conducted by the PTC. • By (month/year), (#) disease intervention staff will attend one training on effective STD interviewing and case management techniques. Short­Term Outcome Objectives • By (month, year), (X%) of STD program supervisors will report that there has been an increased number of professional development (PD) opportunities provided to STD staff. • By (month/year), X% of clinical staff who attend PTC STD intensive training will report in a pre/post instrument administered at the training, an increase in their knowledge of the signs and symptoms of syphilis from (X%) to (Y%). • By (month/year), an increase from X% to Y% of clinical staff who attend PTC STD intensive training will report a score of over 80% in a pre/post instrument on knowledge of Chlamydia screening protocols administered at the training. • By (month/year), (X%) of disease intervention staff who attend PTC training will demonstrate, in a performance exercise, an increase in their interview skills recorded in a skills performance sheet. • By (month/year), (X%) of disease intervention staff who attend PTC training will demonstrate, in a performance exercise, an increase in their case management skills recorded in a skills performance sheet. Intermediate Outcome Objectives • By (month/year), X% of STD supervisors will report that the percentage of clinical staff who adhere to Chlamydia screening protocols has increased from (X%) to (Y%). • By (month/year), X% of STD supervisors will report that the percentage of disease intervention staff who adhere to syphilis interview protocols has increased from (X%) to (Y%). • By (month/year), X% of STD supervisors will report that the percentage of disease intervention staff who adhere to case management protocols has increased from (X%) to (Y%). Long­Term Outcome Objectives • By (month/year), STD clinics with an ‘unacceptable’ quality of services score will increase their quality score by X%. • By (month/year), all STD clinics with an ‘acceptable’ quality of services score will maintain or increase their quality score. • By (month, year), the prevalence of Chlamydia in State X will have been reduced from (X%) to (Y%). • By (month, year), the incidence rate of syphilis in State X will have been reduced from (X%) to (Y%). 102 REFERENCES Centers for Disease Control and Prevention. (n.d.). Oral health infrastructure development tools, Step 2B: Logic models. Retrieved October 17, 2004, from http://www.cdc.gov/oralhealth/library/pdf/logic_models.pdf Harvard Family Research Project. (1999). Learnings from logic models: An example of a family/school partnership program. Retrieved October 17, 2004 from http://gseweb.harvard.edu/hfrp/pubs/onlinepubs/rrb/learning.html MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001, November). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta: Centers for Disease Control and Prevention. Retrieved October 17, 2004, from http://www.cdc.gov/tobacco/evaluation_manual/Evaluation.pdf Poister, T. (2003). Measuring performance in public and nonprofit organizations. San Francisco: Jossey­Bass. United Way of America (1996). Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way. Retrieved October 17, 2004, from http://national.unitedway.org/outcomes/library/pgmomres.cfm W.K. Kellogg Foundation (2001, December). Logic model development guide. Retrieved October 17, 2004, from http://www.wkkf.org/Programming/ResourceOverview.aspx?CID= 281&ID;=3669 Worksheet for Logic Model Component: INPUTS Definition: Inputs are the resources that go into a program (e.g., money, staff, materials). List inputs here: 104 DESCRIBE THE PROGRAM Worksheet for Logic Model Component: ACTIVITIES Definition: Activities are the actual events that take place as part of your program List activities here: Worksheet for Logic Model Component: OUTPUTS Definition: Outputs are the direct products of your program activities; in other words what was produced or delivered as a result of the activities (e.g., seminar conducted, infected clients treated). Tip: Use your program’s process objectives to inform how you write your outputs. Try to link each item in the activity component of your logic model to one or more outputs. ACTIVITIES OUTPUTS 106 Worksheet for Logic Model Component: SHORT­TERM OUTCOMES Definition: Short­term outcomes are the immediate effects of the program in the target population(s), such as changes in knowledge, attitudes, skills, awareness, or beliefs. Tip: Use your program’s short­term outcome objectives to inform how you write your short­term outcomes. Try to link each item in the outputs component of your logic model to one or more short­term outcomes. OUTPUTS SHORT­TERM OUTCOMES Worksheet for Logic Model Component: INTERMEDIATE OUTCOMES Definition: Intermediate outcomes are the effects of the program in the target population that take longer to occur than short­term outcomes, such as changes in policy or in behavior. Tip: Use your program’s intermediate outcome objectives to inform how you write your intermediate outcomes.Try to link each item in the short­term outcome component of your logic model to one or more intermediate outcomes. SHORT­TERM OUTCOMES INTERMEDIATE OUTCOMES 108 Worksheet for Logic Model Component: LONG­TERM OUTCOMES Definition: Long­term outcomes are program effects on the target population(s) that may take several years to achieve. Tip: Use your program’s long­term outcome objectives to inform how you write your long­term outcomes. Try to link each item in the intermediate outcome component of your logic model to one or more long­term outcomes. INTERMEDIATE OUTCOMES LONG­TERM OUTCOMES Figure 6: Logic Model Template for State X Leadership and Professional Development Program Component SHORT­TERMINTERMEDIATELONG­TERM INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES 110 Figure 7: Logic Model for State X Training and Professional Development (PD) Program Component INPUTS ACTIVITIES OUTPUTS SHORT­TERMINTERMEDIATELONG­TERMOUTCOMES OUTCOMES OUTCOMES Increased number of PD opportunities pertinent to the work of STD staff. • Reduced prevalenceof Chlamydia • Reduced incidenceof syphilis APPENDIX WHAT IS A GLOBAL LOGIC MODEL FOR AN STD PROGRAM? A CASE SCENARIO The STD program in State Z applied for funding from CDC/DSTDP to support its Comprehensive STD Prevention Systems (CSPS) program. The application outlined the performance goals, main objectives, and other elements of the program. This information was then used to develop an overall logic model, which provided a framework for program design and expected results as well as a reference for guiding the evaluation. Relevant information on State Z STD Program’s goals, inputs, and activities; and its process objectives and short­term, intermediate, and long­term outcome objectives is listed below. See Figure 1 for the logic model that was developed by program staff and stakeholders. This shows how the list of program information was used to construct the logic model. [Note that the objectives included are not exhaustive, but rather a sample to show you how to connect objectives and logic model components.] GOALS, INPUTS, ACTIVITIES, AND OBJECTIVES Goals • Reduce STD rates by providing Chlamydia and gonorrhea screening, treatment, and partner management to women in publicly funded family planning and STD clinics. • Reduce the incidence of primary and secondary syphilis. • Reduce the incidence of congenital syphilis. Inputs • Funding from CDC/DSTDP and other federal, state, and local sources. • State health department staff. • CDC guidelines and recommendations. • Partners providing technical assistance and collaboration, including federal, state, and local agencies; community­based organizations (CBOs) and their affiliates; city HIV community planning group (CPG). 112 Activities1 • Provide community and individual behavior change interventions. • Provide medical and laboratory services. • Ensure partner services. • Promote leadership and program management. • Conduct surveillance and data management. • Provide or ensure training and professional development. • Ensure a documented STD outbreak response plan. Process Objectives2 • Community/Behavior Change Interventions By (month/year), (#) community/behavior change interventions will be conducted which serve MSM at high risk for syphilis in State Z. • Medical and Laboratory Services By (month/year), all on­site STD clinic laboratories will use Clinical Laboratory Improvement Amendments (CLIA) standards of testing in State Z. By (month/year), (X%) of female admittees to X juvenile detention facilities in State Z will have been tested for Chlamydia. • Partner Services By (month/year), at least one partner of (X%) of syphilis cases will have been initiated for follow­up by the STD program. • Leadership and Program Management By (month/date), the State Z STD Program will have implemented a strategic plan to guide their operations. By (month/year), the STD Program in State Z will have implemented an operation plan to monitor program activities. By (month, year), the STD Program in State Z will have implemented appropriate policies on professional development. 1 Required activities for funding of CSPS programs as stated in PA 04232. 2 Adapted from STD program objectives in the DSTDP Program Operations Guidelines for STD Prevention and the DSTDP 2005 performance measures. • Surveillance and Data Management By (month/year), the percentage of reported cases of P&S; syphilis sent to CDC via NETSS within 30 days from the date of specimen collection will have increased from X% to Y% in State Z. By (month/year), the percentage of reported cases of Chlamydia sent to CDC via NETSS within 30 days from the date of specimen collection will have increased from X% to Y% in State Z. • Training and Professional Development By (month/year), STD Program in State Z will have system in place for the regular assessment of staff training needs. By (month, year), (#) professional development events will have been provided to (#) program staff on syphilis prevention, screening, and treatment in State Z. By (month, year), (#) professional development events will have been provided to (#) program staff on Chlamydia prevention, screening, and treatment in State Z. • STD Outbreak Response Planning By (month/year), STD Program in State Z will have an outbreak response plan with required elements which will be updated every 6 months. Short­Term Outcome Objectives (to be accomplished in 1 to 3 years) • By (month/year), (X%) of patients in STD clinics in State Z will report, on a post­screening survey, an increase in their knowledge of STD consequences, safe behaviors and self­assessment of risk. • By (month/year), (X%) of patients in STD clinics in State Z will report, on a post­screening survey, an increase in their intention to use condoms with their sexual partners. 114 Intermediate Outcome Objectives3 (to be accomplished in 3 to 5 years) • The percentage of adolescents in grades 9 through 12 who report on the Youth Risk Behavior Survey (YRBS) that they abstain from sex will increase from (X%) in (month/year) to (Y%) in (month/year). • The percentage of patients in STD clinics in State Z who report, on a post­screening survey, that they are in mutually monogamous relationships will increase from (X%) in (month/year) to (Y%) in (month/year). • By (month/year), (X%) of patients in STD clinics in State Z will report, on a post­screening survey, a decrease in number of concurrent sexual partners from the previous year. Long­Term Outcome Objectives (to be accomplished in 5 or more years) • Syphilis incidence in State Z will have decreased by at least (X%) from (month/year) to (month/year). • Chlamydia prevalence rates in State Z will have decreased by at least (X%) from (month/year) to (month/year). Adapted from Poister, TH. (2003). Measuring Performance in Public and Nonprofit Organizations; pp 178. Jossey­Bass; San Francisco, CA. WHAT IS A NESTED LOGIC MODEL AND HOW DOES IT FIT INTO THE OVERALL LOGIC MODEL? A CASE SCENARIO The STD program in County Z has received funding from the State’s STD program and other sources to support an activity within its medical and laboratory component. The program activity is to conduct Chlamydia screening among adolescent females who are admitted to the county’s juvenile detention centers (JDCs). In developing the design for the activity, the STD program staff identified the goal of the initiative, the inputs available, the specific activities that needed to be implemented, and the activity’s process objectives and short­term, intermediate, and long­term outcome objectives. This information is provided below. See Figure 2 for the logic model that program staff and relevant stakeholders developed. It shows how the program staff and stakeholders used the listed information to construct the logic model. [Note that the objectives included are not exhaustive, but rather a sample to show you how to connect objectives and logic model components.] GOAL, INPUTS, ACTIVITIES, AND OBJECTIVES Goal • Reduce Chlamydia rates among adolescent females incarcerated in juvenile detention centers in County Z. Inputs • Funds. Funds from CDC/DSTDP, state and local sources, and private sources have been made available. • JDC Staff. Administrators, medical providers, and administrative staff at County Z JDCs will implement the activity. • Partners. Several partners will lend technical direction, assistance, and collaborative energy to the activity. This includes the technical direction of State and County Z STD program staff and health department laboratory technicians who will process specimens. It also includes the organization that has been contracted to train the JDC medical staff in the protocols of Chlamydia screening and treatment, as well as several community stakeholders who will serve on an advisory committee for the project. 116 • Materials. Several types of materials will be used including CDC protocols/guidelines for Chlamydia screening and treatment, prevention educational materials, test kits, and treatment medication. Activities • Professional Development. In order to adequately prepare JDC medical providers to implement the screening activity, contracted organization (X) will train the providers in Chlamydia screening and treatment. • JDC Support. In order for JDC staff to be appropriately involved in the development and implementation of the initiative, the JDCs will authorize the participation of medical staff and others for the activity. • Serving the Target Population. All sexually active female adolescent admittees to JDCs will be offered Chlamydia screening, counseling, and treatment if appropriate. Process Objectives • By (month/year), contracted organization (X) will conduct (#) professional development events with (#) medical providers in JDCs in County Z on appropriate Chlamydia screening, counseling, and treatment with adolescent females. • By (month/year), the administrators of County Z JDCs will authorize that (#) JDC staff be allocated (#) hours of their work week to develop and implement the Chlamydia screening initiative. • By (month/year), (Y%) sexually active female adolescents who are admitted to JDCs in County Z will have received Chlamydia screening and counseling. Short­Term Outcome Objectives (to be accomplished in 1 to 3 years) • By (month/year), the medical staff from County Z JDCs who participated in the training will demonstrate, in a pre/post questionnaire, an increase in their knowledge of the Chlamydia screening, counseling and treatment protocols from (X%) to (Y%). • By (month/year), medical staff from County Z JDCs who participated in the training will demonstrate, in a performance exercise, an increase in their skills in conducting Chlamydia screening, counseling and treatment from (X%) to (Y%), as recorded in a skills performance sheet. • By (month/year), the percentage of JDCs in County Z that conduct Chlamydia screening, treatment, and counseling among sexually active adolescent females as a regular part of the admission protocol will increase from (X%) to (Y%). • By (month/year), (X%) of sexually active adolescent females in JDCs in County Z who receive Chlamydia screening will report, in a post­screening questionnaire, an increase in their awareness of how Chlamydia and other STDs are transmitted and prevented. • By (month/year) (X%) of adolescent females in JDCs in County Z who receive Chlamydia screening will report, in a post­screening questionnaire, an increase in their intention to use condoms. Intermediate Outcome Objectives (to be accomplished in 3 to 5 years) • By (month/year), (X%) of supervisors of JDC medical units in County Z will report an increase in clinician’s adherence to Chlamydia clinical practice guidelines. • By (month/year), (X%) of sexually active adolescent females in JDCs in County Z who receive Chlamydia screening will report, in a post­screening questionnaire, a decrease in their number of sexual partners. • By (month/year), (X%) of sexually active adolescent females in JDCs in County Z who receive Chlamydia screening will report, in a post­screening questionnaire, an increase in their health­seeking behavior. Long­Term Outcome Objective (to be accomplished in 5 or more years) • By (month/year) Chlamydia prevalence among sexually active adolescent females will be reduced from (X%) to (Y%) in JDCs in County Z. 118