Cameron Willis, PhD; Alison Kernoghan, MPH; Barbara Riley, PhD; Janice Popp, MSW; Allan Best, PhD; H. Brinton Milward, PhD

Suggested citation for this article: Willis C, Kernoghan A, Riley B, Popp J, Best A, Milward HB. Outcomes of Interorganizational Networks in Canada for Chronic Disease Prevention: Insights From a Concept Mapping Study, 2015. Prev Chronic Dis 2015;12:150297. DOI: http://dx.doi.org/10.5888/pcd12.150297.

PEER REVIEWED

Abstract

Introduction

We conducted a mixed methods study from June 2014 to March 2015 to assess the perspectives of stakeholders in networks that adopt a population approach for chronic disease prevention (CDP). The purpose of the study was to identify important and feasible outcome measures for monitoring network performance.

Methods

Participants from CDP networks in Canada completed an online concept mapping exercise, which was followed by interviews with network stakeholders to further understand the findings.

Results

Nine concepts were considered important outcomes of CDP networks: enhanced learning, improved use of resources, enhanced or increased relationships, improved collaborative action, network cohesion, improved system outcomes, improved population health outcomes, improved practice and policy planning, and improved intersectoral engagement. Three themes emerged from participant interviews related to measurement of the identified concepts: the methodological difficulties in measuring network outcomes, the dynamic nature of network evolution and function and implications for outcome assessment, and the challenge of measuring multisectoral engagement in CDP networks.

Conclusion

Results from this study provide initial insights into concepts that can be used to describe the outcomes of networks for CDP and may offer foundations for strengthening network outcome-monitoring strategies and methodologies.

Top

Introduction

Interorganizational networks are groups of 3 or more organizations acting together in the pursuit of a shared vision or goal (1). Such networks are important elements of many public health efforts, including population-based chronic disease prevention (CDP) initiatives (2–5). In Canada, government and nongovernment organizations support multiple networks that aim to leverage the expertise, resources, and reach of various agents into lasting population health improvements. However, despite enthusiasm for interorganizational networks, uncertainty remains about “whether and under what conditions networks are actually performing at a level that justifies the costs of collaboration” (6). Although numerous network evaluation frameworks exist (7–10), much empirical work has focused on network structures and processes, with comparatively little research on network outcomes (1–3,11,12). The network outcomes field may be in urgent need of “reliable and valid measures of success at a network level” (9).

A focus on population-based prevention brings a unique set of contexts and challenges to the evaluation of network outcomes, such as developing realistic expectations for outcomes given the dispersed and often delayed effects of prevention activities, the difficult-to-define recipients of population health prevention efforts, and the tendency to attribute outcomes to specific organizations (3). Given these contextual factors, this exploratory project aimed to propose potentially important and feasible outcome concepts that may be useful for further enquiry as outcome monitoring strategies are developed for CDP networks.

Top

Methods

This study, conducted from June 2014 to March 2015, involved 2 phases: 1) concept mapping to identify outcome concepts relevant to CDP networks, and 2) key informant interviews to understand findings in light of contemporary network practice. This study received ethics clearance through a University of Waterloo Research Ethics Committee.

Concept mapping integrates qualitative processes (brainstorming and sorting) with quantitative multivariate statistical analysis, allowing groups to visually represent their ideas (13). We designed a study that used 4 steps:

Step 1. Participant recruitment. Thirty-three networks were nominated at a planning meeting (14) by attendees with expertise and involvement in interorganizational CDP networks. Eighteen of the 33 networks met the following inclusion criteria: they operated in Canada; they involved participants who had research, policy, and practice perspectives; and they focused on CDP population health interventions. The remaining 15 who did not meet these criteria were excluded. Each of the 18 networks included focused on 1 of 4 areas: tobacco control, physical activity, healthy eating and nutrition, or community well-being. The networks also focused on pan-Canadian, provincial, or community-based populations and were funded through a combination of support from the federal government (n = 3), provincial governments (n = 14), and nongovernment organizations (n = 2). Across the networks, sectoral involvement varied widely; each participating network involved different combinations of local public health units; community-based organizations; health care providers; government health, transport, urban planning, and education departments; charitable sectors; academic and research organizations; and individual community members. More than half of individual participants were involved in a combination of research, policy, and practice. The age of participating networks ranged from 4 to 16 years. All members of each participating network received an invitation to participate in Step 2.

Step 2. Online brainstorming. Thirty-two network members completed the brainstorming exercise; 16 reported being a member of their network for more than 2 years (Table 1). By using the Concept Systems Global Max Web-based application (www.conceptsystemsglobal.com) network members provided an unlimited number of responses to the focus prompt, “A meaningful measure of network outcomes is . . .” Data were collected from July 15, 2014, through August 26, 2014. Two authors (C.W. and A.K.) removed duplicate or similar statements, separated statements into single ideas, and clarified language, resulting in a final list of 82 statements that were included in Step 3. During the statement refinement process, only a few statements were removed to retain the breadth and scope of participants’ contributions. Differences of opinion between the authors were resolved by discussion.

Step 3. Online sorting and rating. Leaders from 11 networks, who were responsible for managing or coordinating network activity and who were not involved in the brainstorming activity completed the sorting and rating task. They sorted the 82 statements into discrete groupings based on similar concepts, rated the importance and the feasibility of each statement on a 5-point scale (1 = least important or least feasible; 5 = most important or most feasible), and assigned a name to each cluster. The authors discussed and reached consensus on the final cluster names.

Step 4. Analysis. Standard concept mapping procedures were used to analyze the relationships among statements and to generate maps (13). Statements were plotted as separate points and clustered into groups on the basis of the frequency with which they were sorted together (further methodological details are available elsewhere [13]). A stress value, which reflects the goodness-of-fit for the collected data (measures between 0.205 and 0.365 were considered acceptable), was also calculated (13). We then undertook an iterative process to identify a configuration with an interpretable set of clusters that also preserved the most detail (13). The team examined successively lower cluster solutions beginning at 13 to determine when a merger of clusters was no longer substantively reasonable. The relationship between ratings of importance and feasibility was explored through pattern-matching and calculation of a correlation coefficient (13).

Interviews with key stakeholders from CDP networks explored the consistency of concept mapping results with stakeholder experiences and implications of study findings. Ten key informants from research (n = 1), policy (n = 1), and combined settings (n = 8) were interviewed; length of employment in their current position ranged from 2 to 12 years (mean, 6 years). Interviewees were not involved in the brainstorming or sorting and rating activities. Qualitative data were analyzed using NVivo 10 (QSR International Pty Ltd) with open coding followed by inductive refinement of themes (15). Illustrative quotes from interviews were selected.

Top

Results

Concept mapping

Individual clusters and regions

A 9-cluster solution was determined to be the most suitable fit for this study’s data (stress value 0.280) (Figure 1). The 9 clusters were enhanced learning, improved use of resources, enhanced or increased relationships, improved collaborative action, network cohesion, improved system outcomes, improved population health outcomes, improved practice and policy planning, and improved intersectoral engagement ( Table 2).Individual clusters were grouped into 3 regions: intermediate outcomes, bridging cluster, and long-term outcomes (Figure 1). The 5 clusters along the bottom of the map were considered intermediate outcomes of network performance. Population-level impacts, representing longer-term outcomes, were captured by the 3 clusters in the upper portion. The central cluster, “improved intersectoral engagement,” contains statements that highlight the importance of the coordinated involvement of multiple sectors in CDP networks. This cluster is centrally located because the statements it contains were frequently sorted with statements from all other clusters.

Figure 1. Cluster map showing regions (intermediate, bridging, and long-term outcomes).  [A text version of this figure is also available.]

Importance and feasibility ratings

Table 2 and Figure 2 summarize the average importance rating and feasibility rating for each cluster and describe the correlation between these ratings. Participants rated improved use of resources as the most important and most feasible outcome concept to measure and improved intersectoral engagement as the least important and least feasible measure of network outcomes. Figure 2 also demonstrates a moderately strong relationship between importance and feasibility ratings across clusters, with an overall high correlational value (r = 0.74).

Figure 2. Pattern match of importance (left axis) and feasibility (right axis) demonstrating the relationship in average ratings for both characteristics across clusters. [A text version of this figure is also available.]

Stakeholder interviews

Concept maps and ratings

All interviewees agreed that the concept maps were comprehensive and that all outcome clusters and regions were relevant to their experience of networks. Two participants did not consider the clusters Network Cohesion and Enhanced or Increased Relationships as network outcomes. For example, network cohesion “wasn’t something we were striving for; it wasn’t an intended outcome, but a by-product of it.” Half of participants considered the region of long-term outcomes as the ultimate outcomes of CDP networks, and some suggested the intermediate clusters were steps in the process of achieving long-term impact.

Half of those interviewed expressed strong agreement with the importance ratings. Others questioned the relatively low importance ratings for the long-term outcome region and improved intersectoral engagement cluster, suggesting that these elements were critical for “. . . making sure that what we’re doing has an impact on the groups we’re working with.” Although half of the interviewees agreed with the feasibility ratings, opinions among the remaining 5 interviewees were highly variable, with inconsistent suggestions for higher and lower ratings for both the long-term outcomes and the improved intersectoral engagement cluster.

Key themes

Three themes emerged from analyses of stakeholder interviews: 1) the methodological challenges in measuring network outcomes, 2) the dynamic nature of network evolution and function and implications for outcome assessment, and 3) the importance of measures to describe intersectoral engagement in the context of CDP networks.

Theme 1: Methodological challenges in measuring and attributing network outcomes. Multiple interviewees referred to the difficulties in measuring network outcomes caused by multiple contributing influences (ie, political factors, available funding, government priorities). Although it can be challenging to attribute network activity to outcomes at any point along the development of a network, it may be particularly difficult when assessing long-term outcomes. As noted by one stakeholder, “longer-term outcomes are harder. That’s a practical reason why we focus on the intermediate outcomes. Those are outcomes that are within our control. . . . I can work towards improved practice and planning and overall improved system outcomes but there are other decision makers involved there.” Yet some suggested that the population impact of networks was not only important to capture but also feasible. As one research-intensive interviewee noted, “When planning is done properly at the outset of an activity of the group, the long-term outcomes should be measureable.”

Interviewees also highlighted the difficulties in meeting the needs of multiple stakeholders involved in CDP networks, which by design are often multilevel and multidisciplinary. As a result, a range of individuals, organizations, and sectors operate within and around networks (eg, network leaders, members, funders, recipients), each with his or her own assumptions, expectations, and requirements of what constitutes an important and feasible outcome measure. As noted by one stakeholder, “you work in . . . an environment where organizations have different visions, different mandates, so it’s not always easy to get to common ground.” As such, the emphasis may be less on finding common ground than it is on developing a mutual understanding of needs and perspectives as they relate to network outcomes.

Theme 2: The dynamic nature of network evolution and function and implications for outcome assessment. Multiple interviewees suggested the identified clusters of network outcomes may take on different significance and definitions as networks evolve. Along the pathway toward high-performing CDP networks, “Intermediate outcomes are foundational in order to foster and nurture an environment that provides for upward movement long term.” Although networks in the early stages of development may interpret improved use of resources as relating primarily to reducing duplication, more mature networks may consider this cluster to encompass how network members are using resources for “working together more cohesively along a spectrum.” Similarly, in newly forming networks where “no one wants to give up their turf,” initial resistance to more integrated ways of working may change over time as networks foster trust and reciprocity among members.

These findings reinforce the need to link outcome measures to the stage of network development and to invest in outcome measures that realistically reflect the functions of CDP networks. For example, for CDP networks it may be particularly difficult to directly and immediately “. . . link our work to population health outcomes.” Therefore, alternative markers of network function become important, such as a network’s capacity to improve the practice and policy-planning activities of partnering organizations. Although some consider such outcomes to be long-term markers of performance, more mature networks may consider this an intermediate performance marker that helps to create the conditions for achieving population health impact. Therefore, flexibility may be required when interpreting and applying the outcome concepts identified in this study.

Theme 3: The role of intersectoral engagement for CDP networks. Intersectoral engagement is the coordinated participation and commitment of organizations from multiple sectors and is considered by many stakeholders to be “the foundation for everything to work” in interorganizational CDP networks. Interviewees noted the central position of intersectoral engagement with “all those other outcomes feed[ing] into it.” One example is the link between intersectoral engagement and improved use of resources. This link combines the natural focus of CDP networks on multilevel, multi-organizational participation with the utility networks offer in attracting and reaching new partners and their resources, particularly as organizations are encouraged to “do more with less.”

As a bridging or linking concept, measuring intersectoral engagement was considered an important step in evaluating a network’s ability to attract and retain “a multitude of sectorial players that would bring a great deal of leverage,” whose collective efforts are critical to a network’s “overall effectiveness long term.” Despite its potential importance, challenges remain in how best to capture and report intersectoral engagement. As noted by interviewees, the most useful picture of intersectoral engagement may arise from a combination of concepts, consisting of improved resource use, enhanced relationships, and increased network cohesion.

Top

Discussion

This study aimed to provide preliminary insights into the importance and feasibility of outcomes for measuring the performance of population-based CDP networks. Findings from this study both reinforce and extend existing knowledge of these network outcomes and their measurement. Results from this work are relevant to the design and implementation of prevention-oriented network evaluation strategies and help to surface potential domains for measuring short-term, medium-term, and long-term network outcomes for various audiences (eg, network participants, network managers, network funders). Application of this study’s findings to practical network evaluation will require tailoring through group-based item-refinement procedures.

Consistent with existing literature (7–9,16-19), this analysis identified the importance of networks for improving resource use, strengthening interorganizational relationships and cohesion, and enhancing population health status. Although existing frameworks do not explicitly include these concepts, each correlates with previously described outcome domains: network growth, reduced duplication of activities, greater integration across agencies, improved access to available resources, reduced conflict among partners, greater organizational commitment among partners, and reductions in the incidence of a particular problem. The concepts of enhanced learning and intersectoral engagement identified in our analysis are also consistent with findings from other studies (10,20). Additionally, current results highlight the temporal nature of network evolution and the need to match outcomes to the stage of network development, a finding that is consistent with existing literature related to network development and evolution (21,22), collective impact (23), and system change theories (24).

This study also offers new insights into which types of outcomes may be most important to measure for CDP networks that have a population health focus. For example, participants identified a network’s ability to influence policy and planning and contribute to the outcomes of systems promoting CDP activities as important measures of network outcomes. Previous studies of network performance frameworks did not explicitly identify these types of outcomes. This difference may relate to the composition and intended functions of CDP networks, which, unlike many service delivery networks (25,26), focus on contributing to and supporting coordinated research, policy, and practice as a means to improving population health outcomes (27,28). In these settings, more upstream measures of impact, such as improved policy and planning actions, may be expected.

There are also numerous elements included in existing frameworks that were not identified by our study, including outcomes at the community, network, and organization levels (7–10,29). The limited specificity of our results to particular outcome levels may be due to the study design, which did not attempt to identify or classify outcomes based on their level of relevance or use. An alternative focus prompt referring to community, network, or organization levels likely would have increased the specificity of statements and clusters to particular outcome levels. Future work may explore how the identified clusters can be adapted to particular levels of interest.

Variation in ratings of importance and feasibility between concept mapping participants and interviewees suggests that insights may vary depending on network roles and responsibilities. For example, concept mapping participants rated long-term network outcomes and intersectoral engagement as low on importance and feasibility, whereas interviewees rated both concepts highly. The concept mapping brainstorming participants were largely direct network members, whereas key informant interviewees also included people with broader management, policy, and research orientations for whom long-term outcomes and intersectoral collaboration may have been particularly important. Future studies that involve more participants may yield insights into how these different perspectives influence ratings of importance and feasibility.

The online concept mapping approach used in this study engaged network members from diverse geographical areas. Moreover, key informant interviews enabled in-depth exploration of concept mapping results, including preliminary validation of outcome concepts. This methodology is potentially well-suited to broader application, such as across international jurisdictions where the work of multinational networks may be particularly salient.

A limitation of this study was the relatively small number of participants. A larger group of participants drawn from different networks and settings may have identified alternative concepts, ratings, and insights. A larger sample size would also have permitted stratification of results by a range of variables, such as the perspectives of network funders, leaders, and members. Also, given the exploratory nature of this work, the qualitative analysis did not attempt to reach saturation on any particular domain, favoring capture of the key themes emergent in the data that directly linked to concept mapping results.

The concepts identified in this study provide an incremental step toward better outcome measurement for CDP networks with a population health focus. These outcomes represent potentially relevant areas in which specific indicators, tools, or techniques can be developed for describing network outcomes as CDP networks evolve. Such methodological advancements will serve as important complements to ongoing efforts to develop longitudinal case studies for tracking network evolution and change over time. As these methods continue to be developed, findings from the present study may be of use to those involved in network evaluation and management who are seeking practical guidance on ways to understand the outcomes and impact of their networks.

Top

Acknowledgments

This study was funded by the Canadian Cancer Society (grant 2011-701019). The authors gratefully acknowledge the contributions of all study participants, including the initial contacts who nominated networks for inclusion in the study, the network leaders who distributed the study information throughout their networks, those who completed the concept mapping exercise, and those who completed individual interviews.

Top

Author Information

Corresponding Author: Cameron Willis, PhD, Propel Centre for Population Health Impact at the University of Waterloo, 200 University Ave West, Waterloo, ON, N2L 3G1. Telephone: 519-888-4567 Ext. 36979. Email: cdwillis@uwaterloo.ca.

Author Affiliations: Alison Kernoghan, Barbara Riley, Propel Centre for Population Health Impact and the School of Public Health and Health Systems, Faculty of Applied Health Sciences, University of Waterloo, Waterloo, Ontario; Janice Popp, Faculty of Social Work, University of Calgary, Calgary, Alberta; Allan Best, Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, and the School of Population and Public Health, University of British Columbia, Vancouver, British Columbia; H. Brinton Milward, School of Government and Public Policy, University of Arizona, Tucson, Arizona.

Top

References

1. Provan KG, Fish A, Sydow J. Interorganizational networks at the network level: a review of the empirical literature on whole networks. J Manage 2007;33(3):479–516. CrossRef
2. Luke DA, Harris JK, Shelton S, Allen P, Carothers BJ, Mueller NB. Systems analysis of collaboration in 5 national tobacco control networks. Am J Public Health 2010;100(7):1290–7. CrossRef PubMed
3. Willis CD, Riley BL, Herbert CP, Best A. Networks to strengthen health systems for chronic disease prevention. Am J Public Health 2013;103(11):e39–48. CrossRef PubMed
4. Provan KG, Nakama L, Veazie MA, Teufel-Shone NI, Huddleston C. Building community capacity around chronic disease services through a collaborative interorganizational network. Health Educ Behav 2003;30(6):646–62. CrossRef PubMed
5. Varda D, Shoup JA, Miller S. A systematic review of collaboration and network research in the public affairs literature: implications for public health practice and research. Am J Public Health 2012;102(3):564–71. CrossRef PubMed
6. Kenis P, Provan K. Towards an exogonous theory of public network performance. Public Adm 2009;87(3):440–56. CrossRef
7. Provan KG, Milward HB. A preliminary theory of interorganizational network effectiveness: a comparative study of four community mental health systems. Adm Sci Q 1995;40(1):1–33. CrossRef
8. Provan KG, Milward HB. Do networks really work? A framework for evaluating public-sector organizational networks. Public Adm Rev 2001;61(4):414–23. CrossRef
9. Turrini A, Cristofoli D, Frosini F, Nasi G. Networking literature about determinants of network effectiveness. Public Adm 2010;88(2):528–50. CrossRef
10. Ferlie E, Fitzgerald L, McGivern G, Dopson S, Exworthy M. Networks in health care: a comparative study of their management, impact and performance. In: National Institute for Health Research Service Delivery and Organisation programme, editor. London: Queen’s Printer and Controller of HMSO; 2010.
11. Bevc CA, Retrum JH, Varda DM. New perspectives on the “silo effect”: initial comparisons of network structures across public health collaboratives. Am J Public Health 2015;105(Suppl 2):S230–5. CrossRef PubMed
12. Gibbons DE. Interorganizational network structures and diffusion of information through a health system. Am J Public Health 2007;97(9):1684–92. CrossRef PubMed
13. Kane M, Trochim WMK. Concept mapping for planning and evaluation. Thousand Oaks (CA): Sage Publications, Inc.; 2007.
14. Willis CD, Riley BL, Jolin MA. Improving the performance of partnerships for chronic disease prevention: report on a planning meeting. Waterloo (ON): Propel Centre for Population Health Impact, University of Waterloo; 2013.
15. Boyatzis RE. Transforming qualitative information: thematic analysis and code development. Thousand Oaks (CA): Sage Publications; 1998.
16. Carver V, Reinert B, Range LM. Sustaining tobacco control coalitions amid declining resources. Health Promot Pract 2007;8(3):292–8. CrossRef PubMed
17. Feinberg ME, Bontempo DE, Greenberg MT. Predictors and level of sustainability of community prevention coalitions. Am J Prev Med 2008;34(6):495–501. CrossRef PubMed
18. Feinberg ME, Riggs NR, Greenberg MT. Social networks and community prevention coalitions. J Prim Prev 2005;26(4):279–98. CrossRef PubMed
19. Mandell M, Keast R. Evaluating network arrangements: toward revised performance measures. Public Performance & Management Review. 2007;30(4):574–97. CrossRef
20. Hartley J, Allison M. Good, better, best? Inter-organizational learning in a network of local authorities. Public Manage Rev 2002;4(1):101–18. CrossRef
21. Provan KG, Beagles JE, Leischow SJ. Network formation, governance, and evolution in public health: the North American Quitline Consortium case. Health Care Manage Rev 2011;36(4):315–26. CrossRef PubMed
22. Isett KR, Mergel IA, Leroux K, Mischen PA, Rethemeyer RK. Networks in public administration scholarship: understanding where we are and where we need to go. J Public Adm Res Theory 2011;21(Suppl 1):i157–73. CrossRef
23. Kania J, Kramer M. Collective impact. Stanford Social Innovation Review 2011;2011 Winter:36-41.
24. Foster-Fishman PG, Nowell B, Yang H. Putting the system back into systems change: a framework for understanding and changing organizational and community systems. Am J Community Psychol 2007;39(3–4):197–215. CrossRef PubMed
25. Bunger AC, Doogan NJ, Cao Y. Building service delivery networks: partnership evolution among children’s behavioral health agencies in response to new funding. J Soc Social Work Res 2014;5(4):513–38. CrossRef PubMed
26. Provan KG, Milward HB. Health services delivery networks: what do we know and where should we be headed? Healthc Pap 2006;7(2):32–6, discussion 68–75. CrossRef PubMed
27. Johnston LM, Finegood DT. Cross-sector partnerships and public health: challenges and opportunities for addressing obesity and noncommunicable diseases through engagement with the private sector. Annu Rev Public Health 2015;36(1):255–71. CrossRef PubMed
28. Woulfe J, Oliver TR, Zahner SJ, Siemering KQ. Multisector partnerships in population health improvement. Prev Chronic Dis 2010;7(6):A119. PubMed
29. Popp J, Milward HB, MacKean G, Casebeer A, Lindstrom R. Inter-Organizational networks: A Review of the Literature to Inform Practice. Washington (DC): IBM Center for the Business of Government;, 2014.

Top

Tables

Table 1. Characteristics of Participants in Brainstorming and Sorting/Rating Groups, Concept Mapping Study, Canada, March 2015

Table 2. Clusters, Number of Statements, Cluster Descriptions, and Ratings of Importance and Feasibility, Concept Mapping Study, Canada, March 2015

^a Rated on a 5-point scale where 1 is least important and 5 is most important.
^b Rated on a 5-point scale where 1 is least feasible and 5 is most feasible.

Top