Measuring Interorganizational Collaboration in Communities: Baltimore’s Strategy to Improve Birth Outcomes as a Case St

1. Measuring Interorganizational Collaboration in Communities: Baltimore’s Strategy to Improve Birth Outcomes as a Case Study Donna Strobino and Meredith Matone Department of Population, Family and Reproductive Health Johns Hopkins Bloomberg School of Public Health

2. Infant Mortality Rates: Maryland & Baltimore

3. Gaps in Current Efforts to Improve Birth Outcomes Individual initiatives are funded inadequately and separately Poor coordination of services Lack of adequate primary health care Lack of health insurance Lack of minimum standard of care Limited community mobilization to promote healthy behaviors prior to or between pregnancies

4. General Overview • The Strategy to Improve Birth Outcomes (SIBO) is a citywide and community-led strategy • Reframes the way people, providers, services, and community agencies interrelate around the wellbeing of women throughout their reproductive life cycle (preconception, pregnancy and postpartum) and their infants • Takes a population-based approach to solving intractable disparities in birth outcomes

5. High Impact Service Areas

6. Family League of Baltimore City RFP • SIBO challenges communities to mobilize around improving birth outcomes in a community and increasing demand for and utilization of the eleven high-impact services. • The Family League is funding: • a citywide communications campaign • an evaluation and performance monitoring strategy • resident and community mobilization around infant mortality and collaboratives capable of connecting all women in a community with appropriate care and services.

7. Five Critical Components: 1 Policy mandate emphasizing Improved birth outcomes as a priority of the Baltimore public health landscape. Funding streams to facilitate systems and services to address gaps in coverage, improve effectiveness of practices and policies, and replace inefficient implementation of existing practices with evidenced-based alternatives.

8. Five Critical Components: 2 Reorganize or retool health care delivery systems and practices for mothers and infants (including fathers, partners, families and communities who care for and support them). Assess and refine service provider practices and policies, especially related to risk assessment and create new operational models of service provision, where needed.

9. Five Critical Components: 3 City-wide social marketing intervention to influence the behavior of community residents, with a focus on expecting mothers and their partners and families, caregivers, service providers, and policy makers. Target all citizens

10. Five Critical Components: 4 Community-led focused initiative designed to coordinate already existing, culturally relevant, local health and social services within target areas Customized approach requiring strong community input and collaboration, leading to transferable models for other neighborhoods. NO new services

11. Five Critical Components: 5 A collaborative strategy driven by a set of partnership principles defining the quality and character of stakeholder interactions which guide the tenor of policies, agreements and protocols. Important in defining the community led component Applies to CIT and Steering Committee

12. FLBC RFP for Community Collaboratives: Must Demonstrate the Capacity to: • Outreach to all pregnant women in the community to connect them with appropriate care and services • Participate fully in the new citywide triage system for pregnant women into home visiting programs; • Link pregnant women with appropriate supports and services (home and center based) • 12 targeted communities with 3 awards of $400,000 - 600,000 for the 21 month period April 2010 – December 2011

13. Expected Impact in RFP SIBO’s long term success: improvements in rates for pre-term birth, LBW, and deaths from unsafe sleep in targeted communities and in Baltimore City Tracking of several indicators related to the activities of collaborative teams that contribute to the achievement of the key outcomes. Success also measured by whether communities develop partnerships collaborations to assure access to the 11 high-impact services and link the various components of the health care system and human resources in the community to assure this access.

14. Evaluation Objectives Assess implementation of SIBO core components related to policy, services, communities, and families and individuals; Monitor implementation throughout the duration of SIBO to allow for modifications and trouble-shooting in real time (including the collaboration strategy); and Produce data to understand the impact of the first four core components on infant mortality in Baltimore City.

15. SIBO Evaluation Design • Quasi-experimental design to evaluate changes in indicators in communities funded for the community-led initiative with the other communities with high infant mortality rates not initially funded: funded communities compared to remaining targeted communities

16. Data Source for Collaboratives Data from community partners about collaborations among agencies and community leaders involved in implementation of the community-led focused intervention in targeted communities

17. Overview • Significance • Measurement problem • Collaboration definition, models, research • Evaluation strategies for collaboration • Case Study: The Strategy to Improve Birth Outcomes in Baltimore City

18. Significance Collaboration between health and social service agencies within a community that creates a population-based system of care are ideal because complex issues are inseparable from social and physical determinants of health, which cluster by community.

19. Significance • Proposed benefits of collaboration: • Promote cost-effectiveness • Reduce duplication of services • Improve communication • Comprehensive service provision • Increase community capacity • What is the evidence to support these claims?

20. A Measurement Problem: What is collaboration? • Conceptually: logical and “feel-good” results indicator • Reality: plagued with measurement issues

21. Collaboration Definition “Collaboration occurs when a group of autonomous stakeholders of a problem domain engage in an interactive process, using shared rules, norms, and structures, to act or decide on issues related to that domain.” Wood and Gray 1991 Collaboration in public health… “A relationship between two organizations that involves exchanging information and sharing resources in order to coordinate services for mutual benefit and to achieve common goals, such as facilitating care of treatment of clients.” Friedman et al. 2007

22. Collaboration Models • “Level of Collaboration” models • Attempt to classify the stage, or intensity, of relationships between organizations • Indicators generally include purpose (goals), structures (central body, leadership) and behaviors (communication, budget)

23. Levels of Community Linkage Model, Hogue 1993

24. State of the Research • Most community-level research to date has focused on costs and benefits of collaboration to organizations and/or accounts of the collaboration formation process • Stressors of collaboration in public sector • Characterization of nonprofit organizations likely to participate in formal collaboration • Effective arrangements of collaborative efforts

25. State of the Research • Collaboration research in community health promotion, specifically grant-funded initiatives and public and nonprofit organizations, is underdeveloped • Done little to advance measurement tools

26. Evaluation Strategies • Social Network Analysis • Survey • Thomson, Perry, and Miller Method • Qualitative Analysis

27. Social Network Analysis

28. Social Network Analysis • Social network analysis is a method of studying the linkages among individuals, groups, organizations, and other units (termed actors) • Patterns of relationships are assessed and used to understand the structure, function, and performance of a network

29. SNA in Public Health • Broad applications: • Transmission networks: disease and information • Social networks: social support and social capital • Organizational networks: health systems

30. SNA Methodology: Respondents • Create “sampling frame” of organizations of relevance in evaluation • Known upfront, or • Formative research: snowball sampling, etc. • Designate organization representatives: individual or group • Validity **This methodology is applicable to all evaluation methods

31. SNA Content • Rating scales of relationship type or intensity • Questionnaires often draw heavily from the collaboration models for indicators of relationships

32. SNA Sample Questionnaire

33. SNA Data • Longitudinal • Numeric • Network density, centrality, geodesic distances, directionality • Graphical • Sociograms: visual representations of numeric data • Matrices

34. Sociograms: Cross et al., 2009

35. Surveys

36. Survey Content • Measures more variable in type and complexity • Like SNA, can draw heavily from the collaboration models for indicators of linkage levels or, • Frequency of action measures • e.g. number of communications in the past year

37. Survey Data • Longitudinal • Numeric • Conventional statistical methods • Graphical • Microsoft PowerPoint: models using circles to represent partners and lines to represent linkages (Frey et al. 2003)

38. Thomson, Perry and Miller

39. Thomson, Perry, and Miller 2007 • Goal: to make collaboration quantifiable with an empirically tested measurement tool • Built on the Wood and Gray (1991) definition • Categorize collaboration into five dimensions: Governance, Administration, Mutuality, Norms, Organizational Autonomy

40. TPM Content • 56 item closed-ended Likert scale questions on organizational attitudes and behaviors surrounding each of the five dimensions of collaboration • From the 56-indicator questionnaire, 17 indicators were empirically validated in statistical structural equation modeling • 17 indicators cross-validated in a 2004 study of 133 collaborators in a Family Preservation Program

41. TPM Data • The Thomson, Perry, and Miller strategy measures solely the presence of collaboration using a fixed model of attitudinal and behavioral indicators developed as proxies for the dimensions of an operational definition of collaboration • Collaboration is standardized • More readily compared across initiatives

42. Qualitative

43. Qualitative Methodology • Facilitated focus groups or in-depth interviews • Standard data analysis methods

44. Qualitative Data • Formative • Who are the players? • Process/Outcome • Lists of collaborators, collaborative process info • Contextual information: costs/benefits, barriers/facilitators to collaboration • Mixed-method approaches • In addition to SNA, survey, or TPM • Validate numeric data • Provide explanatory data

45. Evaluation Strategies Matrix Degree of Complexity: instrument development, data analysis techniques and software, and required expertise Cost: material and non-material resources required by the data collection strategy, and the associated burden on the research team and respondents

46. The Strategy to Improve Birth Outcomes in Baltimore City

47. Choosing an Evaluation Method • Data • Ideal…and, • Realistic • Timeframe • Degree of complexity • Cost

48. SIBO Community Collaboration Component Proposed Evaluation Design • Data • Matches ideal data • Timeframe • Case studies of similar scale • Complexity • SNA Evaluator on evaluation team • Cost • Software for large analyses • Quick rating-scale questionnaire administered at time of other evaluation activities (e.g. community surveys)

49. How will we measure success?

50. Cognitive Mapping • Formative • Who are the players? • What should the system look like? • What should the exchanges between agencies look like? • 3 Perspectives • Core Implementation Team • Service Providers • Steering Committee