Valerie Bryan, CSW, PhD Blake L. Jones, LCSW, PhD University of Kentucky College of Social Work

1. Civic Engagement or Token Participation? Perceived Impact of the Citizen Review Panel Initiative in Kentucky Valerie Bryan, CSW, PhD Blake L. Jones, LCSW, PhD University of Kentucky College of Social Work Training Resource Center

2. Objectives for this Presentation • Describe Background and Context of Evaluation Study of Kentucky’s CRPs • Describe Methodology and Results of Study • Give you some “News You Can Use” in your state

3. Citizen Review Panels in Kentucky • Three free standing citizen review panels established in 1999. Two more Panels added since then. At the time of this study, all five regional panels were active, ranging in size from 9 to 17 members. • In 2001, The Cabinet contracted with the University of Kentucky College of Social Work Training Resource Center to provide technical assistance and administrative support to CRPs in their efforts. • Able to do a more comparable analysis across regional Panels

4. Prior Studies on Citizen Participation • To date, only two recent studies have been conducted specifically on CRPs, both with methodological limitations . • Other studies have been conducted pertaining to citizen advisory boards involved in a wide range of initiatives (mental health, law enforcement, environmental cleanup efforts). • Findings suggest that citizen advisory boards are most effective when they have clearly communicated goals and objectives and are given access to needed information to make decisions (Zander, 1993; Callahan, 1999).

5. Other Important Factors * top management commitment to the public participation process * two-way communication and education; * interactive and iterative public participation; adequate resources; * development of provisional trust between agency and public; * giving priority to trust building actions; * openness of the agency (Peelle, Schweitzer, Munro, Carnes, & Wolfe, 1996)

6. Evaluation Process and Design • Evaluation Request was made to the Cabinet for Health and Family Services Research Questions • 1.) How effectively do citizens review panel members perceive their efforts to be in positively affecting child protective services delivery? • 2.) How well do panel members work together collaboratively toward their federally mandated goals? • 3.) How effectively are the citizen review panels and the child welfare agency communicating with each other? • 4.) How responsive is the Cabinet to the panels’ recommendations given in their annual reports?

7. Research Questions • 5.) How prepared are panel members to carry out their mandated duties? • 6.) What do state child welfare agency officials perceive the role of the panels to be in developing and enacting child protection policies? • 7.) What can the panels do to further assist the child welfare agency in developing and implementing effective child protection policies?

8. Research Design A multi-method evaluative design was developed, centered around the investigation of these questions, including interviews with Cabinet personnel, focus groups with citizen review panels, a survey of citizen review panel members, and content analysis of CRP reports and Cabinet responses.

9. Focus Group Procedures • Five focus groups with citizen review panels. A total of 34 panel members participated • Panel members were asked nine questions related to the evaluation aims. • Notes were taken by a practicum student in attendance while the researcher also wrote the responses on an easel board visible to all participants. • These notes were qualitatively analyzed for the identification of primary themes.

10. Survey Development and Procedures • Survey instruments developed from instrument used in previous study • Three subscales measuring 1) information flow between agencies, 2) group cohesion, and 3) self-governance were embedded in the instrument. • The group cohesion subscale was adapted from the Group Cohesion Scale-Revised, developed by Treadwell, Lavertue, Kumar, & Veeraraghavan (2001), and the self governance subscale was adapted from the Governance Effectiveness Quick Check (Gill, Flynn, & Reissing, 2005). • The wording of questions was adapted to more clearly apply to citizen review panel members • The survey was distributed to all 81 panel members via CRP listserv email

11. Survey Procedures, cont… • Two questions were added to assess panel members’ perceptions of adequate community representation on their panels, a requirement of the federal mandate. • Two open-ended questions asked respondents to give feedback on the three largest obstacles to citizen involvement in child welfare and three recommendations for improved partnership with the Cabinet. • Distributed to 81 members via email/ A total of 34 panel members responded, with 30 completing all items (42% response rate).

12. Cabinet Personnel Interview Procedures • Interviews were conducted with Cabinet liaisons, Senior Regional Administrators (SRAs) and state headquarter personnel (Central Office). • Fifteen interviews with Cabinet representatives were completed, in which they were asked questions about their perceptions of the impact of CRPs, the purpose of the mandate which established CRPs, barriers to citizen involvement in public child welfare, and recommendations for future involvement.

13. Content Analysis Procedures • Annual reports prepared by the panels and written responses were available from 2003 through 2006. This process was conducted from October through November 2006. • Content analysis of panel annual reports to the Cabinet and the Cabinet responses to these recommendations was conducted in order to describe and assess the quality of communication exchange between the Cabinet and CRPs.

14. Content Analysis • We wanted to know: 1) what activities were conducted, 2) what recommendations were developed from those activities, 3) how recommendations were addressed, 4) to what extent responses from the Cabinet reflected an adequate recognition of problem areas identified by the CRP reports, and 5) what, if any actions or strategies were planned to implement appropriate and necessary changes.

15. Synthesized Evaluation Findings How effectively do citizen review panel members perceive their efforts to be in positively affecting child protective services delivery? • Panel members’ perceptions regarding their effectiveness in making a positive impact upon service delivery were mixed. • Panel members view themselves as primarily a source of support to the Cabinet and to the Cabinet’s goals, by being expert community partners, advocates, and neutral observers of the child welfare system. • CRP members perceived themselves as more effective when working at the local level on community issues rather than statewide policy matters. • Panel members identified communication problems and a lack of authentic partnership with the Cabinet, as well as resource and membership recruitment issues, as obstacles to reaching their full potential. • Responses to suggestions for more effective citizen involvement indicated that panel members would prefer a more clearly defined purpose and scope of work, and re-emphasized the need for improved communication with the Cabinet.

16. Synthesis of Findings, cont… How well do panel members work together collaboratively toward their federally mandated goals? • CRP members reported working together very successfully toward common goals in their online survey responses. • Panel members who participated in the survey experienced few or no difficulties in working together toward common goals, and that most panel members perceived their CRPs to be cohesive workgroups. • Survey items inquiring about panels’ capabilities to self-govern were also responded to positively, suggesting that panel members perceive themselves as good managers of CRP initiatives and activities.

17. Synthesis of Findings, cont… How effectively are the citizen review panels and the Cabinet communicating with each other? • Findings from all sources suggest that effective communication between the Cabinet and CRPs presents a serious obstacle needing attention. • Panel members identified communication problems as a primary obstacle to fulfilling their duties effectively; all panels wanted to improve the quality and frequency of communication. • Some offered that this could be accomplished through the Cabinet liaisons who attend panel meetings. • SRAs and liaisons echoed these concerns about communication by recommending that more experienced liaisons be appointed to all panels, who could serve as the key intermediary between the Cabinet and the panels more effectively.

18. Synthesis of Findings, cont… • Most survey respondents disagreed with the statement, “Citizen input is sought by the child welfare system early in the policy-making process, before decisions are made,” indicating that the panel members who participated in the study did not perceive themselves to be true partners with the Cabinet. • As a group, they do not believe that they have actual influence upon Cabinet policy makers, which likely affects how effective they perceive their efforts to be. • A primary focus group theme which emerged was the need for Cabinet accountability, most notably with regard to the annual reports. • A few Central Office interviewees expressed concern with the quality of recommendations CRPs offer in their annual reports, however, suggesting that the panels at times make recommendations without really understanding what the process of change may entail.

19. Synthesis of Findings, cont… • Despite these communication difficulties and misunderstandings, Central Office interviewees also recognized that the Cabinet needs to respect the panel members’ efforts. • One respondent stated that the panel members should be “at the table” in the decision making process and they should offer thoughtful input, with the understanding that the Cabinet must make the final decisions about policy and practice changes and improvements.

20. Synthesis of Findings, cont… How responsive is the Cabinet to the panels’ recommendations given in their annual reports? • Findings from focus groups, SRA and liaison interviews, survey results and content analysis indicate that the Cabinet could improve the quality of its written response to CRP reports. • Many panel members criticized the Cabinet’s responses as merely reframing or restating the panel recommendations without directly acknowledging if or how they may address the identified issue. • SRAs and liaisons interviewed also expressed uncertainty about the level of cooperation between Central Office and the panels. • Though content analyses of the panels’ and the Cabinet’s written reports revealed examples of effective communication, several panel recommendations were made without adequate justification and context. • Some of the Cabinet responses failed to mention any potential actions they might take to address the recommendation. • This problematic communication may be negatively influencing the panels’ and the Cabinet’s perceptions of each other and the role the panels can fulfill in improving child welfare practice and policy.

21. Synthesis of Findings, cont… How prepared are panel members to carry out their mandated duties? • Findings from focus groups, survey results, and Cabinet interviews pertaining to the level of CRP member preparedness to carry out their duties were mixed. • Many panel members identified the need for more members who will regularly attend meetings, and the need for more representative membership. A need to retain experienced members was also identified. • Several said they would like to see former clients become panel members but they also voiced concerns over several barriers to their recruitment and retention. SRAs and Central Office interviewees concurred. • SRAs recommended recruitment of new members who were not child welfare professionals, including former clients, to attain a more representative perspective.

22. Synthesis of Findings, cont… • Central Office interviewees were concerned with panel members’ lack of knowledge regarding the bureaucratic nature of the child welfare system and the unrealistic nature of some CRP recommendations. • The federal mandate which established the panels was criticized by all sources as a barrier to the panels effectively carrying out their duties, citing its broad, vague scope, while only requiring panels to meet quarterly. • Cabinet interviewees thought it was impossible for panel members to effectively meet the intent and purpose of the mandate as it is currently written. • Because of this, it is not clear what roles a citizen review panel member must, can, or cannot fulfill to successfully adhere to the mandate, or what activities are sufficient to meet the purpose of the mandate.

23. Synthesis of Findings, cont… What do Cabinet officials perceive the role of the panels to be in developing and enacting child protection policies? • Cabinet interviews indicated that they view the citizens review panel members’ primary roles as those of “objective third party,” agency advocate, staff support, and community partner. • Most SRAs and liaisons viewed the evaluative role the CRPs carry out as valuable, and helped to offset the agency’s “tunnel vision” about policies and practices. • Interviewees frequently emphasized the support and advocacy CRPs provide to the agency.

24. Synthesis of Findings, cont… • Central Office valued their ability to serve as a neutral group who could view the child welfare system “with fresh eyes,” a strength that helps services and practices to evolve. • Some did recall, however, past difficulties and misunderstandings encountered when CRP members misinterpreted the boundaries of their role or brought personal agendas into their CRP efforts. • Like others, Central Office interviewees identified the most beneficial work of the CRPs occurring at the local level. What can the panels do to further assist the Cabinet in developing and implementing effective child protection policies? • Cabinet interviews indicated that they do not view the role of CRPs as that of policy developers or policymakers, but rather as expert community partners who can provide an objective viewpoint.

25. Synthesis of Findings, cont… • It should be noted, however, that the mandate’s language indicates that panels should examine and evaluate existing state and agency policy; in so doing, it would be expected that recommendations they offer would impact the shaping of policy. • Again, the broad language of the mandate may be the source of much miscommunication, role confusion and misinterpretation of CRP responsibility. • Cabinet interviewees indicated that CRPs could help front line workers the most by advocating for human and technological resources. • One Central Office interviewee suggested that because child welfare issues exist beyond the Cabinet’s scope, systemic issues need to be investigated in a broader manner, including other involved elements of child welfare, such as the family courts, public health, and community-based agencies who work with children and families.

26. Synthesis of Findings, cont… • Central Office personnel also wanted to see solutions offered more frequently to the problems CRPs identify in their reports. • They also recommended that CRP members take on the role of intermediary between different parties and providers, explore public relations endeavors, and educate legislators about DCBS work, in the hopes of securing more workers and funding for initiatives.

27. Recommendations Analysis and synthesis of findings from all sources resulted in the following recommendations: • Work to create and pass state-level legislation which would clarify the purpose, role, and duties of citizen review panels, in order to identify the scope of their efforts and in what ways the citizens review panels, the Cabinet, and other partners responsible for child welfare within the state are accountable to each other as a result of the federal mandate. • Improve the level of specificity within the Memorandum of Understanding between the Cabinet by including a feasible protocol for ongoing exchanges of information which will facilitate the ability of all parties to follow through with action plans and to keep all informed about the result of recommendations throughout the change process.

28. Recommendations, cont… • Improve communication between all levels of the Cabinet and the panels through first, adequate preparation and orientation of appointed liaisons who will serve in a consistent manner as an intermediary between the agency and the panels. Secondly, propose this issue as an open topic of discussion among involved parties to identify mutually acceptable ways in which ongoing, constructive communication can be achieved. • The Cabinet, the CRP Program Coordinator, and the panels should consider developing a unified, strategic plan to recruit panel members from a broader base of applicants to assure adequate representation of each region’s citizenry on the panels.

29. Recommendations, cont… • The citizen review panels and the Cabinet should work to identify areas together in which panel members may lack knowledge or training regarding child welfare issues, policies or practice, and the Cabinet should offer ways in which panel members can be educated in these areas at no cost to the members, in order to improve the quality of recommendations offered by all panels every year. The CRP Program Coordinator in partnership with liaisons, SRAs and chairpersons can help to identify areas of need. • The Cabinet should seek to improve the quality of responses offered to each recommendation presented in the panels’ annual reports, addressing all items offered with at the least, a reasoned explanation as to why recommendations are not feasible or timely when appropriate. Action plans, prospective timelines, and updates as they become known should be offered to the panels in response to recommendations which the Cabinet expects to act upon. The recommendation for a follow-up protocol as agreed upon within the MOU as listed above should support this process.

30. Lessons Learned • It was clear that both the Cabinet and the CRPs shared responsibility for the difficulties encountered in working together. • The results support prior study findings on the necessary conditions for effective citizen participation • The obvious communication problems identified led over time to deeply rooted resentments experienced by the CRP members. This led to a sense of powerlessness to change the situation and to make meaningful contributions to child welfare policy and practice.

31. Lessons Learned • As one panel member stated, “The reports and responses should be the start of a conversation, not the end.” Making communication a process rather than an event, and adhering to a systematic exchange of information is the key change needed. • In order to address the offered recommendations, communication problems must first be repaired.

32. References Callahan, K. M. (1999). Citizen participation: The utilization and effectiveness of citizen advisory committees in local government. Unpublished doctoral dissertation, Rutgers University. Gill, M., Flynn, R., & Reissing, E. (2005). The Governance Self-Assessment Checklist: An instrument for assessing board effectiveness. Nonprofit Management & Leadership, 15 (3), 271-294. House Report 104-081, 104th Congress, Second Session. (1996). Jones, B. (2004b). Citizen participation in public child welfare: A multi-state study of citizens review panel members’ perceptions of effectiveness. Dissertation Abstracts International, 65 (07), 2765A. (UMI No. 3140108)

33. References, cont… Jones, B.L., Litzelfelner, P. & Ford, J.P. (2003). The value and role of citizen review panels in child welfare: Perceptions of citizens review panel members and child protection workers. Child Abuse & Neglect: The International Journal, 27 (6), 699-704. Peelle, E., Schweitzer, M., Munro, J., Carnes, S., & Wolfe, A. (1996). Factors Favorable to Public Participation Success. No. AC05-96OR22464, report submitted to the US Department of Energy. Treadwell, T., Lavertue, N., Kumar, V., & Veeraraghavan, V. (2001). The Group Cohesion Scale-Revised: Reliability and validity. International Journal of Action Methods: Psychodrama, Skill Training, and Role Playing, 54 (1), 3-12 U.S. Department of Health and Human Services (1998). Establishment of the Citizen Review Panel Requirement Under the Child Abuse Prevention and Treatment Act (ACYF-PI-CB-98-01). Washington, DC: Author on Team Functioning: An Empirical Investigation. Human Factors, 38:87-100. Zander, A. (1993). Making boards effective: The dynamics of nonprofit governing boards. San Francisco, CA: Jossey-Bass.

34. This presentation based on…. Bryan, V., Jones, B.L., Allen, E. & Collins-Camargo, C. (in press, Child and Youth Services Review). Civic Engagement or Token Participation? Perceived Impact of the Citizen Review Panel Initiative in Kentucky