Download
1 / 33
330 likes | 533 Views
Establishing Baselines – Sources of Information for Indicators and Developing Participatory Indicators. DR Angela Veale, School of Applied Psychology, University College Cork. Overview. Evaluation is a central feature of programme design and project cycle management.
E N D
Establishing Baselines –Sources of Information for Indicatorsand Developing Participatory Indicators DR Angela Veale, School of Applied Psychology, University College Cork.
Overview Evaluation is a central feature of programme design and project cycle management. Aim of session is to reflect on how to identify and develop the best indicators for evaluation purposes. Look at some case studies of psychosocial programme evaluations
Establishing baselines- role of indicators • For each project objective at output, outcome and impact levels, we need to define how we would measure results • An indicator is a simple, clear statement that helps measure and communicate change • If there is an objective there should be a means of measuring if it has been achieved
Intervention Group Waitlist Group no intervention (yet) Standard practice
When standardised measures don’t cut it? UNICEF- Consultancy to Provide Technical Assistance on Pilot to Develop a New Stress Measure for Adolescents and Caregivers UNICEF currently utlising a validated stress scale Programme showing a lack of impacts- may be that existing measures of perceived stress have mainly been developed and validated in the United States and western settings and may not capture the concept of stress in all settings. In order to capture stress as a potential pathway for positive outcomes in poverty alleviation programs and other interventions, it is important to adequately measure stress and thus a A measure of perceived stress suitable for the context is needed- a need for participatory indicator development? How would you go about doing this?
Developing participatory evaluation indicators • The project: Supporting the social reintegration of former child soldier young mothers in Liberia, Sierra Leone & northern Uganda • The approach: Highly participatory; ‘meaningful participation’ was a central principle • How could we know if the programme was successful in having a positive impact on the social reintegration of young mothers? • Who decides what is ‘effective social reintegration’? • How best to measure social reintegration? • Standardised measures possible (e.g. Harvard Trauma Questionnaire; Child War Trauma Questionnaire; Retention in School etc.) used in other studies; not suitable here • Decision to develop participatory and culturally relevant indicators.
Evaluation Methodology Free listing At a workshop with representatives of young mothers from the 3 countries, they generated lists of important indicators of successful reintegration. This generated 47 items Ranking In northern Uganda, indictors were ranked in order of importance. Similar items were grouped together. Low ranking and highly similar indicators were discarded Face validity Remaining items were examined for ‘face validity’ –ambiguous or unclear indicators were discarded. Final survey instrument contained 13 indicators of social reintegration.
Cultural indicators with formerly associated girls, Sierra Leone
Example 2: Participatory indicator development with Syrian Refugee Families in Lebanon • Conduced with male participants and their families who had completed an ‘Engaging Men’ programme (men, wives of men, children of men-adolescent girls and boys) • Men and their wives and teenage children who are due to start the programme • Shanahan, Hajazi, Veale (2016)
Do no harm Facilitators invited participants- existing relationships, ongoing support Informed consent process- written consent Voluntariness Right to withdraw at any time Data protection Confidentiality Psychosocial referral pathways Men and boys and women and girls were interviewed separately Interviews were conducted in Arabic with translation into English. Approval obtained from UCC Ethics Committee.
Participatory Indicator Development Problem analysis (Protection problems affecting men, women, teenage girls, teenage boys and children). Social mapping of vulnerability and situated risk. Free listing and ranking of coping strategies (to reduce stress and support protective capacity) generated and ranked by men, women, teenage girls, teenage boys. Free listing and ranking of conflict resolution capacity generated and ranked by men, women, teenage girls, teenage boys. Reduction in inter-community conflict indicators generated and ranked by men, women, teenage girls, teenage boys.
Problem analysis Male participants explained that due to deprivation and poverty, men fight with their wives and children. They gave specific examples of how after coming home they are greeted by "why didn't you bring food/bread today?” Various men endorsed how wives' "nagging" over their inability to supply household needs compounds their frustration over the situation and causes them to hit their wives. Another source of arguments in the family that may lead to hitting are wives’ frustrations over not being able to visit their family in other parts of Lebanon. This is further exacerbated by if the husband's family are nearby but not the wives. Some men also identified how their guilt and frustration over having to select some of their children to go to school, but not others also frustrates them and increases their risk of fighting with and hitting their wives. Some men also identified early child marriage (around 13-14) as an issue, explaining that they perceive that female children have more expenses than male children (e.g. because they need more clothing). Women identified the impact of stress on men reflecting at home as the most significant family problem, followed by the impact of stress on their children. Women spoke predominantly about verbal abuse with some participants also indicating that physical violence was an issue. Children identified fighting in the home due to the stress their mothers and fathers were under as a significant problem.
Psychosocial problems affecting men Men identified psycho-social problems related to: feelings: male participants identified feeling helpless, weak, broken, being suffocated from bottling up, low mood, feeling pressured, noting how they feel aged because of their burden, noticing how they or other young people have become like old people, feeling that frequent worry would make a "mountain collapse". Feeling like one wanted to "escape one's skin" if for example the children smelled the neighbors barbecuing meat but the fathercan't afford meat. Thinking : male participants reported frequent worrying about the future, worrying about developing chronic diseases like diabetes, or getting a stroke, ruminating about conflict related experiences. Behaviour : increased smoking, irritability, yelling, speaking in a sharp tone, headaches, taking multiple panadols, flashbacks, nightmares, (involuntarily) losing weight, neglecting outside appearance (e.g. not going barber), depriving oneself to provide. Being forced to accept the situation, smoking, nargileh, hash, drinking.
Indicators of reduction in inter-communityconflict When I greet them they would greet me back They would smile at me when I see them in the street Reduction in rents charged to Syrian refugees Less street harassment of young girls by both communities I would visit members of the other community in their homes I would feel safer walking in this area Inter-marriage No shouting or beatings in the streets Groups of friends of Lebanese and Syrians People wouldn’t say ‘refugee’ or ‘Syrian’ in a bad tone I would be allowed to visit my Lebanese friends home Syrians and Lebanese would communicate more often on Facebook and whatsapp
Conclusion • Challenges of participatory indicator development • Time needs to be factored into the implementation model – ideally before starting implementation. • Some sensitive topics are hard to put into words in ways that will be acceptable to ask in a survey-style method • Works best as complementary to standardised measures • Then why do it? • Participants/beneficiaries get to define what they would see as sucessfulprogramme outcomes and impact – gives a deeper insight into how your programme should and could be working. Includes the perspective of those that the programme is ultimately accountable.
Some resources UNICEF (2016) Developing and selecting measures of child wellbeing. Florence: Innocenti Centre. UNICEF (2009) Guide to the evaluation of psychosocial programming in emergencies. Geneva: UNICEF.
