Brief Historical Overview of the Budapest Initiative and Testing Activities

1. Brief Historical Overview of the Budapest Initiativeand Testing Activities 20-22 January 2010 Palais des Nations, United Nations Geneva, Switzerland

2. Measuring health statusfor international comparisons… • What constitutes core health? • What measures should be used? • What standards for producing the data? • Are the data internationally comparable? • What methods should be used for comparisons?

3. Historical Overview:Establishment of the TF and First Meeting • May 2004: UNECE sponsored a Joint Meeting on the Measurement of Health Status with WHO and Eurostat • Broad consensus on a) the need for multiple indicators to provide a full statistical picture of population health, both for individual country use and for international comparisons; and b) the immediate need for an indicator of ‘health state’ • Recommendation to set up a Task Force on the measurement of heath state under the aegis of the UNECE • The first meeting was held in Budapest in November 2005, and following the City Group convention, now known as the Budapest Initiative (BI)

4. Historical Overview:Purpose and Objectives Purpose: To develop a new common instrument to measure health state suitable for inclusion in national interview surveys which will provide basic necessary information on population health. Objectives: • To develop a question set to assess overall health state through a number of domains of functioning; • To describe trends in health over time within a country, across subgroups of a population, and across countries; and • To do so in the framework of official national statistical systems.

5. Historical Overview:Framework, Definitions, Domains • Development of a conceptual framework defining the concepts of health status and health state • Paper: Health as a Multi-Dimensional Construct and Cross-Population Comparability (2005) • Adopted a set of criteria for selecting functional domains • Paper: Criteria For and Selection of Domains for the Measurement of Health Status (2005) • Selection of domains • Developed a question set to measure selected domains • Paper: Conceptual and Logistic Issues in Item Construction and Proposal Questions for Domains (2005)

6. What is Health State? Health State measures functional ability as opposed to other aspects of health: • Determinants and risk factors • Disease states • Use of health care • Environment barriers and facilitators Functioning is measured in terms of ‘capacity’ not ‘performance’ • ‘Within the skin’ • Without the use of aids in a reasonable environment

7. HEALTH STATUS Health states (capacities) Determinants Domain 1 of health Physiological Diseases, Domain 2 Gen etic contribution Physiological risk factors Death symptoms risk factors and and risk and injuries . risk markers markers . Lifestyle / behaviour Domain N1 Physical e nvironment . . Quality of life Domain N 2 Economic e nvironment Wellbeing Social e nvironment Health Status and Health States Health State: an individual’s levels of functioning within a set of health domains.

8. BI Focus and Guiding Principlesfor the Question Set • Measure the health states of individuals; • Operationalize health state as functional ability; • Focus on capacity rather than performance; • Measure across a parsimonious number of health domains, capturing the most variation in health; • Meet high standards of validity and international comparability; and • Simple and clear, and easily translated into many languages.

9. Criteria for the Selection of Domains In order to identify the domains of health to be included, a set of criteria were established: • Relevance: face validity, breadth of domains, importance for population health monitoring, draw on selected key ideas of the ICF; • Feasibility: reasonable for inclusion in health interview surveys, consistency in meaning across multiple social contexts, heterogeneity, parsimony of domains • Measurement: statistical & structural independence, clear series of levels within domains, within/near the skin, suitability for preference measurement.

10. Final 7 Domains for BI Questions • Vision • Hearing • Walking / Mobility • Cognition • Affect • Pain • Fatigue* The original set of 10 domains included dexterity and social functioning/ relationships, both of which have been dropped. Anxiety was combined with depression in the current domain of affect.

11. Criteria and Considerationsfor Item Construction • Number of questions per domain • Ensuring uni-dimensionality • Duration of the recall period • Dealing with assistive devices and medications • Item wording and response categories • Positive vs. negative wording • 4- vs. 5-category response sets • Summary/preference health measures

12. Testing the Question Set

13. Testing the BI Questions Three rounds of testing have been conducted as part of the development of a BI question set. Round 1: • November – December 2006 • BI-only test (4 countries, 3 languages): Australia, Canada, Italy, U.S. • Cognitive testing of all 7 domains • Results reviewed at January 2007 TF meeting • Separate studies and reports, inconsistent analysis methods, some general findings • Outcome: BI-M1 question set, submitted to Eurostat

14. Testing the BI Questions Round 2: • November 2007 – January 2008 • BI-ESS collaborative test (7 countries, 6 languages): Bulgaria, Germany, Portugal, Spain, Switzerland, U.K, U.S. • Cognitive testing of 6 domains (absent Vision) • Improvements: Evidence-based methodology & systematic comparative analysis of patterns • Analysis meeting in U.S. February 2008, all participants • Results documented in report • Outcome: Further changes for cognitive testing recommended

15. WG-BI Similarities Substantive overlap/Differing perspectives: • Question domains include the same basic and complex activities • Disability includes interaction with environment and civil rights perspective Requirements for question sets: • Minimize burden on national data collection • Parsimony in the number of indicators and measures; domains are succinct, clearly defined • Reasonable expectation of high quality responses from samples of the general public, demonstrated validity of measures • International comparability, relevant at national and international level • Focus on aspects of health that are more likely to produce comparable data • Need for cross national cognitive and field testing • Questions must be simple, clear, easily translated into many languages • Amenable to multi-modes of collection

17. Testing the BI Questions Round 3 (Cognitive Test): • January 2009 – March 2009 • UNESCAP-WG-BI collaborative test (10 countries): Cambodia, Canada, Fiji, Kazakhstan, Maldives, Mongolia, South Africa, Sri Lanka, Philippines, U.S. • Cognitive testing of 11 domains (additions: Upper Body, Communication, Learning, and Life Activities) • Improvements: On-line data management and analysis tool • Analysis meeting in U.S. May 2009 • Outcome: Further changes for field test recommended

18. Testing the BI Questions Round 3 (Field Test): • June 2009 – August 2009 • WG-BI-UNESCAP collaborative test (5 countries): Cambodia, Kazakhstan, Maldives, Sri Lanka, Philippines • Cognitive testing of 11 domains (additions: Upper Body, Communication, Learning, and Life Activities) • Improvements: Approx 1,000 field interviews at each site

19. Testing Issues and Methodological Developments

20. General Issues for Testing • Cross-cultural comparability: Do the survey questions work consistently across all countries and subgroups? • Translation comparability: Do terms (both in the question and in the response set) have the same meaning across countries? • Validity: Do respondents interpret questions consistently regardless of country, language, or demographic? Do respondents use the same thought processes to answer questions? • If not, then, why are there differences? What about the countries, languages or demographic subgroups generate different response processes? • How can we “fix” or manage these differences through question design?

21. Objectives for the Test Protocol Purpose: To develop systematic comparable method with joint analysis. • Evidence based • Joint and coordinated interviewing Similar protocol Similar sample Understanding of differences (at a minimum) • Joint and coordinated analysis With interview data Evidence based (as opposed to opinion)

22. Lessons Learned • Semi-structured cognitive interviews offers critical and unique insight into cross-national question performance • Transparency is critical • Of data from interviews • Of the process for drawing conclusions • Data collection oversight • Better data management

23. Example BI chart

24. Q-Notes • On-line data entry and analysis tool • Allows for continuous oversight • Facilitates quick but thorough analysis • Designed around analysis principles

26. Next Steps Toward BI-M2Activities in 2010

27. Next Steps for the BI Questions The BI-M1 did not meet all of the objectives for the question set (some domains absent, some room for improvement). Testing to date has been focused on developing BI-M2. BI-M2 will be based on lessons learned from: • 2009 UNESCAP cognitive and field tests (reviewed here) • 2010 European cognitive tests (Round 4) • 2010 experience with U.S. National Health Interview Survey

28. 2010 Testing - Round 4Comparative Cognitive Interviewing Project Testing will return to the European region, just as in Round 2. • Spain (Spanish) • Portugal (Portuguese) • Italy (Italian) • France (French) • Germany (German) • Switzerland (French, German, Italian) • United States (English, Spanish)

29. Project Group Coalition Testing will be conducted by: • Members of the 1st comparative group (from Round 2) • Methodologists from QEM workshop at NCHS • US and Canada

30. Project Goals • Continue methodological work begun in Round 2 testing. • Narrative interviewing vs. structured probes • Interpretations of vague response categories across cultures and languages • Use of software for data collection. • Examine issues raised at QEM. • Use of mixed-method (PROMIS-WG; cognitive and field tests) • Continue examination of WG/BI extended set, specifically what questions capture. • Patterns of interpretation • Patterns of calculation • Both in-scope and out-of-scope interpretations

31. Domains for Inclusion in the Test • Cognition • Communication • Affect • Learning • Pain • Fatigue • PROMIS subset