What is the evidence that TA improves Global Fund grant performance ?

1. What is the evidence that TA improves Global Fund grant performance ? Aime De Muynck Ret. professor of Epidemiology, ITM, Antwerp, belgium 1

2. Technical Assistance (TA) definition • No generic dictionary definition of TA • However, common dimensions (focus of TA): • transfer of new knowledge along with new technology to the client • developing capacity • producing impacts at multiple levels

3. Basic TA vs. Intensive TA • Basic TA creates readiness for change and facilitates change by providing information and support Duration of basic TA = short term and episodic • Intensive TA is required when new knowledge, skills and abilities are called for Duration= long term

4. TA improves project performance = assumption (need, mission request funded, expert, report approved; thus no proof required) However, little quantitative research to prove that TA improves project/program performance = hypothesis (TBTEAM & GF) Thus: = refutable hypothesis statistical approach needed Ideally experimental design However, only field data available

5. Measuring impact of TA • Very few quantitative evaluations have been carried out to measure impact of TA • Concensus that more research and evaluation methods are needed to determine the relevant dimensions of TA and its impact on changing practices, organisations and systems • Limited scope of reported evaluations; additional data gathering; existing model; appropriate statistical models (PATH analysis, etc)

6. Assessment of TA performance : Quantitative approach 2 independent databases: TBTEAM database GF database

7. TB team database Dec 2011: 3489 records (1 record = 1 TA) of which: 1278 eliminated (not relevant; incomplete) 2211 = database denominator Minor errors : 20% either “cleaned” or “missing item” Thus denominator +/- 2000 records

8. TB team database: Data available title & purpose of mission ToRs Start & end date country organising body status of implementation comment files

9. Purposes of TAs, TBTEAM 2007-11

10. # of TAs per SDA by country

11. TAs by year

12. Duration of TAs

13. Median duration by purpose and WHO region

14. # of TAs vs. burden of disease

15. # of TAs vs. amount of GF grant/country

16. Regression model • TAs/country = -66.2 + 2.2 (partac) + 2.9 (confscore) + 11.0 (log10finsupport) Partac = # of partners active in the country Confounderscore = being a high burden country + MDR country + regional priority country + Expand TB country + TB care country + HIV-TB country

17. Rationale of TAs (planning domain)

18. GF historical grant performance database 1938 records as of Dec 2011 1 record = a baseline or follow-up assessment Available data: grantID, country, GF round, date of start, PR Type, disbursement info, initial and follow-up GF ratings, date of assessment 1033 records (from 2007 onwards) Several records missed grant performance rating Trend info from 105 grants only

19. Assessment of TA impact Three main challenges: 1. Linking the 2 databases 2. Establising the association between TA support and GF grant performance 3. Causal analysis

20. 1. Linking the 2 databases • Challenge: No common key variable • Solution: 1. Creation of a new variable, through recoding existing variable: - purpose of TA [TBTEAM database] - SDA (Service delivery area) [GF database] 2. Creation of a time variable (quarter)

21. 2. Association TA support vs. GF grant performance • Specificity of TA: the purpose of the TA support focused on at least 1 SDAs • Outcome variable : achievement score (%) per SDA • Unit of time : a quarter • Restriction to the period 2007-2011 • Comparison of achievement score in quarters with TA and without (= non exposed)

22. Mean achievement score (%) of SDAs

23. 3. Causal analysis (1) Many factors influence the performance of a grant TA is only 1 of them Ideally: proof of impact of TA through experiment In non-experimental design: control of confounding is essential

24. Causal analysis (2) Procedural approach: • Data on potential confounders (see researches in education, preventive medicine, AIDS, management) • Model of influence of TA on outcome • Robust and valid outcome indicator(s) • Statistics (path analysis, logistic regression, etc) • Analysis of selection and observation bias

25. Causal analysis in this context

26. Proof of cause in non-experimental setting 26

27. Attributable fractionrange : 6% to 23 %

28. Temporal association • For impact to be attributable to TA, TA has to occurprior to measurement of achievement score • How to measure? Achievement score differential:

29. Mean achievement score of planning SDA in consecutive quarters, in exposed and unexposed countries

30. Dose response relationshipDose measured through # of TAsResponse measured through achievement score difference between start and end of SDA activities in a given domain

31. Dose response relationshipDose measured through duration # of TA missions

32. Alternative explanations 1. Hawthorne effect = a form of reactivity whereby subjects improve their productivity, simply in response to the fact that they know they are being studied, but not in response to any particular experimental manipulation. But recent doubts on existence of this effect 2. Uncontrolled confounders: possible although unlikely

33. Bradford-Hill criteria: In summary

34. What to do next ? • Assure commonality of both databases (definitions, criteria, at least 1 common variable) • Establish validity rules • Create routine of validity checking & data cleaning • Establish causal model • Gather routine info on essential confounders • (Routine monitoring of the impact of TAs)

35. Thanks for your attention