Implementation: Pilot testing – and further development – a critical next step

1. Implementation: Pilot testing– and further development – a critical next step David F. Ransohoff MDDepartments of Medicine and EpidemiologyUniversity of North Carolina at Chapel Hill

2. Implementation steps:•certify organizations from p146,7

3. Implementation steps:•certify organizations •what’s missing: how to measure “how trustworthy” is individual CPG from p146,7

4. Howtrustworthy is a CPG? • a. How is “trustworthy” measured(scale) •existential (yes/no)•ordinal (very trustworthy, moderately..., not)•numerical • b. How are the 8 “standards”:•measured individually •‘combined,’ to decide (a) “how trustworthy”? • Doing (a) and (b) is ‘scale-making’

5. Scale-making is commonly done e.g, Apgar Score, or SF-36 These scales “components” have multiple categories. For many scales “coding manuals” are developed.

6. Howtrustworthy is a CPG? Example: What happens if we apply “standards” to CRC screening guidelines,that were a “call-out”: can we tell how trustworthy each is?

7. How trustworthy is a CPG?(CRC MSTF 2008) • ... according to a panel chair:“[T]he process ultimately became politicized and, according to one participant, resembled “sausage-making.”’ • -Cancer Letter, Oct 12, 2008

8. Howtrustworthy is a CPG? Current Instructions to apply standards

9. Howtrustworthy is a CPG? Current Instructions to apply standards Current instructions ask only: “Is everything perfect.” There’s no “threshold,” and no details for individual standards. “Pilot testing” could show whatwould user (CMS? BC/BS? Kaiser?) think or do; how would they use? And then are YOU(IOM) satisfied with how they use them? Do current standards solve the problem we’ve got in this field?

10. Why might we need a more-explicit way to measure “how trustworthy”? • 1. Even if you “certify organizations,” must also be able to judge organizations’ products. (e.g., pharmaceuticals) • 2. Many guidelines will be made by “uncertified organizations” (or do you expect all to stop?). • 3. So a user then needs to know how to judge a CPG – individual elements, ‘composite score’. • 4. #3 was IOM’s original charge. • 5. If this were a pharmaceutical, would it be “rolled out” without #3, and without testing? (Does this solve problem in field?) • 6. We need strong checks-and-balances in an environmentwhen it’s hard for professionals to be “professional.”

11. What a “profession is (via Louis Brandeis and Sam Thier) • Why Sam Thiertalked about this all the time: ‘Society grants privileges to professions (title, income)... in exchange for ‘pursued largely for others’. If a profession doesn’t act right, privileges will be taken away.

12. Example of one profession’s concerns re future • (AGA Institute Future Trends Committee conference, 2006)

13. Example of one profession’s concerns re future • (AGA Institute Future Trends Committee conference, 2006) • ‘The Doomsday Conference’ Separate example: May 9, 2011 AGA Plenary, on screening: “Disruptive Technology”

14. Checks and balances • 1. “Professional organization” wears 2 conflicting hats:1) clients’ interest comes first (Brandeis; Thier)2) professionals’ economic interest comes first • 2. Intellectual conflictexists apart from economics: people “believe in” what they do. (Fowler FJ, et al. Comparison of recommendations by urologists and radiation oncologists for treatment of clinically localized prostate cancer. JAMA 2000;283:3217)

15. Checks and balances • 3. Lessons from evolution of “composition” of ACS-MSTF(Multi Society Task Force)•# generalists/methodologists in MSTF author group 1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594

16. Checks and balances • 3. Lessons from evolution of “composition” of ACS-MSTF(Multi Society Task Force)•# generalists/methodologists in MSTF author group 1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594Why, btw, was MSTF formed? (Sid W; AGA; May 9)

17. Checks and balances • 3. Lessons from evolution of “composition” of ACS-MSTF(Multi Society Task Force)•# generalists/methodologists in MSTF author group 1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594 2003: 2 (RHF, SHW) Gastroenterology 2003;124:544 2008: 0 Gastroenterology 2008;134:1570

18. Checks and balances • 3. Lessons from evolution of “composition” of ACS-MSTF(Multi Society Task Force)•# generalists/methodologists in MSTF author group 1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594 2003: 2 (RHF, SHW) Gastroenterology 2003;124:544 2008: 0 Gastroenterology 2008;134:1570 • The Task Force was “multisociety”; but over time, dominated by subspecialists and lost all generalists/methodologists. • Each iteration of the guidelines was “less trustworthy”; by2008 “very not trustworthy”. • Evolution reflect “inclination”, and power of economic hat. • Anecdotes about guidelines-making are ugly, disturbing, and cleverly hidden. Not ‘good faith’.

19. Checks and balances • The challengeGuidelines that are in clients’ interests:•really need the expertise hat•really need to avoid the economic hat. • Do current standards deal strongly-enough with this tension? I think not.

20. Conclusion and Suggestion • 1. IOM (someone) needs to further develop detailsof standards and of implementation, to provide effective checks-and-balances. • 2. Jefferson and Madison didn’t do their work “suddenly.” It required time/experience to develop checks and balances. Can we establish “process”, mechanism to test, develop standards, “trustworthy” and see if they look like they are solving problem? Who does this? • 3. Guidelines-making is a central issue in “professionalism” in our era; a neutral, professional, authoritative group (IOM?) needs to do/direct this. Can anyone else?