How Does Everyone Else Do This? Search, Metadata, and Taxonomy Practices

1. How Does Everyone Else Do This?Search, Metadata, and Taxonomy Practices Ron Daniel, Jr.

2. Organizational Benchmarking • A common goal of organizations is to ‘benchmark’ themselves against other organizations. • Different organizations have: • Different levels of sophistication in their planning, execution, and follow-up for CMS, Search, Portal, Metadata, and Taxonomy projects. • Different reasons for pursuing Search, Metadata, and Taxonomy efforts • Different cultures • Benchmarks should be to similar organizations.

3. Is Unnecessary Capability Harmful? • Tool Vendors continue to provide ever-more capable tools with ever-more sophisticated features. • But we live in a world where a significant fraction of public, commercial, web pages don’t have a <title> tag. • Organizations that can’t manage <title> tags stand a very poor chance of putting an entity extractor to use, which requires some ongoing management of the lists of entities to be extracted. • Organizations that can’t create and maintain clean metadata can’t put a faceted search UI to good use. • Unused capability is poor value-for-money. • Organizations over-spend on tools and under-spend on staff & processes.

4. Towards Better Benchmarking… • Wanted a method to: • Generally identify good and bad practices. • Help clients identify the things they can do, and the things that stand an excellent chance of failing. • Predict likely sources of problems in engagements. • We have started to develop a Metadata Maturity Model, inspired by Maturity Models from the software industry. • To keep the model tied to reality, we are conducting surveys to determine the actual state of practice around search, metadata, taxonomy, and supporting business functions such as staffing and project management.

5. Goals for this Talk • Provide you with background on maturity models • Provide the results of our surveys of Search, Metadata, & Taxonomy practices and discuss interesting findings • Give you the tools to do a simple self-assessment of your organization’s metadata maturity

6. A Tale of Two Software Maturity Models • CMMI (Capability Maturity Model Integration) • vs. • The Joel Test

7. CMMI Structure • Maturity Models are collections of Practices. • Main differences in Maturity Models concern: • Descriptivist or Prescriptivist Purpose • Degree of Categorization of Practices • Number of Practices (~400 in CMMI) Source: http://chrguibert.free.fr/cmmi

8. 22 Process Areas, Keyed to 5 Maturity Levels… • Process Areas contain Specific and Generic Practices, organized by Goals and Features, and arranged into Levels • Process Areas cover a broad range of practices beyond simple software development • CMMI Axioms: • Individual processes at higher levels are AT RISK from supporting processes at lower levels. • A Maturity Level is not achieved until ALL the Practices in that level are in operation.

9. CMMI Positives • Independent audits of an organization’s level of maturity are a common service • Level 3 certification frequently required in bids • “…compared with an average Level 2 program, Level 3 programs have 3.6 times fewer latent defects, Level 4 programs have 14.5 times fewer latent defects, and Level 5 programs have 16.8 times fewer latent defects”. • Michael Diaz and Jeff King – “How CMM Impacts Quality, Productivity,Rework, and the Bottom Line” • ‘If you find yourself involved in product liability litigation you're going to hear terms like "prevailing standard of care" and "what a reasonable member of your profession would have done". Considering the fact that well over a thousand companies world-wide have achieved level 3 or above, and the body of knowledge about the CMM is readily available, you might have some explaining to do if you claim ignorance’. Linda Zarate in a review of A Guide to the Cmm: Understanding the Capability Maturity Model for Software by Kenneth M. Dymond

10. CMMI Negatives • Complexity and Expense • Reading and understanding the materials • Putting it into action – identifying processes, mapping processes to model, gathering required data, … • Audits are expensive • CMMI does not scale down well to small shops • Has been accused of restraint of trade

11. At the Other Extreme, The Joel Test Developed by Joel Spolsky as reaction to CMMI complexity Positives - Quick, easy, and inexpensive to use. Negatives - Doesn’t scale up well: Not a good way to assure the quality of nuclear reactor software. Not suitable for scaring away liability lawyers. Not a longer-term improvement plan. The Joel Test Do you use source control? Can you make a build in one step? Do you make daily builds? Do you have a bug database? Do you fix bugs before writing new code? Do you have an up-to-date schedule? Do you have a spec? Do programmers have quiet working conditions? Do you use the best tools money can buy? Do you have testers? Do new candidates write code during their interview? Do you do hallway usability testing? Scoring: 1 point for each ‘yes’. Scores below 10 indicate serious trouble.

12. What does Software Development “Maturity” Really Mean? • A low score on a maturity audit DOES NOT mean that an organization can’t develop good software • It DOES mean that whether the organization will do a good job depends on the specific mix of people assigned to the project • In other words, it sets a floor for how bad an organization is likely to do, not a ceiling on how good they can do • Probability of failure is a good thing to know before spending a lot of time and money

13. Towards a Metadata Maturity Model • Caveats: • Maturity is not a goal, it is a characterization of an organization’s methods for achieving its core goals. • Mature processes impose expenses which must be justified by consequent cost savings, revenue gains, or service improvements. • Nevertheless, Maturity Models are useful as collections of best practices and stages in which to try to adopt them.

14. Basis for Initial Maturity Model • CEN study on commercial adoption of Dublin Core • Small-scale phone survey • Organizations which have world-class search and metadata externally • Not necessarily the most mature overall processes or the best internal search and metadata • Literature review • Client experiences • Structure from software maturity models

15. Initial Metadata Maturity Model (ca. May, 2005) 37 Practices, Categorized by Area, Level, and Importance

16. Shortcomings of the Initial Model • No idea of how it corresponds to actual practice across multiple organizations • Some indications that it over-emphasized the sophisticated practices and under-emphasized beginning practices. • The initial metadata maturity model can be regarded as a hypothesis about how an organization progresses through various practices as it matures • How to test it? Let’s ask! • Two surveys to date • Surveys are being run in stages because of large number of practices. • Ask about future, current, and former practices to gather information on progression

17. Survey 1: Search, Metadata, & Taxonomy Practices • The data in this section comes from a survey conducted in the autumn of 2005.

18. Participants by Organization Size

19. Participants by Job Role

20. Participants by Industry

21. Search Practices

22. Metadata Practices These two questions were the only ones with much correlation to organization size

23. Taxonomy Practices

24. Notes from Participants • Validating metadata schema and vocabularies against metadata use scenarios Defining change management and governance policies for both schemas and controlled vocabularies. [Would be helpful if you clarified the definition of taxonomy you are using in the last question, not clear if it's intended to be navigational/browse structures, or controlled vocabularies/thesauri.] • We use a Wiki with categories and page specific tags to locate interesting stuff in our corporate memory • The e-commerce division has a taxonomist on staff who analyzes the configuration of the Verity search on the e-commerce web site. She maintains a taxonomy and works with programmers to change search parameters to optimize the user search. • We have GIS (geographic Information systems) metadata editor and the data is stored in Oracle DB (SDE) together with each stored layer • Approaches to metadata QA an enormous, currently unadressed problem. Gov of [X] has minimum level of compliance required to common metadata scheme (dublin core sub set). Convincing individual depts of value to move beyond minimum, and to evalute quality of what has been done and exploit it and move beyond is sucessfull only in isolated pockets. Central Agencies [X] try but have no stick to use to enforce. Mostly a bottom up effort from an increasingly weary group of sloggers. • Change Control Board Site Registration System being developed • 1) Standard corp taxo (products, capabilities, industries, doc type) drives tagging (from doc repository) to serve up docs/collateral to appropriate pages on internal and external web presences, and to CRM so sales reps can retrieve them through that interface. 2) Standard Corp taxo is used by CMS, doc repository and CRM.

25. Survey 2: Business Drivers, Processes, and Staffing • The data in this section comes from a survey conducted in the spring of 2006.

26. Participants by Job Role

27. Participants by Tenure

28. Participants by Industry

29. Participants by Organization Size

30. Business Drivers: SMT Applications

31. Business Drivers: Desired Benefits Other desired benefits:

32. ROI: Cost Estimation

33. Processes Use of search logs is improving Surprisingly sophisticated Basic data quality and communications need improvement Many solo operators

34. Team Structures & Staffing

35. Salary Survey

36. Notes from Participants (1) • There is the constant struggle with individual [magazine] titles to hire trained librarians or data specialists instead of trying to save money by hiring an editor who can build articles AND create and assign metadata. This is a governance issue we have been struggling with since we have no monetary stake in the individual publications. We make recommendations, but have no higher level authority to require titles to hire trained staff for metadata. • Reporting metrics have become a new area of confusion as we move to portalized pages consisting of objects in portlets, each with their own metadata. • Key organizational issue is that the "problems" that stem from lack of systematic metadata/taxonomy creation are not "owned" by anyone, and consequently have no budget for their solution.

37. Notes from Participants (2) • This is an ad hoc effort, initiated by me, to address upcoming requirements as we move our authoring from a narrative form to a topic-based form.

38. Interim Conclusions

39. Observations (1) • Practices which a single person or a small group can carry out are more commonly used • Not surprising • Very different than ERP/BPR, indicates that information management is not being sold to the “C-level” staff. • People need to question how inclusive their “Organizational Metadata Standards” and “Taxonomy Roadmaps” actually are. • We have found Taxonomy Roadmaps to be an advanced practice, due to a dependence on knowing upcoming IT development schedule

40. Observations (2) • Many of the basics are being skipped • More organizations doing “Spell Checking” than “Query Log Analysis”. • 69% have a taxonomy change plan, but only 41% have a plan for revisiting data if the taxonomy changes. • 45% don’t have a website, but only 36% don’t have a communications plan. • This seems to be linked to the previous observation – things that are easy for an individual get done before things that need an organizational effort, despite their level of ‘sophistication’.

41. Search and Metadata Maturity Quick Quiz • Basic • Is there a process in place to examine query logs? • Is there a process for adding directories and content to the repository, or do people just do what they want? • Is there an organization-wide metadata standard, such as an extension of the Dublin Core, for use by search tools, multiple repositories, etc.? • Intermediate • Does the search engine index more than 4 repositories around the organization? • Does the search engine integrate with the taxonomy to improve searches and organize results? • Are there hiring and training practices especially for metadata and taxonomy positions? • Is there an ongoing data cleansing procedure to look for ROT (Redundant, Obsolete, Trivial content)? • Are tools only acquired after requirements have been analyzed, or are major purchases sometimes made to use up year-end money? • Advanced • Are there established qualitative and quantitative measures of metadata quality? • Can the CEO explain the ROI for search and metadata?

42. Interim Metadata Maturity Model (ca. May, 2006)

43. Next Survey: Tools, Metrics, and Executive Support • What do you want to know about what other organizations are doing in these areas? • Want to know if certain practices at your organization are in place at other organizations? • Send suggestions to rdaniel@taxonomystrategies.com and/or seth@earley.com

44. Recommended Reading • CMMI:http://chrguibert.free.fr/cmmi (Official site is http://www.sei.cmu.edu/cmmi/, but that is not the most comprehensible.) • Joel Test • http://www.joelonsoftware.com/articles/fog0000000043.html • EIA Roadmap • http://www.louisrosenfeld.com/presentations/031013-KMintranets.ppt • Enterprise Search Report • http://www.cmswatch.com/EntSearch/

45. Contact Info Ron Daniel, Jr. 925-368-8371 rdaniel@taxonomystrategies.com