Documentum xPlore: Fundamentals and Troubleshooting

1. Documentum xPlore: Fundamentals and Troubleshooting Chad Peek Designated Support Engineer Worldwide Technical Support EMC – Information Intelligence Group

2. xPlore: Fundamentals and Troubleshooting • Dial-in numbers: • U.S. Dial-in numbers(toll free) 888-643-3084 (toll): 857-207-4204 • Passcode: 34856332 • Country-specific dial-in numbers: http://www.emcconferencing.com/globalaccess/index.asp?bid=302 • Click the handouts icon to download the presentation. • Separate Q&A session after the webinar • The webinar is being recorded • Follow us at: http://community.emc.com/blogs/iigsupportwebinars

3. Agenda • Terminology • Overview of xPlore • Architecture and Dataflow • Installation and Migration • Troubleshooting • Resources

4. Terminology • Instance: An xPlore instance is one deployment of the xPlore WAR file to an application server container. You can have multiple instances on the same host (vertical scaling), although it is more common to have one xPlore instance per host (horizontal scaling). • Domain: A separate, independent group of collections with an xPlore deployment that maps to a single Documentum repository.

5. Terminology • Collection: A logical group of XML documents that is physically stored in an xDB library. It represents the most granular data management unit within xPlore. • Category: Defines how a class of documents is indexed. The definition specifies the processing and semantics that is applied to an ingested XML document. • XQuery: A language for finding and extracting elements and attributes from XML documents and anything that can appear as XML including databases.

6. Agenda • Terminology • Overview of xPlore • Architecture and Dataflow • Installation and Migration • Troubleshooting • Resources

7. Documentum xPlore Overview • DocumentumxPlore is the next generation search for Documentum • FAST is no longer supported. Existing implementations must migrate to xPlore. • Technology Foundation: EMC xDB (native XML database) and Lucene • Compatible with D6.5 SP2 and later • Client's supported by D6.5 SP2 will work without change (with a few exceptions) • Dual mode migration • xPlore & FAST both active (index & query) on the same repository

8. New Features in xPlore 1.2 • 1.2 is the latest version (Dec 2011) • Thesaurus Support • Synonyms, alternate spellings, acronyms • Indexing and query performance • Wildcard performance improvements • Automatic query warmup feature • Language Certifications • Russian, Arabic, Hebrew, and Brazilian Portuguese • Administration Features • Silent installer

9. Agenda • Terminology • Overview of xPlore • Architecture and Dataflow • Installation and Migration • Troubleshooting • Resources

10. Architecture Content Server Index Agent Index Server Repository xDB/Lucene CPS

11. High Level Data Flow 2. Index Agent polls the server for changes. Content Server Index Agent 3. IA transforms metadata and content into XML that the Index Server can process. 1. Content is added to or modified in the repository. 5. IA reports on indexing status. Index Server Repository xDB/Lucene CPS 4. IS processes the XML document. 6. Content Server queries the index (DFC/plugin).

12. Fulltext Objects dm_ftengine_config dm_ftindex_agent_config dm_fulltext_index dm_ftfilter_config dm_ftwatermark For indexing, events like dm_save, dm_checkin and dm_destroy trigger the creation of queue objects dmi_queue_item Monitored by the IA Content Server Content Server Repository

13. These events are registered for object types. By default: dm_sysobject, dm_acl, dm_group and dm_folder All the subtypes of dm_sysobject inherit indexing setting Indexing dm_acl and dm_group allow xPlore to enforce security The dm_folder type must be indexed for location based queries (e.g. FOLDER DESCEND) Enabling and disabling indexing for object types You can use DA to enable or disable indexing. The following query can be used to quickly review the registered events: select registered_id, event, user_name from dmi_registry where user_name like 'dm_fulltext_index_user%‘ The Content Server uses XQuery to query the xPlore index Query plugin is used pre 6.6 Content Server (Cont.)

14. Uses DFC to interact with the CS and the xPlore API to interact with xPlore Migration Mode Repository is indexed in batches using the dm_ftwatermark object Normal Mode Polls dmi_queue_item table for new entries dm_fulltext_index_user Selective Indexing DQL query Object file (a list of r_object_id’s) Index Agent Index Agent Index Server xDB/Lucene CPS

15. Shared Content Storage By default the IA pulls the content from the file system. Mapping the filestore in the indexagent.xml allows CPS to have direct read access which can improve performance. The content storage area must be mountable as read-only by the Index Agent and xPlore hosts. Filters Exclude objects from the index by cabinets, folders or object types. Filters must be in place before migration. Index Agent (Cont.)

16. Components EMC xDB Lucene CPS xPlore Administration Tool Enable/disable auditing Monitor and increase logging OOTB reports Testing features • Index Server Index Agent Index Server xDB/Lucene CPS

17. xDB EMC’s XML database Tracks indexing and update requests Records the location of indexed content Lucene Open source tool that is embedded into xDB Performs query lookup and retrieval of facet and security information CPS (Content Processing Services) Retrieves indexable content from content sources Determines the format and language Parses the content into index tokens • Index Server (Cont.)

18. Using DQL hints with 6.6 and 6.7 clients Newer versions of DFC automatically generate XQuery Turn off XQuery generation by adding the following setting to dfc.properties file: dfc.search.xquery.generation.enable=false Things to Remember

19. Agenda • Terminology • Overview of xPlore • Architecture and Dataflow • Installation and Migration • Troubleshooting • Resources

20. Installation Overview • Read the entire manual before starting the install • Are you migrating or is this a fresh install? • If you are you using FAST, will you use dual mode migration? • Understand which set of steps apply to your situation. • Work through the sizing calculator and the other performance tools available on the EDN • https://community.emc.com/docs/DOC-8922 • Typically you can expect xPlore to use the same resources as FAST, but you should still work through the sizing exercise. • Use the latest xPlore patch • Match xPlore patch with Content Server patch

21. Migration Overview • Migration Options • Immediate replacement of FAST • Variations of the dual mode feature • Dual Mode • Migration feature for customers currently running FAST • Allow you to run FAST and xPlore at the same time for the same repository • Provides simultaneous indexing and query execution (in some cases)

22. Benefits End-user experience remains unchanged. Validation of xPlore index can occur without disrupting business critical processes. Can slowly migrate users over from FAST to xPlore and eventually turn off FAST Minimizes disruption to business and can be done with limited downtime Drawbacks of Dual Mode Added complexity Could potentially require additional hardware Dual Mode

23. Single Content Server and client, both FAST and xPlore actively indexing. Queries are sent to one IS. Advantages No additional Content Servers or clients required Easily toggle query execution between xPlore and FAST Disadvantage Requires a restart of the repository to toggle query target Dual Mode Migration: Single Content Server IA IA CS Webtop IS IS

24. During maintenance window toggle from FAST to xPlore ftengine_to_use parameter in the server.ini Validate the index Execute queries Run FTStateOfIndex job Toggle back to FAST if needed Dual Mode Migration: Single CS Validation IA IA CS Webtop IS IS

25. Two Content Servers and two client, both FAST and xPlore actively indexing. Queries sent to FAST index and xPlore index. Advantage No toggling for query Disadvantages Additional CS and client More maintenance Dual Mode Migration: Two Content Servers IA IA CS 1 CS 2 Webtop 1 Webtop 2 IS IS

26. Testers use Webtop 1 to validate searches against the xPlore index. Webtop 1 connects to CS1 which is configured to send queries to xPlore. ftengine_to_use Slowly migrate users. Once validated, turn off FAST and the extra CS and Webtop instance can be removed. Dual Mode Migration: Two CSs Validation IA IA CS 1 CS 2 Webtop 1 Webtop 2 IS IS

27. Repository A Repository B Migration Considerations • A single xPlore deployment can be shared by multiple repositories. Data is separated into domains. • Ties the future upgrades of the repositories at the hip • By default, queries utilize native xPlore security [RECOMMENDED] • ACLs & Groups replicated as part of IA Re-indexing operation • All configuration options in terms of filters, special character configuration, etc. should be done before the migration to avoid re-indexing. Consolidated Deployment Index Agent Index Agent xPlore

28. Do not run setup scripts twice. Switch from migration mode to normal mode with IA completes the initial indexing of the repository. Stay current with the xPlore software if possible xPlore 1.2 P01 is targeted for release at the end of February. Patches are generally available for all xPlore versions at the end of each month. Things to Remember

29. Performance test your environment using Production data Use the sizing calculator and other tools available on the EDN https://community.emc.com/docs/DOC-8922 Do NOT mix migration and Production usage. Sending queries to xPlore during migration will slow down the process. The resources required during migration are greater than the resources required for normal indexing in most cases. FAST was I/O intensive and xPlore is more CPU intensive Add more CPU’s during and then remove them afterwards SAN’s generally provide the best performance. Things to Remember (Cont.)

30. Agenda • Terminology • Overview of xPlore • Architecture and Dataflow • Installation and Migration • Troubleshooting • Resources

31. Troubleshooting Strategy • Strategy to help troubleshoot the most common indexing issues • Start the indexing process with a test document • Follow the document through the indexing process all the way to the client • Identify the problematic component • Use troubleshooting concepts specific to the component

32. Import New Object Execute the following query to view the queue: select item_name, item_id, task_state, date_sent from dmi_queue_item where name like 'dm_fulltext_index_user%’ order by date_sent desc dmi_queue_item entry does not exist This issue is CS-related Is the type registered properly? Use DA to check the type’s configuration Content Server Content Server Repository

33. If the queue object exists, what is the task_state? A blank state means the object is waiting to be processed Refresh the query Are other dmi_queue_item objects building up in the queue as you continue to execute the query? If the task state never changes, the Index Agent is source of the issue Is the IA started? Check the log if it will not start Is the IA running in migration mode? Index Agent Index Agent Index Server xDB/Lucene CPS

34. When the IA starts to process a queue object the task state will change There are multiple task states Acquired: Object is being processed Warning: Failed to index the content, but the metadata was indexed Metadata will be searchable but not the content Failed: Indexing failed If the object is successfully indexed the IA will remove the queue object For failures or warnings, check the Index Agent log and the Index Server logs dsearch_home\jboss5.1.0\server\DctmServer_Indexagent\logs dsearch_home\jboss5.1.0\server\DctmServer_PrimaryDsearch\logs Index Agent and Index Server

35. Error messages in the log should give you a good indication of the root cause File type is not indexable Content is larger than current settings allow The xPlore Administration and Development Guide has a troubleshooting section that lists common errors. The Search Development Guide has a troubleshooting section as well that is specific to search Powerlink is also good resource for finding information related to the error codes. Index Server: Test Index Agent Index Server xDB/Lucene CPS

36. In some cases the queue object will be removed as if indexing was successful but you are not able cannot find the document Use the ‘Test Search’ option in the xPlore Admin tool to confirm whether or not the object exists in the index search using object’s r_object_id Index Server: Test

37. If you see a result your focus should shift to the configuration of your Documentum environment and your client applications Does the user searching for the content have access to the object Check the repository log to see if the query plugin was loaded Mon Jun 14 21:53:50 2010 031000 [DM_FULLTEXT_T_QUERY_PLUGIN_VERSION]info: "Loaded FT Query Plugin: ...C:\Documentum\product\6.5/bin/DSEARCHQueryPlugin.dll Is the associated format object fulltext enabled? Configuration and the Client

38. Is the client using a fulltext query Using Webtop review the query generated by your search On the search results page, the native query can be viewed by clicking on the magnifying glass as shown below If the native query is DQL then fulltext searches could be disabled Check the following in your dfc.properties files dfc.search.fulltext.enable = false • Configuration and the Client

39. Agenda • Terminology • Overview of xPlore • Architecture and Dataflow • Installation and Migration • Troubleshooting • Resources

40. xPlore Landing Page on EDN: https://community.emc.com/docs/DOC-8945 xPlore Tools (sizing calculator, Bonnie, etc) https://community.emc.com/docs/DOC-8922 xPlore HA Setup and Administration https://community.emc.com/docs/DOC-10111 https://community.emc.com/docs/DOC-10978 Search for ‘xplore’ for several helpful articles, videos and presentations. Posted on the Developer Network

41. Other Resources • xPlore Administration (http://education.emc.com) • How to install, configure, and maintain a Documentum xPlore full-text Index Server and Index Agent • Several xPlore related white papers including migration, disaster recovery and best practices • http://powerlink.emc.com/km/appmanager/km/secureDesktop?_nfpb=true&_pageLabel=default&internalId=0b0140668053a3a8&_irrt=true • xPlore specific offerings from Professional Services