Repository Optimization and TAR PM index rebuild in clustered Weblogic servers

1. Repository Optimization and TAR PM index rebuild in clustered Weblogic servers Refeng Wu CQ5 WCM System Administrator

2. CQ5 - Repository • Database -> Repository • Huge size; maintenance issue • Keeping the clean repository is one of the main task for CQ5 system administration • Challenges: • Third party application server • Websphere 7, JBoss 5.1, Oracle Weblogic 10.3.5, Tomcat 6 • Clustered environments • LDAP integration

3. Challenges • Installation and configuration adjustments • Performance Tuning • Very time-consuming Maintenance process - Repository is very big • Slave repository out-of-syn with Master repository • Server maintenance / catastrophic failure, and could not be restarted smoothly • LDAP issue: • Author instance cannot login, system frozen • Publish instance directed to geometrixx login page • Log Errors: seldom with right solutions

4. Database Maintenance • Monitoring databases: Monitoring the state of the database ensures -- Availability of the Database; Better Performance. • Backup and recovery: To prevent the potential loss of data, set up a backup and recovery plan -- scheduling regular backups; special backup after large data is imported/modified. • Performance Tuning • Reorganizing and generating database statistics: The DB optimizer requires statistical metadata info about tables & the best way to access data. • Re-creating all indexes: to improve query performance when many existing indexes require reorganization. • Managing database connections: define max connection pools for each of services to optimize the balance between resources and requests. • Purging performance profiling data: to restore free disk space and speed up database utilities. • Dropping temporary tables: drop all data from temporary tables/indexes that hold data due to failed job process. • Old object versions: check for old object versions and delete from database tables to restore storage space, enhance performance, and speed up database utilities.

5. Repository maintenance(Database + file system maintenance) • Proactive solution • Check up the log file daily • Final solution • Repository/instance backup daily • Daily Maintenance for Performance Improvement • Indexing and Optimization • Enable hierarchy check • Validate change-set hierarchy • Setup LDAP log • Schedule consistency check • Test clustered PROD repository in QA • Rebuild Index • Running Garbage Collection

6. Repository Optimization • Tar Files: append-only file, disk usage increases • copies data being used from old tar files into new tar files; • deletes the old tar files that contain only old or redundant data • Node parent-child inconsistency • Search index inconsistency • Optimization process: • Merging index files • IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/version • IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/tarJournal • IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/workspaces/crx.default • Rename TarFile: • Renaming /app/CQ5/Author/crx-quickstart/repository/version/optimize.tar to /app/CQ5/Author/crx-quickstart/repository/version/optimizeNow.tar • Optimize TarFile: • /app/CQ5/Author/crx-quickstart/repository/version/data_00331.tar id:331 length:144674816 append:144673792 • Merging index files: • Merging index files for /app/CQ5/Author/crx-quickstart/repository/version ……

7. Indexing • Lucent Index: node content full-text search, resource-consuming, time-consuming • /repository/repository/index • LuceneSearch Index for jcr:system (includes jcr:versionStorage) • /repository/workspace/crx.default/index • LuceneSearch Index for crx.defaultworkspace • TAR PM Index: node index • /repository/workspace/crx.default/index*.tar • Tar PM Index for crx.defaultworkspace • /repository/version/index*.tar • Tar PM Index for version workspace • /repository/tarJournal/index*.tar • Tar PM Index for the cluster journal for Journal PM in 5.4

8. Consistency Check • Consistency Check: repository.xml, workspace.xml • Add parameters to all <SearchIndex...> elements • Add parameters to all <PersistenceManager...> elements • Not solving all corrupted nodes • couple nodes, or too many nodes to be fixed • Index Rebuild • Removing index.tar files from three folders in repository • Do it together with Consistency Check during repository startup • 8G takes less than 30 minutes.

9. Rebuild Clustered PROD Repository • Moving Clustered repository to in QA or Devenvironment • Modify context configure in each server / domain • Weblogicserver bootstrap.properties • Modify cluster configure • change slave cluster.propertyip address • Test the whole process in QA or Dev with PROD Repository • Be prepared for possible results: master fails, slave fails – copy from master, recovery from zip file • Get a solution for each result • Maintenance window and anticipated hours for use

10. LDAP setup • Enable LDAP log info • <logger><appender name="ldap" class="org.apache.log4j.FileAppender"> </logger> • /crx-quickstart/server/etc/Ldap_login.config • Author instance - ldapgroup filter: • (&(objectclass=group)(cn=wcm*)) • Publish instance: • remove group synchronization vs Close User Group • *WARN * ItemSaveOperation: /home/groups/COM/ABB/PROD/CN=Adobe Pro Users,OU=Corporate,OU=Groups,DC=PROD,DC=ABB,DC=COM/rep:lastsynced: failed to restore transient state (ItemSaveOperation.java, line 897)

11. Q&A