Topics

1. NGA Public Web Site Redesign – CQ5 Lessons LearnedNovember, 2012 (David Beaudet, NGA Lead Developer / Architect)

2. Topics • Technical Architecture • Art Data and Image Integration • Other Useful Nuggets

3. NGA CQ5 Technical Architecture • Installation / Upgrade Lessons Learned • CQ 5.4 to 5.5 is a major change – everything is OSGi – some configs buried and some features no longer supported, e.g. RMI no longer seems like an option. • Java 1.7 not supported by CQ 5.5 – unsure about 5.6 – Adobe claims Oracle has a problem in 1.7 that needs fixing • Always disable virus scanner prior to CQ5 install, particularly McAfee, as CQ5 initial install will not complete if McAfee on-access protection is enabled. Maybe run DEV, QA, and Prod on Linux to avoid having to install a virus scanner ? • webDAV doesn’t seem to be completely supported; compounded by Windows 7 shipping with broken webDAV client. • Lucene index occasionally misses nodes – limited to DEV / QA systems? Reindexing may be an option – we have succesfully reindexed. • Start / stop scripts need modification to work properly on Linux • 5.6 seems to have some user interface screens missing, e.g. under Tools. We’re sticking with 5.5 SP 2.1 for now – any opinions about 5.6? • After fresh install, one finds quickly that default memory is insufficient

4. NGA CQ5 Technical Architecture • CQ5 Server Configuration and Operational Concerns • one RHEL 6 VM guest for each of DEV and QA (16 GB RAM / 4 CPUs) • one RHEL 6 VM guest each for prod author and prod publish (16 GB / 4 CPUs) • Lessons Learned • allocate plenty of memory to the JVM for each instance (we use between 4 and 8 GB depending) – use 64 bit operating system otherwise processes are limited to 4GB each • allocate plenty of storage up front – CQ doesn’t perform JCR garbage collection by default – CQ5 halts itself when insufficient storage space is detected – 100GB min • change the default temp directory of CQ5 in start script configuration to a volume with plenty of space • tar persistence is highly I/O intensive - use fastest reliable storage – anyone using SSD? We are using iSCSI on 1 Gbps network • Java patches need to be managed – system library links are severed as part of RHN updates which can lead to CQ5 errors if CQ5 is left running during the patch. • Schedule datastore garbage collection and use incremental backups to minimize impact of backups

5. NGA CQ5 Technical Architecture • CQ5 Clustering • Set up is easy, but result is risky (CQ5 admin class) and expensive • When clustering, understand the types of clustering available and make the right choice based on your requirements (e.g. shared JCR files or synchronized copies). • Conclusions • Frequently, the complexities inherent in software clusters create more downtime than clusters protect against. For small shops especially, that’s almost always the case. • Active / active requires two CQ5 licenses; active / passive might not (ask Adobe).

6. NGA CQ5 Technical Architecture • Web Server Configurations • NGA uses CQ5 behind Apache – HTTPS for author, both schemes for publish – dispatcher only used for publish. • Lessons Learned • Configure mod_proxy to prevent Apache from becoming an open proxy server • SSL configuration in CQ5 requires installing a server certificate into the Java keystore. Unfortunately, Oracle doesn’t include the full set of CA certs that browers do, so your CA’s cert might have to be added to the Java configuration. Moreover, Java updates can replace the keystore which means you have to reinstall the CA cert every time Java is updated. When getting your server’s CSR signed, select a vendor whose CA cert is included out of the box in Java. • Consider creating Apache and dispatcher configuration file templates and automating the creation and distribution of the configs to avoid differences between environments. • Use Apache rather than CQ5 for index page redirection; generally speaking, minimize the number of requests that have to be sent to CQ5 publish instance • Urge developers to test through DEV dispatcher before requested code promotion as filter rejects are easy to identify; create a CGI to expose a long tail of DEV author, DEV publish, and DEV dispatcher logs so developers can see errors.

7. NGA CQ5 Technical Architecture • Dispatcher Configuration • NGA using ver 4.1.2, but 4.1.3 available for download • Lessons Learned • Dispatcher is an application firewall, so it’s probably best to filter out by default and include only the URL patterns that are needed • Check for the text “reject” in dispatcher.log to identify requests that were rejected by the dispatcher filters • Differentiating between the scheme (http vs. https) of the original browser request is impossible to do when proxying the connection to CQ (either with mod_proxy or dispatcher) because the scheme reported by JSP is the scheme of the proxy, not the browser. • Latest dispatcher (4.1.3?) claims to support native HTTPS without require stunnel. Even so, the latest dispatcher still might not support multiple configurations within a single Apache instance, so you might have to install two separate Apache instances on the same machine to detect the scheme properly. • The dispatcher options for flushing the cache are quite limited – you might have to write your own CGI to intercept dispatcher flush requests and handle them yourself – or go with a very small number of stat file levels – we’re still figuring this out.

8. NGA CQ5 Technical Architecture • Author Instance Authentication • NGA looked briefly at SSO with Apache / mod_auth* modules - we use it for other web-based systems; the security model (trusted credential) has been deprecated due to potentially serious security problems • NGA opted for LDAP “out of the box” / we will be transitioning to LDAPS now that it’s working over unsecured LDAP. • Lessons Learned • CQ5 trusted credentials attribute is being phased out as indicated in JackRabbit log files – don’t depend on it - unclear what, if any, mechanism will replace it. • That said, Kerberos (or most any other kind of Apache supported authentication) can be integrated with Apache simply by setting a certain HTTP header in the request after Apache authentication is performed. However, there are security issues with this since headers can be spoofed easily – under current SSO, you must not expose CQ5 instances outside of the local machine. • LDAP (prior to CQ 5.6 I believe) has a bug related to case sensitivity of user names – it’s fixed by a patch – OR – users need to kill the session cookies that are created after their initial login using LDAP.

9. NGA CQ5 Technical Architecture • Publish Instance Authentication • We support native CQ5 users and social users for public authentication • Native CQ5 users are the same users used by the WCM tools, so be sure the default group assignments do not include WCM permissions • oAuth uses the deprecated trusted credentials attribute – so error log fills with warnings about deprecation of trusted credentials attribute • Lessons Learned • oAuth is not a trivial endeavor – there are lots of details and configurations to keep up with plus you’ll be writing some custom code to secure it anyway. • Create specific logging configuration for the oAuth module to send repetitive deprecation warnings to a separate log file or to /dev/null • CQ5.5 SP 2.1 breaks oAuth due to ACL on /etc/cloudservices – you have to adjust it to permit read to /etc/cloudservices/facebook and twitter directories. • HTTPS vs. HTTP and cookie security – might have to get Apache involved to logout users if unsecured cookie transmission is detected – we’re still working on this.

10. NGA CQ5 Technical Architecture • Secure URLs • Need HTTPS for certain form submissions • Avoid abuse of forms that generate e-mail • Lessons Learned • Force HTTPS using Apache and / or with JSPs (if possible) – still figuring out • Use captcha to avoid getting on an e-mail blacklist • User Form Submissions • Need removal of data from publish instances due to gov’t privacy rqmnts • Lessons Learned • Relatively straightforward workflows and workflow launchers can achieve this (auto-reverse-replication and content deactivation); form authors need training • Potential security issue with out of the box form handler (path specification for form element name) – need to lock down permissions for anonymous user and public authenticated users

11. NGA CQ5 Art Data and Image Integration • NGA Art Data – key content for the web site • 110k+ art objects, 20k artists, other entities – highly complex relationships stored in external relational database tables • Redesigned web site has complex art data searches, sort orders, and faceting of art data search results • Data must be synchronized – Art object data changes regularly and new objects are added nearly every day Images of Art Objects – key to a visually rich web site • Images of the art – zoom files are very large, multiple sizes are required, color profile management, specific algorithms required for resizing, must support addition of manually cropped renditions • Image associations with art objects are managed in a different system • Images must be synchronized – images change regularly and new images are added frequently

12. NGA CQ5 Art Data and Image Integration • For art object data, we attempted to build a prototype that would feature: • automatic synchronization of data between relational database and JCR • a JCR content hierarchy able to accommodate 1 million+ art data entities and relationships between them without sacrificing data richness or overloading the JCR • Efficient JCR SQL2 queries that return distinct result sets in a highly customized sort order along with custom facets Findings • Relational database to JCR mapping isn’t straightforward. We realized that in order for JCR queries to perform well and in order to avoid duplicate search results, we would have to store multiple copies of the art object data in multiple JCR hierarchies, possibly many depending on the specific queries. • Custom facet extractors in CQ5 must use the QueryBuilder API – none of the other search APIs support it – and the QueryBuilder implementation requires ht be a requirement of all CQ5 searches actually since CQthat all nodes of the result set be visited to accumulate facets. This mig5 permissions do not seem to be indexed in Lucene – so the larger the result set size, the slower the query performance – others customers have reported similar findings. • A solution involving duplication of data into an unknown number of content hierarchies was deemed to be impractical.

13. NGA CQ5 Art Data and Image Integration What we did • As the complexity of the art object requirements grew, we studied the feasibility of loading all art data and image meta-data into memory. We optimized the memory footprint of those structures and found that it was feasible. The art data bundle loads and caches in RAM approximately 1GB of data. Searches, faceting, and sorting of this data is lightening fast. • What about free text search? We realized that re-implementing the features of Lucene to support free-text search was not something we wanted to do, so we also built a small module to replicate a significantly flattened set of art data in the JCR for Lucene to perform free-text searches on. This data set supports the site search as well as some of the searches on our “advanced collection search” page

14. NGA CQ5 Art Data and Image Integration • For art object imagery, we attempted to: • Use CQ5 DAM to create renditions, including zoom images • Synchronize imagery with the CQ5 DAM using independent Java program invoking JCR APIs over RMI. Findings • Using DAM for renditions ballooned the JCR to over 250GB very quickly – datastore GC might have helped with that, but we didn’t know about it • Insufficient control options over renditions - color profiles are not retained during resizing operations and no out of the box control over resizing algorithms exists • rsync over webDAV didn’t work, so separate program or OSGi bundle would have to be developed to synchronize imagery • Custom workflow processes for images would have to be developed for creating image renditions • What we did • Enhance existing image processing jobs to produce additional sizes; rsync images and serve from a dedicated image and image zoom server • Enhance Art Object JCR bundle to include image data

15. NGA CQ5 Art Data and Image Integration • Lessons Learned • Let CQ do what CQ is good at – authoring, storing, and rendering web content. Resist the temptation to feed a lot of other data sources into the JCR. • Spend the money to consult with an Adobe Architect – it’s worth it. Just be sure they understand how many hours you’ve allocated to each topic so you don’t blow the budget. • Consider using a search engine such as SOLR to power your search features if you have requirements bordering on the complex or if you find yourself struggling with CQ5 JCR queries to get what you need. • The CQ5 DAM is useful for assisting content authors, but until it evolves into something larger, limit it’s use to web content authoring. • Perform data store garbage collection on a regular basis if you find your JCR is growing too rapidly. Remember that CQ backups double your disk space requirement. • Adobe still has a lot of work to do with respect to JCR search. Hopefully the next major version of CQ5 will expose a much richer and better performing search engine to developers.

16. NGA CQ5 Other Useful Nuggets • NGA Developer Workstation Configuration • Java 1.6, latest patch • CRXDE rather than Maven + Eclipse – recommendation from Adobe since small team • CRXDE can be slow and SVN integration is barely sufficient • Missing CRXDE libraries • Disable virus scanner, particularly during install • 4 fast cores + 16GB RAM + fast disk (SSD?) so developer can easily run two instances + tools efficiently – some laptops have only 8 which becomes problematic • local Apache + dispatcher would have been helpful Builds • Scripted build process on shared DEV / build server • Uses curl for all operations • Delete and recreate approach taken • SVN export of apps, design, and environment specific sling:osgiConfig nodes • Bundle build to create JAR files • Package created of all apps, JARs, and OSGI configs; package deployed to targets

17. NGA CQ5 Other Useful Nuggets In hindsight and assuming availability of resources and time, I would have: Specified SOLR as our search solution from the start Sent a few people to CQ5 advanced developer training earlier in the process Sent a few people to CQ5 admin training earlier in the process Committed to Maven + Eclipse rather than using CRXDE for development Would look more seriously at custom oAuth effort rather than using CQ5’s social collaboration module for authentication

18. NGA CQ5 Other Useful Nuggets Questions? d-beaudet at nga.gov 202.312.2755

19. NGA CQ5 Other Useful Nuggets Multiple run modes: Configure your CQ5 instances with multiple run modes (e.g. dev,author) and use sling:osgiConfig nodes to target specific run-mode specific directories (e.g. config.dev.author, config.dev, config.author) Avoid Felix console: don’t apply configurations outside of your DEV activities using the Felix console – it’s better to define those in osgi configs and push them as part of your build. Also, don’t mix modes of applying these configurations – it gets confusing otherwise as changes are persisted differently by each. Avoid memory caching of JCR content if possible (for performance reasons, it’s not always possible) and focus instead on the dispatcher to cache your slowest running pages Use AJAX to avoid cache flushing: For content shared across the site via base page templates (e.g. site-wide alerts), consider using a separate AJAX call in order to avoid having to flush all of your pages from cache. Instead, just cache the individual component instance. Use reference components and train content authors how to create and use reference component instances to avoid having to duplicate a lot of identical content across your site. If you’re programmatically including fixed path reference components in a template, consider keeping that content isolated to a specific directory that requires a developer to change that content; e.g. we use reference components outside of the context of a cq:page for drawers on our event pages.

20. NGA CQ5 Other Useful Nuggets Use client libraries to consolidate and minify your CSS and JavaScript. Separate your JS (under /apps/appname/component path) and your CSS (under /etc/designs/appname) and use categories and embeds (and possibly dependencies) to roll up all of your client libraries into a few consolidated files at publish time. Also, separate your publish components from your author components by checking the WCM mode before including client libs. Otherwise, you might end up (like we did) finding a 7.3MB widgets.js file included for no reason on your home page. Remember that /etc/designs/yourapp/jcr:content contains the list of components permitted for a given template – you can either store it in SVN or let it live on your production authoring instance – but you should choose one approach over the other and adapt build scripts accordingly. Use web browser tools and free on-line web site performance analysis tools to suggest ways of improving the load times of your pages and minimizing the number of requests. Consolidated Search: we had a requirement to spider and show results from non-CQ5 managed web sites within our general site search so we wrote a spider that stores the content of select pages from those sites into the JCR. The web site source is used as a facet so users can opt to select / deselect based on the source web site. SOLR (or another external search engine) could be used for this purpose as well and CQ5 templates would simply wrap the SOLR restful APIs to render results. Use latest dispatcher If at first Adobe support isn’t helpful, be persistent, request a phone call – the quality of response is highly variable – I’ve also heard that specifying “Sling” as the component leads to the best customer support personnel, but I haven’t confirmed that.

21. NGA CQ5 Other Useful Nuggets Avoid using custom name spaces if possible – they cannot be removed from the JCR once created. Avoid creating custom node types for the purpose of simplifying JCR queries – our performance tests show that using attribute / value pairs is usally even faster than a query specifying a particular node type.