1 / 21

MarkLogic Production Deployments

MarkLogic Production Deployments. April 7, 2011. How to make them clean/repeatable/testable. Who Am I?. Brad Rix Brad.Rix@FlatironsSolutions.com Technical Lead for Content Technology Practice

nasia
Download Presentation

MarkLogic Production Deployments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MarkLogic Production Deployments April 7, 2011 How to make them clean/repeatable/testable

  2. Who Am I? • Brad Rix Brad.Rix@FlatironsSolutions.comTechnical Lead for Content Technology Practice • Flatirons SolutionsCorporationFlatirons Solutions is a Colorado based system integrator and long-time MarkLogic partner specializing in XML publishing, dynamic content delivery, and digital asset management for both commercial and government clients.

  3. Why Do I Care About This Stuff? • Problem: • You’ve written some cool XQuery • It does great things that will help your business/customer • But oops: the more you write code, the harder it gets: • New developers break code they don’t understand • Deployments to new environment don’t work and the issues are hard to resolve • More code being written causes more instabilities which causes longer debug/deploy cycles • Which results in you spending less time writing cool XQuery and more time fixing the problems described above • Answer: • Invest some time and engineering to create a build/test/deploy/continuous-integration infrastructure which will greatly improve the ratio of (time doing fun stuff/time doing un-fun stuff)

  4. General Deployment Goals • Repeatable Procedures • Ability to automate deployment of XQuery modules • Ability to automate building of environments to replicate exact configuration among dev/test/production environments • Ability to run unit tests against the code & environment • Automate running of unit tests

  5. Automated Deployments • XQuery modules stored in database • Used ant task to deploy modules • Use Docs HTTP server or create a single HTTP server that has a single module stored that manages deployment • Ant task zips up files and POST them to the HTTP server on the Docs database where the simple XQuery module is deployed. • Setting collections/permissions/etc on the files as they are ingested.

  6. Automated Deployments (ant task) • First zip modules into a single zip file <target name="package“> <mkdir dir="${dist.home}"/> <zip destfile="${dist.home}/${ml. name}.zip"> <fileset dir="${build.home}/src/${ml. name}"/> </zip> </target>

  7. Automated Deployment (ant task) • Properties set in build.properties configurable per environment • Post zip file to preloaded endpoint in Docs database <target name=“deploy"> <exec executable="wget" dir="${dist.home}" failonerror="true" vmlauncher="true"> <arg value="--user=${ml.ingest.user}"/> <arg value="--password=${ml.ingest.password}"/> <arg value="--header=Content-Type:application/zip"/> <arg value="--post-file=${ml.name}.zip"/> <arg value="${ml.db.docs.host}/admin/loadModulesDB.xqy?modulesdb=${ml.modules.db}&amp;serverroot=/"/> </exec> </target>

  8. Automated Deployment (module) Modules parse zip file to get file (snippet): for $zip-input in xdmp:zip-manifest($xquery-modules-zip)//*:part return Get Module: xdmp:zip-get($xquery-modules-zip, $filename, <options xmlns="xdmp:zip-get"> </options>) Then load module into database: xdmp:document-insert($uri, $doc, $permissions)

  9. Automated Configuration • Goal is to be able to configure all database configurations including Application servers/application settings/database settings/range indexes/security roles/etc. • Create a configuration file that contains all of the required information. • Allow the configuration to change and able to re-run on new or existing environment • Run the configuration via ant task or xquery module

  10. Automated Database Configuration • Configuration file per environment • Partial Configuration of database <database name=“MyDBName”> <word-positions>true</word-positions> <element-value-positions>true</element-value-positions> <element-range-indexes> <index> <scalar-type>string</scalar-type> <namespace-uri>http://mhhe.com/meta/resolved</namespace-uri> <local-name>type</local-name> <collation>http://marklogic.com/collation/</collation> <range-value-position>false</range-value-position> </index>…..

  11. Automated Configuration (ant task) • Ant task to call a module to configure the server. • Post the file to an endpoint or have the module load the configuration from a file. <target name="setupdatabase"> <runxquery module="admin/configureEnv.xqy?source=filesystem&amp;uri=${ml.home.url}/Docs/config/environment-config.xml&amp;env=${ml.environment}&amp;restart=true" host="${ml.db.docs.host}" user="${ml.user}" password="${ml.password}"/> </target>

  12. Configuration Module • Configuration module will read the configuration file • Read the XML options and run the appropriate admin services API functions to configure the server/database. • Optionally restart the server depending on parameters that require restart to change

  13. Sample Function to Create Indexes let $database-name := $database-config/@name let $dbid := xdmp:database($database-name) (: remove old ones first :) let $cluster-config := admin:database-delete-range-element-index($cluster-config,$dbid, admin:database-get-range-element-indexes ($cluster-config, $dbid)) let $retv := for $index in $database-config/element-range-indexes/index let $rangespec := admin:database-range-element-index($index/scalar-type, $index/namespace-uri, $index/local-name, $index/collation, $index/range-value-position ) return try { xdmp:set($cluster-config, admin:database-add-range-element-index($cluster-config, $dbid, $rangespec) ) } catch ($err) { if ($err/error:code eq "ADMIN-DUPLICATEITEM") then () else error(xs:QName("ERROR"), xs:string($err/error:message)) } return $cluster-config

  14. XQuery Unit Testing • Know that all existing modules work as expected • Changes in modules still work for previous expected behavior • At Flatirons we developed a java XQueryUnit that utilizes XML Unit for differencing XML files • Can be done using HtmlUnit/XMLUnit/Performance Meters • Framework uses ant to run all unit tests • Incorporated into entire build process so that the deploy target can run the tests to ensure that everything works properly in this environment • To test an http restful call into MarkLogic, write an XQuery wrapper that performs the xdmp:http-get (or post). • Return the data back from the module and compare that to expected output in the configuration file.

  15. Unit Tests (configuration file) • Call a module and expect known return • If XML does not match return, then test will fail <xqu:testxqu:name="HelloWorldInline"> <xqu:xqy_test> <xqu:xqy_test_filexqu:uri="modules/util/testHelloWorld.xqy" /> </xqu:xqy_test> <xqu:expected_result> <xqu:xml_result> <xqu:xml_result_src> <hello> world </hello> </xqu:xml_result_src> </xqu:xml_result> </xqu:expected_result> </xqu:test>

  16. Run Unit Tests • Run ant test test: [junit] Testsuite: com.mhhe.xmlunit.UnitTestSuite [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 3.078 sec [junit] [junit] Testcase: HelloWorldInline took 0.453 sec [junit] Testcase: HelloWorldFile took 0.016 sec [junit] Testcase: TestCreateProject took 2.109 sec [junit] Testcase: TestFavorites took 0.172 sec [junit] Testcase: TestGetAsset took 0.219 sec

  17. Unit Tests (Failed Test) [junit] [junit] Testcase: HelloWorldInline took 0.313 sec [junit] FAILED [junit] Unexpected error received during test 'HelloWorldInline': junit.framework.AssertionFailedError: org.custommo nkey.xmlunit.Diff [junit] [different] Expected text value 'world' but was ' [junit] failed_test [junit] ' - comparing <hello ...>world</hello> at /hello[1]/text()[1] to <hello ...> [junit] failed_test [junit] </hello> at /hello[1]/text()[1]

  18. Automated Build Testing (Hudson) • Continuous build testing with Hudson (http://hudson-ci.org/) or Jenkins (http://jenkins-ci.org/) • Hudson is a self contained web application (just install and run). Includes Jetty web server as part of deployment. • Run builds on a schedule (hourly) as each developer changes code or manual ad-hoc deployments from a web-interface • Build will run full configuration/ code deploy/ unit tests at each interval. Can automate deployments to multiple environments. • If any failure then e-mail user who checked in code (along with buildmeister)

  19. Dashboard (live) • Shows current builds • Allows kicking off new builds • Can see information on success and failed builds

  20. Console Output (live)

  21. Lessons Learned • Tightly type the input and output of all modules • This will detect when function is called with empty or wrong parameters quickly. • Disable Function Mapping • declare option xdmp:mapping "false"; • Unit Test as much as possible

More Related