EOLDAS_ng

1. EOLDAS_ng The new generation Prof. P. Lewis, Dr. Jose Gomez-Dans, UCL/NCEO

2. Task 3: EO-LDAS baseline WP3.1: EO-LDAS v1 consolidation • Autumn 2013/workshop • Publicly available software & documentation WP3.2: Integration of enhanced components • Dec/2013 • Integration of other modelling efforts in variational DA

3. What is EO-LDAS? • generic variational DA system, weak or strong constraint • different observation operators (RT models) • different process models, including regularisation • tutorial documentation -- how to do DA with the system • Demonstrated so far with optical data Lewis et al (2012), Rem Sens Env Lewis et al (2012), ESA SP

6. The concept Define a cost functionJ as a sum of however many components, but typically: Dynamic Model Prior J, J' Observations (RT codes) • Each component calculates the cost and the partial derivatives of the cost function • Use gradient descent minimisation • Need partial derivatives for Observation Operator & Dynamic model • Uncertainty is calculated using the Hessian (matrix of 2nd order derivatives) @ minimum

8. What have we learned with EO-LDAS? • v1 design controlled by configuration files • minimise programming requirements • generic structure probably sound • teach users new metalanguage to describe problem • Too much flexibility leads to inefficiencies • Difficult to add extra functionality • Difficult to maintain

9. EO-LDAS v2 • Assume user knows Python (e.g. I/O, plotting) • Streamline system by reducing flexibility • Simple components - readily extended • Decouple solver from operators • Essentially, list of J, J' EO-LDAS should be a Python library of state definitions and generic operators, requiring the user to interface to other: • ObsOps, • Dynamic Models, • Data, • Plotting, etc

10. Interactive documentation IPython notebooks to provide interactive documentation on library brary http://nbviewer.ipython.org/urls/github.com/jgomezdans/eoldas_ng/raw/master/notebooks/ASimpleSmoother.ipynb

11. Faster models through emulation • First steps in using Gaussian Processes (GP) Provides a fast version of the code • ... but also of its partial derivatives! 1. User generates training set with RT model 2. GP is trained to the above 3. This is used in EO-LDAS • Much more flexible: • if the model can be emulated, any model can go in • no need to waste time on adjoints • very fast • simplifies library structure very much

12. Emulator testing Use IPython notebooks to provide the user with interactive documentation on how to do emulation http://nbviewer.ipython.org/urls/raw.github.com/profLewis/emulate_test/master/emulate_test.ipynb

13. Spares

14. Multi-resolution processing

15. Gotchas Observation Operator gotchas • Typical RT codes are science grade software: • Limited support from the authors • Inefficient • Difficult to follow & modify • Few RT models have adjoints readily available • If the RT model is modified, the adjoint needs to be re-created Dynamic model gotchas • All of the above • Ended up implemented a linear model: if you can express your model as Ax, then you can slot it in • Most dynamic models are highly non-linear, and linearisation requires automatic differentiation • So ended up using Tikhonov-type regularisation