Developing & Maintaining Dynamic Micro-Simulation Models at DWP

1. Developing & Maintaining Dynamic Micro-Simulation Models at DWP Sally Edwards Simon Gault ESRC – 2nd April 2009

2. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

3. Introductions Simon Gault • Statistician, with a strong background in Economics • Extensive experience in developing models • Manages the Forecasting Simulation models team Sally Edwards • Systems / Business Analyst • Predominantly worked in the private sector on large scale IT systems • Manage the Genesis Simulation Model Members of a the Model Development Unit (MDU) which has responsibility for Micro-Simulation Models at DWP MDU consists of 20+ people

4. Micro-Simulation Models • Dynamic Micro-Simulation Models, based on our standard architecture, Genesis: • Pensim2 private & state pension income • Inform Integrated Forecasting model for working age benefit claimants • Individual Benefit Forecasting for RP, IB, DLA/AA • Employment Model, currently being validated • Static Micro-Simulation Models: • Policy Simulation Model (PSM) • Employment and Hours Model (EHM)

5. Key factors taken into Consideration during Development Key factors considered when we designed our first dynamic micro-simulation model, Pensim2 - also valid for our other models: • Simplify the complexity of the model as much as possible, using a modular approach and allowing a simple prototype model to be developed quickly • Several developers work on each model at the same time • Large staff turnover • Models are intended for long term use (e.g. Pensim2 anticipated to have 15-20 year life) so most of the people using/maintaining the models are not involved in their development • Clear documentation, processes, training and user guides are essential • Excel and SAS are the preferred tools, as both are well understood and extensively used among the Analysts at DWP

6. Developing Models at DWP • Overview • Objectives for Pensim2 (first DMS model) • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

7. Original Objectives for Pensim2 • Flexibility8. Efficiency (in use of memory) • Robustness 9. Timeliness • Transparency 10. Done with own expertise • User-friendliness 11. Done with own resources • Speed 12. Ease of handover • Reliable Output 13. Ease of maintenance • Availability 14. Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

8. Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives:

9. Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: Excel based front end – Pensim2 • User friendly, transparentcode in a standard format • Easy to understand and flexible, enabling analysts to change parameters & structure of model without understanding the underlying code. • Easy to maintain and handover • Allowed independence of base data

10. Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: : Excel based front end – Pensim2 • User friendly, transparentcode in a standard format • Easy to understand and flexible, enabling analysts to change parameters & structure of model without understanding the underlying code. • Easy to maintain and handover • Allowed independence of base data SAS code generator - Genesis • Developed with our own expertiseandown resources • Robustgenerator, producingreliable outputs

11. Basic structure of Genesis Dynamic Micro-Simulation Models at DWP

12. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

13. Development Standards Two separate development teams, with different development standards adopted Genesis Model Engine • Developed by IT staff • Used standard IT project development procedures Pensim2 Analytical team • Economists and Statisticians • Less structured approach to development

14. Genesis Model Engine Development Standards Highly structured approach to the development • Project Management Protocol based on Prince2 methodology, although not all aspects were needed • Documentation– 2 types • Development documentation produced at each stage of project, that was signed off and used as input to the next stage • Key Projectdocumentation (*) - maintained and kept up-to-date with each new release • Development Processes –simple process diagrams to provide clear guidelines, particularly useful for new staff • Detailed Requirements – explaining the full functionality of the model engine. Document translates Economist language into IT language(*)

15. Genesis Model Engine Development Standards • Design Diagrams and documentation – used in development stage only • Change Control process–changes estimated and impacted before being accepted/rejected/on hold (*) • Problem Log process–problems recorded, prioritised, fixed/removed from the scope/user error (*) • Validation Process - detailed Test Plan based on Requirements document, used to ensure that each requirement was fully satisfied • Test Pack ensured existing functionality continued to work when changes were made to the model engine (*) • User Guide contains details of functionality written in simple format (*) • Trainingmaterials, inc presentations and training text for self-study (*)

16. Pensim2 Model Development • Pensim2 Analytical development followed a less structured approach • Generally this was appropriate, as detailed requirements were not possible to establish up-front • Effort was predominantly spent on regression analysis • Analysis results reviewed to determine the key components to include and the most appropriate sources for the assumptions • Using the standard Genesis templates, the Pensim2 model was more easy to develop, rather than if a traditional DMS model had been built

17. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

18. Key Aim of Genesis Structure - Simplify Complexity No Hard Coding Rule which allows complete flexibility for: • Structure of the simulation, i.e. order of events processed • Data structures, i.e. data tables, data relationships and variables • Parameters, e.g. indexes, rates, dates • Regression Equations • Selection filters • Static SAS code, if required (e.g. addition of new cases to a model) • Ability to add / remove modules easily Each model is defined in simple Excel spreadsheets, using standard templates, that users can easily understand and modify

19. Key Aim of Genesis Structure - Simplify Complexity Data Access routines • Enable easy access to data so user does not have to write complex code • Default data access will process the current object in the current simulation year • Facility to process a variable belonging to a related object, e.g. partner PAR; Relationships are not hard-coded and new relationships can easily be defined • Data manipulation routines provide commonly required data facilities, e.g. “maximum value during past 10 years” = MA10 • E.g. MA10_PAR_pa_salary returns the maximum value in the past 10 years of the salary variable for the individual’s partner, taken from the pa table Keep Equations simple • Equations, filters & process flows are defined in a simple, standard format, that is unambiguous. Hence the models are easy to maintain

20. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

21. Change Management • Change Management Process is followed for all models • Every change goes through the formal Change Control procedure, i.e. requirements documented, impact assessed, agreed by users & signed off • Change Register listing all changes, each with a separate reference number • For each change to the model, a separate Change Control form is produced • The model code contains the Change ref number, with a comment • Some changes are repeated regularly, e.g. assumptions, so clear documentation is helpful for the repeat change • Questions arise after a change has been implemented - these can be easily answered by referencing the Change Control documentation

22. Problem Management • Problem Management Process is followed for all models • Every problem/error discovered in a model goes through the formal Problem Log procedure, i.e. requirements documented, impact is assessed, agreed by users & signed off • Problem Log Register, listing all problems identified, including those that turn out to be User Errors or misunderstandings; PL ref number assigned • For each problem recorded, a separate Problem Log form is produced • Users are notified of significant problems that affect the model results • Many of the problems are minor and are picked up by the development team – these are recorded formally, even if they do not impact the results

23. User Model • Our models fall into 2 categories: • Maintained & owned by the Model Development Unit • Developed by Model Development Unit; handed over to User team, who then own and maintain the model • Models owned by the Model Development Unit have a User Group and/or a Steering Group consisting of representatives from each team using the model • The purpose of these groups is to agree the content, priority and timing of changes made to the model

24. User Group Purpose Specifically for Pensim2, our largest model: • Change Requests and Problem Logs are reviewed by the User Group, where appropriate • A key user sponsor is assigned to each change. The sponsor ensures that the change is accurately specified & will sign-off testing • The User Group are responsible for signing off new versions of the model before they are released • The users also have a separate User Forum where they present analysis carried out on the model outputs and explain how the results have been used

25. Release Management Again, specifically for Pensim2: • New Releases are implemented approximately every 6 to 9 months • Each Release includes one or two key components, e.g. Private Pension Reform, and a bundle of smaller changes and problem fixes • The timing and content of each Release are agreed with the User Group • When a new Release is built: • Each change/fix is added to the model one at a time • Hence each change is reviewed and signed off separately • Enables us to fully understand and agree the effect of each change. • Time consuming but gives confidence in the results • Clear audit trail • All previous versions of the model remain available

26. Developing Models at DWP • Overview • Objectives • Development standards • Simplifying Complexity • Maintenance protocol • What worked well and Lessons Learned

27. Original Objectives Satisfied • Flexibility8. Efficiency (in use of memory) • Robustness9. Timeliness • Transparency10.Done with own expertise • User-friendliness11. Done with own resources • Speed12. Ease of handover • Reliable Output13. Ease of maintenance • Availability14.Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

28. Objectives Partially Satisfied • Flexibility8. Efficiency (in use of memory) • Robustness9. Timeliness • Transparency10. Done with own expertise • User-friendliness11. Done with own resources • Speed12. Ease of handover • Reliable Output13. Ease of maintenance • Availability14.Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

29. Objectives that proved more tricky • Flexibility8. Efficiency (in use of memory) • Robustness9. Timeliness • Transparency10. Done with own expertise • User-friendliness11. Done with own resources • Speed12. Ease of handover • Reliable Output13. Ease of maintenance • Availability 14.Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

30. What went well • Generic framework has proved successful and easy to use • Genesis architecture has been remarkably robust • Modules have been reused/copied from one model to another • Quicker development of new dynamic micro-simulation models, compared with a traditional build • Automated documentation facility and error checking produced using VBA, listing where variables are used and providing an overview of each module, to ensure consistency • Lessons learnt from Pensim2 development were used when developing subsequent models

31. Lessons Learned Key lessons learnt from Pensim2 development: • Tried to make the model too complicated – particularly the analytical side • We should have developed a set of simple modules for a Phase 1 delivery, then added complexity to the more important modules for Phase 2 • The structure of some of our modules was more complicated than the data would support, e.g. Labour Market process • Insufficient documentation produced by the Analytical team during the development (e.g. Base Data Creation) • Genesis Engine code is difficult to understand and hence modify, although it is robust and does not require much modification • Some Genesis Engine functionality was over-ambitious and subsequently dropped (e.g. Alignment of polychotomous variables)

32. Any questions?