1 / 32

Developing & Maintaining Dynamic Micro-Simulation Models at DWP

Developing & Maintaining Dynamic Micro-Simulation Models at DWP. Sally Edwards Simon Gault ESRC – 2 nd April 2009. Developing Models at DWP. Overview Objectives Development Standards Simplifying Complexity Maintenance Protocol What worked well and Lessons Learned. Introductions.

reidar
Download Presentation

Developing & Maintaining Dynamic Micro-Simulation Models at DWP

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Developing & Maintaining Dynamic Micro-Simulation Models at DWP Sally Edwards Simon Gault ESRC – 2nd April 2009

  2. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

  3. Introductions Simon Gault • Statistician, with a strong background in Economics • Extensive experience in developing models • Manages the Forecasting Simulation models team Sally Edwards • Systems / Business Analyst • Predominantly worked in the private sector on large scale IT systems • Manage the Genesis Simulation Model Members of a the Model Development Unit (MDU) which has responsibility for Micro-Simulation Models at DWP MDU consists of 20+ people

  4. Micro-Simulation Models • Dynamic Micro-Simulation Models, based on our standard architecture, Genesis: • Pensim2 private & state pension income • Inform Integrated Forecasting model for working age benefit claimants • Individual Benefit Forecasting for RP, IB, DLA/AA • Employment Model, currently being validated • Static Micro-Simulation Models: • Policy Simulation Model (PSM) • Employment and Hours Model (EHM)

  5. Key factors taken into Consideration during Development Key factors considered when we designed our first dynamic micro-simulation model, Pensim2 - also valid for our other models: • Simplify the complexity of the model as much as possible, using a modular approach and allowing a simple prototype model to be developed quickly • Several developers work on each model at the same time • Large staff turnover • Models are intended for long term use (e.g. Pensim2 anticipated to have 15-20 year life) so most of the people using/maintaining the models are not involved in their development • Clear documentation, processes, training and user guides are essential • Excel and SAS are the preferred tools, as both are well understood and extensively used among the Analysts at DWP

  6. Developing Models at DWP • Overview • Objectives for Pensim2 (first DMS model) • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

  7. Original Objectives for Pensim2 • Flexibility8. Efficiency (in use of memory) • Robustness 9. Timeliness • Transparency 10. Done with own expertise • User-friendliness 11. Done with own resources • Speed 12. Ease of handover • Reliable Output 13. Ease of maintenance • Availability 14. Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

  8. Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives:

  9. Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: Excel based front end – Pensim2 • User friendly, transparentcode in a standard format • Easy to understand and flexible, enabling analysts to change parameters & structure of model without understanding the underlying code. • Easy to maintain and handover • Allowed independence of base data

  10. Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: : Excel based front end – Pensim2 • User friendly, transparentcode in a standard format • Easy to understand and flexible, enabling analysts to change parameters & structure of model without understanding the underlying code. • Easy to maintain and handover • Allowed independence of base data SAS code generator - Genesis • Developed with our own expertiseandown resources • Robustgenerator, producingreliable outputs

  11. Basic structure of Genesis Dynamic Micro-Simulation Models at DWP

  12. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

  13. Development Standards Two separate development teams, with different development standards adopted Genesis Model Engine • Developed by IT staff • Used standard IT project development procedures Pensim2 Analytical team • Economists and Statisticians • Less structured approach to development

  14. Genesis Model Engine Development Standards Highly structured approach to the development • Project Management Protocol based on Prince2 methodology, although not all aspects were needed • Documentation– 2 types • Development documentation produced at each stage of project, that was signed off and used as input to the next stage • Key Projectdocumentation (*) - maintained and kept up-to-date with each new release • Development Processes –simple process diagrams to provide clear guidelines, particularly useful for new staff • Detailed Requirements – explaining the full functionality of the model engine. Document translates Economist language into IT language(*)

  15. Genesis Model Engine Development Standards • Design Diagrams and documentation – used in development stage only • Change Control process–changes estimated and impacted before being accepted/rejected/on hold (*) • Problem Log process–problems recorded, prioritised, fixed/removed from the scope/user error (*) • Validation Process - detailed Test Plan based on Requirements document, used to ensure that each requirement was fully satisfied • Test Pack ensured existing functionality continued to work when changes were made to the model engine (*) • User Guide contains details of functionality written in simple format (*) • Trainingmaterials, inc presentations and training text for self-study (*)

  16. Pensim2 Model Development • Pensim2 Analytical development followed a less structured approach • Generally this was appropriate, as detailed requirements were not possible to establish up-front • Effort was predominantly spent on regression analysis • Analysis results reviewed to determine the key components to include and the most appropriate sources for the assumptions • Using the standard Genesis templates, the Pensim2 model was more easy to develop, rather than if a traditional DMS model had been built

  17. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

  18. Key Aim of Genesis Structure - Simplify Complexity No Hard Coding Rule which allows complete flexibility for: • Structure of the simulation, i.e. order of events processed • Data structures, i.e. data tables, data relationships and variables • Parameters, e.g. indexes, rates, dates • Regression Equations • Selection filters • Static SAS code, if required (e.g. addition of new cases to a model) • Ability to add / remove modules easily Each model is defined in simple Excel spreadsheets, using standard templates, that users can easily understand and modify

  19. Key Aim of Genesis Structure - Simplify Complexity Data Access routines • Enable easy access to data so user does not have to write complex code • Default data access will process the current object in the current simulation year • Facility to process a variable belonging to a related object, e.g. partner PAR; Relationships are not hard-coded and new relationships can easily be defined • Data manipulation routines provide commonly required data facilities, e.g. “maximum value during past 10 years” = MA10 • E.g. MA10_PAR_pa_salary returns the maximum value in the past 10 years of the salary variable for the individual’s partner, taken from the pa table Keep Equations simple • Equations, filters & process flows are defined in a simple, standard format, that is unambiguous. Hence the models are easy to maintain

  20. Developing Models at DWP • Overview • Objectives • Development Standards • Simplifying Complexity • Maintenance Protocol • What worked well and Lessons Learned

  21. Change Management • Change Management Process is followed for all models • Every change goes through the formal Change Control procedure, i.e. requirements documented, impact assessed, agreed by users & signed off • Change Register listing all changes, each with a separate reference number • For each change to the model, a separate Change Control form is produced • The model code contains the Change ref number, with a comment • Some changes are repeated regularly, e.g. assumptions, so clear documentation is helpful for the repeat change • Questions arise after a change has been implemented - these can be easily answered by referencing the Change Control documentation

  22. Problem Management • Problem Management Process is followed for all models • Every problem/error discovered in a model goes through the formal Problem Log procedure, i.e. requirements documented, impact is assessed, agreed by users & signed off • Problem Log Register, listing all problems identified, including those that turn out to be User Errors or misunderstandings; PL ref number assigned • For each problem recorded, a separate Problem Log form is produced • Users are notified of significant problems that affect the model results • Many of the problems are minor and are picked up by the development team – these are recorded formally, even if they do not impact the results

  23. User Model • Our models fall into 2 categories: • Maintained & owned by the Model Development Unit • Developed by Model Development Unit; handed over to User team, who then own and maintain the model • Models owned by the Model Development Unit have a User Group and/or a Steering Group consisting of representatives from each team using the model • The purpose of these groups is to agree the content, priority and timing of changes made to the model

  24. User Group Purpose Specifically for Pensim2, our largest model: • Change Requests and Problem Logs are reviewed by the User Group, where appropriate • A key user sponsor is assigned to each change. The sponsor ensures that the change is accurately specified & will sign-off testing • The User Group are responsible for signing off new versions of the model before they are released • The users also have a separate User Forum where they present analysis carried out on the model outputs and explain how the results have been used

  25. Release Management Again, specifically for Pensim2: • New Releases are implemented approximately every 6 to 9 months • Each Release includes one or two key components, e.g. Private Pension Reform, and a bundle of smaller changes and problem fixes • The timing and content of each Release are agreed with the User Group • When a new Release is built: • Each change/fix is added to the model one at a time • Hence each change is reviewed and signed off separately • Enables us to fully understand and agree the effect of each change. • Time consuming but gives confidence in the results • Clear audit trail • All previous versions of the model remain available

  26. Developing Models at DWP • Overview • Objectives • Development standards • Simplifying Complexity • Maintenance protocol • What worked well and Lessons Learned

  27. Original Objectives Satisfied • Flexibility8. Efficiency (in use of memory) • Robustness9. Timeliness • Transparency10.Done with own expertise • User-friendliness11. Done with own resources • Speed12. Ease of handover • Reliable Output13. Ease of maintenance • Availability14.Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

  28. Objectives Partially Satisfied • Flexibility8. Efficiency (in use of memory) • Robustness9. Timeliness • Transparency10. Done with own expertise • User-friendliness11. Done with own resources • Speed12. Ease of handover • Reliable Output13. Ease of maintenance • Availability14.Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

  29. Objectives that proved more tricky • Flexibility8. Efficiency (in use of memory) • Robustness9. Timeliness • Transparency10. Done with own expertise • User-friendliness11. Done with own resources • Speed12. Ease of handover • Reliable Output13. Ease of maintenance • Availability 14.Independence of base data • (i.e. desktop model) Pensim2 Feasibility Study document (2002) – Objectives

  30. What went well • Generic framework has proved successful and easy to use • Genesis architecture has been remarkably robust • Modules have been reused/copied from one model to another • Quicker development of new dynamic micro-simulation models, compared with a traditional build • Automated documentation facility and error checking produced using VBA, listing where variables are used and providing an overview of each module, to ensure consistency • Lessons learnt from Pensim2 development were used when developing subsequent models

  31. Lessons Learned Key lessons learnt from Pensim2 development: • Tried to make the model too complicated – particularly the analytical side • We should have developed a set of simple modules for a Phase 1 delivery, then added complexity to the more important modules for Phase 2 • The structure of some of our modules was more complicated than the data would support, e.g. Labour Market process • Insufficient documentation produced by the Analytical team during the development (e.g. Base Data Creation) • Genesis Engine code is difficult to understand and hence modify, although it is robust and does not require much modification • Some Genesis Engine functionality was over-ambitious and subsequently dropped (e.g. Alignment of polychotomous variables)

  32. Any questions?

More Related