The European Commission's Digital Cities Challenge

1. The European Commission's Digital CitiesChallenge • Guidelines for implementation of Digital Transformation Trajectory Step 4: • monitoring and implementation

2. Guidelines for the implementation of step 4 of the Digital Transformation Trajectory: monitoring and implementation

3. Step 4: Monitoring and implementation • You’ve come a long way! • One last step to complete the digital transformation trajectory.

4. Step 4: Monitoring and implementation

5. Step 4: Monitoring and implementation Objectives of the Presentation • Go over the proposed methodology for step 4: monitoring and implementation • Have a detailed discussion on the Digital Transformation Strategy monitoring indicators and performance assessment framework • Next steps and upcoming deadlines

6. Step 4: Monitoring and implementation Now that the full strategy and roadmaps have been designed, you need to develop a system allowing to monitor progress and achievements Preparation 0 Digital vision& ambition 1 Strategy 2 Roadmap 3 Monitoring& implemen-tation 4 Field advisory services Ambition workshop SAT & KPIs Assess-ment visit Strategy workshops Roadmap workshop Monitoring workshop (on-line) Academy Kick-off seminar Vision & Ambition seminar Strategy seminar Roadmap seminar Wave 1 Wave 2 Today The academy seminar is … … a promotion … … not one additional networking event. … a learning and working session … … not a training course. … a recognition by the peers … … not a conference.

7. Step 4: Monitoring and implementation Presentation of general approach Objectives Way of working • Develop a detailed monitoring plan for the digital transformation strategy A • Data collection • Previous interactions with city stakeholders on Monitoring Indicators B • Ideation and formulation • Bilateral conversations with city representatives for the development of M&E procedures and responsibilities, and identifying strategy monitoring indicators ✓ Deliverables • Detailed monitoring plan including monitoring scorecards

8. Three levels of Monitoring Indicators Step 4: Implementation and monitoring Ambition statement indicators Operational Objective indicators Activity indicators

9. Step 4: implementation and monitoring Key definitions • Monitoring indicators: Indicators to be used to monitor the progress and success of your digital transformation strategy. • Monitoring indicators defined in the framework of your strategy are not to be confused with KPIs used during the assessment phase of the digital transformation trajectory. Monitoring indicators will allow you to monitor the progress and success of your digital transformation strategy. • Three levels of Monitoring Indicators which correspond to three levels of results: • Ambition statements: these measure strategy outcomes • Operational objectives: intermediate outcomes (or activity outcomes) • Activities: strategy outputs • Impact is defined at the level of the mission statement: no indicator defined at this level for the time being • Fuzzy borders between outputs, outcomes and impact: you need to make your own interpretation, but be consistent

10. Step 4: implementation and monitoring

11. Step 4: implementation and monitoring Ref: W.K. Kellogg Foundation - Logic Model

12. Step 4: implementation and monitoring

13. Step 4: implementation and monitoring Presentation of general approach • Setting a target for Monitoring Indicators requires doing three things: • Establishing a robust baseline reflecting the current situation: if data is not available, then you need to generate it • Defining the breath of your ambitions e.g. do you want to double growth rates or simply increase them by 5% • Establishing time horizons • If data is not available to define the baseline at this stage, then you should identify the means through which you could develop the baseline at a later stage e.g.: generate data, conduct a study, conduct a survey, etc • Establish a target which seems reasonable at this point in time. Target can always be adjusted later. • Targets should always be accompanied by a timeframe, e.g. supporting the creation of 200 innovative companies over the next 4 years. • Timeframe for output related indicators should be shorter than for outcomes • Outcomes should not be set to appear before 5-10 years.

14. Step 4: implementation and monitoring

15. Step 4: implementation and monitoring • Additionalreading • Technopolis Group, Evaluation of Innovation Activities: Guidance on methods and practices • HM Treasury, The Magenta Book • HM Treasury, Quality in qualitative evaluation: a framework for assessing research evidence (supplementary Magenta Book guidance) • HM Treasury, Quality in policy impact evaluation: understanding the effects of policy from other influences (supplementary Magenta Book guidance) • Vinnova, A Joint Approach for Logic Assessment, Monitoring, Evaluation and Impact Analysis • NIFU, Challenges measuring effects of research and innovation policy interventions in ex-post impact evaluations • TEKES, Better results, more value: A framework for analysing the societal impact of research and innovation • Danish Ministry of Science, Innovation, and Higher Education, Central Innovation Manual on Excellent Econometric Impact Analyses of Innovation Policy • Louis Lengrand & Associés, Smart Innovation: Practical Guide to Evaluating Innovation Programmes

16. Step 4: implementation and monitoring Next steps: developing a monitoring and evaluation plan • Define a performance framework for your strategy including indicators, baseline, targets and means of verification • Start by defining activity indicators thinking about the expected outputs for your activity (i.e. the bottom of the performance framework) • Work your way up (i.e. operational objectives, ambitions) while ensuring consistency across the levels • Compare the results of this work with the indicators you had previously though off (i.e. top-down vs bottom-up approach) and see if it matches • Define the process for monitoring progress within the strategy e.g. regular updates based on monitoring indicators & person/people in charge • Identify options for evaluation i.e. when and what to evaluate (e.g. ex-ante, mid-term, ex-post; internal vs. external; comprehensive vs. topic-specific) • Identify resources necessary for monitoring and evaluation • Define your baby steps: what will have changed in your city one year from now? In one year’s time, what change must have been generated for you to consider to be on the right track? • POINTS TO BE COVERED DURING THE ON-LINE MONITORING WORKSHOP

17. Step 4: implementation and monitoring Next steps: wrapping up • Finalise your strategy reports (including roadmap and governance chapter and activity) based on feedback and peer reviews from the last academy seminar. Deadline: 20 February • Organize the monitoring on-line workshop. Deadline: March 2019 • Additional informal interaction between the experts and the city representatives to wrap up the report • Draft the final chapter of the strategy report: monitoring and evaluation plan + full performance framework + next steps - > Draft final strategy report. Deadline: 15th March • Draft final strategy report taking into account feedback provided by the reviewer. Deadline: 10th April. • Organise a first meeting of your governance / steering bodies, and report back to the DCC on the outcomes of it • Implement – to the extent possible – one pilot activity, and report back to the DCC on the outcomes of it • Participated in the Final Conference: Final debrief with the DCC management team and the EC. 5th of June (Brussels).

18. Contacts • Carlos Hinojosa [carlos.hinojosa@technopolis-group.com] • Pierre Bastien [pierre.bastien@rolandberger.com]  • Morgane VeilletLavallée [morgane.veillet-lavallee@technopolis-group.com] • Jacek Walendowski - project leader [jacek.walendowski@technopolis-group.com]