Bringing results of the E&I project to the production of statistics

1. Bringing results of the E&I project to the production of statistics Pauli Ollila UNECE Session 2011 / Slovenia

2. Four phases of the Editing Project • 1) Internal E&I study at StatFi • Currentfacilities and resources • Existing E&I situations and phases • E&I practices, methodology and tools • 2) External E&I study • - Experiences of otheroffices and institutions • Researchresults and recommendations • Existing methodology and practices • Software • 3) E&I developmentphase • Constructing an E&I framework to be used by statistics • Creating E&I descriptions and recommendations for differentsituationsconcerning statistics making • Testingchosensolutions with statistics • Checkingpossible E&I software alternatives Pauli Ollila / Statistics Finland

3. 4) Distributing E&I information Education E&I instructions E&I models for the types of statistics Non-survey statistics Simple survey statistics Concept and method library Business statistics Multidata survey statistics Economic statistics Consulting Pauli Ollila / Statistics Finland

4. E&I instructions • E&I themes and statistics • Data sets to beused • Data creationphase • Data arrivalphase • Separate data modificationphase • Data merging and furtherprocessingphase • Data content and errorknowledge • Operationsduringediting and imputation • E&I methods and practices • Editing • Imputation • E&I quality evaluation • Documentation and data storage • Testingmethodology Pauli Ollila / Statistics Finland

5. E&I themes E&I themedescribessomesituation in the statistics from the E&I point of view. The themescanvaryfrom general (data arriving at once) to detailed (usingfunction of current and previousvalue for editing). Therearehierarchies of the themes(treestructure). Startingpoint of making statistics and requirements(whatexists and what is wanted) E&I processing in making statistics (what is done) Existing knowledge of data and errors(what is known) • - Resources and equipment • - Combinations of data • - Auxiliaryinformation • Historic data • Requiredresults, estimates and importantsubgroups • - Qualityrequirements • (- Data creationphase) • Data arrivalphase • Editing and otherprocessingin differentphases • Reacting to the informationprovidedbyediting(contacting data providers,, fetchingvalues, furtherprocessing, imputingbased on calculations) • Variablestructures and relations • Noticeableestimates and distributions • Whatkind of errorsthereare

6. E&I models for the types of statistics Non-survey statistics Simple survey statistics Economic statistics Providing the frameworkhow to plan and conduct E&I in the types of statistics with goodpractices and recommendations. Business statistics Multidata survey statistics Inofficial* tool for evaluating E&I situation of statistics • Application utilizing (”ancient”) SAS FSEDIT and descriptions connected to conditions. • E&I answers of the statistics based on the E&I themes are given in the SAS system. The data is permanent, and it can be changed later. • Produces a description (rtf format, readable in Word) based on the answers and a recommendation or a comment to every existing theme and some non-existing themes (in principletailor-made for the statistics). • Mainly for the consultinguse of the editingproject * Notfollowing the leadingprinciple of generality on E&I issues Pauli Ollila / Statistics Finland

7. Concept and method library • Wiki-based application to be used in the internalnet of StatFi • Partially including the samecontents as the instructions, butexpressions are more compact. • Differentways to enter(hierarchical, patterns, alphabetical) • Possible to uselinkseverywhere(blueunderlines) • Elements of librarye.g. • description • examples • methodsavailable • recommendations • programsolution • formula • algorithm • prerequisites for calculation Pauli Ollila / Statistics Finland

8. Education • Part of the StatFi Training Programme in Statistical Skills • Tailor-made courses / theme courses Includingcrucial E&I themes appearing at StatFitogether with E&I methodology and planning of E&I. Consulting and methodological support • Important:Helping statistics applyingavailable E&I knowledge in differentforms • Contributing to differentdevelopmentalprojectsincluding E&I elements • Definingmethodologicalmodulestheoretically for possibledevelopment at the IT department (ifnotalready existing in the software) • Support of existing software (Banff, Selekt) Pauli Ollila / Statistics Finland

9. E&I theme framework with analysing statistics and giving recommendations (example: Finance of housing companies) Key points * Veryhectic data collection and arrivalphase. Notmuchprocessingtimebeforepublication. * Despiteweb-survey data comingalso in variouspaperformats * Notyet a common data base, operationsconnected to E&I spread in twosystems (someroutines with Blaise, others in SAS) Startingpoint of making statistics and requirements(whatexists and what is wanted) * One block of expenses (repairment) provides a vastmajority of errors. Reason: the accountingsystemsdonotclearlyseparatethisby definition. The professionalrespondentscangive the figures, but the ”amateurs” mayprovideveryvaryingerrors. Existing knowledge of data and errors(what is known) * The editingpracticesarebased on somelogicalchecks, errorlistings and a lot of studying the problematicobservations. No use of historic data orauxiliaryinformation. * In practiceallcorrectionsare made bymaking a non-calculativeapproximation. E&I processing in making statistics (what is done)

10. Recommendations: * If the timetable is notadjustable, try to makesome E&I operationsduring data arrival, ifthereareresources for that. * Figure out solutions for gettingrid of paperformats. * A common data basewillcome. Startingpoint of making statistics and requirements(whatexists and what is wanted) * Analyse the repairmentblock of expensesparttogether with otherrelevantvariables in order to getvariablerelations for makingefficienteditrules. Existing knowledge of data and errors(what is known) * Ifpossible, findsolutions for helpingrespondents with the difficultpart (revisinginstructions, questionnaire, getting feedback) * With limitedtime for editing, considerprioritizingobservations to beedited, e.g. byassessing the influence of the observation to the results * The editingprocessshouldbeclearlydefinedbased on substanceknowledge in order to avoidcreativeediting * The changesshouldbeidentifiable in order to calculate E&I qualityindicators. E&I processing in making statistics (what is done)