The role of textual data in a statistical approach for the evaluation of the regulatory impact

1. NTTS 2009 The role of textual data in a statistical approach for the evaluation of the regulatory impact Simona Balbi, Germana Scepi, Giorgio Infante Università “Federico II” di Napoli

2. our aim NTTS 2009 topropose statistical tools for ex ante evaluation of public actions, i. e.helping in identifying groups of interest characterized by different opinions concerning regulatory proposals, in order to identify costs and benefits of different alternative actions outline of the proposal By introducing in Factorial Conjoint Analysis Textual Data Analysis tools (Balbi, Infante, Misuraca, 2008), it is possibleto cluster judges, on the basis of their preference structure referred both to alternative public actions and their free description of a desired intervention

3. theoretical frame NTTS 2009 • Regulatory Impact Analysis • Factorial Conjoint Analysis • Textual External Information • Characteristic words

4. NTTS 2009 Regulatory Impact Analysis (RIA) "RIA encompasses a range of methods aimed at systematically assessing the negative and the positive impacts of proposed and existing regulation“(OECD) • RIA contributes to the creation of an open, transparent, and empirically based regulatory system • (The European Policy Centre) • People consensus is a key factor for reducing the risks of regulatory failure • In political processes, it is important to knowthe actors in favor and the actors adverse to specific interventions

5. NTTS 2009 Factorial Conjoint Analysis Conjoint Analysis (C.A.) seems particularly appropriated since, starting form different regulatory stimuli, it allows to decompose several evaluation dimensions. It enables to estimate the partial utility coefficients of different factor-levels and to define groups of users on the basis of their response similarity • A Factorial Approach to C.A.* allows to get the following purposes: • to synthesize the individual judgments reconstructed directly by the model on the principal axes obtained by their linear combination, • to look for an optimal synthesis of such judgments according to the perceived benefits, • to build two-dimensional graphics (factorial maps, called perceptive maps) for the study of the existing relationships among designed stimuli (normative options), judgments and descriptive levels (core indicators), with the further possibility to underline the different evaluation structures expressed by several groups of judges. *Lauro Giordano Verde 1998

6. A sample of citizens • A sample of experts Y JUDGES g1 g2 ………………...….......gN several criteria: expected benefits, expected public utility, strategic priority indirect net benefits, and so on…… S1 S2 S3 Sq STIMULI OR DIFFERENT OPTIONS ratings or ranking the factors should take into account organizational ,financial,economic and social aspects, criticality and so on.. NTTS 2009 Factorial Conjoint Analysis for RIA* We project and administer a questionnaire for collecting opinions expressed in relation to a set of stimuli simultaneously described by the relevant dimensions (core indicators) *Scepi,Giordano Lauro 2008

7. PUBLIC STANDARDISE PUBLIC With AUTONOMY PRIVATE STANDARDISE PRIVATE With AUTONOMY NTTS 2009 Textual Information - Motivations usually, judges are asked to compare different potential alternatives (in order to ranking or scoring), by the so called full profile method (oldest but still used): a limited number of attributes is used to describe the product or service, but sufficient cards or treatments are shown to one respondent in order to compute his/her utilities coefficients Ex. Which model of Italian University, do you want to work in?

8. NTTS 2009 Textual Information - Motivations the data collection is a critical step in CA: • it is difficult for judges comparing a huge number of potential actions, described in a complex way • a small number of categoricalvariables (with a small number of categories) is rigid A free description of the desired public action can be useful in order to introduce elements do not consider in CA

9. datastructure 1 . . . . . . . G 1 . . . . . . . G 1 . . . . . . 1 . . . . . Z K v p 1 . . . . . . . G 1 . . . . . L 1 . . . 1 . . . Y X s s NTTS 2009 in Z we consider the p characteristics and behaviors of the G interviewed in K we consider the v terms used in the open question by the interviewed X and Y are classically used in the C.A., respectively: the experimental design and the preference data matrices

10. 1 . . . . . . . G 1 . . . . . L 1 . . . Y X s 1 . . . . . . T v 1 . . . . . . . G 1 . . . . . Z p NTTS 2009 thestrategy 1 . . . K is tranformed in the presence/absence matrix T s 1 . . . . . . . G 1 . . . . . L 1 . . . . . . Q v Z is projected as supplementary information after analysing the Q matrix

11. the Q analysis NTTS 2009 the analysis can be seen in a factorial analysis framework by considering the decomposition of Q trough a singular value decomposition in this way we obtain a graphical representation of the terms and the C.A. level, in which we can also consider as supplementary points the information dealing with the interviewed individuals

12. Reforming University in Italy NTTS 2009 • in Italy there is a wide debate concerning with the university system • a sample of 30 university professors, with different backgrounds and experiences, have been asked to describe their ideal university, before ranking different options in a full profile Conjoint Analysis • the levels and the factors used in the C.A. have been chosen by considering different political proposals under debate

13. cultural Formative Target professional NTTS 2009 conjointanalysis – full profile public Management public/private public Teacher Recruitment private Legal value of the certificate YES standardise NO Formative Path autonomous

14. NTTS 2009 Ideal University: partial utilities • public management • public/private man • public recruitment • private recruitment • standardize path • autonomous path • cultural target • private target • legal value YES • legal value NO

15. Graphical representation of C.A. levels NTTS 2009 no legal value professional public standardized autonomous public recruitment private recruitment public/private legal value cultural t1=31% t2=29%

16. Graphical representation of personal characteristics NTTS 2009 > 9 years exp full professor associate professor researcher 4-9 years exp <=3 years exp t1=31% t2=29%

17. Graphical representation of words t1=31% t2=29%

18. Clusteringjudges the proposed strategy allows a final clustering of judges, based on Factorial CA results and described by characterizing words, as usual in textual data analysis here we have identifies 5 groups: • G1: working in a university connected with the external world (società, ruolo, sistema) also by financing (valutare, finanziare) • G2: working in a public university with public recruitment (garantire, pubblico) • G3: working in a university “research oriented” (ricerca) • G4: working in a university “labour market oriented” (di base, professionale, formativo) • G5: The fifth class is pragmatic: not reference to ideal worlds(potere, dovere), the characteristic words looks at results (produttivo, utile) but not at students (studente has frequency equal to 0)!