John Rose 1 Michiel Bliemer 1,2 1 The University of Sydney, Australia

1. Advances in the Construction of Efficient Stated Choice Experimental Designs John Rose 1Michiel Bliemer1,2 1 The University of Sydney, Australia 2 Delft University of Technology, The Netherlands

2. Contents • Efficient Designs Defined • State of Practice in Experimental Design • Efficient Designs for Stated Choice Experiments • Bayesian Efficient Designs • Example • When is an Orthogonal Design Appropriate? • How can I do this?

3. Efficient Designs Defined

4. What are efficient designs? • Based on a design, a survey is composed and the outcomes of the survey are used to estimate the model parameters • The more reliable the parameter estimates are (i.e., having small standard error), the more efficientthe design is 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 experimental design respondents data estimation results

5. What are efficient designs? • The asymptotic variance-covariance (AVC) matrix is an approximation of the true variance-covariance matrix • “Asymptotic” means • assuming a very large sample; or • assuming a large number of repetitions using a small sample • The roots of the diagonals of the variance-covariance matrix denote the standard errors variance- covariance matrix where is the standard error of parameter

6. Asymptotic variance-covariance matrix • Efficient Design: • Generate a design that when applied in practice will likely yield standard errors that are as small as possible 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 experimental design respondents data estimation AVC matrix

7. State of Practice in Experimental Design

8. where is the data or design • The diagonal elements of will be made as large as possible and the off diagonals equal to zero • If we take the inverse of the diagonal elements will be minimised whilst the off diagonals remain zero State of Practice • In linear regression models: variance-covariance matrix = • If X is orthogonal, then

9. where is the data or design • will be maximised State of Practice • In linear regression models: variance-covariance matrix = • If the diagonal elements as small as possible… • And the zero off-diagonals suggest no multicolinearity

10. State of Practice Question: For discrete choice data, what type of econometric model do we typically employ? Question: Is the variance-covariance matrix of the logit model represented by ? Answer: Logit model

11. Efficient Designs for Stated Choice Experiments

12. Efficient Designs and Logit Models • The variance-covariance matrix for logit models is related to the log-likelihood of the model Note that: • In estimation, given the (design) data X and the observations y, one aims to determine estimates such that is maximised[maximum likelihood estimation] • When generating an experimental design, these parameter estimates are unknown • The values of depend on the model used (MNL, NL, ML)

13. Efficient Designs and Logit Models • The second derivatives of the log-likelihood gives the Fisher information matrix: • The negative inverse of the Fisher information matrix yields the model variance-covariance matrix [Hessian matrix of second derivatives] [negative inverse matrix]

14. with Efficient Designs and Logit Models • Example: MNL model with generic parameters(McFadden, 1974) First derivative: Second derivative: Note: y drops out!

15. Assuming that all responds observe the same choice situations, Therefore, the AVC matrix becomes: Efficient Designs and Logit Models • Example: MNL model with generic parameters (cont’d)

16. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 (sample size) 0.1 0 5 10 15 20 25 30 35 40 45 50 Efficient Designs and Logit Models • Example: MNL model with generic parameters (cont’d) “A design that yields 50% lower standard errors requires 4 x less respondents”

17. Efficient Designs and Logit Models Investing in more respondents Investing in better design standard error standard error 0 10 20 30 40 50 0 10 20 30 40 50 (sample size) (sample size)

18. Efficient Designs and Logit Models • Numerical example MNL model: Priors: Design:

19. Efficient Designs and Logit Models • Sample size and designs

20. Efficient Designs and Logit Models Design 1: Design 2: Which design is more efficient?

21. Efficiency measures • In order to assess the efficiency of different designs, several efficiency measures have been proposed • The most widely used ones are: • D-error • A-error • The lower the D-error or A-error, the more efficient the design [determinant of AVC matrix] [trace of AVC matrix] number of parameters (size of the matrix), used as a scaling factor for the efficiency measure

22. Bayesian Efficient Designs

23. Bayesian efficient designs • Efficient design • Example: Find D-efficient design based on priors • Bayesian efficient design • Example: Find Bayesian D-efficient design based on priors

24. Bayesian efficiency measures • Bayesian efficiency is difficult to compute, it needs to evaluate a complex (multi-dimensional) integral • However, it is nothing more than a simple average of D-errors: where are random draws from the distribution function (we take r = 1,…,R draws) Bayesian D-error = Bayesian D-error ≈

25. How to obtain priors? • Prior parameter estimates can be obtained from: • the literature • pilot studies • focus groups • expert judgement • If no prior information is available, what to do? • Create a design using zero priors or use an orthogonal design • Give design to 10% of respondents • Estimate parameters, use as priors • Create efficient design • Give design to 90% of respondents • Create a design using zero priors or use an orthogonal design • Give design to 100% of respondents

26. Example

27. Example Let S = 12

28. Orthogonal Design Db-error = 1.058 N = 316.28

29. Efficient Design Db-error = 0.6617 N = 158.22

30. When is an orthogonal design appropriate?

31. Example Let S = 12

32. Orthogonal Design Db-error = 0.3572 N = ?

33. So how can I do this?

35. Ngene Software

36. Ngene Software