CCMC Corporate Comprehensive Management Consultants

1. CCMC Corporate Comprehensive Management Consultants File-1 “People Drive the Process of Excellence”

2. Workshop on Method Engineering for Assessment Centers at the Pre-conference workshop of the Annual Conference of ACSG, Stellenbosch- South Africa Dr. S. Pandey Corporate Comprehensive Management Consultants India, Bombay 16th March, 2011

3. Objective To provide familiarity with relevant concepts, techniques and skills pertaining to Method Engineering for Assessment center (AC) design.

4. Goals (expectation) • An appreciation program to acquire new AC-community member and refresher to existing members (members include both buyers and suppliers of AC) • Participants should be able to make informed decision while selecting or designing a set of methods for an Assessment center. This implies being able to differentiate good methods from bad methods for a given implementation context. Getting out of the myth that there are good methods irrespective of implementation context • Participants should be able to identify needs for customization and some ideas to customize a method • Ultimately, some participants may chose method engineering as a profession and commit personal energy to advance research and design of AC-Methods, thereby contribute to the community

5. Most essential element of the goal to me today is • Your inquisitiveness (literally, the state of active curiosity) • Our assumption is that for Adult learner “if inquisitiveness is there everything else can be arranged”

6. Session-1 AS practitioner you know the fundamental strength of AC-approach; reduction of error in human attribute measurement. Concepts of errors in the context of Human attribute measurement: • Sources of errors and controls of errors • Method related errors; when your Method is imperfect your measurement data is contaminated by measurement variance and not the construct variance. • Essence of reliability (specially, inter-rater reliability) validity (specially, content and predictive validity) • Measurement of deterministic vs. probabilistic (stochastic) entities – single data point vs. expectancy of a distribution - limits of error tolerance.

7. Session-1: Role of methods in AC • Role of methods in AC • (Methods) Stimulus-Response (behavior) • Concept of behavior variance • Introduction to relevant concepts of experimental design and naturally occurring experiments

8. Who is Method Engineer? • For a detail understanding of Method Engineer’s role refer to ‘Method Engineer’s task flow chart elsewhere in this workshop. • For the timebeing let us note the following. • Method Engineer has the responsibility of design & development of Methods, and also the entire implementation of the AC/DC design. They convert scientific principles into pragmatic/ field application. • Methods & Processes are their domain that mediate between AC-design and AC-goals • Like any other Engineers they often conduct field tests of prototype of Methods & processes of a design to validate earlier laboratory tests and to obtain broader feedback for refinement

9. Sources of Errors in Human Assessment Assessment Error = ƒ (Assessor xAssessment methods x Assessment Factors) • Trained Assessors (ORCE, Factor & Method familiarity) • Accredited (for both Knowledge & skills) for Quality • Standardized methods • Adequate Quality in data generation. [A Method Engineer by design can compensate Assessor’s skill deficiency] • Standard administration process with approved flexibility for customization • Simple in structure but profound in eliciting behavior • Unbiased to Assessees irrelevant background e.g., age, race, gender. • Competency research based Assessment factors/criteria

10. Common Assessment Errors: vis-à-vis Method Engineers responsibility

11. Assessment Errors • Not knowing What to Observe –The answer is Factors Assessment Centre approach ensures relevant factors through: • Analysis of the Job • Deriving a standardized set of Competency Factors to be assessed • Identifying observable behaviors linked to the factors. • Observing, recording, diagnosing the pattern and rating of behaviour

12. Assessment Errors….contd. 2. Not knowing How to Observe- The Methods • Assessment Centre follows rigorously, simple standardized methods that simulate the job partially/fully. 3. Not knowing Who should Observe – The Assessor • Assessment Centre approach helps developing guidelines for assessor selection and training

13. Linking to job Defining Relating to observable behaviors Personal level capability; Namely, Knowledge, Attitude, Skill Rating errors Common Assessment Errors Method Factor Assessor Minimizingerrors • Simplification • Rigor (design) • Standardization

14. Conscious incompetence 3.Conscious competence • Unconscious incompetence • Unconscious competence Energy Level Hi Lo Lo Hi Effectiveness / Results

15. Session-2 Design elements of a battery of Methods - Methods taxonomy and illustration – types of competencies vis-à-vis fitting Methods – types of AC outcome and Method fit – brief ideas of subtler types, as for example, based on degree of rigor or skill levels of Assessors and so on.

16. Session-3 • Design elements of Methods – method specifications elements – Method content design (lead, casting etc.) - Method process design (control) – pilot and standardization (reliability, security, content validity and true validity etc.) – Method testing • Concepts of single stage vs. multi-stage Methods - parallel Methods – sequential supplementary Methods

17. Hands on exercise Design of a prototype on the spot (Any one of the following: case method/ role play method/ on the spot exercise method/ advance method/ simple work simulation)

18. Session-4 • Practice tips – Take home assignment suggestions • Suggestions for applied research – suggestions for basic research

19. Simple criteria to improve Content Validity Exercise • Candidates’ Tasks on the Method should reflect the critical activities of the job • The choice of Methods batteries should reflect Organization’s Values, principles and culture • The Format of the Method and the Process in which participants are asked to respond should match the reality on the job • The difficulty level and complexity of Methods must fit the ‘difficulty level and complexity of the job. • The Task on the Method must be aligned to intrinsic Strengths of the Method (not weakness)