AHELO’s Civil Engineering Strand Reflections, outputs and lessons from the NPM perspective

1. AHELO – What can we Learn from OECD’s Feasibility Study? The Westin Georgetown Hotel, Washington DC July 11-12, 2013 AHELO’s Civil Engineering StrandReflections, outputs and lessons from the NPM perspective HeshamGomma, ADEC Satoko Fukahori, NIER Mary Catharine Lennon, HEQCO Daniel Edwards, ACER Facilitator: Ashley AterKranov, Washington State University

2. Overview of the Session • Introduction (Ashley AterKranov) • Why Civil engineering. • International quality assurance, and the importance of direct measures. • The AHELO experience • Within institutions (HeshamGomma) • Catalytic effects: The Abu Dhabi experience working with institutions and faculty. • Within a jurisdiction (Mary Catharine Lennon) • What information can a jurisdictional report provide, and why is it valuable? The Canadian experience. • Across jurisdictions (Satoko Fukahori) • What can be gained by sharing analytic methods, data, results, etc. across nations? Examples from a three country analysis : Australia, Japan, and Canada. • Lessons and the Future (Daniel Edwards and Mary Catharine Lennon) • FS data limitations, and future assessment design. • Discussion (facilitated by Ashley AterKranov)

3. Measuring if students can think like an Engineerhttp://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf (pp.252-264) The Hoover Dam is a 221-metre high concrete arch-gravity dam in the Black Canyon of the Colorado River in the United States of America. It was built to provide irrigation water, to control floods and to provide water for a hydroelectric power station at the base of the dam. Figure 1. Hoover Dam Figures 2-5. (Images of construction sites, and plans) • Explain why this is a good dam site for hydroelectric power generation. You should discuss at least two aspects. • a) Dam height/high potential energy, b) high flow rate of the river, c) lake capacity, d) minimal social impact, e) characteristics of the rock, f) narrow gorge. • Explain the two main design features that contribute to the structural strength and stability of the Hoover dam. • The maximum electrical power generated by the turbines at the Hoover Dam is 2.08*109.. What is the approximate amount of water that flows through the turbines at this power output, if the power station operates at 90% efficiency? • Imagine that a new dam is being planned today in a different location. Briefly explain two environmental effects of the dam (which could also be upstream or downstream) that an engineer would need to consider in an environmental impact statement.

4. Measuring the Basic Engineering Science Competency http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf (pp.265-266) A load P is applied to a Warren truss as shown below. If the self-weight of the members is ignored, which of the following statements is correct? • Compressive force exists in both the upper-chord member (p-q) and the lower-chord member (r-s). • Tensile force exists in both the upper-chord member (p-q) and the lower-chord member (r-s). • Compressive force exists in the upper-chord member (p-q), while tensile force is applied to the lower-chord member (r-s). • Tensile force exists in the upper-chord member (p-q), while compressive force is applied to the lower-chord member (r-s).

5. International Civil Engineering Strand

6. Guiding Questions • To whom of this of interest and why? • Did the results meet these needs and interests? • What purposes can professional field assessments serve in the future?

7. HeshamGomma Within institutions Catalytic effects: The Abu Dhabi experience working with institutions and faculty

8. Institutions expectations from AHELO • Leadership • Information on the investment efficiency • Performance of their graduates versus local/international peers • Baseline data on their institutions • International visibility and recognition • Faculty • International benchmark • Information about their graduate capacity • Data to review their programs • Students • Know the capacity of their institutions • Participate in international assessment • Support their institutions in international exercise • Visibility of their institutions

9. Engineering faculty perspective on AHELO • Assessment content • Partial irrelevance to Abu Dhabi Civil engineering programs • Captured only some topics • There is a need for Academic Reference Standards at the international level • Scoring rubric • Faculty did not participate in the development process (top-down) • Not always reflecting the student capacity • Data output • Contextual data reflects a realistic picture of the institutions context • The institutions data did not cover learning outcomes and cannot be utilized in reviewing individual institution program • The jurisdiction data is satisfactory for cross institutions analysis and collective program review

10. Outcomes at the institutional level • Institutions have not actively proceeded with data analysis • Responding to ADEC’s invitation, one institution out of the three institutions showed interest • To work on reviewing its curriculum and learning outcomes • To work with employers and other institutions on mapping and aligning learning outcomes with the employers needs • Talks are in place with the other institutions

11. Outcomes of AHELO at the Emirate level • Promoting the assessment culture at the HE level in Abu Dhabi • Developing Academic Reference Standards at the Emirate level • ADEC is on its way to develop two major assessment exercises at the Emirate level for the IT and Education fields

12. Mary Catharine Lennon Within a jurisdictionWhat information can a jurisdictional report provide, and why is it valuable?The Canadian Experience

13. Broad outcomes Collaboration/experience – KEY outcome from Feasibility Study • Countries working together • Institutions working together • Experts working together • Dialogue about learning outcomes expanding Analyses – Secondary outcome (Feasibility) • Existing analyses • OECD international reports on validity and reliability • Institutional reports • Jurisdictional reports --------------------- • Additional analysis: • Jurisdictional reports (internationally leveled data) with international comparisons • Comparative report of Japan, Australia and Canada (recalibrated data) • Combined data report examining contextual strand of Japan, Australia and Canada (recalibrated data)

14. Institutional reporting and analysis • Institutions provided a report on student and faculty context information, student scores and the international average • Institutional reports did not have an impact • Shared though committees • No institutional analysis conducted • Institutions were invited to data analysis discussion/training. • More interested in jurisdiction/international comparative report

15. Ontario’s Jurisdictional Report • Intended to provide Ontario (Canadian) stakeholders: • An overview of the entire AHELO project • A summary of the Ontario experience in administration and implementation • A de-identified cautious analysis of the data • A comparison to Australia • A discussion of the potential value of this type of assessment

16. Working with the data • Challenges of the jurisdictional report • Unable to provide detailed information on student competencies • Sample sizes too small to present detailed information by institution • Cautious not to stretch the data or read too much into it • Benefits from the jurisdictional report • Collaboration with Australia extremely helpful • Gaining better picture of characteristics of Civil Engineering departments and students • Providing programs with comparative information

17. Satoko Fukahori Across jurisdictionsWhat can be gained by sharing analytic methods, data, results, etc. across nations? Examples from a Three Country Analysis: Australia, Japan and Canada (Ontario)..

18. Sharing Data • Memorandum of Understanding • Signed by ACER, NIER, and HEQCO. • Intention of working together in good faith. • Requirement for written consent prior to dissemination of disclosed data and findings from data analysis. • Not intended for ranking purposes • Intended to highlight differences in systems. • Strengths and weaknesses, trends. • Intended to produce information that will benefit institutions in their effort to improve their education.

19. Working with data across jurisdictions • Benefits With caveats on data limitations: unrepresentative data. • Providing comparative information on systemic differences, which should be interpreted within policy contexts, and potentially used to draw policy implications. • Incorporating cross-jurisdictional information within jurisdictional reports (as in the Canadian case) will allow HEIs to potentially globally benchmark themselves, and understand their strengths and weaknesses. • Challenges • Data limitations of the feasibility study did not allow producing the kind of information that HEIs requested for. • Analysis on competence clusters - to be discussed in the next section.

20. Further work underway • Australia, Canada and Japan collaborating on academic papers exploring system differences. • One example underway is analysis focusing on Contextual Variables and the extent to which they explain scores when system differences are controlled for.

21. Daniel Edwards, Mary Catharine Lennon Lessons and the future

22. Key lessons for Civil Engineering GOOD • Introduced a much wider audience to the concept of learning outcomes in higher education. • Gained international consensus on detailed content of assessments. • Initiated new relationships nationally and internationally. Not so GOOD • Output data-wise not as fruitful…next slides explore this

23. Data limitations (Remember: Feasibility Study only!) • Not a nationally representative sample of institutions/programs/students. Cannot draw inferences about countries (or many institutions). • Limited test time, covering multiple items – focus on a wide range of content, concepts, logistics of translation and delivery. Less focus on producing meaningful data (esp. at small group level). • Only one ‘total’ score derived. Not able to analyse by competence clusters. • Tasks more challenging than expert group anticipated, so less variation in outcomes = less nuanced data

24. Future: Assessment design • Identify desired outputs; i.e. competence level detail. • Identify reporting levels; i.e. student, institution, jurisdiction, international. • Structure assessment to suit desired outputs.

25. Example: Eng Competence Clusters • In an ideal case: • Demonstrate the strengths and weaknesses of programs in developing different types of competences. • Analyze the relationships between competence clusters. • i.e. Does competence in the Basic/Engineering Sciences and in Engineering Generic Skills underlie competence in the Engineering Processes?

26. BUT BE REALISTIC! • Recognise that assessing to multiple levels of detail requires many items and very long assessments!

27. Future: Outputs • Identify desired outputs and reporting levels. • Establish agreed data sharing protocols (between governments, institutions or regions – whichever is relevant). • Ensure assessment and framework will deliver desired outputs.

28. These seem simple – but AHELO Feasibility Study makes this seem so… We have the horse…now for the cart!

29. Thank you