1 / 51

Web-based quantitative analysis and reporting of program outcome coverage and student performance

Web-based quantitative analysis and reporting of program outcome coverage and student performance. Paul Van Halen vanhalen@cecs.pdx.edu. PRESENTATION GOALS. Gain/improve understanding Quantitative course-to-program outcome mapping philosophy

umay
Download Presentation

Web-based quantitative analysis and reporting of program outcome coverage and student performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Web-based quantitative analysis and reporting of program outcome coverage and student performance Paul Van Halen vanhalen@cecs.pdx.edu

  2. PRESENTATION GOALS • Gain/improve understanding • Quantitative course-to-program outcome mapping philosophy • Importance of designing course assessment tools with program assessment in mind • Capabilities of open-source software based program assessment framework Quantitative Program Assessment

  3. OUTLINE • Background • Motivation • Implementation • Course Assessment • Program Assessment • Results • Challenges • Future Development • Conclusions Quantitative Program Assessment

  4. BACKGROUND • Oregon University System • 81,000 students • 7 universities: PSU, UO, OSU, OIT and 3 regional schools • OSU and PSU offer engineering degrees • Portland State University • Located in downtown Portland • With 24,000 students, the largest enrollment in the state system Quantitative Program Assessment

  5. BACKGROUND • Maseeh College of Engineering and Computer Science • Electrical & Computer Engineering • Mechanical & Materials Engineering • Civil & Environmental Engineering • Engineering & Technology Management • Computer Science • 70 full time faculty • 1266 undergraduate, 492 graduate students Quantitative Program Assessment

  6. BACKGROUND • Electrical & Computer Engineering • 2 ABET accredited programs: Electrical Engineering and Computer Engineering • 20 faculty • 290 bachelor students • 164 MS and MEng Students • 42 PhD students Quantitative Program Assessment

  7. MOTIVATION • ABET 2000 criteria • False starts • College initiatives • University assessment initiative • What are our program objectives? • What are our program outcomes? • Once we have program outcomes how do we find out how well we address them and how do we perform? Quantitative Program Assessment

  8. MOTIVATION • How does this connect with the grades we give our students? • We should be able to combine the grading data to assess our stated program outcomes both in coverage and student performance • Course outcomes and course assessment are a prerequisite • How do we get from course assessment to program assessment? Quantitative Program Assessment

  9. PROGRAM ASSESSMENT • Assessment in most cases is still focused on course assessment • Convincing faculty to participate in this endeavor is a challenge • Good course assessment does not necessarily compile to a meaningful program assessment, unless the course assessment was planned with program assessment in mind. Quantitative Program Assessment

  10. IMPLEMENTATION • Program objectives • Program outcomes • Program outcomes to objectives mapping • Course outcomes • Course outcomes to program outcomes mapping Quantitative Program Assessment

  11. PROGRAM OBJECTIVES The electrical engineering program has the following educational objectives: • Knowledge. To provide our students with a broad knowledge base in the fundamentals and techniques of the engineering sciences, required for engineering careers in a changing technical environment, to prepare them for successful participation in multi-disciplinary teams. • Application. To provide our students with an in-depth knowledge of the concepts, techniques and tools of the electrical engineering discipline and to impart the ability to apply their proficiency to engineering design and problem solving. Quantitative Program Assessment

  12. PROGRAM OBJECTIVES • Innovation. To provide our students with the ability and desire to continually renew their education in a rapidly developing discipline, enabling them to participate in the research and development of the discipline and to realize their full potential throughout their career. • Community. To ensure awareness of: • the need for personal development, both in discipline related aspects and in terms of understanding the impact of the profession on social and environmental issues. • the importance and benefits of personal involvement in professional societies and local communities. Quantitative Program Assessment

  13. PROGRAM OUTCOMES • Added more specific EE and CpE outcomes to the existing (a) – (k) ABET outcomes. • Created a mapping from the program outcomes to the program objectives. • UPDATE: • Detailed outcomes are TOO fragmented to guide the decision making process • Automatic aggregate of details into (a) – (k) is more helpful in loop closure on program assessment Quantitative Program Assessment

  14. Quantitative Program Assessment

  15. NUMERICAL COURSE TO PROGRAM OUTCOME MAPPING • Instructor in charge of a course has to provide a list of course outcomes as part of a standard syllabus for the course • Instructor also has to provide a weighted mapping of course to program outcomes • Provides relative importance of each course outcome • Provides information on how a particular course addresses program outcomes • Mapping is normalized with respect to credit hours Quantitative Program Assessment

  16. Quantitative Program Assessment

  17. COURSE TO PROGRAM OUTCOME MAPPING • Concepts and limitations of OPAMP-based circuits: 10%; outcome (a) and (c): 50/50 • Concepts of small-signal modeling and amplification: 10%; outcome (a), (c): 50/50 • Analyze and design single-transistor gain stages: 15%; outcome (c): 100 • Analyze and design multistage/differential amplifiers: 15%; outcome (c) and (k): 50/50 • Analyze and design amplifier biasing schemes: 10%; outcome (c) and (k): 50/50 • Frequency response of transistor-based gain stages: 10%; outcome (c): 100 • VCO design project: 30%; outcome (c), (g) and (i): 40/50/10 Quantitative Program Assessment

  18. Quantitative Program Assessment

  19. Quantitative Program Assessment

  20. Quantitative Program Assessment

  21. PROGRAM OUTCOMES SUMMARY • Program outcome mapping from all courses can be combined into a single table • Summary table shows program outcome coverage, or lack thereof Quantitative Program Assessment

  22. Quantitative Program Assessment

  23. PROGRAM OUTCOMES SUMMARY • Observations: • 57.5 credits of required ECE courses in the program • 25.7 credits focused on outcome category (a) • 14.7 credits focused on outcome category (c) • No coverage of outcome category (j) • Coverage appropriate? • Conclusion: Room for improvement Quantitative Program Assessment

  24. COURSE GRADING AND ASSESSMENT • Excel spreadsheet • For every item that is part of the grade • Maximum grade • Mapping to course outcome • Grade for every student Quantitative Program Assessment

  25. COURSE GRADING SHEET Quantitative Program Assessment

  26. COURSE OUTCOME ASSESSMENT Quantitative Program Assessment

  27. VISUAL DISPLAY OF RESULTS • Graphics have been updated to improve communication of results • Use JFreeChart as graphics vehicle • Credit to “Creating Clear and Concise Visual Display of Information”, K.C. Dee, G. Livesay, BAP IX Quantitative Program Assessment

  28. Quantitative Program Assessment

  29. Quantitative Program Assessment

  30. PROGRAM ASSESSMENT RESULTS • Summary of performance for each of the (a) – (k) ABET outcomes • All course in the EE program for which data was available (110 credit hours, 3744 student credit hours: Σ (# of credit hours) x (# of students) • For each of the (a) - (k) outcomes: • Weight in the overall assessment • Grade distribution plot Quantitative Program Assessment

  31. Quantitative Program Assessment

  32. PROGRAM OUTCOME COVERAGE • Table compares the coverage of program outcomes according to course syllabi (required and electives) and assessment data • EE required courses: 66 credit hours • EE elective courses: 132 credit hours • Is coverage appropriate ? Quantitative Program Assessment

  33. EE PROGRAM OUTCOME COMPARISON Quantitative Program Assessment

  34. TECHNICAL DETAILS • Use Apache Software Foundation tools as a framework: • Tomcat is a servlet container/server • Cocoon is a web development framework which provides us with a component-based ability to dynamically generate, transform, serialize web pages from a variety of data sources through “component pipelines” • Forrest is a Cocoon-based publishing framework that transforms data from various sources into a unified presentation, including web site navigation/decoration Quantitative Program Assessment

  35. TECHNICAL DETAILS • Data storage: • All data stored in a RDB, postgresql. • Data stored as a mixture of relational fields and xml text • XML text can be accessed through XPath functions • Every faculty member has his/her own table which is derived from a “master” table. • Faculty has read/write permission on their own table and read permission on the master table enables data collation Quantitative Program Assessment

  36. TECHNICAL DETAILS • Data stored in faculty tables for each course: syllabus, outcome mapping, rubrics/metrics, assessment data • Data entry and manipulation: • Web-based cocoon forms, using AJAX Quantitative Program Assessment

  37. TECHNICAL DETAILS • Majority of pages are dynamic, XSP pages contain data base queries. • XSP formats query results in XML • XML results are transformed in one or multiple stages into HTML Quantitative Program Assessment

  38. TECHNICAL DETAILS • Avoids problems: • Platform independent: faculty on Windows, Mac, Linux, Unix • XML content in database can be fixed, modified, augmented by the database owner Quantitative Program Assessment

  39. Quantitative Program Assessment

  40. Quantitative Program Assessment

  41. FLOW EXAMPLE • http://abet.cecs.pdx.edu/ece/quarterAssessmentList/EE/courseAssessListxsp.html • URL invokes pipeline in cocoon sitemap <map:match pattern="quarterAssessmentList/*/*xsp.xml"> <map:generate type="serverpages" src="{properties:content.xdocs}/courses/quarterAssessList.xsp"/> <map:transform type="xslt" src="{properties:forrest.stylesheets}/ece/assessmentQuarterSelect.xslt"> <map:parameter name="siteurl" value="http://abet.cecs.pdx.edu"/> </map:transform> <map:serialize type="html"/> </map:match> • The first step in this pipeline is the execution of an XSP page: quarterAssessList.xsp • The resulting XML data is rendered into a form through the assessmentQuarterSelect.xslt stylesheet • Submission of this form replaces the current page with a new URL which contains the form results as parameters http://abet.cecs.pdx.edu/ece/programAssessmentList/courseAssessListxsp.html?prog=EE&qrts=200704,&lvl=A Quantitative Program Assessment

  42. FLOW EXAMPLE: quarterAssessList.xsp <xsp:page language="java" xmlns:xsp="http://apache.org/xsp" xmlns:xsp-request="http://apache.org/xsp/request/2.0" xmlns:esql="http://apache.org/cocoon/SQL/v2"> <xsp:structure> <xsp:include>java.text.*</xsp:include> </xsp:structure> <content> <xsp:logic> String uri = <xsp-request:get-sitemap-uri/>; String splt[] = uri.split("/"); String program = splt[1]; </xsp:logic> <title>Assessment quarter selection</title> <program><xsp:expr>program</xsp:expr></program> <esql:connection> <esql:pool>dbname</esql:pool> <esql:execute-query> <esql:query>select distinct quarter from dbtable where rowtype='assessment' order by quarter;</esql:query> <esql:results> <quarters> <esql:row-results> <quarter><esql:get-string column="quarter"/></quarter> </esql:row-results> </quarters> </esql:results> </esql:execute-query> </esql:connection> </content> </xsp:page> Quantitative Program Assessment

  43. FLOW EXAMPLE: assessmentQuarterSelect.xslt <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xalan="http://xml.apache.org/xslt" exclude-result-prefixes="encoder xalan"> <xsl:param name="siteurl"/> <xsl:template match="/"> <document> <header> <style> .body {font: normal 10pt Arial, sans-serif} </style> <title>Program Assessment Plot detail and Quarter(s) Selection</title> </header> <body> <script> function submitted() { var cl = document.qlist; var e = cl.quarter; var m = cl.level; value = ""; var lvl = ""; for (var i = 0; i &lt; e.options.length; i++) { if (e.options[i].selected) { value += e.options[i].value + ","; } } for (var j = 0; j &lt; m.length; j++) { if (m[j].checked) { lvl = m[j].value; } } window.location.replace("<xsl:value-of select="$siteurl"/>/ece/programAssessmentList/" + "courseAssessListxsp.html?prog=<xsl:value-of select="/content/program"/>&amp;qrts=" + value + "&amp;lvl=" + lvl); } </script> <p style="font: bold 10pt Arial, sans-serif">Select plot detail and quarter(s) to include for program assessment</p> <form name="qlist"> <table border="0" class="ECETable" style="padding: 50px; border-collapse: collapse; width: 50%; "> <tbody> Quantitative Program Assessment

  44. FLOW EXAMPLE: assessmentQuarterSelect.xslt (cont.) <tr> <td width="15%" align="right" style="padding: 6px; vertical-align: top; font-weight: bold;">Plot detail</td> <td style="padding: 6px; vertical-align: top; font-family: font-weight: bold;"> <input type="radio" checked="true" name="level" value="A">Program Outcomes Category Summary (a-k)</input> <input type="radio" name="level" value="B">Program Outcomes Detail</input> … </td> </tr> <tr><td colspan="2" style="padding: 6px; vertical-align: top; font-weight: bold;"/></tr> <tr> <td width="15%" align="right" style="padding: 6px; vertical-align: top; font-weight: bold;">Quarter(s)</td> <td style="padding: 6px; vertical-align: top; font-family: font-weight: bold;"> <select name="quarter" size="10" multiple="true"> <xsl:for-each select="/content/quarters/quarter"> <xsl:sort select="."/> <xsl:element name="option"> <xsl:attribute name="value"><xsl:value-of select="."/></xsl:attribute> <xsl:value-of select="."/> </xsl:element> </xsl:for-each> </select> </td> </tr> <tr><td colspan="2" align="center"><input type="button" value="Submit" onclick="submitted()"/></td></tr> </tbody> </table> </form> </body> </document> </xsl:template> </xsl:stylesheet> Quantitative Program Assessment

  45. CHALLENGES • Faculty participation • How do you get faculty engaged? • Version control OR How useful is feedback information when we change the feedback system? • Incorporating constituents feedback Quantitative Program Assessment

  46. UPDATES • Convert data storage from file based in faculty web directories (~/public_html) to a postgresql database • Most data stored in an xml text block in the database • Each faculty member has a table derived from a ‘faculty’ master table • Faculty can modify/update their own table, master user has access to ALL tables Quantitative Program Assessment

  47. UPDATES • Use XSP to retrieve postgresql data • Replace Altova Authentic with Cocoon forms for data entry/update to make the whole process truly web based • Postgresql database provides improved data handling and revision control Quantitative Program Assessment

  48. LIVE ACCESS • http://abet.cecs.pdx.edu/ece Quantitative Program Assessment

  49. FUTURE • Calibrate assessment results: • Incorporate survey and focus group results in the feedback loop Quantitative Program Assessment

  50. CONCLUSIONS: Closing the loop … • The quantitative program assessment outlined in this session provides fact based, data driven feedback in the continuous program improvement cycle mandated by ABET, and facilitates a better quality management decision process • Benefits all constituents Quantitative Program Assessment

More Related