1 / 22

Graduate Programming Examination (GPE) - Experiences and Proposal of GPE rules

Graduate Programming Examination (GPE) - Experiences and Proposal of GPE rules. 黃世昆 交通大學資訊工程系. GPE EXPERIENCES. 到底 GPE 完成了什麼事情？. 大家都會很懷疑到底 GPE 完成了什麼事情？ 不就是 PC^2 架一架、電腦教室一開，就開始了？ 中山與師大早就辦過多次的 two-site contest ？ 這有什麼特別。. 程式「賽場」與「考場」的差別. 賽場 比賽用的（如 ACM 亞洲區賽、大專杯程式競賽） 環境專屬

miron
Download Presentation

Graduate Programming Examination (GPE) - Experiences and Proposal of GPE rules

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Graduate Programming Examination (GPE) - Experiences and Proposal of GPE rules 黃世昆 交通大學資訊工程系

  2. GPE EXPERIENCES

  3. 到底 GPE 完成了什麼事情？ • 大家都會很懷疑到底 GPE 完成了什麼事情？ • 不就是 PC^2 架一架、電腦教室一開，就開始了？ • 中山與師大早就辦過多次的 two-site contest？ 這有什麼特別。

  4. 程式「賽場」與「考場」的差別 • 賽場 • 比賽用的（如ACM 亞洲區賽、大專杯程式競賽） • 環境專屬 • 正式比賽封閉網路 • 比賽機器只能連 server，不能連外、不能連內部其他比賽機器 • 考場（如程式設計上機考試） • 考試用的（如各學科的上機考試） • 環境不是你的、但要清空與考試無關的設定與資料 • 封閉網路（斷網、也不能連其他人的機器）

  5. 賽場、與考場 • 大賽場頂多五年布置一次（但以前師大、中山例外，每年一兩次），小賽場頂多一年一、兩次。 • 我只有主持四次小賽場經驗（三次 PC^2，一次 GPEDomjudge)、一次大賽場經驗(使用PC^2) • 經費：幾萬到幾百萬 • 人力：數十人 • 考場每年多次 • 我有主持小考場近50次的經驗 (20 多次PC^2, 26 次 GPEDomjudge), • 二次大考場（二次 GPE，都使用 GPEDomjudge) • 經費：幾千到幾萬 • 人力：2人（小考場）到10人（大考場）

  6. GPE 要完成什麼？ • 同時開100個程式考場（每個考場容納 20-200 人） • 有限的人力、經費 • 每年可以多次舉辦

  7. GPE Systems 1. GPE Exam Client 2. GPE Judge System (On-line and On-site integration)

  8. Judge Server: GPEDomjudge • on-site judge 方面 • 放棄著名的 PC^2 系統 (最近兩屆 world final 都未採用 PC^2 ) • 換成另外一個 on-site judge: domjudge。 • 我們已進行大部分的翻修，可適應未來頻繁的 GPE 活動與管理， • 具備 on-line(not on-site)/on-site雙重功能的 judge server。 • 可連至以下網站，申請帳號或以 guest 訪客登入測試： • http://progexam.cs.nctu.edu.tw/ • 以往 PC^2 賽前預備費時、狀況頻繁，但採用此 judge server，幾乎隨時可立即比賽，沒有什麼狀況。 • Account 管理（報名、資料轉換）、密碼製作 • PC^2 舊版無法跨 NAT、測資 bug • 沒有 source，無法有效修改

  9. Exam Client Environment • 採用 virtual machine: virtualbox, • VM image: CentOS linux (ICPC world final contest vm image) and FreeBSD • 各地區中心的電腦環境互異，但利用 virtual machine 可快速建立相同的比賽環境。 • 比賽者 login 當地 PC 電腦之後，login script 將自動切換至 remote desktop，進入 local virtualbox 所 boot 的 centos 比賽環境。 • 此 login 帳號將 disable ctl-alt-delete。 • 此環境將經由 VPN 或 IP firewall 限制與 on-site judge 進行比賽連線。 • 未來目標是在半小時內，切換超過1000台的 PC，進入完全相同的比賽環境。 • Oct 11/2010 交大考場，半小時內切換100台 PC為考場

  10. Microsoft Azure Cloud Container and GPE Microsoft will place in Chicago 112 containers holding 224,000 servers, each container with 2,000 servers GPE: distribute 2,000 exam clients into more than 20 sites Same issues: ease of maintenance, ease of resource scheduling Deployment of large number of VMs

  11. Outline • Problem Description • Potential Solutions • Proposed Solution • Experiences of two GPEs • Issues in this GPE • Topics to be Discussed • GPE Operation Procedure

  12. Problem Description • To evaluate the readiness of basic programming capability for graduate level work. • on-site on-line programming exam（現場上機程式考試） • 1500 students for graduate school entrance exam • 1500 computers （必須準備1500台電腦） • Mission Impossible （不可能的任務）

  13. Potential Solutions • Program Understanding Exam • Trace Source Codewith Specified Running Results • Feasible in choice-type questions • Only evaluate the “Reading Capability” : read/modify/create capabilities are quite different • Programming in paper （程式寫在考卷上） • In pseudo code (already in algorithm exam) • Impossible to grade manually and automatically • Multiple times, 200 examines at a time • 8 exams with 8 different problem sets • Difficult to grade in the same ranking/scoring standard

  14. Proposed Solutions • Multi-sites on-site online exam • Each site with 50-200 computers (feasible in NCTU, and many other universities) • Jointly with 10-30 sites • Difficulties • A super-scale programming contest ( 120 on-site teams, claimed to be largest contest in the world) • Yes, zero-judge claimed to hold a 2000 teams on-line contest (not sure if they were on-site)

  15. More Issues • On-site exam • Restricted network connections • Restricted programming environment including software and hardware • The major Problem • Each site with different operating environment • You cannot impact the normal operations of computer rooms in more than 30 sites • You cannot expect service staff of 30 sites to hold a “programming contest” yearly • Where are the programming problem sets ?

  16. The Results of GPE trials on June 9 and Oct 11 • GPE on June 9 • 9 problems • 3 UVA level-1, 2 level-2, 3 level-3,1 level-4 • level-1 (solved in 10 minutes), level-2 in 15 minutes, level-3 in 30-100 minutes, level-4 in 120 minutes • 102 Registered (51 in NSYSU, 51 in NCTU) • 86 attend the on-site exam (84%) • Solved 7 problems (2 students), 6 (7), 5(12), 4(13), 3(17), 2(17), 1(10), 0(8) • GPE on Oct 11 • 9 problems • 1 UVA level-1, 2 level-2, 4 level-3,2 level-4 • level-1 (solved in 10 minutes), level-2 in 15 minutes, level-3 in 30-100 minutes, level-4 more than 100 minutes • 159 Registered (67 in NSYSU, 44 in NCTU, 18 in NTCU, 9 in NCU, FJU 19, 2 in CGU) • 124 attend the on-site exam (84%) • Solved 8 problems (2 students), 6 (3), 5(3), 4(5), 3(4), 2(15), 1(20), 0(79)

  17. Grading • June 9 • 2.3% solved 7 problems (A+) • 10.4% solved > = 6 problems (A) • 24.4% solved >= 5 problems (B+) • 39.5% solved >= 4 problems (B) • 59.3% solved >= 3 problems (C) ------------------- Graduate Programming Level------ • 79% solved >=2 problems (C-) • 90.6% solved >= 1 problems • 9.3% solved 0 problem • Oct 11 • 1.5% solved 8 problems (A+) • 3.8% solved > = 6 problems (A) • 6.1% solved >= 5 problems (B+) • 9.9% solved >= 4 problems (B) • 13% solved >= 3 problems (C) ------------------- Graduate Programming Level------ • 24.4% solved >=2 problems (C-) • 40% solved >= 1 problems • 60% solved 0 problem

  18. Resolved Issues • Unified Software Environment • Restricted Network Connections • Fast Deployment of the Unified Environment • With limited computing resource • Pre-forged Virtual Machines Images • Auto-update VM images • A Revised judge System • Why another judge system ?

  19. A Revised Judge System • Why not contest system in On-line Judge • UVA, TopCoder, Sphere, CodeChef • PKU judge (Binary release) • ZeroJudge (Open Source) • For on-line contest, but not for on-site contest • unpublished problem set • HDOJ : Programming Contest DIY • Why not On-site Judge • PC^2 (binary) • Ad hoc contesting (easy to install server and client) • Hard to manage account/problem set/scoring data • No restrictions on login source (IP) • Domjudge (Open Source) • Web-based • IP-based authentication

  20. Specific Requirements of GPE Judge • Published Problem Set with known solutions and reference implementations • Individual postpone of submission time • Due to instable machines within large number of hosts

  21. Issues in former GPEs • Too many Ajax updates, server jammed and slow response • Incomplete JDK installations of several judge servers • SSL client certificate (generate during auto-update) • screendump/command log regularly (monitor strategy for anti-clone, anti-cheating,...) • VM image partitions (core, fragile, local configuration) • machine crash SOP (adding delay for individual examine)

More Related