iRunner: an educational platform of Belarusian State University

1. iRunner: an educational platform of Belarusian State University Competitive Learning Institute Symposium 02 April 2019 Porto, Portugal

2. Overview • Belarus is a country in Eastern Europe • Belarusian State University (BSU) is our leading universitysituated in Minsk Minsk Porto

3. Overview • iRunner is a multifunctional web platform: • an online judge • a course management system • a contest management system • Available at acm.bsu.by, but there is no public registration • iRunner stands for ‘Insight Runner’ • The name is not related to iPhone or iPad: it was invented much earlier

4. History of iRunner • ViachaslavVdavichenka started the project in 2001 when he was a student • Since 2004 the system has been used in teaching process • Many students and teachers of the Department of Discrete Mathematics and Algorithms (supervised by professor Vladimir Kotov) have made contributions to the system core and its content • iRunner was successful, but it had several shortcomings • Finally, in 2016 the system was completely rewritten (we now call it iRunner 2)

5. Managing Contests • Western Subregional Contest of Northern Eurasia (a stage of ICPC) is annually held on iRunner • In 2018, 47 teams from four countries took part in the contest

6. Managing Contests • iRunner can handle programming competitions of different kinds • Moreover, the platform is used for training secondary school students and serving local Olympiads in informatics

7. Teaching • iRunner is a platform used for teaching university courses • Algorithms and Data Structures; • Programming; • UNIX-like Operating Systems; • Discrete Mathematics andMathematical Logic

8. Teaching ‘Algorithms and Data Structures’ • There are 340 academic problems along with 6.5K test cases that cover different topics of algorithm design and implementation: • Binary search trees, • Dynamic programming, • Approximate algorithms, • etc. • Each problem has a difficulty level (in Belarus we put levels from 4 to 10according to our local grading system) • A solution is a program written in any programming language (usually C++, Java, C#, Python) • Solutions are judged automatically, students get instant feedback

9. Teaching ‘Algorithms and Data Structures’ • Each student gets his own set of individual problems (usually one or two per topic) • A student has to do the following: • think of an algorithm that solves the problem; • talk to the teacher: describe his algorithm, prove its correctness and complexity; • get the OK mark from the teacher; • write a program in his favourite programming language; • submit the program to the automatic testing; • get the OK mark from the system (pass all the test cases).

10. Teaching ‘Algorithms and Data Structures’ • iRunner provides an interface to assign problems to students, to monitor their progress, to calculate statistics… • The system does not replace a human teacher, but it helps him

11. Teaching ‘Algorithms and Data Structures’ • There is also a set of common problems which should besolved by every student • They are used to train basic algorithms (DFS, BFS, Dijkstra, etc.)

12. Technical Details • iRunner consists of multiple modules • Main application • Is written in Python • Uses Django framework • Is fully cross-platform

13. Technical Details • Main worker that executes solutions • Runs on Windows only • Is written in C# and C++ • Uses built-in security features of the OS for sandboxing • Special UNIX worker • Runs on Linux • Is written in Python • Uses Docker containers

14. Key Features of iRunner • Let us name some key features of iRunner that distinguish it from similar packages

15. Storing Full Output • When executing a solution, we put all its output (files, text written to stdout/stderr) to the storage • If the solution is not accepted, the outputs help a teacher to diagnose a bug and to give a piece of advice to the student

16. Storing Full Output • When executing a solution, we put all its output (files, text written to stdout/stderr) to the storage • If the solution is not accepted, the outputs help a teacher to diagnose a bug and to give a piece of advice to the student

17. Content-based Hashing • iRunner uses content-based hashing and does not save the same data multiple times • SHA1-digest is used as a key to store the data 7 97 67 89 78 9 10 34 49 23 43 4 5 12 35 34 48 11 22 12 90 46 17 7 97 67 89 78 9 10 34 49 23 43 4 5 12 35 34 48 11 22 12 90 46 17 006ade87d75c5047e2a61691a38787eadf853d37

18. Rejudges • When test cases are added or modified, it is possible to retest a set of affected solutions • iRunnerfor each solutionstores the history of all runs • Rejudges are transactional • If something goes wrong during a rejudge (e.g. the new-added test is malformed), you can easily rollback your changes

19. Statements in TeX • We use custom TeX-like language for writing problem statements • The engine renders math and highlights code snippets Normal \textbf{bold}\textit{italic} Normal bolditalic \Examples \begin{example} \exmp{ 2 2 }{ 4 } \end{example} Examples

20. Statements in TeX • All the problems in our database look uniform and neat when viewed as HTML

21. Test Case Validation • It is possible to write validators for problems using testlib.hlibrary • A validator is a program that checks input data correctness #include "testlib.h" intmain(intargc, char* argv[]) { registerValidation(argc, argv); inf.readInt(1, 1000000000, "A"); inf.readSpace(); inf.readInt(1, 1000000000, "B"); inf.readEoln(); inf.readEof(); return 0; }

22. Challenges • This technique is useful when adding new test cases • Imagine you already have a set of accepted solutions for a problem • You submit new input data, and iRunner launches each of the solutions over this input • Then you can compare the outputs and find solutions that have failed the test

23. Challenges: Example • Consider the traditional A + B problem • Imagine that there are 10 accepted solutions • You submit the input “2 -3” to the systemand create a challenge • After a few seconds you see that • 8 solutions print “-1” • 1 solution prints “4294967296” • 1 solution crashes on this input • Then you analyze the results and probably add this test to the set of test cases

24. Arbitrary Problems • Many real-life tasks do not fit well into the traditional format of a competitive programming contest problem: • you may want to create multiple files or directories; • you may want to pass different command line arguments; • you may want to interact between processes; • etc. • iRunner lets you create highly customized problems using pytest library

25. Arbitrary Problems: Example • Consider a simple problem from the ‘UNIX-like Operating Systems’ course (training shell programming skills) Write a script in bash called hello.sh that takes a single argument—a name of a file— and writes the string ‘Hello, world!’ into that file. $ ./hello.sh foo.txt $ cat foo.txt Hello, world!

26. Arbitrary Problems: Example • A solution is a script that takes command line arguments and processes them in a special way • In order to judge such solutions automatically, we write a checker in Python language @pytest.mark.parametrize('name', ['foo.txt', 'bar baz.txt']) deftest(solution, name): subprocess.check_call(['/bin/bash', solution, name]) with open(name) as f: assertf.read() == 'Hello, world!\n'

27. Arbitrary Problems: Example • It gets Failed on test #1 AssertionError: assert 'Hello world\n' == 'Hello, world!\n' • Remember the statement: Write a script in bash called hello.sh that takes a single argument—a name of a file— and writes the string ‘Hello, world!’ into that file. • Consider the following solution: echo "Hello, world" > $1

28. Arbitrary Problems: Example • It gets Failed on test #2 -bash: $1: ambiguous redirect • Remember the statement: Write a script in bash called hello.sh that takes a single argument—a name of a file— and writes the string ‘Hello, world!’ into that file. • Consider the following solution: echo "Hello, world!" > $1

29. Arbitrary Problems: Example • It gets Accepted • Remember the statement: Write a script in bash called hello.sh that takes a single argument—a name of a file— and writes the string ‘Hello, world!’ into that file. • Consider the following solution: echo "Hello, world!" > "$1"

30. Plagiarism Detection • Students sometimes try to deceive teachers and submit somebody else’s solutions • Plagiarism detection module helps us to avoid cheating • iRunner compares a new solution to the previous solutions stored in the database and calculates the similarity score • 0.0: original solution, no similar sources found • 1.0: suspicious solution, verysimilar source found

31. Plagiarism Detection • Best-matched solution pairs are then manually reviewed

32. Plagiarism Detection • The system is able to detect: • moving functions up or down, • renaming variables, • adding unused code, • etc. • We use custom algorithm of plagiarism detection • It was developed bygraduating students

33. Quizzes • It is another way to check the student’s knowledge • A quiz consists of several questions

34. Quizzes: Example • Students are asked to choose the correct items from the given options…

35. Quizzes: Example • … or to type their own answers in

36. Quizzes: Autogeneration • We created a framework for making quiz questions in bulk • For example, given an adjacency matrix, the student has to count the number of connected components in the graph • We create plenty of such questions using a special generator • So each student receives his own matrix and has no possibility to cheat classFactory(IQuestionFactory): defmake_question(self, rng): q = SingleAnswerQuestion() # ... returnq

37. Electronic Queue • Classroom lessons are usually limited in time • A lot of students may want to discuss their problems with the teacher simultaneously • iRunner establishes the order the students talk to the teacher during the class

38. Electronic Queue • Queue interface is simple • Students press a special button to join a queue • A teacher then speaks to the studentspicking them up one by one from the queue • FIFO order is not enforced: if required, the queue may act like ‘priority’ queue

39. April Fools’ Day • On April 1st we celebrate the day of our faculty • As a joke, every year we swap colors on accepted and rejected solutions TLE 7 TLE 7 WA 42 WA 42 OK OK

40. Source Code • iRunner is open-source software • Sources are available at BitBucket: https://bitbucket.org/sobols/irunner2 • Unfortunately, now there is not enough documentation about setting up an instance of iRunner (because writing and supporting manuals requires a huge amount of time) • We are going to publish more instructions soon

41. The End • Thank you for attention! Sergei Sobol Developer of iRunner 2, assistant professor Belarusian State University Faculty of Applied Mathematics and Computer Science sobols@bsu.by sergei.sobol.91@yandex.by