1 / 25

Dynamic Benchmarking in Software Development

Learn how the dynamic benchmarking game model is used in software development to assess relative performance, rank software, motivate students, and create a realistic development environment.

mattiecruz
Download Presentation

Dynamic Benchmarking in Software Development

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DynamicBenchmarking Alex Dubreuil Northeastern University dubreuil.a@husky.neu.edu acdubre@gmail.com Software development though competition

  2. This slide intentionally left blank

  3. Contents • Dynamic Benchmarking Introduction • Uses of the Benchmarking Game model • Software Development (CS 4500) • A Lesson I’ve learned Caution: Slide layout may cause drowsiness.

  4. Benchmarking • Assesses relative performance • Typically by running standardized tests • Produces scores which are then compared • SATs • Other options exist • Allowing software to compete directly • Chess game

  5. The Traditional Approach Developer A Software A Static Benchmark Score A Developer B Software B Score B Developer C Software C Score C Parameterized by the domain.

  6. The Dynamic Approach Software A Team A Agent Artificial World (Game) Benchmark A Agent Ranking Software B Team B Agent Benchmark B Agent Software C Team C Benchmark C Parameterized by the domain.

  7. An Artificial WorldAgent’s View Agent Beliefs, Challenges, Problems, Solutions Opponents’ communication, Feedback • Problems: Benchmark output • Solutions: Software output • Beliefs/Challenges: statements about algorithms Administrator Results

  8. Problems & Solutions • Problem communication: • Define an instance of a problem in the domain • Solution communication: • Respond to an opponent’s problem • Administrator has a metric for determining how good a solution is • This metric is well defined and known by all

  9. Beliefs & Challenges • General statements about algorithms • Belief: • Defines a subset of the problems in the domain • Makes a statement about the problems in that subset • Challenge: • A response to a belief of an opponent

  10. Administrator • Opponents’ communication • Filter all communication through the Administrator for security • Filter information when necessary • Feedback: • Inform agents of rule violations • Inform agents of status changes

  11. Administrator • Results • Track state changes through the game • Produce the agent ranking from the end game state

  12. What’s next • Dynamic Benchmarking Introduction • Uses of the Benchmarking Game model • Software Development (CS 4500) • A Lesson I’ve learned If you can read this, you don’t need glasses.

  13. Overhead • Requires mature Administrator, communication system for accurate results • Reuse between domains is possible • Requires new translation for each problem domain

  14. Software Development • Ranks software without a mature benchmark • Dynamic approach excels when a well-defined benchmark does not exist • Creates data to build better benchmarks • Because Agents, not Software, are ranked • Forces developers to consider both their solutions and the problem domain

  15. Education • Motivates students • Mature Administrator/Agent not required • Creates interesting student interaction • Creates a realistic software development environment

  16. What’s next • Dynamic Benchmarking Introduction • Uses of the Benchmarking Game model • Software Development (CS 4500) • A Lesson I’ve learned Yeah, I got nothing.

  17. Specker Challenge Game • The SCG is the basis for Professor Karl Lieberherr’s Software Development class • Uses an arity 3 boolean constraint satisfaction problem (CSP) as our domain • Teams of 2~3 produce the components of an Agent

  18. (Some of the) Skills Involved • Using outsourced tools • DemeterF (developed by Bryan Chadwick) • Component Market • Dealing with users • Underspecified requirements • Source control • Constraint Satisfaction algorithms • Data mining

  19. Added bonus Domain Knowledge Experts Code So what? Programmers Requirements Limitations How-to Non-technical Requirements Gibberish Salespeople Customers Users

  20. It’s a busy class • Traditional grading would not work • The competition keeps students motivated

  21. What’s next • Dynamic Benchmarking Introduction • Uses of the Benchmarking Game model • Software Development (CS 4500) • A Lesson I’ve learned

  22. Administrator Security • Never accept extra input • Transaction: Challenge: ID, Type, Price • vs. • Transaction: Challenge: ID • Check all necessary input • Transaction: Deliver Problem: ID, Problem • Check: Does the Problem match the Type?

  23. General Lesson • Never trust user input • Sanitize data • Protect against buffer overflows

  24. More General Lesson • It’s good to see things before they can do you or others harm • Users you can yell at • Security flaws that don’t cost money • Underspecified requirements

  25. Thank you! Alex Dubreuil Northeastern University dubreuil.a@husky.neu.edu acdubre@gmail.com

More Related