160 likes | 161 Views
http://flic.kr/p/ doFGYA. Ethics and Software Reliability. How we catch bugs: V & V activities. Verification answers: Did we build the system right ? Errors in design (wrt the requirements spec)? Errors in implementation (wrt the requirements spec)?
E N D
http://flic.kr/p/doFGYA Ethics and Software Reliability
How we catch bugs: V & V activities • Verification answers: Did we build the system right? • Errors in design (wrt the requirements spec)? • Errors in implementation (wrt the requirements spec)? • Validation answers: Did we build the right system? • Errors in the requirements spec?
How might we verify/validate a program? • Prove correct (w/ formal math)? • Nope. Too difficult, even for some small programs • Prove a model correct? • Nope. Too difficult to get small enough model that captures all relevant behavior • Run the program to see if it works? • Yup. Testing, that’s what we do
The Testing Problem:How to choose small set of test casesthat reveal all errors? • Fundamental research problem • Essentially unsolvable in general case
Does all “real” software contain bugs? Yup.No software is 100% reliable Let’s consider some examples of reliability failures
Financial software failures • Knight Capital loses $440 mil.: http://www.npr.org/2012/08/18/159082822/in-wall-street-2-0-computers-are-king • L.A. teachers’ pay: http://www.npr.org/templates/story/story.php?storyId=14869684 • Phone bill for $57,346.20 (Minnesota, 2001) • Beef producers lose $15-$20 million due to bad price reports
Data entry/retrieval failures • Bad background checks: http://www.nbc.com/news-sports/msnbc-video/2012/11/rossen-reports-background-check-mistakes-cost-jobs/ • Thousands of voters forbidden to vote because computer miscategorized them as felons (Florida, 2000) • Numerous false arrests due to errors in crime database (NCIC)
Software system failures • Mars Surveyor failure (2007): http://www.npr.org/templates/story/story.php?storyId=6817176 • Therac-25 (1985-1987): multiple radiation overdoses/deaths • Patriot missile system: 28 soldiers killed by SCUD; Patriot never even fires (1991) • Ariane 5 rocket: reuse error causes self-destruct, costing $500 mil.
Software unreliability leads tomoral and ethical questions • Morality: Rules of conduct • What people ought and ought not do in various situations • Different societies may have different morals • Ethics: Rational examination of people’s moral beliefs and behaviors • Reason from facts to commonly accepted values • Ethical theories: frameworks for reasoning • Examples: Subjective Relativism, Cultural Relativism, Divine Command Theory, Ethical Egoism, Kantaniasm, Utilitarianism…
Morality and Ethics From Quinn (2013), p53
Ethical Question Amazon error (UK, 2003): iPaq price listed as 7 EUR instead of 275 EUR Amazon refuses to honor sales—were they wrong? • Utilitarian view: Seller must always honor advertised price • Kantian view: Consumers recognized error and did not act in “good faith”
Ethical Question Should Therac-25 developers be held morally responsible for deaths? • Moral responsibility requires: • Causal condition: actions caused harm • Mental condition: actions intended • Unintended harm counts if due to carelessness, recklessness, or negligence • Quinn concludes “yes” because machine lacked devices to catch/prevent overdose and failed to thoroughly verify overdose wasn’t possible
If perfect software is impossible, what sort of warranties should companies provide? • Shrinkwrap warranties: • 90-day replacement (haha)/money back if softw. fails • Few guarantees (you’ll be able to install it) • Maker has no liability (for harm to your business) • “Don’t blame us if it’s full of bugs” • Are warranties enforceable? • Unfair warranties could violate laws • Court rulings have gone both ways • Should manufacturers be held liable for defects? • Should consumers have to pay for bug fixes?
Hypothetical Question • Medick Inc. developing fancy new mobile software for nurses and other hospital workers • Medick told customers it would be ready next week • Software still contains lots of bugs • Competitor will release new product in a few weeks • If competitor gets to market first, Medick will likely go under What course of action should Medick take? • Perspectives: software engineers, sales force, managers, venture capitalists, and nurses
Summary • All “software” contains bugs • Knowing that perfection is impossible, how to set consumer expectations and assign blame when things go wrong? • Leads to moral and ethical questions http://flic.kr/p/YSY3X
Question • Anyone been a beta tester? • What did it teach you about computer reliability?