軟體測試簡介 All about quality !!

1. 軟體測試簡介All about quality !!

2. Testing • Testing is not important in school homework • Testing is not important in research work to produce experimental statistics for publishing paper • Testing is not important in a research prototype • Testing, however, is very important for a commercial products.

3. Why testing? • Let’s look at some well-known software defects • 台灣高鐵售票系統：系統上線之後當機連連，系統無法應付突然湧現的購票人潮 （no stress testing) • 台灣彩卷系統：類似的問題 • 2000-2005- 巴拿馬國家癌症中心─5個病人接受過量的迦瑪射線照射死亡。15人引發了嚴重的併發症 • 2003 ─軟體造成美國東北部及加拿大停電。5000萬人受影響，3人喪生 • 2000 美國海軍飛機墜落，4人喪生 （控制軟體問題） • 1997 韓國空難，225人喪生 （雷達控制軟體問題） • 1995 美國航空在哥倫比亞撞山159人喪生（導航軟體問題）

4. Statistics • 2002- 美國國家標準局報告─軟體品質每年造成595億美元的損失 （0.6％ GDP） • 國內又如何？ • 台北縣政府校務行政系統案例 • 台師大林口校區學務系統案例

5. Quality Control and Quality Assurance • Software is a product. • Good quality product requires 品質控管quality control (QC) and quality assurance (QA) 品質確保

6. Manufacturing in other discipline • 在製造業的世界裡面 • 產品品質(quality)是很重要的一個因素 • 當你投入非常多成本與原物料時你永遠希望你生產出來100個產品有100個都是可以賣的 • QC (Quality Control) 品質控管 • 製造過程從設計到生產可能有數十道到幾百道程序 • 每一道程序都可能影響到後面的品質 • 如何透過製造流程的改善(process improvement)來提高良率，一直都是製造工業的重要課題

7. How to improve quality? • 在製造過程中如果能夠能夠使用機器，最好的改善品質的方式就是自動化。機器不容易犯錯而且可以不斷地重複單調無聊的工作。機器會出錯的時候通常是由於製造機器的磨損，必須重新校正。 • Ex. 日本的步進馬達精確度超高的秘密。 • Question • 當某部分的工作不能由機器來做的時候，製造過程如何做到品質控管？

8. Assembly line in manufacturing

9. 生產線 • 每個生產線員工只做一件夠簡單到不容易出錯的工作 • 生產線員工不需要夠聰明 • 生產線將複雜的產品製造切割成許多小步驟可以在短時間以及最少的技術內可以完成 • 發現產品的問題並不難.

10. Question? • Is coding a design process or a manufacturing process • Is Code a design or a product? • Again, the analog breaks More questions • If coding is not a design process, we should be able to hire many 生產線女工 to control our quality • If coding is a design process, bad news, errors are inevitable because programmers are always doing complicated design jobs, not simple and easy job.

11. Facts • Now, you should know why software has so many bugs or notorious for being flawed

12. The QC story 產品缺陷率 software 其他工程製造 QC 改進時程

13. A Joke 有天晚上我參加一個老同學的喜宴， 同桌的人有律師、會計師， 只有我一個是電腦業界的人。 為了引起話題，我說我羨慕其他行業的人，像是律師、會計師等，在學校學的那一套終生都受用，每年就算有新法條也不會改太多。 不像我們電腦界的人，每天都在K新的技術手冊，新產品新技術隨時大翻新，改朝換代的速度讓人措手不及，所以學電腦的人看起來都比較蒼老，因為太辛苦了。 我話剛說完，一位會計師馬上回我一句：「你錯了，其實我們最羨慕你們電腦界的人，因為沒有任何行業的消費者像你們電腦界的消費者一樣好欺負。」 頓時我成了眾矢之的， 這些電腦使用者的抱怨全部集中到我身上來。 其中一位仁兄還舉了個有趣的例子， 他說： 「如果傢俱業和電腦業一樣，世界會變成什麼樣子？」以下是他講的故事： 如果傢俱業跟電腦業一樣，比如說我到傢俱店買了一張桌子，搬回家往地板上一放，啪啦一聲桌子就塌了。這時候我不會生氣、不會罵人，我會先自己檢查一下出了什麼錯。

14. 於是我恍然大悟原來是自己的錯， 我就再去買一張「正式版」的桌子。 回家一擺還是啪啦又塌了！ 修了半天還是有問題， 再請傢俱行的技術人員來做仔細檢查， 最後終於發現問題的所在── 「我家的地板和桌子不相容」， 又是我自己的錯， 於是我得趕快幫家裡的地板升級......。 等一切都忙完了， 桌子可以使用了， 我趴在桌上寫字， 心裡充滿了成就感， 我很得意地跟網友分享我修理桌子的經驗， 並暗自慶幸自己在科技的潮流上沒有落伍......。 我會先檢討自己， 是不是我做錯了什麼， 或是我對桌子的使用不夠熟悉。 於是我會去買書來看 (書名可能是《快快樂樂學修桌子》、 《21天學會修桌子》、《修桌子技巧與實例》、 或是《修桌子的聖經》)。 要是書看得不太懂， 我會再花錢去報名上修桌子的課程。 學完之後還是修不好， 我會請其他比較懂得修桌子的朋友來幫忙。 最後沒有辦法， 終於我打電話給原先的傢俱行， (可能還要購買《技術支援方案》)， 結果他們跟我說： 「唉呀！你買到的是搶鮮版啦！ 本來就應該有問題的」。

15. Software Quality • 如同前面所說的，由於軟體產業的生產工具只有人，，人注定會犯錯，很多人還會犯很多的錯 • 不要相信 programmers • This is why testing is so important in matured software industry.

16. 一座橋蓋好了，需要大量的測試嗎？

17. How do you test a DVD player

18. How do you test a software

19. The complexity of software testing • 一般而言軟體有複雜的介面 (including user interface, network interface, file interfaces, etc. ） • 一般而言軟體測試必須考慮更多的情境 • Correct execution paths（正確的路徑） • Incorrect execution paths （不正確的路徑） • 有許多應用軟體的測試其實是一門困難的學問理論，技術，實做，程式能力的要求都不低

20. SDLC (Software Development Life Cycle) • 需求分析 Requirement analysis (marketing research) • 軟體規格寫作 Software function/performance specification • 分析與設計 Analysis and Design (QA 在這時候就要參與專案） • 分析，選定PL，選定 platform, database, network protocols. • 定義軟體功能模組，模組之間的關連，合作，介面 • 程式實做與單元測試 Coding and unit testing (by programmers) • 整合測試 Integration testing (by programmers/QAs) • Alpha testing (by QAs) • Most software functions and features are basically completed • All functions are tested, no functions will be added beyond this point • Serious flaws (high severity) are solved and addressed (show stopper) • Beta testing (by Beta users) • sub-serious bugs are all fixed • Test plan has been completely executed • Bug discovering rate is lower than bug fixing rate • 軟體釋出 Release • Bug discovering rate is lower than bug fixing rate for a long period of time. • The version after fixing bugs has been regressively tested (regression test) • Quality is formally proved by QA team • All documents are ready

21. Common software defects • 70% occurs in design and hard to correct • Software specifications are not precise • Software is complicated • Coding errors (20%) • Function changes which affects other components • 3rd party software is flawed

22.  Early detection of errors Error found Mar 13 Error found Feb 13 Error occurs Jan 13 Work (person-days) 10 20 Time Work proceeds at (say) 10 person- days per month 10 person-days of work has been done assuming the error is not there. Now this must be redone. If error found this late, 20person-days must be redone

23. What to test for a QA (要測什麼？） • 完整性 (completeness) • 軟體是否具備軟體規格書，設計文件中所描述的功能與性能 • 正確性 (correctness) • 可靠性 (reliability) • 相容性 (compatibility) • 效率（efficiency) • 可使用性 (usability) • 可攜性 (portability) • 可變比例性 (scalability) • 易測性 (testability)

24. 迷思 • 軟體有問題是測試人員的錯 • 軟體測試只是一種有效的提高軟體品質手段，但無法百分之百解決軟體品質的問題 • 軟體測試技術要求不高，比程式設計容易 • 兩種人無法類比，程式設計能力好的人，可能可以更勝任軟體測試。 • 自動化測試常需要程式設計高手

25. 就業市場概觀 • 美國的軟體產業現況 • Big software company has their testing team • Small software companies which do not have resources to test can outsource to 3rd party testing company • 台灣的軟體產業現況 • Big software company like 趨勢科技Trend Micro has QA team • Many software companies use programmers as testers – a sad fact!

26. Company Organization Developers Marketings QAs (testing teams)

27. Management Hierarchy Director of development Director of QA Developer manager Developer manager Test manager Test automation Team manager Test facilities manager programmers Test automation programmers Test database programmers programmers testers

28. Important reasons to make QA team and developer team equal and separated • 只有極少數處女座的 programmers 才會有品質（完美主義）的概念 • 叫 programmer 寫完程式，自己找bugs 就如同叫法官判完案之後，承認自己的判案是錯誤的 • 如果他真的找出很多bug ，他決不敢聲張與邀功，因為這代表自己程式寫太爛 • 找的bug越多，表示待完成的工作越多，沒有人會自己找碴，讓自己無法喘息 • 上述「人」的因素使得 • QA 必須在軟體品質上與 programmer 站在對立面 • QA 的升遷管道必須是抓到越多 bug，功勞越高，QA才會努力找碴 • QA 不能歸 developer team 管轄，這就像是司法不能在行政部門底下是一樣的

29. Marketing 恐怖平衡 QA (testers) Developer

30. 軟體測試V模型 回歸測試 軟體功能需求 驗收測試 以此作為驗收測試依據 軟體規格說明書 系統測試 以此作為系統測試依據 以此作為整合測試依據 分析與設計文件 整合測試 軟體模組設計書 單元測試 程式實做

31. Smoke Tests • A smoke test is a subset of the test cases that is typically representative of the overall test plan. • Smoke tests are good for verifying proper deployment or other non invasive changes. • They are also useful for verifying a build is ready to send to test. • Smoke tests are not substitute for actual functional testing. http://www.stellman-greene.com

32. Load testing • To exercise the system to the maximum load that is specified in specs. • The goal is to test whether the system meet the requirement

34. Stress testing • The goal of stress testing is to exercise a system beyond the load in specs to see what it can happen. • require considerable cost and efforts. • often require you to implement a system to test the system. • Load testing is necessary but stress testing is optional. • E.g., a online game server may limit the users to prevent system from crash

35. Let’s back to the reality • Most vendors do not possess a separate QA team • They can fake test plan and report • So? What can we do with the vendor? • Ask for a bug report system (issue tracking system) • Examine the test cases and sampling • Smoke test • Extended test • Critical test cases • Watch the bug report history

36. If you are hired to build a dog house, what ability you should Have?

37. Different Scale of Software Development If you are hired to build a Taipei 101 • What skills and talent do you need to have?

38. Usability Test

39. ACommon story • 各機關不斷的引入各式各樣的軟體系統 • 系統可能內部開發, 可能外包，以外包為例 • 通常由廠商撰寫規格書 • 最低價得標通常是夢靨的開始 • 當系統上線時， 資訊人員通常首當其衝的面對各種狀況 • 廠商將使用人員當測試人員 • 使用人員對於使用上的不方便諉因於自己對軟體的不了解，而不敢提出建言 • 發包單位有結案的壓力 • 測試不完全，廠商與使用人員一路使用一路修正錯誤。幸運的話,共體時艱，不幸運的話彼此怨懟，進入最可怕的驗收夢靨 • 主要原因之一:沒有 usability test

40. Some facts • Most programmers do not possess the knowledge in UI design • Programmers often think UI is not a difficult part, which is wrong in practice, especially in commercial products • Change UI design (e.g., making UI more friendly) often means writing a lot more code, which is not expected. • Programming tasks are often spec-oriented, not user oriented.

42. The failed story

43. Why Microsoft lost its markets? • UI • UI • UI • UI • …..

44. Goals of usability tests • Performance -- How much time, and how many steps, are required for people to complete basic tasks? (For example, find something to buy, create a new account, and order the item.) • Accuracy -- How many mistakes did people make? (And were they fatal or recoverable with the right information?) • Recall -- How much does the person remember afterwards or after periods of non-use? • Emotional response-- How does the person feel about the tasks completed? Is the person confident, stressed? Would the user recommend this system to a friend?

45. Some usability test method • Think aloud protocols • Users are asked to say whatever they are looking at, thinking, doing, and feeling, as they go about their task. • Observers at such a test are asked to objectively take notes of everything that users say, without attempting to interpret their actions and words. • Test sessions are often audio and video taped so that developers can go back and refer to what participants did, and how they reacted. The purpose of this method is to make explicit what is implicitly present in subjects who are able to perform a specific task

46. Eye tracking

47. Eye tracking pattern

48. Investigation room

49. Observations • Most 行政資訊系統 Ihave used are difficult to use, learn, and remember • Poor exception handling • Function-oriented is a common

50. Source Code Quality