1 / 60

How We Test @ Microsoft

How We Test @ Microsoft. How Microsoft Tests Software Today and Where We are Headed Next. Ken Johnston, Director Test Excellence Microsoft Engineering Excellence PSQT North 2006. Making the hard call. Sev 1 Bug Crashes setup and leaves the machine in an unknown state

sutton
Download Presentation

How We Test @ Microsoft

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How We Test @ Microsoft How Microsoft Tests Software Today and Where We are Headed Next Ken Johnston, Director Test Excellence Microsoft Engineering Excellence PSQT North 2006

  2. Making the hard call • Sev 1 Bug • Crashes setup and leaves the machine in an unknown state • On upgrade install it corrupts customer data • Reported by customer • Reported by Dev, Test and PM • Last bug left! • What’s a young test lead to do? Sasquatch

  3. Global Company • Multinational workforce Worldwide customers and partners

  4. Products in Different Categories 260 Products Released in last 12 Months • There is no 1 • “Microsoft Way”

  5. There is No One Way to Ship Many different ship cycles Daily/Weekly Monthly Annually Every 2-3 years More than one product has followed this path • Multiple Slip Schedule

  6. Many Development Methods Used Spiral Iterative/Waterfall Agile Techniques Test Driven Development SCRUM TSP/PSP Team Software Plan & Personal Software Plan

  7. What to do with 50 min left? Look at the people landscape Cover as many techniques and tools as I can Try to squeeze in a little on the future of testing At the end I will point you to additional resources With any luck we’ll have some fun along the way

  8. People, Process and Tools Process People Testing @ Microsoft Tools

  9. How many Engineers are There? More than 31,000 Engineers World Wide More than 7,500 Test Engineers What is the dev to test ratio? Counting full time and temp/vendor Test Engineers Microsoft has more than 1 Test Engineer for every Developer Hired more than 4,000 Engineers FY06 Hired more than 1,000 SDETs in FY06 Too many open positions More than 5,000 open Engineering positions More than 1,000open Test Engineer positions

  10. SDET Profile • Mostly Computer Science Graduates & Industry Hires • Also • Electrical Engineering • Math • Physics • Other Engineering • We look for • Strong Programming Skills • Problem Decomposition Skills • Drive for Results • Passion for Quality • Experience in Test is not the highest order bit

  11. Test Techniques Taught in New Tester • Integration and system level testing • Exploratory testing • Application compatibility testing • Accessibility testing • Security testing • Globalization testing •  Debugging •  Inspections and Review •  Model Based Testing • Functional testing techniques • Boundary value analysis • Equivalence class partitioning • Combinatorial analysis • Structural testing techniques and code coverage analysis • Decision/Branch coverage • Condition coverage • Basis path coverage • Code coverage analysis

  12. The Feature Team Triad Test Simple Descriptions • Program Managers (PMs) research customer needs, competitors and develop the requirements • Developers (SDEs)write the product code to the functional requirements • Test Engineers (SDETs) ensures the product meets the functional requirements Development Program Management

  13. The Quartet for Web Services Test • Operations is critical partner in migration to web services • ~4,500 IT/Ops Engineers Development Program Management IT/Operations

  14. Many More Engineering Disciplines Test Usability IPE Development Program Management IT/Operations Design Creative Content Research Engineering Management

  15. Focus on a Few Processes and Techniques Process People Testing @ Microsoft Tools

  16. Bugs Drive Work

  17. Not all Bugs are Code Defects

  18. The Daily Build The Heartbeat of Microsoft R&D

  19. Build Reporting on Every Build

  20. Too Many Tools Process People Testing @ Microsoft Tools

  21. Microsoft has a lot of Tools

  22. Active Customer Feedback

  23. Play Balance - 01a_look_start 01a_move_start 01a_shield_start 01a_tram_start 01b_1st_start 01b_1st_wv4 01b_1st_wv5_end 01b_atr2_start 01b_atr2_mid 01b_bay1_start 01b_bay1_mid 01b_bay2_start 01b_bay2_mid 01b_arm_start 01b_atr1_start 01b_trm1_start 01b_dck_start 01b_elv_start 01b_lvr_start 01b_gun_start e1_rooftop_defense e2_hunter_intro e3_highway_underpass e4_sniper_alley e5_neighborhood e6_hotel_entrance e7_hotel_hall e8_hotel_exit e9_beach_infantry e10_beach_fort e11_tunnel_entrance e12_tunnel_blockades e13_tunnel_convoy e14_bridge_ascent e15_bridge_summit e16_bridge_descent e17_flooded_tunnel e18_park enc_ledge_lift enc_ledge_lower enc_high_hall_a enc_high_hall_b enc_room_a enc_jail_ini enc_jail_cell_a enc_jail_cell_b enc_jail_down_lift enc_corridors_b enc_tower_a_exterior enc_gardens_a enc_mid_tower e4_generator_room e2_scarab_cove e19_park e20_road_skirmish e21_hospital_seige e22_scarab_intro e23_scarab_boarding 04a_1_landing_zone 04a_2_recycling_center 04a_3_hangar 04a_4_first_halls 04a_5_underhangar 04a_6_second_halls 04a_7_bottling_plant 04a_8_bottling_back 04a_9_dogfight 04b_1_entry 04b_2_disposal_chamber 04b_3_silo_start 04b_4_silo_mid 04b_5_silo_end 04b_6_hall_to_lab 04b_7_flood_lab_start 04b_8_flood_lab_mid_01 04b_9_flood_lab_mid_02 04b_10_flood_lab_end 04b_11_wraparound 04b_12_control_room_01 04b_13_control_room_02 04b_14_cable_room 04b_15_control_return 04b_16_power_core 04b_17_dogfight 04b_18_bottling_return_01 04b_19_bottling_return_02 04b_20_halls_to_under_01 04b_21_halls_to_under_02 04b_22_halls_to_under_03 04b_23_underhangar_return 04b_24_boss_fight_01 04b_25_boss_fight_02 04b_26_boss_fight_03 enc_maus_grand enc_maus_bridge enc_mausoleum enc_maus_midpoint e1_stealthy_insertion e2_cylinder_room e3_ramp_room0 e4_step_room e5_ramp_room1 e6_ghost_ledge e7_kidney0 e8_kidney1 e9_ski_run 04b_27_boss_fight_04 04b_28_boss_fight_05 04b_29_boss_fight_06 04b_30_boss_fight_07 05a_1_landing_zone 05a_2_overlook 05a_3_bridge 05a_4_winding_path 05a_5_temple_entrance 05a_6_tunnel 05a_7_old_temple 05a_8_old_temple_courtyard 05a_9_grotto 05a_10_tower_approach 05a_11_tower1_interior 05b_1_tower1 05b_2_bridges 05b_3_tower2 05b_4_central_platform 05b_5_gondola_01 05b_6_tower3 05b_7_tunnel1 05b_8_sunken_chamber 05b_9_tunnel2 05b_10_island_interior 05b_11_island_gully 05b_12_island_exterior 05b_13_gondola_02 05b_14_temple_exterior 05b_15_temple_halls 05b_16_temple_center 05b_17_temple_retreat enc_insertion enc_hall_a enc_cond_a_sec_a enc_cond_a_sec_b enc_hall_b enc_plug_launch enc_plug_across enc_plug_landing e5_bridge e6_jail e7_scarab_ledge e8_scarab_cove e9_scarab_canyon e11_scarab_boardings e12_antechamber e13_control_room enc_gardens_b enc_tower_b_exterior e10_brute_camp phantom_ledge e3_ramp_room enc_plug_landing_bk enc_plug_landing_lower enc_hall_c_sec_a enc_hall_c_sec_b enc_hall_c_sec_c enc_ledge_a_sec_a enc_ledge_a_sec_b enc_cond_b_sec_a enc_cond_b_sec_b enc_cond_b_sec_c enc_ledge_b enc_ledge_c enc_qz_initial_camp enc_covenant_defense enc_cov_charge enc_vehicle_int enc_vehicle_int_bk enc_qz_ext_a_dam enc_qz_ext_a enc_ext_a_fact_ent enc_crashed_factory_a enc_crashed_factory_ext enc_crashed_factory_b enc_ext_b_fact_exit enc_qz_ext_b enc_qz_ext_b_bk enc_qz_ext_b_exit enc_e21 enc_e22 enc_e23 enc_e24 enc_e25 enc_council_wave_1 enc_council_pedestal enc_council_wave_2 enc_jail_cell_a enc_jail_cell_b enc_council_ini enc_grand_a enc_ledge_ini turbolift_base plasma_chandelier riptide_start riptide_ambush space_cave1 boobag_garden space_cave2 hallway_attack inner_sanctum council_chamber last_room e1_wraith_canyon

  24. Halo 2 Playtest Data “unknown” deaths ≈ 8+ hours in…

  25. Software Quality Metrics (SQM) Software Quality Metrics / Software Quantitative Mechanics Client-side tool which measures statistical usage data Server-side collection, reporting and management tool

  26. SQM in Office 2003 and Office 12 O11 Beta2 O11 Beta 2TR O11 RTM Early O11 RTM All O11 SP1 All M1 Dogfood M2 Dogfood 900 800 700 600 500 Mean Time To Crash (hours) 400 300 200 100 0 OutlookApplication One Note Word

  27. Windows Error Reporting (a.k.a. Watson) Real-world data about user pain Millions of hits every day Walt Mossberg 9/30/99: “Dear Diary: I’m tired of the way Windows crashes and freezes…”

  28. Continuous Feedback Process

  29. Tests and Test Tools Bug Management Test Case Management Automation Focus Static Code Analysis Dynamic Analysis Unit Tests Code Coverage Performance/Load Web Load Testing Model Based Testing Threat Modeling Fuzz Testing

  30. For Bugs We Use Product Studio

  31. Test Case Management Test Case Management Manual and Automated in one system

  32. Microsoft has an Automation Focus Windows Vista + Office 12 > 200 MLOC Most products target ~70% code coverage before ship Some as high as 80% on unit tests Microsoft has >110,000 Test Machines FY07 lab budgets 100 million+ Windows Vista has over 6.7 Million automated test jobs Office 12 has over 1 Million automated test cases Why Automate so Much? Many products must be supported for 10 years Compatibility regression passes Daily build process increases reuse

  33. Every SDET Learns to useDebuggers

  34. Static Code Analysis Static Code Analyzer

  35. Dynamic Analysis – AppVerifier and TruScan Memory Leak Checker Unutilized Variable Checker Race Condition Checker Handle leak checker Buffer Overrun checker Dangling Pointer checker

  36. Unit Testing Integrated Unit Testing

  37. Code Coverage Code Coverage Unit, BVT, Suite, All

  38. Load Generation Alerts and warnings on Systems Under Test Performance measures of test and Systems Under Test

  39. Code Performance Timeline of memory consumption Suspect functions, drillable to code

  40. Web Testing Data substitution Performance breakdown Content validation http request & response View of content as rendered

  41. Model Based Testing • Significant growth in use of Model Based Testing • Courses • Lectures • E-Learning • Active community • MDE – Model Design Environment • Spec Explorer • http://research.microsoft.com/SpecExplorer/ • Built on Spec Sharp (extension to C#) • http://research.microsoft.com/specsharp

  42. Threat Modeling & Fuzz Testing Threat Modeling - analyze the design for vulnerability Model data flows (How to Write Secure Code) S - Spoofing Identity T - Tampering with Data R - Repudiation I - Information Disclosure D - Denial of Service E - Elevation of Privilege Fuzz testing is a method of finding software security holes by feeding purposely invalid and ill-formed data as input to program interfaces. Inputs include: Files Network ports APIs

  43. A Few Other QA Processes

  44. Global Software Company Ship Products World Wide Localize products in >50 Languages Focus of IPE (formerly Localization) discipline Globalization/Global Sufficiency Localization Marketization Geopolitical

  45. LocStudio – the Tool for IPE

  46. Who’s Behind the One-way Mirror? What do we get from Usability? provides perspective onto the total situation reveals nuances of behavior, motivation, and emotion enables comparison and contrasting of attitudes, actions, and feelings uncovers latent needs users can’t articulate fosters user inspired empathy, advocacy, and creativity

  47. Usability Labs Without Walls Virtual Research Lab

  48. Where are We Headed Next?

  49. Metrics then, now, and… Data Sources 2006 Product Studio Test Case P/F Large Checkin Suites Automated per build BVTs Automated test case results Automatic bug filing Stress, Scale, Perf, Icecap Customer feedback systems Watson, SQM PSS Granular call coding Customer impact and cost Magellan code coverage, code churn, code complexity Static Analysis PreFix, PreFast, FXCop Dynamic Analysis App Verifier, Driver Verifier Product Size 1995 • Office 95 • MLOC – ~4M • TC – Unk • Bugs – 268K • Windows 95 • MLOC – ~7M • TC – Unk • Bugs – >5*25K Product Size 2006 • Office 12 • Blocks – 19.1M • TC – ~1M • Vista • Blocks – 34.7M • WTT Jobs – 6.7M Data Sources 1995 Fast forward 11 years RAID Test Case Simple DB Word Docs Excel Files Stress, Scale, Perf PSS

  50. Metrics next… Move from Quality Control to Quality Assurance QC is fixing the bugs we find QA is setting the goals for key indicators that we know will ensure high quality • Data warehouse projects in many product teams • PHI, Team Stats, MSN QA Dashboard, MSN QOS, Exchange, Office, • Cross-correlation of data across large numbers of products • Auto-analysis of data

More Related