Bluetooth Interoperability Testing at ITTC

1. Bluetooth Interoperability Testing at ITTC Dr. Joseph B. Evans Dr. Daniel Deavours Leon S. Searl Information & Telecommunication Technology Center

2. Overview • What is the Bluetooth SIG • Why Test Interoperability • Test Philosophy • Types of Bluetooth Applications Tested • Types of Devices Tested • Bluetooth Interoperability Tests • Test Result Types • General Results So Far • Future Work

3. Bluetooth Special Interest Group • The Bluetooth SIG is a trade association comprised of telecommunication, computing, and network industry companies that is driving the development of a low-cost short-range wireless specification for connection mobile products • Protocols – RF, Link Layer, etc. • Profiles (application APIs) – Serial Port, Object Push, Print, Audio, etc. • Promoters - 3COM, Agere, Ericsson, IBM, Intel, Microsoft, Motorola, Nokia, Toshiba, etc. • HQ in Overland Park, KS • Established in February, 1998 • First Specification release in July, 1999

4. Why Do Interoperability Testing? • Bluetooth SIG would like to obtain snapshot of current state of interoperability of commercially shipped Bluetooth products • Assess the feasibility of having an Interoperability Laboratory • Determine if additional Bluetooth specifications are required

5. Interoperability Test Philosophy • Test interoperability from user point of view • Did it work or fail? • How easy was it to use • User level debugging of failures • See if parameters are correctly set • Were user manual instructions followed? • User can’t tell which which device is responsible for Bluetooth related failure • Save all test data to SQL database for use by SIG

6. Bluetooth Device Categories • Handheld (HH) • Mobile Phone (MP) • Personal Computer/USB/PCMCIA (PC) • Printer (PR) • Headset (HS) • Hands-free (HF) • Human Interface Device (HID)

7. Interoperability Testing • Determine what Bluetooth operations are successful between pairs of Bluetooth capable devices • Business Card Transfer • Meeting Request Transfer • Memo Transfer • File Transfer • Send Fax • Dialup Networking • Data Synchronization • Print Object • Human Interface Device Input • Headset Audio

8. Bluetooth Application Test Types HH-HH Bizcard, Meeting Request, Memo HH-PR Bizcard, Meeting Request MP-MP Bizcard, Meeting Request MP-PR Bizcard, Meeting Request MP-HS Audio, Voice Dial MP-HF Audio, Voice Dial PC-PC Bizcard, Meeting Request, File Exchange PC-PR Bizcard, Meeting Request, File PC-HH Bizcard, Meeting Request, File Exchange, Sync PC-MP Bizcard, Meeting Request, Dial-up Networking, Fax HH-MP Bizcard, Meeting Request, Dial-up Networking

9. Bluetooth Application Test Types • All applications are tested using Public connections • An application from each device pair type test suite is chosen to test the following Bluetooth features for each pair of devices • Bonding • Secure Data

10. Bluetooth Test Organization • Device Pair Suite • There is a Test Suite for each Device Type Pairing (Example HH-PR) • A Test Suite is made up of Step Groups • Step Groups • A Step Group exercises a Bluetooth Application (Bizcard, etc.) between two Bluetooth devices or initializes the devices • A Step Group is composed of individual Steps • Step • Instruction to a Test Operator to perform an application operation on one or both devices • Non-Bluetooth Setup: 1 device, no Step Result Data • Bluetooth Setup: 1 device, may have Step Result Data • Bluetooth Interoperation: 2 device, Step Result Data for Both Devices

11. Example Test Organization • HH-MP Test Suite • Setup for other Step Groups … • Business Card Transfer Step Group • Create Bizcard on HH • Send Bizcard from HH to MP • Alter ‘name’ on Bizcard on MP • Send Bizcard from MP to HH • Remove all Bizcards from MP • Remove second Bizcard from HH • Meeting Request Step Group … • Bonded, Non-secure Step Group … • Public, Secure Step Group … • Dial-up Networking Step Group …

12. Bluetooth Test Step Group Data • Device 1 ID • Device 2 ID • Test Operator Name • Start Date and Time • Unique Execution ID • Support of Device 1 for application tested by the Step Group • Support of Device 2 for application tested by the Step Group

13. Bluetooth Test Step Measurements • Qualitative • Subjective ease of use (0-5 scale) • Ease of Use Comments for subjective scale < 3 • Step comment • Quantitative • Start date/time, Stop date/time • Step duration (seconds) • Action count (integer) • Connection failures/retries (integer) • Success/Failure (boolean) • Group execution ID • Unique step ID

14. Measurement Summaries • Summary data includes the following • Number of device pairs tested • Number of measurement steps • Number of measurement step failures • Percent of step failures • Average subjective scale • Average connection failure loss • Summaries are produced for the following groupings • Total database • Suites • Step groups • Individual devices

15. General Results • 20 devices tested • HF = 1 • HS = 4 • PR = 2 • MP = 2 • HH = 2 • PC = 7 of 10 • 44 device pairings tested (out of 75 possible) • 371 steps with measurements • most of the steps succeeded • 4.2 subjective usability (out of 5) of the steps that succeeded

16. General Results • Terms used by various manufactures are inconsistent with each other • Examples: Business Card, Contact, Address • Inter-operability failure rates correspond to complexity of the user interface • HS & HF = 14.3% failure rate • 1 or 2 button interface • PC = 23.9 failure rate • Graphical user interface • Software, PC user interfaces are still in infancy

17. Future Work • Refined interoperability measurements • Create well designed test application for test operator input (instead of current rapid prototype) • More specific interoperability data captured • Breakup subjective scale • Intuitive interface • User manual • Audio quality • Add new measures • Robustness (PC crashes) • Web-based test operator application • Multiple test operators to reduce measurement bias • Add standard deviation to test summaries in addition to mean