SixthSense: RFID-based Enterprise Intelligence

1. SixthSense:RFID-based Enterprise Intelligence • Lenin Ravindranath, Venkat Padmanabhan • (MSR India) • Piyush Agrawal (IIT Kanpur)

2. RFID • Radio Frequency Identification • Components • RFID Reader with Antennas • Tags (Active and Passive) • Electromagnetic waves induce current • Tag responds • Globally unique ID • Data • UHF (865-956 MHz) • Range – up to 7m • Applications • Tracking, Inventory, Supply Chain, Authentication, … • Novel Research Applications

3. Motivating Scenario Lenin missed an object in the conference room – 2nd floor Scientia

4. SixthSense Goal • Making people, objects and workspaces, the first class citizens of the enterprise computing • Components • Use RFID to capture the rich interaction between people and their surroundings • Combine with other enterprise systems/sensors to make automated inferences • Enable Useful Services

5. Setting Presence Camera People and objects tagged RFID Antennas Calendar

6. Assumptions • Widespread coverage of RFID readers in the workspace • Users are free to pick up new tags and affix them to objects • Can put multiple tags on an object • No dependence on cataloging • Cataloging is an overhead • TagID  Entity • Antenna  Workspace • Error Prone • Tags are fragile – may have to be replaced • Readers/Antenna could be moved Lenin missed an object in the conference room – 2nd floor Scientia 13548234 – Ant 1 15574523 – Ant 1 13548234 – Ant 6 15574523 – Ant 1 • Start with an undifferentiated mass of tags and infer everything

7. Architecture Presence Camera SixthSense People and objects tagged RFID Antennas Calendar

8. SixthSense • Automated Inference • Platform • Programming Model • Applications SixthSense

9. Inference Engine • Person-Object Differentiation • Object Ownership Inference • Zone Identification • Person Identification • Person-Object Interaction

10. Person-Object Differentiation Zone 2 Zone 3 Zone 1 • People can move on their own • Objects move only when carried by a person Co-movement based heuristic • Relative Movement (RM)

11. Object Ownership Inference • Co-Presence • Calculate the amount of time the object is concurrently present in the same zone as a person • Owner is the person with which the object is co-present the most and greater than a threshold

12. Person Identification time Workspace Entrance Event Log-in event xyz abc t1 1 1 1 2 xyz 1 t2 1 Coincidence count

13. Object Interaction (only in zones of interest) • Intra zone • Identify interaction in zones of interest • A person lifted an object • A person turned the orientation an object • Signal Strength of tag varies • Change in distance • Change in orientation • Contact • Monitor variation in RSSI

14. Object Interaction • Sample the RSSI of each object tag every 200ms • Sliding 4-second wide window • Difference between the 10th percentile and 90th percentile of the RSSI • Object is said to be interacted - If the difference > threshold • Minimizing spurious detections • Use multiple antennas

15. Object Interaction Interacted Antenna 2 Antenna 1 Interacted

16. Ensuring Privacy • Enterprise will deploy and manage the system • Expose appropriate set of information • Trust - Analogous to the enterprise e-mail system • Defend against rogue readers • Relabeling approach [A.Juels, 2006] • EPC code rewritten at random times • SixthSense will be aware of the mapping between the old and new tag IDs

17. Implementation Experimental Setup (Real-time feed) Simulator (Trace Generation)

18. Experimental Setup

19. Results • Inter-zone movement detection • Object Interaction • Testbed deployment • To make correct inferences • Average inter-zone movements needed – 4 • Average log-ins required - 3

20. Simulation • Probabilistic model to generate artificial traces • Simulated • Inter-zone movement (walk) • People carrying multiple objects • Log-in events • Untagged people

21. Results • Person-object differentiation • Person Identification • Varying average walk length • Effects of untagged people

22. Person-Object differentiation and ownership • 20 people, 100 tags, probability of walk – 0.1, walk length - 5

23. Person Identification • 10% of users entering workspace simultaneously

24. Programming Model • Callbacks • InterZoneMovementEvent(tagID, startZone, endZone, Time) • ObjectInteractedEvent(tadID, Zone, Time) • Lookups • GetTagList() • GetPersonTags() • GetOwnedObjects(tagID) • GetTagType(tagID) • GetTagOwner(tagID) • GetPersonTagIdentity(tagID) • GetZoneType(zone) • GetTagsInZone(zone) • GetTagWorkspaceZone(tagID) • GetCurrentTagZone(tagID) • GetCalendarEntry(ID, Time)

25. Example • Misplaced Object Alert personTags = GetPersonTags() For each ownerTagID in personTags ObjTags = GetOwnedObject(ownerTagID) OwnerZone = GetCurrentTagZone(ownerTagId) OwnerWorkspace = GetTagWorkspaceZone(ownerTagId) For each obj in ObjTags objZone = GetCurrentTagZone(ownerTagId) if (objZone != OwnerZone && objZone != OwnerWorkspace) Raise Alert

26. Applications • Annotated video • Semi-automated image catalog • Misplaced object alert • Automatic conference room booking

27. Annotating video with physical events • Events • Inter-zone movements • Object Interaction • Tag video feed with events • Person X interacted an object Y • Rich video database • Support rich queries • Give me all videos where Person A interacted with Object B • Application: Integrate with enterprise security camera system

28. Semi-automated Image catalog • TagIDs are not user friendly • Catalog tagID with its Image • User picks up an object • Shows before the camera and takes a photo • Automatic cataloging (TagID, Image)

29. Annotating video with physical events

30. Related Work • Localization • LANDMARC • Indoor Location Sensing Using Active RFID • Sherlock (UMass) • Automatically locating objects for humans • Ferret (UMass) • RFID Localization for Pervasive Multimedia • Platform • Cascadia (UWashington) • Specifying, detecting and managing RFID events • Object Interaction • I sense a disturbance in the force (Intel Research, Seattle) • Unobtrusive detection of Interactions with RFID-tagged Objects • With other sensors • Fusion of RFID and Computer Vision (MSR)

31. Summary • SixthSense • Enterprise Setting • People and Objects tagged • RFID with other enterprise sensors • Components • Automated Inference • Platform • Applications http://research.microsoft.com/research/mns/projects/SixthSense/ Questions?

32. Backup

33. Semi-Automated Image Catalog • TagID-Image Cataloging • User picks up a tagged object • Hold it in front of the camera • Clicks a picture • Automatically identify the tagID of the object

34. SixthSense System Inference Engine, Database, Applications RFID Reader RFID Antennas Calendar Data Presence Data Queries Applications

35. RFID Applications • Industry • Tracking, Inventory, Supply Chain, Authentication • Research • Measurements • Improving reliability, security • Localization • RFID + Computer Vision • Interaction detection • RFID + other sensors

36. Person-Object Differentiation • People can move on their own • Objects move only when carried by a person Co-movement based heuristic • For every tag T, find co-movement tag set {T1, T2..Tn} • m – total inter-zone movement of T • mi – total inter-zone movement of Ti • ci – amount of co-movement exhibited by Ti with T • Declare the tag with the highest RM as person • Eliminate this tags movements • Repeat the algorithm till RM is positive • Tags with negative RM are objects

37. Zone Identification • Individual workspace • If there is one person predominantly present in a zone • Workspace of that person • Shared workspace • If no one person is predominantly present in a zone • Length of time from a person entry to exit < threshold • Reserved shared workspace • Length of time people are present > threshold • Common meeting entries in their calendars • Common areas • Any space that is not classified as one of the above

38. Challenges – Improving Reliability • Multi-tagging scheme • Affix multiple tags in different orientation • Increases the probability that atleast one of the tags being detected • Automatic Inference • Initially assume all tags belong to one giant super object • Fully connected graph • When two tags are detected simultaneously in different zones • Tags belong to different objects • Delete edges between them • Connected components • Set of tags attached to the same object

39. Evaluation – with untagged people

40. Automatic Conference Room Booking • Conference Room Zone is automatically identified • Reserved Space • Automatically book conference room • If it is not reserved • And bunch of people go into the conference room • And spend say 5 minutes

41. Discussion • Privacy • Deployed and managed by enterprise • Limited access to users • Relabeling approach • Economic Feasibility • Passive Tags are cheap • Prices are RFID readers expected to drop (Intel R1000) • Health Implications • Transmitted RF power (up to 2W) is well within safe limits • this question will undoubtedly continue to receive much attention and study