Towards ISO-Space: Annotating Spatial Information in Language

1. Towards ISO-Space: Annotating Spatial Information in Language ISO PWI 24617-4 Semantic annotation framework Part 4: Space James Pustejovsky (chair), Harry Bunt, Kiyong Lee, Inderjeet Mani (Proposed editors) May 31, 2009 ISO Meeting Muenzinger D 430 University of Colorado Boulder, CO

2. Schedule 9:00 Introductions, Member Interests 9:30 - 10:30 Goals, Scope, and Agenda 10:30 – 10:45 Coffee break 10:45 – 12:30 Resources and Data 12:30 – 14:00 Lunch 14:00 – 14:45 Defining a specification 14:45 – 15:00 Action Items

3. WG Participants • James Allen University of Rochester • Rusty Bobrow BBN • Harry Bunt  Tilburg University • David Cooper  NGA • Kiyong Lee Korea University • David McDonald BBN • Annie Zaenen PARC • Graham Katz Georgetown University • James Pustejovsky Brandeis University • Marc Verhagen Brandeis University • Christy Doran MITRE

4. Spatial Awareness in Language The keyhole was below the doorknob. John opened the door and entered the room. The light was to the left of the door on the wall. He walked to the living room. In front of the piano was a rug. On the rug was cat. A woman was at the piano. To her left was a window, from which one could see the garden. In the corner under a lamp, was a chair on its side.

5. Work Item Goals and Scenarios • Building a spatial map of objects relative to each other. • Reconstructing spatial information associated with a sequence of events. • Determining object location given a verbal description. • Translating viewer-centric verbal descriptions into other relative descriptions or absolute coordinate descriptions. • Constructing a route given a route description. • Constructing a spatial model of an interior or exterior space given a verbal description. • Integrating spatial descriptions with information from other media

6. Spatial Awareness Requirements • Topological relations between objects • Orientation and distance between objects • Shape and size of objects • Elevation (Lat, Long values) • Recognition of spatial entities and geopolitical regions • Granularity of the relations • Aggregates and distributed objects • Understanding the dynamics of motion and processes

7. Desired ISO-Space Elements • Regions • Geographic, Geopolitical Places, Functional Locations • Arbitrary Locations • Entities as Spatial Objects • intrinsic orientation, dimensionality, size, shape • Path Objects • routes, lines, turns, arcs • Links • Topological relations • Dimension and Orientation • Metrics • Spatial Functions • behind the building, twenty miles from Boulder • Movements and Spatial Processes • functions from regions to regions

8. Applications

9. Spatial Markup of Images Mapping from Spatial Event Structures to RCC8 relations

10. Spatial Relations in the Image Mapping from Spatial Event Structures to RCC8 relations

11. Image Annotation

12. Image Sequence Annotation

13. Cross-media AnnotationNemrava, Sadlier, Buitelaar, Declerck (2008)

14. Spatial Information in Language • Static Spatial Relations • PP; Mary is on the chair • Verbal; The tree stands in the yard. • Dynamic Spatial Relations • PP; Mary walked to the room. • Verbal; The tree fell. • Polysemy of Spatial Prepositions • over; over the bridge, over the hill. • at, on; on the table/wall, at the computer/party. • in; in the coffee, in the cup, in the bowl.

15. Spatial Prepositions • She lives near the school. • There is an ice cream shop by the store. • An oak tree grows next to my house. • The house is between Elm Street and Maple Street. • I found my pen lying among the books. • The bathroom is opposite that room. • Look towards the mountains.

16. Manner and Path Constructions

17. Language and Spatial Quantifiers • A policeman stands on every corner in town. • A dark cloud covered every house in town.

18. Spatial Relations in Motion Predicates

19. Motion Classes

20. Biking Blogs:http://www.rideforclimate.com/journals/?cat=3 March 7, 2006 Leaving San Cristobal de las Casas four days ago, I biked with Gregg and Brooks for one more day . We climbed over the mountains, and then descended to the east, where a thick green rainforest grew up around the road. We arrived in the town of Ocosingo and we were advised the road ahead was unsafe in the afternoon or night. I did not ask if there was a fire station and went straight to a cheap hotel that I split with Gregg and Brooks. The following morning , as I was planning to ride farther that day, I left at dawn while Gregg and Brooks were still asleep. I biked 30 miles to the town of Agua Azul where I played for 4 hours in waterfalls and clean cool pools. After 4 hours, I was surprised Gregg and Brooks had not arrived. I returned to the main road to continue on, and I asked a police officer if he had seen my friends. A few miles short of Agua Azul, while Gregg and Brooks were slowly climbing a hill, two men with machetes and masks jumped out of the forest. They demanded all of Gregg and Brooks' stuff. I spent the next day with Gregg and Brooks at the ruins of Palenque, walking around ruins of an ancient Mayan city and trying to relax . The following day, Gregg, Brooks, and I threw our bikes atop a van and drove the 90 miles to the border with Guatemala, where we had to take a boat across a river.

21. Military Campaigns:http://www.militaryhistoryonline.com/wwii/dieppe/ In the early morning hours of August 19, 1942, a fleet of up to 250 ships supported by 68 R.A.F squadrons was carrying a force of 6,000-plus men across the English Channel towards the areas that surrounded the city of Dieppe, a port town located in the Pays de Caux region of northeastern France. They would be transported to their target beaches with anticipation from their superiors of accomplishing a series of damaging blows to the German fortifications in Dieppe as well as in the towns, villages, and open areas surrounding Dieppe. But as the minutes approached 0400 hours (4 a.m.), a German convoy approaching from the north would be the first blow to unravel the entire operation. Nine hours later, the convoy would return to its homeports in defeat with high casualty counts. In the years after D-Day up to the present, historians and military officials agreed that Jubilee, as the operation was codenamed, was one of the Allies' greatest military blunders of World War II. The truth is, Jubilee was turning into a blundering operation long before it was executed, due to a series of planning mistakes, miscalculations, and changes made in the weeks and months leading up to August 19.

22. Walking Tours:http://www.leeds.gov.uk/About_Leeds/History/Historic_walks_in_Leeds_city_centre.aspx Leeds Bridge to Temple Mills historic walk This walk begins at Leeds Bridge which crosses the river Aire. Walk down Briggate and you will pass Queen's Court which was a merchant's house. On the other side of the road is a Leeds landmark, Dyson's. This was a famous jewellers and is now a restaurant. Go up Boar Lane to Holy Trinity Church and the Griffin Hotel. There is a Civic Trust plaque where a medieval gate used to stop people going into the Park of Leeds. Mill Hill chapel facing onto City Square is where Joseph Priestley was minister. He is famous for discovering oxygen. There is a statue of him in City Square along with John Harrison and James Watt. There are also statues of 'Morn' and 'Even' and in the centre a large statue of the Black Prince. Go along Neville Street and under the Dark Arches. You can see Bondman Dam which took water towards a number of mills where, during the middle ages, the people of Leeds had their corn ground into flour. Next is the Leeds Liverpool canal and this was finished in 1816 and is 127 miles long. Before crossing over Victoria Bridge is a large building which used to be a granary warehouse. On the other side of Water Lane is a development of offices and flats where the Round Foundry was. Machinery and steam engines were developed here. From Globe Road you can see two towers. The larger one, known as the Giotto Tower (1899), is based on a tower at the Florence Cathedral (1334) and was a ventilation shaft. The smaller tower on the left is a chimney (1864) which is a copy of a 12th century Lamberti tower in Verona. In the distance are rows of terraced houses in Holbeck. This walk finishes at Temple Mills where flax was spun from 1817-30.

23. DAML-Space(Hobbs et al) Rusty Bobrow, Murray Burke, Dan Connolly, Dejing Dou, George Ferguson, Andrew Gordon, Pete Haglich, Pat Hayes,Adam Pease, Steve Reed, Richard Waldinger The Semantic Web requires common ontologies with wide acceptance and use. DAML-S: an ontology of services Development began February 2001 About a dozen people in inner circle Some people have explored using it Institutional status at W3C Version 0.9 just released DAML-Time: a temporal ontology Development began February 2002 Most work by 1-3 people Abstract theory 90% complete Mapping between DAML-Time and TimeML One site “about to” use it Want to build on this experience for a spatial ontology.

24. Aims A widely available ontology of geographical and other spatial properties and relations Provide convenient markup and query capabilities for spatial information in Web resources Adequate abstract coverage of most spatial applications (not necessarily efficient) Link with special purpose reasoning engines for spatial theories and large-scale GIS databases Link with various ontological resources (e.g., OpenCyc, SUMO, ...) and annotation schemes Link with various standards for geographical information (OpenGIS, GML, ...)

25. Structure of Effort Cohn Hayes & Chaudhri Abstract Theory of Space (FOL) etc Complete or Partial Realization in DAML / OWL / RuleML / ... SUMO OpenCyc NLP Extraction Techniques Existing Standards Annotation Standards

26. Some Principles Delimiting the effort: Not a theory of physical objects, properties of materials, qualitative physics Link with numerical computation, don’t axiomatize it Link with large geographical DBs, don’t duplicate them Navigate past controversial issues by Keeping silent on issue Provide easily exercised options Use textbook logic for abstract theory; DAML/OWL-ize predicate and function declarations Provide simple, useful entry subontologies

27. Topics SPACETIME Topology Topology Dimension -- Orientation & Shape -- Length, area, volume Duration Lat/long, elevation Clock & calendar Geopolitical subdivisions -- Granularity Granularity Aggregates, distributions Temporal aggregates

28. Topology Points, arcs, regions, volumes Closed loops and surfaces Ordering relations & “between” in arcs; directions on lines and loops Connectedness, continuity Boundaries & surfaces, interior & exterior, directed boundaries; “airspace above” Disjoint, touching, bordering, overlapping, containing regions (RCC8); location at Holes NOT open and closed sets NOT pathological topologies

29. Dimension and Orientation Abstract characterization of dimension, projections on component dimensions Links w topological notions of dimension Frames of reference: earth-based, person-based, vehicle-based, force-based Relative orientations: parallel, perpendicular Cartesian vs polar coordinate systems, bearing & range Transformations between coordinate systems Degrees of freedom Qualitative trigonometry: granularities on orientations 2 1/2 dimensions: elevation as 2nd class dimension, system mostly thought of as planar Elevation from sea level vs ground level Planar vs spherical geometry

30. Shape 2D vs 3D shapes Linking w shape descriptions in geographical databases Shape descriptors: round, tall, narrow, convex,... Relative shapes: rounder, sharper, ... Same shape as, negative-shape, fits-in Symmetry Links w functionality of shape In artifacts, shape is almost always functional In natural objects, shape often has consequences ? Texture

31. Size Length, distance, area, and volume Precise measures Alternate descriptions of size English-metric conversions Coarse granularities: order of magnitude, half order of magnitude, implied precision, qualitative measures (large, medium, small) relative to comparison set Encoding uncertainty: bounded error, egg yolk theories Uncertainty of location vs imprecise regions

32. Granularity A city can be viewed as a point, a region, or a volume. How should these different perspectives be accommodated? One approach: City is an entity with 3D, 2D, and 0D realizations. User can pick which one(s) to use. Build granularity considerations into spatial ontology from the beginning, not as an add-on.

33. Geopolitical Regions Latitude and Longitude Natural geographical regions: Land masses: continent, island, ... Bodies of water: ocean, lake, river, ... Terrain features: mountain, valley, forest, desert, ... Political regions: Countries Political subdivisions: state, province, county, ... Municipalities: city, town, village, ... Residences and street addresses Other: Indian reservations, regulatory zones, ...

34. Linkages Exploit the large amount of research on spatial representation and reasoning OpenCyc, SUMO, Cohn, Galton, Hayes & Chaudhri, Hayes, Asher & Vieu, Egenhofer, Forbus Axiomatize best of this work in coherent fashion Link with existing large ontologies and annotation schemes SUMO, OpenCyc Ontology should bottom out in existing standards OpenGIS, GML

35. Target Applications As drivers for what has to be represented Flight map system, COA planning, trafficability Travel system involving lat/longs, political divisions, weather Alexandrian Digital Library Geologic and space (NASA) applications (3-D) Cell biology Image interpretation and description Robotics Virtual reality For some of these, we are collecting brief descriptions of the requirements for spatial representation and reasoning

36. Organization degree of community acceptance # of participants

37. Organization time to completion # of participants

38. Organization quality of ontology # of participants

39. Organization daml-spatial mailing list Web page - George Ferguson Coherent construction of abstract theories by small group of people Committee of interested persons in U.S. and Europe Email for commentary / feedback Telecons every 2 weeks to track issues/progress Presentations and discussion sessions at relevant workshops Early realizations of relevant parts of ontology in DAML Early construction of application-oriented entry subontologies

40. Spatial Information in Language • Static Spatial Relations • PP; Mary is on the chair • Verbal; The tree stands in the yard. • Dynamic Spatial Relations • PP; Mary walked to the room. • Verbal; The tree fell. • Polysemy of Spatial Prepositions • over; over the bridge, over the hill. • at, on; on the table/wall, at the computer/party. • in; in the coffee, in the cup, in the bowl.

41. Spatial Prepositions • She lives near the school. • There is an ice cream shop by the store. • An oak tree grows next to my house. • The house is between Elm Street and Maple Street. • I found my pen lying among the books. • The bathroom is opposite that room. • Look towards the mountains.

42. Manner and Path Constructions

43. Language and Spatial Quantifiers • A policeman stands on every corner in town. • A dark cloud covered every house in town.

44. Spatial Relations in Motion Predicates

45. Motion Classes

46. Desired ISO-Space Elements • Regions • Geographic Locations • Geopolitical Places • Functional Locations • Arbitrary Locations • Entities as Spatial Objects • intrinsic orientation, dimensionality, size, shape • Path Objects • routes, lines, turns, arcs • Links • Topological relations • Dimension and Orientation • Metrics • Motions • functions from regions to regions

47. Previous Work • Egenhofer and Franzosa (1991) • Randell et al. (1992) • Cohn and Hazarika (2001) • Nguyen and Worring (2003) • Hollink, Nguyen, Schreiber, Wielemaker, Wielenga, and Worring (2004) • Morarescu (2006) • Mani et al (2008)

48. Qualitative Spatial Reasoning Thanks to Anthony Cohn for slides

49. Qualitative Spatial Reasoning • Many aspects: • ontology, topology, orientation, distance, shape... • spatial change • uncertainty • reasoning mechanisms • pure space v. domain dependent

50. Qualitative Spatial Reasoning • Traditional QR spatially very inexpressive • Applications in: • Natural Language Understanding • GIS • Visual Languages • Biological systems • Robotics • Multi Modal interfaces • Event recognition from video input • Spatial analogies • ...