Access Pattern Analysis, Ideas and Alternative Approaches

1. Crimestat: Performance Tuning Access Pattern Analysis, Ideas and Alternative Approaches Pradeep Mohan

2. Outline • Overview - Crimestat • Motivation • Background • Datasets Description • Distance Calculation • Types of Distance Requests • Function Categories • Access Pattern Analysis • K- Nearest Neighbor Analysis • Hotspot Analysis • K Means • Hierarchical Clustering • Access Patterns – Other Modules • Journey To Crime • Journey To Crime • Crime travel Demand • Problem Definition • Proposed Approach • Voronoi Diagram • TAZ Approximation • K Order Nearest Neighbor • NNH • Network Assignment • Spatial Indexing • Challenges • References

3. Overview - Crimestat • A multi-threaded windows application for crime mapping and analysis. • Main modules of interest • Distance Analysis (K- Nearest Neighbor Analysis) • Hotspot Analysis (Nearest Neighbor Hierarchical Clustering and K Means) • Journey To Crime( Bayesian Journey to Crime) • Space Time Analysis (Knox Index) • Crime Travel Demand (Network Assignment) • Datasets • Multiple point datasets (Ex – Criminal Arrest Record (with location, time) and Crime Incidence Record (with location and time) ) • Traffic Analysis Zones and Road Network data sets (for Journey to Crime and Crime Travel Demand). • Different distance metrics computed between different point sets. (Ex. Euclidean, Spherical, Manhattan and Network.

4. Motivation How are crime incidents clustered together? Courstsey: ESRI. Hot Spots Analysis What are the predicted trips of a serial offender? Journey to Crime Estimation

5. Background - Datasets Description Centroid of a TAZ Courtsey : Ned Levine and Associates

6. Background - Datasets Description Courtsey: Ned Levine and Associates

7. Background - Datasets Description Primary Pointset Courtsey: Ned Levine and Associates

8. Background - Datasets Description Secondary Pointset Courtsey: Ned Levine and Associates

9. Background – Distance Calculation set of N x K points in Euclidean space (or on a network) Distance between all pairs of points. (What is the problem ?) Normal computation takes O(n2) (Why is it hard ?) q1 q2 q3 q4 q5 qk Distance Matrix pn p5 p4 p3 p2 p1 • Do all functions require these distances? • Can calculated distances be re-used? • How do we store them? (in a file) • How do we efficiently search for them? • Is there a single algorithm to calculate all these distances? • Can they be calculated on the fly? • Euclidean Distance • Manhattan Distance • Spherical Distance • Network Distance Calculations in a single threaded application A Distance Cell

10. Background - Types of Distance Requests • given a distance d, find all points separated within this distance (whole dataset!!) • Given a point p, find its k order nearest neighbors. (Ex. p6 is 1st order and p2 is 2nd order). • Find a set P ( of incident points ), given a zone Z, a polygon. • Given a set of Points P and another set Q, find all pair euclidean distance.

11. Background – Function Categories • K - Nearest Neighbor Analysis based modules • Nearest Neighbor Analysis (K order) • Ripley’s K Statistic (simulation) • Point to Point allocation • Point to Zone Allocation • Hot Spot Analysis based modules • K Means clustering • Spatio Temporal Analysis of Crime • Anselin’s Moran • Nearest Neighbor Hierarchical clustering • Risk Adjusted Nearest Neighbor Hierarchical Clustering • Space Time Analysis • Crime Trip Estimation • Travel Demand Modeling

12. Access Patter Analysis Analysis

13. K- Nearest Neighbor Analysis Input: • A set of incident locations, N • K (order of NN) Ouput: • 1- order Nearest Neighbor for all N • K- order Nearest Neighbor for all N Method: 1. For every point (pi, pk)є P, computes the distance. 2. For every particular point quick sorts all distances to get the nearest neighbor. 3. For , K order , top K neighbors of a point are selected. 4. Statistics calculated: mean Random distance, Nearest Neighbor Index etc.

14. Access Patterns • All K-order distances calculated. • Such computations performed on whole dataset. • A distance matrix is calculated – O(n.n) d1 d2 d3 d4 dk

15. Assigning Point to Point , Point to Zone (polygon or grid) Point to Point Assignment Input: A set P of N points and another Q set of M points. Output: An assignment of each point in P to a point of Q. Method: Proximity calculation – For every point in P to every other point in Q. Point to Polygon Assignment Input: A set Pof Npoints and another set Q of M polygons. Output: An assignment of each point in P to a polygon of Q. Method: Point in Polygon for M x N times. A Pre-computed distance Matrix is used currently for distances.

16. Access Pattern Secondary Point Primary Point d4 • Distance of every Pi with every Si is calculated and ordered. • A distance Matrix is used. d3 Si d2 d5 Pi d6 d1

17. Pi Incident Point (in P ) TAZ Centroid Courtsey : Ned Levine and Associates

18. Ripley’s K Statistic Input: Set of N Points. Output: L(t) : measure of second order clustering Method: • Draw a circle around each point, • Collect all points within the radius. • Increase the radius and repeat above operations • Repeat for 100 increments of radius till maximum distance Random Point Set is generated, so distance matrix needs to be recalculated.

19. Primary Point • O1 and O2 are different order radii. • K order radii are computed for each point. • O(N.K) , N Points • Distance calculation O(N.N) (for every new simulation) O2 O1

20. Hot Spot Analysis Modules

21. Mode and Fuzzy Mode Input: • A set of N Points. • A Radius R. (Fuzzy Mode) Output: • Frequency of Incidents at each point – Mode • Frequency of incidents at a radius from a point – Fuzzy Mode Method: • Mode – A frequency count of number of incidents on each point. (For N points) • Fuzzy Mode – A frequency count of incident within a radius around a particular point.

22. Access Patterns: K Means Computing initial seeds Using secondary set of points as seeds Overlay a grid and cell with highest count is seed. Grid approach expensive in O(gridsize x k), k: number of clusters. If a grid size doesn’t produce a cluster another is tried. – worst case Distance measurements of all “n” points with k clusters performed till convergence. O (k x n x iterations till convergence)

23. Nearest Neighbor Hierarchical Clustering (NNH) Input: A set of N points (same file) A search distance, d (random or fixed) No. of simulations, k (order) Min number of points per cluster Output: 1. All k order clusters Conditions: Distance between pairs of points > d Cluster size > = minimum number of points. Method: Compute all pair euclidean distance. Prune based on distance threshold. Computation Saving: Distances have been calculated already. N = 1349, Fixed Distance (dt) = 5 miles Pair Count = 231742 (so many distances evaluated) Courtsey : Ned Levine and Associates

24. Primary Point 1st Order Cluster 2nd Order Cluster Access Patterns

25. Risk Adjusted Nearest Neighbor Hierarchical Clustering (NNH) • Use of baseline variable ( Ex. census blocks). • Interpolate to a grid size based on primary file. (say size N) • Determine absolute densities (of secondary) as points per grid cell. • Proceed as NNH. An O(N.N) for calculating grid parameters, N is primary point set size.

26. Primary Point Spatio Temporal analysis of Crime (STAC) • Area divided into grids • Circle drawn on grids • Circles pruned based on number of points. • Intersecting circles merged.

27. Access Patterns: Other Modules Other Modules Knox Index Mantel Index Distance Matrix re-computed every time for each simulation (simulated point set).

28. Journey To Crime

29. Journey to Crime Input: • A List of incidents committed by a serial offender. • A travel decay function. • A Reference Grid. • Origin and Destination of offenders. (Case 2) Output: The origin of crime (home of offender) Crime Trip (case 2) Observation: O( N(grid cells).Incidents)

30. Courtsey : Ned Levine and Associates

31. Crime Travel Demand • Given Origin and Destination of Criminal – Generate Trips. • Assign Origins and Destinations to Zones (Ex. TAZ, Census Blocks). • Predict trips based on various demand models. • All data points of Primary and Secondary file accessed. • Distance computations are O(N.N).

32. Network Assignment Computation: an Expensive Join between Euclidean Crime trip and a Network based on constraints. Given: • A set of Crime Trips. • A Transportation Network Find: • The actual route on the transportation network Constraints: • Routes are weighted by both distance and time

33. Courtsey : Ned Levine and Associates

34. Courtsey : Ned Levine and Associates

35. Courtsey : Ned Levine and Associates

36. Problem Definition • Given: • set of N Primary Points • set of M secondary points • TAZ or Census Block • A transportation Network • User Defined Parameters • Request for a particular task based on spatial proximity. • Find: • Proximity measure in terms of distance. • Objective: • Define a suitable data structure for storage of input data. • Define a suitable Hierarchical Index. • Define a suitable Join (and Hierarchical Join Index) between different spatial sets. • Define an appropriate storage representation on disk. • Constraints: • Find out all requested proximity measures with lower cost of computation.

37. Related Work q1 q2 q3 q4 q5 qn • Naïve approach to distance calculation • Lazy approach to distance calculation. • R Tree based Index • Hierarchical Join Index • Hierarchical Voronoi based Index Distance File pn Program Address Space p5 p4 p3 p2 p1 p1 • Useful only for Euclidean space. • Networks also need to be stored and accessed separately. • Costly Joins Need to be computed pn R Tree Index Structure

38. Proposed Approach – Voronoi Diagram • Based on spatial proximity of points. • O(nLogn) to calculate the diagram. • Distance of nearest neighbors stored during construction. • A hierarchical index based on voronoi to be constructed. • Voronoi Joins for Euclidean and Networks.

39. Proposed Approach Point to Point and Point to Polygon Assignment – TAZ Approximation Incident Point (in P ) TAZ Centroid

40. K- Order Nearest Neighbor Calculation • For every edge in voronoi, • The sites split by an edge is known . (during computation of voronoi) • There are O(N) edges (for N points). • An edge traversal gives a pair of distance. (which is stored) • Points are kept in an ordered bucket. • Quick sort within the points all its distances. • Also store its neighbor connected by that distance. p2 p3 d2<=d >=d p1 p4 d1<=d p5 Query Point p6 All “k” order nearest neighbors Z Median Center of a Polygon (TAZ) Incident Point ( belongs to set P)

41. Proposed Approach – NNH • Use of an already existing algorithm Amoeba by Castro et al.[2] • O(nLogn) , n is the number of points. • Makes use of Delaunay Triangulation.

42. Proposed Approach - Network Assignment • Use of a Network Voronoi Diagram , Graf and Winter [1]. • Computation of the Voronoi Diagram of the Euclidean Crime Trips ( Origin Destination points) • Computing a Join between the two. • These computations might be expensive. • Eulidean Voronoi’s have been calculated already and stored. Z

43. Proposed Approach - Spatial Indexing Hierarchical Voronoi Indexing [3] • Efficient Paging Mechanism • Spatial Proximity very useful. • Extensible even to networks.

44. Challenges • When to compute the voronoi diagrams? • Repeated computations might be costly. • Need to store the computed distances, voronoi partitions, intermediate results in a file. • Need for a Hierarchical Index : efficient access points and voronoi seeds.

45. References 1. Graf, M.,Winter, S., 2003: Netzwerk-Voronoi-Diagramme. ÖsterreichischeZeitschriftfürVermessung und Geoinformation, 91(3): 166-174. (Network Voronoi Diagrams, english translation) • Castro, E., Vladimir and Lee., I (2000). AMOEBA: Hierarchical Clustering Based on Spatial Proximity Using Delaunay Diagram. Proceedings of the 9th International Symposium on Spatial Data Handling (SDH2000). Beijing, China. • Gold,C., and Angel., P ., 2006 Voronoi Hierarchies LECTURE NOTES IN COMPUTER SCIENCE, pp 99–111, 2006. Springer-Verlag Berlin Heidelberg 2006

46. Thank You