1 / 24

Accelerating External Search with Bitstate Hashing

Accelerating External Search with Bitstate Hashing. Stefan Edelkamp Shahid Jabbar Computer Science Department University of Dortmund, Dortmund, Germany. Model Checking. Given A model of a system. A specification property

alaula
Download Presentation

Accelerating External Search with Bitstate Hashing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Accelerating External Search with Bitstate Hashing Stefan Edelkamp Shahid Jabbar Computer Science Department University of Dortmund, Dortmund, Germany

  2. Model Checking • Given • A model of a system. • A specification property • Model Checking Problem: Does the system satisfy the property ? • Method: An exhaustive exploration of the state space to search for a state that does not satisfy the property. • Problem: How to cope with large state spaces that do not fit into the main memory? Accelerating External Search with Bitstate Hashing

  3. Two Very Effective Methods • External Search: + Uses Harddisk to store the state space. + Implemented on top of SPIN model checker. + Promising: Largest exploration so far took ~20 GB – much larger than even the address limits of most computers. + Pause and Resume support – Can add more harddisks. - Slow duplicate detection phase • Bitstate Hashing (Partial Search): + Uses only a single bit per state for closed list => faster duplicates removal. + What if the size of the open list becomes greater than the internal memory ? - Solution reconstruction is not possible. Accelerating External Search with Bitstate Hashing

  4. Best of the Both Worlds • External Search combined with Bitstate Hashing + Faster duplicates removal. + Open and Closed lists can be saved on the harddisk. + (Partial) Solutionreconstruction. Accelerating External Search with Bitstate Hashing

  5. Outline Directed Model Checking External Search Partial Search External Partial Directed Search Experimental Results Conclusions

  6. Directed Model Checking (Edelkamp, Leue, Lluch-Lafuente, 2004) • A guided search in the state space. • Usually by some heuristic estimate. • Only promising states are explored. • Under-certain conditions proved to be complete. • Shorter error trails • Better for human comprehension • Problem: The inevitable demands of the model .. Space, space and space. Accelerating External Search with Bitstate Hashing

  7. Possible Solution • Use Virtual Memory. • Assume a bigger address space divided into pages. • Saved on the hard disk but are moved back to the main memory whenever they are “called” – Page Faults. • Pages are mapped to physical locations within the main memory and the desired content is returned from the main memory location. Accelerating External Search with Bitstate Hashing

  8. Problem with the Virtual Memory Virtual Address Space 0x000…000 Memory Page 0xFFF…FFF Accelerating External Search with Bitstate Hashing

  9. B Disk External Memory Model (Aggarwal and Vitter) If the input size is very large, running time depends on the I/Os rather than on the number of instructions. M Scan(N) = O(N / B) Sort(N) = O(N/B log M/B N/B) Input of size N and N >> M Accelerating External Search with Bitstate Hashing

  10. External Memory Graph Algorithms • External breadth first search [Munagala and Ranade, 2001]: • Generated states flushed to the disk for every BFS level. • No hash table. • Duplicates are removed by sorting the nodes according to the indices and doing an scan and compaction phase. • Before expanding a layer t, the nodes in the layer t-1 andt-2 are subtracted from t. • O(|V| + sort(|V| + |E|)) I/Os. where sort(N) = O(N / B logM/B N / B) I/Os • [Korf, 2003] presented the breadth first search version for implicit graphs. Accelerating External Search with Bitstate Hashing

  11. Duplicates’ Removal X Y Z A X B C D External BFS X Y Z A X Y Z A t t+1 t+2 Accelerating External Search with Bitstate Hashing

  12. A* Algorithm a.k.a Goal-directed Dijkstra • A heuristic estimate is used to guide the search. • E.g. Straight line distance from the current node to the goal in case of a graph with a geometric layout. • Reweighing:w’(u,v) = w(u,v) – h(u) + h(v) • Problems: • A* needs to store all the states during exploration. • A* generates large amount of duplicates that can be removed using an internal hash table – only if it can fit in the main memory. • A* do not exhibit any locality of expansion. For large state spaces, standard virtual memory management can result in excessive page faults. Accelerating External Search with Bitstate Hashing

  13. Take a closer look h • Implicit, unweighted, undirected graphs • Consistent heuristic estimates. => ∆h ={-1,0,1} It’s a Bucket !! g Accelerating External Search with Bitstate Hashing

  14. Insert state Flush when full Buffer in internal memory File on disk Bucket • A Bucket is a set of states, residing on the disk, having the same (g, h) value, • Where, g = number of transitions needed to transform the initial state to the states of the bucket, • and h = Estimated distance of the bucket’s state to the goal • No state is inserted again in a bucket that is expanded. • If Active (being read or written), represented internally by a small buffer. Accelerating External Search with Bitstate Hashing

  15. External A* [Edelkamp, Jabbar, and Schroedl, 2004] • Buckets represent temporal locality – cache efficient order of expansion. • If we store the states in the same bucket together we can exploit the spatial locality. • Munagala and Ranade’sBFS and Korf’s delayed duplicate detection for implicit graphs. External A* Accelerating External Search with Bitstate Hashing

  16. Complexity Analysis • Internal A* => Each edge is looked at most once. • Duplicates Removal: • Sorting the green bucket having one state for every edge from the 3 black buckets. • Scanning and compaction. • O(sort(|E|)) • Subtraction: • Removing states of orange buckets (duplicates free) from the green one. • O(scan(|V|) + scan(|E|)) Accelerating External Search with Bitstate Hashing

  17. I/O Performance of External A* Theorem: The complexity of External A* in an implicit unweighted and undirected graph with a consistent estimate is bounded by O(sort(|E|) + scan(|V|)) I/Os. Accelerating External Search with Bitstate Hashing

  18. Bitstate Hashing • Single bit per state to represent Closed list. • Guarantees duplicates removal. • But: can remove non-duplicates too – false-positive. • Average probability of a false-positive error during the search: Since the ith element collides with one of the i-1 already inserted elements with a probability of at most (i-1) / m n: size of the state space; m: size of the hash table Accelerating External Search with Bitstate Hashing

  19. External Partial Directed Search H[0 … m – 1]: Bitstate Hashtable fori = 0 tom – 1 H[i]  false Procedure External A* Bucket(0, h(I))  {I} fminh(I) while (fmin ≠∞) g min{i | Bucket(i, fmin − i) ≠ } while (g≤fmin) hfmin − g forallv єsucc(Bucket(g, h)) if (notH[v]) H[v]  true A(fmin),A(fmin + 1),A(fmin + 2)  v Bucket(g + 1, h + 1)  A(fmin + 2) U Bucket(g + 1, h+1) Bucket(g + 1, h)  A(fmin + 1) U Bucket(g + 1, h) Bucket(g + 1, h − 1)  A(fmin) U Bucket(g + 1, h − 1) gg + 1 fmin min{i + j > fmin | Bucket(i, j) ≠ } U {∞} Accelerating External Search with Bitstate Hashing

  20. Experimental Results -1 • Deadlock Detection in Optical Telegraph Accelerating External Search with Bitstate Hashing

  21. Experimental Results -2 • CORBA GIOP – 1 Server, N Clients Accelerating External Search with Bitstate Hashing

  22. Experimental Results -3 • CORBA GIOP – 2 Servers, N Clients 20 GB space requirement Accelerating External Search with Bitstate Hashing

  23. Experimental Results – 4Negative case • Deadlock Detection in Dining Philosophers Time to compute the hash value is dominant than external sorting! Accelerating External Search with Bitstate Hashing

  24. Summary • State space explosion problem can be circumvented by Directed External Model Checking or Bitstate Hashing. • Both approaches have some flaws. • A fusion of both approaches cover individual flaws. • Implemented on top of IO-HSF-SPIN – SPIN model checker with external heuristic search. • Gain in speed up till 9 times. -- THE END -- Accelerating External Search with Bitstate Hashing

More Related