Omphalos Session

1. Omphalos Session

2. Omphalos Session Programme • Design & Results 25 mins • Award Ceremony 5 mins • Presentation by Alexander Clark 25 mins • Presentation by Georgios Petasis 10 mins • Open Discussion on Omphalos and GI competition 20 mins

3. Omphalos : Design and Results Brad Starkie François Coste Menno van Zaanen

4. Contents • Design of the Competition • A complexity measure for GI • Results • Conclusions

5. Aims • Promote new and better GI algorithms • A forum to compare GI algorithms • Provide an indicative measure of current state-of-the-art.

6. Design Issues • Format of training data • Method of evaluation • Complexity of tasks

7. Training Data • Plain Text or Structured Data • Bracketed, Partially bracketed, Labelled, Unlabelled • (+ve and –ve data) or (+ve data only) Plain Text,(+ve and –ve) and (+ve only) • Similar to Abbadingo • Placed fewest restrictions on competitors

8. Method of Evaluation • Classification of unseen examples • Precision and Recall • Comparison of derivation trees Classification of unseen examples • Similar to Abbadingo • Placed fewest restrictions on competitors

9. Complexity of the Competition Tasks • Learning task should be sufficiently difficult. • Outside the current state-of-the-art, but not too difficult • Ideally provable that the training sentences are sufficient to identify the target language

10. Three axes of difficulty • Complexity of the underlying grammar • +ve/-ve or +ve only. • Similarity between -ve and +ve examples.

11. Complexity Measure of GI • Created a model of the GI based upon a brute force search (Non polynomial) • Complexity measure = size of the hypothesis space created when presented with a characteristic set.

12. Hypothesis Space for GI • All CFGs can be converted to Chomsky Normal Form. • For any sentence there are a finite number of unlabelled derivations given CNF • Finite number of labelled derivation trees • The grammar can be reconstructed given sufficient number of derivation trees • All possible labelled derivation trees corresponds to all possible CNF grammars given the maximum number of nterms • Solution: calculate max number of nterms and create all possible grammars

13. BruteForceLearner • Given the positive examples construct all possible grammars • Discard any grammars that generate any negative sentences • Randomly select a grammar from hypothesis set

14. Characteristic Set of Positive Sentences • Put the grammar into minimal CNF form • If a single rule is removed one or more sentences can't be derived • For each rule add a sentence that can only be derived using that rule • Such a sentence exists if G in minimal form • When presented with this set, one of the hypothesis grammars is correct

15. Characteristic set of Negative Sentences. • Given G calculate positive sentences • Construct hypothesis set • For each hypothesis H  G, L(H)  L(G) add + sentence s | s L(G) but s L(H) • For each hypothesis H  G, L(H)  L(G) add - sentence s | s L(H) but s L(G) Generating -ve data according to this technique requires exponential time – Therefore cannot be used to generate –ve data in Omphalos.

16. Creation of the Target Grammars • Benchmark probs identified in literature • Stolcke-93,Nakamura-02,Cook-76,Hopcroft-01 • Number of nterms, terms and rules were selected • Randomly generated grammars, useless rules removed, CF constructs (center recursion) added • A characteristic set of sentences was generated, and complexity measured • To test if deterministic, LR(1) tables created using Bison • For non-deterministic grammar non-deterministic constructs added

17. Creation of positive data • Characteristic set generated from grammar • Additional training examples added • Size of training set 10  20 size of characteristic set • Longest training example was shorter than the longest test

18. Creation of negative data • Not guaranteed to be sufficient • Originally randomly created (bad idea) • For probs 6a  10 regular equivalents to grammars constructed and negative data could be generated from regular equivalent to CFG • Nederhof-00 • Center recursion expanded to a finite depth Vs true center recursion • Equal number of positive and negative examples in the test sets

19. Participation • Omphalos 1st page: ~ 1000 hits from 270 domains • Attempted to discard crawlers and bots hits • All continents except 2 • Data sets : downloaded by 70 different domains • Oracle: 139 label submissions by 8 contestants (4) • Short test sets: 76 submissions • Large test sets: 63 submissions

20. Results

21. Techniques Used. • Prob 1 • Solved by hand. • Probs 3, 4, 5, and 6 • Pattern matching using n-grams. • Generated its own negative data • the majority of randomly generated strings would not be contained within the language. • Probs 2, 6.2, 6.4 • Distributional Clustering and ABL

22. Conclusions • The way in which negative data is created is crucial to judging performance of competitors entries

23. Review of Aims • Promote development of new and better GI algorithms • Partially achieved • A forum to compare different GI algorithms • Achieved • Provide an indicative measure of the state-of-the-art. • Achieved