STS: under the hood

1. Diana McCarthy Erasmus Mundus Visiting Scholar Saarland University STS: under the hood

2. Proposal: Annotation with Alignments • so that we can see where the similarity lies and rationale for scores • sub-alignments • look for consensus on sub-parts • alignments annotated with relation e.g. (just for brainstorming purposes) • = (equivalence/substitutable) • != (contradiction) • → entailment • - (missing) • extra propositional • speculation/certainty • sentiment • assign category (relation) and score to whole text pair • or to sub- alignments

3. Reference: The new system costs between $1.1 million and $22 million, depending on configuration. Candidate: The system is priced from US$1.1 million to $22.4 million, depending on configuration. Annotation (examples from Microsoft Research Paraphrase Corpus: STS pilot task)

4. Reference: The new system costs between $1.1 million and $22 million, depending on configuration. Candidate: score 4.2 The system is priced from US$1.1 million to $22.4 million, depending on configuration. (* good starting point, but we want to look inside the box * brain storming – all annotations done by myself in 20 mins before coming to the workshop.) Annotation (Please note, 1-5 scores off top of my head before seeing guidelines for illustrative purposes only)

5. Reference: [=A The [-X new] system][=B[=D costs] between [=C $1.1 million and $22 million], depending on configuration.] Candidate: score 4.2 [=A.4.2 The [-X] system][=B.4[=D.5 is priced] from [=C.4 US$1.1 million to $22.4 million], depending on configuration.] (* mark alignments between reference and candidate, with category (equivalence =, entails !=, - missing etc...) and score. * Alignments may overlap * May also get non contiguous sections which we can mark with same id (A, B, C etc...)) Annotation with Alignments: (brainstorming purposes)

6. Reference: [=A The hearing occurred a day after the Pentagon for the first time singled out an officer, Dallager, for not addressing the scandal.] Candidate: score 4.9 [=A.4.9 The hearing came one day after the Pentagon for the first time singled out an officer - Dallager - for failing to address the scandal.] (* To save annotators – and systems, could avoid aligning everything. Do sub-alignments where the a subpart differs from the whole, by category or score) Annotation with Alignments: (brainstorming purposes)

7. Reference: [=C U.S.] prosecutors [=B have arrested more than 130 individuals] and have [=D[=F seized][-Y more than] $17 million [-X]] in a continuing crackdown on [=E Internet fraud [-Z and abuse].] Candidate: score ? [=B.5 More than 130 people have been arrested] and [=D.3[-Y] $17 million [-X worth of property][=F.5 seized]] in an [=E Internet fraud [-Z]] sweep announced Friday by three [=C.5 U.S.] government agencies. (* annotators should be allowed to leave parts without annotation. Don't know is important. Also allow for comments on any item. * Could weight according to salience of word, modifier or predicate, syntactic relation, order in sentence (new information towards the end). All depends on goal.) Annotation with Alignments

8. Reference: [=A The company][!=C didn't detail [-specD][=B the costs of the replacement and repairs]]. Candidate: score 4.9 But [=A.5 company officials][!=C [-specD expect][=B.5 the costs of the replacement work] to run into the millions of dollars.] ( * mark speculation somehow, where missing or different type of level) Annotation with Alignments:

9. Components • Need for semantic and non semantic (syntax, pragmatic, extra propositional, extra-linguistic. • Interleaved, but components could provide score on sub-components just as annotators can • Systems mark confidence and components used on sub-alignments with categories (equivalence, contradicts, entails, speculation) • We can learn interaction, rather than assume a priori • Sampling really important, esp if want thin tail rather than just fat head! (Steedman, ACL dinner 2007) • However, all depends on your goal/practical requirement