Classifying Email into Acts

1. Classifying Email into Acts • From EMNLP-04, Learning to Classify Email into Speech Acts, Cohen-Carvalho-Mitchell • An Act is described as a verb-noun pair (e.g., propose meeting, request information) - Not all pairs make sense. One single email message may contain multiple acts. • Try to describe commonly observed behaviors, rather than all possible speech acts in English. Also include non-linguistic usage of email (e.g. delivery of files) Verbs Nouns

2. Some Improvements • With more labeled data (1743 msgs), using 1g+2g+3g+4g+5g features • Using careful (and act specific) pre-processing of message text • Using act specific feature selection scheme (Info Gain, ChiSquare, etc) - Significant performance improvements.

3. Some Examples of 4-grams is fine with mmee is good for mmee i will be there i will look for will look for pppeople i will see pppeople as soon as I $numbex per person i will bring copies our meeting on dday is ok for mmee look for pppeople in will try to keep -numbex i will i will try to i will check my I do not have meet at horex pm horex pm on ddday on ddday at horex pppeople meet at horex to meet at horex would like to meet please let mmee know ddday at horex am ddday at horex pm lets plan to meet would pppeople like to pppeople will see pppeople is fine with mmee numbex-numbex pm can pppeople meet at ddday numbex/numbex is good for mmee and let mmee know know what pppeople think would be able to do pppeople want to do pppeople need to do not want to pppeople need to get please let mmee know pppeople think pppeople need mmee know what pppeople what do pppeople think pppeople be able to pppeople don not want pppeople would be able that would be great Call mmee at home Meeting (noun) Request Commit Req

4. Results

5. Results

6. Request Request ??? Proposal Delivery Commit Parent message Child message Collective Classification: Predicting Acts from Surrounding Acts

7. Content versus Context Request Request ??? Proposal Delivery Commit Parent message Child message • Content: Bag of Words features only (using only 1g features) • Context:Parent and Child Features only ( table below) • 8 MaxEnt classifiers, trained on 3F2 and tested on 1F3 team dataset • Only 1st child message was considered (vast majority – more than 95%) Kappa Values on 1F3 using Relational (Context) features and Textual (Content) features. Set of Context Features (Relational)

8. Collective Classification Model Commit … … … Other acts Request Deliver Current Msg Parent Message Child Message

9. Collective Classification algorithm (based on Dependency Networks Model) New inferences are accepted only if confidence is above the Confidence Threshold. This Threshold decreases linearly with iteration, and makes the algorithm works as a temperature sensitive variation of Gibbs sampling – after iteration 50, the threshold is 50% and then a pure Gibbs sampling takes place

10. Collective Classification Results

11. Act by Act Comparative Results Kappa values with and without collective classification, averaged over the four test sets in the leave-one-team out experiment.

12. What goes next? • Extend Collective classification by using the new SpeechAct classifiers (1g-5g, feat selection) • Online(incremental) and semi-supervised learning – CALO focus. • Integration of new Speech Act package to Minorthird & Iris/Calo. • Role discovery – network-like features + speech act