50 likes | 153 Views
Transfer Learning Evaluation Workshop 2-3 May 2006 Evaluation Data Format & Analysis Clayton T. Morrison, Yu-Han Chang, Paul R. Cohen USC Information Sciences Institute {clayton,ychang,cohen}@isi.edu. Data Format. ( list of IDs and variable settings for each
E N D
Transfer Learning Evaluation Workshop 2-3 May 2006 Evaluation Data Format & Analysis Clayton T. Morrison, Yu-Han Chang, Paul R. Cohen USC Information Sciences Institute {clayton,ychang,cohen}@isi.edu
Data Format ( list of IDs and variable settings for each ((ID repl#) (list of points on curve)) . . . ((ID repl#) (list of points on curve)) ) ( ( ( :<condition-name> (:<variable_1> <value_1>) . . . (:<variable_n> <value_n>) ) . . . ) ( ( ( :<condition-name> <replication-number> ) ( (x1 y1) (x2 y2) . . . (xn yn) ) ) . . . ) ) • We’ll expect x1=0 so we can measure jump-start. • We prefer that x1, x2, … for each curve to be consistent but will interpolate if necessary
An example ( ( ( :exp1-b (:near-far 0.5) (:mtn NIL) (:train NIL) ) ( :exp1-ab (:near-far 0.5) (:mtn NIL) (:train T) ) ) ( ( ( :exp1-b 1 ) ( (0 15.4)(20 28.3)(40 33.1)(60 54.2)(80 56.1)(100 56.2) ) ) ( ( :exp1-b 2 ) ( (0 11.5)(20 23.9)(40 31.9)(60 54.0)(80 55.5)(100 55.9) ) ) . . . ( ( :exp1-b 7 ) ( (0 13.5)(20 28.9)(40 39.4)(60 56.0)(80 55.9)(100 56.2) ) ) ( ( :exp1-ab 1 ) ( (0 35.5)(20 40.4)(40 60.3)(60 62.4)(80 66.7)(100 66.9) ) ) . . . ( ( :exp1-ab 7 ) ( (0 40.1)(20 56.2)(40 59.1)(60 66.4)(80 66.9)(100 66.8) ) ) ) )
Provide TL Analysis Functions • If you want to test any statistics other than the standard ones (transfer ratio, jump-start, etc), simply provide a function that operates on a pair of learning curves: (defun your-stat-fn (curve1 curve2) <your code to compute a statistic f>) where curve1 looks like ((0 10.3) (20 12.8) … (100 34.1)) • We’ll provide code for calculating the standard statistics. • You can provide replacement functions for the statistic (e.g. transfer-ratio), summarization (e.g. average), and asymptote (e.g. max-over-all-data) • We’ll also place all of this code online so you can use it. • To compare your curves, you’ll do something like (process-data :exp1-b :exp1-ab 7 #’transfer-ratio)
End Example of good curve. * bonus points for group that provides us with one