70 likes | 358 Views
Evaluation Plan. Stuart Wheater Arjuna Technologies Ltd., UK. Deliverables. Month 6 Evaluation Plan (D15) Month 12 Updated Evaluation Plan (D16) Month 18 Draft Revised Evaluation Plan (D?) Month 24 Revised Evaluation Plan (D?)
E N D
Evaluation Plan Stuart Wheater Arjuna Technologies Ltd., UK
Deliverables • Month 6 • Evaluation Plan (D15) • Month 12 • Updated Evaluation Plan (D16) • Month 18 • Draft Revised Evaluation Plan (D?) • Month 24 • Revised Evaluation Plan (D?) • “… should evolve in order to provide quantitative evaluation …”
Review Feedback • “The evaluation plan, which in its current form mainly addresses lower level technical questions well need to be updated with the higher level issue, which will allow meaningful measurement of project results” • Draft Revised Evaluation Plan, M18 • Revised Evaluation Plan, M24
Suggestions Functional Evaluation • “Functional issues should be addressed at a much higher level of abstraction” • What can be adapted, and when? • What are the characteristics of a composite service executed? • What about a service can be described?
Suggestions Functional Evaluation (Cont.) • “Functionality should be be evaluated in terms of the initial requirements that have been set out for ADAPT.” • “Technical annex”, not “Call” • “... , by providing a proof-of-concept implementation of standards on top of ADAPT interfaces should give sufficient insight to evaluate the final system.” • ? • “In summary, with respect to functionality, a higher-level grouping of functions and accompanying evaluations criteria is needed.”
Suggestions Non-Functional Evaluation • “… scientifically more interesting, is the evaluation of ADAPT with respect to its non-functional aspects, notably availability and adaptability.” • Appears not to be interested in “standard performance” • “… setting up experiments and evaluation criteria.” • Hinted at, but no detail • “I expect that experiments will be (partly) based on trace-driven simulation, for which reason it is important to know in advance exactly what will be evaluated and how”
Summary • Reorganize into higher-level grouping of functions and evaluations criteria • Determine exactly what non-functional aspects will be evaluated and how