1 / 36

IRENE_DLL: a library to evaluate model performance

IRENE_DLL: a library to evaluate model performance. Presented by: Gianni Fila Research Institute for Industrial Crops Agronomy Section g.fila@isci.it. Topics. Background Overview Using the library: basic concepts Special procedures Application examples Future developments.

silviac
Download Presentation

IRENE_DLL: a library to evaluate model performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IRENE_DLL: a library to evaluate model performance Presented by: Gianni Fila Research Institute for Industrial Crops Agronomy Section g.fila@isci.it

  2. Topics • Background • Overview • Using the library: basic concepts • Special procedures • Application examples • Future developments

  3. 1. Background

  4. Why IRENE_DLL? • Need of reliable model estimates • No standard theory on model evaluation • No standard “boxes of tools” • Plethora of philosophical theories, statistical techniques, and software practices IRENE_DLL:a set of tools all housed in a single, integrated component

  5. What IRENE_DLL can do • Difference-based analysis: indices, test statistics • Regression-based analysis: parameters, test statistics • Patterns detection: “Pattern Index” • Probability distributions: density functions, cumulative distributions (exceedence, non-exceedence) • Aggregation: first level (“module”), second level (“indicator”) • “Shift” analysis (Time mismatch)

  6. 2. Overview

  7. IRENE_DLL in a nutshell • IRENE_DLL is a library of methods and functions to compute a variety of statistics and statistical tests • It consists of ten classes, containing data services, mathematical routines and some special data analysis procedures • All IRENE_DLL classes can be accessed individually (no hierarchy)

  8. Data objects Computing objects Aggregation objects Accessory objects Module DataSelection GeneralRoutines Indicator Pattern DataSelections DistributionFunctions RegressionObject IndexObject Tests Overview Info-display routines They store data to be processed, and expose properties to handle them. Each object holds a group of related functions. They contain methods to perform statistics aggregation

  9. 3. Using the DLL: basic concepts

  10. DataSelection estimated measured (independent) E External data M Handling data (1) • To be accepted by any functions, data (estimated and measured) must be loaded into a DataSelection object

  11. E E E DataSelection 1 DataSelections • DataSelection 1 • DataSelection 2 • DataSelection 3 M M M External data External data External data DataSelection 2 DataSelection 3 Handling data (2) • Whenever multiple series of data are to be processed, it is convenient to use a collective DataSelections object:

  12. DataSelection Computing object Function_1 Function_2 (….) Function_n Required Specs Outputs Handling functions • To use a particular function, you must call it from the parent object, then pass it the data (in the DataSelection format), and the necessary specifications: Outputs from functions are aggregated in a single, collective variable

  13. Paired_Rows Paired_Columns Unpaired_Average Unpaired_One_to_One Function modes

  14. Handling outputs • All functions in IRENE_DLL return a package of outputs • A special collective variable is designed for each type of function • Example: content of the Index_Variable:

  15. Example: computing regression • Compute regression parameters for three arrays of estimated data against three corresponding arrays of measured data Array MyEstimated(1 to 365, 1 to 3) Array MyMeasured(1 to 365, 1 to 3)

  16. Example: (1) load data • Start an instance of a DataSelection object, and transfer your data inside it through the Estimated and Measured properties Dim Nitrates As New DataSelection Loop through: Nitrates.Estimated(i, j) = MyEstimated(i, j) Nitrates.Measured(i, j) = MyMeasured(i, j)

  17. Example: (2) Compute Regression • Start an instance of the RegressionObject, then call the function Regression_LS (least squares method) Dim RegrCalculator As New RegressionObject Dim RegrOutput As Regression_Variable RegrOutput = RegrCalculator.Regression_LS(Nitrates, Paired_Columns, Measured_Variable)

  18. Example: (3) Display results • Outputs are returned as arrays of values Loop from j = 1 to 3 (the number of columns) Print RegrOutput.Intercept(j)       Print RegrOutput.Intercept_StandError(j)    Print RegrOutput.Intercept_Tvs0(j) Print RegrOutput.Intercept_Prob_Tvs0(j)        Print RegrOutput.Slope(j)   Print RegrOutput.Slope_StandError(j) Print RegrOutput.Slope_Tvs0(j)                 Print RegrOutput.Slope_Tvs1(j) Print RegrOutput.Slope_Prob_Tvs0(j) Print RegrOutput.Slope_Prob_Tvs1(j)           Print RegrOutput.F(j) Print RegrOutput.Prob_F(j)

  19. 4. Special procedures

  20. Aggregation of indices: the problem • The need to define synthetic measures of model performance is a major topic of interest in the field of model evaluation. • Giving a solid judgement is often complicated by the need to balance, for instance, departure of estimates from measurements, modelling efficiency, correlation measures, presence/absence of systematic behavior in the residuals, etc. • The user may want to combine all such aspects in only one aggregated index.

  21. IRENE’s approach • Index/test aggregation is set up by IRENE, based on an expert system using decision rules, according to fuzzy logic • This technique is robust when uncertain data are used (e.g. subjective judgements) and allows an aggregation of dissimilar statistics in a consistent and reproducible way.

  22. Aggregation levels hyerarchy • IRENE_DLL supports two levels of statistics aggregation: First level: multiple indices/tests aggregated in one single index (Module) Second level: Multiple modules aggregated in one single index (Indicator)

  23. Module specifications • Aggregation of statistics require the user to introduce specifications derived from his/her expert knowledge, and objectives • For each index to aggregate, the user must specify: • The DataSelection • The Favourable and Unfavourable limit values • The relative weight

  24. r_Pearson, DataSelection, 0.95, 0.90, 1 Add_Index: Module_Value Pattern, DataSelection, 0.2, 0.75, 0.5 Add_Index: RMSE, DataSelection, 0.1, 0.8, 1 Add_Index: How to build a Module Start an instance of the Module object • Add indices to aggregate, with the required specifications Finally, compute module value calling the function Module_Value MyModule (a score between 0 and 1) RMSE r_Pearson Pattern

  25. Soil NH4 (mg kg-1) Time mismatch analysis • There are no specific indices to investigate the uncertainty about possible displacements (delay or acceleration) registered in time series. A time mismatch may be detected by an iterated procedure, “shifting” repeatedly the estimated points until optimal model performance values are found.

  26. Finding a time mismatch • Starting from an observed model performance as initial condition, the procedure runs as follows: • the simulated points are moved backward in time to the maximum anticipation chosen to evaluate the time mismatch • the desired evaluation indices are computed; • the simulated points of one time history are moved of one selected time step forward; • points 2 to 3 are reiterated until the maximum allowed time delay is reached. • The time mismatch is identified by the time step (forward or backward) at which the best value of the evaluation indices are reached.

  27. Handling time mismatch in IRENE_DLL • All functions exposed by IRENE_DLL have an optional ShiftN parameter: • Ex., in the IndexObject object: • Public Function RMSE(Sel As DataSelection, Mode As Mode, [ByVal ShiftN As Long = 0], [ByVal Prob_level As Double = 0.05]) As Index_Variable • By setting a ShiftN value <> 0, the index is computed shifting the estimated points. • We can explore a range of ShiftN values by a loop

  28. 5. Application examples

  29. IRENE_DLL, sample applications • Time mismatch analysis Difference-based indices Pattern Index

  30. 6. Future developments

  31. In the next future: • Integration in IRENE interface • Integration in development frameworks (MODCOM) • Introduction of robust statistics (median-based). • Introduction of randomization procedures (“bootstrap”) • (….)

  32. How to get IRENE_DLL View documentation

  33. The people of IRENE_DLL • Gianni Fila g.fila@isci.it • Research Institute for Industrial Crops • Gianni Bellocchi g.bellocchi@isci.it • Research Institute for Industrial Crops • Marcello Donatelli m.donatelli@isci.it • Research Institute for Industrial Crops • Marco Acutis macutis@ideagate.it • Department of Crop Science, University of Milan

  34. Thank you!

More Related