Identification and Neural Networks

1. Identification and Neural Networks G. Horváth I S R G Department of Measurement and Information Systems NIMIA Crema, Italy

2. Identification and Neural Networks Part III Industrial application http://www.mit.bme.hu/~horvath/nimia NIMIA Crema, Italy

3. Overview • Introduction • Modeling approaches • Building neural models • Data base construction • Model selection • Modular approach • Hybrid approach • Information system • Experiences with the advisory system • Conclusions NIMIA Crema, Italy

4. Introduction to the problem • Task • to develop an advisory system for operation of a Linz-Donawitz steel converter • to propose component composition • to support the factory staff in supervising the steel-making process • A model of the process is required NIMIA Crema, Italy

5. LD Converter modeling NIMIA Crema, Italy

6. Linz-Donawitz converter Phases of steelmaking • 1. Filling of waste iron • 2. Filling of pig iron • 3. Blasting with pure oxygen • 4. Supplement additives • 5. Sampling for quality testing • 6. Tapping of steel and slag NIMIA Crema, Italy

7. Linz-Donawitz converter Phases of steelmaking • 1. Filling of waste iron • 2. Filling of pig iron • 3. Blasting with pure oxygen • 4. Supplement additives • 5. Sampling for quality testing • 6. Tapping of steel and slag NIMIA Crema, Italy

8. Linz-Donawitz converter Phases of steelmaking • 1. Filling of waste iron • 2. Filling of pig iron • 3. Blasting with pure oxygen • 4. Supplement additives • 5. Sampling for quality testing • 6. Tapping of steel and slag NIMIA Crema, Italy

9. Linz-Donawitz converter Phases of steelmaking • 1. Filling of waste iron • 2. Filling of pig iron • 3. Blasting with pure oxygen • 4. Supplement additives • 5. Sampling for quality testing • 6. Tapping of steel and slag NIMIA Crema, Italy

10. Linz-Donawitz converter Phases of steelmaking • 1. Filling of waste iron • 2. Filling of pig iron • 3. Blasting with pure oxygen • 4. Supplement additives • 5. Sampling for quality testing • 6. Tapping of steel and slag NIMIA Crema, Italy

11. Linz-Donawitz converter Phases of steelmaking • 1. Filling of waste iron • 2. Filling of pig iron • 3. Blasting with pure oxygen • 4. Supplement additives • 5. Sampling for quality testing • 6. Tapping of steel and slag NIMIA Crema, Italy

12. Main parameters of the process • Nonlinear input-output relation between many inputs and two outputs • input parameters (~50 different parameters) • certain features “measured” during the process • The main output parameters • temperature (1640-1700 CO -10 … +15 CO) • carbon content (0.03 - 0.70 % ) • More than 5000 records of data NIMIA Crema, Italy

13. Modeling task • The difficulties of model building • High complexity nonlinear input-output relationship • No (or unsatisfactory) physical insight • Relatively few measurement data • There are unmeasurable parameters • Noisy, imprecise, unreliable data • Classical approach (heat balance, mass balance) gives no acceptable results NIMIA Crema, Italy

14. Modeling approaches • Theoretical model - based on chemical, physical equations • Input - output behavioral model • Neural model - based on the measured process data • Rule based system - based on the experimental knowledge of the factory staff • Combined neural - rule based system NIMIA Crema, Italy

15. oxygen temperature System + components (parameters) e S Neural - Model predicted temperature model output components (parameters) temperature Copy of Inverse Model Model measured + predicted temperature oxygen e S - The modeling task NIMIA Crema, Italy

16. „Neural” solution • The steps of solving a practical problem NIMIA Crema, Italy

17. Building neural models • Creating a reliable database • the problem of noisy data • the problem of missing data • the problem of uneven data distribution • Selecting a proper neural architecture • static network • dynamic network • regressor selection • Training and validating the model NIMIA Crema, Italy

18. Creating a reliable database • Input components • measure of importance • physical insight • sensitivity analysis • principal components • Normalization • input normalization • output normalization • Missing data • artificially generated data • Noisy data • preprocessing, filtering NIMIA Crema, Italy

19. Initial database New database Neural network training Sensitivity analysis Input parameter cancellation Input parameter of small effect on the output? yes no Building database • Selecting input components, dimension reduction NIMIA Crema, Italy

20. Building database • Dimension reduction: mathematical methods • PCA • Non-linear PCA • ICA • Combined methods NIMIA Crema, Italy

21. x 2 y 2 y 1 x 1 Data compression, PCA networks • Principal component analysis (Karhunen-Loeve transformation NIMIA Crema, Italy

22. Oja network • Linear feed-forward network NIMIA Crema, Italy

23. Oja network • Learning rule • Normalized Hebbian learning NIMIA Crema, Italy

24. Oja subspace network • Multi-output extension NIMIA Crema, Italy

25. GHA, Sanger network • Multi-output extension Oja rule + Gram-Schmidt orthogonalization NIMIA Crema, Italy

26. x 2 x 1 y 1 Nonlinear data compression • Nonlinear principal components NIMIA Crema, Italy

27. Independent component analysis • A method of finding a transformation where the transformed components are statistically independent • Applies higher order statistics • Based on the different definitions of statistical independence The typical task • Can be implemented using neural architecture NIMIA Crema, Italy

28. Normalizing Data • Typical data distributions NIMIA Crema, Italy

29. Normalization • Zero mean, unit standard deviation • Normalization into [0,1] • Decorrelation + normalization NIMIA Crema, Italy

30. Whitened Original Normalization • Decorrelation + normalization = Whitening transformation NIMIA Crema, Italy

31. Missing or few data • Filling in the missing values • Artificially generated data • using trends • using correlation • using realistic transformations NIMIA Crema, Italy

32. Few data • Artificial data generation • using realistic transformations • using sensitivity values: data generation around various working points (a good example: ALVINN) NIMIA Crema, Italy

33. Noisy data • EIV • input and output noise are taken into consideration • modified criterion function • SVM • e-insensitive criterion function • Inherent noise suppression • classical neural nets have noise suppression property (inherent regularization) • averaging (modular approach) NIMIA Crema, Italy

34. [ i ] n p , k * * y x k k + System [ i ] [ i ] n n + m , y , k + m , x , k [ ] i [ i ] y x k k Errors in variables (EIV) • Handling of noisy data NIMIA Crema, Italy

35. EIV • LS vs EIV criterion function • EIV training NIMIA Crema, Italy

36. EIV • Example NIMIA Crema, Italy

37. EIV • Example NIMIA Crema, Italy

38. SVM • Why SVM? • „Classical” Neural Networks • (MLP) • -„Overfitting” Support Vector Machine (SVM) +Better generalization (upper bounds) +Selects the more important input samples +Handles noise +~Automatic structure and parameter selection • Model • Structure • Parameter Selection difficulties NIMIA Crema, Italy

39. SVM • Special problem of SVM • selecting hyperparameters •  insensitive • RBF type SVM: , C • slow „training”, complex computations • SVM-Light • Smaller, reduced teaching set • difficulty of real-time adaptation NIMIA Crema, Italy

40. Selecting the optimal parameters C=1, =0.05, σ=0.9 C=1, =0.05, σ=1.9 NIMIA Crema, Italy

41. Selecting the optimal parameters Sigma NIMIA Crema, Italy

42. Selecting the optimal parameters Mean square error Sigma NIMIA Crema, Italy

43. 1.4 1.2 EIV-SVM comparison f(x)=sin(x)/x Training points 1 Support vectors Training result of the SVM 0.8 Training result with EIV Training result with MLP 0.6 0.4 0.2 0 -0.2 -0.4 -10 -8 -6 -4 -2 0 2 4 6 8 10 Comparison of SVM, EIV and NN NIMIA Crema, Italy

44. Model selection • Static or dynamic • Dynamic model class • regressor selection • basis function selection • Size of the network • number of layers • number of hidden neurons • model order NIMIA Crema, Italy

45. 12 18 11 16 10 14 9 8 12 7 10 6 8 5 4 6 1 2 3 4 5 6 7 8 9 0 5 10 15 20 Model selection • NARX model, NOE model • Lipschitz number, Lipschitz quotient NIMIA Crema, Italy

46. Model selection • Lipschitz quotient general nonlinear input-output relation, f(.) continuous, smooth multivariable function bounded derivatives Lipschitz quotient Sensitivity analysis NIMIA Crema, Italy

47. Model selection • Lipschitz number for optimal n NIMIA Crema, Italy

48. Modular solution • Ensemble of networks • linear combination of networks • Mixture of experts • using the same paradigm (e.g. neural networks) • using different paradigms (e.g. neural networks + symbolic systems) • Hybrid solution • expert systems • neural networks • physical (mathematical) models NIMIA Crema, Italy

49. Cooperative networks Ensemble of cooperating networks (classification/regression) • The motivation • Heuristic explanation • Different experts together can solve a problem better • Complementary knowledge • Mathematical justification • Accurate and diverse modules NIMIA Crema, Italy

50. Ensemble of networks • Mathematical justification • Ensemble output • Ambiguity (diversity) • Individual error • Ensemble error • Constraint NIMIA Crema, Italy