1 / 24

Ministère de l’Education Nationale, de l’Enseigneme nt Supérieur et de la Recherche

Ministère de l’Education Nationale, de l’Enseigneme nt Supérieur et de la Recherche Evaluation for Human Interaction Technologies Joseph Mariani Director « Information & Communication Technologies » Department French Ministry of Research. Evaluation in HIT.

yvon
Download Presentation

Ministère de l’Education Nationale, de l’Enseigneme nt Supérieur et de la Recherche

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ministère de l’Education Nationale, de l’Enseignement Supérieur et de la Recherche Evaluation for Human Interaction Technologies Joseph Mariani Director « Information & Communication Technologies » Department French Ministry of Research

  2. Evaluation in HIT • Support to LT in France : Techno-Langue (2002) • Support to Computer Vision : Techno-Vision (2004) • Cooperative support to LT in Europe : Lang-Net CELCT Inauguration

  3. Support to LT: Techno-Langue • Report to the Prime Minister (November 2000) • Meeting Ministries of Industry, Research, Culture: June 2001 • TechnoLangue Action • Language technology survey and evaluation • Basic Technological Research (RTB) • Articulate with related existing programs • Technological Research & Innovation Networks (RRIT) • Support Public research + Industry cooperative projects • 3 ICT RRIT: Telecommunications, Software Engineering, Audiovisual & Multimedia (already funded by 3 ministries) • Ministry of Research action on Business Intelligence Tools (VSE) CELCT Inauguration

  4. Basic Technological Research Usage Evaluation Meeting points with technology development Quantitative Evaluation Basic Scientific Research Basic Technological Research (RTB) Application Development (RRIT) Technologies necessitated for applications Bottleneck Identification Research results in quantitative evaluation Technologies which have been validated for applications. Long term / high risk Large return of investment Usability Acceptability Evolutionary CELCT Inauguration

  5. Techno-Langue infrastructure Infrastructural program to support innovative core technologies development, while existing R&D application projects stay with RRIT & VSE (120 M€ / year) TECHNOLANGUE RNRT RNTL RIAM VSE CELCT Inauguration

  6. Courtesy NIST CELCT Inauguration

  7. Techno-Langue Call • Language resources • Spoken/written data (corpus, dictionaries, terminological data…) • Basic Language Processing Tools (Open Source software) • Evaluation • Technology (evaluation campaigns: comparison of systems) • Applications (evaluation toolkits) • Norms & standards • Shared effort to improve French participation • Technological survey • In relationship with on-going actions (Euromap...) CELCT Inauguration

  8. Techno-Langue Call • Steering Committee: • 14 participants – Chair : C. FLUHR (CEA) • Representatives from 3 RRIT + LT experts • 50% industry / 50% public research • Spoken / Written language • Calendar: • Call For Proposals : April 2002, Deadline : June 2002 • Final selection : September 2002 • Contracts notification (Min. Research): November 2002 • International cooperation • Foreign entities may participate in the projects • Should bring their own funding CELCT Inauguration

  9. Results • 52 proposals submitted • 28 projects selected • 94 participants • 33 industry • 39 public research • 11 other (Associations, CEA, French DoD…) • 11 foreign (Bell Labs (USA), NII (Japan), EPFL, LATL…) • Budget: 20 M€ effort- 7.2 M€ public funding (3 years) • Final report Fall 2005 • Public presentation (conference) • Presentation to administration agencies (DoD, MAE…) CELCT Inauguration

  10. Selected Projects • 28 selected projects: • 11 on Language resources (spoken/written) • BLARK (Cf BNC), Fr-En, G, Sp, It, Arabic dictionaries • Specialized (aerospace, automotive…), proper names dictionaries • Aligned corpus (7 novels 19th century litterature in 4 languages) • Children voices and generic voices • 4 on Tools (Open Source software) • Lemmatizer, Chunker, Guesser, Tagger, Parser, Speech & Speaker recognizers, Topic & NE detector, summarizer, term. extractor, Search engine... • 2 on Standards (spoken / written) – support to ISO TC37-SC4 • 1 on Technological survey (Portal) http://www.technolangue.net • 10 on Evaluation : 9 on technology eval., 1 on usage eval. CELCT Inauguration

  11. LT Evaluation • Written language • Syntactic parsing • Text alignment • Terminology extraction • Information query • Machine translation • Spoken Language • Speech synthesis • Speaker recognition • Spoken dialog • Speech transcription / automatic indexing CELCT Inauguration

  12. Support to CV: Techno-Vision • Involves two ministries • Ministry of Defense / DGA • Ministry of Research / Technology Directorate • Basic Technological Research (RTB) • Dual technologies research • Link with RRIT (RNTL, RNRT, RIAM) • Objective : implement a permanent evaluation infrastructure for Computer Vision • Financing evaluation campaigns • Distributing resources and results http://www.technologie.gouv.fr/appel/technovision.html CELCT Inauguration

  13. Techno-Vision Call • Steering Committee: • 24 participants – Chair : Ph.. BOLON (Univ. Savoie) • Representatives from 3 RRIT + CV experts • 10% industry / 60% public research / 30% governmental agencies • All areas of computer vision • Calendar: • Call For Proposals : June 2004, Deadline : August 2004 • Final selection : September 2004 • Contracts notification (Min. Research): December 2004 • International cooperation • Foreign entities may participate in the projects • Should bring their own funding CELCT Inauguration

  14. Results • 16 proposals submitted • 10 projects selected, 50 tasks • 110 participants • 25 companies • 10 foreign (DAG Barcelona, Un. Hong-Kong, Multitel, Un. Reading, Un. Kingston, HWU, ISI, IDIAP, Regim, Un. Illinois…) • Budget: 5.1 M€ effort - 2.1 M€ public funding • 2 year duration • Distribution of development and test data at cost CELCT Inauguration

  15. Selected Projects • ETISEO : video-monitoring • IMAGEVAL : image indexing • ARGOS : Text & people recognition in TV broadcast and video • IV2 : biometry (iris and face recognition) • MESSIDOR : medical imaging (retina ophtalmology) • EVALECHOCARD : medical imaging (echocardiograms) • RIMES : written documents analysis • EPEIRES : symbols in technical documents • ROBIN : object detection in aerospace and terrestrial imaging • TOPVISION : underwater imaging CELCT Inauguration

  16. FP6 and the ERA • 6th Framework Program (2002-2006) • Usual projects • New instruments • Networks of Excellence • Integrated Projects • Construction of the European Research Area (ERA) • 90% of European R&D is conducted within Member States • 10% within European Commission programs • Article 169 (1 proposal / Infectious diseases (3*200 = 600 M€)) • ERA-Net projects (may prepare Article 169…) CELCT Inauguration

  17. ERA-Net • Cooperation & coordination of national or regional research and innovation activities (i.e. programs) • Exchange of information and best practices (projects, priorities, evaluation process, management…) • Strategic activities (complementarities, multinational scheme, barriers identification) • Implementation of joint activities (clustering, joint evaluation, use of testbeds) • Transnational research activities (joint programs, calls, funding...) CELCT Inauguration

  18. ERA-Net • 148 M€ total FP6 budget (2002-2006) • EC funds the coordination costs, not the R&D • First deadline : June 2003. Next: March 2005 • Eligibility: Legal entities from at least 3 MS / AS • Public bodies financing/managing research at national/regional level • Other organizations doing the same (charities) • European bodies (EEIG…(may be sole as a group of public bodies)) • Coordinating Actions (CA) • 3 M€ max. over 5 years max. - 100% costs (incl. subcontracts) • Specific Support Action (SSA) • Preparation of CA: 200 K€ max. over 1 year max. CELCT Inauguration

  19. Language Technologies Program • Language as a specific issue for Europe • Economical, cultural and political challenge • Preserve the EU Member States cultures • Preference for native language (Germany (75%)...) • Allow for communication across member states • 15 countries / 11 languages / 110 language pairs • 1170 translators at the EC - 1.3 Mpages translated in 2001 • 30% European Parliament budget (300 M€) – 500 translators • EU enlargement : 10 more countries since may 1st, 2004 • 20 languages / 380 language pairs • Enormous cost for the EC : even more enormous • Need for the assistance of Language Technologies • Huge effort (# LT * # languages), too large for the EC alone • Should be shared with EU member states (subsidiarity) CELCT Inauguration

  20. Language Technologies Program • LT well fitted with European Research Area (ERA) • The EC supports • the coordination: management, standards, technology evaluation, communication... • the development cost of generic language technologies: • Speech recognition, synthesis, understanding, spoken dialog, language tagging, parsing, analysis, generation, text retrieval, document understanding, machine translation... • Each member state supports the cost for covering its language(s): • Language Resources (essential) : (annnotated) corpus (spoken / written), lexicon (including pronunciations), dictionaries… • Language specific technology development/adaptation CELCT Inauguration

  21. Language Technologies Program France - LR in French - Technology adaptation to French The Netherlands - LR in Dutch - Technology adaptation to Dutch European Commission - Coordination - Standards / sharability - Generic technology development - Technology evaluation Germany - LR in german - Technology adaptation to German Italy - LR in Italian - Technology adaptation to Italian Denmark - LR in Danish - Technology adaptation to Danish Norway - LR in Norwegian - Technology adaptation to Norwegian ………... CELCT Inauguration

  22. Lang-Net • Infrastructural nature • Language Resources, LT evaluation, Standards, Survey • Identify EU countries or regions having similar programs • 11 countries / regions in partnership : France, Germany, Italy, Trento region, Czech Republic, Denmark, Norway, The Netherlands / Belgium-Flanders (Dutch Language Union), Spain, Basque region, Sweden • Austria, Finland, Greece, Iceland, Portugal, Switzerland, UK (contacts) • Extendable to other partners • NMS (Slovenia, Cyprus, Poland, Hungary, Malta, Baltic countries…) • AS (Romania, Bulgaria…) • USA, Japan, South Africa, Israel, Canada… (contacts) • Proposal to be finalized before early march 2005 • Extension in the future to a larger program • including EC FP6 projects (Cf TC-Star, CHIL…), Article 169 on LT ? CELCT Inauguration

  23. Conclusion • A great opportunity & a grand challenge for Europe ! CELCT Inauguration

  24. Conclusion • and for CELCT ! CELCT Inauguration

More Related