110 likes | 311 Views
Sandhan. Indian language search engine. Sandhan – Consortium Project. IIT Bombay (co- ordinator ) CDAC Noida (co- cordinator ) CDAC Pune IIT Kharaghpur Jadhavpur University ISI Kolkata IIIT Hyderabad AU KBC AU CEG Gauhati University DAIICT Gujarat IIIT Bhubaneswar TDIL.
E N D
Sandhan Indian language search engine
Sandhan – Consortium Project • IIT Bombay (co-ordinator) • CDAC Noida (co-cordinator) • CDAC Pune • IIT Kharaghpur • Jadhavpur University • ISI Kolkata • IIIT Hyderabad • AU KBC • AU CEG • Gauhati University • DAIICT Gujarat • IIIT Bhubaneswar • TDIL
Introduction • Cross Lingual Information Retrieval (CLIR) engine for Indian languages • Input: Query in one of the six Indian languages (Hindi, Marathi, Tamil, Telugu, Bengali, Punjabi, Assamese. Gujarati, Oriya) • Output:In Hindi, English and Query Language • Currently in the second phase of the project • Three new languages are added in second phase • Assamese, Gujarati, Oriya • Built on top of Nutch Framework
Software Used • Nutch v0.9 – Framework • Hadoop – Distributed Crawling • Lucene – Indexing • Moses/GIZA++ - Training models • Tomcat – Deployment
Web Font Transcoder Information Extraction Analyzer Fetcher Language Identifier Snippet Translation NE Lookup CMLifier Domain Identifier Summary Generation MWE Lookup Indexer NE Lookup Snippet Generation Translation /Transliteration MWE Lookup Query Formulation Index UNL Index Analyzer
Resources Developed • Language specific analyzers • Stop word List • Bilingual Dictionary ( X-English, X-Hindi) • NE List • MWE List • Transliteration Models
IIT Bombay Participation • Marathi Vertical • Code Integration and Maintenance • MWE Identification • Development of Tracker • Error Analysis • Relevance Judgement
Action Plan • Public release of 5 languages monolingual search engine on April 14th 2012 • Bengali, Hindi, Marathi, Tamil, Telugu • Public Release of remaining 4 languages monolingual search and 5 languages cross lingual search August 15th 2012 • Assamese, Gujarati, Oriya, Punjabi (Monolingual) • Bengali, Hindi, Marathi, Tamil, Telugu (Cross lingual)
Key Achievements • Organized Forum for Information Retrieval (FIRE) 2008, 2010 and 2011 -a workshop for CLIR evaluation for Indian Languages • Demonstrated a basic integrated version of the system at IJCNLP 2008 and ELITEX 2008. Media coverage by ‘The Indian Express’ news paper and ‘Hindustan Times’ (http://www.cfilt.iitb.ac.in/pb_1.JPG) (http://www.cfilt.iitb.ac.in/04_04_2009_010_007.jpg) • Development of a strong and connected research community around CLIR in Indian languages. • Publications in top IR and NLP forum