|
|
|
|
|
Speech & Language Evaluation related links
CLEF Information Retrieval System Evaluation for European Languages. It is the CLIR track of TREC that moved to Europe. ELSE (Evaluation in Language and Speech Engineering) studied the possible implementation of large scale deployment of comparative evaluation in Europe. DISC Developped a set of best-practice guidelines for the development and evaluation of Spoken Language Dialog Systems (SLDSs). In particular the project proposes an evaluation template for guiding the evaluation of a SLDS as well as recommendations for the evaluation of the speech synthesis component. DiET (Diagnostic and Evaluation Tool) was aimed at supporting the evaluation of Natural Language Processing (NLP) applications by producing tools and data for machine translation and computer aided translation of text documents (cf. the DiET flyer and the slides of the 1997 Diet evaluation workshop). It was a follow-up of the TSNLP project (Test Suites for Natural Language Processing). INUSE goal was to define an evaluation methodology of the ergonomy of information systems produced by the European Industry. For more information, see the project description (in french!). MEGATAQ objective was to provide Telematics Applications Projects with user-centred evaluation guidelines and consultancy. TEMAA has developed, from EAGLES results, a framework for the evaluation of LE products. The framework has been formalised and implemented into a workbench whose function is to guide an evaluator in designing and carrying out specific evaluation tasks customized according to different user profiles. A detailed methodology has been worked out for the evaluation of spelling and grammar checkers, and applied to the concrete evaluation of Danish and Italian spelling checkers. EAGLES NLP Evaluation Working Group designed a generic framework for user oriented evaluation of NLP systems, based on feature checklists and applied it to a few examples (Writers Aids, Translators Aids and Knowledge Management Systems). EAGLES final report on: Evaluation of Natural Language Processing Systems is available at: ILC site or ISSCO site, and the material of workshops as well as related evaluation material is available at CST site. FraCaS aimed at bringing about a convergence of current efforts in computational semantics. During the project, a test set of 350 units (sentences, question, answer) to illustrate english semantic phenomena was built.
GRACE was a CNRS project for the evaluation of Part-Of-Speech Tagging of French texts to which initially 21 teams participated. The final results are available in French as well as the evaluation toolkit developped by the project. The Agence Universitaire de la Francophonie and its FRANCIL network organized in collaboration with CNRS a series of evaluation campaign for text and speech analysis applied to French: the ARCs (in French Action de Recherche Concertées ). The topics addressed were: information retrieval, bilingual corpus alignment, automatic terminology extraction, text understanding, speech recognition, spoken language dialog systems, speech synthesis. VERBMOBIL was a German project on real-time speech to speech translation (between German, English and Japanese) during which various components of a spoken language dialog system were evaluated (speech recognition, parsing, language understanding).
In the scope of the TIPSTER text program on Information Retrieval, Information Extraction and Summarization. DARPA organized in collaboration with NIST and ARDA a series of evaluation campaigns on:
On the other hand, the NIST spoken language technology evaluations focused on the following topics:
At SIGIR '98 Panel: Tools for Searching the Web evaluation WEB search tools
 A
lot more soon in this pages...
[CLASS main site home page] [CLASS Evaluation subsite home page] [LIMSI home page] [Last updated: Thu Apr 5th 2001] |