Project ELSE Evaluation in Language and Speech Engineering +-------------------------+ | E L S E - E V A L | LIMSI: PP +-------------------------+ EPFL/DI-LIA: RM+SF Read Me (0.1) Novembre 1999 1. Document(s) Map 1.1 Documentation /Liste des sources d'informations pour la distrib (les docs: que contiennent-elles en gros, ou les obtenir (site web, man des commandes)/ 1.2 Table of Contents /.../ 2. Introduction 2.1 Else Abstract This prospective action aims at preparing a general infrastructure for Language Engineering evaluation in the context of future R&D programmes. ELSE will develop and test a general infrastructure for a task-independent semi-automatic protocol framework for quantitative black-box evaluation of Natural Language Processing systems in a multilingual environment. The protocol will be embodied in a generic blue print that will specify the various components of any instantiation of the protocol for a given kind of natural language processing application. Specific characteristics of systems relevant for particular text and speech processing applications will be reflected in the evaluation policy guidelines. The infrastructure will include the general organization of any evaluation program based upon semi-automatic quantitative black-box evaluation: o) a generic strategy for communication with data providers, customers and participants as well as publication of results, o) methodologies and technical solutions used for test data collection, data formatting and data distribution, for computation, validation and publication of results, and also for valorization of evaluation by-products, o) localisation of the generic aspects of the infrastructure - i.e. what changes and what does not across language boundaries. 2.2 About this document The main goal of this document is to provide the necessary information to understand, install and use the 'ElseEval' environment for the quantitative black-box evaluation of tagging systems. /public visé. plateformes supportées, et config. nécessaires. (actuellement: SunOs 5.6 avec gcc 2.8.1 (+ libstdc++ 2.8.1.1), linux 2.2.12 avec egcs 2.91 (libstc++ inclu) (il faut que je complete -test/adapte- un peu) Maintenance: a qui envoyer les 'bugs-report' et sous quelle forme'./ 2.3 History & Revision This project was started on /.... court blabla sur TagEval .../ This document was started on October 1999, by the LIA /.../ /Vivement les révisions à la version (0.)1 ... / 2.4 Copyrights & license /Else+Grace copyrights/ /Specification de ce qui est couvert par cette license (TagEval) Qqch du genre GPL (?)/ Permission to use, copy, modify, and distribute this software and its documentation under the terms of the /ElseCopyrights|limitations/ is /hereby granted (?) - dire aussi qqch a propos de la limitation de la license, qui ne s'applique pas aux données traitées par le logiciel/. THIS DOCUMENT AND THE SOFTWARE COVERED BY IT ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. THEY COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. /pour les versions alpha-beta/ CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THE DOCUMENT. IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT DESCRIBED IN THIS DOCUMENT COULD BE MAKE AT ANY TIME. 2.5 Acknowledgements and Thanks Thanks go to: o TagEval staff: - Raphaël Rossi, EPFL, DI-LIA (main programmer); - Martin Rajman, EPFL, DI-LIA (project coordinator); - Jean-Cédric Chappelier, EPFL, DI-LIA (additional programmer); - Patrick Paroubek, LIMSI (project coordinator) o GraceEval staff: - Seydoux Florian, EPFL, DI-LIA (main programmer); - Martin Rajman, EPFL, DI-LIA (project supervisor); - Patrick Paroubek, LIMSI (project coordinator & additional programmer) o DiET staff at DFKI; o Gate staff at the Sheffield University; / il faudrait aussi placer, peut-etre ici, l'info sur les outils 'ad hoc' : doxygene (si utilisé)/ 3. Package description 3.1 Structure /description de la structure (ou est quoi - sources, code binaire, données, description des outils (repris du 'man' associé)/ 3.2 Documentation /Ou est la doc (specifique aux modules), et sur quoi porte-elle/ 4. Installation 4.1 Requirements a) Hardware requirements /disk space for compil, for install time to compil with a specific machine (c.f. plateformes actuellement supportées) Others rq (ram -> peut-etre important pour Linux/PC distrib) / b) Software requirements /pour chacun, version mimimum, version 'testée', adresse ou se procurer l'outil: - compilateurs : gcc/egcs 2.8.0 -> 2.95, ... - librairies : stdlibc++ (et lib. C), ... - aligneur-rq : gnu diff, ... Additional software needed: - auto-install : gnu make, ... - doc extractor : doxygene, ... (pour reconstituer la doc, apres modif du code) 4.2 Compilation /En gros: (g)make all; eventuellement compilation spécifique pour les participants, 'experts' et 'évaluateurs'. -> il faudrait 'splitter' la distrib. Aussi info pour compilation 'manuelle' (sans make) Suivant ce que l'on arrive a faire en 'dépoussierage' du code>/ ElseEval should compile without errors or warnings /(hum!)/. If it does not, please send the compilation errors or warnings along wiht a description of your plateform to /... c.f. bugs-report/ / 4.3 Test /(g)make testall ou (g)make checkall; si distrib 'splitée' -> test spécifique pour chaque portion. -> lance automatiquement les outils, sur des donnees connues, et verifie la conformité avec les resultats de references... si pas pareil (crc check p.ex.), diff et bugs-report/ 4.4 Install /(g)make install TARGET=.../ The following binaries should now be available in ...: o) checker: /brief/ o) unifier: /breif/ o) tokenizer: /brief/ /.../ 5. Specific Informations / Presentation des modules, ou plutot lien vers la doc specifique a chacun d'eux (dvi/ps,html,man generee a partir des sources)/