CLEF

This report describes the work done by the QA group of the Language Technology Lab at DFKI, for the 2004 edition of the Cross-Language Evaluation Forum (CLEF). Based on the experience we obtained through our participation at QA@Clef-2003 with our initial cross-lingual QA prototype system BiQue (cf. [1]), the focus of the system extension for this year’s task was a) on robust NL question interpretation using advanced linguistic-based components, b) flexible interface strategies to IR-search engines, and c) on strategies for off-line annotation of the data collection, which support query-specific indexing and answer selection. The overall architecture of the extended system, as well as the results obtained in the CLEF–2004 Monolingual German and Bilingual German/English QA tracks will be presented and discussed throughout the paper.

This report describes our tests on applying statistical translation models for bilingual IR tasks in CLEF-2001. These translation models have been trained on a set of parallel web pages automatically mined from the Web. Our previous studies have shown the utility of such corpora for cross-language information retrieval. The goal of the current tests is to see how we can improve the quality of the translation models and make best uses of them. Several questions are considered: Is it useful to consider the IDF factor in addition to the translation probabilities? Is it useful to further clean the training corpora before model training or the translation models themselves? How could we combine the translation models with bilingual dictionaries? Although our tests do not allow us to answer all these questions, they provide useful indication to several further research directions.

This paper describes the work done by Priberam in the development of a question answering (QA) system for Portuguese. The system was built using the company’s natural language processing (NLP) workbench and information retrieval technology. Special focus is given to question analysis, document and sentence retrieval, as well as answer extraction stages. The paper discusses the system’s performance in the context of the QA@CLEF 2005 evaluation.

Question Answering is a major research topic at the University of Alicante. For this reason, this year two groups participated in the QA@CLEF track using different approaches. In this paper we describe the work of Alicante 2 group. This paper describes AliQAn, a monolingual open-domain Question Answering (QA) System developed in the Department of Language Processing and Information Systems at the University of Alicante for CLEF-2005 Spanish monolingual QA evaluation task. Our approach is based fundamentally on the use of syntactic pattern recognition in order to identify possible answers. Besides this, Word Sense Disambiguation (WSD) is applied to improve the system. The results achieved (overall accuracy of 33%) are shown and discussed in the paper.

This year, besides providing support to other groups participating in cross-language Question Answering (QA) tasks, and submitting runs both for the monolingual Italian and for the cross-language Italian/English tasks, the ITC-irst participation in the CLEF campaign concentrated on the Answer Validation Exercise (AVE). The participation in the AVE task, with an answer validation module based on textual entailment recognition, is motivated by our objectives of (i) creating a modular framework for an entailment-based approach to QA, and (ii) developing, in compliance with this framework, a stand-alone component for answer validation which implements different approaches to the problem.

The CEA-LIST/LIC2M develops both cross-language information retrieval systems and content-based image retrieval systems. The ad hoc and medical tasks of the ImageCLEF campaign offered us the opportunity to perform some experiments on merging the results of the two systems. The results obtained show that the performance of each system highly depends on the corpus and the task: feedback strategies can improve the results, but the parameters used are to be tuned according to the confidence of each system on the task and corpus: for the ad hoc task, text retrieval performs good whereas results of image retrieval are poor. On the other hand, for the medical task, the image retrieval performs better, and text retrieval can improve overall results only with reinforcement strategies.

This paper presents the ITC-irst Multilingual Question Answering system Diogene. The system was used successfully on the CLEF-2003, TREC-2003, TREC-2002 and TREC-2001 QA tracks. Diogene relies on a classical three-layer architecture: question processing, document retrieval, answer extraction and validation. Diogene uses MultiWordNet [8] (http:// multiwordnet.itc.it) which facilitates the transfer of knowledge between languages. For answer validation we used the Web. This year we also used a set of linguistic templates for answering specific questions like definition questions, location questions, and a subset of who-is and what-is questions. Diogene participated in both the monolingual Italian-Italian task and in the cross-language Italian-English task. We also collaborated with the Bulgarian Academy of Sciences in the cross-language Bulgarian-English QA task.

This paper describes the first set of experiments defined by the MIRACLE (Multilingual Information RetrievAl for the CLEf campaign) research group for some of the cross language tasks defined by CLEF. These experiments combine different basic techniques, linguistic-oriented and statistic-oriented, to be applied to the indexing and retrieval processes.

This paper presents the extension of the ITC-irst Diogene Question Answering system towards multilinguality. Diogene relies on a well tested three-components architecture built in the framework of our participation in the QA track at the Text Retrieval Conference (TREC 2002). The novelty factors are represented by the enhancement of the system with language-specific tools targeted to the Italian language (e.g. a module in charge of the answer-type extraction, and a named entities recognizer) and the introduction of a module for the translation of Italian queries into English queries. The overall architecture of the extended system, as well as the results obtained in the CLEF 2003 Monolingual Italian and Bilingual Italian/English QA tracks will be presented and discussed throughout the paper.

LogAnswer is a logic-oriented question answering system developed by the AI research group at the University of Koblenz-Landau and by the IICS at the University of Hagen. The system addresses two notorious problems of the logic-based approach: Achieving robustness and acceptable response times. Its main innovation is the use of logic for simultaneously extracting answer bindings and validating the corresponding answers. In this way the inefficiency of the classical answer extraction/answer validation pipeline is avoided. The prototype of the system, which can be tested on the web, demonstrates response times suitable for real-time querying. Robustness to gaps in the background knowledge and errors of linguistic analysis is achieved by combining the optimized deductive subsystem with shallow techniques by machine learning.

This paper proposes two kinds of relevance measures to rank documents by geographic restriction: scope-based and type-based. The non-geographic and geographic relevance scores are combined using a weighted harmonic mean. The proposed relevance measures and weighting schemes are evaluated on GeoCLEF 2007 dataset with encouraging performance over the standard IR performance. The best performance is achieved when the importance of non-geographic relevance scores outweigh the importance of geographic relevance scores.