OVERVIEW OF THE
CLEF 2007 MULTILINGUAL QUESTION ANSWERING TRACK
4
Danilo Giampiccolo1, Pamela Forner1, Anselmo Peñas2, Christelle Ayache3, Dan Cristea
, Valentin Jijkoun5, Petya Osenova6, Paulo Rocha7, Bogdan Sacaleanu8, and Richard Sutcliffe9
1
2
CELCT, Trento, Italy ({giampiccolo, forner}@celct.it)
Departamento de Lenguajes y Sistemas Informáticos, UNED, Madrid, Spain (
[email protected])
3
4
ELDA/ELRA, Paris, France (
[email protected])
Faculty of Computer Science, University “Al. I. Cuza” of Ia i, Romania Institute for Computer Science,
Romanian Academy, Ia i, Romania (
[email protected])
5
Informatics Institute, University of Amsterdam, The Netherlands (
[email protected])
6
BTB, Bulgaria, (
[email protected])
7
Linguateca, SINTEF ICT, Norway and Portugal, (
[email protected])
8
DFKI, Germany, (
[email protected])
9
DLTG, University of Limerick, Ireland (
[email protected])
Abstract.
The fifth QA campaign at CLEF, the first having been held in 2006. was characterized by continuity
with the past and at the same time by innovation. In fact, topics were introduced, under which a number of
Question-Answer pairs could be grouped in clusters, containing also co-references between them. Moreover, the
systems were given the possibility to search for answers in Wikipedia. In addition to the main task, two other
tasks were offered, namely the Answer Validation Exercise (AVE), which continued last year’s successful pilot,
and QUAST, aimed at evaluating the task of Question Answering in Speech Transcription. The Question
Answering Real Time was also proposed again, after the success obtained in 2006.
As general remark, it must be said that the task proved to be more difficult than expected, as in
comparison with last year’s results the Best Overall Accuracy dropped from 49% to 41,75% in the multi-lingual
subtasks, and, more significantly, from 68% to 54% in the monolingual subtasks..
1
Introduction
The fifth QA campaign at CLEF [1], the first having been held in 2003, was characterized by continuity with the
past, maintaining the focus on cross-linguality and covering as many European languages as possible (with the
addition of Indonesian); and by innovation 1) by introducing a number of Question-Answer pairs, grouped in
clusters, which referred to a same topic and which contained co-references between them, and 2) by giving the
possibility to search for answers in Wikipedia. In this way, the newcomers had the possibility to test themselves
with the classic task, and those who had participated in the previous campaigns had a new challenging factor to
test their systems. In addition to the main task, three other tasks were offered, namely the Answer Validation
Exercise (AVE), which continued last year’s successful pilot, Question Answering for Speech Transcripts
(QAST), aimed at evaluating the task of Question Answering in Speech Transcription, and the Question
Answering Real Time, which made its successful debut in Alicante, in 2006.
In the following sections, the main task and its preparation will be described. A presentation of the participants
and the runs submitted will be also given, together with a description of the evaluation method and the results
achieved.
2
Tasks
Following the procedure consolidated in previous years, in the 2007 campaign several different tasks were
proposed:
1.
2.
a main task, divided into several monolingual and bi-lingual sub-tasks;
the Answer Validation Exercise (AVE), which continued the successful experiment proposed in 2006.
Systems were required to emulate human assessment of QA responses and decide whether an Answer to
3.
4.
a Question is correct or not according to a given Text. Participating systems were given a set of triplets
(Question, Answer, Supporting Text) and they had to return a boolean value for each triplet. Results
were evaluated against the QA human assessments [1];
the QA Answering on Speech Transcripts (QAST), a pilot task which aimed at providing a framework in
which factual. Relevant points of this pilot were:
a. Comparing the performances of the systems dealing with both types of transcriptions.
b. Measuring the loss of each system due to the state of the art ASR technology.
c. In general, motivating and driving the design of novel and robust factual QA architectures for
automatic speech transcriptions [2].
Question Answering in Real-Time (QART), aimed at evaluating the ability of QA systems to answer
within a time constraint. This year two tasks were planned: one exercise via web and another exercise to
be carried out in the Budapest workshop [3].
The AVE and QAST tasks are described in details in dedicated papers in this Working Notes.
As far as the main task is concerned, the consolidated procedure was followed, although some relevant
innovations were introduced.
The systems were given a set of 200 questions -which could concern facts or events (F-actoid questions),
definitions of people, things or organisations (D-efinition questions), or lists of people, objects or data (L-ist
questions)- and were asked to return one exact answer, where exact meant that neither more nor less than the
information required was given. Following the example of TREC, this year the exercise consisted of topicrelated questions, i.e. clusters of questions which were related to the same topic and possibly contained coreferences between one question and the others. Neither the question types (F, D, L) or the topics were given to
the participants.
The answer needed to be supported by the docid of the document in which the exact answer was found, and by
portion(s) of text, which provided enough context to support the correctness of the exact answer. Supporting
texts could be taken from different sections of the relevant documents, and had to sum up to a maximum of 700
bytes. There were no particular restrictions on the length of an answer-string, but unnecessary pieces of
information were penalized, since the answer was marked as ineXact. As in previous years, the exact answer
could be exactly copied and pasted from the document, even if it was grammatically incorrect (e.g.: inflectional
case did not match the one required by the question). Anyway, this year systems were also allowed to use NL
generation in order to correct morpho-syntactical inconsistencies (e.g., in German, changing "dem Presidenten"
into "der President" if the question implies that the answer is in Nominative case), and to introduce grammatical
and lexical changes (e.g., QUESTION: What nationality is X? TEXT: X is from the Netherlands => EXACT
ANSWER: Dutch).
Table 1: Tasks activated in 2007 (in green)
TARGET LANGUAGES (corpus and answers)
BG
BG
SOURCE LANGUAGES (questions)
DE
EN
ES
FR
IN
IT
NL
PT
RO
DE
EN
ES
FR
IT
NL
PT
RO
The subtasks were both:
•
•
monolingual, where the language of the question (Source language) and the language of the news
collection (Target language) were the same;
cross-lingual, where the questions were formulated in a language different from that of the news
collection.
Ten source languages were considered, namely, Bulgarian, Dutch , English, French, German, Indonesian, Italian,
Portuguese, Romanian and Spanish. All these languages were also considered as target languages, except for
Indonesian, which had no news collections available for the queries and, as was done in the previous campaigns,
used the English question set translated into Indonesian (IN).
As shown in Table 1,37 tasks were proposed:
• 8 Monolingual -i.e. Bulgarian (BG), German (DE), Spanish (ES), French (FR), Italian (IT), Dutch (NL),
Portuguese (PT) and Romanian (RO;
• 29 Cross-lingual.
Anyway, as Table 2 shows, not all the proposed tasks were then carried out by the participants.
Table 2: Tasks chosen by at least 1 participant in QA@CLEF campaigns.
MONOLINGUAL
CROSS-LINGUAL
CLEF 2004
6
13
CLEF 2005
8
15
CLEF 2006
7
17
CLEF 2007
7
11
As customary in recent campaigns, a monolingual English (EN) task was not available as it seems to have been
already thoroughly investigated in TREC campaigns. English was still both source and target language in the
cross-language tasks.
As the format is concerned, this year both input and output files were formatted as an XML file (for more details
see [4]).
3
Test Set Preparation
The procedure followed to prepare the test set was much different from that used in the previous campaigns.
First at all, each organizing group, responsible for a target language, freely chose a number of topics. For each
topic, one to four questions were generated. Topics could be not only named entities or events, but also other
categories such as objects, natural phenomena, etc. (e.g. George W. Bush; Olympic Games; notebooks;
hurricanes; etc.). The set of ordered questions were related to the topic as follows:
The topic was named either in the first question or in the first answer
The following questions can contain co-references to the topic expressed in the first question/answer
pair.
Topics were not given in the test set, but could be inferred from the first question/answer pair. For example, if
the topic was George W. Bush, the cluster of questions related to it could have been:
Q1: Who is George W. Bush?
Q2: When was he born?
Q3: Who is his wife?
The Table 3: Document collections used in CLEF 2007.
TARGET LANG..
Bulgarian (BG)
COLLECTION
Sega
Standart
Frankfurter Rundschau
Germany (DE)
Der Spiegel
German SDA
German SDA
English (EN)
Spanish (ES)
Los Angeles Times
Glasgow Herald
EFE
EFE
French (FR)
Italian (IT)
Le Monde
Le Monde
French SDA
French SDA
La Stampa
Itallian SDA
Itallian SDA
Dutch (NL)
NRC Handelsblad
Algemeen Dagblad
Portuguese (PT)
Público
Público
Folha de São Paulo
Folha de São Paulo
PERIOD
2002
SIZE
120 MB (33,356 docs)
2002
93 MB (35,839 docs)
1994
320 MB (139,715 docs)
1994/1995
63 MB (13,979 docs)
1994
144 MB (71,677 docs)
1995
141 MB (69,438 docs)
1994
1995
1994
425 MB (113,005 docs)
154 MB (56,472 docs)
509 MB (215,738 docs)
1995
577 MB (238,307 docs)
1994
1995
1994
1995
1994
157 MB (44,013 docs)
156 MB (47,646 docs)
86 MB (43,178 docs)
88 MB (42,615 docs)
193 MB (58,051 docs)
1994
85 MB (50,527 docs)
1995
85 MB (50,527 docs)
1994/1995
299 MB (84,121 docs)
1994/1995
241 MB (106,483 docs)
1994
1995
1994
1995
164 MB (51,751 docs)
176 MB (55,070 docs)
108 MB (51,875 docs)
116 MB (52,038 docs)
The questions in the set were numbered from 1 to 200, with no indication about whether they were part of a
cluster belonging to the same topic.
Another major innovation of this year’s campaign concerned the corpora at which the questions were aimed at.
In fact, beside the data collections composed of news articles provided by ELRA/ELDA, also Wikipedia was
considered, capitalizing on the experience of the WiQA pilot task proposed in 2006. The Wikipedia pages in the
target languages, as found in the version of the Wikipedia of November, 2006 could be used. XML and the
HTML versions were available for download, even though any other versions of the Wikipedia files could be
used as long as they dated back to the end of November / beginning of December 2006. All the answers to the
questions had to be taken from "actual entries" or articles of Wikipedia pages - the ones whose filenames
normally correspond to the topic of the article. Other types of data (“image”, “discussion”, “category”,
“template”, “revision histories”, any files with user information, and any “meta-information” pages), had to be
excluded.
As far as the question types are concerned, as in previous years of QA@CLEF, the three following categories
were still considered:
a) Factoid questions, fact-based questions, asking for the name of a person, a location, the extent of something,
the day on which something happened, etc.
We consider the following 8 answer types for factoids:
PERSON, e.g.
Q: Who was called the “Iron-Chancellor”?
A: Otto von Bismarck.
Q: What year was Martin Luther King murdered?
A: 1968.
LOCATION, e.g. Q: Which town was Wolfgang Amadeus Mozart born in?
A: Salzburg.
ORGANIZATION, e.g. Q: What party does Tony Blair belong to?
A: Labour Party.
MEASURE, e.g. Q: How high is Kanchenjunga?
A: 8598m.
COUNT, e.g.
Q: How many people died during the Terror of Pol Pot?
A: 1 million.
OBJECT, e.g. Q: What does magma consist of?
A: Molten rock.
OTHER, i.e. everything that does not fit into the other categories above.
Q: Which treaty was signed in 1979?
A: Israel-Egyptian peace treaty.
TIME, e.g.
b) Definition questions, questions such as "What/Who is X?", and are divided into the following subtypes:
PERSON, i.e. questions asking for the role/job/important information about someone,
Q: Who is Robert Altmann?
A: Film maker.
ORGANIZATION, i.e. questions asking for the mission/full name/important information about an
organization, e.g.
Q: What is the Knesset?
A: Parliament of Israel.
OBJECT, i.e. questions asking for the description/function of objects, e.g.
Q: What is Atlantis?
A: Space Shuttle.
OTHER, i.e. question asking for the description of natural phenomena, technologies, legal procedures
etc., e.g.
Q: What is Eurovision?
A: Song contest.
c) closed list questions: i.e. questions that require one answer containing a determined number of items, e.g:
Q: Name all the airports in London, England.
A: Gatwick, Stansted, Heathrow, Luton and City.
As only one answer was allowed, all the items had to be presented in sequence, one next to the other, in one
document of the target collections.
Table 4: Test set breakdown according to question type
BG
DE
EN
ES
FR
IT
NL
PT
RO
F
158
164
161
148
148
147
147
143
160
D
32
28
30
42
42
41
40
47
30
L
10
8
9
10
10
12
13
9
10
T
12
27
3
40
40
38
30
23
52
NIL
0
0
0
21
20
20
20
18
7
Besides, all types of questions could contain a temporal restriction, i.e. a temporal specification that provided
important information for the retrieval of the correct answer, for example:
Q: Who was the Chancellor of Germany from 1974 to 1982?
A: Helmut Schmidt.
Q: Which book was published by George Orwell in 1945?
A: Animal Farm.
Q: Which organization did Shimon Perez chair after Isaac Rabin’s death?
A: Labour Party Central Committee.
Some questions could have no answer in the document collection, and in that case the exact answer was "NIL"
and the answer and support docid fields were left empty. A question is assumed to have no right answer when
neither human assessors nor participating systems could find one.
The distribution of the questions among these categories is described in Table 4.
Each of the question sets was finally then translated into English, so that each group could translate another set
into their own language, when preparing the cross-lingual data sets which had been activated.
4
Participants
After years of constant growth, the number of participants has decreased in 2007 [see Table 5]..
Table 5: Number of participating groups
CLEF 2003
CLEF 2004
CLEF 2005
CLEF 2006
CLEF 2007
America Europe Asia Australia
3
5
1
17
1
22
1
4
24
2 3
17
1
1
TOTAL
8
18
24
30
22
The geographical distribution has anyway remained almost the same, recording a new entry of a group from
Australia. No participants took part to any Bulgarian tasks.
Table 6. Number of submitted runs
Number of
submitted runs
#
Monolingual
Cross-lingual
CLEF 2003
17
6
11
CLEF 2004
48
20
28
CLEF 2005
67
43
24
CLEF 2006
77
42
35
CLEF 2007
22
23
14
Also the number of submitted runs has decreased sensibly, from a total of 77 registered last year to 22 (see table
6). As in previous campaigns, a larger number of people chose to participate in the monolingual tasks, which
once again demonstrated to be more approachable.
5
Evaluation
No changes were made as far the evaluation process is concerned- Human judges assessed the exact answer (i.e.
the shortest string of words which is supposed to provide the exact amount of information to answer the
question) as:
•
•
•
•
R (Right) if correct;
W (Wrong) if incorrect;
X (ineXact) if contained less or more information than that required by the query;
U (Unsupported) if either the docid was missing or wrong, or the supporting snippet did not contain the
exact answer.
Most assessor-groups managed to guarantee a second judgment of all the runs. As regards the evaluation
measures the following measures:
•
•
accuracy, as the main evaluation score, defined as the average of SCORE(q) over all 200 questions q;
the K1 measure[6]:
score(r ) • eval (r )
K 1(sys ) =
r∈answers ( sys )
# questions
K 1(sys ) ∈ IR ∧ K 1(sys ) ∈ [− 1,1]
where:
score (r) is the confidence score assigned by the system to the answer r and eval(r) depends on the
judgment given by the human assessor.
eval (r )
•
6
=
{
1 if
( r ) is
− 1 in other
judged
as correct
cases
K1(sys) = 0 is established as a baseline.
the Confident Weighted Score (CWS), designed for systems that give only one answer per question.
Answers are in a decreasing order of confidence and CWS rewards systems that give correct answers at
the top of the ranking [2] .
Results
As far as accuracy is concerned, scores were generally far lower this year than usual, as Figure 1 shows. In
detail, Best accuracy in the monolingual task decreased by almost 15 points, passing from last year’s 68.95% to
54%, while Best accuracy in cross-language tasks passed from 49.47% to 41.75% recording.
As far as average performances are concerned, this year a neat decrease has been recorded in the biligual tasks,
which went from 22.8% to 10.9%. This was due also due to the presence of systems which participated for the
first time, achieving very low score in tasks which are quite difficult also for veterans.
As a general remark, it can be said that the new factors introduced this year appear to have had an impact on the
performances of the systems. As more than one participant has noticed, there has been not enough time to adjust
the systems to the new requirements.
Here below a more detailed analyses of the results in each language follows, giving more specific information on
the performances of systems in the single sub-tasks and on the different types of questions, providing the
relevant statistics and comments.
80
70
60
50
40
30
20
10
0
Best
Average
CLEF03
CLEF04
CLEF05
CLEF06
CLEF07
Figure 1: Best and average scores in CLEF QA campaigns
6.1 Dutch as Target
For the Dutch subtask of the CLEF 2007 QA task, three annotators generated 200 questions organized in 78
groups so that there were 16 groups with one question, 21 groups with two, 22 with three and 19 groups with
four questions. Among the 200 questions 156 were factoids, 28 definitions and 16 list questions. In total, 41
questions had temporal restrictions. Table XXX below shows the distributions of topic types for groups and
expected answer types for questions.
Table 6: Distribution of topic types and expected answer for questions.
Topic type
OBJECT
PERSON
ORGANIZATION
LOCATION
EVENT
Number of topics
29
18
12
10
19
Expected answer
type
OTHER
PERSON
TIME
OBJECT
LOCATION
COUNT
ORGANIZATION
MEASURE
Number of
questions
45
38
32
25
25
14
13
8
Annotators were asked to create questions with answers either in Dutch Wikipedia or in the Dutch newspaper
corpus, as well as questions without known answers. Of 200 questions, 186 had answers in Wikipedia, and 14 in
the newspaper corpus. Annotators did not create NIL questions.
Table 7: Results
X
U
%F
%T
%D
%L
#
#
#
#
[156]
[41]
[28]
[16]
1
11
10
8
7
23
9.0
24.4
25.6
10.3
10.9
4.9
19.5
19.5
14.6
14.6
3.6
35.7
35.7
32.1
32.1
NIL
uams071qrz
gron071NLNL
gron072NLNL
gron071ENNL
gron072ENNL
15
49
51
26
27
160
136
135
159
161
4
4
7
5
%
[0]
0
#
0
6.3
6.3
6.3
6.3
0
20
20
20
16
Overall
accuracy
W
CWS
R
Run
0.02
0
0
0
0
0.06
0.07
0.02
0.02
7.54
24.5
25.5
13
13.5
This year, two teams took part in the QA track with Dutch as the target language: the University of Amsterdam
and the University of Groningen. The latter submitted both monolingual and crosslingual (English to Dutch)
runs. The 5 submitted runs were assessed independently by 3 Dutch native speakers in such a way that each
question group was assessed by at least two assessors. In case of conflicting assessments, assessors were asked
to discuss the judgements and come to an agreement. Most of the occured conflicts were due to difficulties in
distinguishing between inexact and correct answers. Table 7 below shows the evaluation results for the five
submitted runs (three monolingual and two cross-lingual). The table shows the number of Right, Wrong, ineXact
and Unsupported answers, as well as the percentage of correctly answered Factoids, Temporally restricted
questions, Definition and List questions.
The best monolingual run (gron072NLNL) achieved accuracy of 25.5%, which is slightly less that the best
results in the 2006 edition of the QA task. The same tendency holds for the performance on factoid and
definition questions. We interpret this an indication of the increased difficulty of the task due the newly
introduced Wikipedia collection.
One of the runs contained as many as 23 unsupported answers—this might indicate a bug in the system.
6.1 English as Target
Creation of questions. This year the questions set were radically different from last year. Instead of 200
independent questions, we were required to devise questions in groups. Each group had a declared topic (e.g.
"Polygraph") but unlike in TREC, this topic was not communicated to the participants. As at CLEF last year, the
type of question (e.g. definition, factoid or list) was not declared to participants either.
Table 8: Results
%
CWS
K1
Overall
accurac
y
0
[0]
0.00
0.00
0.00
13.00
11.11
0
0.00
0.00
0.00
13.00
0.00
0
0.00
0.00
0.00
10.00
23.33
0.00
0
0.00
0.00
0.00
7.00
6.67
0.00
0
0.00
0.00
0.00
2:50
0.00
0.00
0.00
0
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0
0.00
0.00
0.00
0.00
0.00
43.33
0.00
0
0.00
0.00
0.00
14.00
R
W
X
U
%F
%T
%D
%L
NIL
#
#
#
#
[161]
[3]
[30]
[9]
#
cind071fren
26
171
1
2
11.18
0.00
23.33
11.11
cind072fren
26
170
2
2
11.18
0.00
23.33
csui071inen
20
175
4
1
10.56
0.00
10.00
dfki071deen
14
178
6
2
4.35
0.00
dfki071esen
5
189
4
2
1.86
0.00
mqaf071nlen
0
200
0
0
0.00
mqaf072nlen
0
200
0
0
0.00
wolv071roen
28
166
2
4
9.32
Run
160 Factoids (in groups) were requested, together with 30 definitions and ten lists. The numbers of temporally
restricted factoids and questions with NIL answers was at our discretion. In the end we submitted 161 factoids,
30 definitions and nine lists. In previous years we have been obliged to devise a considerable number of
temporally restricted questions and this has proved very difficult to do with the majority of them being very
contrived and artificial. For this reason it was intended to set no such questions this year. However, one
reasonable one was spotted during the data entry process and so was flagged as such. Two others were also
flagged accidentally during data entry. Unfortunately, therefore, the statistics can not tell us anything about
temporally restricted questions.
Concerning NIL questions, we have long argued that they tell us very little about the performance of a system
unless it can report the reason why there is no answer. For example, this is a useful system:
Q: Who is the Queen of France?
A: France is a Republic!
By contrast, answering NIL would not tell us whether there was an answer which was simply not found, or
whether no answer in fact exists. Another important point following from this is that NIL questions artificially
boost the performance of a system which returns many NIL answers. For these reasons we decided not to include
any questions with NIL anwers. However, we would like to see ‘Queen of France’ answers being returned in
future workshops.
The grouped nature of the questions had a considerable effect on their difficulty; instead of a series of ‘trivia’
type questions, each with a simple, clear answer, a single topic was effectively investigated in much more detail.
To achieve the goals set by the organisers it was necessary to find topics about which several questions could be
asked and then to devise as many questions as possible from that topic. Each task was surprisingly hard, and an
inevitable consequence was that the questions are much harder this year than in previous years. We had no wish
to set especially difficult or convoluted questions, but unfortunately this arose as a side-effect of the new
procedures.
The requirement for related questions on a topic necessarily implies that the questions will refer to common
concepts and entities within the domain in question. In a series of questions this is accomplished by co-reference
– a well known phenomenon within Natural Language Processing which nevertheless has not been a major
factor in the success of QA systems at previous CLEF workshops. The most common form is anaphoric
reference to the topic declared implicitly in the first question, e.g.:
Q: What is a Polygraph?
Q: When was it invented?
However, other forms of co-reference occurred in the questions. Here is an example:
Q: Who wrote the song "Dancing Queen"?
Q: How many people were in the group?
Here the group refers to the category of entity into which the answer to the first question is known by the
questioner to belong. However, the QA system does not know this and has to infer it, a task which can be very
complex and indirect, especially where the topic is concealed from the participants.
In addition to the issue of question grouping, it was decided at a very late stage to use not only the two
collections from last year (the LA Times and Glasgow Herald) but also the English Wikipedia. The latter is
extremely large and greatly increases the task complexity for the participants in terms of both indexing and IR
searching. In addition, some questions had to be heavily qualified in order to reduce the ambiguity introduced by
alternative readings in the Wikipedia. Here is an example:
Q: What is the "KORG" on which Niky Orellana is a soccer commentator?
Thirdly, we should bear in mind that the Wikipedia varies considerably in size depending on the language, with
the English one being by far the largest. We have not controlled for this fact in CLEF and the consequence could
be that the addition of Wikipedia had a greater effect on difficulty for English than it did for other languages.
Summary Statistics. Eight cross-lingual runs with English as target were submitted this year, as compared with
thirteeen for last year. Five groups participated in six source languages, Dutch, French, German, Indonesian,
Romanian and Spanish. DFKI submitted runs for two source languages, German and Spanish, while all other
groups worked in only one. Cindi Group and Macquarie University both submitted two runs for a language pair
(French-English and Dutch-English respectively) but unfortunately there was no language for which more than
one group submitted a run. This means that no direct comparisons can be made between QA systems this year,
because the task being solved by each was different.
Assessment Procedure. An XML format was used for the submission of runs this year, by constrast with
previous years when fairly similar plain text formats were adopted. This meant that our evaluation tools were no
longer usable. However, last year we also participated in the evaluation of the WiQA task organised by
University of Amsterdam. For this they developed an excellent web-based tool which was subsequently adapted
for this year’s Dutch CLEF evaluations. We are extremely grateful to Martin de Rijke and Valentin Jijkoun for
allowing us to use it and for setting it up in Amsterdam especially for us. It allows multiple assessors to work
independently, shows runs anonymised, allows all answers to a particular question to be judged at the same time
(like the TREC software), and includes the supporting snippets for each submitted answer as well as the ‘correct’
(reference) answer. It also shows inter-assessor disagreement, and, once this has been eliminated, can produce
the assessed runs in the correct XML format. Overall, this software worked perfectly for us and saved us a
considerable amount of time.
All answers were double-judged. Where assessors differed, the case was discussed between us and a decision
taken. We measured the agreement level by two methods. For Agreement 1 we take agreement on each group of
8 answers to a question as a whole as either exactly the same for both assessors or not exactly the same. This is a
very strict measure. There were disagreements for 30 questions out of the 200, i.e. 15%, which equates to an
agreement level of 85%.
For Agreement Level 2 we taking each decision made on one of the eight answers to a question and count how
many decisions were the same for both assessors and how many were not the same. There were 39 differences of
decision and a total of 1600 decisions (200 questions by eight runs). This is 2.4%, which equates to an agreement
level of 97.6%. This is the measure we used in previous years. Last year the agreement level was 89% and the
previous year it was 93%. We conclude from these figures that the assessment of our CLEF runs is quite
accurate and that double judging is sufficient.
Results Analysis. As in previous years there were three types of question within the question groups, Factoids,
Definitions and Lists. Considering all question types together, the best performance is University of
Wolverhampton with 28 R and 2 X, (14% strict or 15% lenient) closely followed by the CINDI Group at
Concordia University with 26 R and 1 X (13% strict or 13.50% lenient). Note that these systems are working on
different tasks (RO-EN and FR-EN respectively) as noted above, so the results are not directly comparable. The
best performance last year for English targets was 25.26%. Nevertheless, considering the extreme difficulty of
the questions, this represents a remarkable achievement for these systems.
For Factoids alone, the best system was CINDI (FR-EN) at 11.18% followed by University of Indonesia (INEN) with 10.56%. For Definitions the best result was University of Wolverhampton (RO-EN) with 43.33%
correct, followed equally by CINDI (FR-EN) and DFKI (DE-EN) both with 23.33%. It is interesting that this
year the best Definition score is almost four times the best Factoid score, whereas last year they were nearly
equal. One reason for this may be that the definitions either occurred first in a group of questions or on their own
in a ‘singleton’ group. This was not specifically intended but seems to be a consequence of the relationship
between Factoids and Definitions, namely that the latter are somehow epistemologically prior to the former1. In
consequence, Definitions may be more simply phrased than Factoids and in particular may avoid co-reference in
the vast majority of cases.
Nine lists questions were set but only CINDI was able to answer any of them correctly (11.11% accuracy).
(University of Indonesia was ineXact on one list question.) Perhaps the problem here was recognising the list
question in the first place – unlike at TREC they are not explicitly flagged. We believe this is not necessarily
reasonable since in a real dialogue a questioner would surely make it quite clear whether they expected a list of
answers or just one. They would not come up with a list question out of the blue.
6.3 French as Target
Details can be found in the online version of these Working Notes.
1
Perhaps it is just a consequence of setting too many undergraduate examination papers!
6.4 German as Target
Two research groups submitted runs for evaluation in the track having German as target language: The German
Research Center for Artificial Intelligence (DFKI) and the Fern Universität Hagen (FUHA). Both provided
system runs for the monolingual scenario and just DFKI submitted runs for the cross-language English-German
and Portuguese-German scenario. The assessment was conducted by two native German speakers with fair
knowledge of information access systems. Compared to the previous editions of the evaluation forum, this year a
decrease in the accuracy of the best performing system and of an aggregated virtual system for both monolingual
and cross-language tasks was registered.
Table 9: Results through the years.
Year
Best Mono
Aggregated Mono Best Cross
Aggregated Cross
2007
30
45
18.5
18.5
2006
42.33
64.02
32.98
33.86
2005
43.5
58.5
23
28
2004
34.01
43.65
0
0
70
60
50
Best Mono
40
Aggregated Mono
Best Cross
30
Aggregated Cross
20
10
0
2007
2006
2005
2004
Figure 2: Results evolution
The details of systems’ results can be seen in Table 10. There were no NIL questions tested in this year’s
evaluation. The results submitted by DFKI did not provide a normalized value for the confidence score of an
answer and therefore both CWS and K1 values could not be computed.
Table 10. System Performance – Details
U
%F
%T
%D
%L
NIL
#
#
#
#
[164]
[27]
[28]
[8]
#
%
[0]
60
48
30
37
10
121
146
164
144
180
14
4
4
18
10
5
2
2
1
0
29.8
24.39
17.07
17.68
3.66
14.81
18.52
14.81
14.81
7.41
39.29
28.57
7.14
25
14.29
0
0
0
12,5
0
0
0
0
0
0
0
0
0
0
0
Run
dfki071dedeM
fuha071dedeM
fuha072dedeM
dfki071endeC
dfki071ptdeC
Overall
accuracy
X
K1
W
CWS
R
0
0.086
0.048
0
0
0
-0.17
-0.31
0
0
30
24
15
18.5
5
The number of topics covered by the questions was of 116 distributed as it follows: 69 topics consisting of 1
question, 19 topics of 2 related questions, and each 19 topics of 3 and 4 related questions. The most frequents
topic types were PERSON (40), OBJECT (33) and ORGANIZATION (23). As regards the source of the
answers, 101 questions from 68 topics asked for information out of the CLEF document collection and the rest
of 99 from 48 topics for information from Wikipedia. The distribution of the topics over the document
collections (CLEF vs. Wikipedia) is as follows: 53 vs. 16 topics of 1 question, 4 vs. 15 topics of each 2 and 3
questions and 7 vs. 2 topics of 4 questions.
Table 11: Inter-Assessor Agreement/Disagreement (breakdown)
# Q-Disagreements
F
D
L X U
Run ID
# Questions
dfki071dedeM
200
20
16
4
0
15
4
1
fuha071dedeM
200
13
10
3
0
7
3
3
fuha072dedeM
200
7
6
1
0
2
2
3
dfki071endeC
200
13
7
5
1
12
1
0
dfki071ptdeC
200
8
3
5
0
8
0
0
Total
W/R
Table 12 describes the inter-rater disagreement on the assessment of answers in terms of question and answer
disagreement. Question disagreement reflects the number of questions on which the assessors delivered
different judgments. Along the total figures for the disagreement, a breakdown at the question type level
(Factoid, Definition, List) and at the assessment value level (ineXact, Unsupported, Wrong/Right) is listed. The
answer disagreements of type Wrong/Right are trivial errors during the assessment process when a right answers
was considered wrong by mistake and the other way around, while those of type X or U reflect different
judgments whereby an assessor considered an answer inexact or unsupported while the other marked it as right
or wrong.
6.5 Italian as Target
Only one group took part in this year to the monolingual Italian task, i.e. FBK-irst, achieving the following
results:
Table 12: Results.
NIL
4
%F
13
15.17
%T
%D
12.5
2.63
% L]
0
14
3
0.0165
-0.0429
Overall
accuracy
160
U
K1
23
X
CWS
W
Correct
irst071ITIT
R
Returned
Run
11,55
More details on Italian as a target can be found in the online version of this working notes.
6.6 Portuguese as Target
Six research groups took part in tasks with Portuguese as target language, submitting eigth runs: seven in the
monolingual task, and one with English as source; unlike last year, no group presented Spanish as source. One
new group (INESC) participated this. The group of University of Évora (UE) returned this year, while the group
from NILC, the sole Brazilian group to take part to date, was absent.
Again, Priberam presented the best result for the third year in a row; the group of the University of Évora wasn’t
however far behind. As last year, we added the classification X-, meaning incomplete, while keeping the
classification X+ for answers with extra text or other kinds of inexactness. In Table 3 we present the overall
results.
Table 13: Results of the runs with Portuguese as target: all 200 questions
Run Name
NIL Accuracy
R
(#)
W
(#)
X+
(#)
X(#)
U
(#)
Overall
Accuracy
(%)
84
103
1
11
1
42.0
11.7
92.3
16
178
0
4
2
8.0
6.3
69.2
12
184
0
2
2
6.0
6.1
84.6
40
158
1
1
0
20.0
8.3
84.6
22
171
1
4
2
11.0
7.3
69.2
Precision Recall
(%)
(%)
diue071ptpt
esfi071ptpt
esfi072ptpt
feup071ptpt
ines071ptpt
ines072ptpt
prib071ptpt
26
168
0
4
2
13.0
7.2
84.6
101
88
5
5
1
50.5
27.8
46.2
lcc_071enpt
56
121
7
3
13
28.0
33.3
23.1
A direct comparison with last year’s results is not fully possible, due to the existance of multiple questions to
each topic. Therefore, 14 presents results regarding the first question of each topic, which we believe is more
readily comparable to the results of previous years.
Table 14: Results of the runs with Portuguese as target: answers to the first question of the 149 topics
R
(#)
W
(#)
X+
(#)
X(#)
U
(#)
Overall
Accuracy
(%)
diue071ptpt
esfi071ptpt
esfi072ptpt
feup071ptpt
ines071ptpt
ines072ptpt
prib071ptpt
61
77
1
9
1
40,9%
11
132
0
4
2
7,4%
6
141
0
1
1
4,0%
34
113
1
1
0
22,8%
17
125
1
4
2
11,4%
21
122
0
4
2
14,1%
92
86
3
5
1
61,7%
lcc_071enpt
44
48
7
3
9
29,5%
Run Name
As it can be seen, the removal of subsequent questions to each topic doesn’t cause a big change on the overal
results, apart from a clear improvement by Priberam. On the whole, compared to last year (Vallin et al., 2007),
Priberam saw a slight drop on its results, Raposa (FEUP) a clear improvement from an admitedly low level,
Esfinge (SINTEF) a clear drop, and LCC kept last year’s levels. Senso (UE) shows a marked improvement since
its last participation in 2005. We leave it to the participants to comment on whether it might have been caused by
harder questions or changes (or lack thereof) in the systems.
Question 94 was reclassified as NIL due to a spelling error, and question 135 because of the use of a word with a
rare meaning. On the other hand, one system saw through that rare meaning, providing a correct answer; we
decided to keep the question as NIL, considering correct both the system’s answer and any NIL answer from
other systems. The same system also found a correct answer to a NIL question, not discovered during the
question creating process; that question was therefore reclassified as non-NIL. In the end, there were 13 NIL
questions.
Table 15 shows the results for each answer type of definition questions, while Table 16 shows the results for
each answer type of factoid questions (including list questions). As it can be seen, four out of six systems
perform clearly better when it comes to definitions than to factoids. This may well have been helped by the use
of Wikipedia texts, where a large proportion of articles begin with a definition.
Table 15: Results of the assessment of the monolingual Portuguese runs: definitions
obj
6
Run
org
6
oth
9
per
9
TOT
30
%
diue071ptpt
6
4
5
4
19
63%
esfi071ptpt
1
0
0
0
1
3%
esfi072ptpt
1
0
0
0
1
3%
feup071ptpt
3
2
4
7
16
53%
ines071ptpt
4
4
6
0
14
47%
ines072ptpt
5
5
6
2
18
60%
prib071ptpt
6
4
6
7
23
77%
6
5
8
9
27
2
3
2
1
8
combination
lcc_071enpt
87%
27%
Table 16: Results of the assessment of the Portuguese runs: factoids, including lists
Run
diue071ptpt
cou
21
loc
31
11
3
2
4
1
2
9
17
3
4
8
3
4
15
combination
16
24
lcc_071enpt
7
11
esfi071ptpt
esfi072ptpt
feup071ptpt
ines071ptpt
ines072ptpt
prib071ptpt
mea
16
obj
5
4
0
0
0
0
0
org
21
oth
per
tim
TOT
26
21
19
160
%
12
8
0
0
1
0
0
14
17
7
1
2
3
2
2
8
12
9
7
2
5
2
2
10
13
65
15
11
24
8
10
78
109
39%
9%
7%
15%
5%
6%
46%
68%
3
10
4
6
48
29%
6
1
1
3
0
0
10
3
0
0
0
0
0
1
11
12
3
6
1
We included in both Table 15 and in Table16 a virtual run, called combination, in which one question is
considered correct if at least one participating system found a valid answer. The objective of this combination
run is to show the potential achievement when combining the capacities of all the participants. The combination
run can be considered, somehow, state-of-the-art in monolingual Portuguese question answering. The system
with best results, Priberam, answered correctly 72.7% the questions with at least one correct answer, not as
dominating as last year. Despite being a bilingual run, LCC answered correctly 14 questions not answered by
any of the monolingual systems.
In Table 17, we present some values concerning answer and snippet size (in number of words).
Table 17: average size of answers
Non-NIL
Answers
(#)
Average
answer
size
Average answer
size
(R only)
Average
snippet
size
diue071ptpt
89
2.8
2.9
25.0
Average
snippet size
(R only)
24.3
esfi071ptpt
57
2.4
2.8
56.3
29.3
esfi072ptpt
19
2.4
2.8
59.7
29.1
feup071ptpt
56
2.7
3.3
59.8
32.9
ines071ptpt
49
3.7
4.8
60.7
33.6
ines072ptpt
47
3.8
5.3
61.7
34.2
prib071ptpt
182
3.5
4.4
49.6
32.4
lcc_071enpt
191
3.4
4.2
45.2
32.7
Run name
Temporally restricted questions: Table 18 presents the results of the 20 temporally restricted questions. As in
previous years, the effectiveness of the systems to answer those questions is visibly lower than for non-TRQ
questions (and indeed several systems only answered correctly question 160, which is a NIL TRQ).
Table 18: accuracy of temporally restricted questions
T.R.Q
correctness (%)
20.0
Non-T.R.Q
correctness (%)
diue071ptpt
Correct answers
(#)
4
44.4
Total
correctness (%)
42.0
esfi071ptpt
1
5.0
8.3
8.0
esfi072ptpt
1
5.0
6.1
6.0
feup071ptpt
1
5.0
21.7
20.0
ines071ptpt
1
5.0
11.7
11.0
ines072ptpt
1
5.0
15.0
14.0
prib071ptpt
8
40.0
51.7
28.0
lcc_071enpt
6
30.0
27.8
50.5
Run name
List questions: a total of twelve questions were defined as list questions; unlike last year, all these questions
were closed list factoids, with two to twelve answers each2. The results were, in general, weak, with UE and
LCC getting two correct answers, Priberam five, and all other system zero. There was a single case of
incomplete answer (i.e., answering some elements of the list only), but it was judged W since, besides
incomplete, it was also unsupported.
6.7 Romanian as Target
At CLEF 2007 Romanian was addressed as a target language for the first time, based on the collection of
Wikipedia Romanian pages frozen in November 2006, and as a source language for the second time, using the
English news collection (Los Angeles Times, 1994 and Glasgow Herald, 1995) and the Wikipedia English
pages.
Creation of Questions. The creation of the questions was realized at the Faculty of Computer Science, Al.I.
Cuza University of Iasi. The group3 was very well instructed with respect to this task, using the Guidelines for
Question Generation and based on a good feedback received from the organizers at IRST4. The final 200 created
questions are distributed according to table 19.
Table 19: Question types distribution in Romanian
FACTOID
DEFINITION
LIST
NIL
QUESTIONS
PERSON
TIME
LOCATION
22
9
10
17
21
7
ORGANIZATION
19
5
MEASURE
COUNT
OBJECT
OTHER
TOTAL
17
20
16
6
21
10
153
30
10
7
Most difficulties in this task were raised by deciding on the supporting snippets, especially for questions
belonging to the same topic. We found unnatural to include answers through “copy-paste” from the text: if the
question requires an answer in the Nominative case, but the text includes the answer in the Genitive case, then
we had to include the Genitive in the answer, even if it is more natural to have the answer in Nominative.
Participants. This year two Romanian groups took part in the monolingual task with Romanian as a target
language: the Faculty of Computer Science from the Al. I. Cuza University of Iasi, and the Research Institute for
2
There were some open list questions as well, but they were classified and evaluated as ordinary factoids.
Three Computational Linguistics Master students: Anca Onofra c, Ana-Maria Rusu, Cristina Despa, supervised
and working in collaboration with the two organizers
4
Without the help received from Danilo Giampicolo and Pamela Forner, we wouldn’t have solved all our
problems.
3
Artificial Intelligence from the Romanian Academy, Bucharest. Three runs were submitted – one by the first
group and two by the second group, with the differences between them due to the way they treated the questionprocessing and the answer-extraction. The 2007 results are presented in Tables 20 below. One system with
Romanian as a source language and English as target was submitted by the Computational Linguistics Group
from the University of Wolverhampton, United Kingdom.
Tables 20: Results in the monolingual task, Romanian as target language
Run
R
W
24
60
60
outputRoRo (1)
ICIA071RORO (2)
ICIA072RORO (3)
X
171
105
101
Overall
Accuracy
U
4
34
39
1
1
0
NIL
RETURNED
12
30
30
NIL
correct
100
54
54
CWS
5
7
7
0.02889
0.09522
0.09522
All three systems “crashed” on the LIST questions. The NIL questions are hard to classify, starting from the
question-classifier (the classifier should “know” that the QA system has no possibility, no knowledge to find the
answer). It would be better to have a clear separation between the NIL answers due to impossibility to find
answer and the NIL answers classified as such by the system. None of the three systems could handle the
questions related under one same topic: the systems returned at most the answer to the first question in a topic.
Assessment Procedure. Due to time restrictions, all three runs where judged by only one assessor at the Faculty
of Computer Science in Iasi, so an inter-annotator agreement was not possible. Based on the Guidelines, all three
systems were judged in parallel. The same evaluation criteria, especially with respect to the UNSUPPORTED
and INEXACT answers, were used.
6.8 Spanish as Target
The participation at the Spanish as Target subtask has decreased from 9 groups in 2006 to 5 groups this year. All
the runs were monolingual. We think that the changes in the task (linked questions and wikipedia) led to a lower
participation and worse overall results because systems could not be tuned on time.Table 21 shows the summary
of systems results with the number of Right (R), Wrong (W), Inexact (X) and Unsupported (U) answers. The
table shows also the accuracy (in percentage) of factoids (F), factoids with temporal restriction (T), definitions
(D) and list questions (L). Best values are marked in bold face. All the runs were assessed by two assessors.
Only a 1.5% of the judgements were different and the resulting kappa value was 0,966, which corresponding to
“almost perfect” assessment [7].
Table 21: Results at the Spanish as target.
NIL
Priberam
%
Overall
accuracy
X
U
%F
%T
%D
%L
#
#
#
#
[115]
[43]
[32]
[10]
#
89
87
3
21
47,82
23,25
68,75
20
3
F
[8]
0,29
-
-
44,5
0,175
-0,287
34,5
K1
W
CWS
R
Run
Inaoe
69
118
7
6
28,69
18,60
87,50
-
3
0,12
Miracle
30
158
4
8
20
13,95
3,12
-
1
0,07
0,022
-0,452
15
1
3
0,03
0,015
-0,224
0,07
0,007
-0.34
11,5
7
UPV
TALP
23
14
166
183
5
1
6
2
13,08
6,08
9,30
2,32
12,5
18,65
-
Best performing systems have obtained worse results than last year due mainly to the low performance in
answering linked questions (15% of the questions) and due to the questions with answer only in Wikipedia.
Table 22 shows that considering only self-contained questions (the first one of each topic group) the results are
closer to the ones obtained last year. In fact the accuracy for the linked questions is less than 20%.
Table 22. Results for self-contained and linked questions, compared with overall accuracy.
Run
Priberam
Inaoe
Miracle
UPV
TALP
% accuracy over
Self-contained questions
[170]
49,41
37,64
15,29
12,94
7,05
% accuracy over
Linked questions
[30]
16,66
16,66
13,33
3,33
6,66
% Overall Accuracy
[200]
44,5
34,5
15
11,5
7
Table 23 shows some evidence on the effect of Wikipedia in the performance. When the answer appears only in
Wikipedia the accuracy is reduced in more than 35% in all the cases.
Table 23: Results for questions with answer in Wikipedia
Run
Priberam
Inaoe
Miracle
UPV
TALP
% accuracy over
questions with answer
only in wikipedia
[114]
40.35%
29.82%
7.89%
7.02%
0%
% accuracy over
questions with answer in
both EFE and wikipedia
[71]
54.93%
42.25%
28.17%
19.72%
14.08%
Regarding NIL questions, Table 24 shows the harmonic mean (F) of precision and recall for self-contained,
linked and all questions. The best performing system has decreased their overall performance with respect to the
last edition (see Table 25) in NIL questions. However, the performance considering only self-contained
questions is closer to the one obtained last year.
Table 24: Results at the Spanish as target for NIL questions
Priberam
Inaoe
Miracle
UPV
TALP
F-measure
(Selfcontained)
0.4
0.13
0.07
0.04
0.06
Fmeasure
(Overall)
0.29
0.12
0.07
0.03
0.07
Precision
(Overall)
Recall
(Overall)
0.23
0.07
0.05
0.02
0.04
0.38
0.38
0.13
0.13
0.38
Table 25. Evolution of best results in NIL questions.
Year
2003
2004
2005
2006
2007
F-measure
0,25
0,30
0,38
0,46
0,29
The correlation coefficient r between the self-score and the correctness of the answers (shown in Table 26)
has been similar to the obtained last year, being not good enough yet, and explaining the low results in CWS and
K1 [6] measures.
Since a supporting snippet is requested in order to assess the correctness of the answer, we have evaluated
the systems capability to extract the answer when the snippet contains it. The first column of table 27 shows the
percentage of cases where the correct answer was present in the snippet and correctly extracted. This information
is very useful to diagnose if the lack of performance is due to the passage retrieval or to the answer extraction
process. As shown in the table, the best systems are also better in the task of answer extraction, whereas the rest
of systems still have a lot of room for improvement.
Table 26. Answer Extraction and correlation coefficient r results at the Spanish as target
7
% Answer
Extraction
r
Run
Priberam
INAOE
Miracle
UPV
TALP
93,68
75
49,18
54,76
53,84
0,1170
0,237
-0,1003
0,134
Final considerations
This year the task was changed considerably and this affected the general level of results and also the level of
participation in the task. The grouped questions could be regarded as more realistic and more searching but in
consequence they were much more difficult. The policy of not declaring the question type means that if this is
deduced incorrectly then the answer is bound to be wrong. Moreover, the policy of not even declaring the topic
of a question group, but leaving it implicit (usually within the first question) means that if a system infers the
topic wrongly, then all questions in the group will be answered wrongly. This should be probably re-considered,
as it is not ‘realistic’. In a real dialogue, if a question is answered inappropriately we do not dismiss all
subsequent answers from that person, we simply re-phrase the question instead. The level of ambiguity
concerning question type in a real dialogue is not fixed at some arbitrary value but varies according to many
factors which the questioner estimates. In CLEF we are not modelling this process at all accurately and this
affects the validity of our results. Finally, co-reference has now entered CLEF. This is interesting and useful but
it might be preferable if we could separate the effect of co-reference resolution from other factors in analysing
results. This could be done by marking up the co-references in the question corpus and allowing participants to
use this information under certain circumstances.
Acknowledgments
A special thank to Bernardo Magnini (FBK-irst, Trento, Italy), who has given his precious advise and valueble
support at many levels for the preparation and realization of the QA track at CLEF 2007.
Anselmo Peñas has been partially supported by the Spanish Ministry of Science and Technology within the
Text-Mess-INES project (TIN2006-15265-C06-02).
References
1.
2.
3.
4.
QA@CLEF website: http://clef-qa.itc.it/
AVE Website: http://nlp.uned.es/QA/ave/.
QAST Website: http://www.lsi.upc.edu/~qast/
QART Website: http://gplsi.dlsi.ua.es/qart/
5.
QA@CLEF 2007 Organizing Committee. Guidelines 2007.
http://clef-qa.itc.it/2007/download/QA@CLEF07_Guidelines-for-Participants.pdf
6.
7.
Herrera, J., Peñas A., Verdejo, F.: Question answering pilot task at CLEF 2004. In: Peters, C., Clough, P.,
Gonzalo, J., Jones, Gareth J.F., Kluck, M., Magnini, B. (eds.): Multilingual Information Access for Text,
Speech and Images. Lecture Notes in Computer Science, Vol. 3491. Springer-Verlag, Berlin Hidelberg New
York (2005) 581–590
J. R. Landis and G. G. Koch. 1997. The measurements of observer agreement for categorical data.
Biometrics, 33:159–174.