Research Abstracts - 2006

Open Domain Question Answering on Newspaper Text

Boris Katz, Gregory Marton, Sue Felshin, Daniel Loreto & Federico Mora

The Problem

NIST's annual TREC Question Answering Track supports the community by providing shared evaluation on a number of important tasks. Factoid and list questions test systems on their ability to answer factual questions exactly, based on a body of English newspaper text. The definition and relationship tasks evaluate systems on their ability to locate the most important facts about a subject, or about the complex relationships between two subjects.

Motivation

Information systems aimed at humans often return documents, paragraphs, snippets of documents, or sentences, and the user must find the actual answer in the text returned. While the surrounding context is often useful to humans, knowing the exact answer can help automatic systems answer more complex questions. For example, if a system can find that "the fifth largest country in Africa" is "Chad", then it becomes possible to answer followup questions, "How many people live there?", and more complex questions, "How many people live in Africa's fifth largest country?".

Previous Work

Since 1992, TREC evaluations [7] have been providing a way to comparatively evaluate systems on a common data set and task. In 1999, a question answering "track" was introduced. A set of "open domain" news articles is provided in advance, and at evaluation time, questions pertaining to the text are given, to be answered automatically. Human assessors then pool the automatic results and decide which answers are correct.

Previous innovations fall into a few categories:

• Using online resources like Google and Wikipedia to find related keywords and query refinements. These methods help find the right documents in the corpus, and then help to extract the best exact answers, on the assumption that those will be the ones most prominently found online.
• Understanding the corpus better before questions are asked: for example resolving names, noun phrase references ("the senator"), and pronouns helps bring the right keywords into close textual proximity of possible answers for a proximity-based keyword search. Textual cues for definitional contexts (for example apposition) indicate good answers for questions like "Who is X?": "X, a leading Z, ...".
• The best systems use reasoning to check that the sentences they use to support an answer can justifiably be said to answer the question. For many answers, this requires much world knowledge.

Our system [3] can be decomposed into five components: data indexing, question analysis, passage retrieval, candidate generation, and answer selection. In the data indexing and passage retrieval components, we have explored several query expansion and document retrieval methods, but their effect on question answering performance is still unclear. In the question analysis component, we identify an expected answer type, and possible paraphrases of the question. During candidate generation, we look for structures that indicate a possible answer. During answer selection, we look for support for each candidate (e.g., from the Web), and choose the set of answers to return. The question answering track is separated into four kinds of questions:

• factoid questions, with an exact named-entity answer ("Who shot Kennedy?")
• list questions, with a set of exact named-entity answers ("What countries did he visit?")
• 'other' questions, asking for important facts not yet explored in the conversation ("What else do you know about John F. Kennedy?")
• relationship questions, that is, questions on complex topics, with a set of complex answers which may be spread across documents ("What communications or conflicts linked Kennedy with Nikita Khrushchev?")

We have fielded separate systems for each kind of question, and we are working to integrate our various strategies into a comprehensive package, and into the START system. [1] [2]

Research Support

This work is supported in part by the Advanced Research and Development Activity as part of the AQUAINT Phase II research program.

References: