In search of the why: Developing a system for answering why-questions

Abstract

Information searching was once a task performed almost exclusively by librarians and domain experts. With the rise of the Internet, searching for information by means of search engines has become a daily activity for many people. The most widespread form of web searching is ad-hoc document retrieval: The user types in a query in the form of a (usually small) unstructured set of key words, and the search engine retrieves a list of pointers to web pages. The web pages are ranked according to their estimated relevance to the input query. Ad-hoc document retrieval and ranking is the core business of the large web search engines Google, Yahoo and Bing. The value of ad-hoc document retrieval for the user depends on his/her information need. For example, the query "flamingos habitat" requires scrolling through at least one of the retrieved documents in order to satisfy the user’s information need. Questions in natural language such as "where do flamingos nest?" tend to be even more specific. In this thesis we address the problem of answering questions that start with why: "why are flamingos pink?" Why-questions require a different approach than factive questions because their answers tend to be longer and more complex. Moreover, because of the complexity of the problem, why-question answering (why-QA) is an interesting case study for a linguistically motivated approach to information retrieval.