Improving Accessibility in an Automated Question-Answering System

Author: Silvia Quarteroni

You may come across many Question-Answering systems where in you ask a question and the system retrieves the answer for you. It does not keep in mind the readability level of the user while presenting the answers. For instance: a primary school child and a History student may need different answers to the question: When did the middle ages begin? Hence it can be considered as a naive way of information retrieval. This paper addresses the problem of accessibility by filtering the answers based on a person's reading difficulty using a Question Answering system.

A Web-based QA system

 A system called YourQA is used as a base system for all the experiments. The QA algorithm follows 3 phases:

Question Processing: The expected answer type for the requested query is determined and then the query is submitted to the underlying search engine (in this case Google, www.google.com)

Document Retrieval: The top 'n' documents are retrieved from the search engine and split into sentences.

Answer Extraction: 

(a) Lexical, syntactic and semantic criteria is applied to the query and to each retrieved document to identify the Candidate answer sentences.

(b) Candidate answers are ordered by relevance to the query. The list of top ten ranked pages is returned to the user in an HTML page.

A personalized QA system

The salient feature of this system is the presence of a User Modelling System.

The interaction between User Model and the core QA module happens in 2 phases:

• During document retrieval phase, the User module provides criteria to filter out unsuitable documents.
• The User module also provides criteria to re-arrange the Candidate answers after answer extraction. It is according to the user relevance to the query.  

User Model

User representation comprises of two aspects:

• User's interest in terms of the answer contents.
• User's preference in terms of answer presentation.

These are modeled using 3 attributes:

1. Age range, {7-10, 11-16, adult}
2. Reading level, {basic, medium, advanced}
3. Profile, p, a set of textual documents, bookmarks and web pages of interest.

 Approaching Readability in Question Answering

Most of the times we assume that a sentence is complicated if it is too long. However, it can be noticed that in Web documents the sentences are usually short and more concise than in printed documents. As opposed to the previous approaches, the language modeling approach adopted in YourQA accounts especially for lexical information. In order to represent the 3 different age ranges defined in the corresponding attribute of the User Model, we define the three following classes:

1. basic, representing a document suitable for ages 7-11
2. medium, representing a document suitable for ages 11-16
3. advanced, representing a document suitable for adults.

Reading Level Estimation

This is done as a supervised learning task. The representative documents for each of the classes are collected and labelled as training instances. Later this is used to classify the unseen documents according to their reading levels.

As a learning model they used Smoothed Unigram Model, based on a representation of data known as Unigram language modeling.  In the test phase of the learning process, given an unclassified document D, a unigram language model is built for the single document D (as done for the training documents). The estimated reading level of D is the language model LMi, maximizing the likelihood L(LMi|D) that D has been generated by LMi. In our case three language models are defined where i = {basic, medium, advanced} and the likelihood is estimated using a function. (Please refer to the paper for equations) 

Reading Level Filtering

The first step carried out during personalized document retrieval is the estimation of the reading level of each document returned by Google in response to the query. Such estimation is conducted via language modeling using the technique mentioned above. 

As there can be queries for which the number of retrieved documents matching a particular reading level is less than minimum number specified, the rules can be relaxed. In particular, if the user's reading level is advanced, medium level documents are considered and in case the threshold number of documents are not met, basic documents complete the set.   The subsequent QA phase of answer extraction therefore begins with the documents left out of the reading level filtering phase.

Evaluation

Evaluation of the reading level estimation was conducted according to 2 criteria:

• Assessment of Robustness of unigram language models created to represent User's readability levels
• Assessment of agreement of users with the system's estimation.

Robustness of unigram language models: This was calculated by randomly splitting all the documents used to create the language models into ten equally sized folds. Then estimation accuracy was calculated.

User agreement with Reading level estimation: The metric used to assess this was called Reading Level Agreement (Ar). Therefore, the number of documents in R rated by the users as suitable for r is as follows:

                                          Ar= suitable (R) / |R|

where R= set of results returned by the system for reading level r.

CONCLUSIONS AND PERSPECTIVES

• In this article, the problem of accessibility in information retrieval is addressed by introducing a QA system able to filter answers based on reading difficulty.
• The reading level estimation technique based on language modeling has an advantage of being applicable to documents in any domain.
• An application has been demonstrated addressing the needs of students primary school, secondary school and adults. However, this model is suitable to model reading level of any user category provided that the training documents are available. 

Hope the paper is "Text Simplified" for you guys to read now ;-)