Related Studies
Early QA system
Early work in Open QA used search engines such as Google as the main information source. For example the AskMSR system developed in 2002, which mainly relies on the page ranking algorithms of search engines. As AskMSR only uses the summaries of the first three searching results as the knowledge source, there is a high probability for this pool of knowledge to miss the answer. Indeed, the response rate of AskMSR system is only about 50%, and overall accuracy is only 34.7%.
Recent researches have been focusing on large, multi-domain KBs like YAGO2 and Freebase, which are mainly curated KBs. These KBs are attractive for QA because they contain highly precise knowledge with little redundancy, which enables QAs to answer questions accurately. However, these systems generally have limited recall, due to the incompleteness of curated KBs. The Paralex system is the first Open QA system to operate over OKB. It uses some learned templates that directly map questions to queries. As one my imagine, this simple method of mapping may result in queries that differ in meaning from the original question, or even don’t make sense.
Recent open QA system
Later a QA system, named OQA, which can be applied to both curated and open KBs, was developed by the same group of people [5]. This system utilizes a set of KBs, which enable it to combine knowledge extracted from different KBs. This technique also allows it to join multiple assertions to arrive at a single answer. Moreover, OQA also combines high-recall data mining techniques with high-precision, hand-written rules to obtain a query which more precisely represents the original question. However, it still have several problems. Firstly, it fails to fully utilize the contextual information to match the entities. Instead, matching is mainly done by approximate string matching, which is a low-accuracy method. Secondly, the machine learning model is not well trained. The ensemble method performs even worse than individual ones, which should not happen if the component estimators are cleverly combined. As a result, the precision and recall are still not high enough for it to be applicable.
Current work of canonicalization
Currently no QA system is built on canonicalized KBs. There have been some works which focus on canonicalizing OKBs, and they mainly use the hierarchical agglomerative clustering (HAC) method to form clusters of assertions, based on some similarity functions to determine whether different assertions should be mapped to the same cluster. However, this process is extremely time-consuming, as the similarity measure is computed between massive pairs of assertions, and the iterative process never ends until a certain threshold of dissimilarity between clusters is reached. As the similarity function is also computed based on common substrings between different entities, it is also not accurate enough. With the low accuracy and efficiency in place, the current canonicalization methods cannot be directly used in building QA systems.