Research Abstracts - 2006

Automatic Grammar Correction for Second-Language Learners

J. Lee & S. Seneff

Introduction

In the past few years, our group has been developing a conversational language learning system [1], which engages students in a dialogue in order to help them learn a foreign language. One important component of such a system is to provide corrections of the students' mistakes, both phonetic and grammatical, the latter of which is the focus of this abstract. For example, the student might say, "*I will like to see flight arrive Dallas next day." The system would be expected to correct this to, "I would like to see flights arriving in Dallas the next day."

Previous approaches to grammar correction focus on parsing ill-formed sentences. To cope with grammatical errors, new mechanisms are incorporated into parsers which, otherwise, are intended for analyzing well-formed sentences. Examples include constraint relaxation in a unification framework [2,3], and error-production rules in context-free grammars [4,5].

A disadvantage with the above parsing-based approaches is that, as more and more types of errors need to be handled, the grammars become increasingly complicated, exponentially growing the number of ambiguous parses. We instead propose a two-step, generation-based framework.

Approach

According to the Japanese Learners' English corpus [7], which consists of transcripts of native Japanese speakers conversing in English, the three most frequent error classes are articles, noun number and prepositions, followed by a variety of errors associated with verbs. Motivated by this analysis, we consider errors involving these four parts-of-speech: articles, prepositions, noun and verb inflections.

In the first step, an input sentence, hypothesized to contain errors, is first reduced to a "canonical form" devoid of articles, prepositions, and auxiliaries. Furthermore, all nouns are reduced to their singular forms, and all verbs are reduced to their root forms. A simple algorithm then expands the sentence into a lattice of alternatives, including all of the alternative inflections, and possible insertion of articles, prepositions and auxiliaries at every word position.

In the second step, a language model is used to score the various paths in the lattice. Both n-grams and a stochastic context-free grammar language model, TINA [6], are utilized in a complementary fashion.

Evaluation

The training set consists of 10,369 transcripts of utterances, produced by callers in spoken dialogues with the Mercury flight domain [8]. The test set consists of 1317 sentences from the same domain, all at least four words long. These utterances, whose average length is 7.6 words, serve as our "gold-standard" in the automatic evaluation.

When using TINA to rerank the 10-best correction candidates proposed by the word trigram language model, the system achieved 91.6% accuracy rate in verb/noun inflections, and 0.80 F-measure on insertions of articles, prepositions and auxiliaries. The trigram model alone, as a baseline, achieved 89.2% accuracy and 0.74 F-measure.

We conducted a human evaluation on the subset of utterances for which a parse was produced (73.5% of the 1317 utterances). Four native English speakers, not involved in this research, were given the ill-formed input, and were asked to compare the corresponding transcript and the corrected output, without knowing their identities. 88.7% of the corrected outputs were judged to be at least as good as the transcript.

References:

[1] S. Seneff, C. Wang, M. Peabody and V. Zue. "Second Language Acquisition through Human Computer Dialogue" In Proc. ISCSLP, Hong Kong, 2004.

[2] F. Fouvry. "Constraint Relaxation with Weighted Feature Structures" In Proc. 8th International Workshop on Parsing Technologies, Nancy, France, 2003.

[3] C. Vogel and R. Cooper. "Robust Chart Parsing with Mildly Inconsistent Feature Structures" In Nonclassical Feature Systems, vol. 10, 1995.

[4] L. Michaud, K. F. McCoy and C. A. Pennington. "An Intelligent Tutoring System for Deaf Learners of Written English" In Proc. 4th International ACM SIGCAPH Conference on Assistive Technologies (ASSETS), Arlington, VA, 2000.

[5] E. M. Bender, D. Flickinger, S. Oepen, A. Walsh and T. Baldwin. "Arboretum: Using a Precision Grammar for Grammar Checking in CALL" In Proc. InSTIL/ICALL Symposium on Computer Assisted Learning, Venice, Italy, 2004.

[6] S. Seneff. "Tina: A Natural Language System for Spoken Language Applications" In Computational Linguistics, 18(1):61--86, 1992.

[7] E. Izumi, K. Uchimoto, T. Saiga, T. Supnithi and H. Isahara. "Automatic Error Detection in the Japanese Learners' English Spoken Data" In Proc. ACL, Sapporo, Japan, 2003.

[8] S. Seneff. "Response Planning and Generation in the Mercury Flight Reservation System" In Computer Speech and Language, vol. 16, 2002.