Research Abstracts - 2006

Equal-party Conversation System for Language Learning

Chih-yu Chao & Stephanie Seneff

Introduction

Computer-assisted language learning has been a topic of interest for decades. For learners who do not have easy access to human instructors, computer applications serve as an important means to expose second language learners to the target language in a frequent manner. While various applications have been created to facilitate reading and writing, very few existing applications focus on enabling learners to carry on daily conversations in the target language.

The system we propose here allows language learners to carry on conversations in a specific domain with the computer application, while practicing sentence formation, speaking, and listening skills in a less stressful environment. The learners may interact with the system over the phone or through the Internet, whichever way they feel more comfortable and convenient. Through a game-like interaction process, learners will be motivated by the scores and will remember the new vocabulary/sentence patterns more efficiently.

Our Approach

Instead of creating the system based on an existing database-accessing domain (such as the Jupiter weather domain system [1] and the Mercury [2] flight domain system), the proposed system has been created from scratch in the domain of activity scheduling. In other words, the user-system interaction will involve activity proposal, confirmation of likes/dislikes, activity negotiation, and schedule checking. Each dialogue may include multiple turns of the above-mentioned actions.

Typically in a database-accessing dialogue system, it is the system that always takes the initiative to ask the user for the information required for database search, and the user simply plays a passive role, providing keywords. The motivation for this activity scheduling domain is to encourage users (hereby used interchangeably with learners) to take the initiative in asking questions and negotiating with a "virtual buddy", so that language learners get to practice forming different types of sentences by playing a different role in each dialogue.

Currently, our goal is to provide English learners and Mandarin Chinese learners an interactive system to practice conversation in the activity scheduling domain. The interaction process can be briefly described as follows:
At the beginning of the game, the user will be presented with a persona, i.e. a schedule from "yesterday" to "day after tomorrow", the activities scheduled at a certain time of day, and the preference of activities. Meanwhile, the system (the "virtual buddy") also has a persona represented internally. Then the user will be prompted to schedule an activity with the virtual buddy, so that both parties can do the same activity together; or the system will start scheduling an activity, and the user will have to respond and/or negotiate based on the given persona. When both parties have reached an agreement, the dialogue is complete, and a new persona will be presented to the user (and a new one to the system as well) for the next dialogue.

Please see the following example dialogues:

The system will be able to keep track of the score the user gets every time a successful utterance in the target language is generated. The user can get help from the system by speaking in the native language, which will be translated into the target language and sent back to the user. At the same time, the system keeps track of the difficulties the user has encountered, so that in the upcoming dialogues, the same material (vocabulary or sentence pattern) will be reviewed until the user succeeds.

Implementation

The interlingua-based approach maps natural language input into a language-independent "semantic frame" representation; this is done by the TINA [3] natural language understanding system, which performs syntatic and semantic analyses. The persona mentioned in the previous section is also represented the same way. Therefore, when the system checks whether a scheduled activity matches the current persona, it in fact checks the keys and values in the semantic frames.

The Genesis [4] language generation system maps the semantic frames into a set of well-formed surface strings in the target language based on the language-dependent generation rules written with the Pluto [5] surface realization module.

During the development phase, the system allows two simulated users to interact with each other. That is, dialogues can be automatically generated by the system, without eliciting input from human subjects. Using automatic dialogue generation, we can acquire a large amount of data with very little cost. The generated dialogues are good for different types of evaluation, such as the coverage of language generation rules, the accuracy of speech recognition, and the quality of speech synthesis.

Future Work

A web audio server is being integrated into the system, so that users will be able to interact with the system as long as there is Internet access and a microphone. Also, a bilingual speech recognizer will be included to determine whether the user input is in the target language, or is a request for help in the user's native language.

The interaction game can be modified into "dual mode", that is, two human users will be able to carry on a conversation through their own computers. The users will not hear the other party during the game; instead, the computer will process the speech input from one user and present the information to the other user (visually and/or in audio). The advantage of such an indirect interaction is that users can still ask the system for help, and the system can filter the users' speech such that only utterances spoken in the target language will be learned and transmitted. It could also give feedback/correction to the user before the input information is sent to the other party.

Since the proposed system is designed for language learning purposes, a series of user studies will be conducted to serve as a reference for design improvement and pedagogical evaluation.

References:

[1] V. Zue, S. Seneff, J. R. Glass, J. Polifroni, C. Pao, T. J. Hazen, and I. L. Hetherington. Jupiter: A telephone-based conversational interface for weather information. IEEE Trans. on Speech and Audio Processing, 8(1):100--112, 2000.

[2] S. Seneff and J. Polifroni. Dialogue Management in the Mercury Flight Reservation System. In Proceedings of ANLP-NAACL 2000, Satellite Workshop, pp. 1--6, Seattle, WA.

[3] S. Seneff. TINA: A Natural Language System for Spoken Language APplications. In Computational Linguistics, 18(1):61--86, 1992.

[4] L. Baptist and S. Seneff. GENESIS-II: A Versatile System for Language GE\eneration in Conversational System Applications. In Proceedings of ICSLP, pp. 271--274, Beijing, China, October 2000.

[5] B. Cowan. PLUTO: A Preprocessor For Multilingual Spoken Language Generation. Master's thesis, MIT, Cambridge, MA, February 2004.