Abstracts - 2006
Equal-party Conversation System for Language Learning
Chih-yu Chao & Stephanie Seneff
Computer-assisted language learning has been a topic of interest for decades. For learners who do not have easy access to human instructors, computer applications serve as an important means to expose second language learners to the target language in a frequent manner. While various applications have been created to facilitate reading and writing, very few existing applications focus on enabling learners to carry on daily conversations in the target language.
The system we propose here allows language learners to carry on conversations in a specific domain with the computer application, while practicing sentence formation, speaking, and listening skills in a less stressful environment. The learners may interact with the system over the phone or through the Internet, whichever way they feel more comfortable and convenient. Through a game-like interaction process, learners will be motivated by the scores and will remember the new vocabulary/sentence patterns more efficiently.
Instead of creating the system based on an existing database-accessing domain (such as the Jupiter weather domain system  and the Mercury  flight domain system), the proposed system has been created from scratch in the domain of activity scheduling. In other words, the user-system interaction will involve activity proposal, confirmation of likes/dislikes, activity negotiation, and schedule checking. Each dialogue may include multiple turns of the above-mentioned actions.
Typically in a database-accessing dialogue system, it is the system that always takes the initiative to ask the user for the information required for database search, and the user simply plays a passive role, providing keywords. The motivation for this activity scheduling domain is to encourage users (hereby used interchangeably with learners) to take the initiative in asking questions and negotiating with a "virtual buddy", so that language learners get to practice forming different types of sentences by playing a different role in each dialogue.
Currently, our goal is to provide English learners and Mandarin Chinese
learners an interactive system to practice conversation in the activity
scheduling domain. The interaction process can be briefly described as
Please see the following example dialogues:
The system will be able to keep track of the score the user gets every time a successful utterance in the target language is generated. The user can get help from the system by speaking in the native language, which will be translated into the target language and sent back to the user. At the same time, the system keeps track of the difficulties the user has encountered, so that in the upcoming dialogues, the same material (vocabulary or sentence pattern) will be reviewed until the user succeeds.
The interlingua-based approach maps natural language input into a language-independent "semantic frame" representation; this is done by the TINA  natural language understanding system, which performs syntatic and semantic analyses. The persona mentioned in the previous section is also represented the same way. Therefore, when the system checks whether a scheduled activity matches the current persona, it in fact checks the keys and values in the semantic frames.
The Genesis  language generation system maps the semantic frames into a set of well-formed surface strings in the target language based on the language-dependent generation rules written with the Pluto  surface realization module.
During the development phase, the system allows two simulated users to interact with each other. That is, dialogues can be automatically generated by the system, without eliciting input from human subjects. Using automatic dialogue generation, we can acquire a large amount of data with very little cost. The generated dialogues are good for different types of evaluation, such as the coverage of language generation rules, the accuracy of speech recognition, and the quality of speech synthesis.
A web audio server is being integrated into the system, so that users will be able to interact with the system as long as there is Internet access and a microphone. Also, a bilingual speech recognizer will be included to determine whether the user input is in the target language, or is a request for help in the user's native language.
The interaction game can be modified into "dual mode", that is, two human users will be able to carry on a conversation through their own computers. The users will not hear the other party during the game; instead, the computer will process the speech input from one user and present the information to the other user (visually and/or in audio). The advantage of such an indirect interaction is that users can still ask the system for help, and the system can filter the users' speech such that only utterances spoken in the target language will be learned and transmitted. It could also give feedback/correction to the user before the input information is sent to the other party.
Since the proposed system is designed for language learning purposes, a series of user studies will be conducted to serve as a reference for design improvement and pedagogical evaluation.
 V. Zue, S. Seneff, J. R. Glass, J. Polifroni, C. Pao, T. J. Hazen, and I. L. Hetherington. Jupiter: A telephone-based conversational interface for weather information. IEEE Trans. on Speech and Audio Processing, 8(1):100--112, 2000.
 S. Seneff and J. Polifroni. Dialogue Management in the Mercury Flight Reservation System. In Proceedings of ANLP-NAACL 2000, Satellite Workshop, pp. 1--6, Seattle, WA.
 S. Seneff. TINA: A Natural Language System for Spoken Language APplications. In Computational Linguistics, 18(1):61--86, 1992.
 L. Baptist and S. Seneff. GENESIS-II: A Versatile System for Language GE\eneration in Conversational System Applications. In Proceedings of ICSLP, pp. 271--274, Beijing, China, October 2000.
 B. Cowan. PLUTO: A Preprocessor For Multilingual Spoken Language Generation. Master's thesis, MIT, Cambridge, MA, February 2004.