Research Abstracts - 2007

Introduction

To learn to speak a foreign language well, it is recommended that the learner practice natural conversation with a native speaker of the language. However, a one-on-one tutor is not affordable for everyone, and the availability and patience of a human tutor is usually limited, too. The systems we are developing serve as an aid for learning a foreign language (currently Mandarin Chinese). By playing the games with a "virtual buddy" on the computer, language learners are provided with exercises in translation and conversation. As the systems preserve the qualities of a typical computer system: being infinitely patient and available any time of day, learners will be able to practice natural conversation repeatedly without embarrassment. Table 1 provides a comparison of a computer vs. human tutor.

The two game systems comprise a translation game [1] [2] and a dialogue game [3], details of which will be provided later. Both games are accessible through a web browser, and the learner's speech is captured during each game in order to support the interaction. The systems may also be installed to run locally on a laptop; that is, no network connection is needed.

Translation Game

The translation game serves as preparation for the dialogue game. The game design is well motivated by the learning approach suggested by Pimsleur [4]. The system presents a set of words/sentences in English within the topic domain, and the user is asked to speak the words/sentences of equivalent meaning in the target language. By playing the translation game, the user is able to internalize the main structures of the target language, as well as its vocabulary, grammar rules, and pronunciation. There are ten difficulty levels in this game: Level 1 contains isolated vocabulary items, and Level 10 contains long and complicated sentences. The user is free to start at any level, and as the game proceeds, the difficulty level is automatic adjusted based on the user's performance. We have currently developed versions of this game in two distinct domains: a flight-planning domain and a schedules-and-hobbies domain.

The user's speech is processed by a speech recognizer and language understanding components [5]. The system compares the meaning representation of the spoken input with one automatically derived from the English translation task. The system then speaks a paraphrase of the user's input in both languages [6] to confirm understanding and to familiarize the user with different expressions of the same meaning. A help function was also designed in the system. When the task is unfamiliar, the user can always listen to the "correct" translation of each task simply by clicking the help button, and then imitate the speech.

Dialogue Game

The dialogue game involves solving a particular scenario by conversing with a "virtual buddy"; that is, the user has to interact with the computer in order to jointly solve the specified scenario. The level of difficulty is determined based on the type of help provided to the user. To be more specific, in addition to the dialogue partner (i.e. the "virtual buddy"), the user is assisted by a robotic tutor, which suggests to the user what might be said in the next turn. The tutor monitors the ongoing scenario and independently plans the user's half of the conversation and updates the suggestion depending on how the context evolves. The user, however, is still free to say whatever appropriate to carry on the conversation. We have implemented a first version of this game in a schedules-and-hobbies domain, where the user and the virtual buddy are tasked with finding a time to meet in the near future to jointly engage in an activity that they both like.

There are five difficulty levels in this game, and each provides the user with a different type of assistance:

• Level 1 – By clicking a button, the user can have the tutor carry out the conversation with the virtual buddy to advance to the next dialogue turn, i.e. the user eavesdrops on a computer-computer conversation.
• Level 2 – The tutor suggests what to say next in the form of text in pinyin (Chinese phonetic symbols) on the screen, and the user can simply read it out to converse with the virtual buddy.
• Level 3 – The tutor presents the suggestion in English text, and the user can converse with the virtual buddy by translating the suggestion into Chinese.
• Level 4 – The tutor presents the suggestion in Chinese characters, and the user can practice recognizing characters while speaking.
• Level 5 – The user stops receiving prompts from the tutor, but can always consult the tutor by typing what s/he would like to say in English or pinyin.

Future Work

The two game systems will be evaluated by two types of basic performance measures: (1) for each system, the recognition accuracy and the translation accuracy serve as an index of quality; (2) calculating the success rate in the translation game and the number of turns taken to complete each dialogue will provide a quantitative view of interaction success. Also, a pre-test & post-test design in the upcoming user study will further confirm the pedagogic value of the systems. More future work of this project involves expanding the domains supported and introducing the games to the classroom setting.

Acknowledgement

This research is supported in part by the Industrial Technology Research Institute and the Cambridge MIT Initiative.

References

[1] C. Wang and S. Seneff. A Spoken Translation Game for Second Language Learning. In The Proceedings of Artificial Intelligence in Education, 2007.

[2] S. Seneff, C. Wang, and J. Lee. Combining Linguistic and Statistical Methods for Bi-directional English Chinese Translation in the Flight Domain. In The Proceedings of AMTA, pp. 213--222, Boston, MA, 2006.

[3] S. Seneff. Interactive Computer Aids for Acquiring Proficiency in Mandarin. Keynote Speech. In The Proceedings of ISCSLP, pp. 1--11, Singapore, 2006.

[4] P. Pimsleur. A Memory Schedule. In Modern Language Journal, 51: 73--75, 1967.

[5] S. Seneff. TINA: A Natural Language System for Spoken Language Applications. In Computational Linguistics, 18(1):61--86, 1992.

[6] L. Baptist and S. Seneff. Genesis-II: A Versatile System for Language Generation in Conversational System Applications. In Proceedings of ICSLP, pp. 271--274, Beijing, China, October 2000.