Research Abstracts - 2007

Problem

Users of cellphones and other mobile devices are faced with an ever-increasing range of capabilities at their disposal, and this development will soon overwhelm traditional user interfaces. New mechanisms are required that enable mobile users to access information, create information and perform actions with ease. We feel that a language-based interface is an ideal candidate for this task: language is not only well-suited to humans—it is expressive, compact and easy to use—but it is also within the capability of machines to handle requests posed in language.

Approach

We have implemented a system called StartMobile that handles a range of language-based requests in mobile environments. StartMobile uses the START information access system [1] [2] to translate user requests into an intermediate representation language called Moebius, and the translated requests are fulfilled by action-completing code that resides on a user's mobile device. This system currently supports a number of activities:

• retrieving contact and calendar information stored on a mobile device,
• retrieving text messages and managing the user's text message inbox,
• placing phone calls from the user's mobile device,
• creating reminders on the user's mobile device or on other users' mobile devices,
• taking pictures with the mobile device's camera,
• accessing position information and displaying associated map and direction information on the user's mobile device,
• retrieving video tutorials for presentation to the mobile user, and
• retrieving general-purpose information for the benefit of the user.

Figure 1 illustrates StartMobile in action. Here, a mobile user has asked the system to perform a search within the contacts database on the user's phone. StartMobile performs the requested search and displays the results to the user.

Figure 1: StartMobile responding to a user's request by performing a search over its contacts database.

Progress

We are continuing to add new capabilities to the StartMobile system. These capabilities will enable the system to access and manipulate information on the user's mobile device and on central servers, to interact with other mobile devices, to vary its behavior by exploiting contextual information, and to control the execution of additional actions on mobile devices and central servers.

Support

This work is supported in part by the Nokia Corporation as part of the Nokia/MIT Lablet initiative.

References:

[2] Boris Katz. Annotating the World Wide Web Using Natural Language. In Proceedings of the 5th RIAO Conference on Computer Assisted Information Searching on the Internet (RIAO '97), Montreal, Canada, 1997.