CSAIL Publications and Digital Archive header
bullet Research Abstracts Home bullet CSAIL Digital Archive bullet Research Activities bullet CSAIL Home bullet

link to publications.csail.mit.edu link to www.csail.mit.edu horizontal line


Research Abstracts - 2007
horizontal line

horizontal line

vertical line
vertical line

StartMobile: Harnessing Language in Mobile Environments

Boris Katz, Gary Borchardt, Sue Felshin & Federico Mora


Users of cellphones and other mobile devices are faced with an ever-increasing range of capabilities at their disposal, and this development will soon overwhelm traditional user interfaces. New mechanisms are required that enable mobile users to access information, create information and perform actions with ease. We feel that a language-based interface is an ideal candidate for this task: language is not only well-suited to humans—it is expressive, compact and easy to use—but it is also within the capability of machines to handle requests posed in language.


We have implemented a system called StartMobile that handles a range of language-based requests in mobile environments. StartMobile uses the START information access system [1] [2] to translate user requests into an intermediate representation language called Moebius, and the translated requests are fulfilled by action-completing code that resides on a user's mobile device. This system currently supports a number of activities:

  • retrieving contact and calendar information stored on a mobile device,
  • retrieving text messages and managing the user's text message inbox,
  • placing phone calls from the user's mobile device,
  • creating reminders on the user's mobile device or on other users' mobile devices,
  • taking pictures with the mobile device's camera,
  • accessing position information and displaying associated map and direction information on the user's mobile device,
  • retrieving video tutorials for presentation to the mobile user, and
  • retrieving general-purpose information for the benefit of the user.

Figure 1 illustrates StartMobile in action. Here, a mobile user has asked the system to perform a search within the contacts database on the user's phone. StartMobile performs the requested search and displays the results to the user.

screenshot of request on phone screenshot of reply on phone

Figure 1: StartMobile responding to a user's request by performing a search over its contacts database.


We are continuing to add new capabilities to the StartMobile system. These capabilities will enable the system to access and manipulate information on the user's mobile device and on central servers, to interact with other mobile devices, to vary its behavior by exploiting contextual information, and to control the execution of additional actions on mobile devices and central servers.


This work is supported in part by the Nokia Corporation as part of the Nokia/MIT Lablet initiative.


[1] Boris Katz. Using English for Indexing and Retrieving. In Artificial Intelligence at MIT: Expanding Frontiers, v. 1; Cambridge, MA, 1990.

[2] Boris Katz. Annotating the World Wide Web Using Natural Language. In Proceedings of the 5th RIAO Conference on Computer Assisted Information Searching on the Internet (RIAO '97), Montreal, Canada, 1997.


vertical line
vertical line
horizontal line

MIT logo Computer Science and Artificial Intelligence Laboratory (CSAIL)
The Stata Center, Building 32 - 32 Vassar Street - Cambridge, MA 02139 - USA
tel:+1-617-253-0073 - publications@csail.mit.edu