Research Abstracts - 2006

End-User Programming for the Web

Michael Bolin, Greg Little, Ricarose Roque, Darris Hupp, Vikki Chou & Robert Miller

Motivation

On the desktop, an application can expect to control its user interface down to the last pixel, but on the World Wide Web, a content provider has no control over how the client will view the page, once delivered to the browser. This creates an opportunity for end-users who want to automate and customize their web experiences, but the growing complexity of web pages and standards prevents most users from realizing this opportunity.

Existing approaches to web automation use a scripting language that dwells outside the web browser, not inside. For an end user, the distinction is significant. Cookies, authentication, session identifiers, plugins, user agents, client-side scripting, and proxies can all conspire to make the Web look significantly different to an agent running outside the web browser. Recent highly-interactive web applications like Google Mail and Google Maps -- sometimes called AJAX applications because they use asynchronous JavaScript and XML -- have made this situation even worse.

Perhaps the most telling difference, and the most intimidating one to a user, is the simple fact that outside a web browser, a web page is just raw HTML. Even the most familiar web portal looks frighteningly complicated when viewed as HTML source.

Approach

Chickenfoot is a scripting environment embedded in the Firefox web browser that provides a platform for automating and customizing web applications through a familiar interface -- as web pages rendered in a web browser. The challenge for Chickenfoot is simply stated: a user should never have to view the HTML source of a web page in order to customize or automate it.

Chickenfoot addresses this challenge in three ways. First, it runs inside the web browser, so that the rendered view of a web page is always visible alongside the Chickenfoot development environment. Second, its language primitives are concerned with the web page's user interface, rather than its internal details. For example, Chickenfoot uses commands like click, enter, and pick to interact with forms. Third, it uses novel pattern-matching techniques to allow users to describe components of a web page (targets for interaction, extraction, insertion, or customization) in terms that make sense for the rendered view. For example, the click command identifies the button to be clicked using keywords from its text label, rather than the name it was given by the web page designer.

Figure 1. Chickenfoot sidebar in Firefox web browser, showing a simple script that automates Google Image searches.

With Chickenfoot, users can automate web tasks, programmatically extract content from web sites, and customize how web pages are presented in their browser.

Some examples of tasks we have accomplished with Chickenfoot scripts are:

• Highlighting SAT vocabulary words in a web page.
• Hyperlinking Java types in sample Java code to their Javadoc API.
• Enabling users to sort HTML tables by clicking on a column header.
• Concatenating multiple pages of search results into one long page.
• Converting web pages into black-and-white so they are easier to read.
• Appending icons after hyperlinks that reflect the file type of the target of the link.

Current Work

Research on Chickenfoot is continuing in several directions:

• Syntax-free programming. One important usability barrier in Chickenfoot is the need to learn and correctly generate the syntax of the language, such as click("search button") or enter("all the words", "disk"). We are exploring ways to eliminate this barrier by allowing the user to enter more freeform statements that are interpreted, such as click search or even all words = disk. Syntax-free programming has the potential to be easier to learn, less error-prone to use, and potentially even more concise than tradtional programming syntax.
• Script recording. Many useful Chickenfoot scripts can simply be demonstrated by pointing, clicking, and typing into the web site itself. These actions are translated into Chickenfoot code which can be incorporated into a script.
• Web services. More and more web sites are exposing programming interfaces, usually in the form of SOAP, XML-RPC, or REST services, but sometimes as Javascript APIs. Chickenfoot scripts can already use these APIs, but we are also exploring how to integrate these APIs into the user interface of the web site itself -- so that, for example, by simply pointing at a field of a search result, the user can easily discover the API calls that would produce that same data.

Acknowledgements

This work was supported in part by the National Science Foundation under award number IIS-0447800. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

[1] Michael Bolin, Matthew Webber, Philip Rha, Tom Wilson, and Robert C. Miller. "Automation and Customization of Rendered Web Pages." UIST 2005, pp. 163-172.  Winner of best paper award.

[2] Michael Bolin. End-user Programming for the Web. MEng thesis, Massachusetts Institute of Technology, June 2005.  Winner of William A. Martin Memorial Thesis Prize for an outstanding Master's thesis in computer science.

[3] Michael Bolin and Rob Miller.  "Naming Page Elements in End-User Web Automation." Workshop on End-User Software Engineering, ICSE 2005.