Research Abstracts - 2007

Motivation

On the desktop, an application can expect to control its user interface down to the last pixel, but on the World Wide Web, a content provider has no control over how the client will view the page, once delivered to the browser. This creates an opportunity for end-users who want to automate and customize their web experiences, but the growing complexity of web pages and standards prevents most users from realizing this opportunity.

Existing approaches to web automation use a scripting language that dwells outside the web browser, not inside. For an end user, the distinction is significant. Cookies, authentication, session identifiers, plugins, user agents, client-side scripting, and proxies can all conspire to make the Web look significantly different to an agent running outside the web browser. Recent highly-interactive web applications like Google Mail and Google Maps -- sometimes called AJAX applications because they use asynchronous JavaScript and XML -- have made this situation even worse.

Perhaps the most telling difference, and the most intimidating one to a user, is the simple fact that outside a web browser, a web page is just raw HTML. Even the most familiar web portal looks frighteningly complicated when viewed as HTML source.

Approach

Chickenfoot is a scripting environment embedded in the Firefox web browser that provides a platform for automating and customizing web applications through a familiar interface -- as web pages rendered in a web browser. The challenge for Chickenfoot is simply stated: a user should never have to view the HTML source of a web page in order to customize or automate it.

Chickenfoot addresses this challenge in three ways. First, it runs inside the web browser, so that the rendered view of a web page is always visible alongside the Chickenfoot development environment. Second, its language primitives are concerned with the web page's user interface, rather than its internal details. For example, Chickenfoot uses commands like click, enter, and pick to interact with forms. Third, it uses novel pattern-matching techniques to allow users to describe components of a web page (targets for interaction, extraction, insertion, or customization) in terms that make sense for the rendered view. For example, the click command identifies the button to be clicked using keywords from its text label, rather than the name it was given by the web page designer.

With Chickenfoot, users can automate web tasks, programmatically extract content from web sites, and customize how web pages are presented in their browser.

Some examples of tasks we have accomplished with Chickenfoot scripts are:

• Augmenting a webmail system so that when a user composes email, the faces of recipients are automatically displayed in the composition window.
• Highlighting SAT vocabulary words in a web page.
• Enabling users to sort HTML tables by clicking on a column header.
• Concatenating multiple pages of search results into one long page.

Current Work

Research on Chickenfoot is continuing in several directions:

• Syntax-free programming. One important usability barrier in Chickenfoot is the need to learn and correctly generate the syntax of the language, such as click("search button") or enter("all the words", "disk"). We are exploring ways to eliminate this barrier by allowing the user to enter more freeform statements that are interpreted, such as click search or even all words = disk. Syntax-free programming has the potential to be easier to learn, less error-prone to use, and potentially even more concise than tradtional programming syntax.
• Script recording. Many useful Chickenfoot scripts can simply be demonstrated by pointing, clicking, and typing into the web site itself. These actions are translated into Chickenfoot code which can be incorporated into a script. We are also using this feature to implement smart bookmarks: browser bookmarks that can return you to any page on the Web, even if the URL of the page isn't directly bookmarkable.
• Style matching. When Chickenfoot is used to augment a web page with new user interface components, e.g. inserting new buttons or content obtained from another web site, then we'd often like the inserted UI to match, as much as possible, the appearance of the rest of the page. We are exploring techniques for copying CSS style rules from one page component to another automatically.
• Making web applications more efficient to use. Many users may use only a few of the features of a complex web application, and never touch the rest. By recording which components (buttons, hyperlinks, text boxes) the user actually clicks on, we can augment the web page to make those components easier to use: enlarging the frequent targets, reducing the contrast of other components to make them less obtrusive, and varying the mouse speed so that frequent targets are easier to acquire.

Acknowledgements

This work was supported in part by the National Science Foundation under award number IIS-0447800, and by Quanta Computer as part of the TParty project. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the sponsors.

References

[1] Michael Bolin, Matthew Webber, Philip Rha, Tom Wilson, and Robert C. Miller. "Automation and Customization of Rendered Web Pages." UIST 2005, pp. 163-172. Winner of best paper award.

[2] Michael Bolin. End-user Programming for the Web. MEng thesis, Massachusetts Institute of Technology, June 2005.  Winner of William A. Martin Memorial Thesis Prize for an outstanding Master's thesis in computer science.

[3] Michael Bolin and Rob Miller.  "Naming Page Elements in End-User Web Automation." Workshop on End-User Software Engineering, ICSE 2005.

[4] Greg Little and Robert C. Miller. "Translating Keyword Commands into Executable Code." UIST 2006, pp. 135-144.