Motivation

On the desktop, an application can expect to control its user interface down to the last pixel, but on the World Wide Web, a content provider has no control over how the client will view the page, once delivered to the browser. This creates an opportunity for end-users who want to automate and customize their web experiences, but the growing complexity of web pages and standards prevents most users from realizing this opportunity.

Approach

Chickenfoot is a scripting environment embedded in the Firefox web browser that enables end-user programmers to customize and automate their interaction with the web by talking about objects they see in a web page.

Most approaches to web automation and screen scraping require the programmer to understand the underlying structure of the web page because they require the user to write syntactic patterns, such as regular expressions or XPATH, in order to identify elements in a web site. Chickenfoot scripts differ because they support the use of semantic patterns, so the user never needs to view the raw HTML of the web page. For example, the name of the HTML element for Yahoo!'s search box is "p," so other automation systems may let the user programmatically set the contents of the search box by writing something like the following:

  html.body.p[3].table.tr[2].input.p = "search terms"

  enter('search box', 'search terms')

The former is tedious to write because it requires the author to plumb through several hundred lines of machine-generated HTML to find the name for the textbox. Further, this script will fail if Yahoo! decides to change the textbox's name or its location in the page.

By contrast, the Chickenfoot code is easier for an end-user programmer to write and maintain because he simply needs to view the web page to write the script. Additionally, Chickenfoot scripts are more robust to changes in the web site.

With Chickenfoot, users can automate web tasks, programmatically extract content from web sites, and customize how web pages are presented in their browser.

Some examples of tasks we have accomplished with Chickenfoot scripts are:

• Highlighting SAT vocabulary words in a web page.
• Hyperlinking Java types in sample Java code to their Javadoc API.
• Enabling users to sort HTML tables by clicking on a column header.
• Concatenating multipage pages of search results into one page.
• Converting web pages into black-and-white so they are easier to read.
• Appending icons after hyperlinks that reflect the file type of the target of the link.

References:

[1] Chickenfoot. http://www.bolinfest.com/chickenfoot/

[2] Michael Bolin, Matthew Webber, Philip Rha, Tom Wilson, and Robert C. Miller. "Automation and Customization of Rendered Web Pages." Submitted to UIST 2005.