Research Abstracts - 2006

Making Dynamic Lists Appear Static by Preserving the Memorable Aspects

Jaime Teevan

Introduction

People commonly interact with lists of information -- search engines return lists of results, incoming emails are listed in a person’s Inbox, news stories appear as lists on newspaper Web sites, and people navigate file systems by listing directory contents. While the changes that occur to a list can be interesting -- a new search result is interesting to the person searching for new information -- changes are often secondary to the primary goal of using information. New search results are inconsequential when a person wants to summarize a set of results or return to a previously viewed Web page. This abstract investigates what aspects of a search result list are memorable and shows that these aspects should only be changed with care, while unmemorable aspects can be changed at will. Because people remember little of the result lists they interact with, we find it possible to preserve the memorable aspects while including new information.

Some search engines support re-finding by allowing people to explicitly search within information they have seen before [1, 2], but these systems do not maintain consistency in result presentation, requiring the users to take a different path to the same information. Information management systems that preserve consistency in dynamic environments permit their users to choose to interact with a cached version of their information space [3, 6]. While employing similar methods to keep the results for repeat queries static would make re-finding simple, it would deny users the opportunity to discover new information. Even though changes to the search results associated with a query can potentially hinder returning to previously viewed information, they benefit users by providing new information.

The approach taken in this abstract is to maintain the information that is important to a person while changing that information that is not. In this way, what is done is similar to a version control system [4, 5]. Version control systems try to keep what is important about one person’s version of a document stable while merging in what is important about another person’s version. Here what is preserved are memorable aspects of the information.

To understand how new search results can be incorporated unobtrusively into a previously viewed search result list, we begin with a study of what 119 people found memorable about search result lists. Based on this study, a model is developed of which aspects of a result list are memorable, and thus should be changed with care, and which are not, and can change freely. After describing the model, a follow-up study of 165 different people is presented that tested the model by asking participants whether they noticed changes that either occurred in accordance with the model or not. Changes made according to the model were less likely to be noticed than changes made naively. Further, the study suggests that apparent consistency is important, because even when the new search results were of higher quality, if the participants noticed a change, they viewed the changed result quality as worse than the original quality.

Recalling Search Results

To discover what people found memorable about search result lists, participants were asked to interact naturally with a list of search results for a self-generated query. After an interval of a half hour to an hour, participants were emailed a brief survey that asked them to remember their result list without referring back to it. Participants were asked to recall their query, the number of results returned, and basic information about each result, including its title, snippet, URL, whether it was clicked, and if so, what the Web page was like.

One hundred and nineteen people participated in the study. Because the recalled rank for a result did necessarily correspond with the result’s true rank two independent coders matched recalled descriptions to actual results, with an 84% inter-rater reliability. A result was considered memorable if sufficient information was recalled to enable both coders agreed on the mapping. Two main factors emerged from the data as affecting how memorable a result was: where in the result list it was ranked and whether or not the result was clicked. This can be seen in Figure 1.

Figure 1: The probability of recalling a result given rank. The probability generally decreases as a function of rank. Clicked results were significantly more likely to be recalled (p<0.01).

How result ordering was remembered was also analyzed, and the results are shown in Figure 2. Recalled rank differed from actual rank 33% of the time. Mistakes tended to be less common for highly ranked results, and the first result’s rank was remembered correctly 90% of the time. The greater weight of the data occurs to the right of the identity line, meaning remembered results were much more likely to be recalled as having been ranked higher than they actually were. Recalled results moved up in the result list 24% of the time, significantly more often than they moved down (10% of the time, p<0.01).

Figure 2: The result's location in the result list as the pariticpant remembered it, compared with the results's actual location. The size of each point represents the number of people remembering that combination.

Preserving What is Memorable

Preserving the feeling of consistency for an individual interacting with changing information does not require maintaining consistency with all information, only with the information that is remembered. Because the previous study showed that people remember very little about the search results they interact with, it is possible to include a large amount of new information in an old result list without disrupting the user of the information.

Merging new results into an old result list requires that the value of the new information be balanced with the cognitive cost of changing the already viewed information. Each result in the original list is assigned a memorability score that represents how memorable the result is based on the previous study. Additionally, each new result that could be added to the original result list is given a benefit of new information score based on the expected benefit the as-yet-unseen result will provide to the user. All permutations of possible final lists that include at least three old results and three new results are considered, and the best is selected.

Testing the Merged List

To test how well lists merged in this way preserve the memorable aspects of the original search result lists, a study was run to investigate how often a changed list looked unchanged to someone who had interacted with the original list. The experimental setup was similar to the first study, except that participants were asked to recognize their original result list rather than recall it. In the follow-up survey participants were presented with a new search result list and asked whether it was the same as the list they saw initially or not.

One hundred and sixty-five people participated in the study. The study showed that intelligent merging enabled new results to be included virtually unnoticed while changes included with na?e mergings were often noticed. Results can be seen in Table 1. This finding suggests it is indeed possible to sneak new information into a result list.

While it seems apparent from the study that new information can be unobtrusively merged into previously viewed search result lists, it is not obvious that people want new relevant results to look the same as old results. When a person searches, he or she is looking for relevant material, so it could be that it is best just to return relevant results regardless of past context.

To explore whether noticeable change is problematic, the perceived quality of the old and new results presented to the participants was compared. The new results incorporated into the original list were ranked higher by the underlying search engine, and thus were more relevant. The new result list was judged by an independent coder to be better than the original result list 81% of the time. Nonetheless, when the participants noticed a change, they were significantly less likely to find the changed result list to be better than the original result list (46% of the time, p<0.01). Further, they thought the changed result list was worse 14% of the time. This indicates that consistency of the type explored in this abstract is likely to be important for providing results that appear relevant.

Conclusion and Future Work

This abstract has demonstrated how to take advantage of the fact that people remember little of what is presented to them to provide new information in an unobtrusive fashion. It is likely that the methods for dealing with change presented here can be applied to many situations beyond search results, and in particular to other interactions people have with lists of information, such as lists of directory files, to-do items, message board threads, news stories and navigational links. While some evidence has been presented that suggests suppressing obvious change makes for a better user experience, the model developed of what is memorable need not only be used to hide changes. It could, for example, be used to highlight important changes. We look forward to further studying when changes hinder user interaction and when they help.

References:

[1] A9. http://www.a9.com

[2] Susan T. Dumais, Edward Cutrell, JJ Cadiz, Gavin Jancke, Ramin Sarin and Daniel C. Robbins. Stuff I’ve Seen: A System for Personal Information Retrieval and Re-Use. In Proceedings of SIGIR '03, pp. 72--97, August 2003.

[3] Koichi Huyashi, Takahiko Nomura, Tan Hazama, Makoto Takeoka, Sunao Hashimoto and Stephan Gudmundson. Temporally Threaded Workspace: A Model for Providing Activity-Based Perspectives on Document Spaces. In Proceedings of HyperText '98, pp. 87--96, 1998.

[4] David L. Hicks, John J. Leggett, Peter J. N?nberg and John L. Schnase. A Hypermedia Version Control Framework. ACM Transactions on Information Systems (TOIS), 16(2): pp. 127--160, 1998.

[5] Kasper Østerbye. Structural and Cognitive Problems in Providing Version Control for Hypertext. In Proceedings of HyperText '93, pp. 33--42, 1993.

[6] Jun Rekimoto. Time-Machine Computing: A Time-Centric Approach for the Information Environment. In Proceedings of UIST '99, pp. 45--54, 1999.