Eye tracking: meaningful or sexy?

By Ruairi Galavan | 9 Comments

We thought it was about time we found out for ourselves if eye tracking is more than pretty heat maps. Can it provide any real value? Is worth the cost? And how does it compare to standard user testing in terms of overhead, prep, planning, execution, and analysis?

Through our research, we found some real value in our eye tracking testing – both in the results and the process itself.

a gaze plot on the New York Times

Retrospective vs concurrent think aloud

Retrospective Think Aloud (RTA) is the method by which we allow users to complete their user test in silence, then replay their eye gaze and allow them to recall their experience. Users can think aloud after the test, as opposed to during it (known as Concurrent Think Aloud).

We wanted to find out if RTA would allow users to behave more naturally, leading to better (and more realistic) test results, thus leading to better design decisions.

Wait, what are you doing to my eyes?

We also had some concerns over what effects the presence of an eye tracker might have on user behaviour. Would users be scared of it, put off by it, or simply in awe of seeing their own gaze replay on screen?

The test

We tested the usability of four news media websites. Users were tasked with accessing football-related content.

gaze plot of the Irish Times

But while we were evaluating how the sites performed against each other, we were more concerned with the eye tracking process.

What additional insights can eye tracking provide?

Through eye tracking, we could see how quickly users found content (i.e. football results) on BBC, but really struggled to find it on Irish Times.

It wasn’t the use of eye tracking that uncovered this issue (we’re pretty sure we would have come to the same conclusion through standard user testing). There were two real benefits to the eye tracking method.

  1. eye tracking data allowed us to illustrate our findings in a new and compelling way
  2. RTA – the much bigger benefit

Staying quiet works

The RTA method allowed users to complete the tasks without any interruptions, which meant they completed them more quickly and more naturally than a standard user test. And the post test ‘think alouds’ was where we got the insight: the gaze replay acted as a great memory-aid for the users, and provided good context for their retrospective reasoning.

Because users could do the test more quickly, we could do more tests — which means eye tracking allows for the potential of a large scale study.

The process

The test design and preparation took a little longer than usual, simply because of the equipment. We suspect that once we’re used to it, the prep time will decrease.

The test facilitation felt very different to our normal method, where we ask many questions as users complete tasks. Conversely, during the eye tracking tests, we took notes, which we later asked the users as they watched their eye-tracking play back.

We quickly learned the value of the pause button (and speed control) during the RTA. The playback was often too fast for the user to keep up with. So slowing down or pausing allowed us to explore certain things in more detail.

While the test themselves take less time, the post-test analysis may very well prove to take longer, if we decide to leverage on any of the many analysis tools that the software provides.

Consensus

Our first impressions of eye tracking have been positive. We see a meaningful analysis with this type of testing. We are already conducting our next study and will be sharing our results, so watch this space. And please share your thoughts and experiences with eye tracking.

 

9 responses to “Eye tracking: meaningful or sexy?

  1. Really interesting stuff. How did you find the calibration process? Do you think that interfered with the user performing their task?

    I completely agree that retrospective is the way to go. I would also be interested to hear what you think is the best way to visualise the results. heatmaps vs gaze-replays, etc.

  2. Sounds like a solid study. Eye tracking is very useful when it is used in the proper way. Did you produce aggregated heat maps for the sites? would be interesting to see those if you update the post…

  3. Hi Ronan,

    Having conducted about 30-40 user tests with eye tracking so far, I don’t feel the calibration process has had any real effect on any user behaviour. Its very quick and easy – it takes about 10 seconds.

    The best way to visualise the results really comes to down to what you are trying to say. Single gaze plots are useful for telling the story of one users experience, whereas heat maps are useful for showing how multiple users engaged with a page.

    But again, I think the real value in eye tracking is how it lends itself to RTA.

  4. RTA can be done without eye track data, but I do think it would be less effective. We are still looking into this, but the question really is how much better is the use of eye track data for RTA. In 2007, Lancaster researchers published results from a study exploring exactly this. Their results indicated that RTA done with eye tracking uncovers a significantly larger amount of usability problems.

  5. I think Think Aloud method is more natural way than RTA. You can’t just tell the participants – “we’re working in silence”. Moreover, in-lab testing it is not a natural environment for a user. But it is a very interesting approach. We have to try.

  6. Hi Dimitry,

    I have to say I disagree. Working in silence is definitely a more natural way of working for the user. We can never achieve an entirely natural setting in the lab, but we always strive to make it as natural as possible. I would be interested to hear your thoughts on this once you have tried the RTA method.

  7. Tobii Technology recently published a white paper comparing four different RTA methodologies which might be intersting to read for you guys. It can be found on their website: http://www.tobii.com/en/analysis-and-research/global/library/white-papers/

    I will be conducting my first RTA study next months. Ruairi how do you typically initiate the RTA session? Do you use an interview script or do you rely on the notes you took while conducting the eye tracking sessions?

  8. Hi Alex,

    Thanks for your comment. We are familiar with these papers already but we felt we needed to investigate things for ourselves too.

    Depending on the test, we would either have a guide script with us of items we are interested in, or we would rely on the notes we took during the test. But more often, it’s a combination of both of these. You can have a script ready, or areas you want to investigate, but other things will always pop up during the test that you want to look at more closely.

Comments are closed.