Remote Card Sorting

Back at the grindstone this week with an interesting foray into card sorting, but this time using a web application while facilitating users (one to one) over conference calls. It’s thrown up some issues, and almost fallen apart at the seams at one point, but I think it’s going to be helpful in the next stage of working out the site’s taxonomy.

It seems that IAs are beginning to polarise on the merits of card sorting. Right now it seems to be a reasonably mainstream technique, but we’re beginning to find flaws in it along the way. Best practice is that if you’re going to do it, it’s the qualitative aspect of what goes on that’s most important during the sort (although the stats analysis is fun).

However, we’re having a really hard time getting users to research on this project, and when only one (one!) actual customer turned up to our first group card sorting session a couple of weeks ago, we had to think of a new direction. So it was that we decided to try remote sessions. After some quick research into online card sorting systems we narrowed down the options to three:

WebCAT: a free web application written mainly in PHP, but it didn’t work on my WinXP machine (didn’t seem to save the results of the sorts) so we had to pass on that. I’d like to try it on a *NIX box at some point though.

WebSort: a commercial, hosted Flash application. The best UI of the ones we found.

CardSword: a free Java application – nice if rather clunky (and very “beta”).

We decided on WebSort ($99 for a one study license), using IBM’s eZCalc to do the cluster analysis from the data it spits out via email after each session. I’d used the latter tool before and am fairly familiar with it now. We then used CardSword as a fall-back in case WebSort went down. Lucky we did, since that’s exactly what WebSort did 48 hours after we paid our money. The fall-back worked using NetMeeting – I shared the application running on my workstation with the participant on each call – but it was only just fast enough for users to operate. Three days into the testing, WebSort came back on line (they’d been hacked).

In retrospect, it was good that we used two systems since each have their strengths and weaknesses. CardSword over NetMeeting had the distinct advantage of me being able to see what the user was doing. Once the user’s session has been saved, however, you can’t go back to see what they’ve done. The data can also only be analysed by CardSword’s analysis, so I had to screen-scrape each session into a spreadsheet for conversion to eZCalc format later (at the time, we assumed NetSort was dead). NetMeeting also caused the usual ActiveX permissions problems, etc. for some users, and was generally slow and flaky – putting an extra 15 minutes onto what were otherwise 45-min sessions for most people using NetSort. In one case we had a user with a Linux desktop (“It says it doesn’t support my browser.” “What browser do you have?” “I’m using Konquerer…”). Apart from some slowness to send us the data for each session, NetSort worked fine for the most part once it was running for us.

At first, we decided that the best way to run the sessions was to stay on the line with the user. While this was feasible with the NetMeeting method as we could watch the session progress, it wasn’t with WebSort, and in any case it was clear that most users wanted to be left alone for a while once they understood what to do. We typically gave them 20 mins in solitude, then came back in to finish up and handle any problems and probe a bit about the groups they’d made. Qualitative data was pretty hard to pick though and we didn’t get much of it.

Some things I’d do differently if we did it again:

– Don’t use conference calling. It’s over complicated for a one to one session.
– Leave a clear hour between sessions to wrap up, take notes and prep for the next one. We had back-to-back clumps one day, and they threatened to overlap leaving no time of any downtime. Booking spare conferences (and NetMeeting sessions!) for overflow is also hard to juggle.
– Assume users won’t read any introductory literature you give them beforehand, however brief, and include a full verbal introduction into the session. Not a single user had read the preparation document we sent them properly, so I had to put them in the picture from scratch each time.
– Turn off your screensaver while sharing apps under NetMeeting! My screensaver locks my workstation, which then also stops the NetMeeting session. We lost one user that way.
– Construct a custom application for ourselves. WebSort is nice, but it’s not reliable. I’d not want to use them again if I didn’t have to.