Usability and Understanding

User testing in London and Milan last week. The scripts we’re using for this are pretty complicated, and the client wants us to cover off a lot of very specific questions about the system, which was pretty tough to do while making sure the user was relaxed enough to give us reasonably truthful answers.

This has led to some complaints from the client that I’ve been asking users the dreaded “leading questions.” On at least one of the sessions, I did find myself lapsing into instructional mode – a bit of a basic mistake for a test facilitator – but that was more to do with the fact that the prototype we were using was so tricky: you have to make sure the user does the “right” things to get into the “right” situation in order to ask some of the questions. For some reason I found myself explaining to the user what I was explaining to myself about how to use the system.

But my failings as a test facilitator aside, it was clear that after a point, if you need to establish whether a user really, truly understands the finer points of what’s going on, there is only so much probing you can do before the poor bugger begins to think that they’re stupid. I’ve been very careful to tell users it’s not a test of them, etc. but more than two or three rounds of “what do you think about…” and “I see, but what if you did …” probing around the same area, and it begins to feel a bit chilly in that camera-infested studio with its large on-way mirror and “relaxing” potted plant.

Perhaps more rehearsing would have allowed me to come up with some lines of questioning to winkle out the coveted understanding. Or maybe it’s just more experience I need. But I get the impression that understanding a system isn’t necessarily a prerequisite for usability. It’s really just about satisficing. For example, when I save an Excel document as CSV, I “understand” that Excel flips up a dialogue with three choices about something. I’ve never bothered to actually read that dialogue properly and work out why it’s asking me that – I just know to press OK and things work out fine. The other choices are probably to cover off edge cases like saving as CSV by mistake and losing formatting etc. but I don’t know. If you asked me if I thought the process of saving as CSV was easy, I’d say it was. There’s obviously something I don’t understand about it, but that doesn’t bother me. Similarly, I suspect my mother doesn’t really understand the file system on her computer – she just saves everything in My Documents* and that’s it. I doubt she would either know nor care how to navigate to the Desktop to save or open something. For that matter, I don’t understand how the NTFS file allocation table works, but I know how to manage files on it using both GUI and CLUI. How far do you have to understand a system before you can use it?

So it is with user testing. Our client is very concerned that we make sure we know whether users fully understand the system and is frustrated I can’t deliver the answer to that question. I wish I had the guts to ask them why they want to know that, but I’m taking their money, so I won’t. I am the Gutless Wonder of the web.

————–

* I wish Microsoft had come up with a better name though. Phone calls after my mum got her new computer went something like: “Just save it in My Documents”, “Your documents, darling?” “No, My Documents… er. No, the place that’s called My Documents, but is in fact yours.”