Transcript: Ask the UXperts: Usability Testing — with Cindy McCracken

Yesterday, by popular demand, we hosted a session in our Slack channel on the subject of usability testing. Our UXpert was Cindy McCracken and she did a fantastic job.

As well as answering questions and handing out valuable advice, Cindy compiled this handy list of resources:

Reading List

• Understanding Your Users by Kathy Baxter, Catherine Courage and Kelly Cain
• Observing the User Experience by Mike Kuniavsky
• Handbook of Usability Testing by Jeffrey Rubin and Dana Chisnell
• The Moderator’s Survival Guide by Donna Tedesco and Fiona Tranquada
• Measuring the User Experience by Tom Tullis and Bill Albert
• Beyond the Usability Lab by Bill Albert, Tom Tullis and Donna Tedesco

Tools for all kinds of testing

• http://ift.tt/15fmjFc
• http://ift.tt/2gEFdjf

User Research Techniques

• http://ift.tt/15URtOX
• http://ift.tt/1YC0NYI

Guidelines for prioritising study findings

If you didn’t make the session because you didn’t know about it, make sure you join our community to get updates of upcoming sessions. If you have follow up questions for Cindy, you can ask them over on our community forums.

If you’re interested in seeing what we discussed, or you want to revisit your own questions, here is a full transcript of the chat.

Transcript

hawk

2016-11-21 17:27

Session starts at 3 pm Wednesday 30 November PDT (or 10 am Thursday 1 December AEST)
You can use the command /tz help to get time zone conversion assistance here on @slackbot3

hawk

2016-11-29 22:55

The beginner’s guide to usability testing: http://ift.tt/2gugHRQ

jacqui_dow5

2016-11-30 21:56

I love usability testing!

hawk

2016-11-30 22:27

For anyone killing time before the session: http://uxmas.com/

hawk

2016-11-30 22:57

Quick overview of how these things go down:

hawk

2016-11-30 22:58

– I’ll introduce @cindy.mccracken
– Cindy will give an overview of the topic and run through some definitions
– We’ll throw it open to you for questions

hawk

2016-11-30 22:59

If things get crazy, I’ll queue your questions in a back channel and Cindy will answer them as she gets through them

jasmine

2016-11-30 23:00

Hey just heard that there’s a session happening today? How do I join?

jonny.bennett

2016-11-30 23:00

you’re in it @jasmine!

hawk

2016-11-30 23:00

We’re just about to kick off so your timing is perfect :slightly_smiling_face:

jasmine

2016-11-30 23:00

Oh! Haha, this is my first time. :smile:

lukcha

2016-11-30 23:01

Welcome @jasmine!

jasmine

2016-11-30 23:01

Thanks! Looking forward to it! :slightly_smiling_face:

hawk

2016-11-30 23:01

Ok, show time!

lukcha

2016-11-30 23:01

We’re all in for a treat. :slightly_smiling_face:

hawk

2016-11-30 23:01

First up, a huge thanks for @cindy.mccracken for your time today – we really appreciate it

hawk

2016-11-30 23:01

And thanks to the rest of you for joining us :slightly_smiling_face:

hawk

2016-11-30 23:02

Cindy recently published a great beginner’s guide on usability testing for us – you can find it pinned in this channel

hawk

2016-11-30 23:02

The formal intro: Cindy McCracken has worked in UX more than 10 years and is in her element when planning studies, conducting research, and analysing data.

hawk

2016-11-30 23:02

Currently consulting with User-View, Inc., focused on UX in the medical and financial fields, she has worked as a senior user researcher at Fidelity Investments, BB&T and iContact. Cindy earned a master’s degree in information science from the University of North Carolina-Chapel Hill.

hawk

2016-11-30 23:03

When she’s not working, you can find her hanging out with her 9-year-old daughter, reading historical fiction, or winding down in a yoga class.

hawk

2016-11-30 23:03

We asked her to come today to talk usability testing because it’s something that you (our community) requested.

hawk

2016-11-30 23:03

So @cindy.mccracken – over to you for an intro to the topic

cindy.mccracken

2016-11-30 23:03

Thanks for the intro, @hawk!

cindy.mccracken

2016-11-30 23:04

Hi everyone! I’m excited to have a conversation today about the details of usability testing.

cindy.mccracken

2016-11-30 23:04

I did my first usability tests for a non-profit when I took over for their webmaster in 2003.

cindy.mccracken

2016-11-30 23:04

I read the book “Designing Web Usability” by Jakob Nielsen, conducted tests, and was hooked.

cindy.mccracken

2016-11-30 23:05

So usability testing became a focus of my career.

cindy.mccracken

2016-11-30 23:05

Usability testing is a very useful technique for learning how well your designs work for people, but the more you get into it, you realize there’s always room for improvement.

cindy.mccracken

2016-11-30 23:05

You can always get better results.

cindy.mccracken

2016-11-30 23:05

There’s a lot to getting the right participants, writing effective tasks, and observing while remaining neutral.

cindy.mccracken

2016-11-30 23:06

The majority of usability testing is called formative testing – where you’re mainly concerned with learning how your product could be improved.

cindy.mccracken

2016-11-30 23:06

You come up with goals to test, and write realistic tasks for participants to get answers to your questions.

cindy.mccracken

2016-11-30 23:07

Of course, there are a lot of variations on the traditional in-person study.

cindy.mccracken

2016-11-30 23:07

For example, remote moderated testing. That gives you a lot more options for recruiting.

cindy.mccracken

2016-11-30 23:07

And then there’s the testing of mobile apps, which is very important these days.

cindy.mccracken

2016-11-30 23:08

And remote unmoderated testing – where you set up tasks in an online tool like UserZoom for hundreds of participants.

cindy.mccracken

2016-11-30 23:08

But for all types of usability testing, you need five things:

cindy.mccracken

2016-11-30 23:08

a design to test, participants, a test plan, a moderator (or test tool), and findings

cindy.mccracken

2016-11-30 23:09

For the design, remember to test throughout the process – from paper sketches to high-level prototypes. That lets you catch issues early.

cindy.mccracken

2016-11-30 23:10

For recruiting, consider who your target users are and where they are, then ask screener questions if you need to be specific

cindy.mccracken

2016-11-30 23:10

The test plan should include all the details of the study, such as the goals, session information (such as how observers can log in), all the tasks, etc.

cindy.mccracken

2016-11-30 23:11

When moderating, of course follow the test plan. Remember to stay neutral in words and actions; make participants feel comfortable; and remind people to talk if they’re not so you can learn.

cindy.mccracken

2016-11-30 23:12

Finally, with findings and presentations, prioritize your recommendations (low, medium, high), and definitely meet with stakeholders in person

cindy.mccracken

2016-11-30 23:12

In person is important so you can make sure the results are being interpreted correctly, and also so you can discuss results with business goals, etc.

cindy.mccracken

2016-11-30 23:13

Remember that in usability testing you are finding problems, not solutions.

cindy.mccracken

2016-11-30 23:13

So don’t get into talking about solutions at your debrief session.

nat

2016-11-30 23:14

There are many types of testing that you can do, do you have a key few you tend to go for first?

cindy.mccracken

2016-11-30 23:14

These are all high-level ideas. I’m curious why you’re here today … what you’re interested in.

cindy.mccracken

2016-11-30 23:14

Yes, I think moderated testing – either in-person or remote – is the best first way to get qualitative feedback

cindy.mccracken

2016-11-30 23:14

You can get great qualitative results about “why” your designs are and aren’t working well.

cindy.mccracken

2016-11-30 23:15

You can do a cafe study to be even quicker.

jacqui_dow5

2016-11-30 23:15

Do you find the findings in remote testing and in person testing give you the same validity?

cindy.mccracken

2016-11-30 23:15

If you do a cafe study, it should be very brief – like 3-5 minutes.

cindy.mccracken

2016-11-30 23:15

Just ask one or two most important tasks.

cindy.mccracken

2016-11-30 23:16

I have found remote testing to be very useful … a lot of times it’s been the only way I could get the right participants.

jacqui_dow5

2016-11-30 23:16

We are having the same

cindy.mccracken

2016-11-30 23:16

The main thing you miss is seeing people’s expressions. Try to make sure they’re talking a lot.

cindy.mccracken

2016-11-30 23:17

You can try to get people to share their faces with their webcams, but sometimes that can feel invasive or awkware

cindy.mccracken

2016-11-30 23:17

awkward

jacqui_dow5

2016-11-30 23:17

We’ve had it where multiple people have been doing the test at once, should we try and prevent this?

cindy.mccracken

2016-11-30 23:17

Yeah, remote is a great way to get more access to people. The incentive can be lower too.

cindy.mccracken

2016-11-30 23:17

You mean two or three people at a computer.

jacqui_dow5

2016-11-30 23:18

Yes! We had one a few weeks ago where we think there were 4 people in the same room around the PC

cindy.mccracken

2016-11-30 23:18

It can just get confusing … but I’ve had that happen too when it was a very import person who wanted others involved.

mpcnat

2016-11-30 23:18

How do you handle usability testing of a mobile app that is being tested by a remote user, and due to logistics and comms issues the facilitator is an account manager from your own company. What tips do you suggest to get some success out of the process?

jacqui_dow5

2016-11-30 23:18

I was worried if one was a ‘boss’ it may influence the others

cindy.mccracken

2016-11-30 23:18

You can try to get them to do it separately if it makes sense

cindy.mccracken

2016-11-30 23:18

Or if it doesn’t, ask that just one person be in control and talking.

lynne

2016-11-30 23:19

Can you explain what you mean by a cafe study? I can make a guess, but it’s the first time I’ve heard this term…

cindy.mccracken

2016-11-30 23:19

so … the remote testing is in person, but the AM is the facilitator?

cindy.mccracken

2016-11-30 23:19

oh wait, remote

cindy.mccracken

2016-11-30 23:19

how is it being done remotely? what method?

cindy.mccracken

2016-11-30 23:20

@jacqui_dow5 – that’s why if they could do it separately it would be better.

cindy.mccracken

2016-11-30 23:20

it’s like in a focus group … where one person might take over

hawk

2016-11-30 23:20

Note: I’ll acknowledge questions that have been queued with a :grey_question:

cindy.mccracken

2016-11-30 23:21

@lynne – a cafe study is where you set up in a cafeteria, or at a mall or somewhere with your design.

cindy.mccracken

2016-11-30 23:21

You have an intro and a task or two for people.

jacqui_dow5

2016-11-30 23:21

Is cafe study the same as geurilla testing?

cindy.mccracken

2016-11-30 23:21

Then you have people do the task on your computer or device, and pay them with a small gift card or something.

cindy.mccracken

2016-11-30 23:22

It’s just usually to get quick feedback on a particular interaction.

cindy.mccracken

2016-11-30 23:22

yes

kaydeecarr

2016-11-30 23:22

What is your technique for analyzing the data after you’ve done the testing?

cindy.mccracken

2016-11-30 23:22

@kaydeecarr – I spend time making sure I take clean notes that are organized and will be easy to analyze.

cindy.mccracken

2016-11-30 23:22

columns for participants / rows for questions.

cindy.mccracken

2016-11-30 23:23

Then I go through and add rows to count the way people behaved (accomplish task, etc.)

cindy.mccracken

2016-11-30 23:23

and add those (this is all in excel)

cindy.mccracken

2016-11-30 23:23

Also, just keeping track of big issues related to goals as we go through the test sessions.

cindy.mccracken

2016-11-30 23:23

Then at the end go through to find data to back up those findings.

cindy.mccracken

2016-11-30 23:24

but I go through everything in case I missed something.

cindy.mccracken

2016-11-30 23:25

@mpcnat How do you handle usability testing of a mobile app that is being tested by a remote user, and due to logistics and comms issues the facilitator is an account manager from your own company. What tips do you suggest to get some success out of the process? (remote participant, local account manager facilitator)

cindy.mccracken

2016-11-30 23:25

just repeating this question.

cindy.mccracken

2016-11-30 23:25

what is the process? can you see the device?

cindy.mccracken

2016-11-30 23:26

are they using webex on an ipad? something like that?

cindy.mccracken

2016-11-30 23:26

or holding their hands in front of a laptop’s webcam?

cindy.mccracken

2016-11-30 23:26

@mpcat – what issues / concerns are you noticing with the facilitator being an AM?

jacqui_dow5

2016-11-30 23:26

I’ve seen examples of where people attach a make shift camera holder to a mobile to record the screen? But would rely on the user making this and having a camera!

nik

2016-11-30 23:27

Is there a big difference between reports/presentations that are based on moderated VS unmoderated tests? It’s my impression that most insights from unmoderated tests comes from direct quotes whereas moderated tests include a lot of other factors, such as facial expressions, body language etc. And how does this affect the conclusions of a study? It seems to me that moderated tests will depend more on “personal interpretation” which can be more difficult to support scientificly.

cindy.mccracken

2016-11-30 23:27

Huh! That sounds interesting.

lukcha

2016-11-30 23:28

Mr Tappy!

cindy.mccracken

2016-11-30 23:28

@nik – In unmoderated tests, using UserZoom or Loop 11, I think most of the results are quantitative

mpcnat

2016-11-30 23:28

It’s a prototype app on a mobile device that the AM has in front of the customer, they have a facilitation script to assist them, that they take notes against whilst the customer performs tasks. The issue is that an AM isn’t familiar with usability Testing, but is great with the relationship, and they feel out of their depth, so do we cut down the tasks to test, but then then that means we lose insight with that customer, that we only see on an irregular basis

cindy.mccracken

2016-11-30 23:28

You would do them if you wanted greater certainty that certain things were problems, or to compare two different designs quantitatively.

cindy.mccracken

2016-11-30 23:29

So I would expect the results to be more charts, that sort of thing.

cindy.mccracken

2016-11-30 23:30

@nik – I don’t know if I’m answering your question. But in moderated testing, you still have findings – like whether people could complete tasks, where they got stuck, etc. So you have real facts to report. In addition to frustrations.

cindy.mccracken

2016-11-30 23:31

@mpcat – Is it in person then? It would help if someone else were taking notes so the AM could focus on facilitating.

jacqui_dow5

2016-11-30 23:31

I have used whatusersdo for unmoderated it was good as the users had a webcam so you got a similar video to moderated testing

cindy.mccracken

2016-11-30 23:31

It gets to be a lot to juggle.

cindy.mccracken

2016-11-30 23:32

Even better, have the AM introduce them to a UXer and have the UXer conduct the test. It can get overwhelming.

cindy.mccracken

2016-11-30 23:32

@jacqui_dow5 – Very cool! I’m going to take a look at whatusersdo. Thanks!

mpcnat

2016-11-30 23:32

yep it’s in person, sometimes there is someone else taking notes, but the feedback from the AM is “can’t you just give me the top 10 things you want to see, there is too much to watch out for”, which I get but since we may only get one go at it with them, it seems like we miss out, the script has tasks and also things to watch for (reactions) from the user on particular screens, this is where they feel overloaded.

cindy.mccracken

2016-11-30 23:33

Have you tried http://usertesting.com? You can create a mobile test for their testers to do on mobile.

hawk

2016-11-30 23:33

If you’re new to these sessions, you can jump in with questions at any time – they don’t have to be in context.

kaydeecarr

2016-11-30 23:33

Do you have any suggestions on what to do when users ask you questions? Like “how do I get to the homepage”?

robbin

2016-11-30 23:34

What’s a good way to practice writing up some questions? I’m new, so my worry is that I’m accidentally going to ask some leading questions.

cindy.mccracken

2016-11-30 23:34

@mpcat – Yeah, that’s definitely a lot to manage. When others take notes, and he doesn’t have to worry about writing down all the reactions, that doesn’t help?

cindy.mccracken

2016-11-30 23:35

@kaydeecarr – You can remind them that you’re there to see how they would do things if they were in their own environment. Ask where they think it would be … or where they would expect to find it

mpcnat

2016-11-30 23:35

It helps but sometimes it’s an AM by themselves on a site in the back of remote QLD :slightly_smiling_face: Thanks for the answers

lynne

2016-11-30 23:35

@mpcnat – can you get the AM to do a practice run with other people in the office? Might make them feel more comfortable with the task.

cindy.mccracken

2016-11-30 23:35

@robbin – definitely don’t ask users what they want … they don’t know.

mpcnat

2016-11-30 23:35

@lynne yep that’s how we are going forward with it. Thanks

cindy.mccracken

2016-11-30 23:36

know what important tasks for people to accomplish are … and ask tasks around that. be specific – like find a vacuum cleaner that gets good ratings and is in a particular price range.

cindy.mccracken

2016-11-30 23:36

What questions are you worried might be leading?

cindy.mccracken

2016-11-30 23:37

also you can follow up each task with a rating question – like how easy or difficult was that task (on a scale)?

robbin

2016-11-30 23:37

When I was practicing with the UX team, I was really tempted to ask things like, “But how do you think you would go back to the homepage?” and when I listened in, I also heard things like, “What do you think this button would do? Go back to the homepage?”

cindy.mccracken

2016-11-30 23:38

Try to keep them focused on accomplishing the tasks. If they’re looking at a button and not commenting, you can say, “What are you thinking?”

cindy.mccracken

2016-11-30 23:39

You’re right – that was leading. :slightly_smiling_face:

cindy.mccracken

2016-11-30 23:39

That is tempting … just try to stay calm and think before you speak. It’s hard to have restraint. I know.

nik

2016-11-30 23:39

You said “prioritize your recommendations”. What do you base this type of evaluation on? Especially, as an external consultant. Do you need access to certain data (e.g. visitors and clicks on specific websites or similar)?

robbin

2016-11-30 23:40

That’s what I thought :smile: I also wondered – how do you help the user feel a little more at ease? When I watch people, sometimes they feel really sheepish and I’m unsure if they’re being truthful. I’ve also been the tester before, so I’m used to saying exactly what’s on my mind because I personally don’t feel weird about it.

cindy.mccracken

2016-11-30 23:40

@nik – that’s a good question. I’ve always had a guide as to what makes something a critical issue vs. high or low, etc.

cindy.mccracken

2016-11-30 23:41

After this session, I can come up with something for you.

cindy.mccracken

2016-11-30 23:41

@robbin – Just try to relax and be friendly, and have a little chitchat at first. And definitely start with easy background questions.

cindy.mccracken

2016-11-30 23:41

Maybe offer them a drink. something like that.

cindy.mccracken

2016-11-30 23:42

@robbin – why don’t you think they’re being truthful?

cindy.mccracken

2016-11-30 23:42

do you think they’re trying to please you?

cindy.mccracken

2016-11-30 23:42

(that was leading)

robbin

2016-11-30 23:42

Hahah! Yeah, I think so – they sort of keep looking at me as if to ask, “Was that right?”

nik

2016-11-30 23:42

@cindy.mccracken That would be very interesting to see! I find myself struggling a bit with this and sometimes spend too much time on issues that might not be very critical

cindy.mccracken

2016-11-30 23:43

@robbin – Yes, that does happen! People think there are right answers. So you can definitely remind people that there really are no right or wrong answers.

cindy.mccracken

2016-11-30 23:43

… that you’re testing the software, not them.

hawk

2016-11-30 23:43

@cindy.mccracken I’d be interested in hearing about some ways to present your findings

cindy.mccracken

2016-11-30 23:43

You want to make sure the design is going to work for them, so you want their open and honest feedback. Really.

cindy.mccracken

2016-11-30 23:44

@hawk – I have done presentations – or results – several ways. One way that didn’t work well in my environments was a long report.

cindy.mccracken

2016-11-30 23:44

I don’t recommend that.

cindy.mccracken

2016-11-30 23:45

I tend to come up with a template for a PowerPoint report, and use that. I love using images and things like callouts to point out findings in the interface.

cindy.mccracken

2016-11-30 23:45

Sometimes color coding helps people see good vs. bad findings too.

cindy.mccracken

2016-11-30 23:45

I’ve tried bulleted lists in email, which can sort of work for agile.

cindy.mccracken

2016-11-30 23:45

But no matter what, it’s critical to present results in person, even if it’s in a casual setting.

jacqui_dow5

2016-11-30 23:45

We’ve had an issue where we are redesigning a system. Our current users have some preconceptions (we are hard to use, so things long winded) so when they test the new software it throws them when something is easy to do and they’ve started doubting themselves thinking we are tricking them! Any way of handling this?

cindy.mccracken

2016-11-30 23:46

You want to make sure everyone’s interpreting correctly and is on the same page about the findings – and agree on how they’ll proceed.

cindy.mccracken

2016-11-30 23:47

@jacqui_dow5 – Tricking them? I’ve never heard of that. I would just try to explain at the beginning of the test that you’ve heard issues and you’re redesigning because of them.

cindy.mccracken

2016-11-30 23:47

Maybe that will help their expectations? Or did you already try that?

jacqui_dow5

2016-11-30 23:48

Yes one comment was along the lines of ‘it’s a lot harder normally, I feel like I need to do more’

hawk

2016-11-30 23:48

There is 10 mins left in the session. If you’re sitting on a question, now is the time to ask!

jacqui_dow5

2016-11-30 23:48

We explain at the start why we are doing it and that this will be new and improved on what they’re used to!

nik

2016-11-30 23:49

Is it important to include “good findings” as well as “bad findings”? Or is it good enough to focus on problems in a report

cindy.mccracken

2016-11-30 23:49

@jacqui_dow5 – interesting to take note on that. that could be a good quote. Oh – which I forgot to mention … I love quotes in presentations! They can really be effective at conveying attitudes.

cindy.mccracken

2016-11-30 23:49

@nik – I actually always try to do both. Definitely.

cindy.mccracken

2016-11-30 23:49

The designers have worked hard on the product, and also they just need to know what IS working well so they keep those things.

cindy.mccracken

2016-11-30 23:49

It shouldn’t be all doom and gloom.

nik

2016-11-30 23:51

@cindy.mccracken That makes sense, I’ll keep that in mind. I guess including some bright sides will also make the team more inclined to want to involve you in a project again some other time :slightly_smiling_face:

bleke

2016-11-30 23:51

@cindy.mccracken Steve Krug talks a lot about getting decision makers to view the sessions. Is it something you normally do?

cindy.mccracken

2016-11-30 23:51

@jacqui_dow5 – Huh. Maybe follow up their – I need to do more – with a question. “Why is that?” or just get them to elaborate so you can learn more from it.

cindy.mccracken

2016-11-30 23:51

@bleke – I’m glad you brought that up!

jacqui_dow5

2016-11-30 23:52

Yes, good idea! It completely threw up in the test!

cindy.mccracken

2016-11-30 23:52

Yes, definitely have observers – as many as possible. It really does help with buy-in and general understanding of the problems when people observe.

jacqui_dow5

2016-11-30 23:52

Threw us*

cindy.mccracken

2016-11-30 23:52

You can simply share your screen and have people log in to the screen-share tool. Have them ask questions of the note-taker or through a different channel so participants aren’t distracted.

cindy.mccracken

2016-11-30 23:53

@jacqui_dow5 – I wondered what that meant.

jacqui_dow5

2016-11-30 23:53

Sorry it’s getting late here!!

cindy.mccracken

2016-11-30 23:53

@bleke – Definitely encourage people to come to as many sessions as they can attend.

cindy.mccracken

2016-11-30 23:53

@jacqui_dow5 – That’s OK! I’m glad you came!

bleke

2016-11-30 23:54

@cindy.mccracken Ok, thanks! :slightly_smiling_face:

lynne

2016-11-30 23:54

I recently attempted some usuability tests in a classroom setting, with multiple high school students using our site at once. It was very hard to manage – we got some good data, but I’m wondering if you have any suggestions on best practices for a session like this?

cindy.mccracken

2016-11-30 23:54

Re: observers – have a few minutes at the end where you ask observers if they have questions, and then ask them of the participant.

cindy.mccracken

2016-11-30 23:54

You don’t want to be interrupted by observers.

cindy.mccracken

2016-11-30 23:54

@lynne – was there a reason for multiple students using the site at once?

cindy.mccracken

2016-11-30 23:55

Like, is that how they normally use it/

cindy.mccracken

2016-11-30 23:55

?

nik

2016-11-30 23:55

One last question from me: Often management views usability testing as a box that should be ticked at the end of product development. Any advice for how to convince management that it should be done as early as possible in a project?

nik

2016-11-30 23:56

I’m hoping there’s one Magic Argument that always works

cindy.mccracken

2016-11-30 23:56

@lynne – It is hard to capture exactly who’s saying what. But if it’s necessary for multiple to be using, maybe focus on their interactions and how they’re doing it.

lynne

2016-11-30 23:56

Yes, it is used by schools in class. But the main reason was because we had an opportunity to do so and it’s hard to get into schools so we tried to make the best of it.

frankenvision

2016-11-30 23:57

Is it ok to make changes after a single usability test? If they are obvious blunders?

cindy.mccracken

2016-11-30 23:57

@nik – hmmm… I wish that too! More or less, do usability testing early as often as possible – even if very simple, affordable ones – to prove how it works and how important it is.

cindy.mccracken

2016-11-30 23:57

In other words, show don’t tell.

cindy.mccracken

2016-11-30 23:57

That worked really well for me at one company where they weren’t convinced of the value until we proved it.

cindy.mccracken

2016-11-30 23:58

If not that, find some good case studies to share …

cindy.mccracken

2016-11-30 23:58

And a lot of data that shows starting early is a best practice. If you wait too late, you can’t do anything with the data.

cindy.mccracken

2016-11-30 23:59

@frankenvision – Usually you will read that you shouldn’t make changes to design … but I think it’s OK if it’s something that’s not working but should have been, and you’re having to talk around it … that sort of thing.

cindy.mccracken

2016-11-30 23:59

That’s not a design change then; it’s fixing a bug.

robbin

2016-12-01 00:00

(Just wanted to say thank you for doing this! This is super helpful!)

cindy.mccracken

2016-12-01 00:00

@lynne – so you were trying to get a lot of kids’ feedback at once to make the most of your time? Here’s one idea … you could have them use the program individually .. and observe as much as you can. Then have a focus group of sorts where they discuss.

cindy.mccracken

2016-12-01 00:00

It’s a thought!

cindy.mccracken

2016-12-01 00:00

Thanks, @robbin! I’ve really enjoyed it.

frankenvision

2016-12-01 00:00

Ok thanks – how many questions should we use for remote usability tests? I think they’re supposed to take 15min for users to complete…

frankenvision

2016-12-01 00:01

What a good question to measure how trustworthy a company is?

cindy.mccracken

2016-12-01 00:01

It might depend on how long it will take to do the tasks. I’d say I’ve seen about 10 tasks on average. Best idea: test it with some people to see how long it takes before sending it out.

lynne

2016-12-01 00:01

That’s kind of what we did, they each had a computer, we gave them tasks to do, and discussed as a group at the end. They were a bit shy about the discussion part.

cindy.mccracken

2016-12-01 00:02

One idea for shyness that I like is first having them write down answers to your questions.

lynne

2016-12-01 00:02

The biggest challenge was trying to observe 20 different people at once. I couldn’t figture out if it was better to choose one or two students to focus on, or roam around the room.

cindy.mccracken

2016-12-01 00:02

Then, having thought it through first, people tend to be more confident.

lynne

2016-12-01 00:02

Nice idea about writing it down – i’ll try that next time!

cindy.mccracken

2016-12-01 00:03

great!

cindy.mccracken

2016-12-01 00:03

@lynne – maybe if that happens again, it could help if you had multiple observers??

cindy.mccracken

2016-12-01 00:03

that’s a tough one,.

hawk

2016-12-01 00:04

Ok all – after this answer that’s a wrap

frankenvision

2016-12-01 00:04

Is it best practice to run a single test to see how it runs or run 3-5 remote usability tests first go?

hawk

2016-12-01 00:04

Remember that if you have follow up questions you can ask at http://ift.tt/1qs7rt7

hawk

2016-12-01 00:04

Someone is there pretty much around the clock

seyonwind

2016-12-01 00:04

Thank you for all the usability advice @cindy.mccracken!
As always, thank you @hawk for hosting :slightly_smiling_face:

cindy.mccracken

2016-12-01 00:05

Thanks everyone! I really enjoyed this.

jacqui_dow5

2016-12-01 00:05

Thank you so much guys! This has been great!

hawk

2016-12-01 00:05

Thanks so much again for your time @cindy.mccracken – you rocked it!

lukcha

2016-12-01 00:05

This has been a great session_thanks so much for your advice and tips @cindy.mccracken :slightly_smiling_face:

cindy.mccracken

2016-12-01 00:05

Of course. And thank you for hosting,

lynne

2016-12-01 00:05

Thanks @cindy.mccracken and @hawk!

hawk

2016-12-01 00:05

And thanks for all the great questions

hawk

2016-12-01 00:05

Have a great morning/afternoon/evening/night all

cindy.mccracken

2016-12-01 00:05

Indeed.

nik

2016-12-01 00:06

Thanks a lot @cindy.mccracken . Good night

nat

2016-12-01 00:08

Thanks!

lukcha

2016-12-01 00:12

Thanks everyone!

The post Transcript: Ask the UXperts: Usability Testing — with Cindy McCracken appeared first on UX Mastery.

by Sarah Hawk via UX Mastery

Maybe I Was Wrong about Java – Part 2

[special] Editor's Note: Being in a Java channel, most of us know the language very well and have been in its ecosystem for at least a couple of years. This gives us routine and expertise but it also induces a certain amount of tunnel vision. In a new series Outside-In Java non-Javaists will give us their perspective of our ecosystem. [/special]

I don't deal with Java much, so I'm investigating how true all my preconceived notions about it are. Last time, I mostly explored user-facing concerns, like speed and size. Results: inconclusive.

But modern Java is increasingly used in places where it's invisible to users: In datacenters, on phones, on my toaster. So perhaps the really interesting questions are about how Java looks to developers.

Java Is Insecure

Java's infamy as a walking security issue dates back to the ancient days where Java applets were a common thing, you trusted that the JVM could effectively sandbox them, and occasionally it couldn't.

Maybe trying to whittle down an entire general-purpose language and its massive standard library to be safe enough to run from the web was a bad idea, but it's a moot point now. I very rarely see Java applets any more. I don't think I even have the NPAPI plugin installed. Firefox doesn't let Java applets run automatically, and is dropping support for them entirely in March; Chrome dropped support last year.

Granted, this was probably in part because Java applets had become more of an attack surface than a useful platform; a CISCO report from 2014 prominently claims that 91% of web exploits were aimed at Java. I think that same year, my then-employer was warning everyone to manually disable Java in their browsers if they didn't specifically need it. If that's the only exposure you get to Java, well, it's not going to leave a great impression.

Hey, hang on. This is supposed to be about the developer perspective. So what about the runtime itself, independent of applet concerns? "Secure" is difficult to quantify, but as a very rough approximation, I can look for the number of CVEs issued this year.

• PHP: 107
• Oracle Java: 37
• Node: 9
• CPython: 6
• Perl: 5
• Ruby: 1

Er, whoops, that caught me off guard. I honestly expected to be pleasantly surprised and clearly proven wrong here, but that list makes Java sound somewhat worse than I thought. I hope there's a great explanation for this, but I don't have one.

Java Is Enterprisey

Ah, another word that everyone uses (myself included) but that doesn't mean anything. It conjures a very specific image, but a very fuzzy definition. I have a few guesses as to what it might mean.

Java Is Abstracted Into The Stratosphere

The abstractosphere, if you will. The realm of the infamous AbstractSingletonProxyFactoryBean.

I'm actually a little confused about this one. Turning to elasticsearch again, I stumbled upon this class, WhitespaceTokenizerFactory. Its entire source code is:

public class WhitespaceTokenizerFactory extends AbstractTokenizerFactory {

    public WhitespaceTokenizerFactory(
            IndexSettings indexSettings,
            Environment environment,
            String name,
            Settings settings) {
        super(indexSettings, name, settings);
    }

    @Override
    public Tokenizer create() {
        return new WhitespaceTokenizer();
    }
}

Okay, sure. You want to be able to create an arbitrary tokenizer from some external state, but you don't want the tokenizers themselves to depend on the external state. Makes sense.

Still, this code looks pretty silly, especially if you haven't seen the other classes that do more elaborate things. The same words are repeated three times; a 38-line file has only two lines of actual code. It's easy to look at this and think Java code goes to ridiculous extremes with its indirection. At worst, I might do this in Python:

@builder_for(WhitespaceTokenizer)
def build(cls, index_settings, env, name, settings):
    return cls()

@builder_for(SomeOtherTokenizer)
def build(cls, index_settings, env, name, settings):
    return cls(index_settings.very_important_setting)

# etc.

I'm handwaving how this would actually work, but there's not much to it. It might even be possible in Java, come to think of it, but probably not pretty or idiomatic. Alternatively, Python code might just have the build on the tokenizer classes themselves. One nice thing about dynamic typing is that code can use a type without depending on it. The tokenizer class can work with IndexSettings and Environment objects without having to import the types or even know they exist. It's a little iffy, but in a case like this where everything's internal, it could make sense.

But given that Java's type system is what it is, I can understand why you'd end up with the above code. What confuses me is this.

Why don't I see the same thing in other languages?

I found this collection of tiny factory classes after about a minute of randomly clicking around in the most starred Java project on GitHub. I'm completely unsurprised by it. Yet I can't recall seeing anything similar in other explicit, statically-typed languages. Where are the tiny factory classes in C++? The most starred C++ project is Electron, and searching for "factory" only finds me code like this, which has a lot more going on. The most starred Objective-C project is AFNetworking, which contains "factory" once — in a changelog. The most starred Swift project is Alamofire, which somehow doesn't contain the word "factory" anywhere!

So while I can accept that layers of indirection and tiny classes are useful for getting along with a C++-style type system, I don't understand why I see them so much more often in Java than even in, well, C++.

Is this a cultural difference? Are C++ developers happy to have a tangled web of interconnected dependencies? Do these tiny classes exist in C++, but live all together in a single file where they're much easier to ignore?

Java definitely seems to live in the abstractosphere, but I can't figure out why it's so different from similar languages.

Java is Tediously Verbose

"Enterprise" makes me think of repetitive bureaucracy sucking the joy out of everything.

Accessors Everywhere

And Java makes me think of accessors. Same idea, really.

private int foo;

public int getFoo() {
    return this.foo;
}

public setFoo(int foo) {
    this.foo = foo;
}

Look at all this code eating up precious vertical space to do absolutely nothing. I could've just said public int foo; and been done with it.

There are three kinds of programmers in the world, distinguished by how they reacted to that last paragraph. Some nodded their heads, and they are probably Python programmers. Some balked that this violates encapsulation, and will balk again when I say that I don't care about encapsulation. Finally, some rolled their eyes and pointed out that a public attribute is frozen into the API and can never be changed without breaking existing code.

Ah, those latter folks might have a point. The trouble is that Java doesn't support properties. "Property" is a horrible generic name for a language feature that's become popular only somewhat recently, but if you're not familiar, I mean this magical thing you can do in Python. If you have a foo attribute that external code is free to modify, and later you decide that it should only ever be set to an odd number, you can do that without breaking your API:

class Bar:
    def __init__(self):
        # Leading underscore is convention for "you break it, you bought it"
        self._foo = 3

    @property
    def foo(self):
        return self._foo

    @foo.setter
    def foo(self, foo):
        if foo % 2 == 0:
            raise ValueError("foo must be odd")
        self._foo = foo

bar = Bar()
bar.foo = 8  # ValueError: foo must be odd

@property is an artifact of great power that transparently intercepts attempts to read or write an attribute. Other code can still work with obj.foo as expected and never know the difference. Even @property itself can be expressed in plain Python code, and there are some interesting variants: Lazy-loading attributes, attributes that transparently act as weak references, etc.

I know Python, Swift, and a number of .NET languages (C#, F#, VB, Boo, ...) support properties. JavaScript is specced as supporting them by now, though I'm not sure how much code relies on them in the wild. Ruby has them, with slightly different semantics. Lua and PHP can fake them. Perl has a thing but you probably shouldn't use it. The JVM itself must be able to support them, since Jython and JRuby exist. So why not Java the language?

It seems odd to me that Java hasn't picked up on this feature that would cut out a lot of repetition. It was apparently proposed for Java 7, but I can't find an explanation of why it didn't make the cut, and now it seems to be very much not a priority.

But Wait, There's More

Continue reading %Maybe I Was Wrong about Java – Part 2%

by Eevee via SitePoint