As part of our commitment to giving back and sharing knowledge, we have partnered with the WordPress Foundation’s community team to run an official WordPress Meetup centered around building more accessible websites with WordPress. This post has a recap of our Meetup that took place on Monday, September 20, 2021, and a video recording of the presentation.
About the Topic
In our Meetup on September 20th, Amber Hinds, CEO of Equalize Digital and an organizer of the WordPress Accessibility Meetup Group, shared resources for hiring or finding volunteer user testers, how to plan for user testing sessions, and how to get the most out of your testing sessions so your team has actionable feedback for improving your website’s usability.
Thanks to Our Sponsor
WP Engine, a WordPress technology company, provides the most relied upon and trusted brands and developer-centric WordPress products for companies and agencies of all sizes, including managed WordPress hosting, enterprise WordPress, headless WordPress, Flywheel, Local and Genesis. WP Engine’s tech innovation and award-winning WordPress experts help to power more than 1.5 million sites across 150 countries.
About the Meetup
The WordPress Accessibility Meetup is a global group of WordPress developers, designers, and users interested in building more accessible websites. The meetup meets twice per month for presentations on a variety of topics related to making WordPress websites that can be used by people of all abilities. Meetups are held on the 1st Thursday of the month at 10 AM Central/8 AM Pacific and on the 3rd Monday of the month at 7 PM Central/5 PM Pacific.
Learn more about the WordPress Accessibility Meetup Group.
Watch the Recording
If you missed the meetup or would like a recap, watch the video below or read the transcript. If you have questions about what was covered in this meetup please tweet us @EqualizeDigital on Twitter, or join our Facebook group for WordPress Accessibility.
Presentation Slides
Click here for presentation slides on Google Drive.
Links Mentioned In This Video
The following resources were discussed or shared in the chat at this Meetup:
- WPEngine
- WPCampus
- 18F Usability Testing Script
- GitLab Usability Testing Script
- Steve Krug Downloads
- UI Access Accessibility Testing Checklist
- Usability Test Plan Dashboard from Medium
- Usability.gov Sample Report
- Hotjar
- Loop11
- Fable, a resource for crowdsourcing blind and low vision users for testing
Article continued below.
Stay on top of web accessibility news and best practices.
Join our email list to get notified of changes to website accessibility laws, WordPress accessibility resources, and accessibility webinar invitations in your inbox.
Read the Transcript
Amber Hinds 0:00
So, let’s see. So, on the announcement side, we have a Facebook group. If you aren’t familiar with it, you can join it, it’s just facebook.com/groups/WordPress.accessibility. And it’s up on the screen, we can also throw it in the chat for you. It’s a great way to connect with people, we’re still kind of growing the community, but we’re trying to build something so that if you have accessibility questions or problems, or you find something interesting related to accessibility in between Meetups, that we can share, and kind of grow the WordPress accessibility community.
Also, tonight, we have live captions, I’m super excited about, um, I’ll talk more about them in a second, but WP Engine has agreed to sponsor our captions for all of our evening Meetups. So we will have live captions at evening Meetups moving forward. We are still working on live captions for the daytime Meetup and as Mark has requested, we’re going to try and see if we can find ASL interpreters as well, because it is important to us that we make this as accessible as possible. Unfortunately, the WordPress foundation does not have funds for that, we, we did try to see if we could go through them. But Emma and I are working pretty hard at trying to find sponsors for the event. If you do have any other suggestions for the Meetup, or need any additional accommodations, you can reach out to us. You can either message us through Meetup, or if it is easier, we are both our first name at equalizedigital.com. So I’m Amber@equalizedigital.com, and Emma is Emma@equalizedigital.com. If you email one or both of us, or message us through Meetup, we’re happy to help you however we can.
So, on the note of today’s sponsor, I just have to give a shout out to them because of course we are super excited that we’re able to have live captions. But WP Engine, if you’re not familiar with them, they are a WordPress technology company. They have their own brand, and then also Flywheel, are two different hosting companies. They offer enterprise WordPress, headless WordPress. They also are the makers of the Local app, it’s called Local by Flywheel, that helps developers work locally on their computers. And then they also own the Genesis Framework, and StudioPress. So they have really great support, that’s one of the things that I love about them, and we, we actually utilize some WP Engine, we have clients on both WP Engine and flywheel, so we’ve had the ability to interact with their support, I think it’s great. And they said that they power more than one and a half million sites across 150 companies. If you want to learn more about WP Engine, you can go to WPEngine.com.
If you’re on Twitter, this is my please request for you, would you tweet at them, @WPEngine, and say thank you for sponsoring live captions for the WordPress Accessibility Meetup? Because I think, like, any kind of feedback like that will help them to stay motivated to want to continue doing it, which of course they’ve said they’ll do it for an ongoing basis. But you know, getting that feedback and hearing from people that it is important and that what they’re doing, you know, matters, I think is really good. So that’s my please give them a shout out on Twitter if you feel up for that.
So, upcoming events. I threw in one that is not ours, because I have been working to organize WPCampus. WPCampus is a two day, online campus that’s specifically focused on WordPress and higher education. And it starts tomorrow. You can still get tickets, they’re totally free, but you do have to reserve in order to be able to watch the live stream. We will have live captions. And we have more than 10 different talks and a community discussion all on accessibility. So, it is focused on higher ed, but there’s definitely things there on the accessibility side that are applicable to any website. And there’s other talks beyond the accessibility talks that would also be applicable outside of higher ed. So, go check that out, 2021.wp campus.org is the website address.
What we have upcoming here in the Accessibility Meetup, “The Business Argument for Accessibility” with Colleen Gratzer of Gratzer Graphics will be on Thursday, October 7 at 8am. And then, on Monday, October 18, we have “Best Practices for Screen Reader Text” with Nick Croft of Reaktiv. That one may possibly change, or we may have to reschedule the date on that one. We will still have a Meetup that day, but tentatively that’s what’s scheduled for that day.
Amber Hinds 5:04
So if you have attended before, normally I introduce the speaker. I, it’s me, so it’s a little weird, but I thought I would just tell you, like, briefly about me and my background, so you kind of know where I’m coming at this from. I am the founder and CEO of Equalize Digital. We’re a certified B Corporation, a WordPress VIP agency partner, and a corporate member of the International Association of Accessibility Professionals. Equalize Digital is actually a relatively new company for us. It’s what we like to call our “pandemic pivot.”
So, I have actually been running a marketing agency, and doing WordPress websites since 2010. And since 2015, I’ve been doing things with accessibility. And we only really, in, last year in 2020, realized that we wanted to be fully focused on accessibility. And that, that is our, our main goal now, and everything that we’re doing is accessibility. And so we made that, that name change and everything last year. So, we do accessible website design and development, accessibility audits, remediations of existing sites, and, of course, user testing.
We also make a plugin called Accessibility Checker that scans websites for accessibility problems, and shows the error reports on the post or page edit screen. There’s a free version of it on wordpress.org. But I’m not really here to sell that today, so, I’m just gonna dive right in. But that’s a little bit about who I am, and my background.
So what we’re going to talk about today is how to run user tests with real world users. I’m gonna start with why user testing is important. First of all, real life users are more likely to reveal issues that you, or developers, or the people who have been working on content in the website will miss. A lot of times we spend so long looking at something that we don’t see, whether it’s a little typo, or we, we spent time thinking “this is what the navigation structure should be, and it totally makes sense to me.” Right? But then, someone else who didn’t, wasn’t sitting in all the meetings, or all the back and forth discussions about what should be in what drop down, and what should these pages be called, they might come to the site, and it does not make sense to them, because they didn’t have all that back and forth, and the history of it. So, bringing other people that are, that actually represent who your users are, will, will surface problems that you otherwise would not have seen.
Another reason why you want to do user testing is people who use assistive technologies or devices every day, will use different shortcuts or features of that technology than non native users. So, I think I’m fairly proficient at NVDA. I’m less proficient at VoiceOver because I only just recently switched to a Mac. But even in those, I don’t use them every day, I use them just to test the websites that we make, or to do accessibility audits of other sites. And because I don’t use it every day, the way I use it is not a natural way that somebody who uses it every day would use it. And so, inherently, you’re going to get a better look at what real usage is like if you actually have the, if you have a native user of that assistive technology.
And then the other benefit of doing user testing is you can test on more devices and software than is in your stack. So, one of our testers uses JAWS. And I personally have never used JAWS, because I haven’t bought it or paid for it, and I’m like, “well, I use VoiceOver, I use NVDA. But JAWS is a very popular screen reader for blind users. And so, it’s important to have the ability to test in something if you know that a lot of your users might be in it. Unfortunately, not every screen reader reads every- everything exactly the same way. They say different things, and so sometimes, it is important, it really is important to test in multiple different software versions. You can also get different devices. So, somebody can be like, “Oh, I’m going to test it on this version of iPhone.” And maybe you don’t have that version of iPhone. So, so really, you can get a more complete picture of everything that’s going on on the website when you bring in users to test it.
Amber Hinds 9:52
So, the goals of user testing. When we do user testing, what we think of is, we want to ensure first, that users don’t get lost, and that they can understand the website’s navigation. So, we ask them questions like, “if you wanted to find out XYZ, where would you go?” And then we just watch. And we see, do they go to the page we think they would go to? Do they find the information? Or if they go to the first page, they don’t find it, do they then go to another page? Or are they not sure what to do? So really just watching how they move around through the site, what they click on, which pages in the content or in the navigation they engage with.
And then you want to make sure that they can intuitively find information. So you think about what’s the most important information that someone needs to get about your business or your clients business, if you’re doing this for a client website. And can that information be found easily and without having to have someone tell them where to go to find that information? Sometimes, we think we have something laid out in an intuitive manner, and then we get users in, and it’s like, no one even goes to this page, because you named it something weird in the navigation.
A really great example I have of this is we had a client who was, who’s an, an artist, and she makes wrapping paper, but she wanted to call it all gift wrap. And, and, or we’ve seen things where they want to give things branded names. And the users don’t, that’s not, in their area where they live or whatever, that’s not what they call it. That’s not what they think of originally. So, if somebody wanted wrapping paper, they would type wrapping paper in the search, but because she hadn’t put the terms wrapping paper on any of those products, they wouldn’t come up for the search “wrapping paper,” even though, you know, gift wrap is like a synonym but because to her, that’s what she called it. So, sometimes what we think makes sense, doesn’t make sense for our users. So, we really want to make sure they can find any important information.
We also want to test to make sure that they can follow the intended funnel, or any path that we expect them to take as they move through the website. So that’s like if we think they’re going to land on the homepage, they’re going to click shop now, they’re going to go to a product, and then they’re going to add it to their cart, and then they’re going to click “check out,” right? Does that actually happen? Is that the path that people take? Can they move through that funnel? So, we want to make sure that they can do that.
And then we want to make sure, of course, along those same lines, that they can complete tasks that are required for any sort of conversions on your site. Now, conversions are totally different, but, depending on the site. Obviously, eCommerce sites, they need to be able to make a purchase and actually complete it, complete the checkout process. If it’s not an eCommerce site, it might be submitting an inquiry form. It might be signing up for an email newsletter. It might be actually clicking a button to follow you on social media. What are the key actions that need to happen on the website in order for you or the client to feel like the website is a success? These are the things that we need to make sure that users are able to complete.
And then of course, along this whole way, we want to test to make sure that users don’t encounter any usability problems or bugs. Is there anything weird that comes up? Did they submit that form and oops! We forgot the confirmation message is blank, because it just says “thank you,” and, and there’s no other message written when there was supposed to be more. Or it went to a 404 page because someone accidentally deleted that confirmation page it was supposed to go to. So, sometimes there are little bugs, especially, I feel like on new builds, where there’s a lot of things that are happening, and just having another user come in and test things, that is not you, they might notice typos sometimes that you overlooked, or didn’t see. Or they might, you know, notice more major errors that are obviously, it’s like, “man, if we had launched with this, it would have caused a problem for us.”
Amber Hinds 14:13
So when do you user test? We, we look at two different times when we include user testing. The first one is if we’re launching something new. The second one is if we’re making an existing site better. So, on the new side, as much as possible, we want to include user testing in our process for that new website build. And I’ll talk a little bit more about where that fits in the process in just a minute.
But on existing sites, there’s a couple of reasons why you might do it. So one is if you’re not seeing conversion numbers that you expect to see. You’re seeing really high bounce rates. If you are seeing a lot of people adding to carts, and then abandoning their shopping carts. If you, just in general are like, “no one’s adding things to carts, no one’s filling out forms,” they’re not doing what you want them to do. So that in and of itself is a major reason to start doing user testing. Because if you’re looking at your site, and you’re saying, “it looks perfect, it all seems to work when I do it,” but you’re not seeing these actions, or you’re seeing really high bounce rates, and you don’t understand why, that’s when you really need to bring in a user who’s representative of your demographic, to find out what they see on the site, and try and get a better feel for why the actions might not be happening. And it can’t be your friend. I’m gonna talk more about this in a minute, but it has to be someone who’s representative of your customer base.
And then, another really good reason, and a huge one, is you need to make sure your website complies with WCAG. WCAG stands for Web Content Accessibility Guidelines. And so, what we like to do is, we like to use user tests as a way of proving accessibility. So we, when we’re building a website, or we’ve audited and remediated a website, we’ve gone through, and we’ve said, “these are all of the WCAG guidelines. And we are going to, we know that we’ve met them.
But accessibility isn’t just about meeting guidelines. It’s about having something that is usable for everyone, and allows people to move easily through the site. And sometimes, you can have something that is technically compliant, but isn’t fully accessible, or isn’t usable, maybe is a better way to put it. So, we like to bring in users we- to confirm, yes, we meet all these guidelines, they don’t see any problems that object, that cause problems with the guidelines. But they also can prove, like, full accessibility.
And then, and then we sort of look at it as we bring users in once we’ve identified and resolved all the issues that we can find on our own with automated scanners. I don’t, I don’t like to bring in users on a website that has a bunch of errors that can be easily found with either our plugin, Accessibility Checker, or with WAVE or axe, which is a-x-e, from Deque, because, one, I just kind of feel like it’s a waste of our accessibility testers’ time. I don’t need them telling me that an image is missing an alt text, right? I want them to really be finding the really minute, and the, sort of the, the specialized problems that really only a user are going to do. So, so we think about user testing as something happening after the dev team, and the content team has said “yes, we think this website is perfect.” Now let’s get a user in, and figure out what can be improved even more, you know? Going beyond what we think is perfection to moving to a more perfect website.
Amber Hinds 18:10
So, on new projects, what this looks like for us, I have an image up on the screen right now, in case you can’t see it, that is a line all the way across from left to right, it shows our new website process. And the way we handle this is we have discovery. And then we do content from our clients. Then we move into a design phase. And then we have development, where our developers are building out the designs. Then we enter testing and debug.
In our company, we do one round of internal testing and debug before we even show it to the client, where the developer says, “Hey, I think this is ready,” and then myself, our COO, and then sometimes Emma, our content person, we all go through, and we do a whole bunch of testing, we do a whole round of fixes, getting things over to the developers. And then, with the clients, we have two rounds of testing and debug, where they go through the site, and they get us a list. Sometimes, in between their second round, their first round and their second round, we’ll bring in user testing. But typically, we do our user testing after we have finished all of our test and debug.
So like I said before, we think the website is perfect. Now we’re going to have users test it. And that’s really because I want to maximize what I get from my users. I don’t want to hear, like I said, small problems. You know, little content issues. Hopefully all the typos are worked out. But the other reason why is because we all, maybe you’ve known if you work on client websites, that sometimes, in the middle of test and debug, you can get some very dramatic changes from clients where they’ll say, “Oh, hey, we changed our mind, rewrite”- like, “here’s the whole new content for this one page.” Like, “we just want to completely replace it.” And sometimes that doesn’t matter, because it wouldn’t be part of something you would test anyway, but sometimes it would be part of what you’d test.
So we really want to make sure that the website is at the point where we feel like it’s a final version. So, we then don’t have to go back and be like, “Oh okay, well, you changed XYZ, so now we need to user test again.” So, so then we would do that, we would have user testing, we would do any remediation that has come out of problems identified during the testing sessions, and then we launch. And then we, the last thing in our process would be training, and providing any sort of post launch support. Of course, you can continue then, on an ongoing basis, with doing any sort of user testing over time.
And what it looks like if it’s not a new website, so, either an existing site that needs improvement, or needs an accessibility audit. The way we approach these, I have the same kind of graphic where it’s a line, and I’m showing the different phases. So, the order of our steps, our accessibility audit, we would do that first. This is where a trained accessibility specialist would go through the website and test everything with automated tools, manual keyboard navigations, and usually two different screen readers. And then we write up a whole report. And then, we typically approach, I have these one, and then the other, so code fixes, content fixes, a lot of times these phases sort of overlap. And it might be like, we have to code something, and then you have to do some content, and then we code something. Or it might be more agile, where we’re coding, and pushing changes live as we go.
But anyway, there’s some code fixes, and some content fixes, and any sort of number of additional internal reviews that are happening as things are being fixed. And then again, once we’re like, “Hey, we think that this is really close to being better,” or we’ve, we’ve solved a lot of the major problems, we think there might still be some problems with accessibility or with the the user flow and the journey through the website, then we bring in users to do testing for us. And then we’ll do another round of remediation. But again, we’re trying to get the major things fixed first.
And then, you might have ongoing accessibility things that happen on an ongoing basis afterwards. But this is where we personally include user testing in those two different types of projects. I think, you know, you, you really have to figure out what works best, either for your organization, or your clients. We’ve found that this helps us to maybe reduce some of the workload if we approach it in this way. So, who should you include in your user tests?
Amber Hinds 23:04
We always, most people think of blind users when they’re thinking about who to include, so people who natively use screen readers. And that’s super important, that’s who we genuinely are working with, but it’s also important to think about other people who come to your site. Again, they need to be representative of your target demographic, but thinking about people with low vision. So these may not necessarily be people who are using a screen reader or are blind, but they could be someone, you know, I think it was David maybe, had said that he was working on the the Fitness Over 50 website, and thinking about if you have older people, they might be using screen magnifier, or they might just have their browsers zoomed in all the time. So, you really want to make sure that your website works well for those people. And it can be important to pull in users that represent the demographic.
Another one could be sighted keyboard only users. So these are people who might have mobility issues. And therefore they can still see everything on the screen, but they’re not able to use a mouse. Or actually, they might not even be keyboard users, they might use alternative keyboards or they might use voice controls, where they’re speaking out and their mobile device or their computer is hearing their voice, and then going to the element that they’re speaking or asking for. So, if that represents your population that could also be an important group to do testing with.
Deaf or Hard of Hearing users would be good to include, particularly if you have a lot of videos and you’re new to captions. This is something that we recommend, we actually have a woman, Meryl, who’ll be speaking later in the year about captions. But she is phenomenal, and like, can give a lot of great information, and also, I know, like, she’s reviewed some videos for us, and if you aren’t sure, you know, how many sounds should I type out? Where should I label people? How should I caption music? It would be really good to include somebody who actually uses captions to get their feedback. You may not have to do this for every video, but in, for the first few videos that you do, I think it’s really important to make sure that the, your captioning style is accurate, and what people who are deaf or hard of hearing would expect to read or see on captions.
If you work with the elderly, or you have customers who are elderly, it is 100% important to include them. This really falls into maybe less about the assistive technology, because a lot of them may just be normally using a computer, but it it falls a lot into having a user journey that makes sense, and is easy to use. I think a lot of, those of us that spend all day, every day on websites, looking at websites, dissecting websites, building websites, like, we just intuitively know things like, if you click on the logo, it will take you back to the homepage. But older people don’t always know things like that, and they can be really confused, and they might say things like, “I have no idea how to get back to the homepage.” So, if this is part of your audience, it’s really important to pull them in and include them in your user tests.
And then another group that you might not always think about is people with cognitive disabilities. If your website is one, that people who have learning disabilities, or traumatic brain injuries might be visiting, you know, let’s say you are a nonprofit that does something with disabled veterans, then it would probably be important to include people who have traumatic brain injuries in your user testing pool, because you really want to have a user testing pool that is representative of your customers, and who you expect to have on the site. And so, that does include people with cognitive disabilities.
Another group, which I didn’t write out here, but potentially thinking about, is language learners. So, if you have a lot of people where English is their second language, and you don’t yet have any sort of translation on your website, maybe you want to think about including them in. So, really thinking broadly about who is your customer demographic? And how can you best make your website work for everyone who will visit it, not just the, quote, “typical users”?
Amber Hinds 27:59
So, once you once you’ve identified who it is that you want to do testing of your website, then you have to figure out how to find them. So, there’s a couple of different ways you can go. The first is you can hire a company. So, like our company, Equalize Digital does this. Knowbility is a nonprofit organization down here in Austin, where I am, that, that does user testing. And, um, the, it’s IAAP, that’s the International Association of Accessibility Professionals, and there’s a member directory, so you can go find companies there.
The benefit of hiring a company to do your user testing is that they will be, they usually have an established process, they can walk you through it. and kind of remove a lot of the legwork on your end, so you can go to them, say “I need my website user tested,” they’ll give you all the different options, you pick a package or a plan that works for you and a timeframe, and then they will set it all up, they’ll run the tests, and either you can be present for them, or they’ll provide you with recordings, they’ll give you a write up, and recommendations for fixes. So, that’s a great option if you want to be super hands off. It can be more expensive than some of the other options, so that’s something you’ll want to consider.
Another route to go is looking for freelancers. So, we had a young woman, Taylor Arndt, who spoke a couple months ago at our Meetup, but she’s a great example. She is blind. She is a WordPress developer, so she’s very familiar with WordPress, but she also does accessibility testing and audits, and she just does it independently rather than as part of a large organization. You can find some of these people through the IAAP member directory as well, or you might try asking, like, on LinkedIn but, uh, freelancers are another way to go.
If you want to manage it yourself, where you’re working with just users from your customer base, let’s say, or people that you find in your community, then, I would say just post it as a job. What is important to know is, generally, you are going to pay user testers. So user testers are not, we’re not asking people to volunteer to give up an hour of their life, or two hours of their life, however long it’s going to be for your specific tests, and give you free advice, or show you, work through your website with you for free. So normally, we’re thinking of this as a job that you’re going to pay people for. So, you could post it on any sort of job board that you would normally post jobs.
Another thought is, is reaching out to either schools or nonprofit organizations. So, we have worked with Texas School for the Blind and Visually Impaired, which is down in Austin, and we hire students from there. And what’s nice about doing that is that they’re able to get work experience, they earn money, and, and then we also have great testers. And we can get students who are brand new to screen readers, and we can get students who are very experienced in screen readers. But you just go to the Career Center, and you talk to them, and you say this is what we’re looking for. And it doesn’t have to be an ongoing thing, like you could just say it’s, this is a one time thing, and they write up a job and post it for their students.
There’s also nonprofit organizations. Like I know, there’s one in our town that works with people with cognitive disabilities, and helps them to find work. So that’s another way to go potentially, if you want to try and find people on your own. So I really think you, when you’re weighing all these different options, you have to think a little bit about how much you want to pay. And then also how much time do you want to put into running the process yourself.
I think if you do have customers who you know fit the demographic of the people that you want to test your website, then you could also reach out to them and offer, you know, either to pay them, or offer to give them a free thing, or a free service in exchange for participating in a user testing session. And that would be great, because they’re actually your customers already. But they may have also already figured out issues on your website, and sometimes it’s better to have totally fresh eyes that have never been on the website before, so they don’t already know the little workarounds.
Amber Hinds 32:39
So once you’ve figured out who you’re going to test for, these are some things that you need to keep in mind. Generally, we’re expecting, if we’re doing testing with people with disabilities, that we’re doing one on one with a tester and a moderator. And then, we’re going to times it by the number of people that we have testing. We like to do at least two testers, because you don’t get a good picture with just one person, but I would say that it’s better to have a larger number. So, five could probably be a good number of users to give you a feel. And they would use different devices, let’s say somebody’s on a Mac, somebody’s on a PC, somebody who’s using Jaws, somebody’s using NVDA, like different screen readers, things like that.
If you have a large website with a lot of things that need to be tested, or it’s for an organization that really wants to be sure, I’ve, I’ve even heard of doing user tests with, you know, 30, 40 different users, so that you can cover the broad range of different user types that you would expect. So, you really have to figure out, you know, how much, how many people, and then think about, as you’re planning it out, that you’re going to typically be doing these one on one where you have a tester and a moderator.
You’re going to want to think about how long the session will be, keeping in mind a few things, one is how, you know, how many items are you expecting them to test? But also, some, some of your users may have limitations that make it challenging for them to participate in a test that is more than like 30 or 45 minutes long, particularly if you’re working with users that have cognitive disabilities. So, really thinking about that. Can it be longer? Yes, it definitely can. We’ve done user testing sessions that were like, over several hours, we just gave them breaks and we fed them food because they came to us in person.
This is actually, I have an image up on the screen, which I think you can see Chr- yeah, I’ve got the pictures of everybody covering Chris, but you can see my partner Chris is standing behind, and he has a camera up on a tripod angling down, looking at Macy, one of our testers, as she works at a desk, um, testing, doing screen reader testing. Um, you know, so, if they’re in person, maybe they can be a little longer. I’d say virtual ones probably are little shorter as far as sessions.
So then, that’s another thing you have to think about, are you bringing them to you, to your office? Or are you doing them over a virtual platform like Zoom, for example? In person, I think is great. I like that we really, I feel like we have better interactions when they’re in person. With COVID, that kind of has gone away. But you can definitely do virtual. If you do do in person, another thing you need to think about is how accessible is your office that you’re bringing people to? You know, is it, is it wheelchair accessible? Is it easy to find or get to? Are you going to pay for transportation if they have to come from somewhere else? Or how you’re, how are you going to reimburse them for those sorts of things? Are you going to provide food if it’s a longer session?
And then you want to think about how will the session be recorded? On Zoom, this is super easy, you just hit the record button, and it will record their face and their screen and everything. Obviously, you can see in the picture, which I’ve described, with the camera up on the boom arm overhead, like we’re thinking about how can we record this, we have to have a whole setup in order to be able to capture what we can see on their screen. And hear that they’re talking. We don’t, in this instance, like we weren’t recording her facial expressions, but she’s talking to us about what she’s doing, and it’s enough for us to hear from her.
You want to think about if you’re doing this for a client, is the client going to be present? If so, you definitely need to have conversations with the client about, you know, what’s appropriate, or what’s not appropriate. Are they just an observer? Are they actually engaging? Typically, we want them to just be an observer, or they have a moment when they can ask questions, but we don’t really want them to ask questions during the session, because it can derail things. If they, or if they are going to, then it’s like, okay, we need to really educate them on how to ask questions in the right kind of way so we’re not leading or causing confusion.
Amber Hinds 37:22
We also want to find out if a signed NDA, so a nondisclosure agreement, is required of the tester. Are they viewing something that is private, not launched yet, and is a secret? In which case, we might need to have the tester sign an NDA, saying that they won’t talk about what they saw.
Is your staging server accessible? So, a lot of times, we password protect websites before they go live. You need to make sure that whatever tool you’re using for password protection can be accessed by someone on a screen reader. Um, some of the WordPress plugins that people use to put their sites into maintenance mode are not especially accessible in order to log in, or they obscure the login in a certain way that makes it challenging for people to log in to the website. If you’re using just like, typical server HT access protection, that should be fine. But there are some credential sources that don’t work for people on screen readers. So that’s something you need to be aware of before you engage in this. Especially if it’s remote, because it can be harder to troubleshoot with somebody remotely.
And then, another thought is do you need to provide email addresses or login credentials for testers? So, if you’re expecting someone to test a, a portal, that’s not a public website, you might need to create a user account for them so they have something to log in as. Or, if you want someone to test making a purchase, and they’re required to enter their email address, do you want them to enter a fake email address? Do you want them to use their real email address because you also want them to go look at the emails they get, and then tell you if they have any problems with the emails they’ve received?
So these are all kinds of things that you have to take into consideration as you’re getting ready to plan the session, and things you might have to prepare in order to make it successful. Whoa, sorry. Okay.
So, if you do do Zoom, which is the way things have been going these days, um, some quick tips on this. One, you want to mute all participants on entry, because, especially if you have observers or clients who may come and go, or um, come, you don’t want to have any background noise that gets recorded. You really only want to have the moderator and the participant with their sound on. I would say even the moderator might frequently toggle between mute, and being able to speak.
You want to turn off the chime for attendees when they join or leave. And then something that’s really important to know, some of you may have been on Meetups where we’ve had, we’ve been demonstrating a screen reader, I know I’ve done it and Taylor’s done it, if you type in the chat in Zoom, Zoom, the screen reader will read out what is said in the chat.
And so, if you have someone screen reader testing, and then let’s say, one of the observers wants to send a message to you, and they, or, you know, they just do everyone, and they type it in the chat, it will read out on the screen reader to the participant while they’re testing the website. And that can be really confusing for people. It will really disrupt their flow, ’cause they’ll think, “wait a minute, what just happened? Is this something that is on the website I’m on?” So you, we say don’t use the chat, just because it’s better to have a blanket rule not to use the chat than to try and be like, “make sure you don’t include everyone.”
And then, you really want to make sure the tester shares their computer’s sound when they share their screen. And you want them to share their full desktop, not a single window, and this is because very frequently we see on websites where there are things that open in new windows, and if they click a link that opens in a new window, like, let’s say you want them to test the social sharing function, and it opens up a new window to share something on Facebook.Well, you won’t have access to that if they haven’t shared their whole desktop, you’ll only have the original window. So for full, making sure you don’t lose anything, you want them to share their full desktop.
So then you need to define the scope of what you’re testing. And first, you need to clearly understand why you’re testing. Are you testing because you are trying to determine if the website is fully accessible because you’re being sued? That’s a whole different scope of what you might test than, um, you’re, even if it’s you’re testing for accessibility, because you want to be proactive, which is awesome.
Amber Hinds 42:25
Um, you want to know, are we trying to meet a certain level of accessibility? So is this a Section 508? website? Is this a website in Ontario that has to comply with the Ontarians with Disabilities Act? Is this, is this something that we’re testing because we don’t think the conversions are as high as they should be? Is this something we’re testing because we’re just getting complaints from customers that the website isn’t usable? So, really knowing why we’re doing the test in the first place helps us to shape what we should be testing, and, um, what the goals are for the testing session.
And sort of in hand with that is, we really want to make sure, you know, does the client have any other specific questions or concerns? Like, sometimes they just want to know, you know, “does my pop up bother people?” The answer is always yes, but, sometimes they need to have users tell them that, right? So, I think, we always say, “Okay, why are we doing this test?” And then we say “what else would you like to know?” Because invariably, you’ll hear “Oh, we also want to know about XYZ,” which didn’t come up when you just asked “why are we doing user testing in the first place?” So ask that question twice.
And then, um, and then once we’ve done that, you want to create a list of the top tasks that need to be tested. So, some of the key things that we like to test for is can people get contact information for the business, um, or the organization, whatever that is. Can they find whatever important information is for that company, whether it’s who they are, what services they provide? If it’s an eCommerce store, can they find out what the refund policy is? Right? Like, any sort of key important information, can that be found? Can they fill out forms, and do the forms engage the way they expect them to? eCommerce functions are obvious. Add to cart, they have to be able to complete a full checkout process.
And then, any other complex functionality that you have on a website. So, if you have a database that they can search and filter. If you have calendars where they need to go see events that are upcoming. If you have tables with a lot of different data in them. If you have something where they can save or favorite items, or put them on a wish list. Any of that functionality would be key things that you would want to test for.
So if you are moderating a testing session, a few tips. One is, as much as possible, you want to be friendly, and approachable. Smile. Smile, even if you are speaking to someone who cannot see your face, and you are on Zoom, because they will hear it in your voice. And I think what’s really important is we want to make people feel comfortable, because what we’re going to ask people to do is describe what they’re thinking as they’re trying to move through a website, and that’s not normal. Like we don’t normally, I don’t normally go to a website and be like, “well, I’m on the homepage, and I need to find your contact information, so maybe I’ll look and see if there’s a contact in the navigation,” right? Like, we’re asking them to really verbalize their thought process, so as much as possible, we want to make them comfortable.
And that goes to my next item that I have here, which is start with some small talk. You don’t have to dive right into it. You can chat and be friendly, you know, relieve some tension a little bit, try to build some rapport. Then we like to tell testers, what the general goals are for the session. So, we’re not necessarily saying “we want to know if you can do blah, blah, blah, blah, blah,” no, we don’t tell them that in advance. But what we say is like, “our goal is to make sure that this is usable for everyone. You’re here because we’re especially interested in making sure it works for people with screen readers, and we’ve done some testing, but we’re hoping that you can provide some extra guidance. If you can’t do something, or you find something, that’s not actually a bad thing. It’s not a reflection on you. That just means we haven’t done a good job, and you’re helping us.” So you want to, them to make sure they don’t feel like this is like a test of them. This is a test of the website, and they’re providing information that can help you to make it better.
Amber Hinds 46:55
And then, while you’re, while they’re doing the testing, try as much as possible to ask open ended questions, not leading questions. Don’t ask yes or no questions, you want to try and ask something that will get them to give you a full sentence and provide extra context. Of course, it seems obvious, but I want to say it, don’t get offended if they say something isn’t good. If they say, “this is bad, this sucks, this is hard to use.” Or if they seem frustrated, you can’t, as a moderator, let that bother you or take it personally.
We only know what we want to know. And I always think like we’re, we’re doing, unfortunately, above and beyond by doing user testing, right? So not every, most people don’t do user testing on their website, so just by doing that, you’re putting in effort, that’s awesome, you are awesome. So, if they say that something doesn’t work on the website, you have to just say, “Okay, you know what, I got the feedback, that’s good, I’m gonna go back and make it better.” And try not to take it personal, even if they’re like, “I really hate this color of green,” and it’s your favorite color of green. Right? You have to, you have to try not to let it bother you like that.
And then, another big tip for screen reader users, if you’re not familiar with screen readers, and even I do this and I use a screen reader on a weekly basis, um, ask screen reader users to slow down their screen reader. They’ll frequently hear it at like, you know, five times speed, maybe faster. And if you’re not used to hearing it, you might not catch what’s being said, and you might not actually notice what’s hanging them up. So, we typically ask them to slow it down to like two times, rather than as high as they might have it. And then we’ll kind of make a joke about how we can hear as good as they can. Because I, I have a hard time, but I feel like, especially because you’re going to be recording, and you may be sharing the recording with a client who’s never heard a screen reader at all, or if it’s, you’re not doing it for client, you’re doing it for your own internal organization, it might be for a marketing director or someone else, or your developer who may not have heard a screen reader, so slowing down the screen reader is good.
So, our general script is, is we want to start by asking their name, and we have them state on the video that they give us permission to record the session. You may or may not, depending upon your organization, also need to have them sign a video release. I’m not a lawyer, so I’m not going to give you any advice on that front. But, um, we always at least have them say on video that they give us permission to record the session so that we have that on the recording. Um, then we talk about the goals for the session, like I mentioned already. Tell, we talk a little bit about honest feedback. Ask them, you know to verbally talk through everything they’re doing and explain, you know, what they’re hearing, what they’re thinking as they’re moving through things, and maybe give them some examples.
Then we’ll ask some warm up questions about who they are, their backgrounds. We ask them to tell us on the video how they’ll be interacting with the website, so, which device, which browser, which screen reader, if they’re using a screen reader. Any sort of demographic information you might want to ask here, but really, you don’t want to be invasive, right? It’s just as much as you need in order to be able to put together a report. Or if you’re looking at multiple users be able to identify, okay, this was the demographics on this person versus this person. Any sort of small talk you can do is great, because again, you want to try and make them feel comfortable and at ease, and not like this is a test or something scary.
And then, and then we go through our task list and the scenarios for the website. And then we ask follow up questions. So, when you’re coming up with your tasks for the testing, I have a couple of do’s and don’ts. So the first do is do use clear and direct language appropriate to your tester’s abilities. This is especially important if you are working with people with cognitive disabilities, but I also say this on the other end, because just because someone is a native user of assistive technology does not mean that they’re dumb or not tech savvy.
So, you really want to make sure that you are being, um, conscious of the language that you’re using, um, and you want to make sure you’re using a language that is understandable, but also takes into account that you don’t necessarily have to explain every little thing. And that’s my second do which is use appropriate terminology and avoid personal references or offensive language.
Amber Hinds 51:54
There are a bunch of great resources on the web, and in some different places I can point you, if you’re looking for, like, words that you should not use. I don’t have them included here, but you definitely want to make sure that you’re being conscious of that.
And then, on the test, some of my don’ts are, so don’t include too many steps in any one task or scenario. So, you want to think about what would a real user do, and how would they get there, and you sort of want it to make sense in the flow of testing. So, if you have a few different goals, they should probably flow into one another, so that you’re not needlessly sending people back and forth across the website.
And then, finally, don’t include subtle hints in your prompts. This is probably the hardest thing as you’re scripting out questions, is you really want to try not to lead people. So, I’m going to give you an example. Um, you want to try and present scenarios where you paint a picture for them of what you ultimately want them to do, without telling them how to do it. I have a screenshot up, that shows an interior page on the Workforce Solutions Panhandle website. This page has a map at the top, and it has filters in a left hand column that show certifications, operating days, non traditional hours. And then in the center, there is a location search, and there are results that show childcare centers. And, so an example, for this page, or this particular site, we would actually maybe, let’s say be on the homepage of the website, and what we would say is “pretend that you are looking for a childcare provider for your infant that is open on Saturday. How would you find that?” And that’s all we would say.
So, but we would know in our head, like what do we expect them to do? So on the homepage, which you cannot see, but you can see the navigation up at the top, we would expect them to probably go to the navigation initially, and maybe go to the childcare option there, and then under childcare, there’s a drop down, which I’m not showing, that says “search childcare providers.” Or, the other way off of the homepage that they might get to it is there is a call out further down on the home page that would also bring them to the childcare page, and then from the main childcare page, they could go to the search childcare providers page.
So, we know there’s two ways from the homepage they could get here. So, the first thing we’re wondering is, can they get to this page? And then the second thing we’re wondering is, can they use those filters to find a childcare provider that takes infants, and there’s filters further down, you can’t see, but, that does child’s age, and is open on Saturday.
And then, we would observe as they move through the website. We might ask questions, but we would try not to lead them unless they got super lost. So, if they couldn’t find a way to the child care page, we wouldn’t abandon this task. We might eventually say, “Okay, here’s what you need to do. Go to the navigation, under childcare, the fourth item is search childcare providers.” And then they get here, and then we say, “Okay, now what would you do?” So, you might still sometimes have to provide some guidance. Hopefully you don’t, because hopefully, your navigation is clear enough.
But then you sort of want to see, you know, what happens when they get to the map? In this particular instance, there’s actually a hidden screen reader text that says, “there’s a map below, click here to,” I can’t remember exactly what the wording is, but basically, it’s like, “there’s a map below this, you can skip the map and go straight to the filters.” So, we want to know, like, do they see that? Do they use that? Do they get hung up on the map? Because Google Maps are crazy annoying for people on screen readers. Um, and not super helpful. So, like, there are certain things that we’re already thinking like, does this work? Can it be made more easier for them? And then we’re going to watch as they move through it.
But you’ll notice that, in the beginning, the way I framed it is “pretend you’re looking for childcare for this specific thing. How would you do that?” Rather than me saying, “okay, go to the childcare page, and then find,” because we want to also test that navigation. We want to test that funnel, and that moving through the website, and actually know, did they go where we, did, is that as intuitive as we think? Obviously, we want to test the filtering, and does that work, too. But we also want to know, how, can they get there?
Amber Hinds 56:53
So, some questions to ask along the way. Again, we want to really try to be open ended. But if some people have a hard time, like, just verbalizing what they’re doing, and what they’re thinking, so sometimes we need to have guided questions. So we might say, “What are you thinking?” If they go off task, or they’re somewhere that we don’t expect them to be, then we might, you don’t want to say, “why did you do that?” Because that implies something sort of negative, right? Like they did something wrong. And really, there is no wrong in this. So instead, we’ll say, “What are you trying to do now?” And then they might tell us what they’re trying to do, and we’ll be like, “oh, okay, we understand why you’re doing that.” Right?
If something happens, and they maybe say, “Oh, this is weird,” or they just seem confused, we might say, “what do you think happened,” to try and figure out what they think happened on that particular page. “What would make it easier for you to find what you were looking for?” So, if they say “I just I cannot find that,” like, ask them that question immediately as a follow up, “what would make, what of, what would have made it easier for you?”
“What would you do next?” is another thing. So, so for example, if they really need our help finding that page, then we might get them there, say, by going to this item, then we say “okay, now you’re here, what would you do next, in order to find these things?” to try and get them on track so that they’re then taking control again of the session. Maybe asking people “what is missing?” “What would you do if I wasn’t here to help you?” If they, like, keep asking for a lot of help, like, we sometimes have to say, “Okay, well pretend I’m not here. What would you do if I wasn’t here?” Because we really want to get like, what would this be like for them if they were in their home, or on their phone, or whatever that might be without someone to help them?
And then, if they say something, and we’re not really sure, like, why, you know, we’ll say things like, “Is there something in particular that made you think that,” like, again, trying to frame things, so they’re, they’re very neutral, and not negative. I’ve heard, I’ve heard someone say, it’s like, what would a therapist say? So you think about like therapy sessions, like, there’s no judgement in what we’re saying, we’re just asking questions to get more information. And that’s kind of how you want to frame it.
So then, after you’ve gone through all the tasks, then we do closing questions. Sometimes we have items that we want to follow up on, but we don’t want to disrupt their flow, so we might circle back to those in their closing questions. So just like, “you mentioned something earlier, and I didn’t want to jump in at that time. Can you tell me more about that now,” like, “you said, this thing happened, and I didn’t really catch that,” or, you’re like, “Oh, that’s weird. But I didn’t want to interrupt you. So can you tell me what was weird?” And maybe they’ll go back to that page, and they’ll explain it to you.
Um, we generally ask them, “How do you feel about the website overall? How does this compare to other similar websites?” So, especially if you know you have top competitors that you think this user might have been on a website for them, like, ask them, does this, you know, “how does this compare to other websites,” or, you know, similar things, because that can give you a feel on if you’re on track, if you’re not on track. “What information or features were missing?” “What would make this website more usable for you?”
I have, and we can 100% throw these up in the chat, I’ll do that in just a minute, I can also throw the slides, or you can screenshot it, but these are very long URLs, but I have a couple of example scripts and reports. 18F, which is a government entity that is specifically focused on accessibility for Section 508 websites, but they have a really great usability testing script. GitLab also has one for writing usability testing scripts. And then Steve Krug, who’s my third list, he has a ton of great resources, including checklists, a usability testing script, and different things there. And then the UI Access accessibility testing checklist, the, what I like about that is it’s, it’s not really like a script, but it’s things you need to think about, or things you need to do in order to prepare for accessibility testing. So that’s super useful.
The Usability Test Plan is, I might actually click on this real quick. Let’s see. This is, some people like to have sort of a one page document. So what this is, is a template on Medium that I found- it’s asking me to sign in. I’m going to say no.
Amber Hinds 1:01:48
And you can put, let’s see if they have an example of one that’s filled in. Can I make it bigger? Nope. Oh, tiny images are no good. Um. Anyway, you can put what you’re testing, what the business case is, what your objectives are, who your participants need to be, what equipment you think might be needed, testing tasks, the different responsibilities, locations and days for testing, and then what the actual procedure would be. So, this is a template that’s available, it’s Creative Commons, and it’s sort of useful if you like this sort of thing for planning out your testing.
And then, the usability.gov, this is a Word doc template that is downloadable and free, and could be used if you’re writing a report. So, if you’re trying to start doing this for clients, and you want to, you know, figure out, like, “what should my report look like,” this would be a great place to start. The last thing I wanted to share, and then I will open up for questions, is the, gathering bulk usability data. I have two resources that I think are sort of interesting.
The, Hotjar we use all the time. Hotjar, what it does is, you put a script on the website, and it, it will record user sessions and it will anonymize them, so if they fill in a form, you won’t be able to see what they wrote in the forum, it just puts asterix there, but you can watch people move through the website. This isn’t fully accessibility testing, it’s more like user testing, or usability or user journey watching. But Hotjar has a free plan that allows you to record 100 sessions a day. So, it’s really great if you’re just trying to figure out, you know, what people are doing on the site, where they’re going. Um, they don’t even know it’s happening. So, it’s, I will say like, the more JavaScript you put on your website, can slower it down, so I don’t know if I recommend just, like, leaving it on your website forever, but doing it for a certain timeframe could be really helpful.
Loop11. I will share this real quick. Loop11, I just found out about this and started playing around with it a little bit. But they do both, it’s software that can be used for moderated accessibility tests where you’re actually engaging with them, but also unmoderated where you build out a flow, and so let’s say you add a customer list, and you build out a flow and you wanted to send it to all your customers and ask them to go through, and I’ll play a video for just a second here. I might have to stop sharing for a minute and make sure that you guys can hear my sound because I might have forgotten to share with sound.
But right now up on the screen, you can see there’s a prompt open over this website, and it says “please see if you can find information on the types of companies and industry sectors, sectors the Fyre Agency has worked on.” And then in the top right corner, there is a woman. So you have the ability to even record people’s faces if you want. The caveat I have about this, I haven’t fully tested it, is, it seemed like it worked okay with a screen reader, but I saw some color contrast issues. I’m not 100% confident, because I haven’t done full testing on it, that you could use it for, like, with blind user testing. It seemed to work fine with keyboard only and no mouse, so you might be able to use it for keyboard only users. But I’m not certain about low vision or blind. So, that’s something if you want to do, I would recommend looking into a little bit more. Just, I don’t want to promise that it will work for that when I haven’t fully tested it.
But let me stop sharing for one second, I will reshare with sound. Because I think I forgot to do that. And then I’ll play this video for a second. And then I’m happy to take any questions. So, this is an example of what, how Loop11 Records information.
Loop11 Online Tester 1:06:15
Okay, so the first task is “please see if you can find information on the types of companies and industry sectors the Fyre Agency has worked with.” I’m going to hide this overlay. I’m looking for information on the types of companies, and industry sectors. So, my guess is that I’m going to look at the, in the, either the About, or the Clients links from the top menu.
I went to About first, just because it was the first one there. I’m reading it, and it’s talking about what they do, one stop shop, expertise to see what skills we draw upon. Okay, well, let me look at Clients then. “Clients, we love them more than our own children.” Okay. Since 2011- 2001, “we’ve had our fair share of clients,” common challenges, how we’re able to help. Okay, so, I guess that does not have the types of industry.
Let me get the overlay back to see, see exactly the information. Types of companies and industry sectors. Well let me look at expertise, then. Expertise is founded on unparalleled knowledge, UX and Research. Their toolkit has, well, it looks like these are the skills that they focus on and the, the, well, the methods that they use to do whatever it is that they’re doing. We’re still looking for types of companies and industry sectors.
I’m gonna go to the home, and I’m just gonna scroll down, and see if anything stands out to me for types of companies and industry sectors. How to keep up and make design decisions quickly. Five tips for conducting guerrilla research. Doesn’t sound right. User experience if PokemonGo. No.
Let’s see, I clicked on the Fyre Agency link. I’m assuming it’s going to take me back here, but just in case it takes me elsewhere. I guess this is a tech company. But maybe I needed to read the About more clearly. This is, a lot of this is, uh, vague. I don’t know what “achieves certainty with us.e” means. It’s possible that’s a tech thing that I don’t understand. But I don’t know what it means. user experience and customer experience.
Amber Hinds 1:08:59
Alright, I’m gonna pause it, but um, this is, I wanted to play this not just because, you know, Loop11, they have, they put that app on, and it allows you to collect it without you moderating, but I think this is also a really good example of sort of how it’s an open ended question. Now, and, she’s doing a good job of talking as she’s going, which is kind of what you want to try and get your users to do.
So, as you’re, as they’re going through the site, you want to try and, in the beginning, you might have to coach them a little bit, it would be like, “tell us what you’re thinking” like, what, “what’s confusing you,” but ideally, you want to get them kind of talking like that as they move through the website. I think, this is a fake website with like a sample video, right? Um, but obviously, in this instance, she couldn’t find the thing.
Eventually, at some point she gives up, and she just says “I can’t find it” and she moves on to the next step. And, and it will record that, and put everything in a dashboard for you and everything. But it’s sort of a neat tool that I just recently discovered, and could be an option for you, depending upon which kinds of users you’re trying to get data on.
So, I am happy to open up for any other questions. I also have up here our link for our Facebook group, please join. Um, my contact information is here. You can also find me on LinkedIn just as Amber Hinds is my LinkedIn name. And if you haven’t joined our Meetup group, please join our Meetup group. But what questions can I answer?
Emma Crowe-Fleming 1:10:36
Um, Mark put in the chat, “question: does Loop11 have automatic captions for when the user announces what they are doing?”
Amber Hinds 1:10:49
Great question. And I don’t know the answer. And I’m going to guess, probably the answer, I mean, this video does not. I would say it is possible that it doesn’t. I only, I only got introduced to this, I’ll tell you, like, just recently because we’re a B Corp, and B Corp used it. And so, I just started playing around with it earlier today, because I was like, “Oh, that was cool, I should mention it in the Meetup.” I’m not sure if it has captions. And if the videos are accessible in that way. I can try and look into it and post in the Meetup.
Emma Crowe-Fleming 1:11:27
Um, Dave has asked, “how much better do you think an in person user test is compared to a user test over Zoom? In what way is an in person test better?”
Amber Hinds 1:11:43
So, I feel like, I feel like people in person maybe get a little bit more comfortable. And I, I like that you can see their face, and see their hands on the keyboard, and see their screen at the same time when you’re in person. Whereas when you’re over Zoom, you might see a little bit of their face and their screen, but you’re not going to necessarily see what they’re doing on the keyboard.
So, I just feel like, from like, a doing testing with people with disabilities perspective, you maybe get a more well rounded picture when you’re in person. But, on the flip side, you’re also more limited to people in your area. You know, we’re really lucky that we have Texas School for the Blind and Visually Impaired not very far away from us, but if you live somewhere that doesn’t have that it might be harder. So that’s where Zoom could be great, because you can test with users that are anywhere in the world.
Emma Crowe-Fleming 1:12:56
Those are the only questions in the chat right now.
Amber Hinds 1:13:00
Okay, does anybody else have anything? You can also feel free to unmute yourself as well. No? I guess, hopefully that means I did a good job.
Emma Crowe-Fleming 1:13:20
We have one more question from Dave. He had asked “how much do you pay student user testers?”
Amber Hinds 1:13:28
We pay them $20 an hour. I think from what I’ve heard, that’s somewhat standard, at least in, here in Austin. If you’re, like, in California, you may need to pay more. I think I saw some jobs for testers in California that were more like $25 to $40 an hour. I think that the law firms in California that look for user testers, when they’re trying to identify websites to sue tend to pay more. But that’s what we pay.
Emma Crowe-Fleming 1:14:12
Alright, and Mark has another question. “How often do you repeat the user tests? Do you invite the same user back after fixing the issues to test the same scenario again?”
Amber Hinds 1:14:28
We have not usually invited the same user back, be- only because I think that then you run into the situation where it’s a little bit similar to us where we’ve built it, and we’re familiar with it, or we’ve figured out how to find the information. So, once that user has figured out how to find the information, or they already know what the problem is, you’re not going to get the same natural response to the problem if it hasn’t been fixed. So, in that instance, we want to, I’m gonna, let me stop sharing, too. In that instance, we generally would bring in someone new. I don’t think it would be wrong to bring the same person back, but I don’t think you would necessarily know was it really fixed? Unless it’s an obvious, you know, where they flagged something for you. So.
And then as far as repeating user tests, I think, you know, on, on our front, it really depends, we’re doing, on, for clients. So, I think it depends on the client budget, to be honest. Like, I wish it was, I wish I could say, like, we’re doing it all the time for everyone. But I mean, we even have new websites we launched where they don’t have the budget for user testing, and so we’re only doing our own testing. I wish every website had user testing. But the reality, unfortunately, is we’re not there yet. Maybe someday we’ll get clients where they all have the budget for that.
And I think, like, on an ongoing basis, I would say that if you’re maintaining a website, what we tell our clients is that if you add any major functionality, or you, like, change your navigation structure significantly, that would be a good time to pull in a user to test it. Or, again, like I mentioned, if you’re not seeing the conversions that you expect to see, or if your bounce rate on the site, anytime that you think the site’s not working for the business, then that’s probably a time to pull in a user.
So, it could be you repeat it once a year, it could be you repeat it in three months when you make some big change. Or it could be that you don’t repeat it at all, because the website’s pretty static, and all you’re doing is adding blog posts, and nothing really major has changed.
Emma Crowe-Fleming 1:17:06
Um, Dave asked, “typically, how long are your user test sessions with student users?”
Amber Hinds 1:17:13
So, it depends a little bit on the size of the site. We are typically doing, in that instance, like, when they came and we did the panhandle website, we actually did, it was almost four hours of testing. But it was like a two hour session, lunch, we bought them lunch, we all hung out and had lunch together, and then we did another two hours.
If it’s ed- but in that instance, they were testing a front end website, and they were testing a portal where users basically, that, that particular portal is for people who are getting assistance. So, like SNAP benefits. And they have to go through the Workforce Solutions to, in order to receive their SNAP benefits to prove that they’re working towards getting a job. So there’s this whole onboarding process with that portal where they have to upload all these documents, and fill out a ton of forms, and sign things, and it had to be accessible because it’s federally funded.
So, it was really testing two websites, it was testing the website in the morning, and then the portal in the afternoon, basically, is the way that worked. I think if you’re doing a tighter test, you maybe only have, you know, five to eight items that you are really trying to figure out, so, let’s say it’s an eCommerce store, pretty much, you know, you could probably get that tested in 30 minutes to an hour, potentially. So, a little bit of it depends on how, how in depth the website is, and what all you need to test and prove functionality on. Do we have any other questions?
Emma Crowe-Fleming 1:19:07
None in the chat right now.
Amber Hinds 1:19:14
Okay, well, I appreciate everyone coming. We will post up a recording-
Emma Crowe-Fleming 1:19:20
-Sorry-
Amber Hinds 1:19:21
-it usually takes us- Oh, sorry was there something else?
Emma Crowe-Fleming 1:19:24
There’s one more question. “How many persons Do you have to test the site?”
Amber Hinds 1:19:30
How many, Gerson, are you saying how many users? Is that what you’re wondering? If you’re saying how many users, then we recommend a minimum of two. Because if it’s just one, one person, then I don’t think you really get a good picture. Like, it could just be that one person was confused about it. So, I’d say a minimum of two, in an ideal world, I think probably around five would be better.
And this is what I said earlier, if you’ve got something really complex, and you really want to, like, cover a broad range of disabilities, then you might be looking at having, like, I’ve seen some where they’re doing like whole user studies, and putting it together, like with quantitative data, and they’re doing like 30 or 40 different users. Which is where maybe something like Loop Net would come into play. Or, or you, you know, hire a company that does that, and they consolidate it all into a giant report for you. We don’t typically do user testing at that scale. But I know that there are organizations that do, like, mass user testing. So.
We, anyway, I really appreciate it, we will have the video up, it usually takes about a week because we have to make sure the captions are all correct on the video, and we will have a full transcript. And that will be posted on our website, and it will be posted in the Meetup. We are continuing to work on, I’m really hopeful that we’re going to have a caption sponsor for our daytime Meetup as well. But that one will be October 8. And if you’re interested, go to WPCampus tomorrow. I’ll be leading the accessibility testing community discussion, and then also a panel, a separate panel discussion as well. And I’ll be manning the WPCampus Twitter so if you tweet at WPCampus tomorrow, I’ll be tweeting back at you, so.
Peter I. 1:21:31
Awesome.
Amber Hinds 1:21:31
I hope everyone has a good night.
