Accessibility Testing: What Automated Tools Will NEVER Catch: Tanveer Khan

[00:00:00] Meetup Introduction/Announcements

Amber: Welcome to WordPress Accessibility Meetup, Accessibility Testing: What Automated Tools Will Never Catch, with Tanveer Khan, who is a Senior SQA and Accessibility Specialist at Arbisoft.

A few things to be aware of. We do have a Facebook group, so if you are interested in connecting with other attendees in between meetups, you can find that at facebook.com/groups/wordpress.accessibility.

It is a great place to share what you’re working on, get feedback, help other folks, and just generally chat about WordPress and accessibility best practices.

Everyone always asks, “Is this being recorded?” Yes, it is being recorded. You will be able to get the past recordings, this recording, all past recordings, and upcoming events if you go to EqualizeDigital.com/meetup.

We have changed our process around these so we can get them out faster, so we should have this recording available to folks later this week, which I’m very excited about.

Also, the other way to get notified about the recording, beyond just watching our website to see when it gets published, is you can join our email list. We send out news and event announcements. If you go to equalizedigital.com/focus-state, you can subscribe there.

And we also release audio from these meetups on our podcast, and you can tune into the podcast at accessibilitycraft.com.

If you have any suggestions for the meetup or you need any additional accommodations to make the meetup work for you, you can contact myself and my co-organizer, Paola, if you email meetup@equalizedigital.com

Who am I? I know I have not introduced myself yet. If you’re new here, my name is Amber Hinds. I’m the CEO of a company called Equalize Digital. We are a mission-driven organization and a corporate member of the IAAP focused on WordPress accessibility. We are the lead organizer for this meetup. We have a WordPress plugin called Accessibility Checker that helps you find and fix problems on your WordPress site. We offer online courses for NVDA and VoiceOver screen reader testing, as well as a course on how to sell accessibility. We do accessibility audits, remediation, and consulting, and lots of education just like this. Our website, as I mentioned before, is EqualizeDigital.com if you wanna learn about any of those things.

We do have a sponsor that I wanna thank today who is generously covering the cost of our live captioning and our transcription, and that is GoDaddy. GoDaddy’s mission is to empower a worldwide community of entrepreneurs by giving them all the help and tools they need to grow online, including a simpler, safer WordPress experience.

GoDaddy provides a managed WordPress experience that is as easy as it is effective. The latest version of WordPress comes pre-installed with exclusive themes, plugins, and tools to get you up and running quickly with automated backups, updates, and malware removal so our pros can spend less time on monotonous maintenance and more time building their businesses. You can learn more about GoDaddy if you go to GoDaddy.com.

I always ask if you are willing on any social media platform that you may be on to tweet them or tag them or comment on one of their things and say, “Thank you for sponsoring captions for WordPress Accessibility Meetup.” It helps to tell them that this matters, and when it comes time for us to ask them, “Hey, will you please renew your sponsorship?” That sort of thing is also evidence that it does work and that people are talking about them, and it helps to encourage them to want to continue supporting our live captions.

Two upcoming events that I want to make sure you are aware of. On Thursday, September 3rd, in this same time slot, Cam Coulter will be presenting How to Learn WCAG. And then on Tuesday, September 15th, the same time again, Maria Maldonado will be presenting Email Accessibility in WordPress. So all those emails that your website sends, how do you make sure that they are accessible?

I am very excited now to introduce our speaker, Tanveer Khan. Tanveer, if you don’t mind turning on your camera, I can pull you up here in just a second for everyone.

There we go. Tanveer is an accessibility expert specializing in web and mobile accessibility testing, WCAG compliance, and inclusive user experience evaluation. He has experience conducting accessibility audits, identifying complex accessibility barriers, and collaborating with development teams to implement accessible solutions across modern web applications.

We are so excited to have you here today, Tanveer. Welcome.

I’m gonna let you take over with sharing. I’m gonna go off camera.

For everyone watching, there is a Q&A module in Zoom. We will come at the end to do questions and answers, so please put your questions in there as we go.

[00:05:43] Presentation Introduction

Tanveer: Sure. Welcome everyone, and first of all, thanks to all of you who are watching and listening to me right now, and thank you for joining me today.

As Amber said , my name is Tanveer Khan, and I’m a Senior SQA Engineer and Accessibility Specialist working at Arbisoft.

Today’s session is about a topic that I think is especially important for anyone who is involved in accessibility, rather in the form of QA development or product design.The title of today’s topic is Accessibility Testing: What Automated Tools Will Never Catch.

Before we begin, I want to make one thing clear that this talk is not against automated accessibility testing, but it is quite the opposite of that. Automation has become an essential part of how we test accessibility today.

What I want to explore is, I want to tell you guys about where automation is extremely effective, where it naturally reaches its limits, and how we can combine it with manual testing to get a much more clear and complete picture of accessibility.

Let’s get started.

That is our agenda of today’s talk.

Before we get into the technical details, let me quickly give you a walkthrough of the journey we will take today. We will first start by looking at how accessibility has changed throughout the years and how automated tools have become an important part of today’s modern software development methodologies.

After that, we will look what automated tools are designed to do and why they are so good at certain types of accessibility checks, and where it reach their limits. After completing that, we will move beyond the code, and we will see that, and we will talk about accessibility from the user’s perspective.

After doing that, we will move to our key question that why does automation have limits, and how would we can reduce them. After that, we will introduce our four areas where automation cannot fully understand the scenario, and how manual testing and humans would be needed to accomplish those four areas.

And then, we will bring automation, and then after that we will see some real-world example, and then we will see how to combine automation plus manual testing. After that, we will see that how to build a complete accessibility strategy in which there will be automated testing, manual testing, and assistive technology testing.

We will summarize the key takeaways after that and move… and we will move towards the live demonstration. There will be a short live demonstration, which I think you would guys, would like it. And then at the end, we will move toward the Q&A.

[00:09:06] Accessibility Had Changed

Tanveer: Before we talk about limitations, I want to talk about how much accessibility testing has changed. Today, we have some powerful tools like axe DevTools, Lighthouse, Wave, Accessibility Insights. They have transformed accessibility testing by making the accessibility checks much faster, repeatable, and they are easy to integrate in our development workflows like CI/CD.

Nowadays, instead of waiting until the end of the project, teams can get the accessibility feedback much earlier. Before it used to be very late, but now due to automated accessibility checks, feedback, getting the accessibility feedback is much simpler and easier and much earlier. And I think that’s a major improvement throughout the years.

Automation gives us consistency. It can gives us repeatability. It can scale across large applications, and it can provide quick feedback when the code changes.

I’m absolutely not against automation, and I absolutely want us to keep using these tools. But this thing lead us to the question which is written on the right side of the slide that “If these tools are so powerful, why do still inaccessible experience still exist?” That’s the question I want us to keep in mind throughout this session, and we will try to find the answer of this question to how to reduce the inaccessible experience.

[00:10:49] What Automated Tools are Designed To Do

Tanveer: Looking at the screen that what automated tools are designed to do. First, we need to understand what accessibility tools are actually designed to do. At their core, these tools perform rule-based validations. What they do, they get a page, they inspect the rendered page, and evaluate things that can be checked against defined accessibility rules. Automated accessibility tools, there are some rule defines against them, and automated tools run on the basis of those rules.

For example, a tool can identify missing alternative tags, it can identify missing form labels, it can also identify color contrast issues, button links, invalid ARIA implementation, missing document language, and some more issues like that.

We have also an image, an example, and it’s so relevant to what I have just said, that if an image has no alternative text, the automated tools can detect them. If a input has no associated label with it, the tool can identify them. If the document language is missing, the tool can also identify that.

These are all objective checks, right? And automation is very good at evaluating implementation against the rule. But, the complete user experience is a much bigger question.

The key message of this slide is automation evaluate code quality, but not the complete user experience.

[00:12:33] Accessibility Isn’t Just About Code

Tanveer: Moving to the next slide, now we are looking at the bigger picture.

I think accessibility is not only about code, that whether our code satisfies technical rule or not. I think user don’t visit a website to pass an accessibility scan. They come to accomplish something, right? Maybe they want to find information, they want to understand some content, they want to navigate efficiently, and they want to interact with controls and complete tasks, and maybe they want to complete a task independently.

These things changes the way we think about accessibility because we think accessibility is just a code-based thing. It’s just a code-based testing, but it’s not the right thing. A page can have technically correct code, but still it can have frustrating user experience.

Think about if a person is using a screen reader, if he’s using a keyboard, using a screen magnification or any other type of assistive technology. The real question is not only does the code meet the rule, the question is can the person actually understand whether the foundation actually work for people or not?

That’s why we have shown a modal over here that accessibility is equal to technical compliance, usability plus user experience.

Technical compliance is, related to code, related to WCAG guidelines. Compliance give us the foundation. Usability and user experience help us understand whether that foundation work for the person or for the people who are using it. That’s the difference between them.

[00:14:26] Why Automation Has Limits

Tanveer: Now, we reach to the boundary.

The boundary is: Why does automation have limits? The answer is not that these tools are weak and this and that. No. These tools are so much powerful at today’s time, and it can be more powerful, and it will be more powerful in the future. But, because some accessibility questions are objective, while other depends on meaning, context, behavior, human interaction and user intents.

On the screen we can see that automation can verify presence of a component or an attribute, anything. It can verify syntax, it can inspect attributes, and it can also evaluate relation between the elements. But these are the things that can be expressed as rules, right? And automated tools work on the basis of rules.

But now, consider the other side. I will ask some question where automation will reach its boundaries. Looking at or where automation cannot fully evaluate meaning, context, behavior, human interaction and user intent. The questions are: Does a button communicate the right meaning? Depend on the context. Does this interaction make any sense? Depends on the meaning. Will the user understand what just changed? Does the focus movement feel logical or not? Can the person successfully complete the task? These are human interactions and where humans interaction and behavior and experience will need it. These questions require interpretation interaction, right? And that is where automation reaches a natural boundary.

A useful way to think about it is, I think automation is excellent at whether something is present or implemented in a particular way, whereas humans are needed to determine whether that implementation actually works for the person who is using it.

The key message is accessibility is not only about whether an element exists, it is about whether people can successfully use them or not. That’s the difference which people are going to… Which people need to understand between accessibility. They think that accessibility is just a one-time thing.It’s just a code base. We have just to follow the WCAG, but not. It’s not the thing.

[00:16:59] The Four Areas Automation Cannot Fully Understand

Tanveer: Other thing just I have mentioned related to it like interaction, context, all these things are related to it, related to accessibility.

Looking at the next slide, if automation has these natural limits, what we have discussed, so where do they appear in practice?

I organized them into four categories: integration, context, behavior, experience.

First, interaction. Interaction is, can user operate every feature? This is about how people actually use the interface, including keyboard interaction and custom controls.

The other is context. Does everything communicate the right meaning? A control can have a technically a correct name, but does that name make any sense where the user encounter it, encounters it? It depends on the human now.

Third is behavior. Does the interface respond correctly? Nowadays, modern application change their states very rapidly. Application open dialogues, update content, display feedback and move focus, open popups.So we need to evaluate whether those behaviors support users or not.

And the final is experience. And experience we all know can users successfully complete their goals, and what are they experiencing in using our product?

These fours are connected to each other. Interaction affects experience. Content affects understanding. Behavior affects predictability. But together, they shape whether the product actually work for the user, and that’s our main concern in term of accessibility. It’s not about WCAG. It’s not about just passing the code, having arias. Our main final output is whether a person or a user or a person with a disability is able to use our product or not.

[00:19:04] Examples Automation Misses

Tanveer: In the next two slides, we will make these categories more concrete with examples.

Now let’s make the framework more practical.

We will start with interaction and context. On the interaction side, manual testing is important for evaluating keyboard workflows, logical focus movement, keyboard interaction with custom components, consistent focus management, and expected keyboard behavior.

First of all, I want to be precise here that automated tools can detect many keyboard-related issues. The point is not that the automation cannot test keyboard accessibility. The point is that humans need to evaluate the complete interaction and whether that workflow make any sense or not.

For example, if you imagine a checkout flow, we have a checkout page where every control is technically keyboard accessible, but after an action, focus moves somewhere unexpectedly. A tool may identify some underlying issues, but actually moving through the workflow reveals whether this experience is logically correct or not.

Now, looking at the context. Automation can identify many links related issues, their links purpose, buttons, headings and structures, heading structure, and this and that. But context asks a different question, and what it asks is it asks, “Does a link make sense where it appears? Is a button have a clear label? Does the reading experience make any sense? Is our reading experiences logical or not? Are the instructions understandable? Is the content organized in a way that helps the user to understand it more easily?”

For example, we have an image over here which is just explaining and written in the it in written in that image what I have just explained.

You can say a checkout page or a simple static page in which we have headers, and there are some parse and manual reviews options. But, in that picture there is a Read More link, and the context with it is, “New arrivals are now in store. Read more.” Now imagine about this link. If you look at the context of that link, I think it’s a valid accessible link, but whether it communicates useful purpose, it depends on the context.

You may know that some people use screen reader, and there is an option using router. You can extract the links from it. So when you extract and see all the links of the page without their context, then read more, and anything that is not mechanics on will be disturbing for the person who is using screen reader because it meaning will not be understandable, and it will be not understandable for a person who is using a screen reader.

The distinction I want you to remember is that detecting something and evaluating it are not the same thing. They are the different things. This automated tool is detecting that, the screen name is correct or not correct, depending on its content. But, evaluating it that if whether the link is appropriate and whether the link is accessible depends on its context is the thing that would only be done using humans.

So manual testing would help us here more than of the automated test because a user know what’s the correct context of this link.

Moving further, we will see some more examples regarding it. Now, we will read about the other two areas, behavior and experiences.

As I say, as I mentioned earlier that modern applications are dynamic. They don’t simply display a page and stop. In today’s applications and modern web applications, focus changes after every actions, contents updates, dialogues open, validation message appears, status message are announced, confident changes.

For behavior, we need to ask questions such as what happened to a focus after a certain user action or after a specific user action? Is dynamic content communicated properly? Are the interactive component describing their states, or does the user receive useful feedback or not?

Automation can verify many technical pieces of these behaviors, but manual testing is needed to confirm that the complete behavior are working as intended or not.

Again, imagine a form where an error message appear visually… where an error message appear visually. We do not need to only verify that the error exists, but we also need to verify does that error message is accessible to the person who is using the screen reader. Or in simple words, in our accessibility world, we say does that message is programmatically associated with the field? Because only in that case the screen reader user know that any error message is assigned with the with form field or not.

On the next option we have experience. Experience is quite simple, and the question arise when you are testing about experience is, can the user complete a task? Is the workflow usable? How much unnecessary cognitive effort is required to understand what is written on the page? Does the overall interaction create confidence? So these are outcome-oriented questions, right? A page can have a technically strong accessibility foundation and still can make a task difficult to implement. So still ultimate question is not simply that did the scan pass? The ultimate question is can the user successfully complete their goals or not?

And that bring us to the solution that automation and manual testing need to work together. I have an image over here. It is also a webpage image in which we have a pop-up which is… which wants our confirmation, confirm shipping address, and it’s a confirmation pop-up in which can answer a confirm button there, and there is also an error message appears on the email field.

[00:26:06] Automation and Manual Testing Work Together

Tanveer: Moving to the next slide, now, that’s my favorite slide. Automation and manual testing work together.

At this point, I want to make the center position of this presentation very clear that this is not about automation versus manual testing. It is about automation and manual testing, right?

Automated testing give us speed. It is repeatable. It’s scalable in large application, excellent for CI/CD, and very good at detecting common issues, and those common issues related to rule-based issues, right?

And on the other side, manual testing gave us context-based issues. It evaluates interaction, validates the user experience, and applies human judgment as well.

The goal is to not choose one, the goal is to combine both.

Let’s imagine a new checkout page where we can do… automation can identify missing labels, issues related ARIA, contrast problems, and other rule-based related issues.

Once those are fixed, now we can test the workflow using only the keyboard to see the keyboard-related issues. Then, we can use a screen reader to verify announcement. We can verify focus behavior and the experience of dynamic content. Finally, after that, we can complete the entire checkout task, and we can say that our complete user workflow is accessible.

Each layers answer a different question. Automation tell us in this regard… in this case, automation tell us that are there technical issues? We can identify it quickly, but manual testing asks, does this actually work when someone uses it? That’s why I have written a goal over here is that the goal is not to choosing one over the other. It is about combining both.

[00:28:22] Building a Complete Accessibility Testing Strategy

Tanveer: Okay, I have told about it that we have to combine it both. But, how do we turn that idea into a real testing strategy? The answer is to build accessibility into a life cycle rather than treating this as a final audit.

Now, I’m describing a complete accessibility testing strategy. First of all, we can start with the design. In the design phase, accessibility should be begun as early as possible. In that phase, we can see the structure, interaction pattern, component behavior, content and focus expectation. We can also verify some color contrast issues.

Then we can move towards the development, where accessibility can help us in using semantic HTML, where appropriate pattern, accessible component design, and accessibility consideration and other components or features are built. We can verify that.

After that, we can move toward automated testing. We can run automated checks during development and where possible in our CI/CD because code changes, and with the change of the code, we can also run an automated test in our CI/CD pipelines. And we can use automation to catch the rule-based issues, and I think it is good at catching that.

Then, we can move toward the manual accessibility testing. We can test keyboard interaction, focus behavior, contextual meaning, contextual structure, and a complete workflow.

Manual testing is needed to test, most importantly, the complete workflows.

After that, we can move toward the assistive technology testing. We can use appropriate assistive technologies, such as screen readers and other relevant tools to validate how the product is actually experienced.

And after that, where appropriate, we can include a person with disability to do some usability testing and accessibility testing in terms of using assistive technology.

The most important, I want to convey a message over here that this thing, this flow should not be a straight line, and it should not be a one-time thing. A single pass through this pipeline is just a checkpoint, not a finish line. A real products, as change continuously. So accessibility testing need to be looped back to the process as the product evolves.

In the first build, we will move from design, development, automated, manual, assistive technology test, user feedback. Then in the another build, when there is some kind of increment, there is some kind of process change, product evaluate, some other components get addition into the product. We should start again our process of the testing strategy.

In that case, we can say that our product is accessible, and because the goal is to make accessibility part of engineering process, not something that happens only once and at the end of the development. That’s not a correct thing. That’s a false thing we assume about accessibility. Accessibility is something that should be ongoing.

[00:32:02] Key Takeaways

Tanveer: Now. we have reached to our key takeaways from this presentation.

Before we move into a live demonstration, let’s bring the whole session down to five key takeaways, so it would be easy for you to remember what we have discussed in this throughout process.

First of all, automated testing is essential but not sufficient. Yes, we have elaborated that we should absolutely use automation because it gave us fast, repeatable and scalable feedback.

Second, is accessibility extends beyond rule-based validation. Yes, we have talked about it. There are questions about meaning, context, behavior interaction, and user goals that requires additional evaluation instead of automated tools.

Third, is human judgment is critical. Yes, a person needs to interpret context, interact with a product, and evaluate whether the experience makes sense or not.

Fourth, is manual testing complements automation. And again, this is not about replacing automated tools. We cannot replace it. It is about using manual testing to answer the questions which automation cannot fully answer. Got it?

And fifth, is better accessibility comes from combining multiple testing methods: automation, manual testing, assistive technology testing, and appropriate user feedback.

Each add a different layer of confidence.

In my opinion, automation answer many accessible questions, but not all accessibility questions. And the goal is to not replace automation. It is to know… We just have to get the knowledge to know where automation ends and human evaluation begins.

That’s a complete strategy for today’s talk that where we can… to gain the knowledge of automation limits and extending the human evaluation. To know about where human will begin and where human is needed, where automated tool will play its role or where a human will play its role.

That’s our complete thing regarding the process and and the message I would love to convey.

[00:34:26] Live Demonstration

Tanveer: Now, we will move toward live demonstration, and we will move from theory into a practical demonstration. And for that, I’m going to use a real website and demonstrate a few accessible scenarios where automated testing gave us useful information, but manual testing help us understand the bigger picture.

As we go through these examples, keep the four areas from our presentation in your mind regarding interaction, context, behavior, and experience, because our live demo would be depend on that, and it will be a small demo.

This is an example page. It’s not related to any website, to any product or this and that. I have just created two page for this to see our issues, our major issues, and it’s not very complex, just a small demo.

First of all, we have product page in which we will talk about interaction and context. When I open the page, there is, it’s about a product catalog. So running my tab on the options, we can see that there is no focus indicator on the options in our header menu. The focus appears on the links, which is in the main section, but not on the top. On the top we have options like Home, Read More, Contact, Shop, but there is no focus indicators.

But when we see the link, the focus is going over there, but we can’t see it on the screen because the focus indicator, the outline, the border is not there. When I inspect my page and run an ARC toolkit, which is our automated tool, when I run a test, there are some errors, it indicating related to normal text contrast, some accessible name and autocompletes, and there are just four issue errors, and the rest are best practices and alerts. It’s not describing anything related to a focus indicator. Let’s run WAVE on this page. WAVE is identifying one error, missing form labels. Not describing anything related to focus indicator.

That’s where we will need human to test related and our context, it depend on that, that interaction is dependent, and we need a human to see whether the focus indicator is there or not.

Another issue we can talk about the links. There are a lot of automated tools like even the axe the… ARC toolkit, which identifies the link purpose. It can detect issues where Read More is used, where something that is not depend on the context is used.

If we read the complete link with its context, maybe the meaning of the link is purposeful. But as I mentioned earlier, if we use a screen reader, and if we extract the links on the page using the router, the link would become Checkout Page, Check Them Out, Grab Yours, Take a Look. So when a person with a disability is extracting those links and navigating on those link the question will arise in his mind that, check them out, what am I checking? Grab yours, what should I grab? Take a look, where should I take a look?

Identically and technically, these links have some meaningful names, but the fact is it’s not depend on the context, and the context and the behavior is not suited over here. That’s why we need a human to identify this thing.

Let’s inspect this. Our toolkit run again, and you’ll see that there is no error regarding the links, and it’s skipping it because according to automated tools, these are meaningful names. And these the they might be meaningful, but in our case, it’s not meaningful because when we extract, in the case of screen reader, it’s not meaningful.

Yes that’s one page. Now, we will move towards the checkout page where we will experience some behavior and issues related to experience.

Now, we have a checkout page on which an email and a shipping address form is output… input fields are given.

First of all, when I click on the proceed to checkout an error message appears that “Please enter a valid email address”, which visually looks good. But the fact is, these email messages, these these valid error messages, is not associated with the label. So a person who is using an input field and he is using a screen reader, he would not know that an error message arise because this message not associated with the our input field. An automated tool cannot detect this because it would only say that he don’t know… he doesn’t know about any… it doesn’t know anything about the error message.

If I add an email in the field and then click on the proceed to checkout, a dialog is open about confirm shipping address. Would you like to add address or send your checkout? There’s two buttonz, Cancel and Confirm. When I navigate it, I see that the focus is navigating out of the dialog, and this will create a frustrating experience that a screen reader user will not know that what happened after when I click the proceed to checkout button. The reason is that the focus is going out, and automated tools does not extract this kind of issues, which is a major thing because user experience is affecting. Its behavior is changing and the behavior of the platform is changing, and user does not know anything about it. That’s a major issue.

When we click on the Confirm, you will see a pop-up message, “Change saved,” which is not being announced by screen reader, maybe due to not use of any live region or any other. And this type of thing is not getting detected by our automated tools.

One issue I just forgot when I was doing in the product catalog page.

If you see visually these links, the links in the header, Home, Read More, Contact, Shop, visually they are in a logical order. But when you run open my inspect page and main, when you see logically so visually they are in a logical order, but when you see in code they are not logical. Their order is incorrect. First, it is going to the home option, then it’s going to the shop which is on the fourth, then it’s coming back to the second option, and then it’s going to the third option. So the logical order, the programmatic logical order is incorrect, but visually it’s correct. But our screen reader runs on logical order, depending on the accessibility logical order, not on the visual order. So that’s the main concern we have in that.

With that, I will wrap up the demonstration, and we are done with our live demonstration as well, and I will open the floor for questions.

I would also love to hear about your own experiences about automated accessibility testing and the kind of issues you have found through manual testing.

And thank you so much for giving me the time. That was all in my data bank right now, which I have used, and thank you so much.

Over to you, Amber.

[00:42:26] Q&A

Amber: Thank you so much, Tanveer. That was fabulous. I really appreciate you doing the live demo, especially showing when a modal opens and the focus stayed on the page, because I see that all the time when I’m doing manual testing. This has been great. We’re-

Tanveer: Thanks.

Amber: … Gonna work through questions that are in the Q&A so please feel free to put them in there. I like to go by most votes, so if you see a question someone else wrote that you- … you can give it a thumbs up, and that will up vote it and get it a little bit closer to the top.

Alice had asked a question when you were talking about user experience. You mentioned how hard is it for someone to understand? Do you have any tools that you recommend that can help with assessing cognitive effort?

Tanveer: That’s the main issue of why I’m here today because I think the cognitive things where we have to understand instruction, I think that’s the main issue where we need humans, right?

As I mentioned in my presentation that automated tools they run on some rule-based verification. There are some rules are defined on that. So defining a cognitive rule on that would be a little bit difficult, but it’s not impossible. I think it would be done in the future. But still, what we can do is we can change our engine, of our rule-based engine. We can tell the automated tools that some scenarios, we can give them some more test data that this kind of instruction is a little bit difficult and this kind of instruction is easy for a person with our experiences. Just like a machine learning, we can make it learn those things that are difficult.

Right now, I don’t have any such specific tool that work on the cognitive because cognitive is something the automated tools can tell you that just… I just mentioned that, check them out. It will say, “Yes, this link is correct.” But when you are cognitively reading the structure, you don’t know about whether this thing is correct or not.

That’s the main thing, I think. And the solution for that is make them automated tools make it learn. Give it some test data to tell them that, yes this is the scenario, this type of thing is incorrect and this type of thing is correct. Yeah.

Amber: Yeah, I think maybe the closest I’ve seen is we did add readability where it checks for the grade level of the language on the page, so that maybe connects in.

But, I think you’re right. It’s really difficult because there’s just a lot of different things that happen that require manual tests and listening- … and the order of operation to really judge is this difficult to understand. And I would always advocate for doing some real user testing. I’m assuming you recommend that as well because sometimes when we’ve built the website, we know how it’s supposed to work and so we think it’s easy to use. And then you bring in someone who’s never seen it before and you might get a very different understanding.

Tanveer: Yep. That’s why we do focus on the user stability testing because I think when a person with a disability, the way he’s using the screen reader and the ways he’s navigating the product or the website, we can’t use that in that way. That’s a real scenario. If I created one thing, I created it because it, it was easy for me, but that’s not logically correct that it would be easy for everyone because everyone thinks differently. Their expertise, their cognitive thinking is different. So yeah, that’s a different thing. Yeah.

Amber: Alex asked, “Will manual testing in one browser usually mean that other browsers would pass too?”

Tanveer: I think not because in my career so far, I have seen that there is some, the default browser issues, right? When you are using Firefox it behaves a little different. When you use Chrome or Safari, it behaves a little different.

In my case, what we do whenever we are auditing a website, we specially mention in our VPAT, in any report, that we have specifically tested on this browser.

The reason is that because maybe in the future, if something arose on a different browser then there would be an issue with that. So we can’t say that every brow… because we know every browser act differently. I used to test on Firefox, but then I came to know that Firefox creates its own focus border, where Chrome shows the focus outline given in the code. That’s a different scenarios. I’m not with that thing. I can’t say yes to this because, yes different browser have different scenarios. They work differently. So 100% not, and maybe we can assume that if it’s passed on Chrome, we assume that it would pass on Firefox and any other thing, but we still specify which browser are we using.

Amber: Yeah, I think probably it’s good to look at browser stats and test on the ones that have the broadest usage, but also look at the audience for the website, and that might tell you if you need to do another one. I know we actually… Yesterday we were just dealing with an issue where a client says, “The videos are broken on our website,” and they’re YouTube videos.

And we’re like, “No, they’re not.” And “I’m looking at them, they’re not broken.” And we figure out, oh, they’re broken on iPhones. Right? But nowhere else. So I think sometimes you have to know what is the audience-

Tanveer: That’s-

Amber: … how are they typically accessing the website, and then make sure you’re testing in that scenario.

Tanveer: That’s why I like the thing, the cross-browser testing. I think the client should. Because every time when we try to do cross-browser testing we run into a time shortage. But I think if a client is okay with that, I think cross-browser testing is much better because we can eliminate such issues you are describing right now, that it’s working on one thing and not working on the other platform or other platform as well. That’s why cross-browser is testing is important, but still we need excessive time for it.

Amber: Peggy had a really interesting question about this testing in different environments. She asked, “Do you test dark mode?” They recently had an issue with the mobile hamburger changing to a color that made it hard to view only in Samsung products.

So how far do you take this testing? Are you testing dark mode and light mode? Are you testing all the different phones?

Tanveer: Yes. We have tested in the mobiles. We definitely test them because I guess mobile is more a useful thing, a usable thing. People use it more oftenly people use it more often than the other technology.

In the mobile case we definitely do test it on the dark mode and the light mode. In the web application we also do that depending on the scenarios.

Some of the clients have also told us that we don’t need dark mode for now because we have not worked on it. And I have seen that many of the platforms, they usually work on one mode, on the light mode or the dark mode.

I have tested it in many platforms, and it gave us a very useful experience and a very interactive experience because it create your experience of using that things. Because whenever there is a color, it’s accessible in one mode, but when you just change, it’s just gone, completely inaccessible.

So yes, dark mode and mode changes testing is very compulsory, and I think it should be added in your testing strategy.

Amber: Corinda asked, “What are your preferred testing tools?”

Tanveer: Testing tools related to automated testing tools. In that case, I like our toolkit because it works much better than other tools like, Oh my God, I shouldn’t have named the tools, but it’s okay. I think our toolkit is working fine for now.

And other than that, I also love to use Wave because Wave is so easy to use. You have just to give the link, and as I have shown in my demo. And all of the issues regarding the structure, regarding the link’s name and different color contrast, it will give you all in one place. But, on the other hand, our toolkit will also lead you to the affected code. That’s, I think, a better thing to use.

Other than that, we have also different tools which runs on their defined tool base. So you cannot answer or filter out only one. For the testing I’m doing right now we are doing cross-browser tools for… if I like our toolkit, I use this. Then if I have time, I run also… I also run the Wave tool. So yes, cross-browser.

Amber: So Wave, and then your toolkit, is that publicly available or is that just an internal tool that you all use?

Tanveer: No, it’s a public tool, and it’s not ours. Okay. It’s a public tool. You can find it everywhere.

Amber: Maybe could you put a link to that in the chat for folks if they wanna check it out?

Okay.

Tanveer: I will, I will.

Amber: I’ll say I also use Wave a lot, especially just on random websites that I don’t have admin access to because it’s quick and easy. But I will pitch, if you haven’t tried our Accessibility Checker plugin, there’s a free version on WordPress.org. And if it’s a WordPress website, it’s…

It can definitely help. I think it’s interesting though, like we talk a lot about using multiple tools because like you were mentioning, Tanveer, it’s all rules-based, and a lot of them have different rules or different logic. And so-

Tanveer: Yeah …

Amber: until, if you’re, especially if you’re learning accessibility, I think it’s helpful to look at reports in multiple until you start to become more familiar, because you might see different passes and failures in different tools.

Tanveer: Yeah …

Amber: and I’ll say too, like a lot of the automated tools are using Axe’s open source rule set, and we use some Axe, but we’ve also added custom rules that are very specific for WordPress. And so I think it’s also good to be aware of what rule set is the automated tool you’re using, ’cause that’s gonna, that’s going to impact how accurate the results are.

Tanveer: Yep.

Amber: Let’s see Okay, this is an interesting follow-up question. Have you ever found any useful AI testing prompts or workflows? And how much do you use AI in your manual testing?

Tanveer: For now, we are using AI mostly in our documentations in creating… recently I have created a script using AI in which we can give clients the AI estimations of how much time is required to complete this audit.

And other than that, we have also created a AI VPAT creator. We can create our VPAT from that. In the testing, we have did some beta testing in which we tried to test our website using AI, but I’m very… What I can say about that? But I think the, the AI give us some false positive and a lot of false negative things right now at this time of stage, and most of the issues were related to the README docs, which I’m very wonder why is that, because I, and the priorities mentioned was not of that of quality. It prioritize critical, and when we see as per WCAG it’s not that much critical. It’s a mild issue.

I think AI has to work more in the terms of accessibility testing, but it is working, and it is working a lot. And I don’t know. We are skeptical in the future.

We don’t know, maybe it start working very quickly. But until now, I think we are as per my opinion, I think we are using it just for the documentation part. And if you want to understand any WCAG compliance, we will, we are using it for that. So not for the testing purpose for now.

Amber: Yeah. I think I’ve found very much like you, which is that it can’t actually think like a human. And so a lot of that context, bigger picture user experience stuff, it just, it can’t do.

Tanveer: Yep.

Amber: Bonzi had an interesting question, which is, “After completing an accessibility audit and remediation, how can I properly demonstrate and justify to my client that their website is now accessible and conforms to WCAG 2.2 level AA?”

Tanveer: I think you can let them test them by themself. And in my case, what we do, we create a VPAT during testing. When we do the audit of a website for any platform, we create a VPAT, and after giving the VPAT to the client, we also give them the coverage of how much the coverage… how much your website is accessible. After that, if they want us to fix those issues, we fix those issue, and after fixing those issues, we again create another VPAT file, which we also call ACR report, Accessibility Conformance Report. So creating again the accessible reports and then they can compare with the first one and the second one. And they can also justify that what issues were before and what are now, and how compliant we are now.

And other than that, you can ask them to do a thorough testing and ask them to get a person with disability and ask it to use it on the screen reader. I think that’s a better way to make your client know that you are… how compliant you are at this level.

Amber: Let me see. I just wanna see real quick some of these questions. A few of them I think we could answer pretty quickly. So someone did ask, “Would using our accessibility plugin conflict with using Siteimprove?”

And the answer is no. We have clients that scan their website with Siteimprove and then also use Accessibility Checker. So that one is fast.

Maybe you could answer this question from Tom: “Is it better to do automated testing first and then manual testing, or manual and then automated?”

Tanveer: Is it better to do automated testing first and then manual?

I think it’s good to do automated test first because it will give you the issues related to the code, right? Whenever you run an automated test, it will give you a list of your rule-based issues like controls. Then it will give you issues related to your structure or invalid ARIA implementation, for which you have to open the dev tool. I think it would be better to do that.

And after fixing those, if you say that now my code is at some instant it is good for now, then you can move to the manual testing. You can use a screen reader test, and you can see that how my website is behaving now. Am I able to complete my work, and my task after fixing those automated accessibility issues?

If not, then you have to mention those and fix those issues. So a complete strategy, in my opinion, would be better if you do automated than manual than assistive tech testing.

Amber: Let’s see. Yeah I tend to agree. Automated is gonna speed you up and help you find obvious problems. Yeah. Vince, I know you had asked a question about custom HTML emails. I would suggest you come to Maria’s talk that’s all about email testing because we definitely don’t have time, unfortunately, to answer that today.

As well as some questions specific to Elementor. But Sandra, I would say the questions you have about how to fix specific things in Elementor, those are great questions to put in our Facebook group, and there’s a lot of folks in there who use Elementor and would be able to walk you through them, especially if you include a link to the website where that problem exists.

Thank you so much, Tanveer. I appreciate it.

Tanveer, if anyone wants to follow up with you, what is the best place for them to get in touch?

Tanveer: I think the best way is to on my LinkedIn. I’m always available there and I love when people message me there and ask me related my experience and discuss different things. And for now I would also love if you guys who are, who have watched me and listened to me if they give me a shout-out and a message or anything they want us to improve or anything, I would love on my LinkedIn, in my DMs or everything, anywhere they want. LinkedIn is the best place. I think if I am if you guys do have our… Yes. Paola just have shared it. Nice work.

Amber: She’s super fast at finding the links.

Tanveer: Yeah. Yes.

Amber: Thank you so much. Thank you everybody, and we’ll see you back here in a couple weeks for another meetup. Bye.

Tanveer: Okay, bye-bye.

Access “Accessibility Testing: What Automated Tools Will NEVER Catch” in Canva.