Skip to main content
SweepHound

Testing Guides

Accessibility Testing: Automated vs Manual

Automated accessibility tools can help identify many common WCAG 2.2 issues, especially problems with deterministic pass/fail logic. They can flag issues like missing alt text or broken heading order, but they cannot evaluate whether a page actually makes sense to a screen reader user. An effective testing strategy combines automated scans, manual review, and assistive-technology testing.

For axe-core specifically, Deque reports automated tests identified 57.38% of issue instances by volume in its dataset across first-time HTML audits; the same report found automated issues for 16 of 50 WCAG 2.1 AA success criteria. Actual results vary by content, WCAG version, and ruleset.

What Automated Testing Catches

Automated scanners excel at rules with deterministic pass/fail logic. These issues account for the majority of violations found in the wild:

  • Color contrast ratios below WCAG thresholds (4.5:1 for normal text, 3:1 for large text)
  • Missing alt text on images
  • Form inputs without programmatically associated labels
  • Heading hierarchy violations (skipping from h1 to h4)
  • Invalid or misused ARIA roles and attributes
  • Missing document language declaration
  • Duplicate IDs in the DOM
  • Empty links and buttons without accessible names

What Automated Testing Misses

The hardest accessibility problems are the ones machines cannot judge. These require a human evaluator, ideally someone testing with assistive technology:

  • Logical reading order — Does the content sequence make sense when CSS layout is removed?
  • Alt text quality — A scanner can detect a missing alt attribute but cannot tell if alt="image" is meaningful
  • Keyboard usability — Tab order may be technically valid but functionally confusing
  • Screen reader experience — ARIA landmarks may exist but the announced content flow may be incoherent
  • Cognitive load — Error messages, instructions, and navigation can be technically present but confusing in practice
  • Dynamic content — Whether live regions, modal focus traps, and SPA route changes behave correctly

Types of Automated Tools

Browser Extensions

Tools like axe DevTools and WAVE run in the browser on a single page. They are fast and free, but they only test what is currently rendered in the viewport. You have to manually navigate to every page and state.

CI/CD Scanners

Tools like axe-linter, pa11y-ci, and Lighthouse CI integrate into your build pipeline. They catch regressions before deployment but typically test a limited set of routes against a static or server-rendered page. They rarely handle authenticated content or client-side interactions.

Full-Site Scanners

Full-site scanners like SweepHound crawl your entire domain, testing every discoverable page automatically. This catches issues that exist deep in your site — blog posts with missing alt text, legacy landing pages with contrast failures, product pages with unlabeled form controls. SweepHound also supports authenticated scanning to reach pages behind login walls.

SweepHound's Dual-Engine Approach

Most automated scanners run a single engine — typically axe-core. SweepHound runs two engines in parallel: axe-core and IBM Equal Access Checker. Each engine has different automated rules and detection heuristics. Running both catches violations that either engine alone would miss.

In internal testing, the dual-engine approach consistently identifies 15-25% more unique violations compared to axe-core alone. IBM Equal Access is particularly strong on ARIA pattern validation and content structure checks that axe does not cover.

Building a Testing Strategy

The most effective accessibility programs layer automated and manual testing at different cadences:

  • Every commit — Run axe-core or Lighthouse in CI against key templates. Catches regressions before they ship.
  • Weekly or on deploy — Run a full-site automated scan (SweepHound) to catch issues across all pages, including content changes from non-engineering teams.
  • Quarterly — Manual keyboard and screen reader testing of your 5-10 most critical user flows: signup, checkout, search, key content pages.
  • Annually — A comprehensive audit covering all WCAG 2.2 Level AA criteria, ideally by a specialist who tests with assistive technology daily.

Automated testing is not a replacement for manual testing. It is a safety net. It helps keep common programmatically detectable issues from reaching production, freeing your manual testers to focus on the harder problems that require human judgment.

Want to see what automated scanning catches on your site? Run a free SweepHound scan and get a dual-engine automated WCAG scan in under a minute.