Five Warning Signs Your Test Automation Is Not Ready for DORA

Test automation can accelerate regression testing and improve software quality, but DORA raises the expectations placed on testing in financial services. For banks and other financial institutions, simply having thousands of automated test cases is not enough. Testing needs to support a broader, risk-based digital operational resilience programme capable of identifying weaknesses, validating critical systems, supporting remediation, and producing reliable evidence. Here are five warning signs that your current test automation approach may not be ready for that role.

test-automation-dora-readiness-proofit

The Digital Operational Resilience Act (DORA) has changed the context in which European financial institutions think about software testing.

DORA requires covered financial entities, other than microenterprises under the relevant provisions, to establish, maintain, and review a comprehensive digital operational resilience testing programme as part of their ICT risk-management framework. The programme must follow a risk-based approach, and appropriate testing must be conducted at least annually on ICT systems and applications supporting critical or important functions.

The regulation also makes clear that digital operational resilience testing is broader than conventional functional regression testing. Article 25 identifies appropriate testing methods including vulnerability assessments, source-code reviews where feasible, scenario-based testing, compatibility testing, performance testing, end-to-end testing, and penetration testing.

This creates an important distinction. A financial institution can have a technically successful test automation programme and still have significant gaps in its wider operational resilience testing approach.

The following five warning signs can help organizations identify where their automation strategy may need to evolve.

Warning Sign 1: You Measure Test Volume Instead of Business Risk

A common test automation dashboard might show: 10,000 automated tests, 95% regression automation, thousands of executions every month. These numbers may demonstrate activity, but they say surprisingly little about operational resilience.

DORA requires a risk-based approach that takes into account factors including the evolving ICT risk landscape, specific risks to the financial entity, and the criticality of information assets and services. This means organizations should be able to connect testing priorities with business and technology risks. Consider two applications.

The first is an internal administrative system with limited operational impact. The second supports payment processing for millions of transactions. From an operational resilience perspective, 95% automated coverage of the first application does not compensate for significant testing gaps in the second.

A more mature automation strategy maps tests to critical business processes, systems, risks, and requirements.

Instead of asking:

“What percentage of our test cases are automated?”

Ask:

“Which critical or important functions can we demonstrate are adequately tested?”

For a bank, that could mean identifying automation coverage across payment processing, customer authentication, transaction management, digital banking, fraud controls, core banking integrations, and other critical services. The objective is not maximum automation. It is automation aligned with risk.

Warning Sign 2: You Cannot Trace Test Results Back to Requirements and Risks

Automated tests generate enormous amounts of information. But information is not automatically evidence. Imagine an organization executes 30,000 automated tests before a major release. Almost all pass.

Management asks a straightforward question:

“Which critical business requirements did those tests actually validate?”

If answering requires several engineers, multiple spreadsheets, and days of investigation, there is a traceability problem.

DORA’s testing framework goes beyond executing tests. Article 24 requires financial entities to establish procedures and policies to prioritize, classify, and remediate issues revealed through testing, together with internal validation methodologies to confirm that identified weaknesses, deficiencies, or gaps have been fully addressed. Effective traceability helps make that process manageable.

A mature Quality Engineering environment should make it possible to establish connections between relevant requirements or risks, test cases, executions, defects, remediation activities, and results.

Traceability improves more than audit preparation. It allows teams to answer practical questions quickly:

  • Which tests cover this critical process?
  • What happened during the latest execution?
  • Which defect resulted from the failure?
  • Was the problem corrected?
  • Was the correction retested?
  • What evidence demonstrates the current result?

Automation that produces thousands of results without providing this context may create speed without sufficient transparency. DORA readiness therefore requires thinking about evidence automation, not just test execution automation.

Warning Sign 3: Your Automation Stops at Functional Testing

Functional regression automation is valuable, but it represents only one dimension of operational resilience. A payment application might successfully calculate transactions, apply business rules, and communicate with downstream services during functional testing.

But what happens when transaction volume increases dramatically? What happens when an external dependency becomes slow? Can the system maintain acceptable response times during peak demand? Does a service recover correctly after disruption?

DORA Article 25 explicitly identifies multiple forms of testing that may form part of an appropriate digital operational resilience testing programme, including scenario-based, compatibility, performance, end-to-end, vulnerability, and penetration testing. If an organization’s automation strategy focuses almost entirely on UI or API functional regression, that should therefore be treated as a warning sign.

Performance is part of resilience. Performance testing is particularly important for critical financial services. A system does not need to crash completely to create a serious operational problem. Increasing response times, exhausted resources, growing message queues, database contention, or overloaded dependencies can progressively degrade a service while basic functional tests continue to pass.

Performance engineering can help organizations understand:

  • transaction throughput;
  • response times;
  • concurrency limits;
  • resource utilization;
  • scalability;
  • bottlenecks;
  • stability under sustained load.

Functional correctness and operational resilience are related, but they are not the same thing. A mature testing strategy needs to examine both.

Warning Sign 4: Critical End-to-End Processes Are Not Tested as Systems

Modern financial services depend on interconnected technology. A seemingly simple customer action – such as making a payment – may involve a mobile application, authentication service, API gateway, payment engine, fraud platform, databases, message brokers, external providers, and reporting systems. Each component may work perfectly when tested independently. The overall business process can still fail.

This is why end-to-end testing remains important even in architectures where teams already have extensive unit, component, and API automation. DORA specifically lists end-to-end testing among the testing approaches that may form part of the digital operational resilience testing programme. The implication for automation strategies is significant.

Teams should understand not only whether individual applications are covered, but whether critical business flows are validated across the systems and dependencies required to deliver them.

Look beyond application boundaries. A mature strategy can combine different testing layers. Unit and component tests provide fast feedback. API tests validate interfaces and service behavior. Integration tests examine interactions. End-to-end tests validate selected critical business flows. Performance tests determine whether those flows remain reliable under realistic workloads.

The goal should not be to move everything into expensive end-to-end automation. Instead, organizations should identify the business flows where end-to-end assurance provides meaningful evidence that component-level testing cannot provide alone.

Warning Sign 5: Failed Tests Are Fixed, but the Remediation Process Is Not Traceable

Finding defects is only part of testing. What happens next matters just as much.Suppose an automated test detects a serious  weakness in a critical application. A ticket is created. A developer implements a change. The test eventually passes. Can the organization demonstrate the complete chain from discovery through classification, remediation, validation, and closure?

Article 24 of DORA requires procedures and policies for prioritizing, classifying, and remedying issues revealed by testing. It also calls for internal validation methodologies to establish that identified weaknesses, deficiencies, or gaps have been fully addressed. This makes remediation part of the testing lifecycle rather than an administrative activity that happens afterward.

Automation should support the evidence chain. A mature environment should make it possible to reconstruct what happened without depending entirely on individual knowledge.

Useful information can include:

  • the test and execution that identified the problem;
  • the affected system or critical process;
  • severity or risk classification;
  • the related defect or remediation activity;
  • changes made to resolve it;
  • the verification or retest;
  • the final result and evidence.

This becomes particularly important when testing complex systems repeatedly across releases. If this information is fragmented across disconnected tools, spreadsheets, email conversations, and individual teams, demonstrating consistent control becomes harder.

A Sixth Question: Is Your Testing Sufficiently Independent?

Although the title promises five warning signs, there is another DORA requirement that deserves specific attention. Article 24 states that tests must be undertaken by independent parties, internal or external. Where internal testers are used, organizations must allocate sufficient resources and avoid conflicts of interest throughout test design and execution.

Independence does not necessarily mean outsourcing every test. It means organizations need an appropriate testing model that avoids relying exclusively on the people who designed a system to determine whether it is resilient.

This principle becomes increasingly relevant as AI-assisted software development and AI-generated testing grow. If developers and AI assistants create both the implementation and the tests from the same assumptions, independent verification becomes even more valuable for critical systems.

What DORA-Ready Test Automation Should Support

DORA does not prescribe a specific automation framework or require organizations to purchase a particular category of testing platform. Instead, the regulation establishes outcomes and requirements around digital operational resilience.

A strong automation capability can support those outcomes when it enables organizations to test critical systems consistently, prioritize testing according to risk, execute appropriate forms of testing, capture reliable results, connect failures with remediation, and reproduce evidence when required.

For some financial entities, advanced threat-led penetration testing (TLPT) is another part of this landscape. DORA establishes TLPT requirements for identified entities, and the associated Regulatory Technical Standards further specify areas including scope, methodology, testing phases, results, closure, and remediation. Test automation therefore needs to be understood as one component of a wider operational resilience capability – not as a substitute for it.

DORA Requires More Than Automated Regression

DORA does not make test automation obsolete. It makes effective automation more valuable. But it also exposes the limitations of automation strategies built primarily around reducing manual regression effort. For financial institutions, testing now needs to support a wider objective: demonstrating that critical digital services can remain resilient in the face of failures, changes, vulnerabilities, performance pressure, and other ICT risks.

That requires more than large test suites. It requires risk-based coverage, traceability, end-to-end verification, performance testing, disciplined remediation, reliable evidence, and appropriate independence.

The most important question is therefore no longer:

“How much of our testing is automated?”

It is:

“Can our testing provide credible evidence that our critical systems are resilient?”

Strengthen Your Test Automation for DORA and Critical Systems

Preparing testing capabilities for DORA requires more than increasing automation coverage. Organizations need an engineering approach capable of connecting automated testing, performance testing, risk, traceability, and operational resilience.

ProofIT has proven references in automated testing and performance testing of complex, business-critical systems across the banking, telecommunications, and aerospace industries.

Our specialists help organizations develop scalable test automation, validate complex end-to-end processes, assess system performance under realistic conditions, and strengthen Quality Engineering for demanding and regulated environments.

Contact ProofIT at business@proofit.tech or +44 73 6049 4722 to discuss how your current testing strategy can evolve to better support DORA and digital operational resilience.

Sources: 1 2 3 4 5