Considerations for Modern Agile Regression Testing

Nowadays, when there is a constant decrease in time windows between software iterations and releases, testing teams are challenged more than ever to build a solid continuous regression testing strategy that can keep up with requirements changes, feedback from production environments, and more. Some teams are leveraging test data analysis, while others are using machine learning and other advanced technologies to optimize their DevOps pipeline. Yet there is still a huge gap in understanding how to continuously deliver value through regression testing cycles.

In this post, I’d like to focus on the topic of strategizing regression testing in an Agile/DevOps environment.

What is Agile Regression Testing?

Agile regression testing ensures high-value continuous regression testing, which allows teams to focus on new functionalities. To achieve this, teams must build regression testing suites with a focus on robust test scenarios, high coverage, and near-zero test maintenance.

From all available definitions of this term, I tend to like this one the most:

“Regression Testing is defined as a type of software testing to confirm that a recent program or code change has not adversely affected existing features”. From this definition, it’s very clear that such testing types should focus on a full or partial set of test scenarios that are executed via trigger or on-demand at pre-defined schedules.

The value of regression tests is very high when they are developed properly according to best practices and cover sufficient functional areas. Regression testing suites can achieve various testing goals such as finding regression bugs, side effects of code changes, and other unexpected issues.

Typically, the common triggers of executing regression tests are around:

1. Code modified due to a new feature that was added or changes in requirements and business flows.
2. Major defect fixing (functional or non-functional) that requires quality assurance that no side effects caused
3. Continuous regression testing (daily/weekly) for risk mitigation

Strategizing Continuous Regression Testing in Agile

Building test automation is a challenging but key enabler for continuous testing and regression testing.

As mentioned above, to make sure the regression suite is of high value continuously, it must be built wisely with a focus on robust test scenarios, high coverage, and near-zero test maintenance. If one of these considerations isn’t kept, it can harm the entire testing pipeline and delay the release trains.

When thinking about a strategy for regression testing in an agile environment, one needs to understand that such an environment constantly changes. New platforms, features, defect fixes and more are added, which means that software regression tests should adjust to this dynamic environment to continue being effective.

Test engineering management needs to focus on the continuous maintenance of the regression suite and determine the following:

1. Which test cases are certified and need to be included in the regression suite, and which should be excluded?
2. What is the execution schedule for the regression and subset-regression suite? (Daily, weekly, per code commit, other)?
3. Which regression tests are executed from the main CI engine, and which (if any) are executed from a different scheduler outside of the CI? End-to-end regression testing vs. sprint level regression tests.
4. What events trigger regression suite maintenance and improvements?
5. What is the time window to complete the regression tests? Are there enough platforms/resources to accommodate these time constraints?
6. Continuously analyze the value of the tests, their flakiness, and more.

Agile Regression Testing Recommendations & Fundamentals

After clarifying some basic strategic considerations and insights around regression testing, here are some best practices and recommendation to take away:

1. Differentiate sprint-level selective regression tests from full regression testing cycles. These vary in scope, platform coverage, time windows, and objectives.
2. Continuously maintain your regression suites to include the high-value functional and non-functional scenarios.
3. Keep in mind test aging, and ensure you leave in your suite the high-priority, high-value test cases.
4. Regression suites are 100% automated. What’s hard to automate and scale should not be part of the suite, or else you’re risking the entire cycle.
5. Agile forces continuous changes of features, requirements – this also implies constant changes to test code – have a process in place to accommodate such modifications.
6. You cannot measure and improve what’s not visible. Ensure you have full visibility into the regression suite reports with detailed views to assess the ROI of the executions over time, per platforms, within CI, and more.

Consider grading your test scenarios within the regression suites so you can properly prioritize what to execute, when, in which cadence, etc.

Get the Most Out of Your Continuous Regression Testing

Continuous testing is enabled by high degrees of test automation. Regression testing in its various types is also fully dependent on robust and high-trusted test automation. To ensure you get value back from the investment in regression testing, you should follow a solid strategy as identified above, and fine-tune it as your product evolves. Make sure you properly communicate between the involved parties around regression testing activities and monitor the outcomes of such periodic executions.

Explore Continuous Regression Testing With Perfecto 

Explore Perfecto Scriptless, the codeless UI test automation tool for web apps. 

Explore Perfecto Scriptless

Related Resources

• 7 Principles of Good Agile at Scale