The Current State & Future Trends of AI in Software Testing

Artificial intelligence (AI) and machine learning (ML) are transforming multiple sectors of the economy, and impacting several aspects of our daily lives.

Workplaces such as those in finance, healthcare, retail, education, and technology are leveraging AI to automate tasks, reduce costs, and make data-driven decisions. In our homes, AI is powering television and movie recommendations, personal digital assistants, security cameras, and home automation.

This blog will focus on how the coming together of artificial intelligence (AI) and software testing have made a huge impact on test automation. Let us discuss this developing trend in more depth. 

AI in Software Testing

AI software testing uses automated software testing tools to leverage AI (machine learning) to produce better results. By incorporating AI into automated testing tools, teams will be able to overcome many of the traditional challenges of automated software testing.

As AI continues to permeate our world, it is becoming more and more critical to validate that these types of systems are functional, safe, secure, performant, available, and resilient. In other words, AI needs testing.

Interested in learning how to incorporate AI automation into your software testing process? Try out Perfecto FREE for 14 days.

Start Trial

Unfortunately, we haven’t seen many advances in the area of testing AI-based systems.

Read Perfecto's Annual State of Test Automation Report

On the bright side, researchers and practitioners are recognizing the potential for AI and ML to bridge the gap between human and machine-driven testing capabilities. As a result, several organizations are developing AI-powered automated testing tools.

Watch: Advancing the State of The Art in AI and Testing

Since 2014, there has been a spike in the number of vendors offering AI-driven test automation services. The majority of these tool vendors are start-up companies targeting system-level testing of mobile applications, and the subject is generating some much-needed buzz in the industry.

Hence, it seems that just like AI needs testing, testing needs AI.

Although some aspects of AI for Software Testing (AIST) are still not receiving enough attention, in the last decade we have seen it emerge as a discipline centered at the intersection of three areas:

• AI-driven automation testing — developing AI tools to test software.
• Testing AI systems — devising methods to test AI systems.
• Self-testing systems — designing software that is capable of self-testing and self-healing.

The intersection of AI and software testing

AI in Software Testing: Current State

The current state of the practice uses autonomous and intelligent agents, referred to as “test bots,” to automate activities such as application discovery, modeling, test generation, and failure detection. A combination of machine learning techniques are used to implement test bots. These include but are not limited to decision tree learning, neural networks, and reinforcement learning.

Back to top

Try Testing With Perfecto

Machine learning allows the test bots to be robust and act under conditions of uncertainty unlike traditional test automation tools and frameworks. Examples of AI-driven automation testing approaches that have formed over the last decade include:

• Differential testing — comparing application versions overbuilds, classifying the differences, and learning from feedback on the classification.
• Visual testing — leveraging image-based learning and screen comparisons to test the look and feel of an application.

Related Reading: Automated Visual Testing for Mobile and Web Applications

• Declarative testing — specifying the intent of a test in a natural or domain-specific language, and having the system figure out how to carry out the test.
• Self-healing automation — auto-correcting element selection in tests when the UI changes.

AI in Software Testing & the Open Source Community

Element location using AI has been incorporated into the open-source mobile testing framework Appium. The project is a collaboration between Jason Arbon, CEO of Test.ai, and Appium’s creator Jonathan Lipps. It uses AI to identify web elements in mobile applications.

Leveraging AI for web element location makes the tests quicker to create and less brittle. Furthermore, the testing community can improve the AI by adding new training data, alternative training methods, more rigorous relevance testing, or new labels.

Related Reading: How to Use Page Object Model in Selenium

The training data is open and available on Kaggle, thereby allowing individuals or organizations to fork the data, clean it up, add it to their proprietary test frameworks, or compete to improve the classifiers.

Related Reading: Test Automation Strategy For Beginners

Tariq King and his team over at Ultimate Software developed and released an open-source framework called AGENT — AI Generation and Exploration iN Test.

AGENT is an ML-based, multi-agent prototype for web application testing. The prototype consists of a collection of intelligent agents that work in conjunction with abstract test generation and web UI classification subsystems. The agents incorporate long short-term memory units trained by human testers.

For any given subsequence of steps through the application, the agents predict sequences of the next possible set of actions and assertions that a human tester might make and automatically execute them on the system under test.

AI in Software Testing: The Future of Test Automation

One of the trends that started this past decade, which is expected to continue, is the use of AI to enhance existing tools and frameworks that target specific testing problems.

Examples at this time include functional testing of web and mobile applications, visual testing of user interfaces, and UI element location and auto-correcting element selectors. Beyond this we should expect to see AI starting to replace entire technology stacks for automated testing.

Related Reading: ML & AI in DevOps: Experts Predict 7 Big Changes

At all testing levels, AI automation testing will take over tasks that require decisions that a human could make in less than a second. Initially, higher-order testing tasks may still require human input or intervention. These are the tasks that require a bit more thought such as test generation, usability testing, security testing, and edge cases.

However, over time as technology progresses, and as the machines are trained on the actions of these higher-order tasks, AI is likely to take over those activities as well and tackle problems that require deeper context. The image below depicts this progression of the AI testing singularity as we move into the next decade.

The AI testing singularity: a projection of the next decade of software testing

How Can We Prepare for Testing AI-Based Apps?

So what can testers do to prepare for a future in which AI tests software? The initial step is to determine whether you are interested in designing and developing AI-driven systems, or being an end user of these tools.

Related Reading: 5 Types of ML & AI Testing Tools

An engineer who is building these types of systems needs to have a good understanding of AI and machine learning. Both MIT and Stanford University have freely available introductory courses on machine learning.

Tariq King and Jason Arbon have open-sourced the materials for an introductory course on AI and ML skills for the testing world, including the slides and tools used during the exercises. It is unlikely that end users of AI-driven testing tools will need to have deep knowledge of machine learning.

Related Reading: Why Test Failure Analysis Is Key to Faster Delivery

Vendors are likely to provide easy-to-use interfaces and APIs for using AI-driven testing features and customizing pre-trained AI models.

This trend is already being followed with the advent of AItools and platforms like Google’s AutoML Vision, H2O.ai, and DataRobot.

Bottom Line

AI is already altering the landscape of testing. And while we don’t know exactly what the future of software testing holds, we can prepare for it by stabilizing and scaling test automation to mature our processes.

Perfecto can help you take your test automation to the next phase, no matter where you are at. With Perfecto, you'll experience:

• AI-powered reporting capabilities.
• Unified testing for web and mobile apps.
• Advanced automation.
• Full toolchain integration.
• All results in one place.
• Shift left testing.

Whether you’re just beginning on automation or you’ve reached continuous testing, Perfecto has the end-to-end testing support to help you accelerate. Try it for free today.

Try Testing With Perfecto

Related Content

• How ML & AI Will Shape the Future of Analytics for App Testing
• What to Look for in Automation Testing Tools
• The Keys to a Strong CI/CD Pipeline