Selenium Testing – here’s what you need to know and consider before incorporating an automation framework to your manual testing operations

Click the link to go directly to the following section:

Introduction – Adding Test Automation to Your QA Operations

So you’ve decided you want to add automation to your manual testing operation. Great, that’s definitely the right move. Especially if your company works in an agile process, there is little else you can do to catch up with development.

Truth be told, testing got a bit damaged by the shift from waterfall to agile. It’s not that coders got all of a sudden so much faster or better; it’s simply that instead of having one or two releases a year, you have anything between 24 to 52 releases. Now go run full regression tests every other week – manually. It simply can’t be done, and so we found ourselves in the situation where testing teams are bound to impossible deadlines.

Thus Selenium test automation was born, out of a need to adhere to shorter development cycles and ever-increasing product versions. That’s one side of the story. The other side is the massive software operations – too big to test manually in any reasonable timeframe.

Before we get started, let’s make a few things clear:

  • There’s no such thing as 100% automation.
  • Test automation is mostly meant for regression and continuous testing.
  • Test automation isn’t about replacing human testers with robots.

This short guide will provide you all the information you need to know in order to reach a smarter decision regarding how to add Selenium automation to your testing operation. And notice we said “how to add”, not “if you add”. Selenium Test automation is here. You should get on board.

We have decided to focus on Selenium test automation, as it is the most common open source code with a steady, growing usage, and adoption worldwide.

These are your two options:

First Option – Build Your Own Selenium Automation Framework

Steps to build your own Selenium automation framework with all it entails – hiring test developers, setting up a test lab and the most difficult part – maintenance. We will cover:

  • Building Automation Framework
    • In-house or outsource
    • Time & cost
  • Framework Integration Into Development
    • Creation & management of new work processes
    • Reporting
    • Versions
    • Test Lab
    • Hiring developers/ teaching code to testers
    • Maintenance of the framework
    • Maintenance of the test flows

Second Option – Codeless Test Automation

A codeless test automation platform that includes a built-in framework, easy creation, Selenium based automated test scenarios, built-in test lab, more resilient tests, and light maintenance.

Now let’s dive into these two options:

First Option – Build Your Own Selenium Automation Framework

Building a test automation framework is a development project all on its own, requiring no less attention and effort. It’s akin to having another development team on your hands, with all that it entails.

A test automation framework, for those who are completely new to the automation game, is the “environment” where the test flows are written in code. Yes, the “automated” part relates to the running of tests, not the creation stage. With automation, every test flow needs to be programmed. Once you have all the needed test flows written, you can run them in bulk through automation.

In-House or Outsource

For building the framework, you have two options – outsource or develop in-house. The two options aren’t that different, but outsourcing saves you the trouble of scouting and hiring the right automation framework specialist.

[Side note: The unique challenge in hiring a framework specialist lies in the fact that this is a contractor position. The specialist comes in, creates the framework, and is off to the next project. Naturally, contractors are less invested in the company and it’s difficult to provide them a sense of commitment.]

With both options though, the framework needs to be built from the ground up and fitted to your product’s specific needs.

Time & Cost

This can be a long and costly process. It’s hard to give exact numbers (it’s like asking “how much does software cost?”), but let’s just say it usually takes between three to six months, depending on the complexity and size of your operation. Now multiply the number of months by the cost of a 2-3 engineers/framework specialists per month and you’ve got a ballpark figure, with outsourcing usually costing more.

Testing Framework Integration into Development

Since a test automation framework needs to be integrated into the overall workflow of development and doesn’t work as an independent unit, you’ll need to create new processes that incorporate the work in the framework and define specific ‘if’s and ‘then’s.

Creation & Management of New Work Processes

By setting up an internal test automation framework you inherently change work processes within your company, or at least in its development sector. This needs to be addressed on all levels and new management protocols need to be written, taught, and enforced throughout your teams.

Since automation fundamentally changes the way testing is planned and executed it has wide-reaching implications on your developed and release cycles.

Reporting

Similar to the point above, you’ll need to establish new bug reporting processes throughout your organization. If until now your reporting was done through an internal system (or simply by excel) you’ll need to integrate your reporting operation into the Selenium automation framework. It is considered a bit difficult because Selenium doesn’t have a built-in reporting accommodation.

Versions

If you work in agile, you probably have multiple versions of your product in various stages of development. To support the agile methodology, you must be able to test each and every one of them independently.

Enabling your automation framework to support multiple versions and being able to keep the related tests separate, is another heavy task. Version management also needs to work seamlessly with reporting.

Test Lab

Connecting to an external test lab or creating one within the framework is another important decision to make.

If you go with an external test lab for executing your tests you will have to pay a monthly fee to the test lab company. These fees run per concurrent sessions and minutes, which add up pretty quickly.

If you decide to build your own execution test lab, first and foremost you must add complexity (complexity = time X money) to the initial setup. In addition, you’ll need servers to run the tests. You can install actual servers, or use a cloud service, like Amazon’s AWS or Microsoft’s Azure.

There’s another important factor here – the multitude of operating systems and browser versions you need to test. If you use an external test lab, you’re covered. They make sure all existing versions are available for you. If you opt for your own test lab, you’ll need to have multiple servers to support the OSs and browsers of your choice.

For example, you can only install one version of IE on a Windows OS. So if you need to test IE6 to Edge, you’ll need 7 different Windows running. These things add up.

Teaching Code to Testers or Hiring Developers

Once you have the framework all set up, the real work of testing begins. Each test flow needs to be written in code. You have two options here:

Teaching Code to Testers

This is, in a way, the most challenging part of it all. Here money isn’t the “magic solution”. You’ll need to deal with people, their needs, wants, insecurities, and pride. Part of the team might feel frustrated with the new assignments, while others empowered. Some of the testers will demonstrate a real knack for coding, while others might struggle. In any case, there are HR elements to this move.

You’re transforming your manual testers into junior programmers, with all that it entails. There’s a learning curve involved here. You’ll most likely need to bring in at least one experienced programmer to lead the way and take care of the more complex test flows and to check and fix the work of the junior team.

Another obstacle is that with Selenium there is no true support from the tool itself. The good news is there happens to be a large community that uses Selenium, so you will be able to find answers, just not from the most reliable source.

Hiring Developers

The different path to take is to fire the testing team and hire “real” programmers. Unfortunately, there is a lot of turnover in this division. A programmer that’s performing test automation is probably looking to scale up and advance to software development. That is to say, test automation is usually a stepping stone for programmers.

This is another HR problem. You are obviously looking for a long-term commitment from your team – you invest a great deal in training them – so knowing that your automation programmer is constantly on the lookout for her next opportunity can be daunting.

Personnel Pros & Cons of Transition to Automation

Testers and Developers

There’s no right or wrong answer here. It’s a set of considerations each organization needs to look at to reach their own decision.

Selenium test automation maintenance

When you say “Selenium test automation”, you assume “maintenance”. Maintenance is a major part of Selenium test automation. This is in many ways the heart of the matter, and an issue many companies fail to recognize in advance. Many automation projects fail because of the heavy burden of maintenance – companies simply aren’t able to handle the workload.

Let’s explain this burden as it comes into play in the maintenance of the framework itself, and the maintenance of the tests.

Framework Maintenance

The framework is the core of the automation operation and it supports all the automation functionalities. Whenever a new functionality is needed by the testers, the framework needs to be adjusted.

For example, if a new browser version is released, the new code needs to be written to allow for the framework to supports test runs on the new version. Same goes for a new Android version, a new Android device, and so on.

Another example of framework maintenance is if the testing team needs to create a new flow that sends a query to a database. If this is a new ‘kind’ of a test (previous tests didn’t send a query to a database) the framework needs to support it, which means that new code needs to be written.

Basically, any new functionality that is being tested, or any new functionality on top of the tests – like a scheduler – needs to be supported by the framework. So framework maintenance is an ongoing task.

Test Maintenance

The idea of test automation is that you create (write code) test flows in advance and then run them – automatically – every time a new feature is added, a new version is released, etc. Test automation is ideal for regression and continuous testing.

When you add a new feature or change an existing one, you’ll need to go back and modify the relevant test flows with respect to the change. Manual testing is simple enough – the tester is made aware of the change and acts accordingly in executing the test. But with automation, you need to re-code all the test flows that touch upon the specific feature.

Writing code for test flows takes time, same as programming, and this is the point that must be understood regarding test automation – the actual run is swift and simple, but the creation of the test flows is not. When discussing “maintenance” in Selenium test automation the critical point is going back to existing test flows and updating them, over and over again.

Second Option – Codeless Test Automation. All in One Test Automation Platform.

Built-in SaaS Framework

With a built-in SaaS operation, there is no need to hire an automation specialist to set up the framework or outsource the job. Nor is there a need to hire a developer for the maintenance of the automation framework. Once you have a built-in framework you can immediately implement and manage your tests.

Maintenance Using Artificial Intelligence

Using a machine learning algorithm, the platform accounts for the vast majority of changes that happen in your application. This cuts down the time and resources you need to effectively maintain your Selenium test flows.

Detailed Reporting in Team/Management Level

Generates detailed reporting on every run, which includes screenshots and videos indicating what needs to be fixed.

Messaging to Notify Developing Team and Testing Team of Issues

Run on a Schedule

Schedule runs for automated test executions.

Built-in Test Lab

On the platform, there is no need to build an internal test lab from scratch or pay for external test labs.

Integrations with CI/CD Tools and Test Management Tools

This enables you to integrate with DevOps and the agile process.

A Selenium Code is Being Created Under the Hood

On-the-Fly test flow creation that does not require coding. There is no need to hire QA developers, which cost more than QA testers, they are difficult to hire and even more difficult to maintain as most wish to progress to product development (see these considerations table). There is a very short learning curve to master the platform. Now you are able to keep your existing team which is familiar with the business and product.

Smart Binding

Bind elements during runtime. Smart Binding technology automatically assigns a binding score to every element, minimizing the number of broken tests. Even if your web app changes, automated tests with this binding are resilient enough to resist breakage.

Light Maintenance

Visual representation of test flows and drag-and-drop canvas for light maintenance.

Use Open-Source Selenium Code

An ability to use any open-source Selenium code and benefit from the Selenium community without the cost of hiring QA developers and the difficult maintenance which causes many Selenium automation projects to fail.

You will be able to:

  • Immediately implement automation to your existing QA operation.
  • Keep your existing QA team, who are specialist in the business and the product.
  • Save the time and effort of building and maintaining an automation framework.
  • Empower your testers who will become automation experts, creating a clear career path and progress. Reduce employees turnover rates.
  • Improve your QA operations KPIs and productivity.
  • Allow your testers to focus on what really matters – what should be tested and why- leading to a better product.