Many blogs, magazines and books already explain the technical difficulties of test automation. In this post I want to explain some of the organisational pitfalls I have come across in my past projects.
For over 15 years I have been working in test automation. First working on existing test automation in organisations that already applied it. The last 10 years I have been introducing test automation in several organisations, small and large. So here are some of the pitfalls I came across:
Development output will drop
This is an important point many people in the organisation, including management, do not seem to understand at the beginning.
Why would development output drop?
- Observability of the SUT needs to be improved
Organisations that don’t have automated testing yet, and maybe only limited manual testing, very often (in my experience) don’t have a good testable SUT.
The SUT needs to be testable, not only from a manual point of view, but also from an automated point of view.
This may mean that ID’s need to be added to webpages, API’s or services need to be added.
What changes are needed are very much different for each application and implementation, but very often the SUT is very observable as is. For example, on one project I worked, the web application was generated using an in-house developed framework. Looking at the HTML sources, it was all just nested DIVs, no ID’s or names for elements. XPath could be used, but that would lead to an unmaintainable mess. Before we could automate on the web UI the developers of the framework had to add ID’s to the elements. Which was quickly done and then automatically applied to all pages.
- Controllability of the SUT needs to be improved
Similar to the observability point, very often the SUT cannot be controlled easily.
Again, it depends on the application itself and the frameworks used.
For example, one application had a Java GUI, made with their own GUI framework based on Swing.
None of the tools we looked at, could read or control that GUI. Worse even, there was not a backend service: GUI and logic entities were compiled together.
In the end we made the Java entities available in test services that we created and we could use those test services to control the SUT.
- Developers might need to support testers
Developers might need to explain the SUT inner workings a bit better.
They might even need to help write test code, for example Page Objects.
Teams need to be aware of the mindshift
There is a big different between manual testing at the end of an iteration (often not very Agile) and nightly automated testing.
With manual testing, the whole organisation is often focused during a short period (a week or so) on regression testing. Developers also make sure they have time available.
With automated testing, there has to be constant attention to the test results and getting defects fixed. A person can work around a defect, but test automation cannot. It can be programmed of course, but that will lower your Return On Investment. So the organisation should be set up to fix defects asap to not hinder the automated tests.
Another example are PageObjects. During the initial test automation for existing features, the test team can implement these. However, for maintenance, it would make more sense that developers are responsible for updating them (or at least the low-level parts like locators). Developers will be updating the code for the pages, they can also update the Page Object at one fell swoop.
Test environment needs improvements
First of all, the test environment needs to be stable.
People can more easily work around issues and quirks in a test environment, however test automation cannot as easily.
For example, a stack existed of many small programs, however, most of the time, not all programs were running. Starting them required opening Putty and type in cryptic commands. It can be automated, but maybe it’s better to have good monitoring in place, also in the QA environment.
Next to the observability and controllability of the SUT, you might also need to improve the observability and controllability of the test environment. Your test framework might need to be able to start/stop/restart the SUT because of configuration changes needed to specific tests. Or you might want to test the behavior when (parts of) the SUT is not responding.
Extra capacity is required
When you are introducing test automation for existing projects, there are still releases to be made before all tests are automated.
This means that until your automated tests are running, you will still have to do manual regression testing. Be aware that during that period extra capacity will be required.
That is one of the reasons I’ve been asked several times to help introduce test automation: it gives the organisation the expertise to set up good test automation and provides temporarily extra resources.