TestOps

TestOps (or test operations) refers to the discipline of managing the operational aspects of testing within the software delivery lifecycle.

Software testing is an evolving discipline that includes both functional and non-functional testing. The first mention of this is regarded in March 2019 where, Ditmir Hasani, CEO of QA Tech consultancy is thought to have coined the term. Increasingly, software testing, especially in agile development processes is shifting to become more of a continuous testing process where software developers, quality engineers, manual testers, product owners, and more are involved in the quality process. As more people have become involved in the testing process and testing projects have grown, so too has the need to advance the discipline of software testing management and to manage the software quality processes, programmers, people, systems, and tests.

TestOps helps teams scale their teams, tests, and quality processes in and effectively.

Elements of test operations (TestOps)

TestOps involves several important disciplines that can be broken down into:

Planning — Helps identify how the software is going to be tested. What are the priorities for testing? How will it be tested? Who will do the testing? In addition, the planning phase should identify the environment for the tests. Will they be run in a test environment, or in production? Production data can be valuable to identify real user flows that help prioritize test cases. The outputs include identifying the type of tests to use, the test automation tools, the timing of the testing, the ownership of the testing at different phases, and the design and outputs of the tests.

Management — Test management includes the organization and governance of the team, the tools, the test environment, and the tests themselves. Tests follow a lifecycle pattern and must be managed whether they are in stages such as draft, active, or quarantine. TestOps helps ensure that the testing processes are efficient and scalable.

Control — As teams and tests grow in number and diversity, they naturally increase complexity. Change control processes such as pull requests, approvals on merges, collaboration tools, and ownership labeling can help ensure that changes to tests are properly approved.

Insights — The data and information you derive from your test automation systems should help inform operational as well as process transformation decisions. Operational information about your testing activities includes the test results (pass/fail), release readiness criteria, failure diagnostics, team testing productivity, test stability, and more. Information that can inform process improvements includes team and individual performance data, failure type trend analysis, test coverage mapping, and risk coverage analysis.

TestOps Features

DevOps integration — TestOps exists to ensure that the product development pipeline has all the testing frameworks and tools needed. It is common for QA engineers to rely on the pipelines that IT puts together without much input. TestOps changes this by owning test activities related to DevOps, allowing QA engineers and developers to have full ownership and visibility of the development pipeline, so they can tailor it to meet their needs.

Cloud integration — Integrating tests and test runs with cloud providers

Enhanced test planning — Automation is not effective if the entire codebase has to get tested every time a line of code is changed. TestOps provides a centralized platform that makes it easier for testers and developers to plan what tests to write, as well as identify what tests to run and when.

Test lifecycle management — Automated tests follow a lifecycle including creation, evaluation, active, and quarantine or removal. The status of the test should impact how it is treated in build automation systems like CI/CD.

Test version control — Processes that help ensure that changes to tests are properly reviewed and approved through capabilities like pull requests in code.

Real-time dashboards — Real-time results and status help the test teams understand the state of software releases and the work that needs to be done to create, approve, or run more tests.

TestOps vs. DevOps

DevOps is a broader, more inclusive concept that includes software feature planning, code development, testing, deployment, configuration, monitoring, and feedback. It was an attempt to integrate some of the disconnected toolchains. Testing is included in the broader DevOps methodology.

TestOps is not simply providing additional emphasis on testing. It is focused on operational aspects of testing that are necessary to ensure that testing, whether performed in development, production, or in its own testing phase, is well planned, managed, controlled, and provides insights to enable continuous improvement.

The siloed working mindset is aimed to be removed from activities such as continuous delivery, software testing (manual and automated testing), environment setups, infrastructure and log management, and built-in security enforcement.