Thoughts on efficient enterprise testing (1/6)

#testing thursday, september 12, 2019

Testing in the enterprise is still a topic that is not as extensively used as it should be. Writing and especially maintaining tests takes times and effort, however, cutting short on software tests can’t be the solution. Which scopes, approaches, and test technology should be pursue in order to make testing more efficient?

I’ve put together a series with my experiences and opinions on enterprise testing, based on many real-world projects. Especially for applications that are considerably more complex then “hello world”, it becomes paramount which approaches to follow. I’ll mostly focus on testing the functional behavior of our applications, that is how well they fulfill our business logic. In the following I’ll explain best practices on how to make testing more efficient, for different scopes and with different approaches:

Ideas & constraints
Unit tests
Use case tests
Code-level integration tests
System tests
Development workflows & pipelines
Test code quality & maintainable tests
Test frameworks & technology

Introduction

Regardless of the different types and scopes of tests, the point of having a test suite is to verify that our applications will work as expected in production. This should be the main motivation to verify whether the system, viewed from a user’s perspective, does its job.

Since human attention spans and context switches are a thing, we need to make sure that our tests run and verify fast, and with predictable results. While writing code, a quick verification, quick as in less or equal to one second, is crucial to ensure a highly productive workflow and that we don’t get distracted.

On a different note, we need to ensure that tests stay maintainable. Software changes very often and with a sufficient functional test coverage, every functional change in the production code will require change in the test scope. Ideally, the test code only changes when the functionality i.e. the business logic changes, and not for code clean-ups and refactorings. In general, the test scenarios need to make non-functional, structural changes possible.

When we look into different test scopes, which we will at more detail, the question arises which scopes to spend more time and effort on. For microservice applications, or any system where we have a significant amount of distribution and integration, integrative tests that verify the system boundaries become more important. Therefore, we need an effective way to verify the overall application during our local development, while keeping the application environment and setup as similar to production as possible.

Principles & constraints

Regardless of the chosen solutions, let’s define the following principles and constraints for our test suite:

Tests need to execute and verify fast, and provide fast feedback. For unit tests without any further integration, we should be able to run hundreds of tests within a single second. For integrative tests, the execution time depends on the scenario, which ideally doesn’t exceed one second.
During development, the tests must provide fast feedback, also on an integrative level. This requires that the test context starts up quickly, or keeps running while we’re writing code. Thus, it should be possible to build up an effective development cycle with redeploy and test turnaround times with less than five seconds.
Tests need to make it able to refactor the production code without significant change in the test scope. Code changes that don’t change the functional behavior of the application should only result in minimal test code changes.
Code changes that do change the functional behavior should equally result in limited test code changes. As an example: “How much effort is it to swap the HTTP boundaries to gRPC, to swap JSON to something else, to even swap the enterprise framework, etc.?”.
The test technology and approach needs to be compatible with crafting proper abstraction, delegation, and code quality, tailored to our business requirements. We need to be able to craft expressive APIs, extend potential DSLs, and to craft the correct abstractions.
The test technology needs to support a “development mode”, that is running the application in a way that enables instant changes and redeploys in an integrated environment, for example “dev” and debug modes of servers, Quarkus' dev mode, Telepresence, watch-and-deploy approaches, and others.
The testing approach needs to be compatible with setting up development and test life cycle individually. That is, the developers must be able to setup and configure their local environment outside of the test life cycle, for example using shell scripts, and then quickly run the test scenarios against an already set-up environment. For reasons of flexibility and reusability, the individual test cases should not manage the life cycle of the test setup.
We need to be able to reuse test scenarios for multiple scopes, for example defining the business scenario once and reusing the setup for system tests, load test, running either locally or against an externally deployed environment. It should be simple to copy the scenarios, that should only consist of a few lines of code, for a different purpose by using a different implementation.

In the next part of the series we will have a look at code-level unit tests and component or use case tests and how they match these principles and constraints.

Found the post useful? Subscribe to my newsletter for more free content, tips and tricks on IT & Java: