Testing as process support
Testing is planned, prepared, performed and evaluated in separate phases within the traditional software development process. However, when it comes to continuous testing the aim is to uncover any risks at the earliest possible point.
Key features of continuous testing include:
- High degree of automation
- Clear assignment of (failed) tests to business risks
Ideally, continuous testing includes both functional and non-functional tests (security tests, performance tests, etc.).
In order to implement continuous testing, we look at the different stages of the software development process:
Testing & Continuous Integration
The term Continuous Integration (CI) was coined by Grady Booch in 1991. CI is the continuous integration of all development activities into the main branch. This measure is used to perform the time-consuming integration of different components in smaller increments. Breaking changes can be recognized early and tracked back to specific changes to be handled accordingly.
Quality assurance measures in continuous integration:
- Static code analysis
- Unit Tests
which are performed by the CI server as part of the build.
Unit tests are often used for code coverage specifications. What may seem like a good idea at first glance can quickly turn into the opposite. Increasing coverage is a considerable task, which does not present a major challenge to most modern development environments (IDEs). However, forcing them to do so quickly can lead to a shift from quality to quantity. The goal is to heighten the awareness of the development team by increasing the quality of unit tests through pair programming sessions and workshops.
Signs of potential for improvement in the unit test area include:
- Unit test results are either not available or only with some difficulty
- Unit tests can only be executed locally, but not on the build server
- Commits to new features are made without associated unit tests
Testing & Continuous Delivery
While continuous integration ends with a tested artifact in a repository, continuous delivery also includes the next steps, namely the automated commissioning and testing of the software in different stages.
According to the motto “If it hurts, do it more often”, the frequently time consuming steps of installation and configuration are formalized and brought into an automatically reproducible and usable form.
The key terms here are “configuration as code” and “infrastructure as code”. The complete provisioning can be automated within a cloud environment.
While in the continuous delivery pipeline, the artifact goes through different stages where environmentally adapted tests are carried out automatically. Due to only small recent changes being tested, it is easier to determine what has caused the resulting problem. If any tests within a stage fail, the pipeline is aborted. The number of stages strongly depends greatly on the different types of integration tests that are executed.
Planned stations:
- Development: Testing the individual artifacts; integration with infrastructure
- System test: Integration of all artifacts within the system boundaries; automated system tests; surrounding systems are provided mocked or virtualized
- System integration test: Environment with connection to surrounding systems; automated end-to-end tests
- User acceptance test: Environment in tandem with surrounding systems; in production-comparable settings (configuration and scaling); used for acceptance tests (also manual), load and performance tests
Testing & Continuous Deployment
As an extension to Continuous Delivery, Continuous Deployment also includes the automated deployment of each approved extension in production.
This enables a significant reduction in “time to market”, while allowing early feedback from customers. With carefully applied testing along the pipeline, few software errors are released into production. Those that managed to slip though can quickly be found and removed due the lesser amount of changes.
Supporting test measures:
- Tests in production: Tests in production include a reduced test set that is executed after each deployment. Testing in production requires early planning, with special attention to the handling of test data and authorization/authentication procedures where appropriate.
- Health checks: Small automated tests (e.g. login) that are carried out periodically.
- A/B tests (especially in web applications): New changes are first made available to a smaller group; only after evaluation of metrics is it decided what variant to adopt.
