General Remarks
Testing shall assure the quality of a product and that development does not break the already implemented functionality. The quality is defined in ISO 9126 for example (see: http://rickrainerludwig.wordpress.com/2014/02/10/software-quality-charateristics-in-iso-9126). Testing is a proof of quality and tests are accepted delivery criteria.
Tests can be separated roughly into different kinds of tests:
- Functional tests
- GUI tests
- Non-functional tests
- Penetration tests
Functional Tests
Functional tests are all intended to run without any GUI interaction. The checks are on functional level and use data to check all kinds of transformation, calculation and handling. These tests assure the correct behavior and results of the application. There are three different kinds of functional tests which are explained below:
- Unit tests
- Component tests
- Integration tests
Unit Tests
Unit tests are intended to test single methods and classes in total isolation on a functional level. Here the implementation of single methods and functions are checked like correct calculations, and correct error handling for invalid parameters. Tests should be performed for valid parameters with a check on the returned results, invalid parameters and the correct error behavior and for corner cases which are to be checked for correct behavior, too.
These tests should be provided by the developers. Unit tests can be created during development as simple ways to check the development code. Test Driven Development (TDD) would be best. Except of file IO no other external infrastructure must be used. Mocking is allowed, if needed, but as it is about single methods and classes, it will not be necessary in most cases.
Since in an optimal setup each method should have at least three unit tests (valid, invalid and corner case parameters), a lot of unit tests are to be expected. Because of this high number of tests, a time constraint should be kept of 0.1s for instance. 10.000 tests (which is around 3000 method, which is 500 classes) would take about 20min. This is a lot of time for a continuous integration test, which uses unit tests.
Component Tests
Component tests test several classes in relationship to each other. In component tests no environment should be involved. The environment is simulated with simple mock-ups. Only the filesystem should be involved in testing to load sample data for instance. For mocking the Mockito framework is used in a lot of projects at PureSol Technologies.
Ideally, these tests would be provided by the developers, too. The scope is also functionality, but with the interference of other classes. Component tests would also be a good check during development.
Component tests can and should be run in a continuous integration way, too. That’s why component tests should also have a time constraint of 0.1s, too.
Integration Tests
Integration tests check the functionality of the product with environment interaction. These tests also include databases and other server systems which will be in place during normal product usage. The environment for integration tests should be as close as possible to the target environment.
Integration tests tend to run a longer time than component tests and unit tests. These tests should run after the normal continuous integration build separately as a single test stage. Integration tests do not have a fixed time constraint.
GUI Tests
The tests following in this section are tests on GUI level. The tests are either pure UI tests or end-product tests.
Pure UI tests just check the correct appearance and functioning of the UI itself. These tests should be quick. Together with the integration test of interfaces like REST, the functionality of the final product is quite likely, but not completely proven.
That’s why, product tests can be performed via UI. These tests check the correct functionality and are intended to test the final product like the customer is going to experience it. The test environment setup and configuration shall be as close as possible to the customer installation.
There are just two possibilities for GUI tests:
- Automated GUI tests
- Manual GUI tests
Automated GUI Tests
Automated GUI tests check the standard behavior of the GUI and are a good tool for end-product testing.Here the most common use cases shall be tests and not only the happy tests, but also the error behavior.
The tests’ maintainability is quite expensive due to a changing GUI. The tests are easily broken and the repair needs to be done carefully. Therefore, these tests should only check the GUI and not the functionality. The functionality is checked with non-GUI Tests.
Manual GUI Tests
These tests are manual test. These tests are very expensive due to the binding of a lot of people for the task. These tests should be minimized to check the aesthetics of the product like colors, graphs, prints and layout. These tests can not be automated.
Non-Functional Tests
Strictly speaking, GUI tests are also non-functional, but here we need to test everything which is not related to the use cases of the product itself.
The possible tests are:
- Endurance tests
- Performance tests
- Resource tests
Endurance Tests (Iteration Tests)
Endurance tests are tests which test one functionality for a large number of times. For example a certain calculation can be performed 1000 times one after another. You can see several effects here which might influence the quality:
- Memory Leaks: After a large number of iterations one can see memory leaks. The function should be tested at least as often as it is expected to be run in a single application startup on client customer site.
- Stability: One can see how stable the function under test is. This can be a mathematical stability as long as the formula relys on some environmental data or the stability in a communication module and so on.
Endurance tests can be coupled with performance tests to save time. During the endurance test the time can be measured for function execution and a statistics can be provided.
Performance Tests
Performance tests test the speed and answer time of applications. Some applications are time critical or time is a measure of service quality. Performance tests assure the correct performance behavior.
Load Tests
Load tests test software applications on heavy load. These tests are important to check how stable an application is on load due to memory consumption, concurrency effects and bottle necks within the applications. Some of these issues would be undetectable without such tests.
Penetration Tests
These tests are done mostly by humans and not robots due to its creative nature. Penetration tests try to break a product through use cases, to show the stability and reliability. The goal is to break the application and to provide information about the breaking mechanism to stabilize and harden the software application.
Other Test-like Tasks
These kind of tests are explained below. Additionally, there are two other kind of tests: Static Source Code Analysis and Code Reviews. Both can be performed by Developers and/or QA and help to keep the maintainability high.