Software testing is one of those amorphous terms that covers a massive range of possibilities. Are we talking about integration, unit, black box, white box, systems, acceptance, regression, load, performance? The list goes on. Unlike a lot of other forms of software development, there is generally no specific point at which it is possible to say “the testing is complete”. It is always possible to add more testing infrastructure.
The difficult point here is generally code testing and test coverage. These goals cannot be achieved simply by implementing an existing system. They consist of code that must be written for each project, often equalling or exceeding the amount of code in the product itself. As such there is an on-going development overhead involved in maintaining good tests. So is this worth it? Should developers spend as much time testing their code as writing it? In our experience, this is almost always a worthwhile effort, with paybacks in testing generally exceeding those from other areas, assuming that a pragmatic approach to testing is taken.
At Spore Lab we believe that while good testing is multi-faceted, it is more important to implement a few testing methods well than try to cover every type of test. To this end we try to focus on the following areas:
Modules should be initially developed using test-driven development where possible.
All modules should have unit tests, covering both positive and negative testing.
Unit tests should be fast, isolated and automatic.
Developers should be presented with unit test failures as soon as possible, ideally before code is committed to the repository.
An integration test suite should be available. When doing embedded development, often integration tests require physical interaction with a device under test. As such these tests are often slightly more difficult to execute.
Test jigs are important here. Integration tests for embedded development require the ability to actuate all the interfaces of the device. Some of these may be genuine connections, and some of them may be faked. For instance connecting to the HDMI input of a device can be done via a USB dongle, however generating a test cellular signal is generally not plausible, and so faking the serial communications for the modem may be a better option.
One caveat to this, is that user interfaces tend to be very difficult to test, and the tests written tend to be much more likely to fail in the future due to normal product evolution. So the time/benefit trade-off for these forms of tests may mean that there are fewer of them with less coverage.
As a possible alternative, or addition, to integration testing, we recommend having some form of fuzz testing. With fuzz testing, pseudo-random input is supplied to the software in an attempt to exercise unexpected corner cases. In the simplest and most common scenario, the test can be as simple as running the application and then supplying it with random input data until either a certain amount of time has elapsed, or the application has suffered some kind of fault.
It is important that the input be only pseudo-random, since if the test fails it is important to be able to recreate the input stream that led to the failure.
If available, some form of back channel can be provided to the test system to allow it to target specific aspects of the interface. Without this it is quite common for random testing to essentially get stuck in a dead end, and not able to get out. It also allows some guidance towards known problem areas - such as exceeding the length of data inputs, or supplying extreme input values, such as integer inputs that might invoke integer overflow.
One nice advantage of having a simple fuzz testing system in place is that they don’t tend to require the same amount of on-going maintenance as other test systems. Since the tests are not targeting specific application areas, they freely scale along with the application, with no additional test code. This can make them very efficient in terms of effort vs. payback.
When bugs are reported, tests demonstrating their existence should be added prior to fixing the bug, where possible.
The combination of low-level unit testing, and high level integration testing means that we get maximum coverage for only moderate investment. It also creates a fairly clear line as to where each type of test is most appropriate.
As with all things software development, this approach is not a panacea. Some software is inherently hard to test, especially when dealing with both user interfaces, and external hardware. Test code is also no different to application code - it takes time to write, and therefore costs money. In every development a balance has to be struck between the progression of new development, and the quality of existing development. A good general principle to use is that at a minimum the infrastructure for testing must be in place. This prevents ‘development friction’ from being a reason to not implement tests. So rather than enforce testing, we instead focus on a system that makes development and execution of testing easy.
For further reading, there are some excellent resources available from http://softwaretestingfundamentals.com/.
The US Government’s National Institute of Standards and Technology also released a thorough report back in 2002 detailing the reasoning behind testing http://www.nist.gov/director/planning/upload/report02-3.pdf.