We have spoken before about testing your software. In particular we have mentioned how if your code isn't tested you can't be confident that it works. Whe also spoke about how the technique of testing and the level at which you test your code will vary based on what you need to test.

What I'd like to talk about this time is about understanding the environment in which your tests exist. Since "nothing exists in a vacuum" it is critical to understand that even if you write beautifully targetted tests, they still exist and execute within the wider context of the computer they are running on.

As you are no doubt aware by now, I have a tendency to indulge in the hoary old developer habit of teaching by anecdote, and today is no exception to that. I was recently developing some additional tests for Gitano and exposed some very odd issues with one test I wrote. Since I was engaged in the ever-satisfying process of adding tests for a previously untested portion of code I, quite reasonably, expected that the issue I was encountering was a bug in the code I was writing tests for. I dutifully turned up the logging levels, sprinkled extra debug information around the associated bits of code, and puzzled over the error reports and debug logs for a good hour or so.

Predictably, given the topic of this article, I discovered that the error in question made absolutely no sense given the code I was testing, and so I had to cast my net wider. Eventually I found a bug in a library which Gitano depends on, which gave me a somewhat hirsuite yak to deal with. Once I had written the patch to the library, tested it, committed it, made an upstream release, packaged that, reported the bug in Debian, uploaded the new package to Debian, and got that new package installed onto my test machine - lo and behold, my test for Gitano ran perfectly.

This is, of course, a very particular kind of issue. You are not likely to encounter this type of scenario very often, unless you also have huge tottering stacks of projects which all interrelate. However you are likely to encounter issues where tests assume things about their environment without necessarily meaning to. Shell scripts which use bashisms, or test suites which assume they can bind test services to particular well known (and well-used) ports are all things I have encountered in the past.

Some test tools offer mechanisms for ensuring the test environment is "sane" for a value of sanity which applies only to the test suite in question. As such, your homework is to go back to one of your well-tested projects and consider if your tests assume anything about the environment which might need to be checked for (outside of things which you're already checking for in order to build the project in the first place). If you find any unverified assumptions then consider how you might ensure that, if the assumption fails, the user of your test suite is given a useful report on which to act.