Software testing lessons from VW

In our last post, we discussed how VW managed to manipulate their vehicle emissions so they could pass the EPA standards, even though in normal use they emitted up to 40 times what was legally permitted. The fallout is continuing, and estimates for the total cost to VW are nearing $100 billion, according to Credit Suisse's worst case scenario. Apart from huge fines, VW will have to compensate 11 million owners for the loss in value of their property, which could cost thousands of dollars per vehicle.

It's a fascinating case, but what has it to do with security software, the main topic of this blog?

Well, it reinforces an important lesson about software testing - specifically the importance of testing software "in the wild". This means testing software under the actual conditions it is used, and so that the software can't detect it is being tested. If a standard suite of test tools or benchmarks is always used, it will be possible to deliberately code to pass these tests, just as VW did. Test suites have their place, of course, but ad hoc testing using a wide variety of inputs coupled with careful examination of outputs is necessary.

In VW's case, the obvious solution is to test vehicles under real world conditions, out on the highway - not just in a test lab. It's going to be very difficult to fake emissions if this is done thoroughly, and if emissions are measured at the end of the exhaust pipe.

For the increasingly popular computerized voting machines, this means testing a random sample of machines on voting day to make sure results tally - it would be easy to insert code that modifies the results solely on pre-determined voting days in the future.

Similarly, secure servers such as CompleteFTP need to be regularly tested in real-world conditions, i.e. when they are actually deployed and performing their intended function. We do our best to make our products as secure as possible, but we can't foresee every scenario under which they might be deployed, not to mention attacking techniques that have not yet been invented. Of equal importance is regular monitoring of servers - primarily examining log files to ensure no suspicious activity is taking place.

Of course, there are a number of preventative measures that can be taken to harden servers so they are less vulnerable - for a detailed discussion, see this white paper.