Like many, I was amazed by Volkswagen’s Dieselgate but at the same time wasn’t surprised either. When governments set strict requirements to be able to sell your product or even subsidise it when it meets them – what do you expect?!

Unit Testing

The way VW cheated on the emission tests made me think of the limitations of the popular unit tests we use for software. Like the emission tests, unit tests are performed on a single feature in a controlled environment. The idea is that the same code should fail or pass a test every time you run it. You want to rule out any other factors so that you know for sure it’s the code that needs fixing, not something else. Very effective, but eventually that software will be used in a dynamic environment with tons of other processes, programs and dependency versions. Just like that car will be used on the road and not on a stationary dyno. This is why unit testing should always be combined with integration, acceptance and other more real-world-like tests.

Manipulating a Unit Test

Now of course there’s a difference between unit testing your own software and having an authority test your car’s emission. Why would you cheat your own tests? Well, I wouldn’t be surprised if the social pressure Spicer describes could also cause people to manipulate tests internally. And a lot of the acceptance tests might be executed by customers, they are often based on tests written by the same engineers that wrote the code. Another example would be how Travis can run your unit tests on GitHub Pull Requests. In this case, a lazy or evil developer could easily manipulate a test to pass even when it shouldn’t ;)

How to prevent another Dieselgate?

The solution to prevent another Dieselgate is actually quite simple. First of all these lab tests should be accompanied by real-world verifications with a maximum allowed discrepancy between both results. But even more simple and effective would be to combine emission tests with tests for the cars behaviour, in particular the engine output. The dyno calibration VW activated during tests not only reduce the emission but also the engine output/power. That is why on the road it would use a road calibration to get drivers the advertised performance. Now if manufacturers would be required to advertise the engine output (and other relevant metrics) measured during the emission tests they could no longer show off with low emission while at the same time also having to deal with marketing a far less powerful car.

What can we learn?

When you’re developing software accept that you can never guarantee it will run on each of the unique environments of your users. Yes, do test your software as much as you can on as many different environments possible, but also equip your software with good error logging and tracing tools so that when your users run into those real-world-problems they have more to feed back to you then just..