Understanding random testing

There are two good reasons to do random testing.

The first is that you’re working with a homogenous or expected distribution, and the cost of letting something defective slip through is pretty low. For example, the chef doesn’t have to taste every single serving before it leaves the kitchen. Quality manufacturing is based on the efficient use of random testing to be sure that each batch is expected to be within tolerance. It works as long as the thing being tested isn’t itself widely variable. If it is, you’ll need to test every single unit.

The second is that you’re trying to send a message to alter people’s behavior. Drug testing or tax audits are examples of this. You can’t test everyone, but you make it clear that there’s a non-zero chance that someone who is outside the rules will get caught.

We do random testing all the time without realizing it. That’s probably a mistake–we should either test every unit (when the stakes are high or when outcomes are unpredictable) or we should trust our people and our systems enough to test very rarely.

It doesn’t make sense to set up a random speed trap. Either measure the speed of every car or measure the speed of none of them.

Sending a message through testing and draconian punishment might make good security theatre, but it’s a waste of time and trust.