Testability is the capability of revealing how well something works. In artificial systems, testing is used to determine whether design objectives are being met. A testable system requires to be observable, where observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
But what happens when the system of interest proves to be, for a variety of reasons, untestable?
Untestability can happen due to the system being, as said above, non-observable: the system is there but there is no way of extracting any information about how well or how bad it is performing. The system might be untestable just because the sole act of testing would cause implications that invalidate the pay-off of the testing process itself—think of nuclear warheads and the political implications of testing them. Or the system might be untestable because the environment where it must perform is not replicable before full deployment.
This introduces us to the concept of “post-deployment testing”: assuming the system is observable, it is only by fielding it in its real operational environment that we are capable of assessing its true performance. No staging, scaled-down or simulation environment will stimulate the system in the same way nor provide the same amount of information compared to taking the thing for a spin.
Complex adaptive systems such as organizations or even a country’s economy follow a similar pattern. No scaled-down environment will provide the same information as the full population—what is more, mockups may provide misleading information. Therefore, policy-making can only truly be tested once deployed to their fullest.
It is only by firing a missile that we can truly assess it performs as expected—ironically its total destruction on target being the final confirmation of success.
It is only by restructuring a team that you get to know if, as Tom DeMarco would say1, it jells.
It is only by launching a product or a company that you get to know if customers want what you offer.
There is a delicate and often overlooked balance between investing time and effort building synthetic environments that will never match reality versus taking the damn thing for a good spin.
Nothing better than testing by using.
https://www.amazon.com/Peopleware-Productive-Projects-Tom-DeMarco/dp/0932633439