State Transition Testing
I have a confession to make: if you’d asked me to rank this year’s sessions in order of interest – a session on testing would have come near the bottom. But that was before I’d heard Peter Quentin‘s fascinating talk.
Peter set the scene by agreeing with the audience that, since the combinations of variables to test in a complex system is nearly always infinite, we need a methodical way to test to an agreed level of coverage, an agreed depth that the testing should be carried out to.
He introduced the concept of state transition using the examples of, first, a light switch which could be on or off, and then a simple digital clock with buttons to switch to display time, change time and change date modes.
He explained three different levels of coverage, "0-switch" or "Branch Coverage" – testing each state transition, "1-switch" or "Switch Coverage" – testing each state transition pair and "n-switch" or "Boundary Interior Coverage" – testing each loop of state transitions ‘n’ times.
He explained since exhaustive testing was impossible it would be up to the business user or the client to determine which level of coverage would be sufficient and that in practice, due to financial constraints, testing often only achieved 0-switch or branch coverage.
In a departure from the normal lecture-only style of meeting, Peter handed round workbooks and set the audience the task of constructing testing trees for, first, a simple electronic toothbrush and then a cassette player.
In questions after the session, we examined the difference between "black box" testing, testing without reference to the source code and "white box" testing where the code would be examined to make sure that certain paths are tested. It was agreed that state transition testing was a black box technique and that a state transition diagram is an abstraction of the system created for the purposes of testing.
Some developers present claimed that they could write code containing errors that would only manifest themselves after, say, 10 iterations of a loop, and that Peter’s method would not find such bugs. To which the answer might be, if you do that deliberately, you’ll soon be looking at your P45! In fact, 10-switch coverage which exercised each loop of state transitions 10 times, would find such errors.
This highlighted the fact that exhaustive testing is impossible and that techniques such as state transition testing exist to try to get some acceptable level of coverage. But that level of coverage is always compromised by the time available to carry out testing. If it was known that a loop may exhibit a fault only after 10 iterations then the tester would no doubt use testing time to carry out 10-switch coverage on that part of the system. If it was not known then indeed this fault may very well go unnoticed until some time after the system was implemented.
Such is the nature of testing.
The lively discussion went on until the lecture theatre had to be vacated, and then continued in the Lamb and Flag.
A copy of Peter’s slides and the workbook is on our downloads page.
