Wednesday, December 23, 2015

Debunking Myths of Exploratory Testing

The article “Exploratory Testing: Tips and Best Practices” appears to present some helpful suggestions about implementing exploratory testing in your project.  However, certain assumptions and statements there indicate that perhaps that author is confused about some fundamental ideas of ET.  Below are a few difficult assertions in that piece, and my comments about them.


Ø  Exploratory Testing: Tips and Best Practices”


BB: We begin with the title.  There are, as is increasingly becoming known, no such things as best practices in the literal sense.  There may be good, recommended practices sometimes, but because different situations require different solutions, we can’t say that a practice is best in all situations, even for exploratory testing.



Ø  “Exploratory testing helps quality analysts and others involved in the testing field ensure systems and applications work for their users.”


BB:  Exploratory Testing does no such thing.  Testing cannot ensure anything.  Testing is a search for information; it is scientific experimentation.  Scientists know that experiments that are designed to confirm an expected value are of far less scientific value than experiments designed to discover how the hypothesis may be disproven (as we learn in BBST Foundations).  Similarly, the result of testing is to report findings and make recommendations rather than “ensuring” that the system works.



Ø  “Exploratory testing is often misunderstood as an approach…”


BB: He has is 100% backwards.  Exploratory testing is exactly an approach.  It is not a technique.  A technique is a way to do something, like following a recipe is  a way to bake a cake.  Exploratory testing isn’t so prescriptive.  By contrast, an approach is a more general concept.  It is the overall manner in which you act.  It is your gameplan, your strategic process, your perspective and your outlook.  Exploratory Testing is not a specific testing method used to accomplish a specific quality goal.  Instead, it is the overall manner in which testing occurs.



Ø  “You are not exploratory testing if you are following a script”


BB: Except when you are.  There is a continuum of exploration, and even when following a script you may be doing it in an exploratory way.



Ø  “The aim of exploratory testing is not coverage – it’s to find the defects and issues in a system that you won’t find through other forms of testing.”


BB: Except when the aim of exploratory testing IS coverage.  Usually, the aim of exploratory testing is to evaluate how much value the product will add or detract from its stakeholders, as the tester learns about the product.  It’s only about finding defects and issues when the information objective is bug hunting.



Ø  “Quite often edge case defects will have a high level of severity even if they are less likely to arise.  This is due to the nature of exploratory testing – it focuses on the parts of a system that are away from the normal usage pattern and are less likely to be well tested.”


BB: Except when the testing missions IS to execute the system in the normal usage pattern, for example in a “Ready for Business” or smoke test situation.



Ø  “Typically, exploratory testing needs a greater level of testing skill and experience than other testing techniques”


BB: As previously mentioned, exploratory testing is not a technique, it is an approach.  But the general idea is true.  ALL of testing requires highly skilled professionals.



Ø  “Keep a clear record of what you did….”


BB:  I agree.  But credit should have been given to Jonathan Bach who presented this material in his talk “How to Measure Ad Hoc Testing” at  STARWest 2000



Ø  “Use exploratory testing alongside automated testing”


BB: This statement assumes that exploratory testing and automated testing are two different things.  But they don’t have to be.  Just as so-called “manual testing” could be done in an exploratory or non-exploratory way; so to could test automation be done in an exploratory or non-exploratory way.  Exploratory automated testing is very powerful.



Ø  “ensure that the system is built to support exploratory testing, e.g. build functionality in slices that can be tested from start to finish as early as possible.”


BB: This is not necessary at all, as long as you focus your exploratory testing sessions around clear test missions.





Questions? Comments?  I’d be delighted to hear from you!



Bennett Weber said...

Good critique. Exploratory testing is like driving a can take off in some direction without specific travel plans, or you an follow a "scripted" route...but nothing prevents you from pulling off the highway to take a closer look at points of interest. Things are not black and white...too many times, testing is viewed in terms of false dichotomies. Testing is automated or manual. Testing is exploratory or scripted. As you point out, it's a continuum.

Blogger said...
This comment has been removed by the author.