Back in 1997 Brian Marick (who later became one of the 17 original authors of the Agile Manifesto) wrote an article called “Classic Testing Mistakes” that had become fairly well known in the so-called QA community at the time. According to Brian’s website, this is the paper that caused James Bach to exlaim, “Brian! I never thought you know anything about testing before!” at that time. It’s interesting to take a historical look at this paper, and see what has changed over the past almost 19 years. There is a lot to examine here, but I’d like to highlight two brief ideas:
Marick starts with the role of testing. “A first major mistake people make is thinking that the testing team is responsible for assuring quality”. I think that by now, many organizations have abandoned that antiquated idea. Quality is the responsibility of the entire organization, we are now told. Indeed, I can’t think of a single place where I’ve worked where “Quality” was not listed on the company’s list of Core Principles. On the other hand, there may be certain organizations in certain contexts where the “tester-gatekeeper” model might still sometimes be valid. Once again, context is the key.
Along the same lines, Marick continues with a second classic mistake: ”Most organizations believe that the purpose of testing is to find bugs” Instead, he explains, there is one key word missing: Testers should be finding important bugs. “Too many bug reports from testers are minor or irrelevant, and too many important bugs are missed”, he writes. This is an interesting distinction, because it appears to be an early formulation of today’s context-driven working definition of quality and testing. The now-famous definition of quality by Jerry Weinberg, “Quality is value to some person,” implies that testing is inherently subjective. What may be considered a bug to one stakeholder might be a key feature to another, therefore, the first order of business for the software tester is to understand what is important to their users so that they can focus on finding the important bugs, if that’s what they need to do.
Please take a look at Marick’s paper. Which of these do you think still apply? Have any of them changed? How?
2 comments:
One thing from Brian's paper that struck me as still very relevant - how usability problems get ignored by organizations. I learned early in my career that usability bugs are likely to be set to a low priority and never fixed. It's similar with robustness problems.
I don't see test organizations being the quality gatekeepers very often. But I have recently had senior managers ask me informally if I thought we should release a product. That tells me that I haven't been providing enough information throughout the development process.
As to the first point, absolutely, quality (and testing) should be done by the whole organization. Interestingly, I've found development groups are often hesitant to do testing other than unit testing...they've been taught that they can't do functional testing, for instance. Truth is, some developers are excellent testers and others are awful testers. I do think it is incumbent on the Test group to train the developers on what does and doesn't constitute good testing.
The second item, about "important" bugs, is a tough one, because unless the tester has all the domain knowledge of the user, it's often difficult to tell what is or isn't important to an end user. For instance, I worked on a system that changed the disclaimer message at the bottom of reports...you know, the "legalese" nobody reads anyway. Unimportant, right? Turns out the client used the whole report, lock, stock, barrel and disclaimer, and fed it into their own system. Which promptly blew up because of the change. So could the tester know that this insignificant change was actually quite important? Probably not. Testers need to log every bug, big and small. Those should be reviewed by the business folks, who should have a better picture as to client impact, at least for client-facing functional bugs,
Post a Comment