Tech Life At the intersection of improbable and unthinkable

The QA Mindset

My first job in technology was a QA internship. The summer between my freshman and sophomore years, I tested the first release of Paradox for Windows at Borland.

As an intern, I started by following someone else’s QA test plan – dutifully checking each test off the list. After a few weeks, I knew my particular area inside and out. A new build would show up, which I’d install via 3.5-inch floppies, and in ten minutes of usage, I’d have a sense – is this a good or bad build?

In QA, there is a distinct moment. It comes once you’re deeply familiar with your product or product area; it comes when you’re lost in your testing, and it comes in an instant. You find a problem, and because of your strong context about your product, you definitely know: Something is seriously wrong here.

Where’s QA?

At the current gig, there’s no QA department. QA as an independent function does not exist and that’s a first for me.

Every company I’ve ever worked at, for the better part of two decades, whether consumer or enterprise software, has had a well-staffed QA function. I’m told this absence is not unique in the Valley. I’m told that both Facebook and Twitter don’t have a QA entity.

Similarly experienced co-workers keep asking, “Shouldn’t we have a QA function?” and while my instinctive response is an emphatic “Yes”, as a member of a rapidly-evolving profession, I need to be open to the idea that software development evolves in ways that may seem counter-intuitive to me. The fact that the team and the company have been successful sans QA is essential and interesting data.

My thesis regarding the necessity of QA has always been: checks and balances. A healthy development team was one that had engineers doing their best to build a great product. These engineers were paired with a team of QA engineers who were doing their best to prove the product wasn’t great. These conflicting goals result in what I consider to be an essential healthy tension between engineering and QA.

Yes, there is often professional conflict between the teams. Yes, I often had to gather conflicting parties together and explain that, “You are both doing your job. No, engineers are not deliberately creating bugs. No, QA is not hating on the product. Yes, we actually have the same goal: rigorously confirming whether or not the product is great.”

The absence of this healthy tension concerns me, but more worrisome is the absence of the practices a thriving QA function builds and maintains. These practices are still with me and are essential to defining and maintaining a high quality product.

They are:

Do I understand the issue?

It’s natural to get rage when software doesn’t work, especially when you paid your hard-earned money to purchase it. Rage is counterproductive to the QA mindset. In fact, it hinders it. The QA reaction to defects is curiosity: Whoa, whoa. Wait, what happened there…? And what follows is a series of interrelated questions that build on each other.

Can I reproduce it? Does it happen every time? What was I doing right before the situation occurred? If there is a crash, does the crash log offer any clues? Based on my knowledge of the product and the code, do I have a hypothesis as to why this might be happening?

In the last decade, software companies have made the process of capturing crashes stunningly easy thanks to the Internet. When your application or operating system crashes, you often receive a dialog delicately apologizing for the crash and asking if you want to submit a crash report. Usually you are asked if you want to add any additional information, and usually you don’t do this, which leads us to…

Can I effectively report the issue?

In this world of auto-submitted crash reports, we the customers have little incentive to provide any information beyond the crash report because we’re mad. Our software crashed, our game was interrupted, our document was lost, value was not delivered. The QA mindset dictates that “Any additional informational, however seemingly trivial, might aid in the resolution of the issue.”

You probably do not take the time to add any additional information when these crashes occur, or if you do, it’s full of rage: “JUST TRYING TO GET WORK DONE HERE PEOPLE.”

Some bugs are slippery. They exist at the intersection of improbable and unthinkable. This is why these bugs are discovered in the wild. Humans do strange shit to software that we could never predict in the controlled setting of our carefully constructed software development environments.

Slippery bugs are a mystery, and there is initially no telling what contributing factors are relevant. In my time in QA, for issues that were hard to reproduce, we prided ourselves on documenting everything, however irrelevant, that might have lead to the crash. Totally clean OS – just installed. Wired connection, wireless disabled. No virus software. All files are local.

You’re not going to do this because you don’t perceive that you have skin in the game. You’re correctly assuming that part of what you were paying for is quality, and you likely haven’t been in QA. You likely haven’t received that mail in the middle of the night from the development team, who has been chasing that slippery bug for the past two weeks where they ask, “Can you try this patch? We think we fixed your bug.”

And they did. Because you took the time to think before you submitted your report, which leads us to our last part of the QA mindset…

Do I perform these actions unfailingly?

The last QA dictate is the most important and the least likely one that you perform. The last dictate is: “In the face of a problem, do you act to correct it? Unfailingly?”

There’s a bias towards system and applications crashes in the observations above because these crashes are the defects that give us the most rage. And while identifying and fixing these crashes is an obvious high priority, there’s a whole other class of defects of equal priority that are less obvious.

My favorite internal application at Apple is a product called Radar. It’s a Cocoa application that served as our bug tracking system, and if you wanted to know what was going on regarding a specific application at Apple, you went to Radar.

For many groups at Apple, Radar was religion. An issue regarding a product was not considered to exist until it was in Radar. If someone asked me, “Have you seen this bug in your product?” My immediate response was, “Is this in Radar?” “No.” “We’re not talking until you’ve filed Radar.” Case closed. For now.

The unfailing rules were:

  • If you see something wrong in the product – however big or small, report it as best you can.
  • It is good form to take the time to report the issue as descriptively and thoroughly as possible, but it is more important to report the issue.
  • It is also good form to check if the issue has been reported by someone else, but it is more important to report the issue.
  • When the issue is reported as fixed, take the time to confirm it as such, because more often than not, it’s not and/or it created another related issue.
  • Failure to follow these rules will be met with an immediate reminder of the aforementioned rules.

For the teams that unfailingly followed these rules, Radar became a powerful tool because it became a credible source of truth regarding the product. The answer to the question, “How is the product doing?” wasn’t an annoyingly vague, “I’m feeling good about it.” The answer was, “We’re fixing critical issues at a rate of 1 issue per engineer per day. We’ve got 14 engineers, we’ve got 308 issues, which means if no issues arrive, we’ve got 22 days of work. Except our arrival rate is 7 a day and it’s increasing. Also, next time you want to know this data, here’s the query you run. Stop wasting my time.”

You are sensing rage in my answer because I’ve spent a career surrounded by well intentioned humans who believed that it was QA’s job to file bugs, and the fact is that quality is a feature, so like it or not, everyone is in the QA department.

QA is a Mindset

In a pre-Internet world, one of the key reasons for a well-defined quality assurance team was the cost of distribution. When you released software, it required producing a pretty shrink-wrapped box full of disks and documentation. This was expensive to build and ship. More importantly, the infrequent yearly release of this shiny box was the sole yearly opportunity to get your software in front of your customers. It could be upwards of a year before you had a chance to right your buggy wrongs.

Thankfully, blissfully, this is no longer the world we live in. At the current gig, we’re releasing the website a couple times a day. We’re releasing apps at a slower pace, but when I say slower pace, I’m talking days… not months… never years. Perhaps our collective ability to not only rapidly detect, but also fix issues within our products, has made us less dependent on relying on an independent QA function?

My concern is that the absence of QA is the absence of a champion for aspects of software development that everyone agrees are important, but often no one is willing to own. Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective. Traditionally, they are known as the folks who break things, who find bugs, but QA’s role is far more important. It’s not that QA can discover what is wrong, they intimately understand what is right and they unfailingly strive to push the product in that direction.

I believe these are humans you want in the building.

Leave a Reply

Your email address will not be published. Required fields are marked *

38 Responses

  1. Andomar 9 years ago

    Years ago I was in a company that didn’t have QA as a separate entity. We merged with a larger company that did. The larger company asked us to run our release by the QA department.

    As an experiment, I send a large release to the customers and to QA at the same time. I made a list of bugs found by each group. The results were very interesting. QA found entirely different bugs than the production users. Of the three major issues, none were found by QA. And you couldn’t expect them to, those issues involved interactions with other software in weird ways.

    QA found many issues with no relevance. If we ran every release past QA, we would figure out the real problems later, and we could fix them slower, because the fix would have to be run past QA again.

    My conclusion was that doing QA actually lowers software quality.

  2. I’m not in the software biz (back in the days of the PARADOX you mention I was a technical writer and trainer for a fortune 10 company) but since then just a user (interesting word in this context).

    In any case, your post reminded me of a book called “Close to The Machine” by Ellen Ullman
    I read it long ago but the intensity still resonates.

    Toward the end when you mentioned the QA mindset of pushing the product toward what is right – I thought that must have been the beauty of what Steve Jobs did. This kind of thinking applies not just to companies that make software. In really good companies it pervades the organization. But to do that it must come from the top.

    But it goes all the way through the organization. I’m reminded of an incident in a retail store where I asked a fellow where a certain item was. As he led me to the right aisle, three or four times he picked up litter or straightened an item that was out of place – all without breaking stride.

    I wished at the time I’d had an opening in my company to make him an offer. Definitely the kind of human I wanted in the building.

  3. Walt French 9 years ago

    Sure would be nice if mortals could access RADAR. Although I’m a registered dev, I’m not doing apps so merely download the latest Xcode and don’t really know the dev systems.

    So fr’instance, I use Calculator o my Mac with speech enabled. “2848 * 1.5=” nicely produces “4,272” on the display, but it speaks just “4”. All responses are truncated at the comma. That’s a bug, a potentially bad one for somebody who needs audio feedback (that I don’t really). Of course, bugs always have the potential to cause memory problems in other apps that use sound or speech, but as a user I have no way of telling what that little gremlin might crash.

    Is this in RADAR? I have no idea. Is there a general user way to report these bugs that result in wrong results but not a hard crash? “Inquiring Minds Want to Know!” ®

  4. Thank you for this article. For 33 years my company has been producing software for financial advisors. I wouldn’t dream of releasing software without our QA department approving it. We want our own humans doing strange shit to our software before we release it. People rely on it to make financial decisions that affect their future.

    Maybe it’s okay for non-critical software to be released as Facebook does. An error in my Facebook feed isn’t going to do me harm. Much software though, isn’t in that category.

    A good QA team is one more layer of defense against an error that can ruin your company.

  5. As a long time reader of your blog and books, I think this is the first I’ve heard of your involvement (and interest) in QA / Testing.

    Everything you say is spot on, and your concerns are valid. My stance on the concerns is that even if a company doesn’t have a QA role, the QA /activity/ still needs to occur. I also believe that there are some productivity gains that come out of a single engineering team (the “checks and balances” you describe above can turn into a bottleneck on many situations.

    I think a no-QA organization where everyone does the same thing fail once they reach any reasonable degree of size or complexity. Instead, an engineering team of generalizing specialists (including those with the “QA Mindset” ) has the potential to make good software more efficiently than a “traditional” test/dev team.

    But, as you can imagine, it’s really easy to do this wrong.

    If you’re really bored (or annoyed) – I ranted about this exact thing last month.

    And slightly related follow-up

  6. Ray Baxter 9 years ago

    Walt French, yes you can file bugs here:

  7. stephane 9 years ago

    There’s a problem with the Radar Story.

    “For the teams that unfailingly followed these rules, Radar became a powerful tool because it became a credible source of truth regarding the product. ”

    The truth is that from the outside, Radar is a symbolic link to /dev/null and this is why too many persons are not willing to report issues through Radar (anymore). So this is only a partial truth, which is either an illusion or a lie.

    Everything should be done to make reporting issues easy, intuitive and fast. And to ensure that the reporters are not wasting their time. Because the more reports you get, the closer you are to the real truth.

    Instead, from the outside, we get a solution where every step of the process is a disappointment.

  8. Kendall 9 years ago

    One of my first jobs was very similar to yours – starting in the QA department at a small company, with a product that ran across many different systems… after a while of developing and maintaining tests for the application, I eventually moved on to development and have been a developer ever since.

    I have been thinking that possibly so many people not thinking they need QA, is because lots of people lean heavily on unity tests now. But one thing I took away from my time as both QA person and developer for the same application, was that any test a developer writes is not worth a lot – they have a lot of assumptions developing which are all too easy to carry into unit tests, which just verify something was written the way the person writing the test thought it was written.

    A real QA department, or even just someone else writing tests for code can find a million things you would not think of, and has far more value than dual code paths to maintain that revolve around the same assumptions.

  9. Apple of My Eye linked to this.
  10. John F 9 years ago

    This is about as good a definition of what a QA team can do, and why quality is everyone’s job, as I have ever seen.

    I don’t break software, I just shatter the illusion that the software is working.

  11. @Andomar, it sounds like perhaps you just didn’t have a great QA team. Or they didn’t fully understand your product yet!

    Even a good QA team needs a while to understand the product fully and the use cases the customer is likely to see.

    What you have is an anecdote, not data. I wouldn’t draw conclusions from it if I were you. 🙂

  12. QA Mindf**k | linked to this.
  13. This story hits on a distinction that needs to be identified. There are two types of QA… the first kind is described by the work I’ve done with smaller shops; this means ad hoc, and targeted testing that addresses scenarios common with the expected use of the software (know your users!). This is generally very effective. Then there’s the other kind of QA, where mega-corporations write Test Plans that are off-shored to a developing nation. Tests are performed by those who are hourly bargains but don’t know or care about the product.

    As soon as the majority of the testing is done by the latter, you can kiss your quality goodbye. (I’m looking at you, large campus in Redmond, circa 2004) When you have the idea of “quality” quantified like a factory process and a budget line, you have lost the plot. Sure, some of it is useful, but without inquisitive humans, you lose the organizational ability to anticipate much of that “stupid shit” within the context of your product.

    In other words, QA isn’t just a function; it’s a representative of your users in-house. If you don’t have user advocates, you will be subject to marketing’s feature creep… a very slippery slope.

  14. QA in a post-QA world linked to this.
  15. My experience with QA is varied, starting with the bureaucratic flavor that appears to relish holding releases hostage all the way to a more fluid, functional team that is tightly coupled with development. I’ve worked with detached, robotic testers and former (fallen?) developers who want to try to push limits and break things. I’ve never been against the function of QA, but I’ve definitely taken issue with implementations.

    Recent history in my industry calls for solid QA, at least as a check and balance against software that could ruin the company in minutes and lead to months and years of legal hell. My industry is staffed with many smart developers who often focus more on shedding nanoseconds and less on handling edge cases or anticipating exotic interactions, so rigorous testing up and down the stack is essential.

  16. JRock 9 years ago

    This is a fantastic article. While I agree that QA is an essential component to delivering a quality product, I find that the quality of QA teams vary greatly. QA teams that I consider to be high quality are usually comprised of full-time employees with years of experience testing a particular product. They also have an intimate knowledge of the software’s purpose. I also think having a relationship with the end users (if possible) assists in proper QA testing.

    When a QA team is farmed out, I find the quality of the testing suffers. Often, these types of QA teams cannot work independently and require guidance to perform their testing. I believe in a healthy separation between the development and QA teams to promote unbiased testing results.

  17. @Andomar, if that QA team found issues of no relevance, then it is because they were allowed to/guided to/directed to. If you create a release and then throw it over the wall to QA and say “Have at it, boys “, then you get the full suite of QA, from cosmetic issues all the way up to major things. Your team needs proper guidance as to what things matter to you, and involvement from earlier stages than a consumer-ready end product where small things are too late to be of value.

    The value in QA is in validating that things are correct as much as it is in finding that things are wrong. If the QA process is suitable and it finds nothing, then that is actually a good thing.

  18. Quality Assurance – Stuart Breckenridge linked to this.
  19. It is true that more and more organizations are moving away from a ‘QA’ organization however I think in order for this change to be truly successful some key elements need to be put in place which include:

    1. Utilization of BDD/Specification by Example. Having the entire team build out contextually rich user stories and acceptance criteria builds the understanding of what the team is going to deliver, everyone shares in the definition and the scope of that work is defined by the Specification acceptance criteria (getting test automation as a by product is simple great value add).

    2. The team, which hopefully includes Software Test Engineers (the new QA role) understands that they OWN the quality of their work, not just the technical excellence that they may build into it, but also making sure that the feature delivers WHAT the customer wants.

    Not having a QA group seems counter intuitive, but when you realize that we have really removed the wall that was always between Developers and Testers and put them together to solve the same thing they actually always wanted, deliver great product.

    There will always be bugs, however as we mature our development and BDD practices I tend to look at those as more undefined requirements or scenarios. that the team didn’t account for (Note – BDD is great for playing the What-if game to uncover these off the normal path issues, define the behavior and you get much further in your understanding of what you are delivering)

  20. Quality Assurance | Stuart Breckenridge linked to this.
  21. The Good Way To File A Bad Bug | Only Best News linked to this.
  22. Terence 7 years ago

    from one analyst to another, truly a great read.

    Thank you for sharing,