Tech Life Quality, features, or time?

Bugsssszzzzzzzz

I’m in the midst of chewing on an article regarding things you absolutely, positively must have in order to design, develop, and ship product and what to do when those things just don’t exist.

The article and my current work situation has reminded me the importance of bug tracking. In each company I’ve worked, the bug tracking system has always emerged as the premiere source of truth regarding the state of the product. For better or worse, it’s often the authority regarding the question, “How close are we to shipping?” The one company that chose to ignore bug data, not surprisnly, failed miserably… releasing poor quality bits on a dictated schedule.

The question is: How do you track bugs for your development process? Doesn’t matter if you’re IBM or Portland Indie Development Shop (PIDS?)… I want to know how you manage your bugs. Spreadsheet? Home grown tool? Professional tool? Let’s hear it…

#

16 Responses

  1. Benko 21 years ago

    We use Mantis. It’s solid, easy to setup, it simply works. I’d prefer GUI frontend, but web gives us platform independence (win, linux, os x).

    I think what’s important are attributes that you can setup for a bug entry, and tracking the discussion around a bug (web, emails). I would love to have a bug tracking integrated with my editor and version system (CVS), unfortunatelly I don’t know about such product yet.

    It would be nice to see what was changed in between two versions, which bugs have been fixed, which occured, and direct links to files that were changed. Now we write this info manualy to mantis entries, but it’s a pain.

  2. Martin Sutherland 21 years ago

    I’ve only ever worked on company internal systems rather than public-facing or shrink-wrapped systems, but here’s what I’ve found: all of the bug tracking tools I’ve worked with, no matter whether they’ve been bought in or home-grown, have been loathed by their users. (The bought-in systems have been: PVCS (now Merant) Version Tracker, FogBUGZ, and NetResults ProblemTracker)

    Developers hate them the most, because they always think they could write a better system themselves. Testers either hate how much data the system forces them to capture, or hate how little data the system allows them to capture, or hate the format the system forces them to capture it in. Project Managers hate not being able to get summary information in the way they like it, so they persuade the DBAs to let them use Access to run their own reports against the underlying database. The Business tends to complain least, because they’re paying attention to the application being tested, rather than the bug tracking tool.

    The uniformity of dislike tells me that it’s probably the *process* of tracking bugs that people hate more than the tracking software itself.

    The other correlation I’ve noticed is that, no matter what the bug tracking tool, the success of a project is proportional to the number of bugs raised.

  3. mike 21 years ago

    At work, we use lots of open source tools

    to build things for clients. Our clients

    are really more consumers of the services

    than the bits generated themselves, if that

    makes any sense. (i.e. they don’t need

    permanent ownership of a given thing, and

    don’t care what it’s written with, so long

    as it works for their purposes for the

    given interval of time)

    Anyway, this has extended to the tools

    we use to manage the development process,

    namely Bugzilla for QA and cvs for revision

    control. We’re still working through the

    process of integrating everything into the

    internal workings of the development flow,

    but have so far had very good success with

    everything.

  4. Mark Eichin 21 years ago

    I’ve seen one commercial effort fail though the use of bugzilla (basically, the excessive detail required on input was a poor match for the project, without judging whether it is a poor match for other projects…) PRMS worked surprisingly well for us at Cygnus for years, but doesn’t match all that well to other environments/cultures (and really it’s surprisingly retro right now.) If your emphasis is on tracking, rather than “bug”, bestpractical.com’s open source “RT” is excellent and has a lot of high end commercial traction – I’ve argued, but not deployed, that request tracking and bug tracking are *different* things, that you have a different set of goals for managing the *report* of the problem (especially when it comes from a customer) than you do when managing the *defect itself* within the code. Unfortunately I haven’t seen any systems that were especially good at the latter (imagine an enhanced cvs annotate with information about nearby bugs, and repository wide “defect-affinity” data, hmmmm.) Much of the resistance to systems like this seems to come from their use by managers as ways of tracking “defective developers” 🙂 Oh, and I’m also using a small one-user roundup instance specifically for security-issue tracking, mostly because it is zero-overhead to get started… odds are that I’ll experiment there, develop a schema and a policy out of that, and then get the schema deployed within the organizationally-supported RT installation…

  5. Alex 21 years ago

    I use Tasks to track both development features and bugs. The heirarchical organization makes sense to my mind and I like being able to see the tree of an entire release (features and bugs). I’ve used Tasks Pro for this with several small development groups and the net result was much more voluntary usage of the system than I saw with the bug tracking systems at several larger companies I was at. I think the ‘thousands of drop downs and too small text fields’ approach you see in many bug tracking systems makes them so slow and painful to use that many developers feel like they are a waste of time.

  6. Alex 21 years ago

    Sourceforge enterprise edition, it’s a great development control tool, and manager like the fact that it plug into MS project, and I like the fact that by now, almost everyone has seen sourceforge.net so training time for everyone is quite low to inexistant. Nothing more boring than to train someone to use the bug reporting system…

  7. charles kens 21 years ago

    Within an iteration, we track defects with failing automated tests – a feature isn’t “done” until it is passing all automated tests and manual testing.

    Outside of an iteration, we track bugs the same way we track feature requests – with index cards.

    We’re aiming to get to the state described here:

    http://martinfowler.com/bliki/VeryLowDefectProject.html

  8. rands 21 years ago

    Random responses:

    1) Bug tracking systems are almost universally loathed by engineers and I think it’s only partly due to poor design/interface. I think it’s also partly due to the fact that bug systems are effectively criticism. YOUR STUFF DOESN’T WORK. It should be constructive criticism, but some times it’s not… I have seen more heated battles in bugs than I have in conference rooms.

    2) I’m a fan of intergrating bug tracking and version control. I like to be able to draw a correlation between a bug, a check-in, and a build. That is a good thing. No company I’ve worked for has successfully done this and I generally avoid initiatives which involve combining bug tracking with ANYTHING ELSE whether it’s version control, testing, tasks, and/or features. (more below)

    3) Bugzilla is a nightmare to use. Huge proponent of web interfaces here, but I need a robust front end to manage bugs… I need to be able to sort by columns and I don’t want to round trip to a damned web server to do this.

    4) A problem with really successful bug tracking system is that the company starts tracking everything in it and then you end up with a system cluttered with bugs, features, tasks, whatever… I’m a database guy and I want to see everything in a database because it gives me a warm fuzz, but overloading bug tracking systems means more work for everyone and a poor signal to noise ratio.

  9. Alex 21 years ago

    1) Well more than the design and the level of maturity of the programmer to withstand criticism, there’s also the fact that a bug report has the fundamental flaw of any report, it’s as good as it’s source. I don’t think it’s a verifiable law, but my understanding is the dumbest people generate the dumbest bug report, and it just pollute the bug reporting system. There’s also the possibility to change it’s use, some people ask for feature, impossible thing, or to change all the color of the button so they are less gray… It’s a human driven system, and it will always be less than perfect I think.

    2) Still haven’t find a good way? Well try until you succeed! You don’t wanna die as a failure don’t you? What if it’s a requirement to have installed a successful bug reporting system to enter paradise? Yeah you’ll be flying first class to hell. You know the definition of insanity is always doing the same thing and expecting different result.

    3) You want an honest to god gui? Welcome to 2004, your web browser is your new operating system, yeah I know it suck.

    4) Why do you look for a good bug reporting system if you don’t want it to be successful? There’s no bad bug report, just a bunch of blubbering idiot.

    You also have to think of a general amount of expected bug report. I mean you’re not gonna read 10000 bug report by yourself, you’ll probably prefer to check at memory dump, diff it and find similarity in crash condition. The type of software matters too, I mean if you go for the general public, you got to expect that they’ll find weak point, and alert you with vague thing like “this is th3 gay3st softwar3 3v3r”.

  10. Martin Sutherland 21 years ago

    Rands: good point about developers hating criticism. I kind of missed the obvious there. It’s far more comfortable to curse the tracking system, or the person who raised the bug, than one’s own code.

    This is an interesting question, though. Most bug tracking systems impose a layer of impersonal-ness between users and the coders. Issues that could be easily clarified with a five minute chat can end up dragging on for days in a game of fixed/test failed ping-pong. It’s the same layer of insulation that you get when you use email instead of picking up the phone, or walking down the hall to speak to someone in person; or when you get behind the wheel of your car. It’s easier to be angry with someone when you’re not dealing with them in real-time or FTF.

    Although bug reporting systems might be a good way for *tracking* bugs, might this mean that they’re not necessarily the best way to actually get bugs *fixed*?

  11. Ian Seekell 21 years ago

    Alex : Good point with “the dumbest people generate the dumbest bug report, and it just pollute the bug reporting system”. Sure, it sucks to go through 10,000 bug reports like you say, but I still apply the Infinite Monkey Principle. If enough people enter enough bugs, some of them are bound to be meaningful.

    There are always numskulls working with bugs; you can

  12. Ken MacLeod 21 years ago

    I do the same as Charles Kens said: Test-first programming, test-driven development.

    Defects have been rare. Most bugs have been fixed immediately (as in: report received, pass to developer, write test, fix bug) and I recall only a handful that were larger and we chose to schedule them as separate tasks.

  13. abi 21 years ago

    My organisation uses PVCS Tracker, mostly because it’s easy to start using. We have relatively few specialist testers (just disgruntled, misplaced developers), so sophisticated software becomes shelfware and we have to rely on the humans. I leave the results to your imagination.

    A lot of the above comments about developers hating bug tracking systems touch on one major aspect: the conflict between developers and testers. The people who have said that hating the bug tracking system is a proxy for hating the testers are onto something. But that’s part of the job, sadly. As a tester, I have to proceed on the basis that there are bugs and my job is to find them. Any other mindset leads to failure and buggy code.

    Of course, the really good developers are the ones who love their work, so a tester’s mindset is not popular. As one of my colleagues said once, it’s like having to tell you that your baby is ugly. Naturally, the systems that I use to tell developers their baby is ugly are going to be unpopular, however specifically they describe and track the ugliness.

    In other words, there will never be a bug tracking system that’s popular with its users. Perhaps that factor shouldn’t be part of an evaluation of the systems in question.

  14. Ed 21 years ago

    We use (a heavily customized version of) Keystone. Works reasonably well. It’s powerful enough for our needs, yet simple.

    We switched to Keystone from Mantis. Mantis was too limited.

    Other bug tracking systems, like Bugzilla, have too many useless features, and they’re too difficult to install. Looking at FogBUGZ, but will probably stick with Keystone.

  15. jm3 21 years ago

    razor. remedy. buzilla. home-grown web-app that was as usable as bugzilla with less features. excel. text files.

    come to think of it, the bug-tracker that’s worked best was entirely hand-written on 8.5×11 paper, but that was on a two-man XP-style 48-hour-straight coding run. we updated our list with fixes, new bugs, and priority adjustments about every 2 hours.

  16. Phil! 21 years ago

    I’ve gone through several iterations of trying to find a good bug tracking software. I work for a small company and am the primary coder for my problem domain (Windows apps; another programmer deals with the AS/400), so I’m just searching for something that works for me.

    The company uses Track-It for trouble tickets. I tried that, and it’s geared towards, well, helpdesk-type stuff. It didn’t have things I wanted for bug tracking (state of the bug, component it applies to, iterated comments, etc.)

    I tried bugzilla. It works well enough, but as the sole user, I have to track all of the entry and handling of the bugs myself, which ends up being a lot of overhead, especially for smaller stuff. I still use it, but for longer-term enhancements or wishlist ideas.

    Currently, I use tdl ( http://www.rc0.org.uk/tdl/ ) with some standardized syntax to track the area in which the bug occurs. tdl is a command line program, so I can easily add entries as I’m on the phone with my boss (who tends to give me stuff in batches).

    A command line interface to bugzilla would be nice, but I haven’t seen any, and I don’t have the time to write one.