—Lisa
We usually recommend experimenting with different tools, using each one for a few iterations, but this is trickier with bug-tracking systems, because you need to port all of the bugs that are in one system to the new one that you’re trying on for size. Spend some time thinking about what you need in a DTS, what purposes it will serve, and evaluate alternatives judiciously.
Lisa’s Story
My team used a web-based DTS that was basically imposed upon it by management. We found it somewhat cumbersome to use, lacking in basic features such as time-stamping updates to the bug reports, and we chafed at the license restrictions. We testers were especially frustrated by the fact that our license limited us to three concurrent users, so sessions were set to time out quickly.
The team set aside time to evaluate different DTS alternatives. At first, the selection seemed mind-boggling. However, we couldn’t find one tool that met all our requirements. Every tool seemed to be missing something important, or we heard negative reports from people who had used the tool. We were concerned about the effort needed to convert the existing bug database into a new system.
The issue was forced when our DTS actually crashed. We had stopped paying for support a couple of years earlier, but the system administrator decided to see what enhancements the vendor had made in the tool. He found that a lot of shortcomings we had experienced had been addressed. For example, all updates were now time stamped. A client application was available that wasn’t subject to session timeouts and had enhanced features that were particularly valuable to the testers.
By going with our existing tool and paying for the upgrade and maintenance, plus a license allowing more concurrent users, we got help with converting our existing data to the new version and got a working system easily and at a low cost. A bonus was that our customers weren’t faced with having to learn a new system.
Sometimes the best tool is the one you already have if you just look to see how it has improved!
—Lisa
As with all your tool searches, look to others in your community, such as user groups and mailing lists, for recommendations. Define your criteria before you start looking, and experiment as much as you can. If you choose the wrong tool, cut your losses and start researching alternatives.
Keep Your Focus
Decisions about reporting and tracking defects are important, but don’t lose track of your main target. You want to deliver the best quality product you can, and you want to deliver value to the business in a timely manner. Projects succeed when people are allowed to do their best work. Concentrate on improving communication and building collaboration. If you encounter a lot of defects, investigate the source of the problem. If you need a DTS to do that, use it. If your team works better by documenting defects in executable tests and fixing them right away, do that. If some combination enables you continually improve, go with it. The main thing to remember is that it has to work for your whole team.
Chapter 18, “Coding and Testing,” covers alternatives and shows you different ways to attack your bug problems.
Defect tracking is one of the typical quality processes that generate the most questions and controversy in agile testing. Another big source of confusion is whether agile projects need documents such as test plans or traceability matrices. Let’s consider that next.
Test Planning
Traditional phased software methodologies stress the importance of test plans as part of the overall documentation needs. They’re intended to outline the objectives, scope, approach, and focus of the software testing effort for stakeholders. The completed document is intended to help people outside the test group understand the “why” and “how” of product validation. In this section, we look at test plans and other aspects of preparing and tracking the testing effort for an agile project.
Test Strategy vs. Test Planning
In an agile project, teams don’t rely on heavy documentation to communicate what the testers need to do. Testers work hand in hand with the rest of the team so that the testing efforts are visible to all in the form of task cards. So the question often put to us is, “Is there still a need for test plans?” To answer that question, let’s first take a look at the difference between a test plan and a test strategy or approach.
The more information that is contained in a document, the less likely it is that someone is going to read it all. Consider what information is really necessary for the stakeholders. Think about how often it is used and what it is used for.
We like to think of a test strategy as a static document that seldom changes, while a test plan is created new and is specific to each new project.
Test Strategy
A strategy is a long-term plan of action, the key word being “long-term.” If your organization wants documentation about your overall test approach to projects, consider taking this information and putting it in a static document that doesn’t change much over time. There is a lot of information that is not project specific and can be extracted into a Test Strategy or Test Approach document.
This document can then be used as a reference and needs to be updated only if processes change. A test strategy document can be used to give new employees a high-level understanding of how your test processes work.
Janet’s Story
I have had success with this approach at several organizations. Processes that were common to all projects were captured into one document. Using this format answered most compliance requirements. Some of the topics that were covered were:
• Testing Practices
• Story Testing
• Solution Verification Testing
• User Acceptance Testing
• Exploratory Testing
• Load and Performance Testing
• Test Automation
• Test Results
• Defect Tracking Process
• Test Tools
• Test Environments
—Janet
Test Plan
The power of planning is to identify possible issues and dependencies, to bring risks to the surface to be talked about and to be addressed, and to think about the big picture. Test planning is no different. A team should think about risks and dependencies and the big picture for each project before it starts.
Whether your team decides to create a test plan document or not, the planning should be done. Each project is different, so don’t expect that the same solution will fit all.
In Chapter 15, “Tester Activities in Release or Theme Planning,” we show examples and discuss alternatives you can use when you are planning the release.
Sometimes our customers insist on a test plan document. If you’re contracting to develop an application, a test plan might be part of a set of deliverables that also include items such as a requirements document and a design document.
Talk of test plans often leads to talk of traceability. Did someone execute all planned testing of the desired behavior on the delivered code? How do requirements and test plans relate to the actual testing and final functionality?
Traceability
In traditional projects, we used to need traceability matrices to determine whether we had actually tested all of the requirements. If a requirement changed, we needed to know that we had changed the appropriate test cases. With very large requirements documents, this was the only way that a test team knew it had good coverage.
In an agile project, we don’t have those restrictions. We build functionality in tiny, well-defined steps. We work with the team closely and know when something changes. If the programmers work test-first, we know there are unit tests for all of the small chunks of work. We can then collaborate with the customer to define acceptance tests. We test each story as the programmer works on it, so we know that nothing goes untested.