Don't Bother With Microsoft Test Manager 2010
74 comments Posted by Eric Jacobson at Wednesday, October 27, 2010After attempting to use Microsoft Test Manager 2010 for an iteration, we quickly decided not to use it. Here is why.
About 3 years ago we finally managed to stop using HP Quality Center (AKA Test Director). We started managing our test cases and bugs as work items in Microsoft Team Foundation Server (TFS). The brilliance behind this transition was that it gave us the ability to attach our tests and bugs to the iteration’s Features/Stories and Tasks. This meant, as Features move in and out of iterations, the related tests and bugs follow; a beautiful thing! Additional benefits are, 1.) Programmers and BAs can easily review our test cases and write reports based on their execution status and, 2.) no more attempting to synch the TFS Features to Quality Center Requirements…the horror! I totally hated that!
It turns out, Microsoft Test Manager 2010, which sits on Microsoft TFS, took away many of the above TFS benefits and added as much overhead as Quality Center.
Stuff We Didn’t Like About Microsoft Test Manager 2010:
- Before one can write tests, one must set up a Test Suite. Test Suites do not update when the Feature set changes. Thus, one must manually keep one’s Test Suite synched with one’s iteration. To further frustrate, without customization, Test Manager does not let one write test cases for Task work items.
- Test cases in Test Manager follow a different workflow than those of TFS. The result is, nobody on your team can see which of your tests pass or fail unless they open Test Manager, which they probably don’t have a license for (unless they are testers). The reasoning behind this is probably Test Manager’s test cases can have test case run execution history (e.g., TestA could be passed in one build and failed in another build at the same time ). Test case run execution history is actually cool. In fact, it was one of the biggest motivators for us to try Test Manager. However, we were hoping Test Manager would trickle down the results to TFS so the whole team could benefit.
- One of the most annoying parts of our Test Manager trial may have been its usability and performance. The screens frequently hung, causing some testers to force quit and re-launch. The navigation was also awkward. It took too many clicks to get anything done.
- To update the execution status of Test Manager tests, one must go through the ridiculous test case executor. This is the thing that shows you each test case step and asks you for a pass/fail result. I can’t imagine anyone actually testing like this. Quality Center had something similar but provided an alternate method of updating execution status (i.e., one could bulk update via a grid).
- Our other gripe about updating the execution status of Test Manager tests is that the test case “summary” does not show. Most testers like to write their test cases as fragments, using the test case work item’s free-form summary tab. The summary tab is preferred over the grid, which forces tests into individual steps with expected results. The big joke is, if you write your tests in the summary tab, it is not possible to see them while running the silly test case executor. So you are presented with a blank test step and asked if it passes or fails.
Stuff We Think We Like About Microsoft Test Manager 2010:
There are two things some project teams are planning on using Test Manager for in the near future:
- Calling our CodedUI test methods and passing in parameters. According the idealistic demo I saw at the Stareast Microsoft booth, this allows manual testers to write/execute automated tests without coding.
- Using test case run execution for regression testing.
Conclusion:
I guess TFS's simple work item model fits our needs for flexible lightweight test documentation with little administrative overhead. Maybe someone can convince me otherwise.
Many testers have chosen to make their jobs stressful by taking on more responsibilities than they should, obscuring their skills with those of others on their teams. Choosing to make your job less stressful will not only help you enjoy testing, it will also allow you to focus on testing, and improve your standing as a test leader. Here are some things I’ve learned over the years…
- When people ask me “Did you QA Certify this for production?”, I remind them the question of when to ship is a business decision, but I can tell them much of what they need to know (e.g., how it currently works under certain conditions, what bugs exist) to make that business decision.
- When I hear users complain about the product not working for them due to the way it was designed, I feel empathy for the users….then I remind myself that I never tell BAs/devs how to build the product or what it should do.
- When I’m faced with really scary technical things to test, I turn to my team. I only have to look stupid to the first person I talk to, because by the time I get to the second person, at least I have what I learned from the first. As I continue to share my test ideas, they gradually change from lame to sophisticated. Soon, I realize everybody else on my team was just as confused as I was.
- Crunch time. I expect it. I chillax at the beginning of an iteration and work harder/longer towards the end. I maintain my work/life balance by keeping my personal calendar free right before and after a production release. Working late with other team members is often just as fun as spending a quiet evening at home with my wife (don’t worry, she doesn’t read my blog).
- Too much to test! Too little time! This one still stresses me out on occasion. But when I’m thinking rationally, I pose the question to my BAs, “I have 2 days left, would you prefer I test these new features or focus on regression testing?”. It trains them to respect my schedule and understand that it is finite. It also shows that I respect their business sense and value their opinion.
- Your product went live and the quality sucks. Okay, you can feel somewhat guilty...along with your devs and BAs. But remember, you didn’t code or design it. Quality can only be added by the programmers (e.g., if you have no code, you have no quality.). If it sucks now, just think about how much it would have sucked before those 471 bugs you caught!
What things do you do to make testing less stressful and maintain your sanity?
"QA"
It’s like there aren’t enough testing-related terms out there so people have to just use the word “QA” to talk about anything in the testing domain.
- “Ask QA to test this.” (It’s a group of people)
- “QA is down again!” (It’s a test environment)
- “We’ll need someone to QA this before it goes out.” (It’s an official action)
- “Is it in QA”? (You’re either asking if someone is testing it or you’re asking if code resides in a specific environment)
By speaking properly, ourselves, perhaps we can change this.
- “Ask the testers to test this.”
- “Environment [insert an environment name, don't name it "QA"] is down again!”
- “We’ll need someone to test this before it goes out.”
- "Is it being tested by the testers?"
Subscribe to:
Posts (Atom)
RSS
