Don't Bother With Microsoft Test Manager 2010
Posted by Eric Jacobson at Wednesday, October 27, 2010After attempting to use Microsoft Test Manager 2010 for an iteration, we quickly decided not to use it. Here is why.
About 3 years ago we finally managed to stop using HP Quality Center (AKA Test Director). We started managing our test cases and bugs as work items in Microsoft Team Foundation Server (TFS). The brilliance behind this transition was that it gave us the ability to attach our tests and bugs to the iteration’s Features/Stories and Tasks. This meant, as Features move in and out of iterations, the related tests and bugs follow; a beautiful thing! Additional benefits are, 1.) Programmers and BAs can easily review our test cases and write reports based on their execution status and, 2.) no more attempting to synch the TFS Features to Quality Center Requirements…the horror! I totally hated that!
It turns out, Microsoft Test Manager 2010, which sits on Microsoft TFS, took away many of the above TFS benefits and added as much overhead as Quality Center.
Stuff We Didn’t Like About Microsoft Test Manager 2010:
- Before one can write tests, one must set up a Test Suite. Test Suites do not update when the Feature set changes. Thus, one must manually keep one’s Test Suite synched with one’s iteration. To further frustrate, without customization, Test Manager does not let one write test cases for Task work items.
- Test cases in Test Manager follow a different workflow than those of TFS. The result is, nobody on your team can see which of your tests pass or fail unless they open Test Manager, which they probably don’t have a license for (unless they are testers). The reasoning behind this is probably Test Manager’s test cases can have test case run execution history (e.g., TestA could be passed in one build and failed in another build at the same time ). Test case run execution history is actually cool. In fact, it was one of the biggest motivators for us to try Test Manager. However, we were hoping Test Manager would trickle down the results to TFS so the whole team could benefit.
- One of the most annoying parts of our Test Manager trial may have been its usability and performance. The screens frequently hung, causing some testers to force quit and re-launch. The navigation was also awkward. It took too many clicks to get anything done.
- To update the execution status of Test Manager tests, one must go through the ridiculous test case executor. This is the thing that shows you each test case step and asks you for a pass/fail result. I can’t imagine anyone actually testing like this. Quality Center had something similar but provided an alternate method of updating execution status (i.e., one could bulk update via a grid).
- Our other gripe about updating the execution status of Test Manager tests is that the test case “summary” does not show. Most testers like to write their test cases as fragments, using the test case work item’s free-form summary tab. The summary tab is preferred over the grid, which forces tests into individual steps with expected results. The big joke is, if you write your tests in the summary tab, it is not possible to see them while running the silly test case executor. So you are presented with a blank test step and asked if it passes or fails.
Stuff We Think We Like About Microsoft Test Manager 2010:
There are two things some project teams are planning on using Test Manager for in the near future:
- Calling our CodedUI test methods and passing in parameters. According the idealistic demo I saw at the Stareast Microsoft booth, this allows manual testers to write/execute automated tests without coding.
- Using test case run execution for regression testing.
Conclusion:
I guess TFS's simple work item model fits our needs for flexible lightweight test documentation with little administrative overhead. Maybe someone can convince me otherwise.
74 comments:
Subscribe to:
Post Comments (Atom)
RSS
I feel pretty much the same way. Thanks for the analysis!
Only after how it looks I wouldn't use such a tool. At lest is free?
I was looking forward to using MTLM but not so much after reading this. Did the option to link the test to an existing bug provide any help to those without access to test manager?
Hi Matt. Not really, because the test execution status can only be viewed from within Test Manager. I may not understand your question, sorry.
Hi Eric - I am sorry to learn that your experience with MTM has not been positive. I am the Program Manager for MTM and wanted to help clarify a few things on your issue list.
1. Test suites can indeed be dynamic and pick up all test cases in your iteration. Have you tried the query based test suite? Or the requirement based suite?
2. To track test pass rates, you can create reports and host them on the sharepoint dashboard
3. There have been quite a few crashes and hangs that were reported - we have fixed 22 such issues in the recently released roll up QFE: Download is publicly available at http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=31858 and details are at http://support.microsoft.com/kb/2387011
4. Bulk update without opening MTR is indeed not currently supported. But I would like to understand a little more about your scenario of why you would not want to see any steps in MTR.
5. Currently, other fields in test cases do not show up in MTR. We are currently looking at a way to solve this problem for future versions
I am glad you like the Coded UI and using the runner for regression testing. Since you mention you use TFS for work item tracking, there are some pretty neat integration scenarios that can light up with MTM - test impact analysis, CI tests etc.
I'll be happy to chat with you in detail about your feedback. Do ping me on http://twitter.com/anutthara for any qs or comments.
- Anu
Anu, thanks for taking the time to respond to my post. It's nice to know you're out there trying to collect feedback about your product. It sounds like you're addressing several of our gripes.
However, on your first point, I don't think it actually works the way you describe. We did try query based test suites, and were surprised to discover they are not dynamic. One must actually manually execute the query to rebuild the test suite to see the changes. I didn't experience this myself, other testers on my team described it that way. So let me know if I'm wrong.
Eric - the deal with query based suites is that refreshing all of these all the time will be a perf hit. So, every 15 mins, a background server job reruns the queries to make sure they are up to date. Your colleagues probably checked within the 15 min time window and hence, they would have had to manually rerun the query.
You can also just hit the "refresh" button while viewing a Query-based suite in MTM, that will pick up any new tests that match the query before the 15 minute timer performs the refresh. (The button is at the top of the right-hand panel and not very large. It's easy to overlook.)
I'm finding the same performance issues with MTM2010 and the biggest frustration is that we can't just edit test case steps from VS2010. I just updated with the VS2010 service pack 1 which I think has the KB fixes mentioned, we'll see. Having the cases open up in a new GUI with limited features pretty much ruins the whole thing, we haven't even gotten as far as running a pass yet, just entering new cases is a chore.
Is there a way to generate a report that shows the linkage between a requirement to a test case to the test case results? That is what I really need so that I can generate a tracability matrix.
Nice discussion on a great topic,pretty informative..
We wanted to use this tool but after reding this report, I will have to rethink.
Hi Anu,
Can you please point me to :
To track test pass rates from MTM, create reports and host them on the sharepoint dashboard
I'm trying to learn Test Manager 2010 - and I agree with at least one point made here --- I see a trend toward many more clicks to get anything done in MS products. (e.g., I'm at least 2 to 3 times faster in Access 2003 vs the newer 2007 or 2010 versions - for this reason alone...). I'm not convinced there are many real users in the vicinity of MS development... but a lot of 'theorists' about what is more efficient... :o)
We are interested in moving to MTM, but are stopped by the fact that the shared steps are not recursive. In Quality Center we use 'template' tests to write tests that are going to be reused heavily. So, when the related functionality changes, we can just update the template tests and all tests using this template are updated dynamically. In MTM, you can only call a shared step, but a shared step can not call another shared step. We would lose a significant amount of data if we were to migrate to MTM.
Also, the workflow to use MTM is extremely disjointed and does not seem to follow how testers perform their work. Who did MS have to beta test this products? Developers?
Eric, did you do any formal training say from Microsoft or their vendor before using this tool. Just curious because I need to make a recommendation to my QA Manager on a Tool to purchase. Thanks.
This is one of the worse cases of bloatware I have ever seen. This software is clumsy, unstable and almost unusable. If you would like to edit a test, you must abort the running of the current script. The edit facility is, again, so slow that it will take a few seconds when you click on each step before you edit it. Oh an don't rely on any handy features such as "Undo", MTM does not have these.
At work we have been forced into this 'upgrade' and if I'm perfectly honest I believe it will decrease the amount of test coverage we will have on each product significantly enough to require the hiring of more staff.
Using MTM is a little bit like being relegated to the early learning centre. If this is the direction the test industry is heading, it might be a wise idea to consider re-training and jumping ship, NOW.
Anonymous,
"Eric, did you do any formal training say from Microsoft or their vendor before using this tool. Just curious because I need to make a recommendation to my QA Manager on a Tool to purchase. Thanks"
No. We read a few MS blog posts about it. I would not purchase this tool. Instead, just piggyback off whatever tool your programmers are already using to track their work. Save your money and take a Rapid Software Testing class.
Hi Eric,
I am trying to create a test case work Item in m TFS.. I dont have MTM. But i am not able to create steps for it. When I click on the Open for Edit button , I am getting the message " unable to open this test Case. please Check that Microsoft Test manager is correctly Installed". Is is possible to create a WI-test case without installing MTM in Visual Studio TFS 2010
Thanks for the post Eric. An additional side note, I have VS2010 Ultimate installed, our team is using M$s new hosted TFS service. When I try to use the Lab features I'm unable to. I receive an error stating, "You are connected to a newer version of Team Foundation Server from an older version of Microsoft Test Manager..." I thought this was prity odd...
Thanks for the post Eric. An additional side note, I have VS2010 Ultimate installed, our team is using M$s new hosted TFS service. When I try to use the Lab features I'm unable to. I receive an error stating, "You are connected to a newer version of Team Foundation Server from an older version of Microsoft Test Manager..." I thought this was prity odd...
QUESTION: Anyone found an alternative to bulk insert test results to TFS?
If yes then please comment this blog post with a link!
I've had similar mixed feelings but I am still trying to find a way to make TFS work for test case (and result) management for my team.
My blocking issue: no way to bulk insert results from Excel, public API, or other method. Even after over a year I haven't seen any codeplex or 3rd party tools to support what Excel can easily handle. Makes me want to try tinkering with the TFS DB tables, but that is often the path to the dark side!
User Scenarios
Scenario#1: Quickly log results for simple tests.
Test cases written with titles that are sufficient to run without reading steps (Feature: state-action-expected). If MTM had a way to change state field in the Test Suite details pane (Plan-Contents-Test suite-test list) I could update my results inline and my core need would be met. If I have a failure or need traces of test execution I'll run the test using existing MTM test runner, but that is the exception rather than the rule for most test cases. My happy path is blocked for tests that are trivial to run!
Scenario#2: Backfill completed test results.
Test pass partially done with 100+ (or 500+) manual tests run but need to be logged in TFS. I cannot select different config from my system then just mark them as Pass. I have to walk through each in MTM runner.
Scenario#3: Non-testers logging results!
Project Managers, Developers, and other team members (possibly even selected internal customers) should be able to contribute. I would love to have them directly log their results. If that isn't possible then see scenario#2.
{rant}Even MTM still does not support bulk result logging after over a year. Even so licensing would block non-testers from logging results. (Please remember only Microsoft and orgs with site license have all product team team staff running VS2010 Ultimate. Even then most Program Managers, marketing, and other team staff should not have to use VS or MTM to log results if they are invited to help run tests.)
{/rant}
FYI:
Minor shortcut to log a result without running each step: With MTM test runner open click the top-right test status icon to drop-down and select the desired status for the entire test. Then move to next test.
Related discussion "Directly log the test result...": http://social.msdn.microsoft.com/Forums/en-US/vsmantest/thread/dca63160-9c4e-4626-9629-e8f94cf20f5a/
If Anyone needs this feature represented as user stories let me know.
-Jai - For reports, please see http://social.technet.microsoft.com/wiki/contents/articles/how-to-create-reports-for-tfs-2010-test-results.aspx
-Anonymous - Editing of test cases while running the test and the related performance issues are things that we are trying to address in the next release – you should be able to have an improved experience there.
-ZephanS - The feature of bulk marking of tests, without opening the test runner, will be there in the next release.
-Kiruthiga - Being able to edit test cases without opening them in test manager is in our backlog
-For other issues please feel free to add your suggestions/comments at http://visualstudio.uservoice.com/forums/121579-visual-studio
Minor update. I heard direct result entry may be supported in Dev11. I have a backlog work item to get Dev11 VM up and this will be my first test. Please reply comment if you confirm or find other solutions.
Hi Nivedita Bawa,
Could u help me to use MTM in my project.I am unable to create connect it with TFS.
Thanks in Advance.
Shankar
Zephan S - Yes bulk marking of tests without opening test runner will be supported in the next release.
Shankar - please feel free to contact me at bawanivedita at gmail dot com with your problem details.
The steps window that compromises between those wanting to see the list of steps readably by default, and those wanting to keep the original narrowest skinniest possible steps window - by exanding the steps window after you mouse click and down arrow three times but then disappears the first two of the steps once you do that, is like an endless horrible insult to the user.
There are some fixes that we fixed in the GDR http://connect.microsoft.com/continue.aspx?pageType=2®Type=2&cru=%2fVisualStudio%2fDownloads%2fDownloadDetails.aspx%3fDownloadID%3d39070&cu=
There are many more improvements in the test steps grid in the next release.
I am a completely new user of MTM. One of the first things I have come across is that apparently I cannot re-order test cases in a test suite which
has been created by adding a user story to my test plan.
I googled this and found a MS help page which specifically stated such test cases could not be re-ordered
See the first sentence here:
http://msdn.microsoft.com/en-us/library/dd997699.aspx
Surely the ability to re-order tests is a very basic requirement. Also it seems I cannot add a new test case within the current order, only at the end, or perhaps have I missed something fundamental here ?
Andy - the current support for ordering is same as mentioned in your link. But ordering in requirements based suites, etc. are already on our backlog, thanks for bringing it up!
Please see http://support.microsoft.com/kb/2656136 for some test steps related fixes.
When is the next patch release for MTM that has the fixes discussed??
How do you update test step status from an associated Coded UI script? I want my script to do this instead of making me go back into the test steps to change the statuses.
Thanks!
How do you update the MTM test step status from a linked Coded UI script? I don't want to manually update my test step status after running automation.
Few YEARS passed, but tehy still not fixed a buf with MTM window resizing.
Sounds stupid, but a tool for managing testing/quality have worst quality of all MS products.
Recently started using this myself and am completely frustrated with the tool.
The Test Case Designer
1. Parameters cannot be reordered in the grid. I can select them and it looks like I can drag them but they don't actually move. Of course the order of the params is last in last on the list. If you have a lot of params then you want to group them so that you can see the relevant data together
2. When I rename a parameter it deletes all the data that I associated with that parameter, why not just change the pointer name and leave the data? Why would I do this? As I am developing my cases from the user stories I realize that a variable needs to be renamed to reflect a change in its scope.
3. The grid for the steps and results is barely functional. I find myself going out to Excel and creating my steps – then one by one importing the cells back into the test designer. Yes, this is a horribly inefficient method but it is better than trying to work in the cells themselves.
4. The steps and parameter grid has a very odd visual defect that makes it so you see no more than 1 step and none of the parameters. I have found that I have to set the width of the screen to about half my monitor width and then slowly drag the height until the grid reforms correctly. My screen isn’t that big (17 inch).
5. Performance is horrible.
6. Since I often write cases that are similar I would like the opportunity to have more than one test case up at a time. Just as I can do with many other applications – Notepad++, Word, Excel, and many other Windows based applications.
7. feature request: I need my summary to be there while I am testing, as a pinnable window along with my user story – also as a pinnable window.
8. Feature request: integrate with Visio. I use Visio to diagram my complex workflows and it would be nice to use it here.
So, this is the tool I have, it appears to be a beta release but it is what we are using organizationally so for the time being we are stuck with it. MS, you really need to put better effort into this and bring it up to releasable standards.
I look forward to your next significant improvement
@Anonymous:
The fixes are available in the VS11 beta release. See http://blogs.msdn.com/b/visualstudioalm/archive/2012/02/29/what-s-new-for-microsoft-test-manager-in-visual-studio-11-beta.aspx
Why do you want to mark the steps of an automated test individually? Isnt marking the test case enough?
Could you elaborate more on the window resizing bug? It would be helpful if you can file all such bugs at http://connect.microsoft.com/ so that we can fix them
I read your page four months ago when we started converting. Well, I agree with everything you say now. This is simply awful software. What is worse is that it is not because of bugginess or awkward usage - although there is plenty of that - but simply because it is architected so very very poorly. It really feels like someone suddenly realized they forgot about QA (again) and slapped something together. The interface doesn't even follow windows standards!!!
Actively looking for replacements now ....
In our organization, the requirements are entered in the TFS with work item type = issue and issue type = business enhancement. How can I create a test suite in MTM based on the requirement in TFS? I mean to ask, is there a way by which I can create a suite that has all the requirements imported from TFS and then create the test cases corresponds to each TFS requirement? If yes, then can you tell me the step by step process for it?
@Anonymous - marking test steps through coded ui script is not possible, but the test case should get marked when you run it in automated mode. There is no 1:1 correspondence between the steps and the coded ui script, hence steps dont get marked.
@Mike - Could you please elaborate a bit on your dissatisfaction? It would be helful to know in specific, the top issues that y ou are facing.
@Deepak - This can be done as described here http://msdn.microsoft.com/en-us/library/dd286578.aspx
Nevita, can i use this to test application develop with Visual FoxPro?
spionst @Anonymous - My guess is that it should work for Visual Foxpro apps. I would request you to do a quick POC to verify.
Hi, I am a new user and just started using UI coded test (record functionality - web application and run for regression purposes) using MS visual studio 2010. My question is can I schedule my Test cases (for example I have recored 20 test cases under a project. Can I set up a schedule to run test for a certain time.
We've been using MTM2010 recently and I was wondering if anyone here knows if there is a link between the parameters and configurations. In my opinion it is logical that an expected result depends on a used configuration. Now it is only possible to use configuration or iterations but not a combination of the two.
Is there a way to edit testcases while executing them manually(step by step) in testmanager?
Thanks,
Naveen
Great post, excelent discussion!
We have some test cases (quite simple in certain form) but that requires a lot of pre-conditions to set up the app to a known state to run that test.
How would we write that test cases ?
Any advice?
(sorry, my english is so bad)
Victor
I just spent over an hour rapid fire clicking to do what should be 1 click: setting a pass result on several hundred tests. The more data you enter in the test steps, the more cumbersome this is, since each test step is click a dropdown and select pass. Tip: hit Ctrl-A then Win-Ctrl-Q (ever heard of the Win key being used as a keyboard shortcut) - pressing these two key sequences will pass every step of the test - then click a hyperlink (either click it or tab 3 times and hit enter not space) - then presss the spacebar to get to the next test (I wonder how many designers fought tooth and nail not to include the ability to press space rather than click the button).
I found you can copy and paste the list of tests into Excel - you cant copy it into notepad - but dont use Ctrl-A when you copy the list of tests or it wont copy - use shift arrow to select the tests in the list.
The designers of this product hate their users waaaay too much. Please note I posted a useful comment prior to this.
I do not believe that the designers 'hate' the users, but I do have to say that this product does not seem to be very well thought out. Perhaps MS works in a way different from most testers in the world, but some of the workflows are backwards. Simple stuff, like being able to suspend a test run is met with questions of "why would you ever want to do that?" Not making test plans and suites into work items is simply criminal - there are three people in my organization who can view them, because they are using VS/Test or VS/Ultimate. Why is there no Undo? Why can I format the History when I cannot format the Summary or the Steps? Just standing in front of a whiteboard drawing lines between sticky notes would have solved a lot of problems before one line of code was written - but as I say, MS must have a test organization that works differently than most others on the planet...
I have accidently deleted test cases with results from test suite. Please let me know if there is any way to recover the deleted test suite along with results and screenshots??
I worked in MS as a tester and I know they are the same as any tester in the word. But this Tool (MTM) was designed by people with no experience in testing or in Project Management. It seems to me that MTM was a Diploma Project in a College somewhere and MS just hired the whole class and asked them to translate the their project from JAVA to C#.
It is so un-usable that I feel pity for the people which have to use it.
It is so non-performant that I fill pity for the Project Managers relaying on such a tool.
One of my beefs in MS testing activity it was the product usability and clearly it still a problem up there in Seattle subs. When will MS learn from Apple about usability?
Is there a way to modify the Test Results, in MTM, to add new fields to the "Summary" and "Details" sections? I found an article for modifying the Test WI template, but not the Test Results template.
http://blogs.msdn.com/b/vstsqualitytools/archive/2011/09/21/customize-your-test-case-work-item-template-to-enable-query-on-test-suites-test-plans-test-categories-and-execution-statistics.aspx
I am using MTM 2012 and I like it. My only beef is when I look at the results, the graph is shows the list of suites on the x axis. The list only shows the last folder from the suite. I would like to show that folder, but also have the ability to show the folders above it. Because my testcases are broken down by module and the modules within UAT, System Test and Performance Folders, so the module names are the same and I can't tell which tests are graphed.
I am using MTM2010 and trying to automate the interaction between web client server apps by recording functions in both of them. Both read the same data from a database and I need to automate that one of them picks up and sends the same information as the other. Is there some way to do this. Every time I try, I can do recorded tests for either application but when I try to move back and forth MTM stops recording my session.
We just decided to start using MTM today at work. I've been struggling to even get this thing to run without crashing. It crashes when connecting to TFS, when adding a New test case, and in several other places. When it's not crashing, it is hard to navigate and sluggish. Why is it so hard to simply add steps to a test? I hope we do not decide to stick with this.
Our company has some teams using QC and some using MTM, with the push to move all to MTM. MTM continues to frustrate me every time I try to use it. Examples:
1. Not able to pause and resume multiple tests (mark tests as Not Completed and then resume later).
2. Not able to view Test Results when looking at a Test Case (in MTM or TFS). You have to use MTM and KNOW what Test Plan and Suite the Test Case belongs to!
3. Not able to see what Test Plan and Test Suites a Test Case belongs to (from both MTM and TFS).
There should be an easy way to view a Test Case and see all its related components (links to Test Plans, Suites, Test Run results, etc)!!!
I hope someone can clarify this for me.
Let's say I have two test suites, Suite 1 and Suite 2, and a single testcase, TC 1.
When I run TC 1 in Suite 1, and it changes status to passed or failed, I want the status of TC 1 in Suite 2 to change as well.
Doing the exact same testcase twice in a row, just for the suites to show what I already know, seems a waste of time, and a poor handling of the testcases by MTM.
What are your thoughts on this? Am I wrong in expecting the same TC to update across suites? Is there a workaround/setting I haven't caught on to?
Though there seems to be more cons than pro in this tool, I still think it would be worth the try.
Eric, thanks for the blog, it has definitely sparked some thoughts and inline with our experience. Given the responses, I am posting my comments/issues hoping to get a response.
We have been using this tool since 2010, mostly for documentation and reference purpose. This time we are starting to do test automation and hoping to leverage the tool in producing results and reports.
Our current challenge is the way the Test Plan Progress Report displays data from the test plan in MTM.
1. We have requirement based suites in our MTM test plan, where each requirement has both manual and automated test cases. When a test build is ran, the manual test cases get converted to active state, eventhough they have already been ran and marked pass/fail.
2. The above scenario throws off our test plan progress report. The report indicates that the manual cases have never been ran.
3. The query based suites in a test plan, works for the above problem, there will be no way to see the requirement-testcase link view.
4. Another problem with MTM is that, if there is a duplicate test case in 2 suites in a test plan, the result is reported twice with one run. This throws off the report again. (same problem as Michael Niebuhr above)
Overall, the fact that testers are having to spend a lot of time trying to figureout a best way to organize a test plan causes an impediment in the process. Especially, in scrum where all test task hours are taken into account.
This is a very informative read about a software testing application. The facts that you have shared are indeed helpful. Thank you for posting.
(part#1 of 6)
Hi Eric,
I am the current product owner for Microsoft Test Manager and I appreciate your candid analysis of "Microsoft Test Manager (MTM) 2010" clearly listing the top
shortcomings in the product - both in your blog post and in the comments section.
I am summarizing my responses/updates on these issues below.
As you can see from below, we have addressed the top pain points with the product, both in MTM 2012 (RTM, Update#1, Update#2, etc.) and MTM 2013 (RTM). We want to
address the remaining items as well in the MTM 2013 Updates train.
An humble request for you - I would really appreciate if you can evaluate the latest version of MTM and then share your new findings either as an update to the
current blog post or create a new post, so that users of your blog can get the updated information.
You can always reach me for any direct feedback or 1:1 discussions at "ravishan at microsoft dot com"
I. CONSOLIDATED RESPONSES FOR ISSUES RAISED THAT HAVE BEEN FIXED
----------------------------------------------------------------
Dynamic nature of test suites:
------------------------------
Requirements based suites (RBS) and Query based suites (QBS) are dynamic in nature, and are automatically refreshed when existing or new test cases are linked to
requirements or match the queries. There is a background job that runs every 15 mins to keep these items in sync. The user can always hit the Refresh button
manually to get an updated set on-demand.
See the following for more info:
How to create tests for PBIs/Userstories/Requirements - http://msdn.microsoft.com/en-us/library/dd286676(v=vs.110).aspx
How to manage test cases using QBS & RBS - http://msdn.microsoft.com/en-us/library/dd286738(v=vs.110).aspx
Viewing test results outside of MTM:
-----------------------------------
Basic reporting information is available in MTM and rich, customizable reports are available as part of TFS warehouse/Analysis cube.
See the following for more info:
http://msdn.microsoft.com/en-us/library/bb385901.aspx#WhatsNewTestingMTMReports
http://blogs.msdn.com/b/vstsqualitytools/archive/2011/10/14/test-case-management-tcm-reporting-frequently-asked-questions-part-1.aspx
http://blogs.msdn.com/b/vstsqualitytools/archive/2011/10/14/test-case-management-tcm-reporting-frequently-asked-questions-part-2.aspx
I acknowledge that there is much more to do in this area and we plan to address them by providing views for recent runs/recent results/run history for a specific
test case etc.
(part#2 of 6)
MTM Usability:
-------------
I acknowledge that getting used to MTM takes a bit of time. That said, we have enabled a lightweight & easy to use web UI that can be used to perform most of the
test management capabilities using a browser.
See the following for more info:
http://msdn.microsoft.com/en-us/library/dd380763.aspx
http://blogs.msdn.com/b/visualstudioalm/archive/2013/02/12/light-weight-browser-based-test-management-and-execution.aspx
MTM Performance:
---------------
Performance of MTM 2010 was an issue and this was a major focus for MTM 2012 RTM release.
See the following for more info:
http://blogs.msdn.com/b/bharry/archive/2011/08/02/a-comment-on-performance.aspx
http://blogs.msdn.com/b/bharry/archive/2012/07/05/test-professional-performance-improvements.aspx
Bulk mark of tests:
------------------
Both, the MTM client and the Web UI, support marking of test case result without opening the test runner and walking through the test steps.
See the following for more info:
http://blogs.msdn.com/b/visualstudioalm/archive/2013/06/05/microsoft-test-manager-customization-of-test-result-fields-and-marking-test-results-as-na.aspx
Pre-conditions/Viewing Summary Info while testing:
-------------------------------------------------
This can be achieved by adding text (with rich formatting) to the summary tab of the test case. This information is shown, with rich formatting retained, in the
test runner in a read only view, below the test case title. Additionally the tester can add his comments to the test case or particular steps during execution.
Edit tests during execution:
---------------------------
Ability to edit test cases while executing manual tests is now possible with both MTM and Web.
See the following for more info:
http://blogs.msdn.com/b/visualstudioalm/archive/2012/11/07/edit-test-case-properties-directly-from-the-test-runner-of-microsoft-test-manager.aspx
http://www.visualstudio.com/en-us/news/2013-mar-4-vso
(part#6 of 6)
List of test suites that a test case belongs to:
-----------------------------------------------
This is something in our backlog and fairly high in stack rank.
Workaround for the above functionality (programmatic):
http://blogs.msdn.com/b/arvindkrish/archive/2012/10/22/find-all-test-suites-test-plans-a-test-case-belongs-to.aspx
The following uservoice item tracks this request:
http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2038943-show-me-all-test-suites-test-plans-a-test-case-be
III. OTHER USEFUL ENHANCEMENTS
------------------------------
Support for multiline and rich text in test cases:
http://msdn.microsoft.com/en-us/library/bb385901.aspx#WhatsNewTestingMTMMultipleLines
Exploratory Testing:
http://msdn.microsoft.com/en-us/library/hh191621.aspx
Clone of Test Plans:
http://blogs.msdn.com/b/visualstudioalm/archive/2013/05/03/cloning-test-plans-using-microsoft-test-manager.aspx
Seamless switching between MTM and web:
http://www.visualstudio.com/en-us/news/2013-jun-19-vso
Automatic updates:
http://blogs.msdn.com/b/visualstudioalm/archive/2012/10/10/managing-automatic-updates-for-microsoft-test-manager-and-microsoft-feedback-client.aspx
Thanks
-Ravi
"ravishan at microsoft dot com"
Query results in TFS 2010 differ from Excel Query Results Using Team Plugin
Below are the steps
•In TFS 2010, create and save a query to retrieve requirement type work items that have a single test case linked to both requirment work items. I am using the Work Items and Direct Link type of Query. (Be sure to have 1 test case linked to multiple requirements)
Team Project = @Project
Work Item Type = Requirement
And linked work items that match the query below:
Work Item Type = Test Case
•Select Open in Microsoft Office, then select open query in Microsoft Excel 2010
•Count the number of rows in Excel and Compare to items found in TFS 2010
•In TFS 2010, there is 1 Test Case assigned to multiple Requirements. The test case appears under each requirement that it is linked to.
•In Excel, the test cases is displayed only once under the first requirement in the work item list. Seems duplicates are removed.
How can I retrieve the exact data as shown in TFS in Excel, including test case work items linked to multiple Requirements?
@opral
I followed up with my TFS counterpart for the issue you raised and here is his response:
The only query types that are supported in Excel are the Flat list and the Tree query. A direct link query is shown as a flat query by design. How would you show the link type for example in Excel, and because work items can show up multiple times in the result what to do if you make conflicting changes?
We have no plans to support direct link queries in Excel.
The only workaround to this problem today is to use the reporting infrastructure.
Thanks
-Ravi
Thanks you for such an informative post. The facts are truly helpful. Cheers!
Hi All,
I would like to use the Recommended Tests Feature to reduce regression efforts for the application which I am testing using MTM.
Can anyone advice what are all the steps which Developers and Testers needs to follow to achieve this .
I know Test Impact setting is needed to have this work . But Still I have some queries If code changes happened in the server how MTM know these are the test cases got impacted due to the code changed.
It would be great if aomeone explains with steps .
Thanks,
Muhsin
Very gοpod article. I'm goihg through some of these issuеs as well..
Also visit my webpage: Ramsey County Dwi Lawyer
@Muhsin, take a look at these posts:
http://msdn.microsoft.com/en-us/library/ff576128(v=vs.100).aspx
http://blogs.msdn.com/b/amit_chatterjee/archive/2009/04/11/diagnostic-data-adapters-changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-collector.aspx
http://blogs.msdn.com/b/densto/archive/2009/06/29/test-impact-walk-through.aspx
Thanks
-Ravi
We are looking at tfs 2013 update 4. Are Test Plans and Test Suites now WI's? If so, can I create a query that would test results for that Plan?
This appears to be one of your most popular posts, at least anytime I search for something related to MTM, this post appears in my search. I'm wondering, since they are up to a 2013 version, if you have any thoughts of a new post with your opinions about the latest release? Just curious. I enjoy reading your blog!
Hi there, what is the latest with implementing an order option in MTM? or where is it located?
I currently have Visual Studio Online Advanced license so can create test plans ec and execute them, I did not see an order option there.
I downloaded a trial version of Test Professional 2013, the documentation indicates there is an Order otpion but I do not see it, so question is, how can I order test cases in Visual Studio Online
This kind of worst application i have never seen in my life. It is hanging all the time + i am not able to control the mouse on runing test cases. The mouse is runing by its on ( MTM is riding on it)
Great article! Can you please share your updated thoughts on Microsoft Test Manager? Please do share if you have any better alternatives that can be easily integrated with TFS and works great for managing testing and test cases.