Who tests the testing tools?
I’ve been using the Team Edition for Developers-variant of Visual Studio 2005 for the past week and have to say that it’s really cool to have unit-testing, code coverage and code analysis all integrated so neatly in the VS IDE. My only gripe is that it feels incredibly rushed. Maybe I have a very short memory, but normally when I install a new version of Visual Studio (I’ve used all versions since 4.2) there are always a couple of bugs, but most of them only show themselves when you do some fairly complicated or unusual stuff (I’ve never done serious ASP.NET-development, I thought I’d mention this because there seemed to have been some big issues with that in previous versions of Visual Studio). This time around however, some of the stuff I’m running into is so basic that I can’t help but think that this is something the developers have certainly run into themselves but somehow the product was released anyway. Below are some of my first experiences
The first thing that went wrong was when I started work on a small new library and decided to generate a unit-test. A unit-testing project was created and added to the solution along with a generated template and some helpful comments. Being comfortable with NUnit is enough to get started immediately (it also makes you wonder why they even bothered to change the names of the attributes. Sure these names are probably better, but why change something that everybody already knows and uses?) Somehow my unit-testing project didn’t build. In fact, none of the attributes rang a bell with the compiler. Even worse, the whole namespace where all this stuff supposedly resides was unknown: there was no reference to the assembly! This was easily fixed by adding the reference myself (even though finding the right assembly took some time, since the namespace is Microsoft.VisualStudio.TestTools.UnitTesting but the assembly is called Microsoft.VisualStudio.QualityTools.UnitTestFramework). Now I know this is probably some glitch that doesn’t always take place (I don’t even want to consider that), but it’s pretty sloppy to have something like this even happen without a proper error message.
After writing a bunch of unit-tests, I got curious about code coverage, so I switched the view in the unit-test results. Unfortunately, code coverage appeared to be disabled, so I opened the respective Test Run Configuration and enabled code coverage. But still no results after running the results. Next step was Google: I found some people claiming that debug runs don’t work with code coverage. This seems to be the case. Would it really have been too complicated to note this on the Test Run Configuration somewhere? Or maybe where it says that code coverage is disabled?
Code coverage itself is really cool, especially in that it often points out some typical corners that are easily forgotten, such as guard clauses for obviously incorrect parameter values. It actually does happen that you find out that you pasted the wrong error message into an exception constructor after troubleshooting an installed application for a while. At least it does to me, so this should teach me to unit-test those guard clause-triggering calls as well. Unfortunately however, the coverage coloring is easily confused in the editor up to the point that restarting the IDE is the only way to get an updated view of the code coverage results.
So all in all, I’m very happy about the features in the Team Edition for Developers IDE, but it’s kind of a weird experience to finally work with a version of Visual Studio that makes it so much fun to test software only to discover that the tool itself could have used a couple more hours of exactly that activity. Oh well, I hear there’s a service pack for VS 2005 on the way?
