Wednesday, February 24, 2010

On the Utility of TestPropertyAttribute

On the project I’ve been working on lately, I’ve been using the unit test stuff that ships with Visual Studio/.NET, in the Microsoft.VisualStudio.TestTools.UnitTesting namespace. For my purposes, I’ve found it to be adequate, and it has the benefit that I don’t need to include any additional libraries.

I recently came across one use of it, however, that took me some time to figure out. Primarily because the documentation on this feature sucks pretty bad. It’s the TestPropertyAttribute, and it’s useful enough to be worth knowing about. So I thought I would share.

If you read the documentation for TestPropertyAttribute, your first reaction might well be the same as mine was: “Huh? That looks like it adds precisely no value over any other custom attribute I might write.” Because the docs show that to retrieve the value, you reflect against the test method it’s defined on, which is pretty silly. However, in experimenting, I found that there’s an equally poorly documented property on the TestContext class called Properties that you can use to get the value of this attribute.

Still, if you only need a value within the TestMethod itself, there’s not much value in it, as it’s no better than just hardcoding the value within the body of the test. Where TestProperty shines, though, is that the properties for the current test are also available during the TestInitialize method. That means you can bung a whole bunch of common setup code into your TestInitialize method, and parameterize its behavior based on properties defined on the tests themselves. A little code might help demonstrate what I mean:


1:
using System;


   2:  using System.IO; 


   3:  using Microsoft.VisualStudio.TestTools.UnitTesting;


   4:   


   5:  namespace Demo


   6:  {


   7:      [TestClass]


   8:      public class ThingTests


   9:      {


  10:          // Note that we have to have a TestContext, and it must


  11:          // be named TestContext


  12:          public TestContext TestContext { get; set; }


  13:   


  14:          // A simple field that we're going to initialize to 


  15:          // some value in the Initialize method.  


  16:          private string _thing; 


  17:          


  18:          [TestInitialize]


  19:          public void Initialize()


  20:          {


  21:              // Obviously we could do something much more complex here.


  22:              if (TestContext.Properties.Contains("thing"))


  23:              {


  24:                  _thing = TestContext.Properties["thing"] as string;


  25:              }


  26:              else


  27:              {


  28:                  _thing = "default"; 


  29:              }


  30:          }


  31:   


  32:          [TestMethod]


  33:          public void TestAgainstDefaultSetup()


  34:          {


  35:              Assert.AreEqual("default", _thing); 


  36:          }


  37:   


  38:          [TestMethod]


  39:          [TestProperty("thing", "non-default")]


  40:          public void TestAgainstCustomSetup()


  41:          {


  42:              Assert.AreEqual("non-default", _thing); 


  43:          }


  44:      }


  45:  }



Basically, we’ve got a field _thing that we usually want to leave at the default value of “default”, but in some tests we want to override to some other value. Moreover, we want to override it in the Initialize method, since that’s where we’re doing our setup that’s common across tests. – maybe we have fairly complicated setup that needs to differ only slightly in some tests. So we use the TestProperty attribute to set a property named “thing” to whatever we like. Then, in the Initialize method, we can use TestContext.Properties to check for the presence of that value and take appropriate action.

I’ve made the code intentionally trivial to more easily demonstrate the concept, but I’m sure you can visualize more realistic scenarios where this might come in handy. For example, I’ve used it to customize which directory my Initialize method operates on.

8 comments:

  1. I find it hard to fathom that someone who has ever used xunit.net or any other non NUnit-family based test system in anger and tried to do without 57 attributes for doing tests couldnt find a cleaner way of expressing such variances in a test that's easier for someone who hasnt digged into the intricacies of all this stuff to grok.

    (I'm a massive fan of all your works down the years and am frankly surprised you deem this cool other than as a hack to add to a big wobbly pile of other hacks).

    And the fact that its included OOTB is balanced by having to install VS on your build server (which is supposed to stop in 2010 but not if you need to do a mstest /publish from what I've seen recently).

    Are you using the Shadow task (Private Accessors) on a build server? Is it stable?

    ReplyDelete
  2. Right, so I didn't say that I'm using VS2010, but I am. And the build machine is actually my dev box. Like I said in the article: this turns out to be a good setup for me, based on the constraints and conveniences of this particular project.

    I have to say I still tend to like something like NUnit better overall, but again, it made more sense for this particular project to use what Microsoft has provided. At the same time, I don't understand your characterization of this as particularly hacky. That's a fair cop for some of the other crap in the VS testing stuff, but this one isn't too heinous from where I sit. Can you explain what you mean - I'm curious to hear your perspective.

    I'm not using the Shadow task, but again, my build "server" has VS installed on it. Which I 100% agree sucks as a general solution.

    Of course, I find pretty much everything in the C# world heinously hacky compared to my latest love, Clojure. :)

    ReplyDelete
  3. I listen to PSCast (and its earlier aborted predecessor!) so am well aware of your sensibilities and depth of thought.

    Tests as described in the Osherove book (for all its faults) and the Meszaros book (which is excellent) and many other places should be as short as possible and rely on as little out-of-band trickery as possible.

    Attributes are the very epitomy of OOBT. Arguably NUnit went too far with them. MBUnit and NUnit defintely did. xunit.net rolls back a lot of this "oh, just read about these 53 attributes and 20 helper bits (SyntaxHelpers.Is, TestClassInitialize vs TestInitialize, Assert.Ignore, Assert.Fail) and you'll be ready to write your first test.

    Having to put

    // Note that we have to have a TestContext, and it must
    // be named TestContext
    public TestContext TestContext { get; set; }

    in a part of your code that is as important as any other part of your codebase is a Bad Thing as people have to understand what it is and dumping auto-generated junk in a #region is not free - there is a tax for it being there in terms of having to skip over it (or understand it before deleting it when you port to xunit.net :P)

    Thus I'd favour an explicit call to SetThingNonDefault instead of something that leans on:
    - adding yet another attribute and the tax it represents to the overall set of thing you have to understand before writing tests
    - strings
    - hard-wired names
    - comments to justify stupid things like hard wired names
    - 20 lines of code including lots of stuff that can fall out of sync (comments, strings)
    On the plus side, there's a lot to be said for declarative things over imperative in general.

    Adding something like this makes it massively over-engineered.

    I'm not disputing that this is a cool trick to be aware of if you're on the road to MSTest nirvana.

    But, like some of the stuff in WinForms. the fact that this sort of trickery and magic is considered a good idea is a Bad Smell. It breeds people thinking there must be a magic answer if only they could find the Bingoogle keyword for it.

    I'm confident I dont need to explain why the Shadow task (testing privates) is bad news. I just wanted to point out that in addition to being bad news in general, it specifically falls over on build servers randomly and very frustratingly.

    I dont know if you've spent a lot of time with xunit.net, but its a seriosly nice piece of kit and I realised that pretty quickly (before reading Meszaros, Osherove and the PragProgs Unit Testing book -- all of which allude to reasons why).

    In xunit.net, if you had a need of this nature, you'd build a custom FactAttribute if this was indeed a real widespread need which was making your test code ugly and hard to fathiom in many places. The diff is that xunit.net wouldnt have got any more complex to grok and your test code wouldnt be as long and ugly at the end of the exercise.

    Now I feel like Scott Bellware except he can clearly type faster so I'll stop now!

    ReplyDelete
  4. Great points, as always!

    First, you're right about the magic TestContext property sucking. I hate it, and the comment is there because my first instinct was to name it Context, which didn't work.

    I also agree that a method like SetNonDefaultValue would be preferable. And I would have used that approach, but it wasn't actually possible in the case that sent me down this road: the custom value is used smack in the middle of the initialization code. And for better or for worse, we have been taking the approach that [TestInitialize] is preferable to calling Initialize() at the beginning of each TestMethod. I can definitely see both sides of that debate, however, and may well swing back the other way, which has been my historical preference.

    What sucks is, I'd love to be able to show y'all the code to see if you agree with my design choices in the face of the constraints I'm operating under - absent those it becomes very difficult to debate the merits of an approach. But this thing I'm working on is going open source in a month or two, so maybe we can revisit it then. And I'd love to hear then whether you think I could have improved on my approach at all.

    That said, I do agree with your sentiments. I have yet to read the books you mention (currently in the middle of "On Lisp", "Clojure in Action" and "Thinking in Clojure", so you can sort of see where my brain is), and I've never used xUnit for real, but I look forward to doing so.

    Thanks again for your insight - always very valuable to me!

    ReplyDelete
  5. I've been neglecting Lisp and F# books in my queue to read the Unit Testing books recently. The Osherove books gets you thinking (but definitely needs a pinch of salt in the same way as MSTest does).

    I have no doubt that in your context this is perfectly valid. And once you've settled on a tool, I'd generally support using features and idioms of the tool in preference to finding a way of routing around it for no good reason. Point is that xunit wouldnt give you something like this to use unless there was a real demonstrated requirement for this that
    a) doesnt have workarounds
    b) justifies the tax of introducing the feature
    c) is going to be widely used

    The sheer number of Attributes and tricks in NUnit/MSTest/MBUnit just isnt funny anymore and luckily a latecomer like xUnit.net can dump some stuff and then be agressive about not putting new stuff in willy nilly.

    Mention of TestInitalize raises the debate about the undue of conflation of fixture and testclass stuff in [MN][SB]?Unit - but you'd need to read Mesxaros for that.

    I wouldnt have seen any of this without porting a suite of MSTests over to xUnit.net and reading the books.

    I look forward to seeing your *real* source and am confident I'll learn from it and not be slewing ranty comments in its direction!

    A good read for food for thought around is the links I have in the comments to http://roundcrisis.com/2010/02/09/test-class-organization/feed/

    ReplyDelete
  6. Oh, there will probably be plenty of room for ranty comments directed at that code. :)

    Seriously, though, intelligent discussion such as this are always welcome.

    I'll have to check out xUnit and the books you mention at my next opportunity. Thanks for the pointers!

    ReplyDelete
  7. Interesting Finds: February 25, 2010

    ReplyDelete
  8. Yet another "feature" of TestContext etc.. which reveals that MStest assembly must be made by very strange people...

    //Use ClassInitialize to run code before running the first test in the class
    [ClassInitialize()]
    public static void MyClassInitialize(TestContext testContext)
    {
    testContext.Propert.Add("mykey", 1 ) ;
    }

    Above will happily work and erase all but "offical" keys as soon as it leaves the method ?!
    So if you thought this method has a TestContext instance so that one can actualy save some class wide testing value, you are wrong.

    Strange thinking beyond belief ...

    --DBJ

    ReplyDelete