Sunday, October 17, 2004

TDD Is Great...Except When It Isn't

I've been a big fan of Test Driven Development (TDD) ever since I first started using it to write FlexWikiPad (FWP) about a year ago. In fact, that was one of the reasons I wrote FWP in the first place, as an excuse to see what the fuss about TDD was. It turned out to be a huge win for FWP, and I've been using it for everything I write since then. Until now.

Having spent a good chunk of my free time in the last two months writing RichTextEditor, my first real WinForms control, I slammed headlong into a problem that many others have had before me: writing a test-driven component that has a complex relationship with its container. You see, the main tenet of TDD is based on the idea that you simulate the client of a component before you write the component. These simulations usually take the form of method calls, since usually what you're testing is a simple algorithmic class.

Sometimes what you're testing is a little more complicated, like a form. In cases like that, testing gets a bit more complicated too, because now you have to simulate a user clicking buttons and entering text. But even that isn't that bad if you separate out the windowy bits of the application (the view) from the bits that react to it (the controller). Then you just test the controller by making calls that simulate events in the view. That's how FlexWikiPad is developed, and I've been very happy with it.

But sometimes, what you're testing is a lot more complicated, like a WinForms control. There, the thing we have to simulate is the windowing system itself, since that's really the client of my control: Windows sends events to my control, and my control sends painting commands back. Making matters worse, at the time I didn't even know what messages to expect, and what should happen in response to them. As a result, I found myself at a loss to write tests for my control.

So I didn't. I tested it the old fashioned way: manually. Was it more error-prone? Yes. Does it make bug regression more difficult? Yes. Did I have any other choice given my time and resource constraints? Not really.

Note, however, what I am not saying here:

  1. That TDD doesn't work. It actually works very well when simulating the container isn't too hard. Which is most of the time.

  2. That TDD doesn't work in GUI scenarios. I've used it very succesfully when building applications. Just not RichTextEditor.

  3. That TDD never works for control development. The limiting factor is being able to simulate the exchange of messages between Windows and your control. If this is complex, at some point simulating it will be too much work. But if it's simple, you might manage it just fine.

  4. That you can't use TDD at all if you're writing a complex control. I actually still used it in RichTextEditor to develop the undo/redo logic of my control, since that stuff is decoupled from the rest of the control, and has a simple interface. It was very helpful for that portion of the development.

Another thing to be aware of is that I never really explored automating my testing process in other ways. For example. a macro recorder could be used to send keystrokes to my control, and possibly to test the results in one way or another. NUnitForms has a Recorder that might help in this regard.

Overall, I'm still a huge fan of TDD, and still plan to use it whenever I can. It's just that now I have a better idea of what “whenever I can” means.


  1. I think you hit the nail on the head when you mentioned other ways of testing - after all, it is TDD not UTDD (unit test driven development).

    One approach I started to use in my previous project was to have lots of ASSERTs, a debug/trace listener that I could query against, and use a record/playback technique (in my case the John Robbins "Tester" utility that was presented over the course of a number of BugSlayer articles). The nominal case is tested by running and checking the specialised log listener recorded no errors or warnings, then as you go along you can test for error conditions by recording appropriate usage, then adding playback and checking into the test script.

    This is clearly not unit testing (system? smoke? integration?), but I'd still say the development was test-driven.

  2. I think that's a great idea. One of the main reasons I didn't go this route was that when I started that particular project, I was just playing around. I could have gone back and retrofitted it, but I was lazy. :)

    In any event, I still believe that there are cases where container-component interaction is so complex that any sort of TDD will start to get very expensive. For example, trying to test logic that ensures a particular piece of text got drawn at x, y location 25, 168 and has a red wavy underline with a solid, one-pixel border might get a little hairy. It's not that it can't be done, it's that it starts to be a net-zero or net-loss proposition.

    But as I said, I still believe in testing as much as you can. You seem to have found a way to automate testing of a large part of your app, which is a good thing.

  3. If the container is that difficult to simulate, why not use the real thing? Have your testing code load your control in a WinForm and play with it (causing events and calling methods to verify that everything stays consistent).

    Obviously, that doesn't allow you to test whether the control sends the right events to its host control but at least you can get pretty extensive test coverage on your own code without tripping on your inevitably imperfect fake host. I say that's better than nothing.

    I agree with you, when the container is complex it becomes very labor-intensive to test your component. Extreme example: my boss actually wants us to implement the entire MAPI specification in order to test our MAPI component (a MAPI Message Store Provider). Please don't laugh.

  4. Sure - when the container is amenable to automation, that can be a handy approach. But as you observe, sometimes it isn't.

  5. I use mock objects extensively, so I know how useful they can be. However, when what you're trying to mock up is the Windows Operating System itself...mock object solutions aren't particuarly helpful.

  6. Hmm. I've written fully tested GUIs without explicitly generating a state machine, so I'm not sure I buy your assertion.

    The article is interesting, however, and clearly a model-drive approach would be useful when the application lends itself to it.

  7. The assertion is a bit extreme I admit it :).

    But imagine that you want to hit the long-run tests, the ones that appear after using 2hours an application or with a sick combination of key strokes. Ideally, you would build a state machine and let run until it crashes (or not).

  8. I think Jens nailed it when he pointed out that there's a distinction between developer-written test, and test team-written tests. I think it's pretty well recognized that both are useful tools in developing quality software. Long-running tests are probably the domain of the test team, rather than the developers, as adding "two hours" to every developer build is not feasible.

    State machine development seems to my inexperienced eyes to be a tool primarily useful for the test team, with the impact to the dev team being *perhaps* to modify their code to make it easier for the test team to inspect application state.

    In other words, it sounds like a great approach as part of the overall process, but I don't think it's going to change the way I code.

  9. I've started reading the text of my program as a test. I check to see that I have made specific connections to framework elements by ensuring that specific interfaces have been coded.

    Then I can keep my state testing within the model. I use an MVC pattern and pass the "container" interface into the model. The code-checking tests verify that the controller has been set up to the view, and that the view is effectively connected (by code) to the the framework.

    I call these tests "Verifiers".

  10. I've written something similar for my clients, but rather than parsing code, I use FxCop to check properties of my assemblies. It seems to be a bit more flexible, as well as being language neutral. Plus FxCop brings so much other goodness to the table.

  11. I agree: you should adopt a test-first mentality and hold on to it whenever you can. However, as was pointed out above, you might be doing test-driven development without doing *automated unit test* driven development. Automated unit tests imply that you have to simulate your client/caller, which can be quite complex. Complex enough to stop you from doing it? Depends on your situation.

    I should also point out that I didn't mean to imply in any way that developers are absolved from writing all the tests they can. Rather, that we should remember that enlisting the test team to write simulations and automation for more complex scenarios - since they have the tools and expertise to do it - is a good idea. It's just about division of labor, not passing the buck.

    As widely available tools mature (e.g. NUnitAsp, NUnitForms), the cost of automation by the developer will continue to drop, and the test team can shift their focus to even more complex scenarios.

  12. That doesn't work when the interaction is complex, with callback relationships and bi-directional message passing.

    A control does not have a type relationship with Windows. It has a message-passing relationship. While that, too, could be mocked, it's a lot more the point of not being worth it.

    As someone else pointed out, you can still be test driven in this case. Just not NUnit test driven.

  13. Yeah, it is difficult to do TDD in some instances. But look at TypeMock.NET from, it is a really good mocking framework.

    I've posted a article the other day on how I solved an issue with a external Web Service depandency. Have a look at



  14. I've certainly used hand-rolled mock objects to the same effect. In fact, I've done it with web services.

    The fact that TypeMock is a commercial product pretty much guarantees that I'll never use it, though.

    And there's a *huge* difference between mocking a component's interactions a web service and mocking a control's interactions with Windows.

  15. I just had to do this, sorry :-)

    I think you are just trying to go with the "i don't have time to test" or "it's to much effort to test". If your job depended on it, you would for sure do whatever it takes to test. In the end, supporting usercontrols is pain and the time invested in testing the code would for sure be worth the investment, no doubt.

    It is such a similar discussion to the one you can hear about data access and SQL.

  16. @Niclas: well, I wrote this article about five years ago - I'd likely do things differently now.

  17. craig-andera, as you have gone through GUI testing long time a go, what's your advice when it comes to test a WPF application with resising adorners & drawing tool ?

  18. It has been over a year since I've done anything Microsoft-related, and much more than that since I've done GUI stuff. I never really got into WPF. So I'm not totally sure.

    I will say you'll probably wind up with a variation of the MVC pattern, where you test your controllers and maybe your models, and have dumb views that don't get tested. But you should go ask Ian Griffiths - he knows a lot about all things WPF.

  19. Thanks, I've sent him an email, and I will put his explanation here, as its in the heart of this thread :-)

  20. Thanks Ian, I wish that your explanations are supported with little examples, or links, it could be more thorough, as its a complex issue indeed.

    May be some one, will be interested in writing an example, showing mocks in action with GUI elements in WPF, it will be a great article indeed.

    Thanks again :-)