2010 - The Year in Review

Forgive me, Internet, for I have sinned. It has been way too long since my last blog post. But I like to have an end-of-year post (here's last year's), so here at the end of 2010 seems an appropriate time to toss out something about what's been going on. This past year was ridiculously full, both from a personal and a professional standpoint. Any one of several of things that happened would have made the year notable.

For starters, we bought a new house and moved into it in June. I love the new place, but of course moving is always difficult, and months later we still have a significant amount of unpacking to do. By itself, moving would have been enough, but my wife and I seem to have a propensity for doubling (or tripling, or quadrupling) down. So at the same time we were moving, we did a major renovation of and rented out our old place, helped my wife's parents move, and took a cruise to Alaska. Right after that, my oldest daughter started Kindergarten, which was quite a shock to the daily routine.

Simultaneously with the move, I finished up at MSDN, having helped to ship VS2010. Having spent almost all of the last decade working in .NET, it made it a great time to make a major change. None of you who read this blog will be surprised that I wanted that change to involve doing Clojure. Clojure is a really interesting language, but also a very new one. We're seeing lots of growth right now, but of course it's a small market compared to what we hope it will be in a year or three. So I was fortunate that I was able to find some Clojure consulting work at Relevance. 

Relevance is a fascinating place. Even aside from the opportunity to do Clojure on a daily basis, I find it to be truly different from anywhere else I've ever worked. They take Agile seriously and (IMO) do it right. And where they don't do it right, well, they have a process for changing the process. It quite literally took me weeks to get used to the pace at which they get stuff done: it is a high-intensity, hard-core, hacker (in the best sense of that word) culture where they also manage to keep doing the right thing by the customer a top priority.

Which is why I am pleased to announce that I will be starting there as a full-time employee on January 3rd. See, I told you that 2010 was a big year: among all the other stuff, it also marks the end of an eleven-year span of being an independent consultant. It was a great run, but I'm genuinely excited about the opportunity to work at Relevance in a full-time capacity. I simply could not pass up the chance to participate in the unique combination of culture, technology, and people. And the people include, of course, my good friend Tim Ewald, who started at Relevance last month. This marks the fourth time we have worked together at one company or another, but the first time we've ever both been employees at the same time.

So 2010 was a year of big changes for me and my family, almost all of them good. What will 2011 bring? I have no idea, but I'm going into it in a pretty good mood. :) 

An Updated Nerd’s Halloween

[Update: corrected a minor typo. Also, I find it heartening that Peter is 70% of the way to his donation goal. Thanks to those that contributed.]

I don’t blog much any more. Partly that’s just blog fatigue (I started in 2003), and partly that’s having two little kids (one of whom just started Kindergarten), a new gig (at Relevance! Doing Clojure!), and having just moved to a new house. But I came across something that’s a pretty interesting, and a bit too long to do full justice to on my Twitter account.

Once upon a time, in a small way, I helped a guy make an awesome Halloween. He emailed me the other day to let me know about his latest project, which is even crazier and more amazing. Check out the picture of what he’s going to try to do:

But he needs some help raising funds. In his own words:

Craig,

It's me again, the guy who likes Halloween more than most humans! This was the post you did on your blog:

http://www.pluralsight.com/community/blogs/craig/archive/2008/10/01/you-think-you-like-halloween.aspx

If you don't remember me [Ed: how could I forget?!], check out the quoted material at the bottom of this email.  I noticed that you're not blogging much lately, but you are using Twitter. I was hoping you could blog and/or tweet about a project I'm trying to do this year. I'm trying to get the word out about my fund raising efforts to build this giant steampunk drilling machine prop (see attached photo) for Halloween this year.  Someone who had emailed me on my site (www.SoCalHalloween.com ) a couple of years ago created this entry on his blog:

http://superpunch.blogspot.com/2010/09/help-fund-building-of-giant-steampunk.html

A plug on your blog (or a tweet) would really help.

I'm using a site called "Kickstarter", which is designed to help do fund raising for any sort of creative project, using Amazon for payments. The rules are really simple: If you make or exceed your funding goal by the specified deadline, you get the money. If you don't make the goal, you get nothing.  In my case, my deadline is October 14, so there are 21 days to raise the money.  Also, people get gifts to thank them for pitching in. Any amount is helpful.  $10? Sounds great.  $20?  Even better.  $50? Say, you *do* care!

Please go to my Kickstarter page for Halloween 2010 to get more information, see how the fund raising works, and (hopefully) donate if you're so inclined: http://www.kickstarter.com/projects/1163328321/halloween-2010-journey-from-the-center-of-the-eart

If there are any problems with line wrapping, here's a shortened URL that will take you to the same page: http://tinyurl.com/2879mqk

I have also modified the main page of the SoCalHalloween website to carry the video, a link to the Kickstarter page, and some pictures.  It also has all the rest of the video and pictures that are always available: http://www.socalhalloween.com

So if, like I do, you think this is a really great idea, consider helping Peter out with a few bucks. Thanks!

Clojure Concurrency Tutorial Available

Update: all six parts are now available. That’s over two and a half hours of Clojurey goodness.

Update: There are now downloadable versions available in wmv and mp4 formats in a variety of resolutions. Just click the download button by each module. Also, the refs module is now up.

A while back, I wrote up some slides on concurrent programming in Clojure that I used at a DC ALT.NET presentation. Since I had them ready, and since Pluralsight has a great infrastructure for doing online training (it’s called Pluralsight On-Demand!), it made sense for me to record the content and make it available. So I have.

I’m happy to announce the immediate availability of the first four parts of my six-part Clojure concurrency tutorial, a screencast. You can find it here. It is freely available, and no registration is required. The final two parts will be available in less than a week.

I wrote the tutorial from the perspective of a .NET programmer who is new to Clojure. I don’t focus on the language itself, but rather the concurrency support in it. However, I don’t assume any prior knowledge of Clojure, so we start by covering just enough language syntax to get us through the demos. From there we dive into vars, atoms, agents, refs, promises, futures, and all sorts of good stuff.

If you’ve been reading my blog, you’ll know I’m a big fan of Clojure. One of the reasons for this - but not the only reason - is its fantastic support for concurrent programming. I like to say that it provides “sanity in the face of concurrency”. If you’re interested to learn more, check out the tutorial, and please – let me know what you think!

The Zero-Argument Overload of Any

I'm a pretty big fan of Linq. I use it all the time in my C# code. But it's a relatively big library, and I've never studied it exhaustively, so from time to time I come across some neat corner of it I've never seen before. In fact, that happened just the other day. 

I find that I'll frequently write test code that looks like this: 

Assert.AreEqual(0, collection.Count()); 

or the inverse: 

Assert.AreNotEqual(0, collection.Count()); 

I've done it often enough that I have thought I ought to write a pair of extension methods on Assert called IsEmpty and IsNotEmpty. That's probably still a good idea, but it also turns out that the Linq extension method Any (normally used to test if at least one element of an enumeration meets some criteria) also has a zero-overload method that simply returns true if the enumeration has any elements at all. So I could rewrite the above tests as: 

Assert.IsFalse(collection.Any());

and

Assert.IsTrue(collection.Any());

respectively. While having Assert.IsEmpty and Assert.IsNotEmpty extension methods would be better (perhaps your test framework already provides them), this is still a handy little thing to know about. 

Implementations of Slideshow in C# and Clojure

As I promised I would the other day, I went ahead and implemented my stupid little slideshow app in both C# and Clojure. It was an interesting exercise, the more so because I’m not entirely sure what conclusions to draw. I may have more to say about this at some point in the future, but for now I thought I’d just put up the code and share some general observations. So, you can find the C# version here, and the Clojure version here. 

The app is pretty simple. It starts a low-priority thread that lazily populates a list of filenames that are descendants of a particular directory. Meanwhile, a timer procedure randomly selects one of the filenames out of the ones that have been found so far and invalidates the screen. The repaint simply draws the image centered on a fullscreen window with a black background. If no images have been found yet, then it prints “Working…” on the screen instead.

The Clojure version isn’t really an app, because I wrote all this code really quickly and didn’t bother giving it a main() function, but other than that they’re about as similar as I could make them, given that one is running against Swing on the JVM and the other against Windows Forms on the CLR. I suppose I could have written the Clojure one against Windows Forms as well using ClojureCLR, but honestly it was simply more interesting to me to use Swing, as I’m not familiar with it. And this is anything but a scientific experiment.

For the C# version, I eschewed the Forms designer to keep it closer to the spirit of the Clojure one. And for something like this, where I’m just blasting raw pixels at the screen, I think that was a reasonable choice.

I wrote the Clojure version first. Perhaps that was a bad choice, since it meant I spent a lot more time on the Clojure one, since I had to figure out all the little details, like how to center an image in a region. It certainly made the C# version a lot easier to write, since I was pretty much just translating. But the C# version probably would have been easier for to write in any event, since I’ve been writing C# professionally for almost 10 years, and Clojure in my spare time for less than two.

One thing that surprised me a bit is that the Clojure version turned out to be not much shorter than the C# version. Indeed, if I had put closing braces on the same line as the statement they close (which is like the Clojure style I use), I’d bet they’re about the same. I had expected it to be quite a bit more compact, since, as I’ve said before, my experience with Clojure has been that it generally lets you be more concise. In this case, I suspect it has to do with the fact that I had a lot of Swing-related code, which doesn’t lend itself strongly to the sorts of concision that Clojure supports. Or it could be the fact that I’m a Clojure n00b, and just don’t know how to structure the code to hit a good balance of terseness with readability. Certainly, I could make it a bit shorter by inlining calls to things like make-frame. Other recommendations as to how better to write either version would be more than welcome.

Both versions of the code are pretty rough, as I wrote them essentially while I was waiting for stuff to install on a VM. I’d definitely clean them up more if I were going to do anything with them, but I figured it was better to get it up here than to wait potentially forever to get it polished. Really, I’m just trying to point out that not all my code looks this crappy. :)

You’ll note that in the Clojure version, I use an atom to store the list of filenames. That’s not because I’m taking advantage of Clojure’s awesome concurrency support, it’s because it was a convenient thing to do. Note that the C# version doesn’t bother with synchronization at all, since I’ve only got two threads, and one is only reading, and one is only appending. If I wanted to get fancier with the threads (e.g. have another thread doing look-ahead loading of the next image), then I might have to start doing synchronization. I’d be in good shape on the Clojure side then, since not much would have to change, and the fact that Clojure values are immutable really helps. I don’t think the C# version wouldn’t need too much work, but I’d have to think about it more to be sure, and it would depend on exactly what I was trying to do.

It’s apropos of nothing, but it made me smile that the C# version required PInvoke to get a fullscreen window, and the Swing one just supported it out-of-the-box.

Being familiar with Visual Studio was a definite benefit when working with the C# code. I’m not very facile with the debugging capabilities of Emacs/SLIME, which is what I use to develop Clojure, and when I had problems it took me a little longer to figure them out. That said, having the REPL was really awesome, and it really makes me hate the Immediate Window in Visual Studio by comparison. It would be interesting to try some of the other Clojure IDEs to see how they stack up.

Neither version of the code has much in the way of comments. Sorry about that. :)

Anyway, that’s my pile of unstructured thoughts. Hope it’s of interest, if not value, to at least some of you. And as I said, further discussion, rewrites, comments, and criticism are wholly welcome.

Creating a Lazy Enumeration of Directory Descendants in C#

Update: it turns out that Clojure has a built-in file-seq function (thanks to Stuart Sierra for pointing this out) that already does exactly this, making Clojure the winner hands-down for brevity :) Seriously though, arguments about “batteries included” aside – although a valid point of consideration – I still think it’s interesting to consider the bits of code below.

Yesterday, I posted some Clojure code that showed how to create a function that would lazily return all the descendants of a directory. Here’s the code again:

(import [java.io File])

(defn dir-descendants [dir]
  (let [children (.listFiles (File. dir))]
    (lazy-cat 
     (map (memfn getPath) (filter (memfn isFile) children)) 
     (mapcat dir-descendants 
      (map (memfn getPath) (filter (memfn isDirectory) children))))))

Towards the end of the post, I tossed off a somewhat off-hand comment:

Two things struck me about writing this code. The first is that it’s really, really compact. It would be even smaller if it weren’t for the calls to memfn, which is just there so I can use a Java method as if it were a Clojure function. The second is that it was a bit of a mind-bender to write, although having written it I find it fairly easy to read. But I suspect both of those things would also be true if I attempted to write the equivalent C# (including the laziness, probably via a “yield return” implementation of IEnumerable somewhere), although it would almost certainly be at least somewhat more verbose.

Well, Chris Sells called me on that one. Here’s the equivalent C# code he came up with:

using System.IO;
using System.Collections.Generic;

namespace ConsoleApplication1 {
    class Program {
        static IEnumerable GetDirectoryDecendants(string path) {
            foreach (var file in Directory.GetFiles(path)) { yield return file; }
            foreach (var directory in Directory.GetDirectories(path)) {
                foreach (var file in GetDirectoryDecendants(directory)) { yield return file; }
            }
        }
    }
}

I think quantitative comparisons of code terseness are generally pretty silly, but - subjectively - when I look at that it seems just as compact as the Clojure version. So I stand corrected on that one. A few other observations:

• Chris included namespace and class statements. Clojure has equivalents for these, but I omitted them. Omitting them from the C# version would make it even shorter.



• The C# version can actually be made more lazy than the Clojure one because of the CLR’s Directory.EnumerateFiles method, as pointed out by John in the comments to yesterday’s post. That’s pretty cool. I could, however, take advantage of this if I used ClojureCLR instead of the JVM-based Clojure.



• I said that writing the Clojure version was a bit of a mind-bender. To clarify, the bendiness came from the behavior of lazy-cat, and I think primarily because I’m relatively new to Clojure. The “yield return” in the C# code – which is the key to the laziness – gave me similar pause when I first encountered it years ago. And although I’m comfortable enough with it now, the two yield return statements in Chris’s code made me pause for a moment to convince myself it would work without consuming stack. So I think there’s probably some inherent complexity in the laziness that – like many things – is no big barrier once you internalize it in either language.

So I haven’t proven anything to anyone, not even myself. That said, having worked with Clojure a bit and C# a lot, I’m still left with the impression that Clojure is generally more compact. It will be a few years before I can say whether that’s a good thing or not (or even whether it’s true), since I tend to think working with a language on that time scale is the only way to really get to know it. I will do two things, however. The first is to show you another bit of code in both Clojure and C#. First, the C#:

private static IEnumerable Fibs()
{
    yield return 0; 
    yield return 1; 
	
    int a = 0; 
    int b = 1; 
    while (true)
    {
        int temp = a + b; 
        a = b; 
        b = temp; 
        yield return b; 
    }
}

Next, the Clojure:

(def fibs (lazy-cat [0 1] (map + fibs (rest fibs))))

Hopefully it’s obvious from at least one of these that they’re meant to compute the Fibonacci sequence.

Maybe someone can come up with ways to make the C# more compact: I would welcome such suggestions. And I’m not trying to have a Clojure-versus-C# contest here – those are stupid, like saying, “Which is better: a hammer or a drill press?” But language comparisons are still interesting in that you generally wind up learning something you didn’t know about at least one of the languages.

The other thing I’m going to do is to implement my slideshow app (the thing that had me writing the directory recursion function in the first place) in both languages and post them here. Until Chris posted, I was pretty sure I knew how they’d stack up. Now I’m just curious! Maybe you are, too.

Thanks, Chris, for keeping me honest.

Creating a Lazy Sequence of Directory Descendants in Clojure

Update: it turns out Clojure has a built-in function file-seq that does (almost) exactly this. The difference is that you get a lazy sequence of File objects, not paths. Hooray for "batteries included"!

I was playing around with Clojure this morning, attempting to write a little photo viewer application. As part of this, I wound up writing a function that produces a lazy sequence of all the files that are descendants of a given directory. I was pretty happy with how it turned out, so I thought I’d share it. Here’s the code in its entirety: 

(import [java.io File])

(defn dir-descendants [dir]
  (let [children (.listFiles (File. dir))]
    (lazy-cat 
     (map (memfn getPath) (filter (memfn isFile) children)) 
     (mapcat dir-descendants 
      (map (memfn getPath) (filter (memfn isDirectory) children))))))

The key bit here is the lazy-cat, which returns a lazy sequence – it doesn’t evaluate the second expression (in which I walk into the children of the current directory) until someone asks for the next element in the sequence. In other words, even though this function looks recursive, it’s not. Because lazy-cat returns right away, the “nested” call to  dir-descendants happens after the stack has been popped. This is totally awesome, because it means I can call dir-descendants and it’ll return right away. So I can start displaying pictures without having to wait for the entire directory tree to be enumerated. 

Two things struck me about writing this code. The first is that it’s really, really compact. It would be even smaller if it weren’t for the calls to memfn, which is just there so I can use a Java method as if it were a Clojure function. The second is that it was a bit of a mind-bender to write, although having written it I find it fairly easy to read. But I suspect both of those things would also be true if I attempted to write the equivalent C# (including the laziness, probably via a “yield return” implementation of IEnumerable somewhere), although it would almost certainly be at least somewhat more verbose. 

 

On the Utility of TestPropertyAttribute

On the project I’ve been working on lately, I’ve been using the unit test stuff that ships with Visual Studio/.NET, in the Microsoft.VisualStudio.TestTools.UnitTesting namespace. For my purposes, I’ve found it to be adequate, and it has the benefit that I don’t need to include any additional libraries.

I recently came across one use of it, however, that took me some time to figure out. Primarily because the documentation on this feature sucks pretty bad. It’s the TestPropertyAttribute, and it’s useful enough to be worth knowing about. So I thought I would share.

If you read the documentation for TestPropertyAttribute, your first reaction might well be the same as mine was: “Huh? That looks like it adds precisely no value over any other custom attribute I might write.” Because the docs show that to retrieve the value, you reflect against the test method it’s defined on, which is pretty silly. However, in experimenting, I found that there’s an equally poorly documented property on the TestContext class called Properties that you can use to get the value of this attribute.

Still, if you only need a value within the TestMethod itself, there’s not much value in it, as it’s no better than just hardcoding the value within the body of the test. Where TestProperty shines, though, is that the properties for the current test are also available during the TestInitialize method. That means you can bung a whole bunch of common setup code into your TestInitialize method, and parameterize its behavior based on properties defined on the tests themselves. A little code might help demonstrate what I mean:

   1:  using System;

   2:  using System.IO; 

   3:  using Microsoft.VisualStudio.TestTools.UnitTesting;

   4:   

   5:  namespace Demo

   6:  {

   7:      [TestClass]

   8:      public class ThingTests

   9:      {

  10:          // Note that we have to have a TestContext, and it must

  11:          // be named TestContext

  12:          public TestContext TestContext { get; set; }

  13:   

  14:          // A simple field that we're going to initialize to 

  15:          // some value in the Initialize method.  

  16:          private string _thing; 

  17:          

  18:          [TestInitialize]

  19:          public void Initialize()

  20:          {

  21:              // Obviously we could do something much more complex here.

  22:              if (TestContext.Properties.Contains("thing"))

  23:              {

  24:                  _thing = TestContext.Properties["thing"] as string;

  25:              }

  26:              else

  27:              {

  28:                  _thing = "default"; 

  29:              }

  30:          }

  31:   

  32:          [TestMethod]

  33:          public void TestAgainstDefaultSetup()

  34:          {

  35:              Assert.AreEqual("default", _thing); 

  36:          }

  37:   

  38:          [TestMethod]

  39:          [TestProperty("thing", "non-default")]

  40:          public void TestAgainstCustomSetup()

  41:          {

  42:              Assert.AreEqual("non-default", _thing); 

  43:          }

  44:      }

  45:  }

Basically, we’ve got a field _thing that we usually want to leave at the default value of “default”, but in some tests we want to override to some other value. Moreover, we want to override it in the Initialize method, since that’s where we’re doing our setup that’s common across tests. – maybe we have fairly complicated setup that needs to differ only slightly in some tests. So we use the TestProperty attribute to set a property named “thing” to whatever we like. Then, in the Initialize method, we can use TestContext.Properties to check for the presence of that value and take appropriate action.

I’ve made the code intentionally trivial to more easily demonstrate the concept, but I’m sure you can visualize more realistic scenarios where this might come in handy. For example, I’ve used it to customize which directory my Initialize method operates on.

2009 – The Year in Review

With only six posts the entire year, 2009 was my lightest ever for blogging, although I continue to be active on Twitter (here). I don’t see that changing in 2010. Still, as is my custom, I post herewith a review of 2009, both personal and professional.

Looking back on my review of 2008, it seems that 2009 was a year of relative stability for me. I continued to work on pretty much the same stuff: primarily MSDN, but a few side projects whenever I could wedge in some free time. And outside of work, family consumes the bulk of my time, although I still manage to run and to occasionally do something else fun (like fly the Black Shark – ping me if you want to join us!).

The MSDN work has remained interesting, particularly in giving me the chance to work on Visual Studio 2010 itself. That’s right, I have code that will ship in the box! Specifically in the bit that supports offline help. It was quite an experience to work on so large a project, and I’m definitely looking forward to installing the final bits when they are released later this year.

A few random highlights from the year:

• I bought an iPod Touch, which I totally love. I use it constantly, and if it ever broke I’d buy another on in a heartbeat.
• Clojure remains the most interesting programming language I know. I donated twice in 2009, and would encourage you to do the same (donating just once would be fine). Even if you don’t care about Clojure - although any serious programmer owes it to themselves to check it out - I assure you that Rich is doing a lot to advance the state of the art in concurrent programming and is worthy of your support.
• I read a graphic novel version of “The Hobbit” to my five-year-old…four times.
• We made some progress on the basement, but am still (somewhat depressingly) not done.
• I started using git seriously, and think it’s pretty cool.
• I ran in a track meet for the first time since college. And didn’t suck…as much as I thought I would.
• I started playing bass again. Badly, but I don’t care.
• I saw four movies in an actual theater: Earth, Terminator, Harry Potter, Star Trek. That’s roughly four more than I usually see, at least since the kids were born.
• We did lots and lots of travel: I was away from home for a cumulative total of over two months last year, including one trip to Asia. Alice and the girls were away for around three months if you add it up, including a second trip to Asia without me. Crazy.

As for what 2010 holds, I’m honestly not sure. On the personal front, it would be great to finish the basement. But bigger than that by far looms the shadow of Kindergarten, which our older daughter will enter in the fall. I’m sure it’ll be a big transition for everyone around here, particularly in the mornings, which tend to run on the late side most days. But we’ll handle it like we handle everything else. And I’m sure we’ll keep traveling, but hopefully not quite as much.

On the work front I really have no idea. I’m continuing to work with MSDN, and that will go on for at least a bit longer, but the life of an independent contractor is filled with uncertainty, and there’s the distinct possibility I’ll have to do a fairly serious (and possibly lengthy) search for work at some point. But as I’ve said before, it’s the rare challenge that doesn’t pose an opportunity.

Good luck to you in 2010!