Announcing ShadowSpawn

One of the great things about working at Relevance is having every Friday to work on non-client work. I've been spending mine on various open-source projects, like Artifact and a core.logic port of The Reasoned Schemer. But I haven't forgotten my Windows roots entirely. One of the things I've been trying to do is to continue to improve hobocopy, the open-source backup tool I wrote that can copy in-use files. I initially released it about five years ago, and in that time it has been downloaded something like 100,000 times. In the half year since I re-launched it on github, it has been downloaded around 15,000 times. So clearly, people are using it, and this is a reasonably successful open source project. 

But as the project has picked up steam again I've noticed that the things people want tend to fall into two categories: 1) fixes for mysterious issues with the Volume Shadow Service that I can't do much about, and 2) feature requests around the copying part of hobocopy. I'm sort of stuck on #1, but on #2 I could actually roll up my sleeves and implement copying of streams, or correctly enabling the SE_BACKUP privilege, or any one of the myriad other things that go into correctly copying a file from point A to point B; it's a surprising complex problem. 

Or I could cheat. 

For a long time now, I've been contemplating an alternate approach. Rather than have a piece of software that would make a shadow copy of a volume and then make copies out of it, I thought it would be handy to have a piece of software that would simply make a shadow copy, temporarily make it available on some drive letter, and then run an arbitrary command [1].  If that arbitrary command happens to be Robocopy, then you've got all the advanced copying ability you could ever want. But you could just as easily use notepad if all you wanted to do was look at a file that was locked. 

Thus was born ShadowSpawn. I worked with the ever-excellent Kim Wolk, and this last Friday we got everything to the point where it works reasonably well. We've slapped a 0.1.0 version label on it and made it available for download. I'm not using it in my nightly backups at home yet, the way I do with hobocopy (I plan to make the switch soon), so consider it to be beta: we expect issues, but it should generally work.

Here's a simple example of how to use it: 

shadowspawn C:\some\directory Q: robocopy Q: D:\some\other\directory

That would snapshot your C: drive, mount the snapshotted version of C:\some\directory at Q:, run the robocopy command against it, and remove the Q: mapping when robocopy finished. More details are available at the website.

Anyway, we hope you find it useful. Feel free to ask for help on the mailing list, or to make a feature request or bug report on the issue tracker.

[1] Indeed, a tool called VShadow lets you do exactly that, albeit in a slightly different way. But I had additional reasons for wanting to take on this project. 

Hobocopy Lives!

 

It’s hard to believe it has already been almost over four years since I wrote
a little tool called hobocopy. The name was a pun on the truly-awesome robocopy, which does a great
many things and is a truly useful tool. Hobocopy doesn’t do a tenth what
robocopy does, but it does do one thing that robocopy can’t:
copy files that are currently in use. It does that by using the Volume Shadow
Service, which is the same facility Windows uses to create Restore Points. You
can read more about hobocopy here
and here.

I worked on hobocopy for a while, but as often happens, I got pulled onto
other things. I tried to see if someone else would pick up the torch, but was
unable to find anyone. Still, at least some people were able to get some use out
of it: there have been something like 100,000 downloads from SourceForge as of
this writing, and I get regular but infrequent questions about its use. So I
felt bad that I had let development languish.

Well, now I work for Relevance, and
they have a
very progressive policy on open source work. Of course I have far more
projects I want to tackle than I could do in a decade of Fridays, but still, I
figured that hobocopy deserves some love. So I’ve been spending a bit of time on
it lately. I've created the hobocopy Google group, and I've moved the code to its
new home on GitHub.

As you might imagine, there’s still a lot to do, but I’ll be
chipping away at it as I’m able to. Help of any sort (including bug reports!)
is, of course, welcome. 

 

2010 - The Year in Review

Forgive me, Internet, for I have sinned. It has been way too long since my last blog post. But I like to have an end-of-year post (here's last year's), so here at the end of 2010 seems an appropriate time to toss out something about what's been going on. This past year was ridiculously full, both from a personal and a professional standpoint. Any one of several of things that happened would have made the year notable.

For starters, we bought a new house and moved into it in June. I love the new place, but of course moving is always difficult, and months later we still have a significant amount of unpacking to do. By itself, moving would have been enough, but my wife and I seem to have a propensity for doubling (or tripling, or quadrupling) down. So at the same time we were moving, we did a major renovation of and rented out our old place, helped my wife's parents move, and took a cruise to Alaska. Right after that, my oldest daughter started Kindergarten, which was quite a shock to the daily routine.

Simultaneously with the move, I finished up at MSDN, having helped to ship VS2010. Having spent almost all of the last decade working in .NET, it made it a great time to make a major change. None of you who read this blog will be surprised that I wanted that change to involve doing Clojure. Clojure is a really interesting language, but also a very new one. We're seeing lots of growth right now, but of course it's a small market compared to what we hope it will be in a year or three. So I was fortunate that I was able to find some Clojure consulting work at Relevance. 

Relevance is a fascinating place. Even aside from the opportunity to do Clojure on a daily basis, I find it to be truly different from anywhere else I've ever worked. They take Agile seriously and (IMO) do it right. And where they don't do it right, well, they have a process for changing the process. It quite literally took me weeks to get used to the pace at which they get stuff done: it is a high-intensity, hard-core, hacker (in the best sense of that word) culture where they also manage to keep doing the right thing by the customer a top priority.

Which is why I am pleased to announce that I will be starting there as a full-time employee on January 3rd. See, I told you that 2010 was a big year: among all the other stuff, it also marks the end of an eleven-year span of being an independent consultant. It was a great run, but I'm genuinely excited about the opportunity to work at Relevance in a full-time capacity. I simply could not pass up the chance to participate in the unique combination of culture, technology, and people. And the people include, of course, my good friend Tim Ewald, who started at Relevance last month. This marks the fourth time we have worked together at one company or another, but the first time we've ever both been employees at the same time.

So 2010 was a year of big changes for me and my family, almost all of them good. What will 2011 bring? I have no idea, but I'm going into it in a pretty good mood. :) 

An Updated Nerd’s Halloween

[Update: corrected a minor typo. Also, I find it heartening that Peter is 70% of the way to his donation goal. Thanks to those that contributed.]

I don’t blog much any more. Partly that’s just blog fatigue (I started in 2003), and partly that’s having two little kids (one of whom just started Kindergarten), a new gig (at Relevance! Doing Clojure!), and having just moved to a new house. But I came across something that’s a pretty interesting, and a bit too long to do full justice to on my Twitter account.

Once upon a time, in a small way, I helped a guy make an awesome Halloween. He emailed me the other day to let me know about his latest project, which is even crazier and more amazing. Check out the picture of what he’s going to try to do:

But he needs some help raising funds. In his own words:

Craig,

It's me again, the guy who likes Halloween more than most humans! This was the post you did on your blog:

http://www.pluralsight.com/community/blogs/craig/archive/2008/10/01/you-think-you-like-halloween.aspx

If you don't remember me [Ed: how could I forget?!], check out the quoted material at the bottom of this email.  I noticed that you're not blogging much lately, but you are using Twitter. I was hoping you could blog and/or tweet about a project I'm trying to do this year. I'm trying to get the word out about my fund raising efforts to build this giant steampunk drilling machine prop (see attached photo) for Halloween this year.  Someone who had emailed me on my site (www.SoCalHalloween.com ) a couple of years ago created this entry on his blog:

http://superpunch.blogspot.com/2010/09/help-fund-building-of-giant-steampunk.html

A plug on your blog (or a tweet) would really help.

I'm using a site called "Kickstarter", which is designed to help do fund raising for any sort of creative project, using Amazon for payments. The rules are really simple: If you make or exceed your funding goal by the specified deadline, you get the money. If you don't make the goal, you get nothing.  In my case, my deadline is October 14, so there are 21 days to raise the money.  Also, people get gifts to thank them for pitching in. Any amount is helpful.  $10? Sounds great.  $20?  Even better.  $50? Say, you *do* care!

Please go to my Kickstarter page for Halloween 2010 to get more information, see how the fund raising works, and (hopefully) donate if you're so inclined: http://www.kickstarter.com/projects/1163328321/halloween-2010-journey-from-the-center-of-the-eart

If there are any problems with line wrapping, here's a shortened URL that will take you to the same page: http://tinyurl.com/2879mqk

I have also modified the main page of the SoCalHalloween website to carry the video, a link to the Kickstarter page, and some pictures.  It also has all the rest of the video and pictures that are always available: http://www.socalhalloween.com

So if, like I do, you think this is a really great idea, consider helping Peter out with a few bucks. Thanks!

Clojure Concurrency Tutorial Available

Update: all six parts are now available. That’s over two and a half hours of Clojurey goodness.

Update: There are now downloadable versions available in wmv and mp4 formats in a variety of resolutions. Just click the download button by each module. Also, the refs module is now up.

A while back, I wrote up some slides on concurrent programming in Clojure that I used at a DC ALT.NET presentation. Since I had them ready, and since Pluralsight has a great infrastructure for doing online training (it’s called Pluralsight On-Demand!), it made sense for me to record the content and make it available. So I have.

I’m happy to announce the immediate availability of the first four parts of my six-part Clojure concurrency tutorial, a screencast. You can find it here. It is freely available, and no registration is required. The final two parts will be available in less than a week.

I wrote the tutorial from the perspective of a .NET programmer who is new to Clojure. I don’t focus on the language itself, but rather the concurrency support in it. However, I don’t assume any prior knowledge of Clojure, so we start by covering just enough language syntax to get us through the demos. From there we dive into vars, atoms, agents, refs, promises, futures, and all sorts of good stuff.

If you’ve been reading my blog, you’ll know I’m a big fan of Clojure. One of the reasons for this - but not the only reason - is its fantastic support for concurrent programming. I like to say that it provides “sanity in the face of concurrency”. If you’re interested to learn more, check out the tutorial, and please – let me know what you think!

The Zero-Argument Overload of Any

I'm a pretty big fan of Linq. I use it all the time in my C# code. But it's a relatively big library, and I've never studied it exhaustively, so from time to time I come across some neat corner of it I've never seen before. In fact, that happened just the other day. 

I find that I'll frequently write test code that looks like this: 

Assert.AreEqual(0, collection.Count()); 

or the inverse: 

Assert.AreNotEqual(0, collection.Count()); 

I've done it often enough that I have thought I ought to write a pair of extension methods on Assert called IsEmpty and IsNotEmpty. That's probably still a good idea, but it also turns out that the Linq extension method Any (normally used to test if at least one element of an enumeration meets some criteria) also has a zero-overload method that simply returns true if the enumeration has any elements at all. So I could rewrite the above tests as: 

Assert.IsFalse(collection.Any());

and

Assert.IsTrue(collection.Any());

respectively. While having Assert.IsEmpty and Assert.IsNotEmpty extension methods would be better (perhaps your test framework already provides them), this is still a handy little thing to know about. 

Implementations of Slideshow in C# and Clojure

As I promised I would the other day, I went ahead and implemented my stupid little slideshow app in both C# and Clojure. It was an interesting exercise, the more so because I’m not entirely sure what conclusions to draw. I may have more to say about this at some point in the future, but for now I thought I’d just put up the code and share some general observations. So, you can find the C# version here, and the Clojure version here. 

The app is pretty simple. It starts a low-priority thread that lazily populates a list of filenames that are descendants of a particular directory. Meanwhile, a timer procedure randomly selects one of the filenames out of the ones that have been found so far and invalidates the screen. The repaint simply draws the image centered on a fullscreen window with a black background. If no images have been found yet, then it prints “Working…” on the screen instead.

The Clojure version isn’t really an app, because I wrote all this code really quickly and didn’t bother giving it a main() function, but other than that they’re about as similar as I could make them, given that one is running against Swing on the JVM and the other against Windows Forms on the CLR. I suppose I could have written the Clojure one against Windows Forms as well using ClojureCLR, but honestly it was simply more interesting to me to use Swing, as I’m not familiar with it. And this is anything but a scientific experiment.

For the C# version, I eschewed the Forms designer to keep it closer to the spirit of the Clojure one. And for something like this, where I’m just blasting raw pixels at the screen, I think that was a reasonable choice.

I wrote the Clojure version first. Perhaps that was a bad choice, since it meant I spent a lot more time on the Clojure one, since I had to figure out all the little details, like how to center an image in a region. It certainly made the C# version a lot easier to write, since I was pretty much just translating. But the C# version probably would have been easier for to write in any event, since I’ve been writing C# professionally for almost 10 years, and Clojure in my spare time for less than two.

One thing that surprised me a bit is that the Clojure version turned out to be not much shorter than the C# version. Indeed, if I had put closing braces on the same line as the statement they close (which is like the Clojure style I use), I’d bet they’re about the same. I had expected it to be quite a bit more compact, since, as I’ve said before, my experience with Clojure has been that it generally lets you be more concise. In this case, I suspect it has to do with the fact that I had a lot of Swing-related code, which doesn’t lend itself strongly to the sorts of concision that Clojure supports. Or it could be the fact that I’m a Clojure n00b, and just don’t know how to structure the code to hit a good balance of terseness with readability. Certainly, I could make it a bit shorter by inlining calls to things like make-frame. Other recommendations as to how better to write either version would be more than welcome.

Both versions of the code are pretty rough, as I wrote them essentially while I was waiting for stuff to install on a VM. I’d definitely clean them up more if I were going to do anything with them, but I figured it was better to get it up here than to wait potentially forever to get it polished. Really, I’m just trying to point out that not all my code looks this crappy. :)

You’ll note that in the Clojure version, I use an atom to store the list of filenames. That’s not because I’m taking advantage of Clojure’s awesome concurrency support, it’s because it was a convenient thing to do. Note that the C# version doesn’t bother with synchronization at all, since I’ve only got two threads, and one is only reading, and one is only appending. If I wanted to get fancier with the threads (e.g. have another thread doing look-ahead loading of the next image), then I might have to start doing synchronization. I’d be in good shape on the Clojure side then, since not much would have to change, and the fact that Clojure values are immutable really helps. I don’t think the C# version wouldn’t need too much work, but I’d have to think about it more to be sure, and it would depend on exactly what I was trying to do.

It’s apropos of nothing, but it made me smile that the C# version required PInvoke to get a fullscreen window, and the Swing one just supported it out-of-the-box.

Being familiar with Visual Studio was a definite benefit when working with the C# code. I’m not very facile with the debugging capabilities of Emacs/SLIME, which is what I use to develop Clojure, and when I had problems it took me a little longer to figure them out. That said, having the REPL was really awesome, and it really makes me hate the Immediate Window in Visual Studio by comparison. It would be interesting to try some of the other Clojure IDEs to see how they stack up.

Neither version of the code has much in the way of comments. Sorry about that. :)

Anyway, that’s my pile of unstructured thoughts. Hope it’s of interest, if not value, to at least some of you. And as I said, further discussion, rewrites, comments, and criticism are wholly welcome.