Announcing HoboCopy

For a while now, I've been unhappy with the state of my backup strategy. I have a simple script that runs ntbackup every night on a rotating, 14-day schedule: one full backup followed by six incremental backups, then repeat with a different target. In this way, I can always recover to any day in (at least) the last week.

 

So why am I unhappy? Two reasons:

 

1. ntbackup isn't exactly robust - I've seen all sorts of failures, and restoring files is sort of flaky.


2. ntbackup locks its information up in a proprietary format. I just want my files, dammit, not some .bks file.

 

Being a cheap bastard with decent programming skills, I started looking at what it would take to wire up my own solution. I thought about using robocopy, the king of copy tools. But it has one major drawback: it can't copy any files that are locked by another program. Some of my friends solve this problem by shutting down all their programs every night, but I knew that there was no way I (or my wife) was going to remember to do that, and of course the one time I'd forget was the day my hard drive would tank. Plus, what about programs like SQL Server that I don't want to ever shut down?

 

Obviously, it's possible to copy files that are in use. After all, ntbackup does it. So I started poking around, and came across VSS. The good VSS (Volume Shadow Service), not the unbelievably crappy one (Visual Source Safe). The Volume Shadow Service is a very cool piece of Windows XP/2003 that lets you "snapshot" a hard drive, creating what's essentially a point-in-time image of what's on the disk. You can then copy files from that image at your leisure. Better, it's done in an efficient manner, so that it doesn't actually copy anything unless someone does a write, and even then it only copies at a block level. Which is good, because otherwise you'd need 50GB free to snapshot 50GB of data.

 

But it's even better than that. VSS includes an API that programs like SQL Server 2005 can use to find out when a snapshot is about to occur. When so notified, VSS-aware programs can flush their state to disk, so you get a consistent backup. Of course, not every program is aware of VSS. For those, you get what the docs call a "crash consistent" snapshot. Translation: whatever the hell was on the disk. In my book, that's still better than not backing up at all. After all, my computer seems to do just fine after a BSOD, which is no different.

 

Armed with the Platform SDK docs, I set out to achieve my goal of a VSS-based backup utility. I would have liked to have used robocopy to do the copy, but I couldn't get it to copy using source paths like \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\data\backup\SmtpSend\Properties\Resources.resx,

which how you get to the files in the snapshot.

 

Next, I tried to write a managed tool that would do the copy. That wouldn't have been too bad - recursive file copy is pretty simple to implement. Unfortunately, the VSS API is completely broken for CLR interop - the main object you need to access implements one COM interface,  IVssBackupComponents. But if you try to query that object for that interface, it returns E_NOINTERFACE. Which is wrong, wrong, wrong. And also means that there's no way to use the object from straight managed code.

 

So I decided to write the tool in unmanaged C++. Maybe I could have done it in managed C++, or a combination of C++ and C#, but in the end I decided to make this a project that would help me get a little rust off my C++ skills. Plus, the documentation suggests that paths starting with \\?\GLOBALROOT aren't safe to use from managed code, although it worked fine when I tested it. The end result is HoboCopy. You pass it a source directory and a destination directory, and it makes a recursive copy using VSS. I've run it against my entire hard drive, and it's able to copy everything except files I didn't have permission to copy (e.g. some stuff under Documents and Settings), and for those situations I've got the /skipdenied switch. Some day I'll add a switch that enables the SE_BACKUP privilege to make that behavior even better. Unless one of you wants to submit a patch. :)

 

Speaking of patches. As of now, the project is hosted at SourceForge under the MIT license, so have at it! (Make sure you download the right one - there are separate binaries for XP and W2K3.) Just go easy on me if you look at the source - it's been years since I wrote much C++. :)

FolderShare + Bazaar = Tasty

As a consultant, I face the occasional problem of having to work on source code that lives behind my clients' firewalls. Often, getting access to their VPN is difficult, like when the engagement is very short and the VPN red tape copious.

 

I've tried emailing zips, setting up a WebDAV share, and various other strategies, all of which had one shortcoming or other. The best solution is, of course, a source control system of some sort, but even that I've had problems with (e.g. the combination of the client's firewall and my ISP blocks the port(s) we need).

 

For a couple of weeks now, I've been trying a new strategy. Using FolderShare, we create a shared folder that we both have write access to. Then we turn it into a Bazaar repository. On each end, we pull from and push to this folder/repository, making it look a lot like a source control server. And so far, it works great. It even runs over HTTPS, which should make any suspicious sysadmins slightly happier.

 

I imagine one could do the same thing with CVS or Subversion [1], although I haven't tried it. Bazaar seemed like a reasonable choice for two reasons: a) it's more "folder-y" than most SCC systems, so it seemed like a natural fit, and b) Sam Ruby thinks it's cool, so I wanted to try it out. :)

 

[1] You might even be able to make it work with SourceSafe. But then you'd have to use SourceSafe, and why would you use something that's both worse and more expensive?

Inside MSDN - Consuming MSDN Web Services

Well, it's early September, and you know what that means: time for the October MSDN issue. I've been checking the website frequently to see when the new issue would be released, because I've got an article in it! You can read it here. It's the latest installment of "Inside MSDN", a column that talks about how the suite of applications and services that make up MSDN itself is built. My article covers how to use the MTPS Content Service, which I announced here on CraigBlog a few months ago.

 

This is my second MSDN article, but my first solo one. I wrote the first one with Tim on an obscure feature of COM+. ("What's COM+?" you say? Yeah, I feel the same way.) 

 

I hope I can be excused for thinking it's sort of cool to have published in MSDN Magazine. :)

Console Text Selection Via the Keyboard

I was hanging out with some Pluralsight guys last weekend. Dan Sullivan was talking about SQL Service Broker, so I was already paying close attention, but then he did something that made me sit straight up my seat…he selected text in a console window using only the keyboard!

 

I've said it many times before: I'm a keyboard kind of guy. So it has always bugged me when I have had to take my hands off the keyboard to select text in a console window. Well, now I don't have to any more. Maybe I'm the last guy to figure this out, but apparently when you go into mark mode in a console window, you can use the arrow keys to move the selection point, and shift-arrows to extend the selection. Way cool!

 

Just to summarize, here's how I'd select some text in a console window, using only the keyboard:

 

Alt-Space, e, k  [Enter mark mode]

Down, down, down, right, right, right [Move the cursor]

Shift-right, shift-down, shift-right [Extend the selection]

Enter [Copy the selection to the clipboard]

 

Yay! Thanks Dan!

Producing Open Source Software - A Review

I just now finished reading "Producing Open Source Software: How to Run a Successful Free Software Project". It was, in a word, excellent. Really, really good. And it's available in its entirety for free online.

 

I definitely learned a lot reading this book, despite having had a couple of years' experience helping to run FlexWiki now. I was happy to see that we already follow many of his recommedations, but I also know I'll be changing a few things we do based on what I read.

 

Even setting aside the fantastic advice about how to set up the technical infrastructure, I found Mr. Fogel's insight into the culture of free/open source to be truly insightful. It's no wonder that Subversion rocks so hard with guys like this behind it.

 

If you haven't already, read this book soon.

Getting CLR Security Right - Seeing Double

This one bit me the other day. Fortunately Dominick was able to straighten me out rather quickly.

 

What I was trying to do was to update a build script to run off of a shared drive (don't ask). Of course, when running .NET code like NAnt and the other 90 tools we use in the build, you need to elevate privileges or your tools are unlikely to work with the decreased permissions they get by default when running from a network drive. That much I knew. Where I got burned was that some of our tools are compiled for .NET 1.1, and some for 2.0. Well, policy for 1.1 is completely separate from policy for 2.0. So I had to change it in both places.

 

Luckily, I didn't waste too much time on this. It helps to be willing to bother your friends. :)

It's (sort of) Alive!

Just a few minutes ago, I fired up my web browser and pointed it at the FlexWiki code I've been hacking on. To my complete amazement, the home page actually rendered! It was riddled with errors, and it crashed when I tried to edit a page, but the blasted thing actually showed me HTML that wasn't the ASP.NET error page…I practically peed myself.

 

As you may have read here before, I've been working on refactoring FlexWiki for quite a while. I started tracking my hours in October of 2005, but I really started this effort a few months before that. I generally try to work on it every day, but the realities of work and family mean it's much less often than that. Still, I see I've put in over 100 hours on it. If nothing else, that makes me feel pretty good about having "paid" for all the open source software I use. :)

 

There's a metric boatload of things to be done before FlexWiki 2.0 is ready to go, of course. I think my next few months will look something like this:

 

1. Fix the 10,000 bugs I introduced.


2. Develop a performance test suite to characterize the behavior of the application.


3. Run the test suite.


4. Analyze the results.


5. Make improvements, including the caching provider that was part of the original reason for doing this massive refactorization.


6. Goto 3 a lot of times.


7. Write the security module that got me started on this hellroad.


8. Goto 1 a lot of times.


9. Release!


10. Goto 1 a lot of times.


11. Take a break from FlexWiki. Maybe to work on FlexWikiPad. :)

 

Still, this really feels like turning a corner. Yay me.