Measure, then Optimize

I posted yesterday about this article, which talks about the performance of various operations in the CLR. I said it was a good article, and I still think it is. But Ian Griffiths wrote me up to take issue with the fact that that's all I said - he felt that the article in and of itself does not actually tell you anything directly useful...and I agree.

Ian and I have both done more than our share of optimization, and we've both arrived at the same set of rules:

1. Don't do anything intentionally stupid when first writing the code, but 
2. Don't spend a lot of time trying to write really fast code up front. Instead,
3. Measure, then optimize the slowest thing.
4. Repeat until performance is good enough.

These rules will hardly be surprising to anyone that's succesfully done performance improvement work. But they surprise the hell out of a lot of people nonetheless. "I thought writing fast applications was all about knowing which sorting algorithm to use and which data structure to pick?" Not really.

I used to work at a mortgage company where I sat next to Jon, a good friend of mine. People would often come to me with a C++ program and tell me that it was too slow. The first question I would ask them is, "Have you talked to Jon yet?" When they said, "No," I'd tell them to go away. See, Jon used to work at Oracle, developing bits of the database, so he knew SQL up and down. He would routinely make queries that originally ran in half an hour, run in 30 seconds. As a pretty good C++ programmer, I could expect to decrease execution time by about 10%, or maybe 25% if I was really lucky. A sixty-fold performance improvement was out of my league...but Jon did it all the time.

As another example, while profiling a web service I've been writing, I found that the following line of code was the slowest thing in the app:

RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(cp);

And I mean it was by far the slowest thing in the app - making this call less frequently had a significant impact on throughput. I don't remember what the timing on the call was exactly, but I never would have guessed that a constructor call could take as long as this did.

All this goes to explain why I claim that the article is useful, but not directly useful: because when it comes time to optimize, you have to measure the slowest thing and fix that. Anything else is a waste of time. And you'll probably be surprised by what the slowest thing is. And it's likely not going to be slow due to any of the things from the article - at the point where you're worrying about the performance of fields versus properties, you've almost certainly already optimized a whole bunch of other stuff that's going to be the dominant factor. But at the same, there may come a time when knowing that the C# as operator is twice as slow as the cast operator in some situations might save you a few hours.

The life of a performance optimizer is a tough one: you have to know everything about how the app works, down to the silicon, since you never know what the bottleneck is going to be. (This is an outcome of the Law of Leaky Abstractions.) But since this is waaaay too much to keep in your head for even a trivial app these days, we need to just make our best guess when first writing the app, then measure to zoom in on the places where bad things are happening.

Oh, and don't forget step #4: stop optimizing when performance is good enough - you're just wasting money after that. Of course, that assumes that you can get a definition of "good enough" from the customer, but maybe that's a post for another time. ;) In the meantime,