The MTPS Content Service - A SOAP Success Story

When I wrote the MTPS Content Service (was that a year ago already?) I didn't really expect people to use it much. I figured there'd be a few small and interesting apps - for instance, I wrote msdnman mostly for fun - but that it would largely be a useful but little-used service.

 

I was quite wrong.

 

The other day I found out the usage numbers for the service. The conversation went something like this:

 

Kim: "Hey, I wanted to tell you the stats are in for the content service."

Me: "Oh really? I'm curious - what are they like."

Kim: "Here, I'll email you the numbers."

Me: "Wow! It's only been a year and there's been 1.3 million hits - that's a lot!"

Kim: "No, that was for yesterday."

Me: "…"

 

I was quite literally speechless. Which, if you know me, you know is a rare thing. :) That's fifteen requests per second, so perhaps I can be forgiven my momentary surprise. (Traffic has since subsided to a mere 600K requests or so per day - an average of around seven requests per second.)

 

I have to say I'm fairly proud of having written a web service that can sustain those sorts of loads. Of course, the credit is not due all nor even (close to) mostly to me: the service is a fairly thin wrapper around the MTPS system, and I was only one part of the team that did that. Plus, without a crack ops team it doesn't matter how good your software is - you're not scaling. Kudos to those guys as well. All I had to do was not screw up the translation to SOAP. :)

 

We don't really have good data on what applications are creating all this traffic. We do know that the recently-released PackageThis is responsible for somewhere around a third of the traffic. We know that because PackageThis is the only app currently sending an AppID header with their requests. An AppID header is just a short string that identifies the application making the request. This is something we added rather late in the development cycle, never really documented, and made optional. So maybe it isn't surprising that it's not showing up in a lot of the requests. Hopefully we can fix that at some point.

 

Sandcastle also uses the service, and I wouldn't be too surprised if it accounts for the bulk of the remainder of the traffic. They're adding an AppID header soon, so we'll have better data then. It certainly will be interesting to see - personally I'd love it if there were still a whole bunch of unknown apps in the traffic pattern, as it would suggest that lots of people have found good ways to use the service. Which was our hope when we wrote it.

FlexWiki 2.0 Security Bug Alert

I noticed something weird on one of the wikis I’ve upgraded to FlexWiki 2.0. Even though I’d locked down a namespace to only allow authenticated users to edit, I was still seeing new topics getting created. Sure enough, when I tried it myself, I was able to create a new topic even though I wasn’t logged in. Fortunately, the problem only seems to manifest with new topics: editing of existing pages is still correctly prevented by the security provider.

 

After a bit of digging, I figured out that the problem is with the way permissions are handled for nonexistent topics. Basically, users were granted full control over nonexistent topics. The correct behavior is for nonexistent topics to be given the default permissions for the namespace, as once they are created, that’s what they’ll have (absent explicit permission statements). I’ve coded the fix and submitted it – it’s present in build 2.0.0.49 and forward.

 

Note that the fix makes the wiki secure by ensuring that unauthorized writes can't happen, but that the UI is still somewhat wanting: You're not told that the write is going to fail beforehand. I'll make that change soon. I just wanted to get a patch out to solve the underlying problem as quickly as possible.

 

If you’ve deployed FlexWiki 2.0 you should seriously consider upgrading to this latest build. Available here.

Microsoft vs. TestDriven.Net Express

TestDriven.NET is the only Visual Studio extension I ever use. It's fantastic. So I was really bummed to read this, which tells about the legal trouble Jamie has been having with Microsoft over his truly useful product.

 

I'm no legal expert, so I have no idea which side is "right" here (assuming that it's even as black and white as that). All I know is it would suck if Jamie had to stop making TestDriven.Net, or even got so annoyed that he didn't feel like working on it any more, and that he has my sympathy. I've been in situations where legalities influenced technical decisions in distasteful ways many a time, and it always offends my sense of software aesthetics.

 

I wanted to blog this because I think people should be aware of it. It's days like this I seriously consider ditching as much proprietary software as I possibly can and living the free and open software lifestyle for real.

The Secret To Software Estimation: Lie Like Scotty

I see that Scott Hanselman posted about software estimation. Excellent advice, as usual. Here's my somewhat less serious advice, in the form of a quote from "Star Trek III: The Search for Spock":

 

Kirk: "How long to re-fit?"

Scotty: "Eight weeks. But you don't have eight weeks, so I'll do it for you in two."

Kirk: "Do you always multiply your repair estimates by a factor of four?"

Scotty: "How else to maintain my reputation as a miracle worker?"

Kirk: "Your reputation is safe with me."

 

I always multiply my initial gut-reaction estimate by four. So if I think it'll take a day, I tell the client it will take a week. If I think it'll take a week, I tell the client it'll take a month. This works really well for a couple of reasons:

 

1. My initial gut-reaction estimate is almost always wrong by a factor of two or three.


2. When I tell the client it's going to take a week and it takes three days, they're thrilled.

 

The funny thing about all this is that it still works even if I tell the client what I'm doing beforehand. Weird but true: I tell them a week, they know I actually think it'll take a day, but they're still happy when it's done in three.

 

Of course, this really only works after I've established a track record of delivering solutions to hard problems. And my miracle worker reputation gets a bigger jump from not telling them beforehand, so I usually don't. Except for the ones that read this blog.

FlexWiki 2.0 Beta 1 Released

I've just posted the latest FlexWiki bits on SourceForge. Get 'em here.

 

I seem to say this every few months, but this is a pretty major milestone on the long, long road to FlexWiki 2.0. Specifically, this is the first release where essentially everything from FlexWiki 1.8 works again. So the RSS feeds, the newsletters, and the administration pages have all been repaired, which is nice. The one thing that's still broken is the SqlProvider. Since most people use the FileSystemProvider, I don't think that's a huge deal. I'm fixing the SqlProvider even now, and it will be present in FlexWiki 2.0 Beta 2.

 

The major new feature in Beta 1 is the security infrastructure. FlexWiki now sports topic-level security. That means you can put something like this

 

AllowRead: user:candera

DenyEdit: anonymous

 

in a topic and it'll work. You can also do this sort of thing on a per-namespace basis, or wiki-wide. It works with either Windows or Forms authentication, and I've even thrown in a simple implementation of some login/logout pages for people that want to get something simple set up for Forms authentication.

 

If I seem excited about the security stuff, it's because I am: the whole reason I ripped FlexWiki apart (starting nearly two years ago!!) was to make it possible to add security in a maintainable fashion. Now it's done.

 

You can read more about the new security features here.

 

As far as the road ahead, there are really only two tasks remaining before I can shove FlexWiki 2.0 RTW out the door:

 

1. Fix the SqlProvider.


2. Analyze and tune performance.

 

I've already started the SqlProvider work, and I don't expect too many problems. Performance could be easy or could be hard - I'll need to start by comparing the performance of the 2.0 bits with the 1.8 baseline. That'll tell me how much work I have to do.

 

But any way you slice it, the release date for 2.0 RTW is a whole lot closer than it used to be.

Thinking Like Tim

I talk to Tim on a regular basis. So when he outed himself as a REST convert lately, it actually wasn't a huge surprise to me - we've been talking a lot lately about what the last few years have taught us about web service programming, and how more and more SOAP/WS-* just doesn't seem to be all that and a bag of chips. Still, the key part of the analysis in his latest post - that you can use HTTP as a way to publish application states and the transitions between them - really struck me as something significant, and brought a lot more clarity to the whole problem.

 

Here's where my thinking is at: Basically every SOAP web service I've written in the last four years could have been done just as well as a straight HTTP interface. And a several of them would have been significantly better. The ones that couldn't are very clearly RPC style, for better or for worse. I've been feeling this especially acutely lately as I try to optimize one of the services I've written. I found myself implementing HTTP over SOAP, which just seems stupid, not to mention clumsy.

 

For me it's really not about whether the thing is "REST" or not. I find the term REST to be too laden to be useful. I think more in terms of "SOAP via toolkits" (SVT?) versus Plain Old XML over HTTP (POX). After all, I'm perfectly willing to use a SOAP Envelope as long as I still get to use HTTP GET. But at the end of the day, ignoring particular terminology, I agree with Tim's analysis: SVT is for stuff I'd use DCOM for in the old days, not for putting up a large-scale, public-facing, web-based, programmatic interfaces.

 

I realize that programmers love their tools and toolkits, but honestly, when you build big giant systems sometimes you have to do things the "hard" way. It always surprises me when my clients complain that complex problems contain complexity in the solution. But in any case, I'm not even convinced that throwing away toolkits like WCF will raise the complexity level - there's a lot to be said for being closer to the wire. Plus, things like Linq for XML look really tasty when you consider how they might help with this style of programming.

 

I do really like the analysis of REST as Representational State *Transition*, as opposed to the T being for "Transfer". To me, viewing the design exercise as figuring out how to expose a connected state graph (i.e.

both transitions and states are explicit) brings a lot of mental order to what I already knew I wanted to do anyway. In particular, it helped me understand how a POX interface is useful for writes...because for reads it's dead obvious: GET kicks ass, especially if you care at all about performance. For a long time I was hung up on trying to figure out how to use SVT on write, and GET on read, because that seemed to make the most sense. But if the thing is just your application state where each state has an address, it's pretty obvious that it's okay to POST to a URL to cause a state transition. Or even for that state transition to significantly alter the URL space.

 

Of course, I have yet to build a large, real system using this new approach. I'd love to, though, because more and more it seems like the right thing. And no battle plan survives first contact with the enemy - witness SVT. The big question is whether the big concepts will prove out for me personally. Hence my eagerness to put my customers' money where my mouth is :).

 

And before you point it out in the comments, I should say that I'm aware of the efforts to make WCF more amenable to straight HTTP-style programming. It's just that I'm dubious about the viability of exposing transport semantics explicitly via an API that's all about abstracting transport semantics. It seems to me that if you're going to remove a layer of abstraction, you should actually remove a layer of abstraction. But again, I'll need to build something real with it before I can draw any supportable conclusions.

Converting Messages to Text in Outlook 2007

In Outlook 2003, you could open an email message and convert it from HTML to text by hitting Ctrl-Shift-O. That was handy, because I generally prefer to bottom-quote, and that's a pain in the ass when the email is HTML. So when I installed Outlook 2007 I was annoyed that Ctrl-Shift-O didn't work any more.

 

After a bunch of fumbling around, I finally discovered that if you open a message and edit it (Message->Other Actions->Edit), you can then change it to plain text (Options->Plain text). Well, being a keyboard guy, I got pretty tired of doing that, so now I have the following key sequence committed to muscle memory:

 

Enter (open message)

Alt H X E (edit message)

Alt P L (convert to plain text)

Alt H R P (reply)

 

Which, honestly, sucks, but is maybe 100 times as fast as doing the same thing with the mouse.