AppEngine Thoughts, or “How to support a cloud” | Kyle Baley

:

AppEngine has new pricing as of November 7 which has generated much discussion, most of it as exciting as deciding which wine to serve with raccoon (answer: dandelion). So with the segue established, I shall pontificate on what I believe is the single biggest thing keeping it from being a truly great product and a worthy competitor in the cloud hosting space.

First I’ll get out of my system a laundry list of lesser issues:

1. Confusing pricing/configuration

This could be a product of my Microsoft indoctrination but I still have trouble figuring out how we get charged for stuff. It’s based on usage which is a great model on paper. But the usage is broken down into, among other things:

  • Front-end instance hours
  • Back-end instance hours
  • Datastore Write Operations
  • Datastore Read Operations
  • Datastore Small Operations
  • Stanzas Sent
  • Channels Created

Furthermore, you can tweak things like the minimum and maximum idle instances and the minimum and maximum pending latency.

All this requires a lot of research and testing. And the only way to test various settings is: a) in production, or b) using load testing with a separate paid environment. Both of which will end up costing you dollars. But it must be said, those dollars probably won’t add up to the cost of setting up your own test environment. So far, we spend about the cost of a smoothie every day for all versions of our app (depending on where you find the ingredients).

2. No built-in backup and restore

Backing up the datastore is your responsibility. Now, it’s easy to set up a batch process running nightly. But you’re almost guaranteed to hit your Datastore Read Operations quota on a nightly basis once you reach a certain size. Also, the new-fangled High Replication datastore makes things more interesting by not actually being supported for this scenario.

3. No built-in reporting mechanism

The only way to interact directly with your data is through code you’ve written yourself or with GQL, a SQL-like language for the datastore. But you quickly hit limitations. First is that you can return only 20 results at a time (which you can up to 200 by adding a limit parameter to the URL…manually). Second is that you have to create indexes for fields you want to filter or order by. Doesn’t lend itself to ad hoc querying.

4. Admin console bugs

This one irks me almost as much as the fact that I have to restart my computer whenever I update the glorified Notepad that is Adobe Reader*. Too often for my tastes, when clicking around the console, I’ll get a general “An error has occurred” page. And the error page is not a pretty one:

AppEngine Error

Even when it does work, until recently, I had to use Firefox or Internet Explorer to access one of the pages (Datastore Admin) because it didn’t load in Chrome.

Which leads me to the point of this post. The one killer issue in AppEngine that keeps its status below world class hosting environment:

Support as a second-class citizen

If you didn’t notice at first glance, take another look at the error page above. In particular, at the URL at the bottom, which is where the “report” link goes. It’s a Google Group page for AppEngine. To their credit, they’ve addressed many of the issues I’ve had with Google Groups recently. Even so, for a world-class hosting environment and especially for apps we’re paying for, I’d much rather see something like this.

Also, my experience with the group site has been pretty dismal. Much of the time, my questions get not a single response except mine. Even reporting issues on the Google Code site leads to sporadic responses. I get slightly better averages on StackOverflow.

This came to a head for us recently when we converted from Master/Slave to High Replication late one Saturday night. Several of the problems I outline here occurred that night, including the 1990s error screen above. And I couldn’t find a single email address or phone number anywhere that I could contact for help and be assured of a response with even an unreasonable time. I made a post to the forum about an issue we saw the next day. The link is included above in the list of posts that have received no response.

Another support-related area that could use work is the roadmap. Something I’ve noticed in all dealings with the AppEngine team is an almost fanatical abhorrence of delivery dates. Even at Google IO, the best we could get from the team was “we’re working on it” with a lot of nervous laughter when someone asked about SSL access on custom domains, which is one feature we’ve had to make decisions around. I gather it’s a hard problem but even if they said “probably 2012”, that would at least indicate to us “okay, it’s not anytime soon, time to decide which is more important in the short term.”

Had I written this post a couple of months ago, after our High Replication migration, it would have been a lot more acidic in tone. For a few weeks after that, I was actively checking out Amazon Web Services. A Microsoft rep reached out to us serendipitously that same week wanting to talk to us about Azure. If he had not dropped the ball and postponed our meeting at least three times, this here blog thingy might be back in .NET land by this time. (Likely not, but this wouldn’t be a blog if I didn’t make grandiose unsupportable claims that back up my argument.)

These days, I’m not so much frustrated as I am disappointed. For all its faults, there is a lot of good being offered by AppEngine. Like many startups, we have discussions about Google and AppEngine and the general consensus is that we’re happy to see Google focusing on what they’re good at. (Do I lose points if I point out my subtle digs?) But when it comes to support for AppEngine, it feels like it’s run by engineers for engineers. Yes, support is boring and customers can be confrontational and much of the time, the answer is a variation of “you’re doing it wrong”, etc, etc. But it’s not just another IoC container; it’s a cloud hosting platform. I believe it needs a higher level of professionalism than what I’ve seen so far.

Kyle the Untenable

*But not nearly as much as the fact that said update invariably adds an icon to my desktop. C’mon Adobe, who actually opens Reader and then opens a PDF file?