Blog Joyent

Cloud Computing Infrastructure for Facebook Developers on Dell Servers. What does it mean?

[Note: I’m going to be writing some longer posts about the needs of a cloud computer. This first post details the need for reliable supply. Please take this in the spirit it’s offered: a survey of one company and its experience in the market.]

I wrote recently that the operating system doesn’t matter anymore for developers of internet applications. Joyent’s announcement last week that we would be providing free cloud computing infrastructure to Facebook developers is further evidence that this is true. We spent a considerable amount of time optimizing Joyent Accelerators for Facebook, not for OpenSolaris (the OS we use). A Facebook developer can sign up for a free Joyent Accelerator here, get their application up and running within a few minutes, and be on their way. We did this a couple times with a couple customers last week at the Dallas Facebook Developer Garage. One of them had been kicked off their hosting provider because they were actually using some of the “free” bandwidth that over-selling hosting providers market. They were up and running on Joyent Accelerators within minutes of getting their log-in. The operating system didn’t matter in the process.

Joyent the Meta-manufacturing Company: We Need Stuff On-demand

If the operating system doesn’t matter to developers, the server under the operating system shouldn’t matter either. In fact, that is true to the extent that there are many makers of server hardware and they are mostly interchangeable. But in another way, the servers does matter to developers. Can the server hardware manufacturer your development stack runs on scale at the velocity of your application? While most Joyent customers don’t worry about whether we have enough power and cooling, they will need to worry, at a point, whether we have enough CPU, RAM, and storage. And, since Joyent doesn’t make any of those consumables, we have to turn to businesses that do. Joyent is, in this regard, a meta-manufacturing company. We manufacture a compute cloud upon which developers can run web applications and scale them. Our Facebook offering, and the scale issues that can face developers on Facebook, meant that we had to have parts suppliers for our cloud computer that could also scale.

A Brief Excursus: What Happened with Joyent and Sun?

When I got off the stage last week at the Dallas Facebook Developers Garage having made the announcement about Facebook and Joyent and Dell, one of the consistent questions was: “What happened to your relationship with Sun?” We continue to use Sun technology for critical parts of our infrastructure. The most obvious is our choice of Solaris Zones as virtualization building block of Joyent Accelerators. We continue to believe this is a better choice than embracing Xen (though others don’t always agree, ironically). This doesn’t mean we’ll never use Xen. We still don’t support .NET, a technology that is officially supported for Facebook developers. Xen would help with that. But so would VMWare.

One problem with Sun continues to be the sales model. In order to ensure dependable supply, we had to sign-up to buy large numbers of servers whether we actually used them or not. Sun put the risk onto the customer. We came to the conclusion that Joyent can’t buy from Sun if Joyent can’t buy direct from Sun. Faced with the tsunami that the Facebook opportunity represented, we couldn’t/didn’t know how big a pre-buy to make. It was too big a risk.

This on top of the fact that we wanted to buy the new Sunfire X4150 (dual socket, quad-core) but nobody in the channel (that hated word, “channel”) could tell us when we could get them. This for a model that had been announced weeks back. Why not just continue with the Sunfire X4100s we’ve been using. Well, for one, the X4150s allow us to cram tons of storage into the server, thus side-stepping many of the iSCSI issues (target) we have had with OpenSolaris. It’s one reason Joyent has been buying NetApps. And, again, the Facebook opportunity meant we would be building out significant infrastructure. To put it into the context of a systems manufacturer: our servers are Joyent’s CPU. We didn’t feel like installing the 586, when the 686 was freely available on the market.

Jonathan Schwartz has said these problems would be fixed, but they haven’t. And we don’t see anything to indicate that they will. We’ve talked with all sorts of Sun sales people. They put us into a special group for internet companies. We have made personal appeals to senior executives at Sun (that generally are answered…thanks for that). We’ve passed out bottles of 18-year old scotch. But the fact remains: every time Joyent engages Sun sales, they can’t really sell me something. The channel gets in the way. This is unfortunate, and ironic, for a company that did $1 billion in direct sales in a year (within three years of being founded).

So we called Dell

We had test systems FedEx’d to us and confirmation that Joyent would be able to run our stack on their stack within 36 hours. Once we had given the green light, the systems were in our data centers in two days. And OpenSolaris is supported on these systems according to a recent news item. We have a direct sales representative at Dell who is amazing. We don’t have to work the Dell organizational chart because our sales representative just gets things done. There’s not much to say about the Dell relationship because it is drama-free. Joyent’s relationship with Sun wasn’t. A cloud computing company needs reliable, drama-free supply. It’s that simple.

Update (Nov 24)

I got this interesting anecdote from a reader:

I found your recent acquisition of a large batch of PowerEdge 2950s to be quite interesting. We’re a smaller Dell customer, about 10 servers per month (SC1435 and 2970) and have made two attempts in the last 18 months to purchase (or at least get some pricing) on Sun servers. Both attempts including one about a week ago led to unanswered emails and the eventual passing us off to a VAR, which we specifically asked for direct sales when we spoke to a front level sales associated at sun.

Just trying to get a sales associate at sun to give us a similar system to our Web/Database builds was like pulling teeth, we were told to sign up for sun startup essentials before we could get into conversation. Three weeks later one of their VARs from Texas (we’re in Portland) was knocking on our virtual door.

My assumption was that our purchasing volume was just too low for the interest of direct sales from Sun, however, your comments on the Joyent/Sun situation seem to indicate it being more of a culture issue in the realm of Sun sales management.

Good luck with your new hardware and keep up the great work at Joyent.