Node.js on the Road: Node.js @ Yahoo

How Node.js has united and changed the Front End Developer culture at Yahoo.

Eric Ferraiuolo Principal Software Engineer, YUI

I want to first answer the question of, "does Yahoo use Node.js in production to serve web pages?" And the answer is yes, we definitely do. We've been actually doing this since Node 0.4, so for a long time. And one really interesting thing is like when we experienced the 0.8-0.10 transition, it was very smooth for us, and that really showed us that the platform is really ready, like we were pretty early adopters for it.

But when we saw that transition, we knew like this stuff is getting pretty solid, and so we started putting a lot of stuff on it. A lot of the new Yahoo-based products that you've seen, our properties, digital magazines, we have Yahoo Tech, David Pogue is now at Yahoo, writing for Yahoo Tech; and Yahoo Food, and these are more advanced single page applications, but also work as content sites, it's a nice hybrid.

Those are being served through Node. Yahoo Answers, famous for awesome answers, uses Node on some parts of it. Yahoo Screen, which is Yahoo's video playing property who now I guess can stream everything from old SNL clips, which is pretty awesome, that's all running on Node. But then we also have other stuff that you guys are probably really familiar with which is like Flickr.

So Flickr is going through re-writing a lot of the stuff that they have, they actually are in a state where they have a hybrid application. Some of it is in PHP, but some of the new stuff, especially the photo page which is the most highly trafficked page on Flickr to show a single photo, that's in Node. And then there's a lot of other components that you see on other pages that are being rendered from a Node process that are intermixed into the Yahoo homepage, and Finance, and these sorts of other sites.

So, we're using it a lot. We're pumping through about 25,000 requests every single second through Node, through the servers that we have. And this is still a minority of Yahoo's traffic. So we're definitely pushing it. We're pumping lots of request through Node.js, and so the questions are, "oh, does Node scale. Is it performant?"

Yes, it definitely is. You can make it scale, and it's very performant, and every property that we've moved over to the Node.js stack has seen an increase in performance, so we haven't moved something over and then saw it dip. And the way that it's set up, it's very easy. The architecture that you can create with Node.js

makes it very easy to scale horizontally, which we definitely have to do at Yahoo.

So what are we using it for?
Well, the most common thing that I'm going to talk to you about is application server. So we're serving dynamic webpages out through Node.js servers, where there's a bunch of developers writing, a lot of times they're using express for a lot of these as well, so it's like express powers tons of Yahoo which is kind of crazy to think about from a few years ago, and people are writing routes, and they're writing templates with handlebars, and stuff.

So it's what all of us are used to. I do a lot of opensource development, I use a very similar stack that Yahoo uses. I even wrote my wedding website using Node.js and opensourced a lot of parts, and now Yahoo is using those packages. It's kind of cool. What's that?

[audience] Yahoo is getting married?

No, no, no, for my wedding website. And we're doing a lot of interesting stuff now with server-client coordination. So we're trying to be really fast at rendering an initial page view on the server, but then giving everything to the client's side of the application to take over from there and really run the show.

This is how you can move between photos on Flickr hitting the arrow key, and they're just right there. We're not doing full page reloads. We're trying to use the best of both worlds from the single page app plus rendering an initial view on the server, and Node.js is making this really awesome to do.

We're also writing a lot of dev tools, so being a front end engineer, it was like alien to me, how do I write a simple dev tool? What is make? What are those tabs? How does that all work? We have a lot of itches we want to scratch, we're using tons of JavaScript on the client side now, it's like how do we build this stuff concatenate it, minify it?

We wanted that in our control, so now we're just writing tons of tools—command line tools—we're leveraging a lot of like grunt tools, build system of the month tools, whatever. We are able to use, and what's awesome is like there're some people at Yahoo that are still using Windows and like developers and when they have to do Windows it's like hey, our tools work, and they appreciate that. So we're also doing a lot with continuous integration and continuous deployment, so we have this entire system set up, you can think of it like Travis and Heroku-like stuff that we have internally and all this stuff that we wrote here we rewrote using Node.js.

So a lot of this internal infrastructure about deployments, doing testing, doing code coverage, all these kind of things are all tools written in Node. I don't know, are people doing testing here of their JavaScript code?

Yeah, like not very many.

[TJ]Please do more, please do more.
So there's like a pretty good code coverage tool that is called Istanbul, and the way that it's different—I'm not sure exactly where the name comes from, so don't ask me—but the way that it's different is that it does branching coverage, so it will check it uses Esprima, does a parse AST, and then crawls that around and makes sure that like every code branch, so if you have an if statement, was the else condition met, and tested, and all this stuff. So the guy who wrote that is actually the guy who wrote our internal testing tool, that like all the Yahoo projects go through, so that's pretty awesome. That's all in node on NPM, Istanbul.

And yeah, so I
could go a lot deeper into like how Yahoo is using Node, what our internal infrastructure looks like.

But what I realized is, there is a different thing that I want to touch on here, and this is how Node.js has actually changed the developer culture within Yahoo, especially from front end engineers, and there was this post last year by Nicholas Zakas, I'm sure most of you probably heard of him, he actually used to work at Yahoo, and he talked about this idea of Node.js as the new web front-end, and there's this quote here, I think I'd like to just read it real quick, it really captures what I'm trying to get at, it really puts words to what we were doing.

So this is from Nicholas Zakas, he was saying in his blog post, "To me, Node.js was never about replacing everything on the server with JavaScript. The fact that you can do such a thing is amazing and empowering, but that doesn't make it the right choice in every situation. No, to me, I had a very different use in mind: liberating the back-end UI layer from the rest of the back-end", and that's really what we were realizing we were doing, and loved that he wrote this blog post, you can find it on his website. And what essentially he's showing here is this idea that, yes, let's as Front-end developers, let's take control of our part of the server side and let's write it in a common language, let's use JavaScript, let's use Node to power this. Because we're writing JavaScript in the browser anyways, but a lot of times for a lot of applications there's this like back-end side or the server side of the front-end, and whatever is going on in API land whether it's Java or maybe Node.js as well or whatever,

we just don't really care, because we're just a consumer of that backend API, and there's the back-end engineers, and the database engineers who are really good at that stuff. We should let them use whatever they want, but we don't have to go to them and say, oh, can you help us write out this piece of content in this HTML in your JSP page or something?

So, we want to take control of that, and that gives us a great opportunity to also expand our skill sets. We can do a lot now. We have control of our part of the server. And we can be self reliant on implementing the types of features we're trying to do. And Node.js has actually made this possible I think, because of how easy it is to be successful with it.

I copied this code from the Node.js website on the home page, and from this little bit of code, you can create an HTTP server, and this is so powerful. You can just paste this into a file, and run it with Node, and you have an HTTP server. That's super successful, very early, and this is way less code than I would write for a normal button click handler in the browser.

And you're like accomplishing something that you weren't really able to do before as a front-end engineer, and I think that's huge. That was very big for me personally, to be able to do stuff like this, and feel like I know JavaScriot really well. I love that I can use it in this new way. A criticism of this is, well if you start doing everything in Node, then you're going to have to rewrite everything, reinvent every single wheel that there is?

But I think that there is another side of this which is, you now have this clean slate. You can rethink a lot of the things you were doing before, especially for new projects, we all do this anyways, we rethink our architectures a lot, and think about where we can learn new things or try out something new.

And Node.js gives us that opportunity, and I think that that's also very powerful. But then we also get to leverage tons of open source software. So open source and Node.js and the Node community are very synonymous with each other almost. We have NPM with 60-70,000 modules, packages on there that you can just install and start using.

So that's also very powerful. So not only can you contribute to this community, you can use everything that's there, and it gives you a chance to try out new things, rethink ideas, rethink architectures, and that's something that I've found to be very powerful, and a lot of other people at Yahoo did as well.

Essentially the way I think about it, it's like a power up for us frontend developers. We are now powered to be able to do these things on our own. That comes with responsibility, so we have to be good with like cross-site scripting stuff, avoid all those things, be good about that, but learning about that is very good.

It allows engineers to grow. It allows you to go from a junior engineer to a senior engineer and onward, and I really think that this is a big difference that we're seeing, and it gets everybody on the same page, like we're writing JavaScript now. We know JavaScript if we're writing web applications because you kind of have to, and it allows us to move a lot faster.

I have a concrete example of this with Flickr. I've been working closely with the Flickr team, and they have this PHP/Node hybrid app that I was talking about. They're trying to transition all their stuff over to Node over time, but they're taking a very pragmatic approach to doing so. Like one example here is the comments feature, so you can add comments to a photo on Flickr, and what was really interesting when I was talking to them about this, is they have a back end API for dealing with all the data stuff, but they want to render it on the server, they want to render it on the client, they want to be able to handle adding new comments, and all this stuff, right?

They're like, yeah, this one guy will go over and implement that feature, and it's like whoah! that's crazy. So this one person, this one engineer can go over and implement the entire feature from the server side part of it, communicating with the back-end API to the user interaction on the browser, and to me that was like an aha moment, that this is very powerful. It really is giving people the ability to sort of like be this self-reliant, implement these features, and Flickr's even taking it further where a lot of their leaf features, are these like end features that they have in their stack, they're writing it in isomorphic ways, so the same code that's running on the server is running on the client.

So this person is just implementing code that is talking to the APIs the same way that it is on the server in the Node runtime as it is running in the browser. And that's allowing them to move very quickly with these features that they're trying to implement. And I think it's a great example, and what we've seen in the past at Yahoo, these are three of the major organizations development, like big groups of developers within the company. We have media which is like, and everything, those sites. We have Search, which is the oddball out of that one, and it's on it's own. So these used to be all on their own stack, and Flickr on its own stack, and they were sharing some things, but they were all having their own frameworks. This has been the case, I think from what my manager told me, for like ever at Yahoo,

they've never all been on the same platform.

So my team, which is a platform team, we really wanted to get all these teams in the same stack, and they all wanted to use Node.js, everybody was experimenting with it, and so we were able to do it. They are all now sitting on Node. There's some common and open source components that are sitting on top of Node that we helped work out with them.

And what we wanted to do from our team's perceptive is like look across, what are each of these teams doing, and realizing there's a total set of common problems here, we know you guys are doing this. We talked with the search team, we know they're doing this. Talked with the Flickr team, they're trying to figure out the same thing.

So we said to them, let us solve those problems for you, and we can then do it and provide this common layer. And then each of these applications, or sets of applications need a slightly different framework for everything they're trying to do, or they just have a different way that they they're engineering things internally in the teams.

In this way they can build up from there, and really focus on their features, and what they were trying to build, while we could, at the platform level, really make sure that these lower pieces on top of Node, we're very solid for them to use. And this has caused like really great collaboration now between everybody in these different organizations within the company, and really building this community around Node.js, around JavaScript, just us, frontend engineers really getting together more, and talking about more things, and figuring out things together.

And you know what I've really come to find out is that, Node.js has really unified and changed the frontend developer culture at Yahoo. And I think that that is really awesome and a testament that it's a great platform, and we definitely love using it here at Yahoo. So, thanks.

Sign up now for Instant Cloud Access Get Started