I want to first answer the question of, "does Yahoo use Node.js in production to serve web pages?" And the answer is yes, we definitely do. We've been actually doing this since Node 0.4, so for a long time. And one really interesting thing is like when we experienced the 0.8-0.10 transition, it was very smooth for us, and that really showed us that the platform is really ready, like we were pretty early adopters for it.
But when we saw that transition, we knew like this stuff is getting pretty solid, and so we started putting a lot of stuff on it. A lot of the new Yahoo-based products that you've seen, our properties, digital magazines, we have Yahoo Tech, David Pogue is now at Yahoo, writing for Yahoo Tech; and Yahoo Food, and these are more advanced single page applications, but also work as content sites, it's a nice hybrid.
Those are being served through Node. Yahoo Answers, famous for awesome answers, uses Node on some parts of it. Yahoo Screen, which is Yahoo's video playing property who now I guess can stream everything from old SNL clips, which is pretty awesome, that's all running on Node. But then we also have other stuff that you guys are probably really familiar with which is like Flickr.
So Flickr is going through re-writing a lot of the stuff that they have, they actually are in a state where they have a hybrid application. Some of it is in PHP, but some of the new stuff, especially the photo page which is the most highly trafficked page on Flickr to show a single photo, that's in Node. And then there's a lot of other components that you see on other pages that are being rendered from a Node process that are intermixed into the Yahoo homepage, and Finance, and these sorts of other sites.
So, we're using it a lot. We're pumping through about 25,000 requests every single second through Node, through the servers that we have. And this is still a minority of Yahoo's traffic. So we're definitely pushing it. We're pumping lots of request through Node.js, and so the questions are, "oh, does Node scale. Is it performant?"
Yes, it definitely is. You can make it scale, and it's very performant, and every property that we've moved over to the Node.js stack has seen an increase in performance, so we haven't moved something over and then saw it dip. And the way that it's set up, it's very easy. The architecture that you can create with Node.js
makes it very easy to scale horizontally, which we definitely have to do at Yahoo.
So what are we using it for? Well, the most common thing that I'm going to talk to you about is application server. So we're serving dynamic webpages out through Node.js servers, where there's a bunch of developers writing, a lot of times they're using express for a lot of these as well, so it's like express powers tons of Yahoo which is kind of crazy to think about from a few years ago, and people are writing routes, and they're writing templates with handlebars, and stuff.
So it's what all of us are used to. I do a lot of opensource development, I use a very similar stack that Yahoo uses. I even wrote my wedding website using Node.js and opensourced a lot of parts, and now Yahoo is using those packages. It's kind of cool. What's that?
[audience] Yahoo is getting married?
No, no, no, for my wedding website. And we're doing a lot of interesting stuff now with server-client coordination. So we're trying to be really fast at rendering an initial page view on the server, but then giving everything to the client's side of the application to take over from there and really run the show.
This is how you can move between photos on Flickr hitting the arrow key, and they're just right there. We're not doing full page reloads. We're trying to use the best of both worlds from the single page app plus rendering an initial view on the server, and Node.js is making this really awesome to do.
We wanted that in our control, so now we're just writing tons of tools—command line tools—we're leveraging a lot of like grunt tools, build system of the month tools, whatever. We are able to use, and what's awesome is like there're some people at Yahoo that are still using Windows and like developers and when they have to do Windows it's like hey, our tools work, and they appreciate that. So we're also doing a lot with continuous integration and continuous deployment, so we have this entire system set up, you can think of it like Travis and Heroku-like stuff that we have internally and all this stuff that we wrote here we rewrote using Node.js.
Yeah, like not very many.
[TJ]Please do more, please do more. So there's like a pretty good code coverage tool that is called Istanbul, and the way that it's different—I'm not sure exactly where the name comes from, so don't ask me—but the way that it's different is that it does branching coverage, so it will check it uses Esprima, does a parse AST, and then crawls that around and makes sure that like every code branch, so if you have an if statement, was the else condition met, and tested, and all this stuff. So the guy who wrote that is actually the guy who wrote our internal testing tool, that like all the Yahoo projects go through, so that's pretty awesome. That's all in node on NPM, Istanbul.
And yeah, so I could go a lot deeper into like how Yahoo is using Node, what our internal infrastructure looks like.
But what I realized is, there is a different thing that I want to touch on here, and this is how Node.js has actually changed the developer culture within Yahoo, especially from front end engineers, and there was this post last year by Nicholas Zakas, I'm sure most of you probably heard of him, he actually used to work at Yahoo, and he talked about this idea of Node.js as the new web front-end, and there's this quote here, I think I'd like to just read it real quick, it really captures what I'm trying to get at, it really puts words to what we were doing.
we just don't really care, because we're just a consumer of that backend API, and there's the back-end engineers, and the database engineers who are really good at that stuff. We should let them use whatever they want, but we don't have to go to them and say, oh, can you help us write out this piece of content in this HTML in your JSP page or something?
So, we want to take control of that, and that gives us a great opportunity to also expand our skill sets. We can do a lot now. We have control of our part of the server. And we can be self reliant on implementing the types of features we're trying to do. And Node.js has actually made this possible I think, because of how easy it is to be successful with it.
I copied this code from the Node.js website on the home page, and from this little bit of code, you can create an HTTP server, and this is so powerful. You can just paste this into a file, and run it with Node, and you have an HTTP server. That's super successful, very early, and this is way less code than I would write for a normal button click handler in the browser.
And you're like accomplishing something that you weren't really able to do before as a front-end engineer, and I think that's huge. That was very big for me personally, to be able to do stuff like this, and feel like I know JavaScriot really well. I love that I can use it in this new way. A criticism of this is, well if you start doing everything in Node, then you're going to have to rewrite everything, reinvent every single wheel that there is?
But I think that there is another side of this which is, you now have this clean slate. You can rethink a lot of the things you were doing before, especially for new projects, we all do this anyways, we rethink our architectures a lot, and think about where we can learn new things or try out something new.
And Node.js gives us that opportunity, and I think that that's also very powerful. But then we also get to leverage tons of open source software. So open source and Node.js and the Node community are very synonymous with each other almost. We have NPM with 60-70,000 modules, packages on there that you can just install and start using.
So that's also very powerful. So not only can you contribute to this community, you can use everything that's there, and it gives you a chance to try out new things, rethink ideas, rethink architectures, and that's something that I've found to be very powerful, and a lot of other people at Yahoo did as well.
Essentially the way I think about it, it's like a power up for us frontend developers. We are now powered to be able to do these things on our own. That comes with responsibility, so we have to be good with like cross-site scripting stuff, avoid all those things, be good about that, but learning about that is very good.
I have a concrete example of this with Flickr. I've been working closely with the Flickr team, and they have this PHP/Node hybrid app that I was talking about. They're trying to transition all their stuff over to Node over time, but they're taking a very pragmatic approach to doing so. Like one example here is the comments feature, so you can add comments to a photo on Flickr, and what was really interesting when I was talking to them about this, is they have a back end API for dealing with all the data stuff, but they want to render it on the server, they want to render it on the client, they want to be able to handle adding new comments, and all this stuff, right?
They're like, yeah, this one guy will go over and implement that feature, and it's like whoah! that's crazy. So this one person, this one engineer can go over and implement the entire feature from the server side part of it, communicating with the back-end API to the user interaction on the browser, and to me that was like an aha moment, that this is very powerful. It really is giving people the ability to sort of like be this self-reliant, implement these features, and Flickr's even taking it further where a lot of their leaf features, are these like end features that they have in their stack, they're writing it in isomorphic ways, so the same code that's running on the server is running on the client.
So this person is just implementing code that is talking to the APIs the same way that it is on the server in the Node runtime as it is running in the browser. And that's allowing them to move very quickly with these features that they're trying to implement. And I think it's a great example, and what we've seen in the past at Yahoo, these are three of the major organizations development, like big groups of developers within the company. We have media which is like yahoo.com, and everything .yahoo.com, those sites. We have Search, which is the oddball out of that one, and it's on it's own. So these used to be all on their own stack, and Flickr on its own stack, and they were sharing some things, but they were all having their own frameworks. This has been the case, I think from what my manager told me, for like ever at Yahoo,
they've never all been on the same platform.
So my team, which is a platform team, we really wanted to get all these teams in the same stack, and they all wanted to use Node.js, everybody was experimenting with it, and so we were able to do it. They are all now sitting on Node. There's some common and open source components that are sitting on top of Node that we helped work out with them.
And what we wanted to do from our team's perceptive is like look across, what are each of these teams doing, and realizing there's a total set of common problems here, we know you guys are doing this. We talked with the search team, we know they're doing this. Talked with the Flickr team, they're trying to figure out the same thing.
So we said to them, let us solve those problems for you, and we can then do it and provide this common layer. And then each of these applications, or sets of applications need a slightly different framework for everything they're trying to do, or they just have a different way that they they're engineering things internally in the teams.
And you know what I've really come to find out is that, Node.js has really unified and changed the frontend developer culture at Yahoo. And I think that that is really awesome and a testament that it's a great platform, and we definitely love using it here at Yahoo. So, thanks.
Node in Production
See techniques for deploying a large-scale, high-uptime production cluster.