Node.js on the Road: Node.js & Yahoo

Node.js on the Road is an event series aimed at sharing Node.js production user stories with the broader community. Watch for key learnings, benefits, and patterns around deploying Node.js.

Node is relied on at Yahoo to support our products and engineers. This talk covers how Yahoo chooses technology for products, how Yahoo uses Node today, and the challenges we've faced on the way.

Reid Burke, Senior Software Engineer, Platforms

My name's Reid Burke. I work at Yahoo. And what I do for Yahoo, is I make—I work on platforms at Yahoo. So, what that means for me, is I make tools that other engineers use. And so, normally, when I'm talking about Node, I'm talking to other people at Yahoo, and I'm giving them tips for this crazy thing that means a lot of things for a lot of people, and normally it's also very boring and it's mostly tips and I don't get to talk about things that Node works really well for, or why we use it or any of this stuff. So this is not what I'm going to talk about. I'm going to have some tips at the end, but I would like to talk to you about why we use Node, where we use it, problems that we have with Node, and where we're going with Node. And I think it'll be really great; I think you guys are going to learn something. So this is me. This is what I do on weekends in front of my house. There's also a slack line not pictured but it's pretty cool.

I work at this little known startup that no one's ever heard of, and so when I was approached about this talk, about giving this talk, they wanted to me to focus a lot on production use of Node, and that's awesome. We use it in production, so I have a lot to talk about, but I thought it would be useful to explain to this audience that production at Yahoo has meant something different than production at other companies I worked at at the past.

And the big thing about this, is most people think of production as something that keeps your company running, and at Yahoo, this means stuff that people actually use and what makes Yahoo money, right? So this is what production is, but it's also for us, we have a lot of engineers—thousands and thousands of engineers, and it's very important that we keep them productive and working, and that's a big part of what my job is all about.

So, and we do this with various tools that are, not just tools that people use on their computers, but tools that are available as services to engineers, which I'll get to talk about a little bit later. So it's both of these things our production. I work a lot with the dev part, the part that keeps engineers working, but of course in working with them, I know a lot about how Yahoo uses Node in production for keeping business going.

So I'd like to share why we chose Node, and what goes into choosing a technology like Node for the things that it does, and so this is great. I saw this a few years ago, back when Instagram was it's own company, and they wrote this blog post about why they chose what they chose to figure out how to do sharding of IDs in their database; common problem when you get that big.

What's interesting about them though, is that they're very small group, right? There are a few engineers that are doing a lot of stuff. And they were going over—they looked at a lot of different ways of solving this problem, a lot of ways that other companies did it, including looking at a way that Flicker solved this problem, but instead they chose something that was in the domain of tooling that they already used.

They used a feature of their database they were already using, because it's something that they understood very well, and was something they already trusted, and I think that's a huge deal. When choosing technology, we want to choose stuff that Yahoo understands really well, and coming from a strong frontend discipline at Yahoo for the last 10 years, we understand JavaScript really well but Node is new, and so we didn't know that at first, so we got to use it over a period of time. But when we look at what's used at Yahoo, we look at stuff that's been around for a while, it's stuff that didn't come overnight. We don't use a lot of new stuff, we use stuff that we understand really well.

Sometimes that's newer, sometimes it's very old. So at Yahoo the general thing that, and this isn't just a thing at Yahoo but I saw this and I thought that this is—when someone told me this, I thought this is great and this is exactly what Yahoo does, our products are what people should be noticing. People should not be noticing that we use Node. People should be noticing that this stuff works and it's awesome, but that said, Node is great. We've been using it for a long time. I started using Node for real work at Yahoo when it was at version 0.1.101, back when packages had to be installed either with NPM or TJ Holowaychuk's package manager, and I forget the name of this package manager, but anyway, we're not there anymore thankfully we have NPM. So we've come to understand Node over years, and when I started using it and when a lot of people at Yahoo started using it, it was for dev tools. It's for making stuff work on your box. The first thing I wrote with it was a test runner, and that worked really well, and over time, we came to understand it more. We wrote bindings that brought Yahoo's technology that's used across all of their different languages that you'd use like PHP, Java, those common things are brought into Node. We found a way to serve Node securely, and so now we use it for a lot, and I'll share with you where we use it.

The first thing (and what a lot of the people who've talked tonight have talked about so far) has been hosted apps, websites, or stuff that powers websites, and that's a really big deal. We also use it though to power our continuous integration and continuous delivery pipeline. What that basically means is, this is what we use to build, test and continuously deploy upon successful testing, not only our Node projects but really anything at Yahoo that's new.

Winds up going through a pipeline, a build pipeline that's powered by Node, which is really cool, and we also use it for what I've already been talking about, dev tools.

So I want to talk
a little bit about hosted applications. So we wind up using Node in quite a few places. The most important ones are these three: Homepage, My Yahoo and Flickr. And on all these places, Node doesn't power everything; it actually powers newer components of these sites. So for Homepage, when you go to the Yahoo Homepage, you'll notice that there's a lot of pieces, a lot of parts. There's a big thing at the top of the page that has a lot of pictures, there's news stories, fulfilling that afterwards, there's ads, My Yahoo has a lots of different things you can customize the page with, and so some of these parts of the page are generated by backend services that emit HTML that's ready to go on the page we just put in there, and that comes from Node.

And so
newer things that are made this are replacing stuff in legacy languages and legacy platforms, and are getting much better performance without as many servers which is all great, because it works very well for the type of problem we're throwing at it which is just basically server side rendering of pages.

uses—every time you visit a photo page on Flickr, that's being served by Node, and slowly the legacy code base is getting replaced with things in Node, and it's a really exciting project. We also use it for advertising and targeting. It's a really big thing; it provides a lot of support around those things. One thing that probably gets the most use, the most requests per second at Yahoo, is a video quality beacon. And so every time a video plays, video isn't being served by Node necessarily but the—every time you do an event on a video, it'll send that through a service that's backed by Node.

This is interesting because a
lot of things on this page I talked about so far, I guess, except for targeting, are all very cacheable, and so this one isn't so it's where we probably get the most unique hits and stress test Node probably the most, is on this kind of thing. And this is how much we get per second on just hitting Node, and this is after it goes through a cache already, so this is pretty big. We use it a lot.

Like I
said before, we use Node to push Node, so what I mean by that is, like everything that's written in Node you basically are hosting it on a platform that's where everything is, kind of, like a software or platform as a service kind of thing. And so we use Node not only for building the thing but also for pushing it to the various places like it needs to go, like from your machine to your, or from your CI machine to say a staging environment, then production.

like I said, we have this internal hosting platform for everything that uses Node, we use to kind of coming—this is kind of the first language or the first platform at Yahoo, where we are having things as kind of very self-service for everything, including Node itself. Or if you're using something that's in a different language, you typically have to think on how do I get enough VMs on his thing or other thing, you don't have to think about that as much with new stuff. We're kind of giving people just the tools that they need, and not having to think about operations stuff in the same way that you used to.

So this is a really big thing for Yahoo that's new, and it's working really well for us so far. So this is a topic. The building and testing and continuously releasing. This is something I care about a lot at Yahoo, and we don't use Node for everything in this kind of thing. It touches almost every part of it, but we don't use it for everything.

In fact, there's also still a lot of projects at Yahoo that were made before we introduced changes to our continuous integration system. So there's still a lot of stuff at Yahoo that's built on the back of Jenkins, and Jenkins isn't going away. In fact, the current system that we're using that we're trying to get as many projects on at Yahoo as possible, is still—fundamentally, Jenkins is sitting somewhere handling builds. And this is because we didn't want to write a completely new CI system.

This is already one that's, well, many people have tried and they ultimately will use Jenkins and that's for a reason. It would be interesting to me to fix some of the, not have to use Java or whatever, but that's not so interesting to us, and instead, or rather before I get into that, I want to mention that the problem with us using Jenkins and having one Jenkins everywhere, is that it sucks. Like having one Jenkins powering every project at Yahoo is never going to work.

And as we're getting to that point, we understand that somehow we have to wind up scaling out what Jenkins is doing, and also as someone who's as an engineer who has tried to administer Jenkins for my team a couple of years ago, I realized that it's performance is horrible once you try to keep a lot builds around, which is something you would want to do if you want to compare performance of something that's built in the past to something that's current or it just—it's just not good.

It's not great for a lot of things, but it's really good at building stuff. So we want to keep that part, but not necessarily have an interface. Point people to one, like here's the Jenkins master you want to use. So my big thing I've been working on this whole year, is in putting a Node layer on top of that so that we can say if we need to shutdown a whole Jenkins master, we can just move all the project over somewhere else.

No one ever visits Jenkins, so no one will notice because they're all looking at this. Anyway, so that's something I think is going to be really awesome, and I hope you guys will think so too. So, like I said, someone asked me just today, I was explaining this to them, and they were asking, what if people need Jenkins plug-ins or CI plug-ins, or whatever?

What we're moving towards is instead of people having to write something and that plug-ins to Jenkins, we instead have a Yahoo continuous integration engine that's all built in Node, and it has various points where you can plug in your own stuff. Like, so if you want to extend the functionality of our CI system, you can do so by writing modules with Node.

So that's how people extend it, so this is not just for Node projects, like I said, this is for anything at Yahoo, especially native apps for mobile devices, and it's pretty cool. So if you need something, you just have to write some JavaScript, you don't have to learn Java or bring out a Java IDE and make sure you're testing on Linux. Like, if you want something, you can just—people are at Yahoo mostly, I mean, almost everybody knows JavaScript and so this works really well for us instead of having to get familiar with the Jenkins APIs.

And what's really great about this, is that they can run it on their own machine. So all someone will have to do on their Mac even, is run this command and then they would be able to run this command line utility locally and it would just build their project, which is really cool. So not everyone will do this, I just want to explain this is what Yahoo winds up doing, and its a hard problem we're still working on: scaling out CI and making something sane where people don't have to set up their own CI system, because that just sucks.


So when we're
hosting things, when we're hosting applications and have this common internal platform, we want to make some choices that kind of fit what most people are going to wind up doing for their application that prevents serious bugs from getting out there. One thing that we do that's just common, that we've done for almost everything we've ever used at Yahoo in production, is limit access to the file system. And we have a module that does that you can use if this is interesting to you. It's open source, and it gets the job done, so the gist of it is at the bottom of the page, because you can't access this, somehow we screwed up and someone can find a way to get something in there which is horrible that, to read their own files, can't happen because we have this restriction in place which is really useful.

We also don't allow child_process in production so there's usually not a need for it. If you do need something like that, we usually aren't writing it in Node for serving a hosted application. We still use it a lot for dev tools and CI and CD. And this is a really big deal for us, and it's been talked about already tonight, is we use a lot of native add-ons to Node, and the reason why we use a lot of native add-ons is particularly because every platform at Yahoo uses consistent resources for like, input validation, crypto, storing secrets, those kind of things. So we keep saying by using this abstraction, and thankfully, this abstarction won't be needed anymore. This is called, I think it's, nan stands for Node.js abstraction, something. It's not a number. But this is really useful, basically, if you're using, if you today need something in production used with Node, we have a lot of things that we need to support old versions of Node, we have not that old but the last Node stable 0.8. They're still used in a lot of places and we want to not have to keep moving around for those things today before the next version of Node comes out with a stable API, so.

This lets you—it's a helper so it doesn't do everything for you but it makes it a lot easier to keep your add-ons working across many versions of Node, so use that. So basically, yeah, we have this platform that handles, like, whenever you want to push something with Node, you get CI as a platform, and that's a really big deal. We use Node for it, and if you are building something that needs to target native APIs or APIs in Node, you use nan.

Finally, I want to talk about devtools. So we use this for local utilities. The biggest benefit for us is that, if someone wants to try out their build on Windows, or someone wants to run their testing tools on Windows, or something else, there's a lot of people who use Windows, this just works, its fantastic, it's very easy to install, I mean, all you guys know this already, and we have an internal registry that makes it really easy inside the company for people to download and use something that was made just for use inside of the company, and we do that with extensions to NPM that uses NPM at its core, but it's basically just NPM preconfigured to use our internal registry. It has great up time, and because we serve everything in flat-files, and Dave who is our Node.js architect at Yahoo wrote this and it's really great.

It doesn't make your own registry, but this is the kind of philosophy that we have, is just do something that's really simple and not serve a mirror that's powered by Couch, but just have just everything as flat-files, and I'm very happy to say that if you ever use Node in Australia or need to install something, it's actually powered by this now, so that just happened just a couple of days ago. So I'm hoping that more people can leveravage this technology that came out of Yahoo for having just great experience of installing NPM packages everywhere in the world, so I'm looking forward to more of this happening. And you saw ynpm install command earlier. This is basically how someone would install it. It's also a Node module and you just point it to our internal registry to get that and then you have it. So, that runs in your box just like we have that CI run locally as well, so that's pretty cool.

And problems.

So, yeah. I'm going
to talk a little bit, and this is where you might get something out of this talk that you—this is where I give the tips and stuff, but I'm going to just talk about a few. This isn't everything, but these are the biggest things. One is really big for Yahoo; one's really big for me. And so, yeah, when you deploy Node, you wind up running into things that you didn't expect.

The biggest thing at Yahoo that we run into in production is memory. Like we are horrible at, apparently, writing JavaScript in a way that doesn't leak with closures because JavaScript is a language that seemed to optimized for creating those kinds of mistakes which is awful because I love JavaScript, but this is the biggest thing. Like CPU, not our problem. Memory, very much our problem, and it's not Node's fault, it's our fault.

So it's very difficult to debug this, it's very hard, I don't have any good advice for you. I do say that like, if you can use tools that Joyent provides that get the job done for a lot of things. For if you're trying to understand where leaks are coming from in your program, you can take stuff from your production Linux infrastructure, and then put it on SunOS with Joyent's MDB utilities and get going there, but we can't do that for everything, so that's unfortunate.

So that's just hard, if you have a solution to that, please talk to me because I'd love to hear from you. Errors are something you have to handle really well, and when you start building anything that's bigger than something small or just something that's very limited in scope, you're going to have a lot of errors that come out of your application.

And you have to deal with those really well, and so if you get anything out of this talk, please take care of your errors. If you get an error, this is kind of all from, by the way, from Joyent's production guidelines. They have a great guide on their website on their developer section for not only errors, but for a lot of common production practices in Node. So I'd encourage you all to check out Joyent's developer site for more elaboration on this. But if you basically get an error that you don't expect, there's all kinds of problems that could be going on. You should be crashing, you should be doing what I told you earlier for dumping, and if you don't know what something is, even if it is something that you expect to happen like a network problem or something else, if you don't understand where those are coming from, if your program can not decide like what this error means, then you don't have a working program. Your program is an accident. And I'm guilty of this. And I've been tearing my hair out thinking there has to be a better way, and I looked at Joyent's website and I saw that yes, there is a better way, I'm about to show it to you. Wrap your errors, please. So this is inspired by hapi, it doesn't really matter what this is, but say if I have this handler that's doing stuff, like it's getting something from the database, so I have this, this is the handler part, it doesn't really matter what this stuff does, but typically when you're building something that's not small, you're doing a lot of steps. Say, I have to make sure that the the user ID requested is a number, so that's one thing that has to be done, now I have to do some things that are asynchronous, then I have to decode that and show it.

There's a lot of places that where things can go wrong, and so we do something like this where in a reply something happens that will either handle the error or actually show the response. And so what happens, though, inside the router (and this is inspired by hapi, but this is basically something that I think hapi does really well, and we use this for the projects that use hapi inside of Yahoo) is that we show different things, like different stuff will happen based on what that error was. So if it's your fault, if you didn't specify the right input, getById would wrap that error in a way that it would define where that error came from. If it's our fault, then it's unwrapped, which means we would just show a generic message to the user like 500, and then the handler would crash if it's not recoverable like it used to.

So, anyway to make sense of this what I just explained to you, if the error is wrapped, meaning that it's your fault, means that we shouldn't just show a 500 error, we should show something else that's more descriptive, then we will do that by replying with the appropriate status code and a sanitized message that demonstrates what went wrong in a way that user can understand, but will log the actual error from before it was wrapped for our use, right?

So this means that we don't have a lot of conditional logic all over our code where we show different response.send(something else) based on what kind of error we got, or we're doing this in many different places. Instead we just wrap it then pass it along, and the error goes all the way back up. And then if there was something that was our fault, we just show a generic message, but we log the error for our own use.

There's many, many different ways you could do that, and if you don't understand what I said, and this didn't make any sense to you, I understand. But look at—these are two things that I've looked at that do this and are ways for you to look at this and understand how to better handle errors, and also look at Joyent's developer resources which are awesome.

So, at Yahoo, production is in a lot of different places, but I think that, really when you're thinking about your company even if it's small, really think about like how are you—if you're making something that's used internally, think of that as production, like you don't want that stuff to go down. I've given you some resources for how you can look at problems, or understand problems that might come up. I've showed how Yahoo uses Node and our future of using Node is really, we're finding more and more uses for it all the time around…

My team is, for enhancing developer productivity, and I think, the further we go with Node as something that is used and understood by more and more people, especially as you bring in more people inside of the company that understand JavaScript really well, it's going to be a bigger deal in the future. It's great, it's worked well for us for a long time, and I'm really excited for where Joyent takes Node and where the community takes Node, and yes, so production winds up being in a lot of places, more places than you might expect, so make it safe, make it easy, make it enjoyable. A lot of the commands I showed you are just one liners that people can run, that's kind of where we come from when making dev tools. We want to make things that are really easy for people to use, things that are really difficult to mess up, and things that just make other engineers happy, and then which by extension make our users really happy.

So, thank you so much. We would love for you to consider joining Yahoo because its awesome. And yeah, we're hiring for everything. If you care a lot about Node, if you care a lot about really any kind of engineering, we have a need for everything, and we really wish you were here. And thanks a lot.

Sign up now for Instant Cloud Access Get Started