Node.js on the Road: Q&A

Node.js on the Road is an event series aimed at sharing Node.js production user stories with the broader community. Watch for key learnings, benefits, and patterns around deploying Node.js.

Question and answer session for Node.js on the Road from San Francisco, 2/27/2014.

Dav Glass, Node.js Architect, Yahoo
Jeff Harrell, Director of UI Engineering, PayPal
TJ Fontaine, Node.js Project Lead, Joyent

OK, so let's open it up for questions, as TJ said. I think it's really important that we hear from you guys, I'll play Phil Donahue although I now realized that I just shouldn't have made that reference, that's a old thing. It's something that doesn't exist anymore. They used to use these contraptions much more frequently, yes, is the caller there?


[audience] Not exactly a Node question but
this is for the Paypal guy, you and LinkedIn seem to love the dust.js, anything you want to say about dust.js?

Not Node specific, I guess. We can probably talk more offline on it. Honestly in our case, part of the adoption was javascripttemplating, but also LinkedIn was catering and we were working with them on it.

So it was an easy sell in that case. We actually use it too. Yeah, you get your own mic dude. There's actually a couple of companies that are using it. We actually use that with the Yahoo people and the LinkedIn people and Benefit Focus, so it's getting a little bit of traction.

[audience] Again,
for Jeff, what we notice is that, when people start writing Node, they approach some code that's written in, let's say, Java or something else, and it's already legacy, it's a little crusty, and has a lot of things.

You would want to rewrite it anyway. So how much of that, how much of improvement that you got because you actually rewrote it, how much is actually Node contributing.

I don't remember if I was talking with you or someone else about this earlier.
It's hard to measure that, right? A lot of things we're building are apps that were built previously. We're not exactly porting them over. We're seeing a speed increase obviously from Node itself, but there is a speed increase because we're also re-architecting the apps at the same time. But I don't know that there is a sane way to measure that. At the end of the day, it does't matter as much because there's a speed increase, so everyone is happy.

[audience] I'm
just curious, what's the roadmap like for the Node Crypto module.

The Crypto module, there's all sorts of areas. At the very least we can continue to map on a one-to-one scale what's already provided by openSSL, so that's like low hanging fruit that we can get out there, but there are also like wrapper interfaces that we can provide a better interface to—a much more approachable interface for some of the common routines that people will find themselves doing. Other more specific areas are, we have a GitHub issue that just came in this week that I've been talking with MasterCard as well about getting hardware certificates off it, so like when you're using these hardware Crypto devices, they actually don't have a file on disc for where the certificate and the keys actually are, so that's a very specific instance of where we would be adding interfaces to make sure that we can adopt that. In as generic of a way that we can, so like approaching it on a one-to-one scale of like, this is what the openSSL API provides, we're trying to make sure we can provide that in the same way for everybody else.

[audience] Hi, so this is for anyone who wants to answer, but I know Jeff specifically mentioned there's kind of this seven steps where you're like a Java developer and then you're worried about things. We've fully adopted Node and it's great, but I still do have this concern, in the JavaScript world on the client side, we use closure, we have this compiler, effectively that type checks some of the stuff for us, and in Node land it's a little bit scary. So even though I think we're on the right path, do you do anything about that, like type checking or that type of safety?

I'm actually, you guys may
have thought something on that as well, so jump in after me, I guess. In our case, there's a couple of thoughts, right? There's a lot of teams that do go down the path of like, typescript and things. We've actually taken an alternate stance and said, we really want our teams to embrace JavaScript and so as a large part like JavaScript doesn't have type safety, and depending on who you are may or may not be a large concern, but we've actually, instead of wanting them to go down the path of some sort of method of build time type safety, just again, asking them, embrace the language. Learn the language, and then learn the intricacies of the language, and maybe after that, then we can start talking about other options. At the moment, everything is just vanilla JavaScript.

You guys have any thing?

I like JavaScript. That's what we write.

Use JavaScripts.

[Bryan Cantrill] So on that actually,
I mean one of the things that we have found anyway, is that with Node things tend to be smaller and you tend to have many more components. I'm wondering for you guys at PayPal and Yahoo, if it's the same kind of thing, that Node makes it so easy to make a small component and have that thing talk over the network.

Then you can kind of reason about that independently, and frankly, you don't have this kind of million-line monstrosity where type safety becomes much more important because architecturally you have no other way of preventing damage. I think with Node, and I'd be curious to hear from you guys, if that's your experience as well.

In my experiencen Node is very much the UNIX philosophy and it drives you to build smaller things, is that true for you guys too?

Yeah, absolutely.

I think, yeah, you think about that.

Yeah, actually it's part of
the, I don't know one of the steps I skipped, right, it's the stop building mini-ish monolithic apps and start building small pieces. And once you remove that complexity, things like type safety aren't as important. If your code blocks are like 20 lines of code, and then it's a module, and you go use that everywhere.

So it's a really great point actually, yeah.

I remembered now. Test your [xx].
That's what I was trying to say.

That's good, I like it a lot.

[Bryan] And these smaller compartments make them
more testable too, when you've got a small component you can test on it's own. Because for us now, years in, I'm surprised how few problems we have had because of the lack of type safety.

Don't say that, what is your…

Hey, you got a wood pillar, there's a wood pillar right there.

When I say we, I don't mean you. It's just
in general, like the software that we have built using Node, and it's like, we have all sorts of bugs in our own software. It doesn't tend to be, you don't look at a bug, if only I had type safety here. I only have a bureaucrat here to tell me not to do this.

[audience] It's probably more of a newbie question, but how do you guys deal with process management? So obviously, when I'm developing, it's very easy to be like, node app.js, yes, go. Once I go into production, what are you doing to keep multiple processes alive, how many are you putting on each core on the server that you're on, are you operating inside of VMs, like, what does that actually look like for PayPal and Yahoo?

Well so for us, we've actually open sourced our launchers under the hood, they're up on We actually have a C daemon that launches our processes, and one of the things that we did for performance, was instead of having Node write access logs, it dumps it to the C daemon, the C daemon takes care of it. So then it's out of the event loop and it doesn't have to deal with it anymore and it can continue pushing that kind of throughput push that we're trying to push. And then, like I said in my talk, we have the domains that will catch if anything happens and it will spin it out, kill the process and it will launch another.

I think we run one per core, just as usual and that works perfectly fine for us. We run bare metal a lot, but we also, in one my other talks, I talked about the yroot that we have, and it's basically a Yahoo-ified change root, and that's what a lot of our virtual host are or basically a change root that gets run inside of those.

In our case, most of our stuff now days, especially the Node apps are being launched in VMs. It's one per core, obviously, VMs will be spun up with a couple of cores, but not like a huge amount, and then we'll use things like PM2 to actually manage the process from there, and so it's basically a dispatches to your process in that case.

Talk a little bit, I
know Brian can go in more depth, but what we do at Joyent, is actually, we have this wonderful little interface called SMF, which I don't know if you've ever checked into it, it's a beautiful piece of software that I'm sure you'd all love, basically, we use that.

We follow a very similar kind of path of like one process per core, we tend to stay away from the cluster module and we expose—we run those processes all on their own port, and then have a different load balancer in front of that. But probably the one key factor I'd love to be able to tell you guys about, make sure you understand, when you're running in production, there's a flag that we got upstreamed into V8, it's abort on uncaughtException which is an amazing thing, such that when you throw an exception, you get a core file, and that core file has all the state right then and there when your process died. This is by far one of the most crucial pieces when you're running in production of being able to figure out where your app went off the rails and why, and then how to quickly fix it and iterate over that.

And once you start doing that in
production, you start adopting that in your development process, and you actually just start iterating on your apps faster and you're able to get to production faster with the new version, so it's like a very useful part of our process, management, is also part of our development strategy.

[Bryan] And just to flush out, because you asked an OS question and I'm an OS guy, so I've got to give you the OS answer. So, what TJ is describing is called the Service Management Facility, SMF. So in SmartOS, a service is a first class notion, and it's baked into the operating system and has been for a long time, because ultimately every process re-starter is broken. The only way to do it correctly is to actually bake it into the kernel, because otherwise, who watches the watcher kind of a thing, right, and so the OS kernel can properly restart things and understands all of the reasons that processes die, because a process is just a fiction created for you by the OS kernel.

So it knows when its fictions fail. And so, we do everything with SMF. So if you deploy on SmartOS, SMF is already there. The most regretable aspect of SMF, and this tells you how old it is, that the manifests are specified in XML and yes, yes, I agree, like just stop, I know. And in fact, one of our engineers, Dave Pacheco has put together a little goober that allows you to write these things in JSON, which is much more pleasing to the eye, so you don't have to deal with the DMV that is XML.

But anyway, there are other ways to do this, of course, but that's the way we do it and just, on TJ's point, for us, the abort on uncaughtException has been, I mean, even for us, I should say, has been hugely eye opening, because we have kind of just deployed that in production in the last six months, and whenever a Node process dumps core, that gets automatically uploaded to a system like analyze it, and all of a sudden, we just get these deluge of core dumps from all of this pain that was happening in silence, effectively.

So it's amazing, when you want to take that your environment to kind of that final mile of total robustness, you really need that --abort-on-uncaughtexception. Huge, huge feature. Yep, sorry, over there.

[audience] Yeah, you guys work
for big companies with big budgets and all, and I'm kind of working for a startup here in San Francisco. I'm trying to convince everyone to start using Node.js. I tried screaming "Java sucks, Node rules," but that doesn't really seem to help a lot, and whenever I ask them what's wrong with Node.js, and they're like, if you want to write your services in Node.js, you're going to be the guy that goes to the server and fixes when stuff breaks, it's kind of un-maintainable they say, so what's your experience with that, is it maintainable?

I was actually going to say push them aside and go do it.

[audience] I did, actually.

I guess there's two parts to that,
right, like how do you convince them, and then is it maintainable. And so, I mean, obviously, I don't think Yahoo and Paypal and any of the other companies would have adopted if it wasn't really maintainable. It's just, is your team knowledgeable enough to maintain it?

So that's a knowledge question rather than the, yeah, yes it is. As far as like convincing them, we actually ran into the same thing where we got laught at early on and so, same scenario, doesn't matter if it's a small company or a large company, our way of proving them that it was feasible was just doing it. Not asking for permission, just doing it and launching it and I didn't elude to it too much, but the app we built, we built it in Java and Node, and then we basically perf tested them against each other, and once they passed all functional tests, and it was like, sweet! we did it quicker, dropping the Java app. That was a couple of months in, but it was an easy sale, and it was a business sale, not a technology sale.

You can argue technology all day long. People are going to have their pet languages they like. When the business people, the ones that actually have the money, when they're like I can do this quicker, they're going to be like, score, let's do this, let's do it quicker. I don't know if you have thoughts on that too.

Yeah, for us, we have thousands of engineers, and they all know JavaScript. And to make people use it, I just stand up and yell a lot. I'm a big guy, people listen when I start yelling.

You're not even anywhere near most of these guys.

Well I was, yeah that's even scarier.

[Bryan] Do you show them the forearm tatts? I feel the forearm tatts are kind of…

I think in terms of also, one thing that Jeff that you've mentioned that I think is kind of a common theme, is that—this is probably true to Yahoo as well, that a way to get the inroads is to say, okay, yeah, yeah, I get it, not maintainable, lets put that aside. Let's just build a prototype of the system. We can kind of agree that its faster to build stuff in this right, it's really fast. Let's just build a prototype, because I think, what a lot people find, and what we found too actually, is you build a prototype, and you're like, the prototype seems to be working really, really well, and we actually just threw a lot of load at it and, now we know we turned off 30 machines or whatever have you, and I think for a lot of big companies and small companies, the inroads have been through that small prototype.

So, I think one of the great things about the Node community, is that it's very—and Node itself, it's very pragmatic and utilitarian, so don't end up in a religious conflict. Just go build something.

We get plenty of religious conflicts
inside of Node already, if that's what you need…

As the Node turns.

[audience] I have a question about Paypal
infrastructure, and you said that you use TLS to hit nginx and then Node, and then you talk to a bunch of backend services, right?

So I'm just wondering what kind of protocol or RPC you used to talk to the backend services, and what's your experience with that.

The services themselves are
a combination, depends on the service, they're either a combination of just straight HTTP REST. Some are binary and then over HTTP, and then some are message based, so it to some level depends on the nature of the service and/or even who built it sometimes.

[Bryan] That's a great question, Dave, you want to answer that question? What are the interNode protocols?

So the interNode protocols for
us is HTTP most of the time, and then for any of like I mentioned the logger, or the monitor that monitors all the processes, that actually uses UNIX [xx] to pipe all that around. That way it's much faster, and we don't have to worry about writing all that other stuff, but 90% of what we do is HTTP. We're internet companies, right?

Yeah, it's just kind of standard protocol.

There are a lot of things
that people ask about what to do when to start with it, actually start from a different way like what's the easiest for me to get up and running and the easiest—and that also means like what's the easiest to debug, and HTTP is so well known, you have all the tools already there to be able to figure out what's going on, you just pull up TCPdump or wireshark or whatever it is that you're going to use you see the wire protocol actually happening, and it's just so straight forward. JSON over HTTP, done.

One comment to add to that is that, depending upon the—a lot of the newer services we have, the internal ones are HTTP and REST, the older ones are binary. And so just to add to that point, we actually have historically incurred a lot of complexity by doing a binary format and the specific one we did, and so every time we spun up something, it was like, now we need to integrate with that again, and we need to go build something around it, and so yeah, just using something plain vanilla like off the shelf that was standard and super easy in that regard.

[Bryan] Do not suffer from premature optimization with respect to protocol, and it's an embarrassing problem, many people have it, it's fine. You just wanted—so you don't want to optimize a problem that you don't have. That said HTTP, and what we've done for our services, all HTTP, and I think with TJ's point, HTTP first. Let me do it on HTTP, and if I could do it on an HTTP, great. And then if the data, emphasis on data, the D word, if the data indicates that I'm spending a lot of my time in parsing overhead, which can happen, that's unusual but it can happen.

You've got services that are so hot, and we do have a couple that are so hot, that actually a binary protocol is not just something that someone read on hacker news, but actually something that is worth doing. So we actually developed something called node-fast, terrible name, I know. But Mark Cavage at Joyent developed something called node-fast, which gives you the binary protocol without the ridiculous bureaucratic overhead of IDLs, and limited language bindings, and all the other crap that comes with things like Thrift, and Avro, and all this other garbage spewed forth by companies that should really, really have known better. But so you can check out node-fast if you want a binary protocol, but just to TJ's point and all what all these guys said, is to start with the HTTP and demonstrate that HTTP is not working for you because it's so nice to debug.

Not to mention
HTTP is easy to scale. You can cache, you can route, you can round robin, you can distribute it. It's a lot easier than if you had something that was binary. You'd have to re-implement that and you would have to figure out how to make one of those things under the hood take half a request.

[Bryan] You can say, we can
scale these networks with this HTTP things, you think. It got legs, it may have legs.

Just a little bit.

[audience] This question is for Paypal.
So I just found out that you guys are working on a Kraken open source Express-like thing. Is that what you guys are basing your production on, or is it just a side project?

Imminently. I have
no timeline. Let's say by the time 12's released, we'll have Kraken 1.0 out, and it will actually be significantly different because we've gotten a lot of community feedback since we released it, but the gist of it is that, we wanted to go Express for a multitude of reasons but when we started down on that path, we scaled out 200 teams.

We scaled out to a couple teams, and we had teams that were building their Express apps very differently. And so they became hard to maintain, plus at some point, we shimmed in Paypal-ism logic into there, like how to decrypt stuff and things like that. So Kraken actually solved a lot of that for us, and it brought sanity to our Express apps. Actually, I think I was just told I couldn't say the word sanity by my peers. Anyways, so yeah, that's actually being used for all of our apps right now. And so at the end of the day, they're all really just Express apps. Kraken doesn't change much of that, it just provides config and other things to it .

Alright, if you
guys want to speak to the other in terms of frameworks you are using, are you guys…?

We have this thing called Mojito. The new version of Mojito is the one that I played a bigger part in. And it's basically a bunch of Express middleware. So we went the route of being able to share middleware between all of these different teams, and then for us, we actually handle our administrative stuff in Manhattan.

So Manhattan does that stuff for us. The engineer, when they're working on their local machine, everything is exactly the way they would expect. They can get request.params and query and search, everything that would normally come through, but what we've done in Manhattan, is we've stopped all the bad stuff before it got there.

So we prevent any of the stuff happening, before it even gets down to the Node app, so these guys don't have to worry about that, because we have the same security across all of our platforms. So we just put it at that level so that it keeps them from breaking as it comes through. Not to mention we have encryption stuff, being able to decrypt and encrypt certain pieces of information. All of that happens in Manhattan before my developer even gets it, so they just get to play with regular stuff. It's all nice and pretty, it's cool, it's done, it's ready for them to use because it's all been taken care of by the server itself.

We'll start like a fight.

We were talking about earlier about banging
each other with chairs, It's also less about, in our case, less about security, as much as it's just about the commonality middleware that could be say, experimentation, or how do you deal with your config. One of the things we allow, is people to, based on their environment, set up say I'm using memory sessions here or whatever else over there.

That's part of what we're dealing with.

We actually have a pilot
grunt task to do most of that for them. People share their grunt tasks back and forth of how they set their configs up, because we have multiple databases and we have multiple different systems, and so…

I'm going to get Dave to use Kraken by the end of this.

I have a shirt, I do.
I actually had a Kraken shirt before a Node shirt, just so you know.

Settle down.

I'll get you 10 Node
shirts if that's what the problem is. I got one in my backpack.

I'll have both of you using Restify by the end of this anyway.
So at Joyent, Mark Cavage also wrote Node Restify, which is a very similar kind of middleware based HTTP server. We started with Express, moved away into our own thing to specifically design these API end points, but with debugability and observability as a first class feature in the framework itself. So you're able to, while your app is running, be able to see route start, route end, handler start, handler end at a very finite, very discreet path of what's going on and be able to figure out what your app is doing, where it's spending it's time, either on a route basis or on a handler basis.

And you can do this all external to the process and DTrace without any overhead in production, and it's just what we designed for, we're aiming for the fact that we probably wrote something wrong and now we need to figure out why and what's going on. And so that's what we're going for with Restify.

[Bryan] Restify has been
huge for us, as TJ says, I think another important meta point, and I think it's a philosophical difference frankly between Node and some of the other environments out there is—so there's a lot of choice up here, you got a couple of different ways of doing it, and Node is really not about forcing you to choose any one of those, but about putting a couple of choices in front of you, different tradeoffs were made and it's not that one should exist and the other should become extinct or that we need to have a giant feud and agree on one mediocre thing instead of three great things that are greater at their own.

[Bryan] I mean, Node
is all about the right tool for the job, which I think is actually amazing—so TJ, in terms of what he's doing in the Node core, it's really only the core of Node, and allowing that module ecosystem to continue to bloom. I think Node really fervently believes in the democracy of ideas that you don't always see out there [Java]. Oh, sorry.

I'm just going to add a thought to that too, just to tie it up. I think that ties back into your comments earlier about modules. Small modules that do little things, because then that introduces composability, right? I know that's something we've been stressing a lot with our systems where we don't want to have like a large monolith system inside of PayPal, we want people to just be able to of swap in any little piece they want, because hey! we don't necessarily know what they want.

[audience] Just a quick question about, I guess for you and
TJ is, so, you're going to improve documentation, was that to make the current API reference more robust, or add tutorials like 10gen to increase MongoDB usage, like start their own university? So, I was just wondering what that meant when you talked about improving documentation?

Sure, I'm going to put a small button in there, there's going to be a large blog post, not large, there's going to be a blog post that talks about this end to end about what I mean when I say these things, but for a preview of that kind of conversation. so the API reference documentation does a reasonably good job of achieving what you need to do to get started to a certain point of using the module and the interfaces that we've got defined for you, but there's a whole world of best practices that—so not best practices, let me be careful.

So Joyent already
has information about how we use Node in production, and what we consider are our best practices for using Node. Yahoo has similar information. I'm sure PayPal also has information that they share within their engineers. Joyent has put their's online. But there's also conversations that could host about the patterns that you can use that are either already expressed inside of Node and understanding that information. So, it's more like a pro's introduction to Node and what's going on with these APIs and how these parts fit together and why.

There's another part of this documentation
story that doesn't actually belong on but that we want to foster and then make sure that we're facilitating in the community to actually do, which is this idea of stacks.

These are all third party modules that we composed together and this is what we do, and that's not necessarily content that's going to live on, but it's important information that everybody is constantly asking for us, like there're people who come to the roadshow to talk to PayPal and Yahoo to find out how they're, and Joyent, about how we're using Node in production, and that's also an experience that we want to foster in the community of getting that production story out, and so you have three kinds of documentation that need to be improved.

Two that are going to live on, that are all going to be community driven through GitHub issues and pull requests, and then a third that has yet to be determined how the community is going to decide to build that infrastructure and maintain it. It probably looks something relatively similar, but it's, so the stuff that's on is specifically for Node and then there's a world about, a world of documenting the rest of the Node ecosystem.

[audience] So would Joyent be sort of sponsoring like maybe like teaching Express, sort of…

At the moment, Joyent—we have Node core support and we have support packages that you can purchase as part of running in our public cloud or in our private cloud, but we, at the moment don't have any offerings for teaching. Node.js as a core project, wants everybody to be able to find the information to do that, and it's not that Joyent's sponsoring the development of any kind of documentation for Express, or Hapi, or Kraken, it's just that we want to make sure that we're facilitating the community and providing those kind of resources that are actually useful to develop those environments that people can actually get off the ground and running.

[audience] OK, thank you.
[audience] TJ, why do you hate domains, and how would you improve them? And then for the group, what is your experience with domains in production?

Yeah, boy, I really stepped in it.
OK, so domains are a useful kind of feature at times; they need be used very sparingly. So when you're in a scenario of API endpoints, when there's not a lot of shared state between—you just came up here specifically to troll me. I can feel it, it's just like…

So when you're using domains, you need to make sure you understand the consequences of what we're talking about. So domains are there specifically to wrap around an unexpected error that you did not handle. So the consequences of that are difficult to understand, so when you know your app end-to-end, and know that there's not a lot of shared state that's actually happening here, it may be safe for you to be like, I'm going to close the listener, and I'm going to let everything spin out, and I'm going to shut the process down and then restart. But if what you're doing—if what you're writing in your Node app is like a chatroom kind of environment where there's a lot of shared state between all of the connections that are around, you don't necessarily have the best idea where in that transaction something might have been happening, then decided to die.

So there is not necessarily a clear rollback path, and it's very application specific about what that actually means. So, I don't want people to look at domains as a panacea like look! I never see uncaughtException anymore because I just use domains and the problem has gone away. It's actually much more complicated than that.

So also the there's a whole bunch of semantics about nested domains and all kinds of other things that, from a technical side of things, make it more difficult. So there's both a rhetorical answer to that and a technical answer.

I don't think I can add much to
that knowledge wise, obviously, but we use them a little bit—sparingly though. They actually haven't made it into any of our major chunks of code.

Honestly, example wise I think you provided a great example earlier, so I would just stick to that. That's a good example.

Yes. So, you all heard it, so I don't need to repeat it. I just don't want to get in the way of you two arguing. It's fine.

Don't worry about him.

[audience] I'm kind of fascinated by the
fact you guys onboard new JavaScript kiddies. When you do this, do you give them a workbench? Do you let them, assuming you are hiring people already know that they probably have their own workbench and I know it's a religious war but you know it's like Vagrant, obviously NPM, Brew, Bower, grunt vs gulp, did you have a preferred workbench you give these people or do you just let people figure it out on their own?

Alright, I have a list of "this is how you should build stuff," and most people do that. I have a Dav's list, I mean this is what Dav does, I've been in the Node community a long time, right? And this is how I work, so I just have a list of stuff that that's like this is what you could do, but usually it's the teams that revolve around that, so the teams that are building these things they decide as a group.

We are going to use this, or we are going to use that, because it doesn't matter how they build their stuff. It just matters that they built the modules somebody else can use. So I really don't care, but it's pretty much around the group of people. It's just like tabs vs just spaces. Who cares? Really, seriously it's the people that you work with that matter.

[Bryan] As long as you agree with me, I don't care at all.

What I'm saying is it comes down to
a team. it comes down to the people that you work with everyday that you have to look at this code with, you have to share this code with, and you have to build this thing with. That's the people that really matter, because those are the ones that you are interacting with day to day.

The outcome is just a package, it's a tar file that you just throw up somewhere. Who cares how it got there. It got there, and you're sharing it. That's the whole point.

Yeah, actually that's a really good point.
So in our case, since we have a lot of people are moving over to Node and aren't as familiar with it, we do provide a template of tooling that we ask them to use, however much to your point we don't force them to use it, there's limited touch points with our system, so, yeah, at the end of the day, as long as you can run it with Node, it's all good. And so thus some teams have deviated and gone down their own path of how they want to deal with stuff. I think a lot of it is, if you're onboarding new people don't let them go crazy maybe, but do give them a footprint, but don't enforce it essentially.

[audience] So you talked a little bit about the TLS dropping down more to the openSSL, kind of down to the native level. Are there tradeoffs to doing that? Like, talk about the dative add-ons, will more things be doing that, and is that something that people should kind of keep in mind when they're working on things that might be processor heavy?

So yeah, kind of like the centralization decentralization cycles that happen, there is code that goes back and forth between C++ and JavaScript, and we share back and forth, how much logic is in C++, how much logic is in JavaScript. We generally—in Node Core we generally err to try and do as much as possible in JavaScript, because it's a dynamic language, the jit is actually very capable about optimizing for how you're actually using it, whereas when you come down to the C++ layer, you've decided this is exactly how it's going to be used and there is nothing more.

In the case of TLS what we have: we were getting data in off of the native layer from our net interface, comes up into our Streams interface in JavaScript says, oh! I'm a TLS connection, I need to go back down into C++, and then oh! I've got clear data to come out on the other side and it comes right back up into JavaScript, and then we start passing it to you, so from that kind of a stand point it's frustrating and that was an easy win, very low hanging fruit for us to get a lot more performance out of our TLS layer, without sacrificing our front end API. As far as is more going to go that route, it's difficult to say. It's all about what's going on in a production app and where we can actually solve those problem, so we generally don't start with like, it all must be JavaScript or all must be C++. It's just about let's find out where the pain points are; let's take that data and actually act on it in a way that's appropriate to the answer, instead of just being like, our gut reaction is, if it's all on JavaScript it's faster, if it's all on C++ it's faster.

So hope that answers that part.

I've actually talked to Trevor
quite a bit about some of this, because we have a unique situation like that, like I said, we're an infrastructure place, so I said we have some of these modules that are written in other languages that we have to then compile. Well, one of those is our cookie processor.

So we now have to take this traffic in through C that has to come up into JavaScript to load the module that then loads the C module with, parses of all that staff and then sends it right back up to JavaScript. Same situation.

So we want
to be able to do all of that before any of that request actually make it back into JavaScript. That way we're all in that C level for those hardcore things that have to happen, especially—even header scrubbing, you people scrubbing headers and fixing those things. That bounces back and forth like two or three times; it's pretty bad when you're doing you know 18,000 request a second on this thing.

Yeah, and there's definitely a way to solve that kind of problem, especially from the binary add-on side, that doesn't require us to be particularly evil, but utilizes the interfaces that we already have existing today, so all of these things—the connections that come in, they're all expressed in libuv terms, and then what we need to be able to view is make sure that if you're passing something from JavaScript down to a binary module, we actually allowed you to unwrap that handle so that you can operate on it directly. So that way if you know in this particular case that this is yours and you want to actually manage on your own, then that's the kind of interface that we need to provide. Just be like OK, here it is, I need that connection, and now Node is going to disavow any knowledge of it and I'm responsible for parsing this data and then getting it back up into Node.

That's also another easy win for us to provide for people who're doing those kinds of used cases.

[audience] So I'm pretty new to
Node.js, and in rewriting some stuff I'd done in other languages, I soon found myself in callback hell, and I wanted to ask you guys, do you guys use any kind of promises libraries or async libraries in production or is it just better design patterns?

Wait, before we get too far down
this religious war, I'm going to say callbacks are the way. No.

I will second that.

To the earlier point,
between promises, callbacks, etc, we actually all agree that writing code that you're comfortable with is by far the most important thing to do. So make sure that is—you all said it in ways, don't start with me. That's by far the most important thing, before we get into the religious war, make sure you're writing code that you're comfortable understanding and maintaining, and that is by far more important that any other part of this religious war. Beyond that, I would like to tell you about a module called VAsync, which is inspired by Async, but comes with a debugability and observability as a core feature. I don't know if you've caught a little bit of my mantra here. When you're developing Node itself, you actually have to care about debugging all the time, so from my standpoint, I need to make sure that those things are designed with that as a first class feature, so VAsync, it's out there, it's in the world, have fun.

[inaudible audience]
[Bryan] That's VAsync. I'm sorry Michael, were you trying to say something?

I also think, in terms of
callback hell, there are some modules like VAsync that can help you out there. Also, don't feel obligated. If you're over in that kind of 80th column, something is wrong, and you can actually assign functions to variables and actually you can get out of callback hell and make your code a lot cleaner—yes, I am a hard-tab dead ender and proud of it, in terms of white space, we're hard tabs and 80 columns.

Part of the reason for that is, that only allows you so many levels of indentation before you're actually forced to refactor, and there's something to be said for that because in Node its very easy to get kind of the 134th column over here, and it's like if you would stop making God angry for a moment, and get back to 80 columns the way he or she intended, your code would be cleaner. Sometimes you can just apply craftsmanship as well in addition to things like VASync.

There is, and you can view that, but another thing to keep in mind that as you're in that nested structure, if you reverse your thinking just a little bit, and closures are a very important part of the language, and they're very useful and very powerful, but if you move and hoist that function out from there and move it from being a closure, you actually solve a bunch of other kinds of problems that you could introduce later.

So instead of just relying on the closure to save your state, pass your state around, you actually are not keeping as much state around, and you're avoiding the resource leak that eventually is going to kill your Node process because V8 only gives you 1.5 GB of heap. By doing that refactoring when you're in that callback hell, you're actually saving yourself so many other kinds of issues down the road, but write code that you feel comfortable maintaining. Or you feel comfortable with somebody else looking at.

If 10 depth of callbacks is a problem for to let somebody see that, then yeah, probably avoid that at all costs.

There's also the way of looking at it like, we're finally getting into a really good language to do async, and a lot of those things could probably be doing at the same time. So you don't need to nest a lot of these things. You have to look at your data and see, does this really depend on the thing that was calling it? Can I not call them all at the same time and wait for them all to be done? Which will be a lot faster. And then there's also pull the thing out and make it another module so it doesn't do all that again.

So there's a whole lot of that, it also depends on just how you think about it, and that's one of the things that we've had to look at, was that people get into that mind set of A, B, C, D, and that's not how it is, because C and D don't give [xx] about A and B, so they can both be run at the same time. What matters is that they are all done at the same time and then I have my complete amount. We look at that, because when you look at one of Yahoo's pages there's like 50 boxes all over the place. And that's all content coming from different places.

We couldn't do that if you went A, B, C, D, E—it's just not scalable, and that's and an important lesson that people need to learn, is being able to spit those things out and understand when you do that, and when is appropriate to do that.

[audience] Are there any plans for incorporating
documentation for the V8 API itself into the Node documentation?

So that's an excellent question. So the answer to that is the C binary add on layer, so by creating this blessed interface that we are willing to support going forward, we're going to document that and make sure that you understand how to use that. That will be the supported way to actually build your modules and continue going forward. That's not going to prevent you from V8 and using more advanced features of the V8 API, it's just that as far as documentation is concerned we can only document the stuff that we can control, and V8 is moving at an incredible speed, no care for anybody else involved.

And you should watch their commit history because it's like land, land, land, revert, revert, revert, land, re-land, revert re-land, revert land. It's an insane development cycle, so it's very difficult for us to continue to paper over that for you. The best way we can do that is to provide a separate API entirely, and the reality is most of your binary interfaces are using only like 1 or 2% of the V8 API, and that's the API that going to support and it's going to be easy. I'm going to release it here around the time of 12 as well. The stable C API will actually exist and you'll be able to work on 8, 10 and 12, and then it will be a first class feature going forward beyond 12, so we're not going to document V8, we're going to document the C API.

[Bryan] Note that TJ said C not C++. God is very happy.

[audience] So continuing on that, and this may be like a weird use-case for Node, I was wondering does Node care or maybe at some point start to care about embedded devices and running it more specifically on arm architecture CPUs, so when I'm messing with hardware, I want to run Node on it and be able to use a rest API, to do stuff with my hardware?

Absolutely we—so Node does care, I would say Node loves you, but apparently NPM took that. We care very much about being able to provide you that kind of interface. We're not ready yet to provide official builds for all the different architectures. The matrix of builds for ARM-based kinds of environment is very difficult to keep up with, and if you want to donate a ton of hardware to me personally I would love to help you out, but we have TooTallNate, he actually does our Pi builds on a lagging basis, when he gets gets time to do them.

So we want to provide interfaces, and we want to make sure it's still working on there. It's not a first class platform at the moment, but we get contributions from people all the time, making sure that it's working not just on ARM, but also working in Android and all the other kinds of ancillary operating systems that exist in those infrastructures as well.

So we care, and we integrate those as people who have time and resources to fix those problems bring us those fixes.

[Bryan] Alright wow, maybe one or two more questions I've been, I don't want to, what's that?

You tired of running around through everybody?

No I'm loving the running. I could do this all night. I'm ready to go, if you want to, I'm happy to go all night. I just don't want to keep folks too long, but these questions, I have to say are terrific. We got a lot of good questions, and I think we've trolled, we've hit every major troll, we've done white space, we've done promises, and we're on it.

[audience] So we've been using Node on our web services at Bleacher Report for a while now. Everything is great; we're really happy with it. But we were originally a ruby shop, and some people have got pretty excited about elixir recently, and that's opened the door to conversations about go as well, and I'm wondering if you're looking at those languages and looking at what's good about them, and incorporating those things.

You should talk to the guy that just recently helped port them.

[Bryan] So I recently worked to
port go to SmartOS. So I've been way up inside—Josh Clulow and I, along with Eram unpronounceable last name, brilliant engineer—working together on porting go, and going way up into go, which was a really an interesting experience for me. At was as Josh had the [xx] that when you go into the actual innards of go, you need to think of yourself as—it's like steam punk because there's so many things, it's basically the plan 9 build chain from 1981, that is run—anyway, so it's its own kind of strange world in miniature. Playing around with go is very interesting, coming from the Node perspective, and fortunately TJ allowed me to kind of explore languages outside the marriage. It's cool, we've got an open relationship with Node, and my conclusion from go is that you look at the go community and you look at the Node community, with some notable exceptions, there's not a lot of motion between the communities and that's because I think they both represent modern ways of thinking about programming.

They both represent asynchronous paradigms. Yes, they are slightly different and yes, we can have arguments about whether Rob Pike is being deliberately ignorant about 20 years of systems research, but it's not worth doing that, because the truth is that both of these things are modern environments and my attitude when porting go was this is actually fine, It's great.

There's not a huge delta over Node. I mean I would never actually build anything in it, not as an indictment of Go, but just because it's so similar to Node in terms of its modern thinking, and if you look at the go community, it's people coming from other communities, it's coming from Python, it's coming from—I think go is picking up a lot of C++ people who are finally realizing that they've been living in a ghetto for 15 years, and the—sorry, not to cast too much judgement, but I think to answer your question, I think that go and Node are friends really. They are both fighting common battles against legacy ways of thinking about the system.

So I don't think you need to feel like you're betraying Node, if you're building something in go, is that fair TJ?

No it's absolutely right, you should
be building, first off not only should you be writing Node in whatever form that actually makes you feel comfortable, CoffeeScript if you have to, go for it. You should also be using the tool that's right for whatever it is that you're trying to solve, and if what you want to do is go out and explore with go—we're going to hit them all—go out and use go, or use rust and experience them. And there's plenty enough research and development happening on all these different platforms, that we can all learn from each other and make sure that we're all being state of the art, like it doesn't really need to be as much of a religious war as some of the people in some of the communities want it to be. We want to make sure that, yeah, we're keeping an eye on what it is that those other frameworks and platforms are doing and making sure that you're able to build at least that quality of service in Node, and that's my stand point as the lead.

[Bryan]Go made my marriage stronger.

OK, any other questions or trolls? This is going to be an Adam I/O question I think.

I have a question about debugging in Node. Actually, I know that you can do console logs, and maybe even use Node inspector, but apart from that, is there any other way to do proper debugging in Node?

Any of you guys want to take that?

Go right ahead, man.
Go right ahead.

As a matter of fact there's a bunch of great tooling out there that I'd love to talk to you about from Joyent, but we—so, one of the—debuggability is a very important issue, especially from the question about is it maintainable, so there are third party modules out there that you can use and logging is certainly something that everybody needs, that you should be doing anyway, but it always presupposes that you knew in advance where this was going to be a problem, and that's generally not the case when you're in production. If you knew it was going to be a problem, you would've solved it already.

So what we've available to you from Joyent's stand point is MDB, which is a way to look at core files or the running process and it's exact state and be able to actually inspect the JavaScript heap and look at your call stack, and see where you are, and normally when you pull up a Node code dump or anything like that, or attach GDB whatever, you just see a bunch of hex mess once you start getting into V8 land of like generating code. We have modules that actually deconstruct that information and reconstruct useful JavaScript stacks, such that you can actually see where your Node process died. You can also see, jsstack -v, now which Bryan would want to make sure you know that, because the way Node works, your source code always in that core file as well, so when your process died at that point in time, and you do -v you actually see the function definition, so I don't know if you know this but JavaScript allows anonymous function closures all over the place and so sometimes when you're debugging this it's like, it's just this random function anonymous good luck. So with -v you actually see the function definition that was actually being executed when it failed.

You can inspect, you can find JS objects, you can see confined objects by the constructor, by the property type. Also when you're defining your objects, make sure you prefix your property name, such that when you're trying to find objects that match this thing, you're not calling it domain and then having nothing but Node all up over your crap. So you prefix it with something useful, so that you can find your way back. Leave yourself breadcrumbs. And then the other big thing is DTrace, which is a dynamic tracing platform that Brian actually wrote, and created while he was at Sun. But it's crucial to being able to debug those hard to find, hard to hit problems in production, so you can define your—you figure out what your question is you need to ask while your Node app is running. You can have DTrace running while in production, and not even worry about the overhead of it, and then when your Node app actually hits that problem, you can get the state right then and there, and see the stack, and do other kinds of very useful things.

Or maybe you just want to profile your application so there's all kinds of tooling that you can use. If you go to, you can find a bunch of documentation about that and we go out and we do meetups all the time. I just did one this morning at Yahoo about MDB and how to inspect core files, which you're able to do not just on us, but also with your Linux cores, so like if you run recent versions of .10 64 bit, you can actually take those core files, bring them to Manta, and we will be able—we'll actually help you inspect inside your Node process and see whats going on.

[Bryan] TJ this sounds too good to be true. May I see a demo of this afterwards?

You can totally see a demo afterwards, either from me or the creator.

[Bryan] Any other questions?
Guys, thanks again. The questions were terrific. I'm really impressed how people stuck it out here. I know we've been kind of over our allotted time, and I thank all three of you. Thank you very much TJ. Obviously thank you for your, well thank you for getting rid of your mullet first of all, but thank you for your leadership and guys, Jeff and Dav, thank you for everything you've done for Node.

I don't know how you folks feel. I think that when you can actually talk to practitioners who've deployed this stuff, and ask them the tough questions, ask them about the dirty laundry, ask them about what they use, I think that it's really valuable information that you can't get any other way.

So guys thank you very much and thank all of you.

Sign up now for Instant Cloud Access Get Started