So yeah, so like they're saying, I definitely want to start introducing Dow Jones and myself. A lot of people think Dow Jones, they think of a Dow Jones Industrial Average, they think of Wall Street and we do do some stuff there, but mostly we're a media company, we do the Wall Street Journal, we do Barron's, we do magazines, we do lots of different content in media, and websites, and that's our primary business, so we've been doing that for about 130 years, 125 years on the Wall Street Journal.
The reason I point this out and that we're media company is because if you ask people internal to the company, "what does Dow Jones do?" primarily especially these guys here in the front row are going to say, it's a technology company, and that's because we've been innovating for a very long time. We invented/we were one of the first people to use the 5-inch ticker to deliver news. Before that people were kind of just running around Wall Street saying, hey the price of IBM changed. And we are also were the first to deliver the newspaper across the nation via satellite. So this is a tradition we have, and a lot of technologies that we use today are part of that but certainly Node.js. This was the Node.js of 1914.
So the big boy is WSJ. If you view WSJ articles, if you view the mobile website, if you view the technologies section front, if you view the markets page, it's all served through Node. So you're talking about tens of millions of requests on a daily basis. Of course it depends on the new cycle and things like that, and we're running on four servers, so we're dead in between. For WSJ, we've lots more Node servers than that. And there's a lot of other products that we have, so Barron's has a portfolio application that's running completely in Node in production, the streaming stories kind of sits under WSJ. A product I've been working on for a majority of those three years is this product called Real-Time, which is how we deliver—it's the only way we deliver data directly to clients, so we have a lot of feeds like the Dow Jones news wires that go through like a Bloomberg terminal for example, but in this product we deliver those wires directly and much faster to the customer, and we deliver a lot of financial data from like our MarketWatch platform and things like that.
The stuff in the middle, MarketWatch and Factiva, they're not fully in production yet, but they're both flagship products. And MarketWatch, for example, the mobile app is currently in production and they are right now in Minneapolis coding and writing the website over in Node.js, so they have a big redesign and they're going do that whole thing in Node.js.
Factiva is more of a strategic thing. We have things within the institutional platform that are currently in Node, but Factiva is kind of evaluating which pieces will be replaced. We're big onto switching as much as we can over. And then it's not just our production apps, but our internal applications too, so we have a big chat application called DJ Hub, and one of the developers here today, and that's completely using Node, it's a bit different stack than we use for the websites, but very interesting. Our commerce platform, which is super important, obviously, that's how we do all of our business with our customers.
It has a good chunk in Node now, and they are actually rewriting the entire service API layer in Node right at this moment. And lots of other little internal systems, I just point out one here DMI which is a feed management system for risk and compliance data that we have, so a lot of internal tooling is also Node.
So, hoow do we get there, right? I think our history is an important part of this, and something that other companies either at any stage of development, and we're still developing Node, right? We make changes everyday and are constantly having debates within the company at where it should be used and how, but we grew it organically, so we started with this project called WSJ Social, which was a Facebook news reader, and that was in 2011, or early 2011 around Node .4 and we basically, it was totally ground up, I could say things about scale-ability and re-usability and all this stuff we told management, but essentially Yuri came to me and said, "dude, I convinced them to use Node," so we did that and it was that success.
So we decided to go with Node there as well, and it went just as well. But the important part is, in the middle of this project we realized a couple of things. One, we needed to build a foundation, right? If we wanted to scale this across the entire organization, it wasn't good enough to just say, oh, Team A, you build the Node app this way, Team B, you build…
Dow Jones is huge. We literally do Java, C, C#, Ruby, PHP, Pearl. We have hundred of different teams all across the globe, and just saying, "yeah, go use Node" wasn't sufficient. So we actually decided to build a framework and a whole stack around that which I'll talk a little bit about in a minute. And another important piece during that Real-Time project is we were working in coordination.
That DJ Hub, that chat app that I talked about earlier, we were working in coordination to get that integrated into the real time product, and the lead of that project at the time, who I was talking with on a daily basis, discussing Node, helping him set up his machine, teaching him some new things and vice-versa,
we were doing pull requests back and forth—he became our CIO.
So, once that happened, it kind of solidified everything, because now we had this organic growth, and we had our C-level supporting us completely, and a lot of excitement building around it, and of course a lot of news, because once the C-level is excited about something, you know it's going to come down the pipe a lot faster. And after that it just took a couple of months to commit (so this is maybe now a year and a half ago on the time-frame) and we decided we had to do a couple of things which was, we basically made it the gold standard, so if you want to make a web app at Dow Jones, it's probably going to be in Node and you have to have a good excuse why it needs to be in another language.
And we also actually made a core team that's sole job is to kind of evangelize and build things in Node and we have this framework and this stack which we call Tesla, which they're the stewards of, although we have a very open source model within Dow Jones too. So there's maybe what, 20, 30 developers really pull requests to that framework.
So now you kind of know what it is we do, and how we got there, but what do we do? So this is the one part of the talk where I get to talk a little bit of technical stuff. I wish I had a lot more of time. Hopefully we can cover it in the Q&A. We have our own NPM, so from day one we decided that both to insulate our code, and to remove any dependency on the public NPM, we would deploy our own.
So we kind of dug in, it's all open source and we deployed two versions. We have a dev version and production version. Dev has everything, and production is just the stuff we trust that's going to go into production, and that's evolved over time too. So it used to be just the copy, now it's a proxy, so that we can save space and be more efficient and things like that especially as NPM grows out of control and Isaac tries to fix it.
And GitHub, this is a lot more vanilla thing, we just have an internal VM that we host GitHub on, but obviously it's integral to what we do in Node. But most of our—actually we have split our Tesla framework allows us to take deployables now from either GitHub or NPM, but it used to be primarily NPM. Another thing is, all of our development, and deployment tools, and services that we use as developers, they are also all in Node.
So when you do JSHint, or you do any kind of testing, that's all tooling that we developed, and it's the same thing that runs in Jenkins when you deploy to your environments, right, and everything has hooks all the way through, so I might do a check on my machine with local tools which are very quick and then deploy to Jenkins. I'm guaranteed to get the same result there and get that deployed out to production, and then part of Tesla stack that we have, we also have services on the machine that are helping us with deployment and management of code, and they have hooks too, so that Jenkins can just call in for example with CI and say, hey, restart this box because there's new code.
So Tesla, this is a big thing for us, again I wish I could talk about it a lot more. Essentially a couple of things, we didn't think at the time (it's evolved a lot since) that the frameworks were good enough for enterprise, not because they weren't quality, but because just like the Node community wants, you have very small targeted things that you're trying to do. But in enterprise again if you let people go wild, they're just going to—you're going to wind up with infrastructure spaghetti, right?
You're going to have all this different stuff. So we wanted to have a more robust framework that did a lot more than just routing or templating. We wanted to do all of that together in one place, and we wanted to make it really simple and really modular. So I could wax poetic, but essentially we want to have a developer be able to work in this huge page like a wsj.com, and touch just the tiny piece and in one place his LESS, his CSS, his JS, front end and back end, his unit test, everything are always across projects, they're always going to be in the same play structure, the same way, and he doesn't have to worry about the whole ecosystem around him. He knows that everything is just going to work. We also do the usual stuff that I kind of eluded to: logging, and caching, and routing, and testing. We help the developer do all that kind of stuff. And I'm just going to highlight one specific thing about Tesla, that I really love, which is the dependency injection.
So dependency injection is something you don't see a lot in Node. It helps you keep your code really modular, it helps keep it really, really testable and isolated. It's part of that keeping things modular, because you just get things instantiated for you in your code, and it does a couple of interesting things.
So one thing is, it let's us hot swap code. So in production, if you're using logger V1 and at a system level the developers say V1.5 is ready, API is the same, everybody can consume this thing, we can in production in real time, just switch out that one module and the following request will use logger 1.5 instead of 1.0, and that's because all of the requests are being routed through this framework, and all the dependencies are being grabbed by us and returned to the individual code. So if you ask for 1.0, and in the next iteration of your code you ask for 1.5, you get 1.5.
And it also—dependency injection is interesting because it's request scoped, and that's the only scope we allow, and this helps solve some problems that modules and the community like CLS are trying to solve, where they are tying to pass context down through all different levels and maintain it. Dependency injection, you don't need to do that.
If you have a user service that has your user data, and that user data is request scoped for that user, you can reliably just inject it anywhere in the whole stack and you're going to get that user. So if you want to, for example, log something about the user but you're already like in utility method, after utility method, after utility method, you can just say, oh, give me the user service, what's the users, I don't know, whatever, some kind of information about him, right, some kind of ID, hash, or whatever. So that's pretty, darn cool. What happened? My computer is disobeying me. OK.
So what did we learn? So this is kind of the benefits and pain-points, although we honestly haven't had many pain-points with Node. We haven't add any debugging fiascos or, like, rampant memory leaks, at least not in production. We've found everything pretty much in dev and QA. But the first and biggest message I want to communicate is that anyone can do this. If Dow Jones can do this, anyone can do this, and that's because we have such a crazy organization. We basically have start-ups inside our organization, we have these huge like institutional, there's this huge enterprise thing that mirrors a lot of Wall Street companies, and we have, like I said before, a myriad of languages and technologies.
So one story for anyone can do it is that, just recently, six months ago (and a few of the guys are here) we had to move 100–150 .NET developers over to Node for this huge project we were trying to get done in a few months. So you can imagine how that went, but they're all really happy with it now.
Another couple of things I want to encourage or share, is building your own. So I think building your own stuff is really beneficial, but you also just have to beware that it could come with some down-sides, and usually those down-sides are being outmoded or outdated. So to give some context, I talked about those development tools we had, right?
When we run testing or linting or whatever in the code, that has to go in some order and do some operations right? So what did we need? We needed a task manager, except Grunt was barely in its infancy, Gulp didn't exist. So we built our own. It's very cool, it's a little naive compared to some of those things, but now what do we do?
So we have a debate actually going on in the company right now, do we continue using it? Do we replace it with Gulp (is the one we happen to be looking at), do we do some weird hybrid approach, where we do ours, and then we tie in if the users want to do Gulp in addition to that. But of course there's lots of benefits, like NPM is pretty—it's had its problem, but it has worked out for us and achieved the goals that we wanted and we can tailor it exactly to our organization.
And last is ownership. When I say ownership, I really mean contribution; owning the community, like us all owning everything. I think that's super important going to things like this, contributing code back so if you look over (you wouldn't know it because we all kind of go out and contribute on our public accounts), but we've contributed to the MongoDB driver, to Hogan, to NPM, to XML serializers, to all sorts of libraries in the community because we needed them to work in the production, and they didn't. They fell down in some way. So went out and we did that, and that was a great experience, meeting the people, and talking to them. That's another thing about this community, is just how awesome everybody is. We really haven't had any—we've had some egos, but nothing we couldn't get over. So yeah, so basically get out there, get involved, listen to things like Nodeup which I'm hoping I can go on, and yeah, just be involved.
And the last thing is, I hope we can talk a little bit more about the technology in the Q&A after, but I want to thank the whole community. So I don't think Dow Jones could be where it is, or any of these companies could be where they are, if this community wasn't as enthusiastic as it is, if it didn't have the work ethic it does, and if it wasn't as intelligent as it is. I think that the growth rate and all of this that we're experiencing right now is just amazing and unparalleled.
Node in Production
See techniques for deploying a large-scale, high-uptime production cluster.