Node.js on the Road: Gary Borton and Alex Vernacchia

Node.js on the Road is an event series aimed at sharing Node.js production user stories with the broader community. Watch for key learnings, benefits, and patterns around deploying Node.js.

Gary Borton and Alex Vernacchia
Software Engineers

I'm Alex.

And I'm Gary.

And Gary is actually going to
be handling the whole computer the whole time, so that's going to be great for me. So we are software engineers at ExactTarget, here today to talk a little bit about Node, as you've obviously heard. So, who exactly is ExactTarget? It might not be a common household name, but I guarantee you that everybody in this room has gotten something from us.

And so we are a cross-channel marketing company that does primarily enterprise B2B, and some of our customers include Microsoft, Walmart, we actually have, Kevin's here to present to you guys a little bit later about it, Expedia, and so these are pretty big customers, and so they do a lot in their own space respectively, but what exactly do we do for them?

Well, we send a lot of messages, and so when I say messages, I mean email. We're not a spam company, don't worry. I just mean social media messages like Facebook, Twitter, Pinterest, and whatever social media network really pops up, we get on it really quickly. And then lastly, we have been moving into the mobile space really recently, and we send a lot of sms, mms and push notifications.

So while we might not have hundreds or millions of people using our software like Facebook, or Twitter, or Walmart, or Yahoo, we do have hundreds of thousands of users that use our software to plan and execute their marketing campaigns on a daily basis. So if you kind of still don't know who we are, Aziz actually helped us out and we got acquired by Salesforce last year, and so now we're actually branded the Salesforce ExactTarget Marketing Cloud.

So now, it actually all started for us with Node, and among other things, about 18 months ago, which is about the time I actually started working there, which is great, and it's been quite the ride so far. So, it started with one team, and this one team wanted to accomplish two things. They wanted to, first, they wanted to distance themselves from our huge monolithic .NET backend build that powers all of our REST APIs, all the services that we use, and so we have a lot of developers as I'm sure you guys can understand that break this all the time, and so it really inhibits what we do, and it can be detrimental to our release and feature process.

So that's the first thing they wanted to do, and the second thing they wanted to do, was enable us developers. They wanted to give us the runtimes and services that we felt comfortable with, and we were easily able to use to develop on. And so with this we set up our own internal PaaS, and we actually use a product called ActiveState, not ActiveState, Stackato from ActiveState, and so with that we got all these cool new runtimes and services.

We got PHP, we got Ruby, we got Java, Python, but most notably, we got Node, and things just took off from there. And so for about 6 months we used Node internally for tooling, for testing our APIs, and everything that goes along with that, but then after that, we actually launched our first production app to Node, or to production with Node.

And it's called HubExchange. And so HubExchange is the app marketplace where users can install apps into their own accounts and use them from third parties. And it has a public and private facing side, and so it gets hammered pretty hard. So after that, light bulb went off in our head, and it was like, yes.

Node works. Wow. Surprising, right? Who would have thought? It's so new. Okay, great. So let's keep on going and see where we can go with this. So a few months later, we had our big user conference called Connections, and for the keynote we had to create a demo. So what do you think we chose to use? Node, it was really easy.

It allowed us to rapidly iterate, allowed us to have access to our data, it was super simple. Demo went off without a hitch. And so we got this code base from this demo, and we set it aside, and we were kind of like, yeah, sweet, let's use this later, let's see what we can do. And so a few months after that we launched our second app called Journey Builder to production.

It uses Node, and Journey Builder is the actual future of our company, and it allows our customers to interact with people on a very fine grained level. So they can interact based on events, based on almost anything, and then they can push them through different flows. So after all of these three things, we were like, yes, Node, viable. We have a lot of people hitting it, we've proved it in production, let's keep on going. So we took this codebase that we developed for our demo and turned into a generator, and we used Yeoman, and we turned it into it, and so now our devs can have easy access and can set up their apps very, very quickly.

And so, with this came a transition outside of the department that Gary and I work in, because we work in our product department, and we have this other organization which is totally separate from us, called Services. And what they do, is point solutions for all of our clients and custom integration and stuff like that.

So once they got access to this generator, it just took of. They started using it for everything. They switched off their static landing pages and started using it for that. So it's really cool to see how something so useful can transition from one end of the company to the other end of the company without much work.

So, now we are consistently releasing apps to production using Node, and for example, Gary actually just released his first production app a few weeks ago, so that's great.

Thank you.

So what you see here is the ExactTarget product offering breakdown. All this stuff on the left is our products that were tied to our monolithic .NET backend.

So they can run a little bit of server side JavaScript using Jint, which is JavaScript interpreter for .NET. And then about a year ago, everything would have been on the left hand side of this. So note, all the apps on the right hand side obviously are things that are getting written using Node.

So right now, we're about 50-50, and all our frontend applications that are being written are now being written in Node. This is actually how we use Node from a frontend developer's perspective. So on the left hand side we got the .NET backend that runs on .NET and Magic. Then we have on the right hand side, the UI backend that communicates with our. NET backend via REST, and what this separation allows us to do, is it allows the frontend developers to have a finer grain of control over our continuous integration process.

So that means when we commit, we get builds. We get faster builds, we get more iterative feedback, our unit tests run more often. And then the generator that Alex was talking about actually gives us the rest of this stuff in the form of like unit tests, we have a launching point. The generator includes a scaffolding and a nice launching point for all our unit testing.

So it's already there, it's just up to us to continue it. We also get compilation which normally doesn't happen in the frontend, but we can use Grunt to do RequireJS optimization, JavaScript minification, LESS compilation. This is really easy for all of our developers, it's just a single command, Grunt, it does all this build process for us. Node also manages all of our session information. So if we need to write stuff that consumes the REST API, a lot of our teams use Backbone, and we use Backbone models. So our models don't actually have to know anything about the token. They don't need to know who you are, when your session times out, if your session is even valid anymore, they don't need to handle any of that.

They just hit the REST endpoint in Node, and then Node handles routing that back to the backend. The generator also gives us baked-in security. So we get CSRF support, we get cross-site scripting support. The individual devs and the individual teams don't have to worry about any of that because it's already built in. Obviously, we get dependency management in the form of NPM and bower.

So both our Node dependencies and our frontend like jquery_handlebars, that stuff is managed for us. And then, the last one is probably most important from a frontend perspective. We get the cross-domain proxy. So our frontend apps run at a separate domain than our backend, so reaching across that domain, UI side has always been a pain, but with Node we actually hit our own apps and then proxy that on the backend so we don't have to worry about any of that stuff.

Okay, so now you know how we use Node, but what does Node exactly allow us to do? And first and foremost, it allows us to change our development life cycle, no longer do we have to depend on this gigantic .NET build or have a local VM that we have to process or proxy our data to. We can actually proxy all of our data from our QA environments and our production environments if need be.

So this is great, and it really allows us to rapidly prototype, which is one of the small steps that lead to continuous deployment which us devs are pushing for, which would be great. So like Gary said, it allows us to optimize all of our files to the best way we can. We use Grunt a lot, we can optimize our JavaScript, our images, our class and CSS, and what not.

When before our old testing scripts only covered maybe one or two of those things, and so we weren't delivering the best experience to our customers that we could. So the main thing that Node allows us to do, is really differentiate between releases and deploys, which are two separate things, and I know a lot of companies have problems differentiating between these.

I know, ours does. So we're able to deploy our apps to new URLs that very few people know about, test out all those things, and then make sure it's good to go, and then map over the new URLs to the one that our customers use. And so this enables us to do a lot, and it's really great. And so one of the last big things that Node really allows us to do, is develop as if we were our partners.

We go through the same exact process they do, registering our app in app center, configuring our session management and everything like that. So if problems arise, we probably have seen it before, and we're able to get fixes out really quickly. In addition to this, if they come up with a problem or find something wrong, it's actually really easy for us to dive in because we've been using this technology and are able to diagnose it really quickly.

So instead of a couple of weeks turnaround, maybe we'll get one or two day turnaround.

One of the other things it allows
us to do, is we no longer have to install the .NET backend. So it used to be that you would have to pull down the monolithic build locally. It would have to be in a clean state, and you'd have to run it, and then you can authenticate there.

Node allows us to run our frontend server anywhere we want, and then we can connect to testing environments, QA environments, and even production environments very, very quickly, very, very simply. Our switch to Node, though, wasn't without its own unique challenges. One of the big challenges is the generator that we use, it's up to the individual teams and product owners to continue to get these changes.

So if we were to run the generator today, and two weeks later there was a security fix, it's up to us to watch the generator and vet whether or not we want to grab these changes. Another big problem, is that with all this freedom that Node gives us, it can be kind of hard to standardize, so Alex's team might use one library and I might use another.

For example, he might choose to use angular, and I might be using handlebars, or not handlebars, backbone. He might be using handlebars, and I might be using mustache, so minor problems there.

Over-optimization can be
an issue too, so like we've been saying, we use Grunt to minimize all of our files and turn them into single files to get rid of multiple network requests, but the problem there is that even if we were to use the same version of, let's say, jQuery, if you went from his app to my app, you wouldn't have jQuery cached anymore because it's served from our single file.

So we got to look at the trade offs there. We're also sort of facing an if-it-works-why-change-it mentality, and this is largely due to our process. So the frontend is now separated from our backend, but we still have to follow the same release cycle. We can't do a major release without them doing a major release, we're not allowed to do hotfixes without them doing hot fixes because all the code goes out at the same time.

And then the last problem, also a process problem, probably one of our biggest, is that we have one team that does all of our deployments. So they do the .NET backend deployments, they do all the frontend app deployments, and with all the freedom that Node gives you, not all the teams follow the same deployment process.

So some teams might have their dependencies checked in, other teams might have them checked out and need the deploy team to install them. Some teams might have a release branch, other teams might have a release fork. Some teams have an entirely separate release repository. So we have different deploy steps for the same deploy team, and that can be hard to manage.

So we do have these issues, and we're working really hard to fix them. And so what we're doing for the last topic Gary mentioned, is we're trying to standardize what teams across ExactTarget do to release their code. So right now our release team probably isn't too happy with us since all of the apps we have running in production do something a little bit different.

And so we're actually almost done with a standard process that we can give to all of these teams in order to make it easier for them, so hopefully, they don't hate us as much which might be a problem. So the next thing that we face, is if it works, why fix it or if it works, why change. So one of the big things that we have to do, is educate people.

Everybody has their preconceived notions about what Node can and can't do, maybe they read a blogpost where the author hates it, or maybe they read a blogpost where the author loves it. So they have their preconceived notions, but they really don't actually know exactly what it can do. So I'm sure everybody in this room knows that concurrent web requests, it's great, but for intensive data crunching, it's pretty poor.

So what we try and do, is continually tell people what exactly it can and can't do, and it's actually been helping us a lot, and which brings me to my next point is, our management. They're starting to finally come around. As long as we're educating them, as long as we get our work done, as long as we're able to prove that it's viable and scale, and optimization, and everything that goes along with it. They're actually on board with us, and that's really great because it actually makes their life a lot easier also.

And so most of these problems might be technical problems, but when it comes down to it, it's really a procedural and communication problem, and that's what we're working really, really hard to actually fix at the moment. So we've been talking a lot about this generator that we created, so we're actually going to take a little bit of time to show you it.

So, let's get setup here real quick. So I already started it, and I can't see the screen, so I'm just going to go through it real quick.

Can anybody not see?

Do I need to make it bigger for anybody?


We're good.

So I'm just going to go through it. It doesn't ask any actual programming questions or implementation question, so it makes it really easy for people to get started. And so we actually provide two ways of authentication internally, so we're going to use the one that our clients will use, we're going to add our unit testing and test in Chrome, and then we're going to go through our ET specific data, so like, our app center data which handles all our token generation and session management, and we'll get to our security questions.

That really doesn't look like a security question, so none of these really look like what they actually are, so we're going to keep on going, and then it created it, and it would have installed it for you, do an NPM install and a bower install, but we're not going to go through that right now. So I actually installed it in a different folder, if I could spell today.

So I'm just going to run the grunt process real quick, and what it'll do, is it will go through all of our files, run our lint on it so we can kind of enforce standards between teams. It will run our unit tests, which we have by default baked in already, and then it'll run a complexity report, to make sure we're kind of producing maintainable code, so we give it and we hand off our products to people, they can actually work on it.

And then it'll compile our JavaScript, minimize our CSS and optimize our images. So that's what the build process is. So, we're actually going to start it real quick, and see what it does. So, it's going to start, it's on that port, I'm going to switch over to my browser, I'm going to load our SSO URL, and hopefully, oh! do I not have internet?

This is awful, technical problems. Oh man, this is what I get for trying to do live coding, right? Man, all right, so. So, I guess, I'll just talk through it. So, in this link it would've actually authenticated us and it would have popped us right into our Node app. And so one of the cool things that we do with Node, is we put a facade over all of our rest APIs. So, for example, this is one of our new rest APIs, and we have two, and so in this URL, we're going to go to the end, and find it. So this is all I have in my URL, if I were to load it, it would load. And as you could see, there's no tokens, there's no session management, there's no nothing.

We don't have to manipulate headers or any of that.

Yeah. And I promise I
didn't do anything outside of here. And then we have our other REST API which is totally different, different URL, different scheme, and we would load this, and it just works. But the difference between these two, is the way that you have to authenticate with them, you have to send different headers, you have to go through a different process, but Node makes this really, really easy for us, and we're actually in the process of putting a facade over our SOAP API which a lot of our customers do use at the moment.

So let's get back to the presentation.

Alright, so the future of
Node at ET. Well, essentially the summary is that all the new apps that we're building are using Node, and we're continuing to evaluate the best software, the best practices surrounding Node. Our managers are finally coming around, we're proving to them that we can get stuff done faster in Node and be more iterative, and then have a better confidence in our software when we finally push to production, and we really don't see it stopping any time soon.

So, thank you.

Sign up now for Instant Cloud Access Get Started