July 01, 2013 - by Mr. Max Bruning

Share:

This past week, I attended Nodeconf 2013 at the Walker Creek Ranch north of San Francisco. At the conference, 8 different sessions were held, each session repeated 8 times to 40 different people each session. I presided over a session on using DTrace to help debug and troubleshoot problems in node.js applications. The sessions lasted 1 hour each, and they were meant to be as hands-on as possible. Since I was presiding over one session 8 times, I did not get a chance to see any of the other sessions. Hopefully, some of the other presenters will also be blogging about what they did. In this post, I'll talk a bit about what we did in the DTrace session.

The slides, lab descriptions, and lab files are available at https://github.com/max123/nodeconf. My presentation started with a quick overview of DTrace: what DTrace is and what it can be used for. As I mention in the slides, DTrace is a dynamic tracing facility. It can be used to trace all aspects of a node application, from the operating system, through the node engine and libraries, to the node.js scripts themselves that make up the application. DTrace allows you to trace code as it is executing. Trace points (called probes), can be dynamically inserted into the code at runtime, or can be statically defined. DTrace is useful for performance analysis, debugging, code coverage, and in general, finding out what your code is doing. DTrace is safe for production use, and when it is not in use, there is zero overhead. Even when you are tracing, the probe effect tends to be very minimal, depending on what you are tracing, and what actions you take when the probe fires.

Conference sessions were meant to be as hands-on as possible. Following the brief introduction to DTrace, attendees were given a set of 5 labs. In trial runs of the material before the conference, we thought that if students could finish the first lab, it would be enough to get them started, and would probably take the time allotted. I finished the introduction to DTrace material in about 30 minutes, leaving 30 minutes to do the labs, which worked out well. Most everyone finished the first lab.

Before I write about the labs, I want to spend a few minutes describing the classroom setup for the labs. The ideal setup would have been to have attendees go to www.joyent.com and sign up for a free trial machine in the Joyent Public Cloud. However, it was not clear beforehand that the internet capabilities at Walker Creek Ranch would be sufficient for this. So, I set up COAL (cloud on a laptop) on my laptop. This is a tool that Joyent uses internally for testing and debugging that allows us to run an entire SmartDataCenter (SDC) on a laptop. SDC is the same software that we run on the Joyent Public Cloud (see Joyent Compute Service). Very briefly, SDC consists of a "headnode" that runs the cloud orchestration software, and one or more "compute nodes" where customers run their virtual machine instances. For COAL, I had one 4GB headnode, and 1 6GB compute node. On the compute node, I provisioned 40 SmartOS instances. Each SmartOS instance had its own IP address and stack. Each instance exists in its own zone. Each zone looks like a complete machine to users within that zone. They cannot access other zones, and there are resource controls so that they cannot effect other zones within the machine. To do the lab work, each student ssh-ed into a different SmartOS instance (zone), and followed instructions.

At a high level, these are the labs:

1. The first lab is one for which I wrote a blog post here. This was the only lab that could be done entirely on Mac OS X (the others require SmartOS). And this is the lab that everyone who attended was able to complete.
2. The second lab uses the DTrace probes that come with node-restify. The DTrace example is at https://github.com/mcavage/node-restify/tree/master/examples/dtrace. The DTrace probes built into restify allow you to trace entry and return from any routes you have set up.
3. The third lab was to create a flamegraph for a running node process. Flamegraphs give you a graphical representation of where your code is spending its time. See here for an example, and see this for a "how-to" for doing this for node apps. For this lab, the flamegraph will be much more interesting if the node process is busy (i.e., you may want to generate load so that the process is actually doing something).
4. The fourth lab uses DTrace (and some other SmartOS tools) to examine a memory leak. It will never be this easy...
5. In the fifth lab, you use mdb(1) to examine objects within a node application. See this post for a description.

I am currently adding more labs, as well as more details to the nodeconf material in order to offer a course on debugging and performance analysis of node using DTrace. A course description (and sign up) should be on the Joyent website sometime this week.

As for the conference, it was great! I want to thank TJ Fontaine, Joshua Clulow, and Emily Tanaka-Delgado of Joyent for all of their kind help with assisting classes of 40 people, the staff at Walker Creek Ranch, and Mikeal Rogers and the nodeconf staff for giving me the opportunity to do this. I'm hoping to be back next year.