MDB and Linux

January 30, 2014 - by TJ Fontaine

In addition to our role as corporate steward of Node.js, Joyent operates one of the largest deployments of Node. Because Node is so crucial to our infrastructure, we developed state of the art tooling to debug it. These tools include our v8 module for mdb and DTrace ustack helpers, which together provide unprecedented observability into your Node application. They allow you to see all of your application's state without modifying your application, which makes them especially useful in production, where you often won't know which piece of application state to expose beforehand.

Until now these tools were available only for debugging Node applications running on our infrastructure (SmartOS and other illumos based installs), but we're excited to announce that you're now able to debug core files from your Linux environments as well.

This is the result of a hackathon that Joyent hosted for all of its engineers this past November. Everyone spitballed ideas and voted on which projects they wanted to work on for the next week. The intent was to have a working prototype by the end of the work week. Max Bruning and I decided we wanted to be able to load a core file from a Node.js Linux process and be able to run ::findjsobjects on it in mdb. As a result of this project, Joyent was able to launch a support offering for those running Node.js on 64-bit Linux.

These examples all use Joyent's Manta service, so if you haven't already, sign up and start your free trial today. Also make sure to install and configure the Manta client utilities. You can also run this demo on your own SmartOS or illumos based install provided you have the appropriate libproc, but I find Manta the easiest way to bootstrap.

You should be running the latest stable version of Node.js (or at least v0.10.24), make sure it's 64-bit, that you obtained the binaries directly from http://nodejs.org/ and that your environment is configured to produce core files. (ulimit -c unlimited will direct your current shell to write out a file that is as large as required.)

There are a few ways to get a core file from your Node application.

  • Run Node as normal and it aborts on its own.
    • This is the worst case scenario. Either Node, one of its dependencies, or a binary module has a programming error that resulted in an abort. In those cases please report the problem to either the core team (if it was Node) or to the module author.
  • Run Node with --abort-on-uncaught-exception.
    • This directs v8 that anytime you fail to catch an exception to exit uncleanly and produce a core file. This is how Joyent runs all of its Node processes in production.
  • Trigger a core file at runtime by calling process.abort().
    • Be advised this will cause your Node your process to exit.
  • Use a platform tool to generate a core file.
    • On SmartOS and related systems there's a utility gcore which will generate a core file without causing the process to exit.
    • On some Linux installs you may also find a gcore script, it works by attaching gdb and running generate-core. Be advised that gdb will pause your process until you tell it to continue.

Demo Time

For our purposes here's the script I will use to generate a core file:

var obj = {
  myproperty: "Hello World",
  count: 0,
};

function increment() {
  obj.count++;

  if (obj.count === 1000)
    throw new Error("sad trombone");

  setImmediate(increment);
}

setImmediate(increment);

Here's how I ran it:

$ uname -a
Linux 7527bd77-ab3e-474b-ace7-eed6053931e7 3.1.10joyent-ubuntu-10-opt #1 SMP Fri Jan 20 09:55:31 PST 2012 x86_64 GNU/Linux
$ ulimit -c unlimited
$ node --abort-on-uncaught-exception t.js
Uncaught Error: sad trombone

FROM
Object.increment [as _onImmediate] (/data/test/t.js:10:5)
processImmediate [as _immediateCallback] (timers.js:330:15)
Trace/breakpoint trap (core dumped)
$ ls -alh core
-rw------- 1 root root 12M 2014-01-28 18:37 core

Once you have your core file, we'll need to get the core file and the actual node binary uploaded into Manta.

$ mkdir debug
$ cp $(which node) debug/
$ cp core debug/
$ tar cz debug | mput ~~/stor/debug.tar.gz

You can use this little wrapper script to launch an mlogin session, designed to drop you immediately into a debugging session.

$ cat << EOF > mmdb.sh
#!/bin/bash

mlogin \
  -s /NodeCore/public/linux-cores/mdb-linux.sh \
  -s /NodeCore/public/linux-cores/libproc64.so \
  -c 'echo "::load v8" > ~/.mdbrc &&
      tar xzf \$MANTA_INPUT_FILE --strip-components=1 &&
      bash /assets/NodeCore/public/linux-cores/mdb-linux.sh node core' \
  \$@
EOF
$ chmod +x mmdb.sh

This script expects you to be using a tarball that has the node binary literally named "node" and the core file literally named "core" like we built above. It's using a prebuilt asset of libproc64.so which has been built with Linux support, as well as an asset mdb-linux.sh here is its source which merely launches mdb with the patched libproc64.so preloaded.

Start your interactive debugging session by running mmdb.sh with the location in Manta you uploaded the tarball to.

$ ./mmdb.sh ~~/stor/debug.tar.gz
 * created interactive job -- 97b0b830-bafe-edb6-a271-b6a308a1a947
 * waiting for session... / established
mdb: warning: librtld_db failed to initialize; shared library information will not be available
V8 version: 3.14.5.9
Autoconfigured V8 support from target
C++ symbol demangling enabled
>

We're in a debugging session, have the v8 module loaded, it's recognized version 3.14.5.9 of v8 and we can now start performing some of our normal debugging techniques.

> ::jsstack -v
7fffa26a02b0 v8::internal::OS::Abort+0xe
7fffa26a0380 v8::internal::Isolate::DoThrow+0x2c3
7fffa26a0390 v8::internal::Isolate::Throw+9
7fffa26a03d0 v8::internal::Runtime_Throw+0x44
7fffa26a03f8 0x344a26e06362 internal (Code: 344a26e062c1)
7fffa26a0428 0x344a26e76411 increment (328d77f0c679)
    file: /data/test/t.js
    posn: position out of range
7fffa26a0460 0x344a26e743ee processImmediate (328d77f0c7a1)
    file: timers.js
    posn: position out of range
7fffa26a0498 0x344a26e0d507 
7fffa26a0520 0x344a26e06116 
7fffa26a05b0 _ZN2v88internalL6InvokeEbNS0_6HandleINS0_10JSFunctionEEENS1_INS0_6ObjectEEEiPS5_Pb+0xf2
7fffa26a0600 v8::internal::Execution::Call+0x106
7fffa26a0670 v8::Function::Call+0x112
7fffa26a06e0 node::MakeCallback+0x60
7fffa26a0740 node::MakeCallback+0x59
7fffa26a0780 _ZN4nodeL14CheckImmediateEP10uv_check_si+0x4c
7fffa26a07b0 uv__run_check+0x42
7fffa26a0810 uv_run+0xe0
7fffa26a08a0 node::Start+0x152

There's our stack, now let's find our sentinel object we defined

> ::findjsobjects -p myproperty
137289672551
> 137289672551::jsprint
{
    myproperty: "Hello World",
    count: 1000,
}

And there you have it, you're now able to debug your Node.js applications that run on Linux with our existing tooling.

Consider the possibilities:

  • grab core files at regular intervals and perform differential v8 heap analysis
  • process aborted from too much memory you can see exactly what was in the object space
  • after a missed an exception you can see the precise state at that moment
  • get the stack from a Node process stuck on CPU

It's important for us to be able debug our Node.js applications, so as we build and improve the tooling we use, we will continue to make sure you are reaping those same benefits. Your success with Node is important to us.

You can learn more about debugging Node.js applications with mdb. Also check out our other pages about designing, deploying, and debugging your Node.js applications. You can find the patch for libproc that enabled debugging Linux core files in this gist.

Want to learn more about our cloud offerings? Request a Cloud Specialist