January 30, 2014 - by TJ Fontaine
In addition to our role as corporate steward of Node.js, Joyent operates one of the largest deployments of Node. Because Node is so crucial to our infrastructure, we developed state of the art tooling to debug it. These tools include our v8 module for mdb and DTrace ustack helpers, which together provide unprecedented observability into your Node application. They allow you to see all of your application's state without modifying your application, which makes them especially useful in production, where you often won't know which piece of application state to expose beforehand.
Until now these tools were available only for debugging Node applications running on our infrastructure (SmartOS and other illumos based installs), but we're excited to announce that you're now able to debug core files from your Linux environments as well.
This is the result of a hackathon that Joyent hosted for all of its engineers this
past November. Everyone spitballed ideas and voted on which projects they wanted to work on
for the next week. The intent was to have a working prototype by the end of the
work week. Max Bruning and I decided we wanted
to be able to load a core file from a Node.js Linux process and be able to run
::findjsobjects
on it in mdb. As a result of this project, Joyent was able to
launch a support offering for
those running Node.js on 64-bit Linux.
These examples all use Joyent's Manta service, so if you haven't already, sign up and start your free trial today. Also make sure to install and configure the Manta client utilities. You can also run this demo on your own SmartOS or illumos based install provided you have the appropriate libproc, but I find Manta the easiest way to bootstrap.
You should be running the latest stable version of Node.js (or at least
v0.10.24), make sure it's 64-bit, that you obtained the binaries directly from
http://nodejs.org/ and that your environment is configured to produce core files.
(ulimit -c unlimited
will direct your current shell to write out a file that
is as large as required.)
There are a few ways to get a core file from your Node application.
--abort-on-uncaught-exception
.
process.abort()
.
gcore
script, it works by attaching gdb
and running generate-core
. Be advised that gdb will pause your process until you tell it to continue.For our purposes here's the script I will use to generate a core file:
var obj = {
myproperty: "Hello World",
count: 0,
};
function increment() {
obj.count++;
if (obj.count === 1000)
throw new Error("sad trombone");
setImmediate(increment);
}
setImmediate(increment);
Here's how I ran it:
$ uname -a
Linux 7527bd77-ab3e-474b-ace7-eed6053931e7 3.1.10joyent-ubuntu-10-opt #1 SMP Fri Jan 20 09:55:31 PST 2012 x86_64 GNU/Linux
$ ulimit -c unlimited
$ node --abort-on-uncaught-exception t.js
Uncaught Error: sad trombone
FROM
Object.increment [as _onImmediate] (/data/test/t.js:10:5)
processImmediate [as _immediateCallback] (timers.js:330:15)
Trace/breakpoint trap (core dumped)
$ ls -alh core
-rw------- 1 root root 12M 2014-01-28 18:37 core
Once you have your core file, we'll need to get the core file and the actual node binary uploaded into Manta.
$ mkdir debug
$ cp $(which node) debug/
$ cp core debug/
$ tar cz debug | mput ~~/stor/debug.tar.gz
You can use this little wrapper script to launch an mlogin
session, designed
to drop you immediately into a debugging session.
$ cat << EOF > mmdb.sh
#!/bin/bash
mlogin \
-s /NodeCore/public/linux-cores/mdb-linux.sh \
-s /NodeCore/public/linux-cores/libproc64.so \
-c 'echo "::load v8" > ~/.mdbrc &&
tar xzf \$MANTA_INPUT_FILE --strip-components=1 &&
bash /assets/NodeCore/public/linux-cores/mdb-linux.sh node core' \
\$@
EOF
$ chmod +x mmdb.sh
This script expects you to be using a tarball that has the node binary
literally named "node" and the core file literally named "core" like we built
above. It's using a prebuilt asset of libproc64.so
which has been built with
Linux support, as well as an asset mdb-linux.sh
here is its
source
which merely launches mdb with the patched libproc64.so
preloaded.
Start your interactive debugging session by running mmdb.sh
with the location
in Manta you uploaded the tarball to.
$ ./mmdb.sh ~~/stor/debug.tar.gz
* created interactive job -- 97b0b830-bafe-edb6-a271-b6a308a1a947
* waiting for session... / established
mdb: warning: librtld_db failed to initialize; shared library information will not be available
V8 version: 3.14.5.9
Autoconfigured V8 support from target
C++ symbol demangling enabled
>
We're in a debugging session, have the v8 module loaded, it's recognized version 3.14.5.9 of v8 and we can now start performing some of our normal debugging techniques.
> ::jsstack -v
7fffa26a02b0 v8::internal::OS::Abort+0xe
7fffa26a0380 v8::internal::Isolate::DoThrow+0x2c3
7fffa26a0390 v8::internal::Isolate::Throw+9
7fffa26a03d0 v8::internal::Runtime_Throw+0x44
7fffa26a03f8 0x344a26e06362 internal (Code: 344a26e062c1)
7fffa26a0428 0x344a26e76411 increment (328d77f0c679)
file: /data/test/t.js
posn: position out of range
7fffa26a0460 0x344a26e743ee processImmediate (328d77f0c7a1)
file: timers.js
posn: position out of range
7fffa26a0498 0x344a26e0d507
7fffa26a0520 0x344a26e06116
7fffa26a05b0 _ZN2v88internalL6InvokeEbNS0_6HandleINS0_10JSFunctionEEENS1_INS0_6ObjectEEEiPS5_Pb+0xf2
7fffa26a0600 v8::internal::Execution::Call+0x106
7fffa26a0670 v8::Function::Call+0x112
7fffa26a06e0 node::MakeCallback+0x60
7fffa26a0740 node::MakeCallback+0x59
7fffa26a0780 _ZN4nodeL14CheckImmediateEP10uv_check_si+0x4c
7fffa26a07b0 uv__run_check+0x42
7fffa26a0810 uv_run+0xe0
7fffa26a08a0 node::Start+0x152
There's our stack, now let's find our sentinel object we defined
> ::findjsobjects -p myproperty
137289672551
> 137289672551::jsprint
{
myproperty: "Hello World",
count: 1000,
}
And there you have it, you're now able to debug your Node.js applications that run on Linux with our existing tooling.
Consider the possibilities:
It's important for us to be able debug our Node.js applications, so as we build and improve the tooling we use, we will continue to make sure you are reaping those same benefits. Your success with Node is important to us.
You can learn more about debugging Node.js applications with mdb. Also check out our other pages about designing, deploying, and debugging your Node.js applications. You can find the patch for libproc that enabled debugging Linux core files in this gist.