Mesmerizing mdb at Node Summit

December 10, 2013 - by Ben Wen

Last week at Node Summit 2013, we were treated to an exhibition of sparkling systems showmanship by Joyent’s SVP of Engineering Bryan Cantrill. And all of that while riding the command line of the MDB debugger, natch. The broader motivating topic was describing the three-year path that Joyent took in adopting and implementing Node.js into our mission-critical systems. The storytelling culminated in a demonstration of JavaScript-aware post-mortem analysis on a Linux-based Node.js runtime, a previously impossible activity.

Truth is, most implementations of systems software are not fodder for scintillating demos. Without context, they are not too different from watching authorship of a Neanderthalian libretto of guttural vocalizations ("grep awk sed *.ps1, ugh!"). But for this audience, there very much is the appreciation of the crush depth that one descends to in making mission-critical systems actually work, especially in finally eradicating the corner-case bugs that page us at 3am on a Sunday morning.

That, it turns out, only slightly overstates the motivation for Joyent to invest heavily in creating a production-quality system for Node.js. As Bryan describes in his presentation by being stewards of both the application runtime (Node.js) and the underlying operating system (SmartOS), Joyent was able to mutually reinforce both products.

The fast-paced (and entertaining!) tour of production features for Node.js ranged from static DTrace probes in Node.js Core in August 2010, stopping along the way to include work in SmartOS's debugger, mdb, to parse otherwise opaque JavaScript stack frames and objects. One of the newest tools uses the ephemeral compute and persistent storage in Joyent Manta as not only a place to store core dump files, but also as a place to effectively analyze and instantly and interactively debug them without having to spin up a VM. The 30-min talk, peppered with colorful imagery and exhortations of a live-running system to yield demo-worthy data, closed with the demonstration of a new capability that was developed over a few days in a recent Joyent internal hackathon by TJ Fontaine and Max Bruning.

One of the most powerful tools in understanding post-mortem output of a misbehaving Node.js program has been the collection of tools built into mdb, including the nearly auto-magical ability to scan all memory to find and identify JavaScript objects and then make sense of them. Because of the level of integration into the underlying SmartOS operating system to perform these low-level functions, porting mdb to other systems, including Linux, has been a non-starter. So Linux users of Node.js, while able to use Node.js package contributions like Restify for APIs, Bunyan for logging, and Fast for lightweight HTTP, we have not been able to benefit from the post-mortem analysis and dynamic tracing in SmartOS. Bryan announced that at one of these, post-mortem analysis, had been successfully run to ground.

In the finale, Bryan took post-mortem data from another system running Node on Ubuntu Linux, a system that does not natively have these analytic tools, and was able to perform the same deep post-mortem analysis by using the ephemeral compute within the Joyent Manta Storage Service. Opaque JavaScript objects in the Linux stack were made transparent with a few commands. Somewhere the proverbial demo gods, seemingly satisfied with what was sacrificed in their name, were smiling.

The new Linux functionality and demo script are available publicly at this link.

(More in a future blog post!)