Walmart Node.js Memory Leak Part III

Hey, Memory Leak: Checkmate.


TJ Fontaine, Node.js Core Contributor, Joyent

Walmart came to us and were like complaining, we've got this memory leak, and we can't find it. It's totally Node's fault, it's totally Node's fault, and I was like, no, it's going to be your JavaScript, it's always your JavaScript. Eran is like, no, no it's your fault, it's your fault, he finally a couple of weeks ago, narrowed it down, and he's like, it's leaking 8 megs a day. For Walmart, that's a really, really slow leak, and so for Node just trying to find this down.

I went at length about this on Thursday night, so if you want to talk about it, we can, but anyway, here's this simple straight forward HT client, it just connects the server, server just listens, and ends the connection immediately like that. So there's nothing special about this. We start running their script, and it's like, RSS is growing, growing, growing, growing, and growing and you're like what's going on, and after a bunch of analysis we figured out that what's happening is that we're creating a bunch of HandleScope data, or we're leaking inside of HandleScopes, and we couldn't figure out where that was.

So yesterday, after reading a lot, a lot, a lot of the V8 source code, we came up with this little dscript here, what's it's going to do is, it's watching for all HandleScope constructors, and increasing its state variable, and then for a destructor, decreasing state variable for both of the HandleScope constructors that exist in the source, and then we are going to come down here and we're going to watch for make callback, and if no HandleScope has been created, we are going to print out the stack for where we called make callback without first creating a HandleScope, and then we're going to print out the results of that. So we're going to run that dscript here, and then intense, do I have multiple of these running, or did I typo something in here?

That's not what I meant. This demo's going great! So you run this little script, and in 10 seconds, we will have this sucker dead to rights of where Node actually is screwing up. Come on, print out, anytime, boom! So here we see that HandleWrap OnClose called make callback without first setting a HandleScope. So this little guy, for everybody here who's using node on 10, who has a socket that disconnects, you've been leaking four bytes of memory every time, and you didn't know it, and that's what I spent my time reading a lot of the 8 source code doing. So you can look forward to this in 10.22 which will happen Tuesday.

Good job TJ.

Sign up now for Instant Cloud Access Get Started