Tracking down a memory leak involving multiple DDP connections

help
1

I have a network of multiple “nodes” running Meteor v1.5.4.2 (due to dependencies). Each of these nodes is supposed to be able to communicate with the others to fetch statistics etc. This is done using Meteors ddp-client server side on the nodes that should get information from the others.

This seemingly worked well, but when we started provoking it with a lot of changes in the network (meaning a lot of connections come and go) the memory gradually builds up until it freezes up and belly flops. I have limited experience resolving memory leaks, but by looking at heap snapshots, I found that there’s a buildup of objects called “Connection” (pasted below). Under strings I also find a lot of strings containing the certs and CRL’s used in the DDP connection, leading me to believe theres an issue with my code involving the connection handling. Ive tried to list the highlights below, removing a lot of minor logic involved.

I am at a little bit of a loss as to further approach this, so any suggestions, thoughts or ideas would be most welcome.

Thanks in advance.

Heres a quick run down of how its connected:

if(Meteor.isServer) {
    connectionHandler = new DDPConnectionHandler();
    Meteor.setInterval( () => connectionHandler.checkNodeConnections(), 5000);
}

export const DDPConnectionHandler = function() {
    this.connections = [];

    this.checkNodeConnections = () => {
        // Logic to add or remove the node connections in this.connections
        // Looping pr. node to handle
        const node = {...} // Details of the node to add/remove

        // Add new conncetion
        this.connections.push( new DDPConnection(node) );

        // Remove connection
        const index = currentConnections.indexOf(node.id);
        this.connections[index].disconnect();
        this.connections.splice(index, 1);
    };

}

export const DDPConnection = function(node) {
    let self = this;

    // setting up variables to use, pw, user, url ... etc. 
    
    this.connection = DDP.connect(url, { /* certs etc. for SSL */ });

    this.connection.call("login", {/* login details */}, (error, result) => {
        if( !error ) {
            // Wrap in timeout to space out the stats calls
            Meteor.setTimeout( () => { self.initNodeStats(); }, randomNumber );
        } else { /* No luck */ }
    });

    this.disconnect = () => {
        this.connection.disconnect(); // also tried with .close()
    };

    this.subscribe = (collection) => {
        // function to fetch other data
    };

    // Initialize and start recieving default basis ndoestats from current external nde
    this.initNodeStats = () => { this.getStats(); };

    this.getStats = () => {
        self.connection.call('getStats', {}, (error, result) => {
            if( error ) { /* No luck */
            } else if ( result ) { /* Pass data to handlers */ }
        });
    }

}

Connection objects in heapdumps

Connection
    _stream::ClientStream
    __proto__::Object
    _outstandingMethodBlocks::Array
    __flushBufferedWrites::()
    map::system / Map
    _methodInvokers::Object
    properties::(object properties)[]
    _bufferedWritesFlushAt::system / Oddball
    _bufferedWritesFlushHandle::system / Oddball
    _lastSessionId::system / Oddball
    _retryMigrate::system / Oddball
    _userId::system / Oddball
    _version::system / Oddball
    _versionSuggestion::system / Oddball
    onReconnect::system / Oddball
    _supportedDDPVersions::Array
    _userIdDeps::Tracker.Dependency
    _bufferedWrites::Object
    _documentsWrittenByStub::Object
    _methodHandlers::Object
    _methodsBlockingQuiescence::Object
    _serverDocuments::Object
    _stores::Object
    _subsBeingRevived::Object
    _subscriptions::Object
    _updatesForUnknownStores::Object
    _afterUpdateCallbacks::Array
    _messagesBufferedUntilQuiescence::Array
    _resetStores::system / Oddball
2

I think you’ll need to ensure your server code is able to delete the connection object(s) when it gets a client disconnect. Take a look at Meteor-onConnection and ensure the returned stop method is called on disconnect to ensure the connection callback is unregistered.

Caveat I’ve not tried this myself (yet).

1 Like
3

Hi, and thanks Robfallows. That is a good tip, I believe I somehow considered that to be called on the server-side of the connection. I will give it a closer look. Thank you!

1 Like
4

Ive looked into this. The function you mention is ment for the server side of the connection. My problem arises on the servers that is acting as a client. Meaning its the Connection object on the client-side of the connection that is not deleted/collected.

1 Like
5

Update, after digging some more.

I seem to be getting a buildup of “methods” in the “_outstandingMethodBlocks” attribute on the Connection objects. Which is defined on line 129 here:

Maybe theres some timeout setting I could use to stop them from being stored there?