Timeouts when trying to query an external Mongo (non-Meteor) server from Meteor method

help
1

So, I’m building an internal tool using Meteor in which I need to create some reports using data that’s residing on an external Mongo database (not a Meteor app).

I’m facing a weird timeout issue when trying to query the remote DB (not Meteor).

I’m not using pub/sub, but am trying to query the data inside a method to retrieve the initial dataset and return the data in the method response, so that I can do the necessary filters and transforms on the client. I don’t want reactivity for these queries (I’m worried about the impact of long-polling if the number of connections increase and since the data is static once queried, I can live without reactive updates), which is the main reason I’ve opted to try to use methods.

Based on what I’ve googled, I should be able to do what I need to using MongoInternals.RemoteCollectionDriver, but the behaviour is very inconsistent, and most often the call timeouts.

I’d like to know what is the right way to non-reactively query an external Mongo instance from Meteor server side and send the query results to the client consistently.

I’m not having any issues querying the same DB from the same dev system using RoboMongo or other Mongo client. It’s only an issue when trying to programatically query from my code.

Here’s what I have in my method definition:

// /server/method.js
import { MongoInternals } from 'meteor/mongo';
import { check } from 'meteor/check';

const dbUrl = 'mongodb://xxxxxxxxx/dbname';

const db = new MongoInternals.RemoteCollectionDriver(dbUrl);

Meteor.methods( {
  'getStats' (range, dates) {
    if (!this.userId){
      throw new Meteor.Error('not-authorized');
    }
    check(range, Array);
    check(dates, Object);

    let query = {
      data: {$in: range},
      datetime: {
        $gte: dates.start,
        $lte: dates.end
      }
    };

    let res = db.open('collection').find(query).fetch();
    console.log(res);
    return res;
  }
});

I’m calling this from the template (I’m still using ViewModel and Blaze [yes React may be better, but no I’m not looking to move ATM]) as follows:

// client/client.js
getData: function(){
    let dates = {};
    let self = this;
    dates.start = new Date(this.startDate());
    dates.end = new Date(this.endDate());
    Meteor.call('getStats', range, dates, function(err,res){
      if(err)
        console.log(err);
      else {
        console.log(res); // It's enough if I can reliable dump the response in the browser console for now
      }
    });
  }

The getData function is triggered on a button click like so:

// /client/client.html
<button class="button is-medium is-info" {{b "click: getData"}}>Get Data</button>

I’d really appreciate any insights on how to make this work. Thanks in advance.

2

I would still use pub/sub jf I were you, but with a very hight polling interval and the collection’s _driver option set to your mongo internals external mongo collection.

This allows you to use all of meteor’s out of the box functionality. I have done some implementations with external mongo databases ( other then meteor’s default mongo databases). Just raise the polling interval to a level from which you believe wi never happen (like once a day).

One of the biggest advantages is that minimongo will help you cache the data so that other users will get the same results from memory and not directly from the database which can potebtially save a lot of load.

Try this:

import { MongoInternals } from 'meteor/mongo';
import { check } from 'meteor/check';

const dbUrl = 'mongodb://xxxxxxxxx/dbname';

const db = new MongoInternals.RemoteCollectionDriver(dbUrl);

const stats = new Mongo.Collection("stats", { _driver: db });

Meteor.publish('stats', function(range, dates) {
    if (!this.userId){
      throw new Meteor.Error('not-authorized');
    }
    check(range, Array);
    check(dates, Object);

    const query = {
      data: {$in: range},
      datetime: {
        $gte: dates.start,
        $lte: dates.end
      }
    };

  return Stats.find(query, {
    pollingIntervalMs: 1000*60*60*24 // 1 day interval
  });
});
3

Ahh brilliant. I didn’t know you could control the polling interval on the query. Will definitely try this out.

I don’t quite follow on this:

I thought minimongo is only on the client. How does this help with server-side caching?

4

The publications in combination with minimongo form an in memory cache of query results. Whenever someone subscribes to the same publication, a check is made against this cache and that data will then be published avoiding hits on your database.

5

Got it. Thanks so much. Really did help solve this.

1 Like
6

Whoops, spoke to soon. So the moment I added field filter and sort options to the list it started breaking again. It’s still erratic as hell.

It occasionally delivers data to the client, and when it doesn’t at times it looks like the server connection times out at times. At other times, I’m able to query the db on the server-side but the return data to the client doesn’t work. At that time, the entire process locks up until the mongo connection times out. And the publication doesn’t respond to new changes.

I really don’t understand what’s triggering this behaviour. I’m on Meteor 1.5.2.2 (latest). Any ideas on what I’m doing wrong?