MongoDB sharding and Livequery


#1

Hello,

I have heard that Livequery does not yet support mongodb sharding. And the Livequery page says we would need to route DDP manually (https://www.meteor.com/livequery). I love the reactive-ness of Livequery and think its a great feature.

However, my question is this: Is there a way to disable both mongo oplog tailing and mongo polling in my meteor app. (That is, disable livequery)? That way if we used a sharded mongodb cluster, we wouldn’t have to route DDP manually.

I ask this because the mobile app we are working on at my company only shows the user their data, and at this point the data only changes when they do so from the mobile app. So there isn’t any need for the mobile app to observe for changes in the data on the server.

Thanks in advance for taking the time to read and consider this question!


#2

meteor add disable-oplog - https://github.com/meteor/meteor/wiki/Oplog-Observe-Driver


#3

Thanks. I tried running that command, but I don’t notice any difference. I tested locally and also deployed our meteor app and tested there. When I open the app on 2 different computers, I see the changes made from one computer on the other computer.

I am assuming that once the livequery is essentially disabled then you won’t see live changes any more from one computer to another. Is that correct? Or is there another way to test that our meteor app is not using livequery and therefore compatible with a mongodb sharded cluster?


#4

There is a livequery package, too. You could try removing that.


#5

Ok, I am noticing a small difference with the disable-oplog package added. When I make a direct change to the mongoDB it takes around 5 seconds to show up on the UI, rather than being instantaneous. So it seems that with oplog disabled, it is now doing the older style polling of the mongo DB. I want to disable the polling as well. Any ideas?

As far as the livequery package – all I see is a mongo package, I don’t see a distinct livequery package. I wonder what the exact name would be for that


#6

Meteoruses the DDP messaging protocol to communicate with the server (preferably over websockets) and connected clients either take advantage of the oplog observe (near realtime) or polling (5-10 second delay).

But, if I understand correctly, what you are trying to achieve is a typical request response model where you refresh your data manually.

You could remove ddp, livequery etc, but there may be cases in your app that you find it valuable. Besides they may (and possibly will) be introduced as dependencies from other packages.

So, the best way to achieve what you want is to not use the publish/subscribe mechanism. This way, there will be no server initiated automatic communication between the client and the server.

Of course this then brings in the question about how one would get the data from the server. You could use either one of:

  • Ajax calls made to serve http endpoints (by means of packages such as restivus and simple-rest)
  • Method calls which return data snapshots.

In either case, to be able to better utilize the client side reactivity based goodies, you can load the returned data onto client side collections.


#7

livequery is part of the meteor-platform package. Running meteor show meteor-platform currently gives:

Package: meteor-platform@1.2.2
Maintainers: mdg
Implies: blaze, check, ddp, deps, ejson, fastclick (web.cordova), jquery, launch-screen (web.browser, web.cordova),
         livedata, logging, meteor, mobile-status-bar (web.cordova), mongo, random, session, spacebars, templating,
         tracker, ui, underscore, webapp

If you want to remove livedata, you’ll need to remove meteor-platform first and then add back the individual component packages of meteor-platform that you want.

When I did remove meteor-platform to a new Meteor app I got the following:

autoupdate             removed from your project
base64                 removed from your project
binary-heap            removed from your project
blaze                  removed from your project
blaze-tools            removed from your project
boilerplate-generator  removed from your project
callback-hook          removed from your project
check                  removed from your project
ddp                    removed from your project
deps                   removed from your project
ejson                  removed from your project
fastclick              removed from your project
geojson-utils          removed from your project
html-tools             removed from your project
htmljs                 removed from your project
http                   removed from your project
id-map                 removed from your project
jquery                 removed from your project
json                   removed from your project
launch-screen          removed from your project
livedata               removed from your project
logging                removed from your project
meteor-platform        removed from your project
minifiers              removed from your project
minimongo              removed from your project
mobile-status-bar      removed from your project
mongo                  removed from your project
observe-sequence       removed from your project
ordered-dict           removed from your project
random                 removed from your project
reactive-dict          removed from your project
reactive-var           removed from your project
reload                 removed from your project
retry                  removed from your project
routepolicy            removed from your project
session                removed from your project
spacebars              removed from your project
spacebars-compiler     removed from your project
templating             removed from your project
tracker                removed from your project
ui                     removed from your project
url                    removed from your project
webapp                 removed from your project
webapp-hashing         removed from your project

Which is quite a list!


#8

@robfallows, I removed the meteor-platform package and added back everything except the livedata package but it didn’t make any difference. When I update a document in the mongo DB directly, it still updates on the interface. Livequery is still active. I think it is part of the mongo package itself.

@serkandurusoy, I was able to return data from a Meteor.methods call, but I can’t seem to insert that data into the mini-mongo (client). When I insert the data it somehow inserts on the server’s mongo and also doesn’t even show up in the UI on the client. I’m must be doing something wrong. Do you have some example code of how to do this? Here is what my code currently looks like:

// simple-todos.js
Tasks = new Mongo.Collection("tasks");

if (Meteor.isClient) {

    Meteor.call("getMyTasks", function (error, result) {
        if (!error) {
            Tasks.insert(result);
        }
    });

    Template.body.helpers({
        tasks: function () {
            if (Session.get("hideCompleted")) {
                // If hide completed is checked, filter tasks
                return Tasks.find({ checked: { $ne: true } }, { sort: { createdAt: -1 } });
            } else {
                // Otherwise, return all of the tasks
                var findResult = Tasks.find({}, { sort: { createdAt: -1 } });
                return findResult;
            }
        }
    });
}


Meteor.methods({
    getMyTasks: function () {
        var findResult = Tasks.find({ private: { $ne: true } });
        var fetchResult = findResult.fetch();
        return fetchResult;
    }
});

#9

Try this,

if (Meteor.isClient) {
    // simple-todos.js
    Tasks = new Mongo.Collection(null);

    Meteor.call("getMyTasks", function (error, result) {
        if (!error) {
            Tasks.insert(result);
        }
    });

    Template.body.helpers({
        tasks: function () {
            if (Session.get("hideCompleted")) {
                // If hide completed is checked, filter tasks
                return Tasks.find({ checked: { $ne: true } }, { sort: { createdAt: -1 } });
            } else {
                // Otherwise, return all of the tasks
                var findResult = Tasks.find({}, { sort: { createdAt: -1 } });
                return findResult;
            }
        }
    });
}


Meteor.methods({
    getMyTasks: function () {
        var findResult = Tasks.find({ private: { $ne: true } });
        var fetchResult = findResult.fetch();
        return fetchResult;
    }
});

What I did here is

  1. Declare the collection only on the client
  2. Declare it without a name, so meteor does not try to send it back to the server

http://docs.meteor.com/#/full/mongo_collection has quite detailed information on this subject.


#10

Cool! Thanks so much! That worked, @serkandurusoy. I used that concept and re-arranged my code a little to have a separate local collection from the server collection. Here is my final code from the Simple Todos Meteor app in case someone else will find this useful:

// simple-todos.js
var Tasks;


var methodsObject = {
    addTask: function (text) {
        // Make sure the user is logged in before inserting a task
        if (!Meteor.userId()) {
            throw new Meteor.Error("not-authorized");
        }

        Tasks.insert({
            text: text,
            createdAt: new Date(),
            owner: Meteor.userId(),
            username: Meteor.user().username
        });
    },
    deleteTask: function (taskId) {
        var task = Tasks.findOne(taskId);
        if (task.private && task.owner !== Meteor.userId()) {
            // If the task is private, make sure only the owner can delete it
            throw new Meteor.Error("not-authorized");
        }

        Tasks.remove(taskId);
    },
    setChecked: function (taskId, setChecked) {
        var task = Tasks.findOne(taskId);
        if (task.private && task.owner !== Meteor.userId()) {
            // If the task is private, make sure only the owner can check it off
            throw new Meteor.Error("not-authorized");
        }

        Tasks.update(taskId, { $set: { checked: setChecked } });
    },
    setPrivate: function (taskId, setToPrivate) {
        var task = Tasks.findOne(taskId);

        // Make sure only the task owner can make a task private
        if (task.owner !== Meteor.userId()) {
            throw new Meteor.Error("not-authorized");
        }

        Tasks.update(taskId, { $set: { private: setToPrivate } });
    }
    , getMyTasks: function () {
        if (!Meteor.userId()) {
            return Tasks.find({ private: { $ne: true } }).fetch();

        } else {
            
            return Tasks.find({
                $or: [
                  { private: { $ne: true } },
                  { owner: this.userId }
                ]
            }).fetch();

        }

    }
};


if (Meteor.isClient) {

    Tasks_client = new Mongo.Collection(null);
    Tasks = Tasks_client;
    // This code only runs on the client

    Template.body.helpers({
        tasks: function () {
            if (Session.get("hideCompleted")) {
                // If hide completed is checked, filter tasks
                return Tasks.find({ checked: { $ne: true } }, { sort: { createdAt: -1 } });
            } else {
                // Otherwise, return all of the tasks
                var findResult = Tasks.find({}, { sort: { createdAt: -1 } });
                return findResult;
            }
        },
        hideCompleted: function () {
            return Session.get("hideCompleted");
        },
        incompleteCount: function () {
            return Tasks.find({ checked: { $ne: true } }).count();
        }
    });

    Template.task.helpers({
        isOwner: function () {
            return this.owner === Meteor.userId();
        }
    });


    Template.body.events({
        "submit .new-task": function (event) {
            // This function is called when the new task form is submitted

            var text = event.target.text.value;

            Meteor.call("addTask", text);

            // Clear form
            event.target.text.value = "";

            // Prevent default form submit
            return false;
        },
        "change .hide-completed input": function (event) {
            Session.set("hideCompleted", event.target.checked);
        }
    });


    Template.task.events({
        "click .toggle-checked": function () {
            // Set the checked property to the opposite of its current value
            Meteor.call("setChecked", this._id, !this.checked);
        },
        "click .delete": function () {
            Meteor.call("deleteTask", this._id);
        },
        "click .toggle-private": function () {
            Meteor.call("setPrivate", this._id, !this.private);
        }

    });

    Accounts.ui.config({
        passwordSignupFields: "USERNAME_ONLY"
    });

    Meteor.methods(methodsObject);


    Meteor.call("getMyTasks", function (error, result) {
        if (!error) {
            check(result, Array);

            var len = result.length;
            for (var i = 0; i < len; i++) {
                Tasks.insert(result[i]);
            }

        }
    });

}

if (Meteor.isServer) {
    Tasks_server = new Mongo.Collection("tasks");
    Tasks = Tasks_server;

    Meteor.methods(methodsObject);

}

#11

Hello Aabeyta, how do you connect to your shared cluster? I have this error: “TypeError: Cannot read property ‘master’ of undefined”

My connection string is “mongodb://user:password@mongos1:27017,user:password@mongos2:27017/db”


#12

Hi Keo,

The way I connect to my mongoDB is from my application server. I put the MONGO_URL as an environment variable. Then when the app starts it connects to the mongoDB server (separate server from the application server) My connection string looks a little different than yours, Here is kind of what mine looks like. I hope this helps.
mongodb://user:password@fe8756987_a0.server.com:47021/database_name?replicaSet=tk-hp5738396

I have used a connection string that looks like this that works as well.
mongodb://user:password@fe8756987_a0.server.com:47021/database_name


#13

Do you confirm that you connect to a sharded cluster? that fe8756987_a0.server.com is a mongos?


#14

I’m sorry, I misunderstood. Ok, to answer your question, I am not using a
sharded cluster, so fe8756987_a0.server.com is not a mongos. I am using
regular mongodb collections with replication, but no sharding yet.