Why does my meteor refuse to activate oplog tailing?

jesperwe · December 22, 2015, 10:01am

I am trying to set up MongoDB OpLog tailing on newly commissioned server.
Using Meteor 1.2.1, MongoDB 3.2 under Ubuntu 14.

I have configured my MongoDB install to run in master mode, and when I start MongoDB I see in its log:

    2015-12-20T12:52:34.069+0000 I CONTROL  [initandlisten]     target_arch: x86_64
    2015-12-20T12:52:34.069+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", master: true, net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
    2015-12-20T12:52:34.095+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
    2015-12-20T12:52:35.202+0000 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.$main
    2015-12-20T12:52:35.203+0000 I STORAGE  [initandlisten] The size storer reports that the oplog contains 122 records totaling to 7867 bytes
    2015-12-20T12:52:35.203+0000 I STORAGE  [initandlisten] Scanning the oplog to determine where to place markers for truncation

I have created a user for oplog access, and I can connect manually:

    client@hh-web1:/var/log/mongodb# mongo 127.0.0.1:27017/local -u oplogger -p oplogger --authenticationDatabase admin
    MongoDB shell version: 3.2.0
    connecting to: 127.0.0.1:27017/local
    > show collections
    me
    oplog.$main
    startup_log

I start meteor as follows:

    ROOT_URL=https://foo.com/ MONGO_URL=mongodb://127.0.0.1:27017/foo MONGO_OPLOG_URL=mongodb://oplogger:oplogger@127.0.0.1:27017/local?authSource=admin meteor --production --settings server/settings.json

As the server starts I see the connections open in mongodb.log, including:

    2015-12-20T13:20:33.589+0000 I ACCESS   [conn40] Successfully authenticated as principal oplogger on admin

…but still Meteor complains and will not start:

    W20151220-13:20:33.834(0)? (STDERR) /home/shing/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:245
    W20151220-13:20:33.834(0)? (STDERR)                                             throw(ex);
    W20151220-13:20:33.835(0)? (STDERR)                                                   ^
    W20151220-13:20:33.835(0)? (STDERR) Error: $MONGO_OPLOG_URL must be set to the 'local' database of a Mongo replica set
    W20151220-13:20:33.835(0)? (STDERR)     at Error (<anonymous>)
    W20151220-13:20:33.835(0)? (STDERR)     at [object Object]._.extend._startTailing (packages/mongo/oplog_tailing.js:216:1)

What am I doing wrong?

hwillson · December 22, 2015, 2:48pm

You’re not using Compose.io …

In all seriousness though, if MDG themselves are having issues with Mongo hosting, I’d highly recommend letting others worry about it. Life is too short …

The above being said, what roles did you give your oplogger user? Maybe confirm they match the roles mentioned in Meteor’s Oplog Observe Driver docs.

jesperwe · December 22, 2015, 5:58pm

I just love Slashdot style responses

$ mongo
MongoDB shell version: 3.2.0
connecting to: test
> use admin
switched to db admin
> db.getUsers()
[
    {
        "_id" : "admin.oplogger",
        "user" : "oplogger",
        "db" : "admin",
        "roles" : [
            {
                "role" : "read",
                "db" : "local"
            }
        ]
    }
]
>

Seems to match well…

sgroebler · January 14, 2016, 2:23pm

I had the same issue. I needed to setup a replica set to get it working, even if it just contains one member because you might be working locally, which is btw. what the meteor command does as well.

The error message you have is a bit confusing because the same error is thrown in two different scenarios:

github.com

meteor/meteor/blob/dc3cd6eb92f2bdd1bb44000cdd6abd1e5d0285b1/packages/mongo/oplog_tailing.js#L185


  }
  var f = new Future;
  self._catchingUpFutures.splice(insertAfter, 0, {ts: ts, future: f});
  f.wait();
},
_startTailing: function () {
  var self = this;
  // First, make sure that we're talking to the local database.
  var mongodbUri = Npm.require('mongodb-uri');
  if (mongodbUri.parse(self._oplogUrl).database !== 'local') {
    throw Error("$MONGO_OPLOG_URL must be set to the 'local' database of " +
                "a Mongo replica set");
  }


  // We make two separate connections to Mongo. The Node Mongo driver
  // implements a naive round-robin connection pool: each "connection" is a
  // pool of several (5 by default) TCP connections, and each request is
  // rotated through the pools. Tailable cursor queries block on the server
  // until there is some data to return (or until a few seconds have
  // passed). So if the connection pool used for tailing cursors is the same
  // pool used for other queries, the other queries will be delayed by seconds

and

github.com

meteor/meteor/blob/dc3cd6eb92f2bdd1bb44000cdd6abd1e5d0285b1/packages/mongo/oplog_tailing.js#L216


  self._oplogUrl, {poolSize: 1});


// Now, make sure that there actually is a repl set here. If not, oplog
// tailing won't ever find anything!
var f = new Future;
self._oplogLastEntryConnection.db.admin().command(
  { ismaster: 1 }, f.resolver());
var isMasterDoc = f.wait();
if (!(isMasterDoc && isMasterDoc.documents && isMasterDoc.documents[0] &&
      isMasterDoc.documents[0].setName)) {
  throw Error("$MONGO_OPLOG_URL must be set to the 'local' database of " +
              "a Mongo replica set");
}


// Find the last oplog entry.
var lastOplogEntry = self._oplogLastEntryConnection.findOne(
  OPLOG_COLLECTION, {}, {sort: {$natural: -1}, fields: {ts: 1}});


var oplogSelector = _.clone(self._baseOplogSelector);
if (lastOplogEntry) {
  // Start after the last entry that currently exists.

I guess your problem is the second one. For me it seems that meteor does not support standalone master oplog tailing.

jesperwe · January 17, 2016, 8:27am

This seems like an unnecessary limitation, and the comment above is not correct I would say.
So I created an issue, we’ll see if the devs have any opinions.

abernix · May 19, 2016, 7:36am

An oplog does not exist without a replica set. You don’t have to have secondary servers, but you do need an oplog – which is stored in local. In a way, Meteor acts as a secondary server to your primary. I replied in the issue with more information.