RFC: No oplog vs. oplog vs. redis-oplog when self-hosting

deployment
1

After years using Galaxy+Atlas for deployment, I have been offered an Ubuntu server to use to deploy a Meteor app for my university department. The app is not mission critical, does not require high availability, and there are only about 500 users (with low concurrent usage—25 concurrent users under normal usage with bursts to 75 or 100). The MongoDB database contains less than 100,000 documents, and any single user subscribes to no more than 1000 documents.

Seems like it should be easy to self-host, right?

So far, I’ve been using Docker to set up Meteor and Mongo on this new server. My in-progress documentation of this process is available here:

As you can see, I’ve just got to the point of trying to figure out oplog tailing. I’ve spent the past day or so reading on the topic, and am a little confused as a result. Lots of pages are years old, so I am not sure I have current information. I am hoping the awesome Meteor community can help me with the following questions:

(1) I have read that under certain circumstances, oplog tailing reduces performance. Would that be possible in my situation? Do I need to A/B test my performance with/without oplog tailing?

(2) I am aware of redis-oplog as a “drop-in” replacement for oplog tailing. By “drop-in”, does that mean I have to set up oplog tailing as a prerequisite to redis-oplog, or can I simply install redis-oplog and not bother to set up oplog (i.e. configure for a replica set, define the mongo oplog environment variable, etc)?

(3) Would redis-oplog be well-suited to my needs?

(4) What are the key performance indicators that I should be monitoring in order to determine that whatever choice I make is a good (or bad) one?

(5) More generally, what are people using these days for application performance management when self-hosting? I am spoiled by my years with Galaxy and its Kadira-based APM. I saw Monti APM—are people having good experiences with that? Other ideas?

I may run back to Galaxy with my tail between my legs after learning that I am terrible at DevOps, but so far I am finding this to be an interesting educational process. I will be happy to share my lessons learned with the community.

Thanks,
Philip

2

Have you considered using meteor-up (mup) to simplify your deployment? All it needs are SSH details into a box and it can do all the docker install, nginx setup, meteor & mongo install. oplog tailing seems to work for me with the addition of a single line to my mup config, and as you can see from that post oplog increased performance significantly.

I can highly recommend monti-apm. From the same author as meteor-up. The free tier is great, but the $5 tier with extended logs is well worth it (plus @zodern deserves it!)

Keep persisting! There’s no need for Galaxy with the app you describe. I have production SAAS apps with more users than you’re describing on a $5/mo Digital Ocean instance and it barely breaks a sweat and has 100% uptime.

3 Likes
3

@philipmjohnson

  1. With your low number of users - oplog tailing should be fine (caveat: if you have a lot of observers, could be an issue). Btw, you can set single mongo instance as a replica set. You don’t need multiple instances to get oplog.
  2. To use redis-oplog, you also need to run redis server in the same network – again, I think it’s overkill
  3. (see above)
  4. As @wildhart mentioned, Monti-APM would do the job. The best indicators are typically delay for methods / subs – everything boils down to that: how fast do you serve your users. You may care about memory / cpu if you have a weaker / shared hardware
  5. As far I know, Kadira-based is the common. There are a few attempts to use more generic nodejs based monitors, but these look like custom implementations which are incomplete vs Kadira
1 Like
4

Thanks so much for the quick replies, @wildhart and @ramez!

I spent yesterday exploring MUP, and you’re right: it’s way easier and better than my hand-rolled Docker approach. (https, force-ssl, and oplog tailing “for free” are definite winners).

My new deployment approach is documented here:

As noted on that page and here, there is a problem with the Monti APM agent that prevents me from using it at the moment. I have signed up for the paid tier anyway in hopes this will encourage @zodern to provide a fix. :slight_smile:

2 Likes
5

Until monti-apm is fixed, you could try downloading the 2.32 relase into your packages folder, and then update its package.json file as per this commit.

6

That seemed like a great plan! I downloaded the 2.32 release into a local packages/montiapm:agent directory, and updated the package.json, but when I ran meteor add montiapm:agent I got the following error:

$ meteor add montiapm:agent
montiapm:agent: updating npm dependencies -- debug, kadira-core, pidusage, evloop-monitor, lru-cache, json-stringify-safe...
/Users/philipjohnson/.meteor/packages/meteor-tool/.1.11.1.11tawgt.8emlk++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:218
      throw error;
      ^

Error: ENOENT: no such file or directory, rename '/Users/philipjohnson/github/radgrad/radgrad2/app/packages/montiapm_agent/.npm/package-new-22w2xn.eratf' -> '/Users/philipjohnson/github/radgrad/radgrad2/app/packages/montiapm:agent/.npm/package'
    at Object.renameSync (fs.js:741:3)
    at /Users/philipjohnson/.meteor/packages/meteor-tool/.1.11.1.11tawgt.8emlk++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/tools/fs/tools/fs/files.ts:1638:23
    at Object.renameDirAlmostAtomically (/Users/philipjohnson/.meteor/packages/meteor-tool/.1.11.1.11tawgt.8emlk++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/tools/fs/tools/fs/files.ts:1073:7)
    at completeNpmDirectory (/tools/isobuild/meteor-npm.js:811:9)
    at createFreshNpmDirectory (/tools/isobuild/meteor-npm.js:768:3)
    at Object.meteorNpm.updateDependencies (/tools/isobuild/meteor-npm.js:117:7)
    at Object.compile (/tools/isobuild/compiler.js:128:19)
    at /tools/isobuild/isopack-cache.js:354:30
    at Object.enterJob (/tools/utils/buildmessage.js:388:12)
    at IsopackCache._loadLocalPackage (/tools/isobuild/isopack-cache.js:309:18)
    at /tools/isobuild/isopack-cache.js:243:16
    at Function.time (/Users/philipjohnson/.meteor/packages/meteor-tool/.1.11.1.11tawgt.8emlk++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/tools/tool-env/tools/tool-env/profile.ts:284:12)
    at /tools/isobuild/isopack-cache.js:242:17
    at Object.enterJob (/tools/utils/buildmessage.js:388:12)
    at IsopackCache._ensurePackageLoaded (/tools/isobuild/isopack-cache.js:233:20)
    at /tools/isobuild/isopack-cache.js:75:14
    at /tools/packaging/package-map.js:57:7
    at Function._.each._.forEach (/Users/philipjohnson/.meteor/packages/meteor-tool/.1.11.1.11tawgt.8emlk++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:191:9)
    at exports.PackageMap.eachPackage (/tools/packaging/package-map.js:49:7)
    at IsopackCache.buildLocalPackages (/tools/isobuild/isopack-cache.js:74:24)
    at /tools/project-context.js:890:25
    at Object.enterJob (/tools/utils/buildmessage.js:388:12)
    at ProjectContext._buildLocalPackages (/tools/project-context.js:889:18)
    at /tools/project-context.js:300:9
    at Object.enterJob (/tools/utils/buildmessage.js:388:12)
    at ProjectContext._completeStagesThrough (/tools/project-context.js:290:18)
    at /tools/project-context.js:282:12
    at Function.run (/Users/philipjohnson/.meteor/packages/meteor-tool/.1.11.1.11tawgt.8emlk++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/tools/tool-env/tools/tool-env/profile.ts:289:14)
    at ProjectContext.prepareProjectForBuild (/tools/project-context.js:281:13)
    at /tools/cli/commands-packages.js:2160:20
    at Object.capture (/tools/utils/buildmessage.js:283:5)
    at Command.func (/tools/cli/commands-packages.js:2159:27)
    at /tools/cli/main.js:1528:15 {
  errno: -2,
  syscall: 'rename',
  code: 'ENOENT',
  path: '/Users/philipjohnson/github/radgrad/radgrad2/app/packages/montiapm_agent/.npm/package-new-22w2xn.eratf',
  dest: '/Users/philipjohnson/github/radgrad/radgrad2/app/packages/montiapm:agent/.npm/package'
}
~/g/r/r/app (issue-310|●86✚2) [1] $

The packages/montiapm_agent/.npm directory exists, but is empty.

A little googling led me to Local packages ENOENT .npm missing #7711, but not sure how to proceed from here.

Ideas? I don’t want to back off to a prior version of Meteor, which I’m kind of suspecting is the ultimate cause of the problem.

7

Just want to note a happy ending to this problem. @zodern responded very quickly to my issue report:

And, just like that, my problem disappeared.

While I still think that Galaxy has a crucial role to play in supporting larger scale Meteor applications, my initial experience with mup and monti-APM lead me to believe that these packages, in combination with (say) Digital Ocean droplets, provide an excellent low-cost alternative for experimentation and small-scale apps.

4 Likes