@evolross, we never cease learning! Thanks so much for this. We just started pulling out reactive-publish and are already seeing the positive effects (speed, less needless recomputes, less redis-oplog errors, better control).
The needless recomputes were killing us! Now we just added 20% more users to our servers (mind you, lots more code and testing before we push to prod)
@evolross can you provide an example of how you write your custom publications? I’m not sure I understand how manually adding/updating/removing would be more efficient than a well written reactive-publish.
Qualia rolled out redis oplog last night and our average db query execution time went from 300ms -> 3ms… And that’s without using any of the special redis oplog features. Thanks @diaconutheodor!
We run a multi-tenant architecture, so we went from having thousands of open oplog cursors to <10 (will be zero soon). So deploying redis oplog was particularly transformative for us.
Wow @veered that’s an awesome improvement! I have been trying redis-oplog out on my beta servers for a couple of days but it is actually slower than normal oplog for us. I’ve fixed some issues with my reactive publishers but method response time is still slower than normal oplog. It also hasn’t helped with my servers locking up when the CPU load gets too high… If I get more than 70 users on a pod the CPU usage suddenly jumps to 100% and stays there until the pod is killed by my health checker. I’ve got some more debugging to do haha.
What version of meteor and node are you using? I’m thinking that node 8.12 might be a problem per this issue https://github.com/meteor/meteor/issues/10216. I initially upgraded to 8.12 because I thought I was running into the fiber explosion problem, but perhaps something else is going on.
We’re on Meteor 1.6 and Node 8.12.
Redis oplog didn’t have any impact on performance when deployed to a single server. But when we deployed redis oplog to all of our servers the load on the db decrease dramatically. And that’s why we had such massive improvements.
Edit: It’s worth noting that each server is connected to a different db and a different Redis server. Our performance improvements were because we no longer needed to have open oplog cursors on the db.
@imagio A bit off-topic, but sure. Here’s an involved reactive join-type example custom publication.
Say you have a reddit style message board and you want to publish all the Users including name and email who commented on a given Thread. Just for fun and to illustrate the flexibility and power, let’s also publish the count of each user’s Comments on that Thread. Let’s say each Comment contains the userId and the threadId. But a Thread document does not contain any knowledge of which Users commented or how many Comments the User created. So we’ll have to query to gather all that up.
If you used normal pub-sub, you would break reactivity because logically you’ll need to do a find on Comments passing a threadId, then you’d need to only get the unique userIds from those Comments, query the Users for their name and email, somehow tally up a comment-count, and then pass that all back via a reactive cursor. Bad news… none of that can be reactive. Hence your use of reactive-publish.
However, if we used observe on the Comments collection and break all this up using .added, .changed, and .removed, and think about it as Comments get added, updated, and removed, we can pull this off easily without any extra packages and have it be totally reactive. Anytime you can eliminate relying on a package, it’s usually more optimal/efficient.
The publication looks like this:
Meteor.publish("usersAndCommentCountsByThread", function(threadId) {
check(threadId, Match.OneOf(String, null));
let self = this;
// Check the thread for existence
let thread = Threads.findOne(threadId, {fields: {_id: 1}});
if(thread) {
// Create an object to track userIds that are already published and their comment counts on the thread
// NOTE: There's a lot of ways to do this part depending on what you want to do in you pub
let usersAndCommentCounts = {};
// Query and observe comments since that is what will affect users and their comments count
let commentsHandle = Comments.find({threadId: threadId}, {fields: {userId: 1}}).observe({
// This code fires any time a new comment is added that matches the above query INCLUDING
// any EXISTING comments that match this query
added: function(addedComment) {
// Check if this comment's user is already in the users object, if not, add it
if(!usersAndCommentCounts[addedComment.userId]) {
// If this is the user's first comment, add the user's id to object of users and set their
// comment-count to one
usersAndCommentCounts[addedComment.userId] = 1;
// Query the user getting their name and email
let user = Users.findOne(addedComment.userId, {fields: {name: 1, email: 1}});
// Load the count onto the queried user
user.commentCount = usersAndCommentCounts[addedComment.userId];
// Now publish the user and their comment-count. Note that we're publishing the comment's
// userId since we're actually publishing users in this pub
self.added("usersCommentCounts", addedComment.userId, user);
}
else {
// Else, simply increment this user's comment-count
usersAndCommentCounts[addedComment.userId]++;
// Publish the changed comment-count
let updatedFields = {commentCount: usersAndCommentCounts[addedComment.userId]};
self.changed("usersCommentCounts", addedComment.userId, updatedFields);
}
},
// This code fires any time a comment is updated that matches the above query
changed: function(changedComment, priorComment) {
// There's nothing to add here because if a comment is only changed/updated it doesn't affect
// the list of users or their comment-count. So this whole block can be commented out. I just left
// it in to demonstrate. If we were also doing something like counting "total words" across
// all comments per user, then we would want to use this to update a word-count if they changed
// one of their comments. If you were doing something here, it would look like this.
// Do your logic for updating your counts, etc. and create an object of updated fields
// let updatedFieldsObject = {wordCount: ...};
// Call this.changed with the existing document id and updated fields object
// this.changed("usersCommentCounts", changedComment.userId, updatedFieldsObject);
},
// This code fires any time an existing comment is removed that matches the above query
removed: function(removedComment) {
// Check if this user has any additional comments (could query actual DB here for comments by this
// User and do a count OR rely on our own tracked count)
if(usersAndCommentCounts[removedComment.userId] === 1) {
// Since the user has no more comments on this thread, remove the user and
// comment-count from our object
delete usersAndCommentCounts[removedComment.userId];
// Remove the User from the publication since they don't have any more comments on this thread
// Note that we're passing the userId of the comment since we're publishing users in this pub
self.removed("usersCommentCounts", removedComment.userId);
}
else {
// Decrement the user's comment-count
usersAndCommentCounts[removedComment.userId]--;
// Since the user's comment-count updated, but the user is still being published, we need to
// only update the publication with the new data
let updatedFields = {commentCount: usersAndCommentCounts[removedComment.userId]};
self.changed("usersCommentCounts", removedComment.userId, updatedFields);
}
}
});
// Mark the subscription as ready
self.ready();
// Stop observing the cursor when client unsubscribes
self.onStop(function () {
commentsHandle.stop();
});
}
else {
self.ready(); // If no thread, mark publication as ready
}
});
As I said, it’s a lot more code. But this is what is happening under the hood by Meteor when you don’t define your own .added, .changed, and .removed handlers. But when you do, you can see you have ultimate flexibility. You don’t have to do this with every publication, but when you need it, it’s a great tool.
The last thing you may be asking is, what the heck is that first argument "usersCommentCounts" that is passed into each publication function? That is simply the string name of the Collection that you’re publishing too. The cool thing about this is you can use any of your existing document Collections here. So if you have a Collection called Comments defined somewhere like Comments = new Meteor.Collection('comments') and you were doing a custom publication that was ultimately returning Comments, you can simply pass "comments" as this argument. I don’t know what the string name of Meteor.users is (only because I’ve never custom published to Meteor.users). But if you publish to an existing Collection like this, I don’t think you can mix your custom publication with another publication of the same document type. So, more often than not, you’ll just want to simply make a new client-only collection to hold your published documents, which is what I’m doing in the example above. So somewhere in client code you’ll simply have this:
Meteor.startup(function() {
// Create client-only collection to hold special custom publications
UsersCommentCounts = new Mongo.Collection("usersCommentCounts");
});
Now you can publish directly to this client-only Collection and never have to worry about it mixing with one of your client-server mongo Collection publications. On the client, to subscribe you’ll just have:
Meteor.subscribe("usersAndCommentCountsByThread", threadId);
and then of course to find the documents you’ll have:
UsersCommentCounts.find();
With the above, you can publish all kinds of crazy stuff mixing DB documents/fields, custom counters, 3rd party API request data… all kinds of combinations. You might create a publication that doesn’t even publish any DB data, just stuff you generate on the fly based around DB document data.
Hopefully this opened your mind a little if you didn’t already know about this.
@evolross Thanks so much for the detailed reply! That is definitely helpful and I think I can apply that technique to some of my publications. I was already doing something similar to publish counts for badges on my navigation.
So that I don’t pollute this thread with off topic things I have started a new thread to discuss this further. I would appreciate it if you could take a look at it! Reactive joins without using any packages
Any future plans integrating Redis 5.0 and use Redis Streams for redis-oplog? It seems like it’s gonna solve Meteor scaling issues even further…
Guys, look, we have an ‘events.files’ collection, that has thousands of inserts per second.
MongoDB is being updated by Node.JS. We are manually sending a change to the Redis from an insert callback. The documents all have the same taskId field, which is a Mongo object.
We tried to do it the following way and we see the event in a debug mode of redis-oplog.
But the collection is not reactive and re-subscription doesn’t happen on Meteor side🙁
redis.publish('events.files', JSON.stringify({
e: 'i',
d: {task: taskId}
}));
What are we doing wrong? Please heeeeeeeelp!
@martineboh is there any difference in their pub/sub api ?
@dimanjet the document does not look like that. It contains the _id of the document as well! Look how we do it: https://github.com/cult-of-coders/redis-oplog/blob/master/lib/mongo/lib/dispatchers.js#L78 Sometimes the source-code is your best documentation! 
Hope you solve it.
Thank you! It makes things a lot clearer.
I wonder if it is possible to make bulk insert and publish all of them by one common field? We don’t know the _id here yet, because we have an array of several thousands elements. I don’t think would be a good solution to make publish for the every document.
We’re also in active process of trying tulip/oplogtoredis solution. This program tails the oplog of a Mongo server, and publishes changes to Redis. But change events are not seen back in Meteor yet. Will try that too.
No support for bulk elements. A thousand pushes to redis per second is nothing in my humble opinion But if you like you can implement that yourself, I can offer you some guidelines, and do a PR.
Oplogtoredis is genious but is good if you have lots of sources updating mongodb.
Our production app regularly does a thousand pushes through redis-oplog a second and I can confirm that it’s nothing - especially for Redis. It barely makes a bump on performance charts on Compose.io’s Redis product. How Meteor handles that action… that depends on what those one thousand pushes are doing.
Security
I am still trying to comprehend the security implications with Redis Oplog. This is the Security Specs for Mongo Atlas (if the Oplog is used with them): Mongo Atlas
Is there anyone experienced with TLS and encryption within Redis? Has anyone used Redis with something other than naked http?
Tx.
On AWS, we used a redis instance (elasticache) in the same private network as the EC2 instances in EBS, so we had no concerns for naked http (and it’s less CPU intensive). If you have multi-region servers, you can enable in the AWS UI secure connection (i.e. over ssh tunnel). I think it’s the standard way of doing things with redis as it does not come with built-in encryption (at least not the versions we are using)
You can, of course, do it all yourself, but that can turn costly and risky quickly if you don’t have the expertise in-house.
It’s not encryption but you can add IP white-listing to your Redis deployments. You make a good point though… will have to look into securing Redis more.
I’m just curious if SyntheticMutator is working for anyone who is using Meteor 1.8. None of the methods work for me as per this issue.
How is the compatibility with Astronomy? Does it work out of the box or do you need to do a lot of manual hookups?
It’s just standard pub/sub @jorgeer it should work without any change just like normal pub/sub (This is what we mean by full BC). Unless astronomy is doing some weird hacks to the internal live processing of Meteor.
@diaconutheodor I just remember reading this https://github.com/cult-of-coders/redis-oplog/issues/235, which is closed but I wasn’t sure if it was solved or if there are outstanding problems that are not resolvable because of the way astronomy uses the collections.