In terms of what should underly our reactivity - IMO the most appealing solution is what @radekmie created in his changestream-to-redis package. I guess the only downside is you need another server/process running this along with your redis db. Assuming I understand it correctly, it also has an advantage over the standard implementation in that non-meteor projects writing to your DB will also trigger reactivity which I believe was an issue in the standard setup - but don’t quote me on that 
.
2 Likes
It’s a bit off topic, but from your experience can you tell how big the impact of using redis-oplog really is? I know the answer will depend on a ton of conditions, but roughly speaking: will this eg. affect response times for methods and pubs?
For those who still need it, I’ve updated the package to work with Meteor 3.0. Any news when Update tests by StorytellerCZ · Pull Request #173 · Meteor-Community-Packages/meteor-publish-composite · GitHub is going to be merged?
2 Likes
As you’ve mentioned, it depends on different things. If oplog-tailing is causing you performance issues, you can see immediate performance gains just by replacing it with redis-oplog. The default install gives you reactivity for _id-based mutations and queries.
For advanced usage, you can then proceed with granular reactivity only for very specific mutations in a collection. The most extreme version of this is reactivity on mutations only on a single document depending on how you query that document in your publication. This means that reactivity will “not be triggered by all mutations” in that collection or even in that single document. Even MongoDB change streams cannot do this granularity of reactivity. This granularity can provide tremendous performance gains versus oplog-tailing.
For most things, the performance gains depend on the developer and his knowledge on how to use any library/package.
In terms of usage:
Oplog-tailing reactivity = changestream-to-redis + redis-oplog reactivity
That combination becomes a true drop-in replacement of the default meteor reactivity. Adding this combination as core packages and part of official Meteor documentation is an appealing solution for Meteor
In terms of performance:
redis-oplog only > redis-oplog + changestream-to-redis >>> Meteor oplog-tailing
Caveat: I have not used changestream-to-redis in a production app with real heavy traffic
My two cents regarding redis-oplog: I do think it should be supported. I’ve implemented it in a few large projects (both as a consultant and as a developer) and it helps immensely and immediately.
Sure, it involves additional complexity, as it requires an additional Redis instance. But it’s only one $5-10/month instance for a few thousand dollars deployment (or even more). Or at least I never heard of anyone who needed a bigger one.
Yes, that’s true. But again one $5/month instance for the whole deployment (I wrote about it in here). That’s because there’s virtually no state in there.
I slightly disagree here. In my opinion it goes like this:
- Performance:
redis-oplog+changestream-to-redis>redis-oplog>>> oplog-tailing - Latency:
redis-oplog>redis-oplog+changestream-to-redis>>> oplog-tailing
That’s because without changestream-to-redis the app server has to push things to Redis, which also cost some CPU cycles. Sure, the results are there sooner (thus, better latency), but we’re talking about a few milliseconds here (remember to put Mongo, Redis, and the app as close together as possible).
At aleno, we do use bulkWrites a lot, and in our case changestream-to-redis is the only viable solution. The same goes for people sharing database with other services, as then Meteor doesn’t know that it should push something to Redis.
And of course, it’s not a silver bullet either – for more complex queries, server still has to refetch tens of documents, so “oplog spikes” are still a thing (although manageable). Using FULL_DOCUMENT with protectAgainstRaceConditions: false helps, but it only extends your runway (and slightly increases costs).
1 Like
@radekmie, for those who don’t use bulkWrites on reactive collections or external mutations, are we going to gain performance if we install changestream-to-redis in our redis-oplog implementation?
It depends on how much is your app pushing to Redis, i.e., how many write operations you have. If there’s a constant stream of those, then it will definitely benefit from it. I can’t tell if it will be visible on CPU/RAM chart (i.e., I can’t guarantee a jaw-dropping CPU gains), but I saw a few % in one project (back then it was with oplog-to-redis, but it would be the same with changestream-to-redis).
1 Like
Merged just now, but still will need more work on the tests to move them to GitHub Actions.
Can you please give feedback on this: Fix for conflicting cursor events by gabrielcazacu96 · Pull Request #177 · Meteor-Community-Packages/meteor-publish-composite · GitHub
If we could merge that as well then the only remaining issue is to decide what the version bump should be.
1 Like
Thank you. I also just sent a tiny pull request, with the test dependencies fix I mentioned in a comment under the previous pull request. See here: Fixed tests dependencies by manueltimita · Pull Request #178 · Meteor-Community-Packages/meteor-publish-composite · GitHub
I will have a look at #177 these days.
1 Like
Since I got your attention, maybe you can review Meteor 3.0 compatibility by manueltimita · Pull Request #99 · percolatestudio/publish-counts · GitHub? I’ve applied the changes you’ve requested. Thanks!
Regarding all the discussions here on Oplog, I recently published a PR which also improves the performance aspect of the oplog tailing a lot: New mongo package options to optimize Oplog tailing performance to include/exclude certain collections by Twisterking · Pull Request #13009 · meteor/meteor · GitHub
In regards to packages: We also use Grapher quite heavily at Orderlion, but are now replacing it with the successor @bluelibs/nova - @diaconutheodor and his team really did an awesome job there!
I personally am now a huge friend of Mongo aggregations. When you have multiple collections to “join” and have nested stuff, this can become a huge pain in the a** to get right in my experience.
The Aggregation tab of the recent version of MongoDB compass has been very helpful when creating Aggregation queries
+1 here, RedisOpLog is crucial part. Even for the apps/devs who use reactivity only when absolutely necessary.
2 Likes
We’ve recently started using aggregations more heavily again, particularly with migrations which are really nice to be able to test in Compass. What about with publications though. Have any of you used jcbernack:reactive-aggregate (or edit) tunguska:reactive-aggregate (which seems more up-to-date) ?
I’ve used its successor GitHub - robfallows/tunguska-reactive-aggregate: Reworks jcbernack:reactive-aggregate for ES6/ES7 and Promises in projects before. It worked nicely. I can’t speak to its performance under heavy load though. Curious to hear other experiences.
That is a great idea! So why not remove all publication features from the core and only provide methods?
I think it’s a great idea if you just remove all the unique features of Meteor and adapt Meteor to other frameworks like NextJS. That way we’ll have the 100th Javascript framework with the same features 
2 Likes
Without any heavy load, there is already a significant waiting time due to the way livedata_server.js handles publishing, refer to :
I think @storyteller was economical with the details. The reywood:publish-composite package used to be employed for all sorts of work that GraphQL and a few other tools came to solve in recent times. But reywood:publish-composite remains quite useful in a lot of situations.
If you had a look at the 6 pull requests merged since Oct 2023, some of them pretty large, you’d have a different opinion : Issues · Meteor-Community-Packages/meteor-publish-composite · GitHub