I’ve run into caching a little bit during the stream. Definitely something that needs more exploration as I’m looking to replace subs-cache package and similar.
Subs and methods caching, together with general caching via PWA and so on is something that will have to be explored. I would say a big part of the overall performance story that needs improvement. @xet7 mentioned that he got performance boost from using the deprecated appcache package, if that is the case, the I think that requires an investigation what from that is still working or could be updated as we really never got a proper replacement for that package. He also mentioned that they are using pretty much all the other caching packages out there. Would be great to have these things in core.
Next is the testing story which I feel has deteriorated recently.
But these are different topics. I will create topics for these to discuss these further.
Have you used redis-oplog in a production app? Seems you don’t because it does reactivity “using atomic/granular observers.”
You are mixing client and server caches.
If I remember correctly, the challenge here was the size of the bundle and the limits of localstorage. Proper setup with serviceworkers for caching assets will be a good replacement
Someone posted about grounddb before which looks like a good foundation. We have been using a very basic localstorage-based caching in the client and having something similar integrated with subs/methods will be a big help than manually doing this
This is a great topic to follow.
I think reywood:publish-composite and GraphQL are not same thing but we are using Mongo Aggregations when reactivity is not required. If we need, reywood:publish-composite is a good solution but could have side effects.
redis-oplog is the most loved and most required package we ever needed. It solves nearly all of the performance issues of reactivity. I think it should be in the core as mongodb 
I think the same way for other packages and looking to remove picker as soon as possible.
Perhaps the way you read it makes you feel that I mix things that should not be mixed but it is just the way you read it. Will explain again: I believe that reactivity should be very limited and I give priority to having more data stored on the client. Since I get this data via Methods, I’d prefer to have a Redis in between my Mongo and my App Server. Since I host my DB with Atlas, hitting queries into a Redis would resolve the concurrent connections problem that requires expensive tiers with Atlas.
Perhaps you are right, client and server caches are mixed but not in the way you might have intended to put it.
As for the Pub/Sub, I used them in the past, I only use one subscription at the moment but planning to kill that as well and remain with only the Meteor.userId() subscription but I might turn this off too. I am referring there to a technique (reactivity should be done at minimum and using atomic/granular observers) and not to a capability. I mention that because there are some developers who write here about not getting their 1000 documents after some changes in the DB.
I think you could focus on the text and not the writer. Think of me as a bot. It doesn’t matter what I mix or what I use, what I seem.
For caching, the discussion splits to here:
Let’s continue here with packages that it might be possible to ditch or replace when working on the Meteor 3 upgrade.
Yes, it must be part of the core.
We’re quite heavy users of publish-composite but would like to be less so. Certainly some of these uses could be replaced with aggregations… but there are some which we need to keep reactive. natestrauser:publish-performant-counts could be rewritten to use countDocuments underneath? But something to publish a reactive count is still needed - I’d imagine a lot of people have the typical ‘unread’ count etc.
The others I agree 
In terms of what should underly our reactivity - IMO the most appealing solution is what @radekmie created in his changestream-to-redis package. I guess the only downside is you need another server/process running this along with your redis db. Assuming I understand it correctly, it also has an advantage over the standard implementation in that non-meteor projects writing to your DB will also trigger reactivity which I believe was an issue in the standard setup - but don’t quote me on that 
.
It’s a bit off topic, but from your experience can you tell how big the impact of using redis-oplog really is? I know the answer will depend on a ton of conditions, but roughly speaking: will this eg. affect response times for methods and pubs?
For those who still need it, I’ve updated the package to work with Meteor 3.0. Any news when Update tests by StorytellerCZ · Pull Request #173 · Meteor-Community-Packages/meteor-publish-composite · GitHub is going to be merged?
As you’ve mentioned, it depends on different things. If oplog-tailing is causing you performance issues, you can see immediate performance gains just by replacing it with redis-oplog. The default install gives you reactivity for _id-based mutations and queries.
For advanced usage, you can then proceed with granular reactivity only for very specific mutations in a collection. The most extreme version of this is reactivity on mutations only on a single document depending on how you query that document in your publication. This means that reactivity will “not be triggered by all mutations” in that collection or even in that single document. Even MongoDB change streams cannot do this granularity of reactivity. This granularity can provide tremendous performance gains versus oplog-tailing.
For most things, the performance gains depend on the developer and his knowledge on how to use any library/package.
In terms of usage:
Oplog-tailing reactivity = changestream-to-redis + redis-oplog reactivity
That combination becomes a true drop-in replacement of the default meteor reactivity. Adding this combination as core packages and part of official Meteor documentation is an appealing solution for Meteor
In terms of performance:
redis-oplog only > redis-oplog + changestream-to-redis >>> Meteor oplog-tailing
Caveat: I have not used changestream-to-redis in a production app with real heavy traffic
My two cents regarding redis-oplog: I do think it should be supported. I’ve implemented it in a few large projects (both as a consultant and as a developer) and it helps immensely and immediately.
Sure, it involves additional complexity, as it requires an additional Redis instance. But it’s only one $5-10/month instance for a few thousand dollars deployment (or even more). Or at least I never heard of anyone who needed a bigger one.
Yes, that’s true. But again one $5/month instance for the whole deployment (I wrote about it in here). That’s because there’s virtually no state in there.
I slightly disagree here. In my opinion it goes like this:
- Performance:
redis-oplog+changestream-to-redis>redis-oplog>>> oplog-tailing - Latency:
redis-oplog>redis-oplog+changestream-to-redis>>> oplog-tailing
That’s because without changestream-to-redis the app server has to push things to Redis, which also cost some CPU cycles. Sure, the results are there sooner (thus, better latency), but we’re talking about a few milliseconds here (remember to put Mongo, Redis, and the app as close together as possible).
At aleno, we do use bulkWrites a lot, and in our case changestream-to-redis is the only viable solution. The same goes for people sharing database with other services, as then Meteor doesn’t know that it should push something to Redis.
And of course, it’s not a silver bullet either – for more complex queries, server still has to refetch tens of documents, so “oplog spikes” are still a thing (although manageable). Using FULL_DOCUMENT with protectAgainstRaceConditions: false helps, but it only extends your runway (and slightly increases costs).
@radekmie, for those who don’t use bulkWrites on reactive collections or external mutations, are we going to gain performance if we install changestream-to-redis in our redis-oplog implementation?
It depends on how much is your app pushing to Redis, i.e., how many write operations you have. If there’s a constant stream of those, then it will definitely benefit from it. I can’t tell if it will be visible on CPU/RAM chart (i.e., I can’t guarantee a jaw-dropping CPU gains), but I saw a few % in one project (back then it was with oplog-to-redis, but it would be the same with changestream-to-redis).
Merged just now, but still will need more work on the tests to move them to GitHub Actions.
Can you please give feedback on this: Fix for conflicting cursor events by gabrielcazacu96 · Pull Request #177 · Meteor-Community-Packages/meteor-publish-composite · GitHub
If we could merge that as well then the only remaining issue is to decide what the version bump should be.
Thank you. I also just sent a tiny pull request, with the test dependencies fix I mentioned in a comment under the previous pull request. See here: Fixed tests dependencies by manueltimita · Pull Request #178 · Meteor-Community-Packages/meteor-publish-composite · GitHub
I will have a look at #177 these days.
Since I got your attention, maybe you can review Meteor 3.0 compatibility by manueltimita · Pull Request #99 · percolatestudio/publish-counts · GitHub? I’ve applied the changes you’ve requested. Thanks!
Regarding all the discussions here on Oplog, I recently published a PR which also improves the performance aspect of the oplog tailing a lot: New mongo package options to optimize Oplog tailing performance to include/exclude certain collections by Twisterking · Pull Request #13009 · meteor/meteor · GitHub
In regards to packages: We also use Grapher quite heavily at Orderlion, but are now replacing it with the successor @bluelibs/nova - @diaconutheodor and his team really did an awesome job there!
I personally am now a huge friend of Mongo aggregations. When you have multiple collections to “join” and have nested stuff, this can become a huge pain in the a** to get right in my experience.