Here is a list of packages that you might no longer need in Meteor 3 or you want to replace during this migration.
communitypackages:picker
Use WebApp instead as it is now express and can handle the routing that Picker was providing.
reywood:publish-composite
Especially if you don’t need the data reactivelly. While this is a great packages it can have serious side effects and performance issues. I highly recommend that you review its usage, if the data it provides are really needed reactively and if not consider either GraphQL or MongoDB aggregations and data retrieval via methods.
natestrauser:publish-performant-counts and similar
See Collection.countDocuments and Collection.estimatedDocumentCount for a replacement. It won’t be reactive in a method, so not a full replacement, but for most part it will be fine. Would love to see an upgraded package that could return reactivelly.
I am replacing Grapher (not GraphQL) with Mongo Aggregations in all my projects and I like it more every day.
Another package that should no longer be required in most projects in Meteor 3 (in the future in general) is redis-oplog or other oplog solutions. Not because it is not great but because (I believe) the concept is wrong. Tailing data is very resource expensive and in general, reactivity should be done at minimum and using atomic/granular observers, events, streams. I think most projects do not need reactivity and the best practice is to use local/global stores with purge/update algorithms on the Client. I mention this because I believe Meteor needs a cacheing layer for Methods and Redis has a solution for this. In React at least, reactivity should be done at View/Component level using change streams directly via React hooks.
I posted a bit about it here: Performance report - #14 by paulishca
In general I think we need more demos and POCs, data models for developers to adopt for fixing development problems. Doing complex things is hard in any framework but this is where Meteor could go one step further.
I follow a couple of JS Development accounts in TwiXter and I like how they promote how to do things with NextJS. I don’t use Next and not planning to but I find a lot of good knowledge in those videos. I think we need this for Meteor.
I’ve run into caching a little bit during the stream. Definitely something that needs more exploration as I’m looking to replace subs-cache package and similar.
Subs and methods caching, together with general caching via PWA and so on is something that will have to be explored. I would say a big part of the overall performance story that needs improvement. @xet7 mentioned that he got performance boost from using the deprecated appcache package, if that is the case, the I think that requires an investigation what from that is still working or could be updated as we really never got a proper replacement for that package. He also mentioned that they are using pretty much all the other caching packages out there. Would be great to have these things in core.
Next is the testing story which I feel has deteriorated recently.
But these are different topics. I will create topics for these to discuss these further.
If I remember correctly, the challenge here was the size of the bundle and the limits of localstorage. Proper setup with serviceworkers for caching assets will be a good replacement
Someone posted about grounddb before which looks like a good foundation. We have been using a very basic localstorage-based caching in the client and having something similar integrated with subs/methods will be a big help than manually doing this
I think reywood:publish-composite and GraphQL are not same thing but we are using Mongo Aggregations when reactivity is not required. If we need, reywood:publish-composite is a good solution but could have side effects.
redis-oplog is the most loved and most required package we ever needed. It solves nearly all of the performance issues of reactivity. I think it should be in the core as mongodb
I think the same way for other packages and looking to remove picker as soon as possible.
Perhaps the way you read it makes you feel that I mix things that should not be mixed but it is just the way you read it. Will explain again: I believe that reactivity should be very limited and I give priority to having more data stored on the client. Since I get this data via Methods, I’d prefer to have a Redis in between my Mongo and my App Server. Since I host my DB with Atlas, hitting queries into a Redis would resolve the concurrent connections problem that requires expensive tiers with Atlas.
Perhaps you are right, client and server caches are mixed but not in the way you might have intended to put it.
As for the Pub/Sub, I used them in the past, I only use one subscription at the moment but planning to kill that as well and remain with only the Meteor.userId() subscription but I might turn this off too. I am referring there to a technique (reactivity should be done at minimum and using atomic/granular observers) and not to a capability. I mention that because there are some developers who write here about not getting their 1000 documents after some changes in the DB.
I think you could focus on the text and not the writer. Think of me as a bot. It doesn’t matter what I mix or what I use, what I seem.
We’re quite heavy users of publish-composite but would like to be less so. Certainly some of these uses could be replaced with aggregations… but there are some which we need to keep reactive. natestrauser:publish-performant-counts could be rewritten to use countDocuments underneath? But something to publish a reactive count is still needed - I’d imagine a lot of people have the typical ‘unread’ count etc.
The others I agree
In terms of what should underly our reactivity - IMO the most appealing solution is what @radekmie created in his changestream-to-redis package. I guess the only downside is you need another server/process running this along with your redis db. Assuming I understand it correctly, it also has an advantage over the standard implementation in that non-meteor projects writing to your DB will also trigger reactivity which I believe was an issue in the standard setup - but don’t quote me on that .
It’s a bit off topic, but from your experience can you tell how big the impact of using redis-oplog really is? I know the answer will depend on a ton of conditions, but roughly speaking: will this eg. affect response times for methods and pubs?
As you’ve mentioned, it depends on different things. If oplog-tailing is causing you performance issues, you can see immediate performance gains just by replacing it with redis-oplog. The default install gives you reactivity for _id-based mutations and queries.
For advanced usage, you can then proceed with granular reactivity only for very specific mutations in a collection. The most extreme version of this is reactivity on mutations only on a single document depending on how you query that document in your publication. This means that reactivity will “not be triggered by all mutations” in that collection or even in that single document. Even MongoDB change streams cannot do this granularity of reactivity. This granularity can provide tremendous performance gains versus oplog-tailing.
For most things, the performance gains depend on the developer and his knowledge on how to use any library/package.
That combination becomes a true drop-in replacement of the default meteor reactivity. Adding this combination as core packages and part of official Meteor documentation is an appealing solution for Meteor
In terms of performance:
redis-oplog only > redis-oplog + changestream-to-redis >>> Meteor oplog-tailing
Caveat: I have not used changestream-to-redis in a production app with real heavy traffic
My two cents regarding redis-oplog: I do think it should be supported. I’ve implemented it in a few large projects (both as a consultant and as a developer) and it helps immensely and immediately.
Sure, it involves additional complexity, as it requires an additional Redis instance. But it’s only one $5-10/month instance for a few thousand dollars deployment (or even more). Or at least I never heard of anyone who needed a bigger one.
Yes, that’s true. But again one $5/month instance for the whole deployment (I wrote about it in here). That’s because there’s virtually no state in there.
I slightly disagree here. In my opinion it goes like this:
That’s because without changestream-to-redis the app server has to push things to Redis, which also cost some CPU cycles. Sure, the results are there sooner (thus, better latency), but we’re talking about a few milliseconds here (remember to put Mongo, Redis, and the app as close together as possible).
At aleno, we do use bulkWrites a lot, and in our case changestream-to-redis is the only viable solution. The same goes for people sharing database with other services, as then Meteor doesn’t know that it should push something to Redis.
And of course, it’s not a silver bullet either – for more complex queries, server still has to refetch tens of documents, so “oplog spikes” are still a thing (although manageable). Using FULL_DOCUMENT with protectAgainstRaceConditions: false helps, but it only extends your runway (and slightly increases costs).
@radekmie, for those who don’t use bulkWrites on reactive collections or external mutations, are we going to gain performance if we install changestream-to-redis in our redis-oplog implementation?
It depends on how much is your app pushing to Redis, i.e., how many write operations you have. If there’s a constant stream of those, then it will definitely benefit from it. I can’t tell if it will be visible on CPU/RAM chart (i.e., I can’t guarantee a jaw-dropping CPU gains), but I saw a few % in one project (back then it was with oplog-to-redis, but it would be the same with changestream-to-redis).
.