Without any heavy load, there is already a significant waiting time due to the way livedata_server.js handles publishing, refer to :
I think @storyteller was economical with the details. The reywood:publish-composite package used to be employed for all sorts of work that GraphQL and a few other tools came to solve in recent times. But reywood:publish-composite remains quite useful in a lot of situations.
If you had a look at the 6 pull requests merged since Oct 2023, some of them pretty large, you’d have a different opinion 
: Issues · Meteor-Community-Packages/meteor-publish-composite · GitHub
Read the sentence again. The key word is IF. The main point being that you should think about your data. NOWHERE did I mention anything about removing pub/sub (especially from the core - totally ridiculous point in all aspects). Also for when you need them reactivelly the point is to think better on how to manage and structure the data than using reywood:publish-composite and just get everything on the client. If not handled properly this will just create more issues. @illustreets got it right.
This topic is here to suggest packages that have better alternatives, or approaches. Especially if they are legacy or maintainers are not supported properly.
Yes, I’ve read the IF.
**reywood:publish-composite**
Especially if you don’t need the data reactivelly, then it is better to migrate to GraphQL.
That is the point. If we use publish-composite, we NEED the data reactively. So mention a none reactive solution makes no sense here. We have methods for none reactivate data.
Yeah, I understand that. But GraphQL isn’t reactive (out of the box). If there are better packages for publishing multiple collections I would agree with you.
Great then it does not concern your use case (do not assume that everyone is using that package responsively as you do), so why do you assume that I want to remove pub/sub from the core?
Given the age of the package there are many old apps that have pub/sub for all the data and don’t even consider using methods or other non-reactive approaches for data that they don’t need reactivelly.
Others got the point well and we moved on updating publish-composite based on this thread. I don’t understand the hostile tone, why you reached that conclusion, but I’m going to update the opening post to make things more clear for future readers.
Do note, that since there is another thread about this that you have created I will remove any discussion (or if I can figure out how, move it to the other thread) on this topic here as it is off-topic to the thread here and there is other thread to discuss this.
The comment was not meant to be an attack, I respect the work you are doing for the community. The solution you proposed (GraphQL) made me assume that there is no longer a focus on libaries that work with reactive data. And publish-composite is still the proposed solution for relational data. If there is a better package or better solutions that support reactivity, I fully agree with you that publish-composite can be replaced.
Honestly we need a more dedicated maintainer for publish-composite or a newer solution. For many Reactive Aggregations (I do recall seeing a package for that somewhere) I think would cover most use cases, but then there are others where you simply need the data in minimongo in which case I currently don’t know of a better solution. Which is what I hope someone might suggest in this thread.
I’ve been at it for a while, though I cannot claim I know it very well. That package still has secrets as far as I’m concerned. But I didn’t find it hard to work with it either. It’s a combination of time and lots of testing. I’m happy to keep contributing to it for the foreseeable future. We use it in production for a few lightweight, but essential publications of multiple related collections.
or a newer solution
For what it does, provided it’s small scale, hard to see what could replace it.
Some of wish to be helping this but since we are limited in knowledge. If you can give some lesdons on it or if there is a lesson give a link to understand this situations that some of us will start to help. Thanks.
I’m one of those devs who use reactivity only when absolutely necessary but Redis Oplog isn’t crucial for me. I do understand it’s crucial for those who use a lot of pub/sub but it’s not crucial nor necessary if you only have one or two pub/sub and you know how to use use them correctly.
The missing qualifier here is having a busy application requiring multiple server instances. Oplog tailing already eats resources proportional to the number of server instances, even without considering your pub/sub.
1 Like
True, but using this package may improve load sustainability on high-load projects reducing maintenance and servers costs. So it really depends from size of the project and its complexity
1 Like
- Determine which area you want to contribute in (specific packages, Meteor itself, community organization, docs, etc.)
- If packages or Meteor itself, go to that repository and familiarize yourself with the contributing guidelines and the code.
2a. Optional, but highly recommended, familiarize yourself with the key people in the project. - Take on an issue that you have in your project or a small issue that you can do. Don’t be afraid to reach out and ask the maintainers to guide you.
- Create a PR with your contribution.
- Repeat, get involved in the issues discussions, and so on.
Each project is a bit different, but the key is to start solving issues and contributing. Another option is to sponsor the maintainers and get involved in the discussion about the directions of the project. Feel free to reach out to me in PMs if you have specific ideas in mind and want more directions.
I think another package that we should recommend for people to move away from (over the long term) is underscore. There is an ongoing effort to remove as much of it from core packages and update our Meteor version to a newer version (though there are still some bugs). The issue being that often you can get the specific function from NPM packages and there is also:
1 Like
It depends on what you are doing. For some database operations, you want an “opt out” option. An example: If you remove a huge amount of temporary data (f.e. 100.000 documents or more) at once, your Meteor application will crash. Why? Because Meteor will process every single remove event when you use the integrated oplog integration. With redis-oplog, you can call remove via the native database methods (collection.rawCollection()). In this case, Meteor will “ignore” the removed documents because the updates are not sent to the Redis instance.
Hi, I only use rawCollection throughout my whole data heavy backend app, since many years before Meteor added transactions. So guess what, what you described isn’t a problem at all for me.
FYI - I have 14 millions docs in my database
How do you do reactivity with just rawCollection()?
How many concurrent server instances and users do you serve?
I mostly don’t and it’s not needed. There’s some data that is sent from the Backend app via the “StreamyMsg” package that informs the Frontend about data updates which is then displayed, so that’s eg used on progress bars or to update displayed data.
My app is very data heavy as it keeps DNA data but there aren’t many concurrent users (but data grows fast from those few users) which some other people have at the opposite site of the spectrum of apps (meaning several thousand of users accessing more or less the same data). So the M10 Atlas server was always fine with it, even during peak times it never broke a sweat.
Remains to be seen if that scales in the future or if I need to add Redis-Oplog at some point in the (far) future.
My whole point of my answer was that there’s isn’t one single use case in Meteor apps and everyone has the volume that they need Redis-Oplog. On the contrary I think, my guess is that there are many smaller apps in Production that can achieve the necessary reactivity whilst avoiding pub/sub like Count Dracula avoids the sunlight 
Hence my view that there’s nothing wrong with keeping Redis-Oplog out of the Meteor core for now and keep Changestreams in view in the meantime as an alternative, or whatever changes might happen in the future.
Im summary, you are not using Meteor’s reactivity. With the same logic in your conclusion, you do not even need the pub/sub feature in core, whether it is oplog-tailing, changestream, or redis-oplog.
2 Likes