New package - jam:pub-sub - Method-based and Change Streams-based pub/sub, and subscription caching

announce
21

So what happens if we delete temporary data (100.000+ docs) per day? Can we „opt out“ so that Meteor ignores the changes from the remove operation?

22

That is insane!! great job @jam I will ping the team so that we can start testing it right way, it is awesome to have this feature in a package

2 Likes
23

I did a quick review of your code regarding the change streams and here’s a few points:

  1. I see you added a comment about connection pool, but from my experiments (*), having a change stream per publication is simply not going to work.
    • If you’d use an M30 instance on MongoDB Atlas (it’s already quite large), you’ll have 3k connections limit. Yes, that means less than 3k different change streams available in total.
    • If you think that’s a lot – it may be, for some apps. At aleno we have more than 160k active publications in peaks, and some of publish more than one collection (hence would require more than one change stream).
  2. The updateDescription is not that simple and there are both disambiguatedPaths and truncatedArrays which require additional handling.
  3. (about the YMMV part) If you’ll have multiple change streams being affected by the same operation, then this one operation will be sent to your app multiple times. On the other hand, oplog will get only one.
    • In an extreme case, you could have a multiplier of a hundred or even more, leading to worse performance than oplog tailing.

(*) I hope I’ll be able to share what I’m working on soon… We’ll see :crossed_fingers:

7 Likes
24

Thanks for taking a look and sharing your thoughts! I was inspired the Change Streams performance data you posted previously and look forward to seeing what you have cooking.

  1. Technically this solution isn’t using a change stream per publication. It’s a change stream per unique match filter for a collection. So if your app has a lot of users subscribing with the same match filter it would reuse the same change stream. That being said, maybe that still won’t get the job done for some apps and if you’re scaling horizontally it won’t be as effective in reducing change stream creation (though currently scaling oplog horizontally won’t work as I understand it). Curious if you or anyone else has a better solution here.
  2. Thanks I’ll take a closer look at this.
  3. This is a good point. Welcome ideas on how to solve.
25

Yeah, I saw this deduplication and it’s very good that it’s there.

I have some “middleground” solution, but more on that later.

That’s basically the same problem as 1, but the other way round – the fewer change streams you have, the more processing the listeners have to do.