Rethinking Meteor – managing subscription state

core
1

If Meteor were built from scratch today, would it still opt to keep all client subscription state on the server and use merge box to process diffs?

When Meteor was created, relying on stateful servers for subscriptions was possibly the only solution that made sense, but this decision came with two serious tradeoffs:

  1. Limitations on scaling.

There are two big factors that place limits on scaling Meteor — oplog tailing and merge box. Oplog tailing should be replaced with change streams, which hopefully will eliminate the current restriction on scaling horizontally but they also present their own issues to solve. Merge box eats up resources storing all subscription state on the server and diffing data.

  1. Modeling data is at odds with Mongo.

Sounds ridiculous right? Meteor was made for Mongo. But there’s a big caveat: Mongo encourages embedding data when possible and discourages joins for the most part (though they have definitely made strides here) whereas Meteor encourages normalization due to its top-level-only diffing which necessitates joining. I believe deep diffing was ruled out due for being even more resource intensive than it already is.

Resolving these issues are key to solving Meteor’s notorious scalability issues. As I mentioned in another forum thread, if Meteor had a better realtime scaling story, it would be much more sought after, especially with the current trend of syncing over fetching and collaborative, local-first apps. Meteor is poised to be a go-to solution here when an authoritative server is needed (which is basically all B2B apps and non personal-use apps).

Could Meteor subscriptions benefit from a paradigm shift?

Things are much different today with clients that are crazy powerful relative to 13 years ago.

What if state and diffing was moved to the client and the server kept a limited amount of state? The different publication strategies opened the door somewhat to this line of thinking. What if we took it to an extreme?

The upsides of this shift:

  1. Distribute more work to today’s powerful clients for true edge computing
  2. Less resource usage on the server (RAM and CPU)
  3. Reduction in costs for your business
  4. Scale horizontally
  5. Potentially open the door for deep diffing where Meteor could align itself more with Mongo in its encouragement of embedding and denormalization (where it makes sense of course)

The potential downsides:

  1. Increased bandwidth
  2. Increased latency (maybe)
  3. If you’re serving users with underpowered devices, this might be problematic.
  4. ???

But if we take a local-first approach with optimistic UI to an extreme then there is zero latency. Bandwidth might also be a negligible issue if you’re careful with your projections and caching your subscriptions.

Now, you might be saying: “we already do this, we just use methods and avoid pub sub like the plague” but when you do this, you give up one of the things that makes Meteor great — syncing data rather than fetching — and you introduce more complexity for you to build and maintain. You’ll forego Minimongo and likely be re-inventing the wheel with your own client-side stores.

This was one of the impetuses for jam:pub-sub – keep the things that are great about syncing data into Minimongo and eliminate the drawbacks of traditional pub sub. To that end, it enables subscription caching and provides an easy way to use mongo change streams, meaning that oplog tailing could be disabled. In the most recent version of the package, you can also significantly reduce state on the server with the serverState config. You can essentially opt-in to what I’m describing above, today. I believe it could be a huge improvement for many Meteor apps and its functionality would be a great addition to Meteor core. For really large-scale realtime apps, maybe a solution involving Redis / Valkey or a new service that sits in-between your app and mongo, similar to ElectricSQL, will be required.

Curious to hear what people’s thoughts are in 2025. Have you experimented with something along these lines? What advantages and challenges have you experienced? What other ideas do you have to solve Meteor’s scalability concerns?

3 Likes
2

I was about to ask that this should be made into code or contributed to the core instead of it being mere thoughts then I scrolled down to serverState :sweat_smile:

EDIT: I think folks would be more inclined to try out your package if you can pair with with metrics illustrating the performance gains you were able to achieve. Maybe akin to GitHub - meteor/performance

2 Likes
3

For sure. To be clear though, this thread is meant to discuss ideas to help Meteor solve some fundamental limitations and reach its full potential not to convince people to try the package. :slightly_smiling_face:

It would be great to have help from interested community or core team members to put the package through different scenarios.

4

I think this is an area of Meteor where it should be optional. The developer should be able to toggle managing subscription state on the server or the client.

Considering Meteor is a full-stack framework, the “common” state logic is all JS, agnostic to if it runs on the client or the server.

Mobile apps would benefit from having subscription state stored on the server, where more traditional desktop/business applications could utilize client performance.

2 Likes
5

I still haven’t gotten to trying out the .once publications (though I am very much planning to shortly), if I understand though that’s almost having the best of both worlds too, as long as having a peer trigger an update isn’t necessary, right?

The one thing I’m guessing (though I still might have a misunderstanding on my end) is that for .once publications you have to resubscribe when you expect a change to be ready to pull, which has sone similarities with TanStack Query’s mutate feature, or (more passively) passing a query a key that includes reactive state (eg Vue refs).

Meteor’s already pretty groovy with reactivity; maybe it’s trivial to mimic that TanStack Query behaviour in Metoer with jam:pub-sub just by finding a nice place to hook a resubscribe in?

I have to learn more about mergebox and change streams to appreciate the rest; for now I would blindly take what you recommend for granted, test it in practice then note if the behaviour makes sense per my intuition :sweat_smile: