Independent Subscription Modes for Meteor Collections

core
1

Problem Statement

Meteor’s publications share data in a single Mini-Mongo collection on the client. While this can be convenient, it introduces complications under certain scenarios.

Scenarios for Improvement

There are two main use-cases where isolating publications of the same collection into different client-side instances would be beneficial:

1. Pagination Challenges

Currently, pagination is problematic for a collection when multiple subscriptions exist on that collection. To elaborate:

  • Suppose a client subscribes to 10 elements from a collection. The pagination works well as long as that’s the only active subscription.
  • If a second subscription fetches an additional, non-overlapping item from the same collection, the client’s Mini-Mongo instance will now have 11 items.
  • This discrepancy complicates the client’s pagination logic, as it is challenging to discern which 10 items belong to the original pagination query.

2. Non-Meteor App Integration via DDP

When integrating non-Meteor apps using DDP, the current implementation requires the use of Mini-Mongo (or its reimplementation) in the external app. This constraint stems from the fact that all subscriptions for a given collection merge their data into a single client-side Mini-Mongo collection. Having the ability to segregate subscriptions could simplify this integration by allowing the server to control precisely what each subscriber sees.

Proposed Solution

Introduce a new subscription mode in which each subscription to the same collection is isolated from others. This would give developers more control over how data is managed on the client, enabling easier pagination and improved integration with non-Meteor apps.

2

Some days ago I was working on grapher and realized that it uses a package to workaround on this issue partially. The package name is Subscripcion Scope and it does what it says in the name, it scopes docs to a specific subscription.

The package is very old and uses coffee-script, but it still works. Maybe an update to the package might be a good start for this.

3

I could see how this would be useful. As a workaround, I think you could use the reactive aggregate package

4

Some notes:

  • Your pagination query should have conditions on the client (we share them partially with the server) so they are the same code. Then you have not the trouble you run into.

  • If you use something like: publish composite your can use transform, it will allow you to add a flag field:

That way it’s easy to find your data from that publication. You can just search for the flag if you subscribe once. You could even supply the flag value from the client as a subscription param to make sure they don’t mix. (Make sure to give each flag a separate fieldname so when merged the same doc can be used for multiple groups of data).

That being said: It could be quite interesting in some cases, the example of grapher is a good one.

But the generic concept of minimongo saves you sending the same doc many times which also has it’s advantages. So like all engineering decisions it’s a trade-off.

This really depends on the needs from the external app. In most cases we see an API in terms of REST, GraphQL etc. to be more suitable for external apps unless it’s really for getting the live-data of course. Depends strongly on the target audience of the external apps and their knowledge. There are npm modules available for using full DDP in external apps (check the React Native one) if you want to go that route.

5

It shouldn’t be too hard to create a publication which returns subscription information - e.g., the IDs are the subscription IDs and the result is an array of document IDs. If you need more detail than that, e.g., not just the IDs but which fields are associated with each subscription, that might be a little harder.

6

What could the condition be? I mean, if I am on page 42 with a limit of 10, but there are 11 documents in mini-mongo, I think there is no way for the client to know which 10 come from the pagination query.

7

Assuming you used the transform option to add a table: true key to the collection, then it’d be as simple as

 this.handle = Meteor.subscribe('items.paginated', TableConfig.get('search'), {
      limit,
      offset,
      sort,
    }, filter);


    const data = Items.find({ ...filter, table: true }, {
      limit,
      sort,
    }).fetch() || [];

(Just pulled this out of a repo I have, didn’t test it)