Clarification on Oplog Subscription Execution with Specified Fields

bgramaje · October 23, 2024, 11:07am

Hi Meteor Team,

I have a question regarding how Meteor handles oplog subscriptions for constantly changing data from large collections. Specifically, does Meteor automatically know which subscriptions to execute from the oplog if specific fields are specified at the time of the subscription? If I have too many elements, does it know which element to target to notify the subscription, or does it need to execute every single subscription for each entity in the collection?

Thanks for your help!

radekmie · October 28, 2024, 10:07am

It does not “know” it, it has to be checked. Every oplog entry is checked with every observer (think observer = cursor, but it can be deduplicated). The result may be…

…“definitely yes”, e.g., { $inc: { a: 1 } } with projection: { a: 1, b: 1 }.
…“definitely no”, e.g., { $inc: { c: 1 } } with projection: { a: 1, b: 1 }.
…“maybe”, e.g., $inc with $sort and $limit, as the document may change the result (this is slightly more complex, but you get the idea). This one results in a refetch of the cursor (not always, but most of the time).

I’m not sure what do you refer to as “element” here, but once the above detection is done, the oplog entry is “applied”, resulting in zero or more DDP messages sent to the client. It’s done in a smart way, optimized for the lowest possible number of messages (at a cost of server processing time).

Why are you asking exactly? Are you facing some specific problem?

rjdavid · October 29, 2024, 7:00am

Per field reactivity is possible with redis-oplog

github.com

cult-of-coders/redis-oplog/blob/master/docs/finetuning.md

## Fine-Tuning

Now that we have a generic idea of how this works, we can make some fine-tuning that will **dramatically** increase the speed of your highly loaded app.

### Custom Channels

A very common scenario where this can be used is a chat application. Usually when we create a chat app, we have a parent "Thread",
and each message will have threadId pointing to our thread.

```js
Meteor.publish('messages_by_thread', function (threadId) {
    // perform additional security checks here

    return Messages.find(selector, {
        channel: 'threads::' + threadId + '::messages' // you can use any conventions that you like for naming them, it's not relevant
        // any name you choose just make sure it cannot conflict with a collection (threads), or a direct listening (threads::threadId)
    });
})
```

This file has been truncated. show original

We have an extreme case of a per key of a document, which we make reactive if that document is updated and all subscribed clients are updated.