Incomplete publication data

help
#1

Hi Everyone,
I have a problem that’s been bugging me for several weeks already.

In my app I have a collection called StateCollection and I publish it like this:

return StateCollection.find({
            mac: mac
});

The documents are rather big, but I use all the data, so I don’t have any fields section - I use all of the data. The mac filter is just to show data that relate to the current user (similar as if I used userId).

It used to work OK for a long time, but recently what started happening is that some parts of the document won’t arrive on the client. If I log the documents server side, they are OK, but on the client side, some of the “sub-documents” are missing. This may not be relevant, but specifically, the documents have a field called status, which is an object and it’s supposed to contain (among other stuff) another object called system. And that’s the problematic part. This system field is completely missing on the client side (although if I debug-print the document server side inside the publication function, it’s all OK).

Now I think this has something to do with collection merging which is potentially faulty. Here’s why:

At some point I added a different publication that’s intended for Admins to get some overview of the user data. This one is called lastUsersActivity It looks like this:

return StateCollection.find({}, {
            fields: {
                "mac": 1,
                "status.lastUpdated": 1,
                "lastDataUpdate": 1
            }
        });

I.e. it does no filtering (fetches all the users), but it only gets some minor portion of the documents (especially notice the status.lastUpdated field).

Now the problem I’ve described above only happens if the lastUsersActivity publication is used before state publication, i.e. if the admin first loads a page that uses lastUsersActivity and only then opens another page, that subscribes to state publication. In the opposite order there is no problem.

So, my guess is that somehow meteor thinks that user who subscribed to lastUsersActivity already has those data and does not propagate those to the client. That is however not true… If you look at the definition, only status.lastUpdated is sent not, the entire status object. And that may be the reason why status.system is missing.

Can anyone confirm if this could be a problem?

Also, I’ve read all the articles on subscription / publication stuff, like the docs or this one and watched all tutorials I could find but none of those mentions this kind of problem.

This is meteor 2.2. I would spend the effort of migrating (I’m rather afraid of what it might break) if I knew the problem was addressed, but I found on such mention in the changelog.

Would appreciate any help I can get…
Thanks!

#2

I think the explanation for this is in the Meteor.subscribe docs:

Currently, when multiple subscriptions publish the same document only the top level fields are compared during the merge. This means that if the documents include different sub-fields of the same top level field, not all of them will be available on the client. We hope to lift this restriction in a future release.

There was a feature request open to improve this: Support merging subfields in DDP (or separate multiple subscriptions instead of merging) · Issue #67 · meteor/meteor-feature-requests · GitHub

1 Like
#3

Hi zodern, thanks for the quick response. Looks like exactly the problem.

So, what are my options? That repo is read-only and it doesn’t look like the feature request would be implemented any time soon…

Do I have to subscribe to everything? That’s not good, would probably be a massive overhead. There are like 700 users, so publishing all of the data to the client doesn’t feel right…

//edit: For now I worked around it by moving that specific field I need to get to the top-level. Ugly as hell, but gets the job done. I am still interested in any progress on the feature request though…

#4

By subscribing to something, it means that you need two things (observe an update and pull that update) which is basically … one :slight_smile: - have the most updated state in your view.
This might work if your update rate per observed field is not too high (such as in gaming or stocks, IOT, etc): You create a field updatedAt that you update every time you change something in a document.
You only subscribe to that field and if it changes, you pull your data with a method. You can also keep tracking of what was updated. For instance:

{
   updatedAt : Date,
   updated: 'this.that.the_other_one'
}
/* the cost of writing this to DB is insignifiant when compared with the cost of publishing large objects.*/

If for instance you have 15 keys in a document and only 3-5 are getting updated, you only pull those with your method. In the client side you can merge objects very easy and efficient.
Another option is to redesign your DB schema so that you can subscribe to some data in another DB while more static data stays in the original DB.

What happens to you now it basically tells you: “Your design is inefficient”.