How can I design this composite publication better

help
1

Hi folks,

I’ve got a particularly absurd publication that sets off warning bells in my head.
Just how bad is it to have 3.5 levels of nested children in a single publication?
What alternatives are there?
Is it time to bite the bullet and use grapher or Apollo? (I’d rather not have to refactor the entire data layer)

publishComposite('renderData.byProject', function(projectId) {
    check(projectId, String);
    const userAllowed = ProjectSecurity.isEditableBy(projectId, this.userId)
    return [
        {
            find() {
                return Projects.find(userAllowed ? { _id: projectId } : null);
            },
        },
        {
            find() {
                // Check here as well to be sure
                return Pages.find(userAllowed ? { projectId } : null);
            },
            children: [
                {
                    find(page) {
                        return Modules.find({ pageId: page._id });
                    },
                    children: [
                        {
                            find(mod) {
                                // Stupid Mediacollection doesn't return real cursors
                                return MediaCollection.find(
                                    (mod.content && mod.content.media) || null
                                ).cursor;
                            },
                        },
                        {
                            find(mod) {
                                return Sets.find(
                                    (mod.content && mod.content.set) || null
                                );
                            },
                            children: [
                                {
                                    find(set) {
                                        return MediaCollection.find({
                                            _id: { $in: set.mediaFiles },
                                        }).cursor;
                                    },
                                },
                            ],
                        },
                    ],
                },
            ],
        },
    ];
});
1 Like
2

I would guess the main potential issue here would be performance? If that’s the case, it would make sense to try it out with some real world data before making any big changes. If the number of documents being returned is not very large, then perhaps just rolling with it is the best option.

1 Like
3

Have you given this a look yet?

I haven’t actually used the package but first chance I get I wanna give it a test run with the Socialize package set and see how well it performs.

1 Like
4

Yeah, it’s not a problem yet. It just set off my spidey senses that something was wrong here.
This is pre-production so don’t have any data yet

That looks great! Not sure if it supports the level of nesting in our data model but I’ll take a look

5

Does it have to be reactive? Otherwise use a method and assemble the data yourself…?!

const project = Projects.findOne(projectId)
const pages = Pages.find({ projectId })
const mods = Modules.find({ pageId: { $in: pages.map(page => page._id )} })
const sets = Sets.find({ _id: { $in: mods.map(mod => mod.content ? mod.content.set : null) })
// and so on
1 Like
6

your database server is going to explode for this nested publication, especially if it’s publish-composite and if it runs pollingObserver :laughing::laughing:

7

If you fork the publish composite package and change this function https://github.com/englue/meteor-publish-composite/blob/master/lib/publication.js#L92 to be:

    _republish() {
        if (EJSON.equals(this.cursor._cursorDescription, this._getCursor()._cursorDescription)) {
          return;
        }

        this._stopObservingCursor();

        this.publishedDocs.flagAllForRemoval();

        debugLog('Publication._republish', 'run .publish again');
        this.publish();

        debugLog('Publication._republish', 'unpublish docs from old cursor');
        this._removeFlaggedDocs();
    }

then it will be a loootttt more efficient. I should probably PR this onto the main publish composite repo.

3 Likes
8

It doesn’t look too bad to me, especially as the queries aren’t too complex, and some query by _id, which is the most efficient way to use Meteor’s pub/sub. If you want to optimize, I could suggest:

  • switching to methods, and using pub/sub where real-time is needed
  • filter out unnecessary fields
  • set limits if necessary

If you’re worried about publishComposite, or have real performance issues because of it, I suppose it would not be too hard to manage individual subscriptions using Tracker.

9

Thanks everyone!

Since being reactive is only a nice-to-have, I’m going to switch it to a method and publish a reactive flag to notify the client if content updates are available (and a flag to force the client to fetch new data).

That should give a nice balance between the two without massive memory usage

2 Likes
10

Purely for selfish reasons - this would not be a bad idea!

1 Like
11

Especially since other packages like grapher depends on publish-composite so even more ppl would benefit from it.

1 Like
12

Why not using mongo aggregate and lookups? aggregate/lookup is more efficient since it does the join at the DB (I think Grapher is making use of it) instead of fetching everyting and joining at the server.

If you want it to reactive, you can use reactive-aggregate package.

2 Likes
13

The benefit of grapher for me is that I can easily swap reactive (publication with publish-composite) and non-reactive (method) queries.
The reactive queries have another extra: When I have a list of items and I create/delete/edit one of them, I get the updated list “for free”, meaning I don’t have to deal with refetchQuery. If I use non-reactive queries I have to either refetch everything after every change or write a bunch of helper functions to update my “local cache”.

14

Curious, has anyone replaced publish-composite with reactive-aggregate? Would love to understand performance implications and know of any gotchas. :slight_smile: