Best publish-subscribe pattern for large collections with complex search criteria?

help
1

Hi all,

This is probably a simple question to answer, so I hope I’ve phrased it clearly…

I have a large collection consisting of documents with a relatively complex schema. On the client, I need to be able to search this collection and return results for combinations of values (for instance, search for documents that match chosen values for ‘age’,‘gender’ and ‘location’).

Now, I obviously don’t want to publish the whole collection to the client. However, how can I only publish and subscribe to a subset if I need to be able to search through the entire server-side collection for all documents that match the given search criteria?

So for instance, if I only want to publish ten results matching my multiple search criteria at a time (and then increments of ten more via some or other pagination system), how would I write my publication and and subscribe functions? I’m imagining I would I do all the heavy lifting on the server by passing all the search variables to the server-side publication function and returning a limited set of results (ten), but then what happens when I request more results via pagination? Do I stop the current subscription (SubscriptionName.stop()) and create a new one with a skip variable every time I call the pagination function? That doesn’t seem right to me. Is there some sort of reactive variable I set that the server picks up and uses to automatically send more data?

If this sounds confusing, it’s probably because I’m confused! I can write some pseudo-code if that helps clarify my question. In short though, I’m looking for a pattern that works for returning collection data based on multiple search criteria in a way that’s compatible with pagination (i.e. get the first ten matches, then the next ten, then the next ten…)

Thanks!

2

Can’t you just perform the search during the subscription?

3

I propably know what you mean. Basically the answer is yes, just subscribe again based on a client-side reactive-var with the paging data. Meteor will try to reuse it but because the limit changes the query will be completely rerun again. This is just fine and happens a lot in many applications.

I try to solve stuff like that with a load more approach rather than using paging (lazy me). That even works if you have multiple subscriptions in parallel.

In my case I opened 2 subscriptions with different selectors limited by limit+1 which ends up on a maximum of limit*2+2.

On the client i count() and check if there is more than limit+0 objects. If there is, I show a loadMore button. Complex sorting is applied on the client.

But for Paging you would need a custom publication returning ids per page I guess. Have a look at the internals of https://github.com/aldeed/meteor-tabular

4

Great, thanks all - this is helpful :slight_smile: