[Best practice] Multiple publish/subscribe with the same collection?

Hi,

Sorry for the disturbance, I’m trying to understand what should be the best way of fixing my current issue in a personal project in terms of code quality & performance.

I actually use the Account Password to store users in my application, which work fine.
I also use meteor-astronomy to add some helpers to my user collection in order to ease the development.

Here is my user class (simplified to be digest enough) :

export const User = Class.create(
{
	name: "User",
	collection: Meteor.users,
	fields: {},
        helpers: {},
        behaviors: {}
};

The thing is, I’m trying to build an administration panel to manage the registered users in the form of a paginated table.
Paginated table and all work fine, but I encounter some issues with two publications.

First, I have an intercom available in every pages of my app which allow the current user to do multiple things (read notifications, messages, etc), which needs the current user information I publish with a first publication. Here is the related publication :

Meteor.publish("user_intercom", function()
{
	let user = User.findOne({_id: this.userId});

	if (Obj.isFalsy(user) || !user.isActionAllowed())
		this.error(Logger.subscriptionError(ErrorCode.PERMISSION_DENIED, {connection: this.connection, userId: this.userId}));

	return User.find(this.userId, {fields: UserPublisher.getAllFields(user)});
});

Then, I have the table of users in which I display registered users, which needs said users information I publish with a second publication.

In the first time, I only published the cursor from both collections, but encountered the fact that my client side collection had 50 +1 users per page (the 50 from the table and the one from the intercom, except when the intercom is supposed to be on the table for the page set [50 in this case]).
To solve this, I assumed I had to publish this list in a client side dedicated collection, which I then use to populate my table. Here is the publication for this behavior :

Meteor.publish("user_admin_list", function(limit, skip)
{
	let dskip = skip > 0 ? skip : 0;
	let dlimit = limit > 0 ? limit: 50;

	Mongo.Collection._publishCursor(User.find(
	{
		_id: {$ne: this.userId}
	},
	{
		limit: dlimit,
		skip: dskip,
		sort: {createdAt: 1}
	}), this, "users-admin-list");

	this.ready();
});

This works, but there is some issues with this as well.
First, it doesn’t allow me to use the Schema provided by Astronomy on this client-side collection.
Secondly, I’m a bit worried of introducing many client-side only collection just to solve this particular issue, which I may encounter a lot of time during further development.
A simple solution (even though I find it particularly ugly), would be to count users in the table before displaying and removing current user if there is more than 50 users in the set.

What is the best way of handling such use case ?
Thanks for your reading and response :slight_smile:

You may want to cross post this on the repository for the package you’re using if you haven’t already.

I did but this is not actually a problem with the package (even though some underling issues are being resolved in the package at the moment).
The problem would be the same if I used plain Mongo.Collection.

The real problem here is I published users from two sources and I’m trying to avoid being polluted by the first publication (even though I can’t remove it) when fetching my users client side.
The solution proposed in many post I’ve seen is about using a client side collection to get the result of the first publication and another for the second. This is what I did, and this works fine (except for the underling issues encountered due to the package).

But my question is : is it really the only way ?
Isn’t there a way of filtering published data from source ?
Isn’t there a better pattern to achieve my use case ?
What about performances issues related to this usage of client side collection ?

UP.

Does anyone know anything about this ?
Thanks for your help.

Did you have a look at this package: https://atmospherejs.com/percolate/find-from-publication?

I had the same issue and this package worked pretty well for me. I don’t know if it works well with Astronomy.

@vuhrmeister
I kind of manage to achieve what this package is proposing by extending my Schema to multiple Astronomy classes thanks to @jagi.
I now have multiple collections client side, and fetching in the right one when appropriate, solving the initial use case problem :slightly_smiling_face:

My main concern now is to be sure I’m not taking this problem through the wrong end, though.
Is my problem something people often/always encounter while working on large application with Meteor ?
Will it introduce performance issues in the future ?

I would like to hear from people with app in production and how this kind of issues are managed in there app, to know if my way of thinking with Meteor is as it should be :slight_smile:.
Is your app doing well in prod ? Have you ever encountered issues afterward ?

@axelvaindal You should have mentioned me in this discussion in the first place so I would help you solve your problem without doing hacks :).

Actually having 50 + 1 users from two publications is an entirely normal thing and you don’t have to use client only collections to deal with it. You should know one thing: client collection is a subset of some part of data of the server collection. It doesn’t necessarily mean that one collection = one publication. You can have data from several publications in one collection and you can still make things work properly.

What you have to do is to make another client only query. I assume that you’re probably using ReactiveTable package but if not, the same rule applies anyway. In this case, you can’t just provide data for the table in the following way: User.find(). Instead, you have to provide some selector that will limit the number of documents to 50: User.find(selector). Where selector will be the same (or similar) selector as the one in the publication. In your case:

User.find({
  _id: {
    $ne: Meteor.userId()
  }
});

@jagi this only works in such an expected scenario where you know what data is in the collection. If there is more data it gets harder to filter them out.

I had a scenario where I needed to publish all documents with only one or two properties. In a paginated table I then couldn’t figure out anymore what to display.

Is your app doing well in prod ? Have you ever encountered issues afterward ?

Yep, it’s doing well with the find-from-publication package. The advantage is that you have all documents in one collection. If you split it up into several collections then you might be sending more data over the wire than necessary.

Another way would be to add a property to the documents on publish and then filter for it on the client. E.g. you could add forList: true and then on the client User.find({ forList: true }).

First of all, thank you for your time helping me solve this issue, both of you :slightly_smiling_face:.

@jagi I’m aware I could publish those two publications in the same collection, this is actually what I am doing almost anywhere else in my app, but this cannot work in this particular scenario.

I still have an issue which is the 50 + 1 published users in my client side set.
I cannot restrict from getting everything but the actual connected user because, well, the actual connected user may be in the published set of 50 users.

Here are the possibilities :

  • Both publications, 50 + 1 users (the connected one and 50 others)
  • Both publications, 50 users (the connected one is actually part of the set in the 50).

If I were excluding the connected one every time, this would be an issue because it is supposed to be in the table at some point and excluding it would make me have 50 users almost everytime but 49 when the connected one is supposed to be available as well. Not mentioning I actually sort the users based on dynamic data and can’t predict at what pagination the connected user will be visible in the table.

Limiting to 50 won’t do as well because my users need to be sorted and thus the +1 could be in the first 50, thus excluding a user which should have been in the visible set.

A friend of fine just advised to check if there is more than 50 users in my fetch, if so, exclude the currently logged in one. This works, but I feel like I’m solving the problem with laziness and ugly code base and I don’t like it :smile:.

To overcome this difficulty, my solution was to publish the list in a temporary client side collection, in which I’m sure I will always have my published set in the correct order with the correct fields, thus my needed help for applying my schema to multiple collection with your package.

@vuhrmeister I originally wanted to find from publications as well, but wasn’t sure my Astronomy schema would be working with this, and moreover thought the MDG provided some way of solving this (common ?) issue without the help of an external package.
I’m actually controlling each published data in the client, so the amount of unneeded one in the client may be negligible, although in my actual code it is true my connected user is sent twice to the client.

Add a property on publish would work (and I didn’t even know it was possible), and solve this issue, but I think my schema won’t allow it and the field wouldn’t be available client side if not registered in Astronomy first, which is not a good idea because I would have to do this for each time I encounter this issue (which is also the case with the client side collection, this is why I’m looking for another way).

To resume :

  • Filtering the connected one is not possible due to the data not being predictable in terms of order
  • Filtering by limit is not possible due to sort-able nature of the data which may be exclude the wrong one
  • Multiple collection does the trick, but worries me about performance issue or being not the good way of solving this issue
  • Temporary added field may do the trick, but seems incompatible with meteor-astronomy schema if not previously registered (@jagi may confirm or correct me if I’m wrong). I’m curious however to know more about this, if you have a code sample or something :slightly_smiling_face:

Probably “transient” field should solve your problem but you have to check it out. You can read more about transient fields here http://jagi.github.io/meteor-astronomy/v2#transient-fields

@vuhrmeister I was developing many Meteor apps and never had a problem with not being able to select a subset of documents that were needed in the given situation but maybe I just haven’t dealt with such a situation.
However, I can agree that sometimes having in one collection data coming from the several publications might be hard to manage but not impossible. But that’s why we’re switching to GraphQL in our apps because it’s way more intuitive. So @axelvaindal depending on how big your project is I would recommend using Apollo for GraphQL and not worrying about pubs/subs.

@jagi Transient field won’t help with the current matter, because transient field are still registered field in the schema. If I could have a mean to get fields which were not previously defined in the Astro.Class, this would work. However, I seem to remember everytime I publish a doc from the database, the Classname.find() retrieves the document but omits fields it is not aware of (which is normal), so it won’t work.
More precisely, it will work in terms of code, but as well as with a normal field, and I would like to avoid adding fields for the sole purpose of solving an use case issue.

I need to have a well advanced version of my app in the next three months, and I’m actually developing this alone, so I don’t have time to learn how to use GraphQL and replace the current codebase. This is a good idea however, so I will add this to be evaluated for the after release refactoring, may be it will give me some perspective.

For now, I will stick with the minimal amount of client side collections and classes necessary to achieve what I need done, unless you, @vuhrmeister or anyone else has more information about what the best practice would be with such scenario :slightly_smiling_face:

I can’t think of more options right now. Both, find-from-publication and adding separate property to the published documents are a good way, imho.
Since I don’t know Astronomy, I can’t tell you more on how to integrate with it.

Well, let’s just stay with that then. Thank you both for insight :slightly_smiling_face: