Advice on performance of potentially large collections

performance
1

My background is front-end and I’m currently working on my first Meteor project. Im loving how quickly I can build out the back-end I want to make sure I’m following best practice around a few things.

Im making a social network with groups. Each group can have events. Users can join groups and also join events.

I have separate collections for users, groups and events.

A user has fields for groups and events they are attending. Ive manually set the group IDs as they are predefined, but users can create events so Im using Meteor’s _id field for these:

{
  name: 'James Smith',
  group: [1,2,4],
  events: [‘nca45R4um67RQDBSG’, ’GKs3bFv9BipJkyxr3’]
}

Events have a field showing which group they are part of:

{
  name: ‘some event’,
  location: ‘somewhere’,
  group: 2
}

On a group page I need to show all the upcoming events from that group and all the users who have joined the group. Im loading all of the events and users into the React component and then filtering by the group field to determine which ones to show on the page.

On the event page I also need to show all the users who are going. So again I load all of the users into the React component, but this time filter by the events field.

This is working fine for me as I develop locally but is this likely to scale?

I also need to allow users to comment on both groups and events. If I have a new collection for comments this could get very large over time. Is there a rule of thumb around performance for database sizes / organisation?

Thanks!

2

There’s no silver bullet for performance. But the main thing would be to set your secondary indexes up for your queries, enable MongoDB oplog and you’ll probably need to shard later on too. These are pretty much the basics for scaling MongoDB, but you could always go further of course. However, that would require a bit more thought on your end and how you do that is up for debate / down to your use-case.

1 Like
3

Thanks for that, Ill do some reading up on those.

In the short term (and given I don’t yet know how popular my service will be) does my approach seem broadly OK?

4

Yeah your approach seems fine to me. It’s how I would do it anyway :slight_smile:

5

Unfortunately I wouldn’t advise storing your data this way, especially if your arrays could grow quite a bit. You might take a look at the way the packages in my Socialize package set approach modeling their data.

6

@jamesmad I would actually go with @copleykj’s way of structuring your data. Looking at his packages, I agree with him! In fact it looks like his packages would work for you out of the box :slight_smile:

And @copleykj good job man, those look great! (I’d honestly not come across your socialize packages before)

1 Like
7

Thanks! I think it’s pretty common for people to not have heard of them. Maybe after the next release I need to spend some time on marketing :blush:.

Sadly I don’t think they would be specifically useful in this case, as there currently aren’t packages for groups or event listings. These are planned, but have taken a back seat to a large upgrade to the set. Due to this upgrade, if anyone were looking to start a new project with the packages, I would advise waiting till the release due to the amount of changes you will encounter.

1 Like
8

@copleykj Im sure its my lack of experience but Im struggling to get my head round what Im looking at in your packages. Do you have a a blog, any medium articles etc?

No worries if not. I have some other Meteor tutorials I’m working through so maybe this will be clearer to me after I complete them.

9

I don’t currently have any posts or articles. The basic concept is to normalize data that needs to be normalized and store it relationally. This tends to run counter to the Mongo mantra of embedded documents, but embedded data like this can cause some not so nice issues, such as not being able to properly paginate your data or having to sending huge documents over DDP.

For your use case I would probably use a collection for groups, group members, events, and attendees, with group members and attendees storing relational data.

As far as comments and performance of large collections… Mongo (Humongous) was actually created to handle very large sets of documents, so size of the collection should not be an issue as long as you index properly.

Implementation may be a bit daunting, especially when you need to implement comments and such across multiple other data sets. (Groups, Events, Photos, Posts, etc…). This is the very reason I started the Socialize packages.

Since you need comments on groups AND events, you could create a Group class and Event class, which extend the Commentable class and both will inherit commenting functionality.

Using the upcoming API this would look like this…

import { Meteor } from 'meteor/meteor';
import { Mongo } from 'meteor/mongo';
import { CommentableModel } from 'meteor/socialize-commentable';
import { LinkParent, LinkableModel } from 'meteor/socialize-linkable';
import SimpleSchema from 'simpl-schema';

const GroupsCollection = new Mongo.Collection("groups");

const GroupSchema = new SimpleSchema({
    "userId":{
        type:String,
        regEx:SimpleSchema.RegEx.Id,
        autoValue:function () {
            if(this.isInsert){
                return this.userId;
            }
        },
        denyUpdate:true
    },
    "createdAt":{
        type:Date,
        autoValue:function() {
            if(this.isInsert){
                return new Date();
            }
        },
        denyUpdate:true
    },
    "groupName":{
        type:String,
    }
});

class Group extends CommentableModel(LinkParent) {
    //Add any instance methods here
    owner() {
        return Meteor.users.findOne({ _id: this.userId });
    }
}

//Attach the collection to the class so we can use BaseModel's CRUD methods
Group.attachCollection(GroupsCollection);

//Register the Group class as a potential Parent of a LinkableModel
LinkableModel.registerParentModel(Group);

Now you can use your group class…

const awesomeGroup = new Group({ groupName: 'Functional Programming' }).save();

awesomeGroup.addComment("Welcome to my awesome functional programming group!!!");

const anotherGroup = GroupsCollection.findOne();

anotherGroup.comments().fetch()
4 Likes
10

This is great @copleykj. And your explanation is fantastic.

Out of curiosity, what did you make the socialize packages for? Is it for a personal project? There’s clearly been a lot of work put into the packages :+1:

1 Like
11

Originally the friendships and messaging packages started out as parts of contracted app that never came to fruition and just sat unfinished. At some point I came across a post, not unlike this one, asking about the best way to model friendships in an app. That made me realize that I had code just sitting around that might be useful to people. Now, about 3 years later there are 15 packages, and a few more that I have planned (groups, event listings, photos/albums), time and funding permitting.

3 Likes