Adding indexing to minimongo

announce
1

Hi all,

For one of my projects I noticed that slow lookups in minimongo were a major factor in making the templates render slowly, so I created a package that adds indices to minimongo. I thought it might be of interest to some others as well, so I published it on atmosphere as helfer:minimongo-index. You can try it out by running meteor add helfer:minimongo-index in your project directory.

The package is still experimental at this stage, but in our app where we had a collection with 415 elements, we managed to speed up the find() function by a factor of 20, with corresponding effects on the rendering time.

More information is in the readme on the package’s github repo.

If you have large collections on the client and think minimongo might be slowing down your rendering time, give it a try and let me know how it works. If there’s enough interest, I can develop it into a fully fledged indexing package that also speeds up sorting (it doesn’t right now).

PS: If anyone knows of a better way to modify meteor internals other than redefining the functions with your own code, please let me know!

13 Likes
2

Sounds very interesting. Could you post some benchmark data or an example repo that we could run and fiddle with?

3

Great project - would be great to see it continued with full index support including sorting. I might even be able to help out a bit

4

Very cool! Keep up with it, I’d love to use this stuff in production applications.

5

Very cool, would love to see some benchmarks.

6

An alternative approach could be to ditch minimongo entirely, as it can’t handle a large amount of documents anyway (I tried putting in 10k documents and failed miserably, I just wrote about it here

Using lodash (or pure js), you can easily query such a set with no noticable lag.

2 Likes
7

How would you do it exactly?

8

Something like this:

userdata = UserCollection.find({}).fetch()

result = _.find(userdata , function(o) { return o.age < 40; });

This would first dump all user data to a regular js array of objects. Then using the lodash collection syntax you can query this data.

1 Like
9

The problem with the lodash approach that you described is that you now need to write all the tracker logic yourself or lose the reactivity. I actually considered doing just that in my project but then decided that putting a cache in front of what’s essentially already a cache of the same size does not make much sense. It’s better to add missing features to the current cache, and so I decided to add some basic indexing to minimongo.

1 Like
10

too bad it’s not ready for sorting, is there an alternative for minimongo beside loadash/underscore? thinking about using redux.

11

Is anyone interested in reviving this package?

2 Likes
12

I am interested. Though I would say it would be good to have this as native Meteor feature. Recently I have been experimenting with multiple servers and server-only minimongo as cache and figured out that the lack of indexes is actually a deal-breaker to not proceed with my original idea. While looking to the code, adding indexes to minimongo does not seem a too complicated task, no?

13

I don’t know. If you have specific implementation ideas maybe we can start discussing here or in a new issue here https://github.com/meteor/meteor-feature-requests

14

It is already there: https://github.com/meteor/meteor-feature-requests/issues/66

Would be great to have this.

1 Like
15

Agree it would be great to have support for this in core. Is anyone aware of alternate approaches? In my use case, I’m fetching data from an external API and putting it into a client side only collection (I’m not interested in storing the data permanently). I’d like to avoid inserting duplicate data which is where a minimongo index would be handy.