Improve speed of pub/sub of ~5k items from Mongo collection

help
1

Hi there,

So I am currently working on a Meteor app which has a huge database of items behind it. These db will later on contain millions of items, which are then filtered by a supplier_id. (there are many suppliers where each one is selling on average 5000 items).

When the user chooses a supplier from a list, these 5000 items should then be published to the client to be used inside an autocomplete search field. Think of a simple list where the user can add more and more items he wants to order and everytime he is using this autocomplete search field.

I am using the iron:router and subscribing like so:

return Meteor.subscribe("mongoItems", this.params._id);

The publish function:

Meteor.publish("mongoItems", function (supplierid) {
	return MongoItems.find({ supplier_id: supplierid }, { item_ref: 1, item_name: 1, _id: 1 });
});

It all works, but the data of the subscription takes about 4-5 seconds on my MBP on localhost to reach the client which is not really acceptable to take that long until the user can start typing. I am using an index on the supplier_id field and when I use .explain()it shows that mongo is indeed using the index properly and the find() operation itself only takes 6ms! So the bottleneck is the data reaching the client. DOM Manipulation is not the problem as in my tests the collection is only pushed into a JS array.

I also tried using numtel:mysql awesome MySQL package and put this huge list of items into a MySQL DB. 1 million in total, this time about 1400 items matching my chosen supplier_id. --> result: blazing fast! The data reaches the client instantly!

So I guess my question is:
Why is the Mongo pub/sub so slow when transfering about 1/3 of the same data (1400 vs 5000 items) coming from an SQL db is instant?! and in both cases the queries themselves seem not to be the problem as they are in single digit milliseconds.
What am I doing wrong? I would love to use Mongo for my “items table” as well for a number of reasons but this whole delay of the pub/sub is a dealbreaker.
Can anyone help?

Thanks a bunch in advance … Meteor rocks!! :smiley:

cheers, P

2

Related: https://github.com/meteor/meteor/issues/5633

3

Thanks for that answer!
I am now using a method to get my items array and it really did speed up the whole thing to “SQL level”! awesome!! :slight_smile:

Can you maybe just help me with the best implementation?
Currently I am doing this:

server:

Meteor.methods({
	'getMongoItems': function(supplierid) {
		return MongoItems.find({ supplier_id: supplierid }, { item_ref: 1, item_name: 1, _id: 1 }).fetch();
	},
});

client/route_controller.js:

onBeforeAction: function () {
	try {
		allItems.length;
	} catch(err) {
		Meteor.call('getMongoItems', this.params._id, function(err, res) {
			allItems = res;
		});
	}
	this.next();
}

Is this the “correct way” to do it?
Thanks again!

EDIT:
One more thing: Just to be sure that I understand it correctly: I obviously do not have a published subscription now to my MongoItems collection but I still CAN do minimongo update()s to this collection to update a certain item in it and this will be pushed to the server. is this correct?!

4

For autocomplete, i do a custom subscription which is updated with the search terms entered.
Also you don’t want to run the search when you KNOW (less than three characters entered ?) that it will give you too many results.

5

well, why not do search server side?
there are few packages.
and it is much more mobile friendly etc to transfer only matching documents, not all for given supplier

6

Could you provide some of these packages you may use yourself? :slight_smile:

7

Could you provide some sample code on how to implement this custom subscription incl. updating it the proper way? Would help me a lot! Thanks! :smile:

8

easy-search, typeahead, autocomplete …
or just type search to atmosphere and check what u want :smiley:

9

Sure.
One simple example :

//on server
Meteor.publish(“foo_lookup”, function(fooLookup){
if (fooLookup && fooLookup.length>2){
return Foos.find({foo:{$regex:fooLookup $options:‘i’}});
}
}

//on client
Deps.autorun(function(){
var fooLookup = Session.get(“foo_lookup”);
Meteor.subscribe(“foo_lookup”, fooLookup);
});

1 Like
10

with some $limit for result count, you cant show many on client anyway.
and if changing query for every keyup method is called anyway.
with some auto pagination when u scroll down on client…

but yes, it start to be little tricky for newcommers :smiley:

thats why packages rocks

11

Thank you! :smile:
I think I will for this one!

12

Thank you, this helps a lot! :smile:
Especially the Deps.autorun() part is new for me.

13

It is Tracker.autorun now :smiley:

1 Like