I have run into some trouble with my meteor application.
Previously I had a few documents containing alot of data. In order to add some flexibility I have changed the data structure so that much of the data in the previous large documents have been put into a separate collection now containing many documents.
However, after doing this change my subscription cant even complete loading! The CPU of the server goes up to 100% and then the DDP connection resets and the subscription starts over again.
In order to verify that this was indeed a problem I created a test app. I then created two collections, one called LargeDocs and one called ManyDocs. The total size of both these collections should be approx the same. In ManyDocs I create 40 000 documents with some random data. And in LargeDocs I only create 40 docs but with size 1000 greater per document than ManyDocs.
I then compare then compare the subscription time for the two versions. I have also tested with different versions of Meteor and this is the results:
Total size large docs = 10197857 chars Total size many docs = 10240335 chars **1.6.1 - LargeDocs [ms]** 1905 1589 1544 1754 **1.6.1 - ManyDocs [ms]** 7565 7472 7504 7104 **1.8.1 - LargeDocs [ms]** 2737 ms 2802 2812 3030 **1.8.1 - ManyDocs [ms]** Fails to subscribe. The DDP connection resets and the subscription is restarted
So for meteor 1.6.1 the total subscription time is 4 times larger when moving data into another collection with many documents.
I know there is a performance issue with meteor 1.8.1 so that is nothing I want to address in this question. I just want to understand why there is a difference when subscribing to many documents vs large documents when the total data still is the same? Any ideas?