Meteor app production bundle slower than dev-environment?

help
1

I have observed a problem in our production environment of our application. I have noticed that on some server methods the production bundle runs up to 10 times slower than in dev-environment! What I mean by slow is for example one server method I have which does alot of CPU-instensive work aswell as alot of I/O so I should state that it is not a client-side issue here.

Before I realized that it was actually the production bundle which was the issue, I thought that it was hardware issues on the production instance but eventually I tested deploying the production bundle on the same computer as which I develop and I found that it was slow there aswell.

I have made some tic-toc timings to the see what is going slow and for example simple find-queries by id take alot more time on the production-bundle. NOTE: I have not had time yet to analyze all parts of the code so I am not sure whether is only is database-issues or if all code is running slower.

Anyone else experienced this issue or has any idea what the problem could be?

Info: I am running Meteor 1.5.1.
Node versions tested: 4.2.6, 4.8.2

2

Sounds like it could be an issue with Mongo?
Have you looked into Mongo performance on the production machine?

3

I have made further analysis now.

It seems that the issue is related to the mongo db somehow but I dont understand how.

So the place where I see the issue is a Meteor method which makes a batch job, i.e. computing aswell as alot of database reads and writes. I have timed some operations to find the problem.

Consider this simple query:

    let start = new Date();
    const items = MyCol.find({
      _id: {
        $in: arrayOfIds
      }
    }, {
      sort: ["field_1"],
      fields: {
        field_1: 1,
        field_2: 1,
        field_3: 1,
        field_4: 1,
        ...
      }
    }).fetch();
    let end = new Date() - start;
    Log.debug(`Query took=${end} ms`)

(Note: this is just one of the querys that seems to run slow)

Here we fetch items based on ID. This should basically be instant, ~1ms, which it is when I run dev mode. But If I bundle the app and run it on the same machine as which I develop this simple query takes 30-60 ms!

So I then tested to run the exact same query in the mongo shell and it takes 0 ms. That says that there is no issue with the query per se.

I then created a simple Meteor method which only executed this query alone to find if there was some special problem with Meteor and this database fetch. But this also took around 1 ms on the bundled version. This shows that the there is no general issue with a bundled app.

So this says to me that there is some problem when running multiple I/O in a batch job. But this is not an issue when running in development. And bundled app for production should not run slower than dev right? Atleast not 4-10 times slower.

I then tried to rule out possible issues with node version and mongo version. I even tried to run the bundled instance with the node version that comes packed with Meteor and same for Mongo. Issue remains.

I have also tested upgrading from Meteor 1.5 -> 1.6 but the issue remains.

NOTE: All tests has been made on the same machine as I develop. So the issue is not hardware related.

I can not be the only one that has experienced this issue?

4

Have you checked the MongoDB query analyzer and profiler on your production database?

You may have indexes in development that you have not added in production. For example, do you have an index on field_1? Any field used for sorting should have an index.

5

Thats not it. I restore my production database in dev so indexes are the same.

Also many of the queries I have tested only have one element in arrayOfIds.

It should return in O(1)

6

There is something in the batch job which slows down all database queries. I tested commenting out the line which updates the collection:

MyCol.update(item._id, updateDoc);

After commenting out that all queries run faster. What could be the reason for this? I mean in meteor the queries run synchronous so after the update it should be done, shouldnt it? But since many queries run slow after this it seems like mongo is still busy doing something in the background or node is. Any ideas?

What is the difference rellly between a production bundle and running meteor run

NOTE: I start the mongodb in the exact same way as meteor run does, i.e

~/.meteor/packages/meteor-tool/.1.5.2.3iob8k.uipt8++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/mongodb/bin/mongod --bind_ip 127.0.0.1 --port 3001 --dbpath ~/workspace/myApp/app/.meteor/local/db --oplogSize 8 --replSet meteor --nojournal