HTTP Requests time out with Production Compose Database only


#1

I’ve been working for two days on an issue where I can’t make HTTP requests from server code (local or production). I’ve narrowed this down to one situation — when I’m running my meteor app using my production database which is hosted with Compose.

Here’s my test process. I’m setting MONGO_URL to get my local Meteor installation to use a remote database. I change this for each test (or use a new Terminal tab to reset it). Then I’m starting a local meteor server with the exact same settings file every time using meteor --settings ../settings/development.json.

Then I’m opening a new tab and running meteor shell. Into the shell I paste the following code which makes a request to an API. Note that this server code does not call any Meteor libraries. It’s just using the Node request library — https://github.com/request/request

var request = require('request');
var myRequest = request({
  method: 'POST',
  url: 'https://api.nylas.com/send',
  timeout: 5000,
  auth: { user: 'REDACTED', pass: '' },
  json: {
    to: [{ name: "Tim Fletcher", email: "tim@timothyfletcher.com"}],
    body: "Request test",
    subject: "Request test"
  }
}).on('response', function(response){
  console.log(response.statusCode);
}).on('error', function(error) {
  console.log(`Got error: ${error.message}`);
});

I tried running this code in a variety of different ways. It only fails (ETIMEDOUT) when I’m running it using my production database.

This code works perfectly when I run it:

  • Within a normal node REPL i.e. not related to Meteor at all.
  • Within a meteor shell with local (built in) Mongo database.
  • Within a meteor shell remote (MLab) Mongo database (sandbox, no Oplog I think)
  • Within a meteor shell remote (staging/Compose) Mongo database (with Oplog)

It fails every time with a timeout when it’s run:

  • Within a meteor shell with local meteor / remote (production/Compose) Mongo database (with Oplog)

I’m struggling to see how the database instance can have any effect on an HTTP request, particularly when an identical database at the same host seems to work fine. Any ideas? I’m really lost on this one. What would you do to narrow this down further?


#2

Ouch - this looks really painful. A few questions:

  1. Which version of Meteor are you using?

  2. With compose.io, are you signed up with their “MongoDB Classic” or “MongoDB” plan?

  3. Can you share your production MONGO_URL / MONGO_OPLOG_URL settings (private details omitted of course)?

Since you’re seeing an outgoing connection timeout, I’m wondering if you’re running into oplog issues with the newer compose.io infrastructure (what they now call the “MongoDB” plan). Compose has talked about these issues a bit here (basically it comes down to Meteor’s mongo drivers being out of date - updates are coming soon though). Maybe you’re polling your DB at compose (instead of using the oplog) and you’re hitting a max number of concurrent outgoing requests limit with Node. You can try adjusting the allowed concurrent outgoing requests by tweaking Nodes HTTP library https.globalAgent.maxSockets setting (which is 5 by default). Definitely a long shot though …


#3

Hey Hugh — Thanks for chiming in.

Which version of Meteor are you using?

1.3.2.4

With compose.io, are you signed up with their “MongoDB Classic” or “MongoDB” plan?

I’m 99% sure it’s the Classic plan. The $18/month for 2 nodes one. The production and staging databases are each a single node under the same deployment.

Can you share your production MONGO_URL / MONGO_OPLOG_URL settings (private details omitted of course)?

Replica Set URI from Compose — MONGO_URL

mongodb://<user>:<password>@candidate.17.mongolayer.com:11147,candidate.54.mongolayer.com:10927/production?replicaSet=set-56956bccd17b2decf90008b5

The staging one (that works fine) is identical to this except the database name is ‘staging’ instead of ‘production’.

I’ve not been setting MONGO_OPLOG_URL while running my tests but this is what I have in my production settings for deploying to Galaxy.

mongodb://<user>:<password>@candidate.17.mongolayer.com:11147,candidate.54.mongolayer.com:10927/local?authSource=production

I’ll try your other recommendation and creating a new database/migrating my data / turning off oplog access. Thanks again.


#4

I moved the entire production database to mLab and ran my local installation with the remote mLab. Same thing — I can’t make the HTTP request. I can only think it’s something in the database’s data now…


#5

I’ve worked it out!! It is nothing to do with the DB provider and all to do with the data.

My app uses Nylas to sync customer email accounts so I can detect replies to emails that customers send via my app. For each customer that connects their email account, I open a stream to Nylas so they can send me delta updates on what’s going on with that account e.g. new draft created / email sent / reply received. It seems that I can only have 5 streams open at once. Any more and I can’t make HTTP requests from the server, or at least until one of the streams disconnects. This explains why only my production server is having the issue because the dev and staging servers are syncing < 5 account.

So the question is — How can I get around this? I assume this is a hard limitation of Galaxy so maybe I need to roll my own servers. Ugh.


HTTP request times out with Galaxy
#6

Glad to hear you found it @timfletcher! I bet the max of 5 connections is caused by Node 10.x’s HTTP maxSockets default size of 5 connections. I’m not sure how you’re connecting to Nylas, but here’s a quick example of how you can adjust the maxSockets setting when making a request:

...
const http = require('http');
http.globalAgent.maxSockets = 50;
const request = require('request');  // request uses http under the hood
...

#7

Ha. Yes, I just came here to tell you that I used this exact code into my app and could make requests even with 5 accounts syncing. Phew. Thanks for your help on this!!! It’s been a rough first half to the week. Haha.

I also ready on SO that newer versions of node won’t have this limitation.