Recipe: scalable high-availability setup with Meteor-up + Digital Ocean load balancer

#1

Hi,
Just sharing this here because I had quite some trouble setting this up.
Goal: use Digital Ocean droplets + load balancer, deploy with Meteor-up and get free Letsencrypt certificate.
Why? DO increased their 5USD droplets to 1GB, perfect for Meteor. And I like simple managed load balancers.

It turned out to be quite difficult because the order of steps really determines the success of getting Letsencrypt and your load balancer working. I have failed a lot of times so here’s the magic recipe:

Step 1:

  • Create a 1Gb droplet
  • Set proxy in mup.js for SSL, as per the instructions
  • Now set DNS of your domain to the IP of that droplet, make sure that the domain matches in mup.js
  • mup setup & mup deploy

Result

  • Should now have Letsencrypt and force ssl for the domain. Optionally check this with mup proxy logs-le

Step 2

  • Create a DO load balancer, USD20/month (but it’s managed, availability and easy to configure, imo better than nginx). Make sure it’s in the same region as your droplet!
  • Assign the droplet from step 1 to the load balancer
  • Set redirect https 443 to passthrough. This makes sure https traffic is redirected to your droplet, so encryption till your droplet, not till your load balancer. It also avoids needing to use DO’s DNS - I prefer Cloudflare.
  • Set health check to TCP port 443.
  • Now change the DNS of your domain to point to load balancer

Result

  • Load balancer should show droplet as healthy.
  • Your domain now points to load balancer and should redirect just fine, loading your app

Step 3

  • Now create a live snapshot of the droplet
  • Create a new droplet from the snapshot
  • Assign the droplet to the load balancer
    (Here you can make as many droplets as you like)
    (This snapshot step could potentially be skipped, I haven’t tried. But as Letsencrypt is already configured on the droplet of step 1, I feel this is a more safe approach)

Result

  • Load balancer should show all the droplets as healthy
  • Your domain now points to load balancer and should redirect just fine, loading your app

Step 4

  • Update mup.js to include the 2nd droplet
  • Now you’re ready to do mup deploy

Result

Your domain now points to load balancer and should redirect just fine, loading your app.

You now have a 40USD/month high-availability setup (hooray! :partying_face:) that is easily scalable as you can just clone droplets and add them to your load balancer.

Oh and did I mention this is about half the price of Galaxy? :muscle:

The only downside is that DO doesn’t support Ireland/Dublin as a region, so if you use mLab, that will decrease your performance a little as the closest is probably London. But then again, only AWS does Dublin (Galaxy is on AWS) so what can you do… For US your options are better.


Misc:

Check status for proxy ssl:
mup status

Letsencrypt logs
mup proxy logs-le

Clearing dns host cache in Chrome
chrome://net-internals/#dns

12 Likes
High Availability - Galaxy vs DIY?
#2

Thanks for sharing, this is awesome :slight_smile:

1 Like
#3

@satya couple of questions about load balancer. Does it support sticky sessions? If yes, do you know which algorithm is used to distribute clients across app nodes (ip_hash or something else)?

#4

@satya Do you know how to connect different regions? i.e. USA+EU?

Because without different zones it isn’t high-availability completely)

I put my few cents: You can save 15$(20$DO load balancer vs 5$DO droplet) using nginx instead of LB.

  1. create droplet 5$(Ubuntu), install nginx+letsencrypt:
    https://gist.github.com/afrokick/ff4890a34c71e96707f6d331e04e1eb6

  2. nginx config file(/etc/nginx/sites-enabled/default):
    https://gist.github.com/afrokick/0ce702aed854879bd231580e3463f64e

example.com - your domain
app1.example.com, app2.example.com - app’s domains. I’m using droplet’s private ip instead of domain.

  1. reload nginx and enjoy it)
#5

yes

No I haven’t checked but you can Google it or ask DO

#6

No it doesn’t support multiple regions. Cloudflare’s load balancer can do it

Your nginx setup encrypts traffic until your LB right? So from nginx till your droplets it’s unencrypted, I’m not sure how secure that is even when using private ips?

And if your LB is a single droplet, then it’s a single point of failure right? If it goes down, your app goes down.

#7

DO documentation https://www.digitalocean.com/docs/networking/private-networking/overview/#features

Yes, from nginx to app traffic not encrypted, but it is in the same private network. You can configure a firewall(it is free DO feature) to filter input/output traffic for your droplets, so only proxy can communicate with app servers.

Yes, it is single point of failure as well as have all infrastructure in one datacenter)

#8

DO’s load balancers are managed, don’t depend on a single instance. But yes, a single data centre. I personally feel it’s worth the additional 15$, but it’s a personal decision I guess.

But, DO’s advice is to not decrypt SSL on your load balancer, but do pass-through SSL- also for internal traffic.

I’m curious how do you deploy if your firewall only allows traffic from your LB?

#9

For those who are interested. Full description of LB features is here: https://www.digitalocean.com/docs/networking/load-balancers/overview/

It does support sticky sessions based on LB’s own cookies which it stripes out during routing.
And it supports round robin and least connections routing strategies.

Thanks @satya for sharing your experience.

2 Likes
#10

From Digital Ocean:

When balancing encrypted web traffic, there are two main configuration choices: SSL termination and SSL passthrough.

With SSL termination, SSL requests are decrypted at the Load Balancer and sent unencrypted to the backend. This places the slower and more CPU intensive work of decryption on the Load Balancer and simplifies the management of the SSL certificates. However, the traffic between the Load Balancer and the backend is unencrypted, which means it is visible to any neighbors on that network segment.

The other option is SSL passthrough, which sends SSL connections directly to the backend. Unlike SSL termination, the requests remain encrypted and the decryption load is distributed across the backend servers. However, the certificate information must be replicated on every server. In addition, you cannot add or modify the HTTP headers, so you may lose the client’s IP address, port, and other information contained in the X-forwarded-* headers.

We recommend SSL passthrough to secure traffic between the Load Balancers and backend Droplets.`

So what I read here is: don’t count on private networking for security!

#11

Hm,

As of July 18, 2018, DigitalOcean private networking isolates communication at the account or team level between Droplets located in the same datacenter region.

As I understood: only your(account or team) droplets can communicate with each other in private network.

Of course, you should enable firewall to prevent access from public. I allowed only ssh and proxy<->app communications.

But it is your decision to use SSL termination or passthrough.

#12

Ah cool, that’s new :slight_smile: Thanks for the info!

1 Like
#13

FYI: DO just decreased pricing of their Load Balancers from 20 to 10 USD :slight_smile:

1 Like
#14

Very nice write up of your setup, thank you very much.

I had a very similar setup about a year ago but ran into problems with the oplog features of Meteor. My servers had very high CPU usage and kept on crashing.

In the end I had to go with 1 server that had multiple cpu`s and high memory, after this the CPU usage was almost nothing and I had no further issues.

So my question…what did or are you doing to mitigate this? Do you have Oplog enabled?

#15

Yes, of course, no production without oplog.
But we’re using mLab so they can have fun with CPU issues :slight_smile: