Recipe: scalable high-availability setup with Meteor-up + Digital Ocean load balancer

Hi,
Just sharing this here because I had quite some trouble setting this up.
Goal: use Digital Ocean droplets + load balancer, deploy with Meteor-up and get free Letsencrypt certificate.
Why? DO increased their 5USD droplets to 1GB, perfect for Meteor. And I like simple managed load balancers.

It turned out to be quite difficult because the order of steps really determines the success of getting Letsencrypt and your load balancer working. I have failed a lot of times so here’s the magic recipe:

Step 1:

  • Create a 1Gb droplet
  • Set proxy in mup.js for SSL, as per the instructions
  • Now set DNS of your domain to the IP of that droplet, make sure that the domain matches in mup.js
  • mup setup & mup deploy

Result

  • Should now have Letsencrypt and force ssl for the domain. Optionally check this with mup proxy logs-le

Step 2

  • Create a DO load balancer, USD20/month (but it’s managed, availability and easy to configure, imo better than nginx). Make sure it’s in the same region as your droplet!
  • Assign the droplet from step 1 to the load balancer
  • Set redirect https 443 to passthrough. This makes sure https traffic is redirected to your droplet, so encryption till your droplet, not till your load balancer. It also avoids needing to use DO’s DNS - I prefer Cloudflare.
  • Set health check to TCP port 443.
  • Now change the DNS of your domain to point to load balancer

Result

  • Load balancer should show droplet as healthy.
  • Your domain now points to load balancer and should redirect just fine, loading your app

Step 3

  • Now create a live snapshot of the droplet
  • Create a new droplet from the snapshot
  • Assign the droplet to the load balancer
    (Here you can make as many droplets as you like)
    (This snapshot step could potentially be skipped, I haven’t tried. But as Letsencrypt is already configured on the droplet of step 1, I feel this is a more safe approach)

Result

  • Load balancer should show all the droplets as healthy
  • Your domain now points to load balancer and should redirect just fine, loading your app

Step 4

  • Update mup.js to include the 2nd droplet
  • Now you’re ready to do mup deploy

Result

Your domain now points to load balancer and should redirect just fine, loading your app.

You now have a 40USD/month high-availability setup (hooray! :partying_face:) that is easily scalable as you can just clone droplets and add them to your load balancer.

Oh and did I mention this is about half the price of Galaxy? :muscle:

The only downside is that DO doesn’t support Ireland/Dublin as a region, so if you use mLab, that will decrease your performance a little as the closest is probably London. But then again, only AWS does Dublin (Galaxy is on AWS) so what can you do… For US your options are better.


Misc:

Check status for proxy ssl:
mup status

Letsencrypt logs
mup proxy logs-le

Clearing dns host cache in Chrome
chrome://net-internals/#dns

20 Likes

Thanks for sharing, this is awesome :slight_smile:

2 Likes

@satya couple of questions about load balancer. Does it support sticky sessions? If yes, do you know which algorithm is used to distribute clients across app nodes (ip_hash or something else)?

2 Likes

@satya Do you know how to connect different regions? i.e. USA+EU?

Because without different zones it isn’t high-availability completely)

I put my few cents: You can save 15$(20$DO load balancer vs 5$DO droplet) using nginx instead of LB.

  1. create droplet 5$(Ubuntu), install nginx+letsencrypt:
    https://gist.github.com/afrokick/ff4890a34c71e96707f6d331e04e1eb6

  2. nginx config file(/etc/nginx/sites-enabled/default):
    https://gist.github.com/afrokick/0ce702aed854879bd231580e3463f64e

example.com - your domain
app1.example.com, app2.example.com - app’s domains. I’m using droplet’s private ip instead of domain.

  1. reload nginx and enjoy it)
2 Likes

yes

No I haven’t checked but you can Google it or ask DO

1 Like

No it doesn’t support multiple regions. Cloudflare’s load balancer can do it

Your nginx setup encrypts traffic until your LB right? So from nginx till your droplets it’s unencrypted, I’m not sure how secure that is even when using private ips?

And if your LB is a single droplet, then it’s a single point of failure right? If it goes down, your app goes down.

DO documentation https://www.digitalocean.com/docs/networking/private-networking/overview/#features

Yes, from nginx to app traffic not encrypted, but it is in the same private network. You can configure a firewall(it is free DO feature) to filter input/output traffic for your droplets, so only proxy can communicate with app servers.

Yes, it is single point of failure as well as have all infrastructure in one datacenter)

1 Like

DO’s load balancers are managed, don’t depend on a single instance. But yes, a single data centre. I personally feel it’s worth the additional 15$, but it’s a personal decision I guess.

But, DO’s advice is to not decrypt SSL on your load balancer, but do pass-through SSL- also for internal traffic.

I’m curious how do you deploy if your firewall only allows traffic from your LB?

1 Like

For those who are interested. Full description of LB features is here: https://www.digitalocean.com/docs/networking/load-balancers/overview/

It does support sticky sessions based on LB’s own cookies which it stripes out during routing.
And it supports round robin and least connections routing strategies.

Thanks @satya for sharing your experience.

3 Likes

From Digital Ocean:

When balancing encrypted web traffic, there are two main configuration choices: SSL termination and SSL passthrough.

With SSL termination, SSL requests are decrypted at the Load Balancer and sent unencrypted to the backend. This places the slower and more CPU intensive work of decryption on the Load Balancer and simplifies the management of the SSL certificates. However, the traffic between the Load Balancer and the backend is unencrypted, which means it is visible to any neighbors on that network segment.

The other option is SSL passthrough, which sends SSL connections directly to the backend. Unlike SSL termination, the requests remain encrypted and the decryption load is distributed across the backend servers. However, the certificate information must be replicated on every server. In addition, you cannot add or modify the HTTP headers, so you may lose the client’s IP address, port, and other information contained in the X-forwarded-* headers.

We recommend SSL passthrough to secure traffic between the Load Balancers and backend Droplets.`

So what I read here is: don’t count on private networking for security!

1 Like

Hm,

As of July 18, 2018, DigitalOcean private networking isolates communication at the account or team level between Droplets located in the same datacenter region.

As I understood: only your(account or team) droplets can communicate with each other in private network.

Of course, you should enable firewall to prevent access from public. I allowed only ssh and proxy<->app communications.

But it is your decision to use SSL termination or passthrough.

Ah cool, that’s new :slight_smile: Thanks for the info!

2 Likes

FYI: DO just decreased pricing of their Load Balancers from 20 to 10 USD :slight_smile:

3 Likes

Very nice write up of your setup, thank you very much.

I had a very similar setup about a year ago but ran into problems with the oplog features of Meteor. My servers had very high CPU usage and kept on crashing.

In the end I had to go with 1 server that had multiple cpu`s and high memory, after this the CPU usage was almost nothing and I had no further issues.

So my question…what did or are you doing to mitigate this? Do you have Oplog enabled?

1 Like

Yes, of course, no production without oplog.
But we’re using mLab so they can have fun with CPU issues :slight_smile:

1 Like

Hey @afrokick,

if I want to use the droplet IP should I need to put it in place of example.com? I don’t have a domain yet and i’m not using ssl for now. I just want to change the default sites-enabled/default file to test my Meteor app. I guess that if I don’t have a domain name I need to use the default file, right?
I don’t understant how to config the nginx default file and the mup file. Can you please post your sample config files?

Just replace app1.example.com:3000 to droplet’s public/private ip. Check firewall before it to open port.

Hi I managed to setup correctly the load balancer on DO based on this great article but I’m having issues with the proxy_protocol and how I have to setup correctly nginx via mup so I can have the real IP forwarded on my droplets.
Anyone has faced this before and managed to do it?

Thanks

Sorry to resurrect an old thread, but I’d really like to follow this approach as I am using exactly the same stack (DO + Cloudflare).

The only thing making me doubt is the documentation page on DO’s website. It has a warning message in the Passthrough section saying it is not compatible with Sticky Sessions. Any feedback on that from anyone using this setup?

(How to Configure SSL Passthrough :: DigitalOcean Documentation)