Our current set up is 10 Meteor instances running with an nginx load balancer.
We use AWS spot instances to reduce costs but occasionally the instances are replaced which seems to cause issues:
The spot instance loses its elastic ip so it can no longer receive traffic. I’m not sure how to debug this but it seems like nginx still passes traffic to the instance. Do I have to have something set in nginx to do the health checks, and if the health check fails, to avoid sending traffic to this instance?
Anyone else using spot instances with Meteor? Is it a bad idea?
The problem with spot instances, as you have found, is that they get replaced with as little.as 2 minutes warning. And depending on the instance type you select, you might not even get another one when it goes down (they’re cheap because they’re excess capacity, when there is no excess, there are no spot instances)
Do you know if the instance id stays the same after an IP change? If not, look at an application load balancer and register targets by instance id. Either way, You shouldn’t need elastic ips as they are global public ips And if you’re internally load balancing you’d only need a private IP.
If the instance id doesn’t remain the same, you could always make the first step of your application startup be to register itself with the alb via the API. This is a little bit of a pain to setup but is doable. Your first step should be to check the instance id. You can also check for other services in AWS that might make this process easier. ECS or app runner may work for you though I’ve always had trouble getting app runner to work
A better alternative might be to pay for reserved instances. They can be as much as 60% cheaper and if you know you’re going to need the capacity (e.g., you’re not auto scaling) this can be a good option. If you pair that with a burstable instance type, like t3 you can get a kinda half auto scaling setup where you’re paying for lower capacity but can burst to higher as needed.
