Copy all this. But this line “At this point, CloudFront goes to the origin…” The origin is the ALB (application load balancer), not an individual container (whether old or new).
So my above argument is that during a deploy, if a new container is available, then any requests (including from CloudFront) would only go to new containers because the ALB would route them to new containers if available.
Origin is your domain not your ALB. Your ALB does not contain any Meteor bundle files.
https://docs.aws.amazon.com/cloudfront/latest/APIReference/API_Origin.html
Did you read the docs you linked?
An origin is the location where content is stored. Use
CustomOriginConfigto specify An Elastic Load Balancing load balancer
What I meant is that from CloudFront’s point of view, the origin is https://app.someapp.com which goes to the ALB, not any set Meteor server container (whether old or new). This is important. Because the ALB makes sure CloudFront doesn’t go to an old container when a deployment is in progress. It would only go to a new container and thus have all the new assets (e.g. JS bundle, CSS, /public files). That’s my whole point.
I think we’re talking in circles as this point as I still don’t see how CloudFront would request old assets given everything I’ve stated. And our new ECS-based system is working good in test so far. So… I’m fine to drop this.
You sound like you got offended or something. You are free to drop anything you want to drop whenever you want it.
We, except you apparently, were not talking in circles nor trying to convince of something, just trying to help. Your origin is your domain it will pass through ALB but still your origin is your domain not the ALB.
Your balancer will try to … balance. If your new server is loaded, your ALB will not keep sending connections to it because is … new. Will continue to balance the load.
What others try to tell you is that there are situations/chances where you might not have the bundle files available.
Good luck with your project and I am glad you dropped it.
Dude you know you can just use nginx right? It’s free. Most of the net’s run on it since about two decades now. Come on get with the program already!
This is what I’ve been asking. So the ALB wouldn’t necessarily send new connections to new containers just because a deploy is happening. If the new containers are loaded, then a new connection, like from a CDN requesting a JS bundle, could then go to an older container.
MOSTYL RESOLVED: So, it turns out as soon as we used a CDN to serve code and css, this problem has mostly been resolved.
We still slightly over-provision and start multiple containers, but now we start 4 containers at a time instead of 11. We do this b/c 1 new container coming online still pukes as it ramps from 0-1000 connections in a very short time being forced to swallow every single new connection to the app.
Since adding the CDN, each container can handle significantly more connections. It turns out Galaxy was just really, really bad at serving up app assets. Now they don’t have to.
Special shot out to @filipenevola at zCloud for asking why we weren’t using a CDN yet. His team is working hard and smart.
I think any elastic compute cloud would be struggling to deliver assets not only Galaxy, but …
Galaxy for the premium price they charge over AWS should provide you with real data for every business client. I don’t know how they build their things over AWS but if they use shared application load balancers, you would be sharing network resources with other clients.
My understanding is that your problem was limited understanding and top and detailed view of your infrastructure. AWS offers these type of monitoring graphs:
At level of individual EC-2 virtual machine:
At level of Beanstalk (all EC-2 running in the beanstalk):
At ALB level (application load balancer):
I think Galaxy needs to start providing infrastructure intelligence data if they want to stay in the game or allow connectors behind their interface so one can take monitoring data directly from AWS. “10%” doesn’t say much, or “OK” in green color or other minimal info.
