Heavy processing design

help
1

I am looking for some suggestions on how I should tackle the following workflow from some more seasoned \ experienced developers.

I have a method that receives an array of organization ID’s. There can be several hundred or even thousands of ID’s in the array. I then need to find all the organizations based on the ID’s, and grab an AWS SNS ARN that is stored in each organization’s document. Once I have that ARNs, I need to send a Push Notification to each ARN of the organization.

These will be pretty compute intensive and I am trying to see what is the best approach to implement. I’ve looked at the following:

  1. Micro-Service, move this code to a separate Meteor instance
  • Problem - how would I make this a private API with Galaxy hosting? If I deploy using galaxy, its exposed to the world unnecessarily
  1. Use AWS Lambda function
  • Problem - would need to allow access to the DB from more IP’s\endpoints. It is currently locked to the Galaxy IP’s
  1. Bigger Galaxy servers, just keep things monolithic and increase the capacity of the Galaxy instances.
  • Problem - this may increase cost quite a bit

Looking for any feedback or suggestions on how you may have solved a similar issue.

Thanks.

2

Why is it a problem for you to grant access for more IPs to your MongoDB? It’s a purely administrative action. Yours is a clear cut case for a microservice architecture. If I were you, I would definitely go for AWS Lambda — though I would evaluate first if Google Functions aren’t better for any reason.

You can rule out (1), as adding just a single additional Meteor instance as a microservice container is per se not scalable: what do you do if that single instance is out of CPU / RAM due to an increased load as your business grows or the computation requirements get more demanding?

So if you think in this direction, you should conceptualize this solution in a fashion that allows for scalability of your microservice containers. But if you’re already on this path, why building it on your own, if you can have AWS Lambda or Google Functions?

(3) is what I would never do: with just scaling up you’d be living on borrowed time.

3

Thanks for your feedback @peterfkruger.

Regarding #1, this would be a new Meteor cluster hosted on Galaxy with autoscaling turned on. So it would be very scalable and not based on a single instance.

As to why AWS and not Google Functions? The App currently sits on Galaxy (AWS) and the database sits on Mongo Atlas (also AWS), so I figured it would be best for performance and costs to keep the Lambda function in AWS as well.

I will start looking into the AWS Lambda route.

Thanks.

4

In case anyone is in a similar situation, you may want to investigate AWS EventBridge if you are using Atlas:

Basically, this will allow a Lambda function to be called from Atlas whenever a document is inserted, updated or deleted. I can then do my heavy processing there.

1 Like