I’ve been trying to focus on optimization lately, since we’re planning on growing the users of our app soon and the top priority is scaling. I’d asked a similar question for handling importers, but having a little bit of a different problem now.
Our app basically needs to track any updates on products from various sources and put them in to a queue to execute in order. Some product lines do need some additional updates to run (which may make the function heavier at times). Many updates queued up at once during peak hours, and It’s important for us to work through that queue as fast as possible, in the order received.
I’m concerned that this is going to block the event loop too much.
I know one option is to split this part of the code in to a separate app as a recent Meteor article on background jobs stated. But in term of making the functionality itself most efficient and not blocking the event loop as much as possible, what would the best approach be?
Also we use Galaxy for hosting, so hoping to find a solution that would scale well by resizing/adding containers, if possible.