I am writing a meteor app where it will fill up forms in multiple local websites using x-ray package. The idea here is that user will do it once and the app do the ‘heavy lifting’ multiple times. Anyway given the connection to those sites can be slow, what will be the best model to do this job asynchronously? I am expecting hundreds to thousand concurrent use so I will have to think about that too.
I am thinking something in the line of message queue and workers on other thread or even a separate microservice in python/go communicated over REST api but I am not sure if this is the best way and looking for suggestions. Thanks in advance.