How to manage a cluster of Meteor web scrapers services?


#1

I have a big monolithic application and about 50 different websites that need to be scraped periodically. I’ve yet to do anything with micro-services, and I think its seems reasonable to have 50 different Meteor apps (one for each website), that would boot up once a day, do its web scraping duty, connect to main server, call a bunch of Meteor.methods to write to the database, and then gracefully shutdown.

The question now is how should I go about doing this? I have pretty much ZERO dev ops experience so I’d appreciate anything you have to say. I think meteorhacks:cluster is a pretty obvious place to start, but I don’t know where to go from there…


#2

Do you really need separate meteor app for each of them? Cause synced cron can schedule them and I think also distribute over instances.
I would just write 1 scrapper app, 1 main function for every site and than schedule them in some cron package like synced cron


#3

So you’re saying just have 1 scrapper app, and leave it running idle for most the time with all the cron jobs schedules for different times? I suppose that works. That would be 49 less apps I’d have to worry about starting up and shutting down haha.


#4

Well, I just suggested 1 more possible way how to do it.
You know if it fits your usecase.