I have a big monolithic application and about 50 different websites that need to be scraped periodically. I’ve yet to do anything with micro-services, and I think its seems reasonable to have 50 different Meteor apps (one for each website), that would boot up once a day, do its web scraping duty, connect to main server, call a bunch of Meteor.methods to write to the database, and then gracefully shutdown.
The question now is how should I go about doing this? I have pretty much ZERO dev ops experience so I’d appreciate anything you have to say. I think meteorhacks:cluster is a pretty obvious place to start, but I don’t know where to go from there…
Do you really need separate meteor app for each of them? Cause synced cron can schedule them and I think also distribute over instances.
I would just write 1 scrapper app, 1 main function for every site and than schedule them in some cron package like synced cron
So you’re saying just have 1 scrapper app, and leave it running idle for most the time with all the cron jobs schedules for different times? I suppose that works. That would be 49 less apps I’d have to worry about starting up and shutting down haha.