I was wondering which is the best approach to inserting data into the mongodb used by a meteor application. The use case is this:
I will crawl some websites for information including a rss feed about once or twice per hour. If I have new articcles (which I need to check in the Meteor mongodb) I will add these.
My alternatives are:
Setting up some REST-like method inside the Meteor app which handles a new content (checking if it is really new and adding it to database).
Having only the NodeJs apps fully outside meteor except access to the meteor database, inserting new content (and checking for old) every time it is run.
Having some cron-like job inside Meteor which does the fetching of content/rss and inserts into Meteor mongodb.
Which approach would you use and why? Is it hard to connect to the mongodb instance from external nodejs apps?
Take a look at the percolate:synced-cron package. You could create a cronjob using this package that crawls for your external data, then stores it in Mongo. Since it’s all contained within Meteor, you’ll be able to leverage your existing Collection functionality (making it easy to check if similar data already exists, add new data, etc.).
That looks really interesting. Will it interfere much with the “website” part of the meteor application. Sure my jobs are not that resource/cpu intesive (download a webpage and extract a json document and insert into db).
It really depends on your application structure, load, server infrastructure, etc. SyncedCron jobs run on the server, so they won’t directly interfere with any of your client side code. The Meteor server component (Node app) is usually the lifeblood of your application though, so if you run regular intensive cron jobs they will end up slowing down your server’s ability to respond to client requests in a timely manner. If that happens, you’re probably better off splitting your scheduled jobs out into a separate service/application, that talks to the same Mongo DB instance(s) as your Meteor app. I recommend starting by keeping things simple though, and use SyncedCron within your app. You can always split the cron jobs out to run on their own later (if needed), scale up/out your hardware, etc.