Running a constant background task

Sorry on that, that was not clear to me from your answers.

Now we know it’s about Twitter messages it might be interesting to look at the way Kadira handles it’s incoming streams of data. You receive the incoming data on separate servers (with separate apps) which can be meteor but also native nodejs for example.

Then you merge/filter/whatever do to the data and then you store it somewhere. For example in Mongo. Meteor will just take the already prepared data and show it to your client.

If you don’t do that it will constantly restart if you deploy a new version. Also it will become a scaling issue (which is not premature in this case with Twitter). If you do it in Meteor itself (for example in a proof of concept) for sure look into fibers and the function unblock (http://docs.meteor.com/#/full/method_unblock). That will move code out of the loop so it continues quickly.

Since this is the top Google result for “meteor background jobs” and SyncedCron is deprecated, I thought I’d mention that the new Steve Jobs package is designed just for Meteor and supports this kind of use case (and others too)