Question: How to poll a 3rd party API to store in a collection periodically

help
1

I am looking to build an aggregator that pulls in multiple 3rd party APIs on a periodic basis, once per day let’s say, that will store the information in a collection. Then I can just use the publish/subscribe method to show that in my react components on the client side. I would prefer to do it this way, rather than loading a bunch of data from different end points to display at the point of request, it would be faster if it was all on the local mongo db I am using.

So to be clear: What is the pattern to set up custom methods (ideally for each 3rd party API), that will go grab their data, once per day, then store it in the collection. I would need to customize each API call because they have different formats and I want to normalize them. I am interested in this problem alone, but if there is something related to subscribe/publish concepts in React with your approach, be sure to include it.

Thanks in advance!

2

There are a few things to consider if you have a more advanced setup, i.e. multiple app servers, where you don’t want the jobs to run from every server instance etc. For basic syncing, there are several full examples in this book, Meteor Design Patterns - would recommend. They go through full examples with the Stripe API, with text/code samples. It may be easier to see full examples than reading comments.

Regarding multiple servers and syncing batch jobs, we moved to managing those tasks with scheduling with a messaging/queue system with iron.io/amazon sqs and scheduled workers, iron worker/AWS lambda functions to send job messages to a Meteor cluster.