I’m building an app that seeds Collections with a bunch of data from various external APIs. I’ve wired it all up and it works – but it feels dirty and I’m wondering if there’s a better approach. I’d love guidance from others on the best approach here. Maintenance and extensibility are my key concerns.
1) My initial pattern
get starter data from local file
– iterate over & Insert a new Document for each iteratee
get API_1 data
– match on starter data, Update matched Document
get API_2 data
– match on starter data , Update matched Document
… rinse & repeat
2) Alternative pattern
Create a local container Object, e.g. var dataContainer = {obj1, obj2, obj3}
Build these objects up locally (don’t make calls to the db)
When the container Object is fully built up, then iterate over and make Insert calls for each iteratee to seed the Collection
Looking at my code I’m starting to think #2 makes more sense. Thoughts? Or maybe there’s a completely different, more optimal, pattern?
In case it’s clarifying, here’s pseudo code for my approach right now:
// grab starter data from local file
// iterate over and insert new Documents into Collection
function initDestinationCollection() {
var countries_starter = JSON.parse(Assets.getText("countries_starter.json")),
_.each(countries_starter, function(country) {
Destination.insert({
country_code: country.country_code, // this is used as a foreign_key
prop1: data
// ...
})
}
}
// get data from API
// iterate over & match on foreign_key (country_code)
// then update Collection with new data
function API1() {
var response = HTTP.get('url'),
data = response.data;
_.each(data, function(countryData) {
// match on country_code, then add new data
Destinations.update(
{ "country_code": countryData.country_code },
{ $set: { /* new data */ } }
),
}
}
// rinse & repeat with various 3rd party data sources
function API2_And_Beyond() {}
If not, I’m not sure how much sense it makes to insert/update piecemeal. Though this doesn’t really speak to performance/efficiency so much as what makes sense to actually persist in your db.
Good point. The first time I “seed” the app with data it’s not valuable unless it’s complete - my app is entirely dependent on this data set. But thereafter, I’m only updating this data every X days.
So here’s what I’m thinking:
initCollection fn that builds the Collection w/ all of its expected properties. Will only run once in the app’s lifecycle.
update fn’s for each API I’m getting data from. These run every X days.
Looks something like this:
// call once in the App's lifecyle
function initApp() {
initDestinationsCollection();
updateApi1();
updateApi2();
updateApi3()
}
// call every X days
function updateDestinationsCollection() {
updateApi1();
updateApi2();
updateApi3();
}