Meteor way to handle data feed imports


#1

Hello,

We are getting xml feeds and update(upsert) feeding them to collection A.
Than these records should be inserted into B and if there is already same record with other source xml, just extend that record in B with duplicates field and push info about source to it.

I was writing methods for this yesterday and realized I am dumb and doing it wrong way.

Now I have it as cursorA.observe, added: and changed: callbacks and it is working as a charm, without need for any methods and you can even change it from mongo and it updates nicely, test nicely.

Is this correct meteor way? Cause it still feels little odd to run cursorA = find… so you can run observe as background task on server side.

And collection hooks seems to handle just meteor instance calls/etc, not mongo external stuff.
And there is plan to mess with DB from various sources.


#2

What’s wrong here is your unwillingness to admit success. You had a challenging problem, solved it and tested it thoroughly.

And you tell me you’re worried that this is not the correct meteor way.

Well… I’ll tell you the correct meteor way.

Abstract what you have done into a meteor package so you and anyone can do what you have done with a few lines of code.

In the meantime, I invite you to read this inspiring blog post by Sacha Greif.


#3

Well, there are some things which are not very reusable in package, like this piece
In EshopProducts collection are all various product variations from various eshops
And we are matching them to Products collection master products by adding product_id.
Which will insert few fields from EshopProducts document to Products.eshop_products array.
And tracking changes over time.
And I am in the middle of writing and refactoring it, as you can see

Meteor.startup ->
  handle = EshopProducts.find({}).observe(
    changed: ( newDoc, oldDoc ) ->
      console.log 'change observed'
      if newDoc.product_id? 
        if newDoc.price isnt oldDoc.price or newDoc.URL isnt oldDoc.URL

          productMatched = Products.findOne
            _id: newDoc.product_id
            eshop_products:
              $elemMatch:
                eshop_id: newDoc.eshop_id
                itemId: newDoc.itemId

          if not productMatched?
            #push initial value
            #this should never run as we are creating it in product_id update branch
            console.log "pushing new outside of product_id update branch, should never run"
            Products.update { _id: newDoc.product_id },
              $push:
                eshop_products:
                  eshop_id: newDoc.eshop_id
                  itemId: newDoc.itemId
                  price: newDoc.price
                  URL: newDoc.URL
          else
            #updating records
            Products.update { _id: newDoc.product_id, eshop_products: $elemMatch:
              eshop_id: newDoc.eshop_id
              itemId: newDoc.itemId },
                $set:
                  ### rewriting whole record, bull...
                  "eshop_products.$":
                    eshop_id: newDoc.eshop_id
                    itemId: newDoc.itemId
                    price: newDoc.price
                    url: newDoc.url
                  ###
                  "eshop_products.$.price": newDoc.price
                  "eshop_products.$.URL": newDoc.URL

        else
          # other changes than price/URL
          if not oldDoc.product_id? and newDoc.product_id?
            #when we added product_id after pairing to master product
            #initial pushing eshop_product to child array
            Products.update { _id: newDoc.product_id },
              $push:
                eshop_products:
                  eshop_id: newDoc.eshop_id
                  itemId: newDoc.itemId
                  price: newDoc.price
                  URL: newDoc.URL
      else
        # not paired yet, we dont care
  )