[Solved] Is there a way to force commit a doc to mongodb? (Proper use of async method calls)

[ktaggart]

I have a meteor app that stores an array of values in mongodb that are used to create a heat map. The data is generated via a python script, using python-shell from a method.call.

This all works.

The issue is that after I do a db.insert of the data, the doc does not show up until I refresh the browser:
meteor:PRIMARY> db.maps.find().count()
0

(browser refresh)

meteor:PRIMARY> db.maps.find().count()
1

Once the browser is refreshed, the heat map loads the data and renders.

When I generate subsequent data sets, I see exactly the same thing (naturally); I need to refresh the browser to commit the new data to the db.

Any thoughts on why the insert does not commit? Am I only seeing the db cached on the client in the meteor mongo terminal? Is there a way to force a commit without refreshing the browser each time?

Thx,

-k

[rhywden]

It’s not entirely clear what you’re doing here - you first speak of inserting data via python then you speak of the browser needing to refresh and finally you drop something from the console?

Maybe you could put up some actual code which shows how exactly you insert the code and what your publications/subscriptions look like?

[ktaggart]

My post seems pretty clear to me…

I am using python-shell to execute a python script from within meteor. The output of the python script is stored in mongodb.

The problem is that after I insert/update mongodb, the record does not get committed to the db until I refresh the browser, and I provided the mongodb console output, both before and after browser refresh to demonstrate this.

So, my question is, is there a way to force commit to the db?

Here is the relevant code:

Meteor.methods({
        'mapUpdate': function() {
            const bound = Meteor.bindEnvironment((callback) => { callback(); });

            var pyFile = 'sar_server.py',
                pyFilePath = Assets.absoluteFilePath(pyFile),
                pyFolder =  pyFilePath.substr(0, pyFilePath.length - pyFile.length);

            let options = {
                mode: 'text',
                pythonOptions: ['-u'],
                scriptPath: pyFolder
            };

            PythonShell.run(pyFile, options, function(error, results) { 
                var map = [];
                for (var i = 0; i < results.length; i++) {
                    map[i] = [results[i]];
                }

                bound(() => {
                    if (error) throw error;
                    var exists = Maps.findOne({_id: 'rfid1'});

                    if (exists) {
                        // DEBUG
                        console.log('rfid1 exists');
                        Maps.update({
                            _id: 'rfid1'
                        },{
                            $set: {
                                map: map,
                                updatedAt: new Date()
                            }
                        });
                    } else {
                        Maps.insert({
                            _id: 'rfid1',
                            map: map,
                            createdAt: new Date()
                        });
                    }
                });
                // DEBUG
                console.log('map inserted');
            });
        }
    });

-k

[rhywden]

That’s your method. Where and how did you setup subscriptions/publications?

Because it sounds to me like you didn’t setup a proper subscription and thus don’t get the automatic synchronization.

As such, a data refresh only upon complete reload of the page would be a natural consequence.

[ktaggart]

I subscribe in heatmap.js, which is where I make the method call:

Template.heatmap.onCreated(function() {
    Meteor.subscribe('maps');
});

Template.heatmap.onRendered(function() {
    ...
    Meteor.call('mapUpdate', function(error, result) {
        ...
    });
    ...
});

I publish in maps.js, which is where the previously discussed method (mapUpdate) is located:

Meteor.publish('maps', function() {
    return Maps.find();
});

-k

[ktaggart]

So, to get back to my original question, is there a way to force commit a doc to mongodb?

-k

[hluz]

Well, looks to me that you are calling the mapUpdate method from the onRendered callback… This is a strange place to have a method call. It usually is triggered from a user action, and normally in a template event handler… I don’t know / understand what you are trying to achieve, but remember that the onRendered callback will only execute typically when you first render the template screen (which of course happens wen you refresh the page). How/where are you getting the data (from minimongo) to show up on the template?

The way you formulate the question may be very clear to you, but it does not make sense to me and probably to others here, since there is no such think as “committing” the document to MongoDB. The update is requested in the collection.update / collection.insert statements and it is committed to the db as a result of these…

[ktaggart]

I am making other method calls from onRendered that remove db records and they work as expected. The reason I am calling from onRendered is because there is no user interaction at all - it is meant to be fully automated.

I cannot go into too much detail as this is a research project for my company, but essentially this is a data visualization (heat map) app that will eventually update from live data transmitted from a drone.

Functionally, the drone gathers data, packages it as small data.json file and transmits it via POST to the meteor app. The json data is saved to a mongodb collection, ‘Tags’. That json data is then written to the filesystem and is the input to the sar_server.py script referenced in the previous code snippet, which generates a 200x200 array of data points and stores that in the maps collection which is used to generate the heat map. As new data comes in, the heat map has to update without user interaction, hence calling from onRendered instead of a user-initiated event.

The method call in question is triggered by the insertion of the data.json object into the tags collections as such:

var handle = Tags.find().observeChanges({
        added(newDoc) {
            // DEBUG
            console.log('tag: ', newDoc);
            // get latest record from db
            var thisTag = Tags.findOne({}, { sort: { _id: -1 }, limit: 1 });
            var x = thisTag.tag.groundtruth.position_x;
            var y = thisTag.tag.groundtruth.position_y;

            // call server method to create heatmap data
            Meteor.call('mapUpdate', function(error, result) {
                if (error) throw error;

                // get map db entry
                var thisMap = Maps.findOne({ _id: 'rfid1' });
                var map = thisMap.map;
               
                // json to csv
                var csv = Papa.unparse(map, {
                    newline: '\r\n',
                    dynamicTyping: true
                });

                // create heatmap
                createHeatmap(csv, x, y);

                // remove tag from db
                Meteor.call('removeTag', thisTag._id);
            });
        }
    });

The template is very simple:

<template name="heatmap">
    <div id="rfid-heatmap"></div>
</template>

It is just a div that holds a d3js map.

Again, it all “works” except for the map data not being committed to the maps collection without a browser reload.

It is important to note that the method call ‘removeTag’ functions exactly as it should, removing the record and committing the change immediately. It’s just the ‘mapUpdate’ call that doesn’t commit.

Any suggestions on how I can make this work without user interaction would be greatly appreciated.

-k

[rhywden]

Okay, we now got your subscription.We’re missing one last piece of the puzzle:

Where exactly do you put your Maps.find({}) method?

Also, this struck me:

As new data comes in, the heat map has to update without user interaction, hence calling from onRendered instead of a user-initiated event.

Erm, this will actually not update without user interaction. Because it only updates on onRendered which it only does when the user interacts (in your case: refreshes) with the page.

[ktaggart]

Where exactly do you put your Maps.find({}) method?

It’s in a file called maps.js, where the method updateMap resides. They are both shown in previous posts.

Erm, this will actually not update without user interaction. Because it only updates on onRendered which it only does when the user interacts (in your case: refreshes) with the page.

It should. Consider this code fragment from a previous research project I worked on:

Template.tab.onRendered(function() {
  // initialize chart
  var chart;
  var data;
  var svg = d3.select('#lineChart');

   // observe changes to mote collection and update chart
   const updated = Motes.find({});
   const handle = updated.observe({
      changed() {
          // get mote data
          data = getMoteData();
          // update chart
          svg
            .datum(data)
            .call(chart);
      }
      // more chart stuff
      ...
  });

This is a d3js line chart that updates each time new data comes in via POST, triggered off the db changed method. There was no user interaction in this app either, it just displayed categorical data transmitted from remote sensors. Instead of calling a server-side method, I am getting the data from a client-side call, getMoteData().

This is pretty much exactly what I am attempting in the new app, with the major exception of trying to write data from python-shell to the database from a method call originating in onRendered.

This continues to be the issue; how to commit the data…

-k

[rhywden]

I’m really not sure that you’re doing this right. There’s no such thing as “committing data” in MongoDB. At least not with the simple approach you’re showing us here

Either the write to the DB succeeds and or it doesn’t. And if it doesn’t you’ll get an error.

Since you’re calling the update method on the server you’re immediately writing the data to the database - so either your update doesn’t get called in the first place or you’re looking at the wrong thing at the wrong time.

[ktaggart]

I’m really not sure that you’re doing this right.

No argument from me, something is clearly wrong, but I have no idea what…

Either the write to the DB succeeds and or it doesn’t. And if it doesn’t you’ll get an error.

No error and I can console.log the result. the data is correct - it just never gets written to the db.

Since you’re calling the update method on the server you’re immediately writing the data to the database - so either your update doesn’t get called in the first place or you’re looking at the wrong thing at the wrong time.

I am looking at the console.log output and the mongo shell. As stated, console.log prints out the correct info, mongo shell shows no data until after I refresh the browser.

As my “simple approach” isn’t working, do you have an alternative that might? Perhaps a more complex approach? That is why I am asking for help, after all…

-k

[rhywden]

I’m not sure why you’re refreshing the data only on interaction by the user (i.e. at the onRender stage).

You’re saying that new data comes from a drone which posts this via a POST request.

For me, this would be the natural point where to refresh data - a successful POST initiates all the calculations (or maybe a CRON job / timer if that would be computationally heavy).

On the user side you then only have a simple pub/sub connection to the data.

[robfallows]

There are a couple of problems with this code:

1. The method returns before the Python script finishes executing. That in turn means the client call will complete before data is available. It will complete its execution, and because of the next issue won’t see any error.
2. If there is an error in the method, you don’t make it available to the client.
3. You have a potential race condition in your test for existence and subsequent action. That may not matter if you are running only one server instance and one client.

The code below solves 1 and 2. The race condition can also be solved, but is slightly more complex (and may not be necessary). Note, I have not sanity checked your map generation.

Meteor.methods({
  mapUpdate() {
    const pythonShellRun = Meteor.wrapAsync(PythonShell.run, PythonShell);

    var pyFile = 'sar_server.py',
      pyFilePath = Assets.absoluteFilePath(pyFile),
      pyFolder = pyFilePath.substr(0, pyFilePath.length - pyFile.length);

    let options = {
      mode: 'text',
      pythonOptions: ['-u'],
      scriptPath: pyFolder
    };

    try {
      const results = pythonShellRun(pyFile, options);
      const map = results.map(result => [result]);

      const exists = Maps.findOne({ _id: 'rfid1' });

      if (exists) {
        // DEBUG
        console.log('rfid1 exists');
        Maps.update(
          {
            _id: 'rfid1'
          },
          {
            $set: {
              map: map,
              updatedAt: new Date()
            }
          }
        );
      } else {
        Maps.insert({
          _id: 'rfid1',
          map: map,
          createdAt: new Date()
        });
      }
      // DEBUG
      console.log('map inserted');
      return true;
    } catch (error) {
      throw new Meteor.Error('E123', error.message);
    }
  }
});

Your template subscription should use this.subscribe, not Meteor.subscribe unless you intend handling stopping the subscription yourself. Failure to stop the subscription means potential client and/or server memory leaks.

onRendered is not the normal place for a one-off Meteor.call. That’s usually done in the onCreated. Technically, either will work in a similar manner, but onCreated happens earlier and. Note also that normally onCreated and onRendered only run once - they are not reactive. It’s possible to make them re-run with an autorun, but you need to be sure you understand why you’re doing that.

Using a method to populate a collection and a subsequent pub/sub to track changes in that collection is a widely used technique. However, you must ensure that the client code watching for this is running in a reactive context (e.g. a temnplate helper, or an autorun). If you don’t have a reactive context the client will never see the updates.

If you don’t use a helper, one trick is to use a ReativeVar in the onCreated and test in in an autorun in the onRendered:

Template.abc.onCreated(function() {
  this.flag = new ReactiveVar(false);
  Meteor.call ('xyz', (error, result) => {
    if (error) {
      // handle error
    } else {
      this.flag.set(true); // or this.flag.set(result) if appropriate
    }
  });
});

Template.abc.onRendered(function() {
  this.autorun(() => {
    if (this.flag.get()) {
      // data's ready do what's needed
    }
  }
});

[ktaggart]

This is great stuff and works perfectly. To be honest, I’ve never quite gotten the async/await/promise concepts very well (this is my second meteor app, so still learning how it all works) despite reading your blog posts on the subject. Your tuition is exactly what I needed, thanks!

As I review my previous app in which I was getting remote data and displaying it in a line graph, I see that I used a Tracker.autorun method on a Session var in a similar matter, but failed to implement the proscribed solution in this app - along with the correct way of implementing the method itself.

Live and learn.

Thanks again Rob - now onto the next hurdle…

-k