One-sided client-server sync

mobile
1

I am programming an android Meteor Mobile Cordova app for data collection. This is currently for
an academic research framework, but it will be translated
to a clinical environment in due course.

The Meteor clients need to be able to work offline most of the time and
create persistent entries in the local minimongo for each individual
data collection event. Also, at any one time the app should be able to
display on the client the list of data collection
events currently stored locally. When put on the network with the
server, the clients should be able able to (ideally automatically) send
their data to the server and delete their local copies. Ideally the
server should send some form of acknowledgment back
to client about safe receipt of data. The clients should be completely
independent from each other and only communicate with the server.

I have two issues that I have not managed to tackle so far. The first
issue is to make the clients completely independent, specifically
avoiding sync-ing data from one client to all the other. I have tried to
restrict access by subscribing each data collection
session to only the document in the collection pertinent to that
session. However, when doing that, I don’t seem to be able to access
(for browsing) all the data events currently stored. The second issue
relates to the ability to sync only with the server
and then delete local copies of the data.

This is the first app I implement in Meteor. I have gone through the
tutorials and scoured the discussion forums. However, I could not find
any resource that addresses issues similar with mine. I would really
appreciate your help and/or advice.

2

Did you try GroundDB? It’s not perfect but one of the most close solutions ready as a package I know of.

3

I use GroundDB to make the database persistent between offline session and it works fine. But it does not, as far as I know, deal with the issue of syncing client-to-server only - am I missing something?

4

Ok so the second issue is actually solved already with GroundDB?

The first issue is to make the clients completely independent, specifically avoiding sync-ing data from one client to all the other.

In general will sync all data which is published. So if you publish document A to 2 clients then they will both sync.

I have tried to restrict access by subscribing each data collection session to only the document in the collection pertinent to that session.

That seems to be the most relevant solution yes. So let’s say user X works on document A he is the only one subscribing to that document and so he is the only one getting it.

However, when doing that, I don’t seem to be able to access (for browsing) all the data events currently stored.

I am not totally sure on how you data model looks but yes, if the user is not subscribed to all data he won’t receive all data. Which is what you seem to want but I think there is more story behind it. Can you explain a bit more about who should receive which data? And how you publish / subscribe that currently?

5

Apologies for not explaining clearly in the first place and many thanks for taking time to help me. I’ll try to explain a bit better the model I am trying to implement.

I have a sever, behind a firewall and restricted to a local network because I am dealing with sensitive data. Let’s say that the server has a SESSIONDATA collection to store data collected from clients. I have an unspecified number of mobile clients that have to work offline. When clients are first setup they should connect to the server to register and receive a client ID to be stored locally and appended to any subsequent id for SESSIONDATA documents generated on that client - I am not sure what framework within meteor I can use to achieve this.

Let’s say that we registered CLNT01 and CLNT02 and then these two clients collected offline a series of sessions, say CLNT01S01, CLNT01S02 and CLNT02S01, CLNT02S02, CLNT02S03, that are added to the corresponding local minimongo version of SESSIONDATA. Between subsequent runs and possible re-starts of the mobile device the documents have to be persistent in the local SESSIONDATA - I achieved the persistence using GroundDB. While the clients are offline, users should be able to browse local SESSIONDATA and list all documents on that client added since the latest upload to the server. This browsing is trivial if clients subscribe to the entire SESSIONDATA collection. However, if each offline client session only subscribes to the document specific to the current data collection session (say current session only subscribes to CLIENTS02), then I don’t know how to browse all current documents from current local SESIONDATA (that is, get the listing of CLNT01S01 and CLNT01S02).

When online with the server, both clients should be able to send their data to the server, receive notification of receipt and delete the uploaded data from their local database. No data from sever should be pushed onto any client. With my current knowledge of the metero platform I haven’t found the mechanism by which upload is acknowledged and uploaded data is deleted from clients, while remaining persistent on the server.

6

Let’s talk a bit more in simple Meteor terms. We can add more collections and exceptions later.

A client is a user. So on first setup the user logs in. He now has Meteor.userId() so we can know who he is.

There is a collection with sessions. A user can have one or more sessions. He makes them offline. Session is not the right name since it conflicts but works for the example.

So let’s first publish the data to each user:

// shared
Sessions = new Mongo.Collection("sessions");

// server
Meteor.publish('userSessions', function() {
  return Sessions.find({
    userId: this.userId,
   "processedByServer" : { "$exists" : false }  // will explain this one later on
  });
});

// client
Meteor.subscribe("userSessions");

They will, in your setup, not receive any data at all. The client will subscribe, but the search will result in zero results.

So now we have the basic data, standard Meteor stuff, and this should work. You can, and already have, GroundDB to keep it working offline so just keep that part.

So, now we are offline working, let’s insert:

// Off course here you add the real data also.
Sessions.insert({ userId: Meteor.userId(), sessionNumber: 'S02' });

Now onto your next requirement:

When online with the server, both clients should be able to send their data to the server

So here we go online again. That will automatically send the data from the client to the server.

receive notification of receipt and delete the uploaded data from their local database.

You could do that by the following:

// server
var query = Sessions.find({ "processedByServer" : { "$exists" : false } });
var handle = query.observeChanges({
  added: function (id, session) {
    // Here you receive the new document from the client, do what you want with it
    // Because we set processedByServer = true it will not be sent to the client again (see publication)
    Sessions.update({_id: id}, { $set: { processedByServer: true}});
  },
});

// client - Only needed if you want to show a notification that the server removed the document from the client. If you don't need that notification you can skip this.
var query = Sessions.find({});
var handle = query.observeChanges({
  removed: function () {
    // here you know the report has been removed from the client
    console.log('report got removed from client');
    // it is removed because the publication no longer publishes this because the server set processedByServer to true
    // And to be specific the data is now removed from minimongo
  },
});

No data from sever should be pushed onto any client.

Also solved, not sure really no data is communicated but it will be removed from the collection. If you want to know exactly how read for example: https://github.com/meteor/meteor/blob/devel/packages/ddp/DDP.md

With my current knowledge of the metero platform I haven’t found the mechanism by which upload is acknowledged and uploaded data is deleted from clients, while remaining persistent on the server.

As far as I understand your post this is all solved with the example above.

Depending on the sensitivity of the data I am not sure how well-deleted the data is on the client. It works in a functional way but I don’t know it’s really overwritten or something. Also caches might be an issue for data security.