Is GroundDB the best for offline storage?

[brunoneumann]

Hello @banjerluke! Sorry to bother you, but could you give an example of how to use your code?

[banjerluke]

@brunoneumann

…could you give an example of how to use your code?

First, since this isn’t a package or anything, you’ll have to drop the files into your codebase somewhere and do an import { GroundedCollection } from 'path/to/GroundedCollection.ts';. You’ll also need to install Dexie and underscore: npm install --save dexie underscore. (You can swap out the throttle implementation if you don’t want to use underscore.)

I kept the API more or less compatible with ground:db, so start with the ground:db README and whatever else you find about ground:db – GroundedCollection.ts is more or less a drop-in replacement in terms of API. I did rename some of the more “internal” functions to more self-explanatory names (i.e., saveDocumentToMemory, etc).

The basic idea is that there’s an in-memory collection that you can access via minimongo, and a localstorage-backed collection that is loaded initially and then kept in sync.

I cannot stress enough how helpful it was to go through the source of ground:db line by line, which is what I did years ago when I first implemented offline data storage (well before my rewrite). It was intimidating at first, but when it was a “black box” my implementation really suffered. Once I understood what was happening under the hood, I was able to make my app much better. So I would definitely recommend doing the same with my code.

That said here’s some actual usage code pulled from my app so you can get an idea of basic usage:

    this.subscription = Meteor.subscribe(options.publicationName);
    this.groundedCollection = new GroundedCollection(storageDBName);
    await this.groundedCollection.waitUntilLoaded();
    await waitUntilReactive(() => this.subscription.ready());
    this.groundedCollection.observeSource(this.liveCollection.find());
    this.groundedCollection.keep(this.liveCollection.find());
    this._offlineReady.set(true);

    // then interact with this.groundedCollection like a normal Meteor collection
    this.groundedCollection.findOne();

Btw, the waitUntilReactive call is to a super-useful utility function I made that converts a Tracker autorun into a promise that’s called when the function returns a truthy value:

export const waitUntilReactive = (condition: () => any): Promise<any> => {
  return new Promise((resolve) => {
    Tracker.autorun((comp: Tracker.Computation) => {
      const result = condition();
      if (result) {
        comp.stop();
        resolve(result);
      }
    });
  });
};

[brunoneumann]

Hello @banjerluke ! Thanks for this excellent explanation. I’ve never tried Dexie before, good to know there is such a tool for IndexedDB. You did a nice job creating the bulk operation and keeping the usage similar to grounddb, I’ll test it for sure.

About grounddb, there is an effort to bring it to the meteor community packages, according to this post: https://github.com/GroundMeteor/db/issues/212. No reply by creator yet.

[cmoore]

@banjerluke , I got your code to work to save data to indexedDB. It works great! I can read that data as well. But I seem to be missing something? I’m not sure how to sync this up to MongoDB? Is this only a pure offline solution that doesn’t sync with MongoDB?

[banjerluke]

@cmoore I updated the demo code above with some comments to explain a bit about what’s going on. Meteor does the syncing with the server, GroundedCollection syncs with the in-browser copy that Meteor provides. It’s pretty automatic for read-only access. If you want to make changes back to the database, you should use Meteor methods as usual. If you want to be able to make changes offline, this means queueing up a list of methods to call when online, and it all gets rather complicated and situation-specific; I do this for a few collections but I use slightly different approaches depending on the nature of the collection. The general approach, though, is that I have additional GroundedCollections that store a list of records to be inserted, updated, and removed, and then when Meteor.status().connected == true I loop through them and call the relevant insert/update/remove methods on the server.

The original version of ground:db did automatic method-queueing, I believe, but they dropped all that for version 2 because it was problematic. I do think this is a situation where it’s good to build your own solution so that you deeply understand what’s going on; too many tricky edge cases to rely on a “magic” solution (which I don’t even think exists here).

[cmoore]

@banjerluke , thanks for explaining more. I’m going to deep dive into this because I want an app to be able to work offline and then sync up when online. This is a great start! From what I can tell, I can build out a Meteor app the “normal way”, then easily adjust it with GroundedCollection to make the data available offline, and then put in an effort to que the changed data and sync it up when Meteor is connected again.

[cmoore]

ok, I’ve got my data available offline and coordinating with the Meteor data. Now it’s time to figure out queing the data when offline and syncing when Meteor is connected again.

When that is done, I will create a demo app and publish it to github for all to view.

[paulishca]

This thread might be of some use. In summary, if you are offline, use functions, when online, use methods and understand the security risks and some scenarios (e.g. user has access to write “something”. User goes offline. Meanwhile, user looses right to write “something”. User writes that “something” offline. Then when online users syncs and there is a mismatch in user writes which leads to data inconsistency.

Setup meteor app to be offline first

I have an app that must be offline first. I have researched this and implemented as much as I can. with my current implementation, I realise method will retry when connection is gained and sometimes produces multiple results of the method call, and thia is because; the method are not available on the client when action is called at offline status. there’s no feedback to user that the method failed. I have looked at grounddb, but I believe it doesn’t solve my methods problem or maybe I stil…

[banjerluke]

I do all access control on the server, which is probably where it should be happening in any case. I never thought about the situation where a method gets called that requires a permission that the user loses while offline – I guess it would fail repeatedly, which is something I catch (using Meteor.apply() instead of Meteor.call()). Anyway, hasn’t come up for me yet but that’s a good point.

For what it’s worth, my offline implementation got a lot less messy when I moved everything into a wrapper class (one for each collection) that handles the data fetching and manipulation, and another class that specifically handles the offline stuff. For the most part, the rest of the app interacts with the wrapper class without caring whether it’s actually online or offline. The class figures out what to do with each function call based on whether we’re online or offline. Hence the references to this.groundedCollection in the code above, for example. No other class on the client interacts with the collection or minimongo directly.

Here’s an example of one of those inner classes that deals with offline stuff. It’s a generic class that gets subclassed (further down in the paste) for specific collections. In reality I have a bit of other app-specific code in here which I’ve removed, so as written this code is UNTESTED and is given “as is”. But it is at least pretty close to what has been working well for me.

interface StorableType {
  _id: string;
  unsynced?: boolean | string;
  createdAt: Date;
  updatedAt: Date;
}

export abstract class OfflineCollectionBackendBase<T extends StorableType> {
  groundedServerRecords: GroundedCollection<T>;

  protected recordsToInsert: GroundedCollection<T>;
  protected recordsToUpdate: GroundedCollection<T>;
  protected recordsToRemove: GroundedCollection<{ _id: string }>;

  constructor({
    indexDBName,
    insertionsDBName,
    updatesDBname,
    removalsDBName,
  }: {
    indexDBName: string;
    insertionsDBName: string;
    updatesDBname: string;
    removalsDBName: string;
  }) {
    this.groundedServerRecords = new GroundedCollection<T>(indexDBName);
    this.recordsToInsert = new GroundedCollection<T>(insertionsDBName);
    this.recordsToUpdate = new GroundedCollection<T>(updatesDBname);
    this.recordsToRemove = new GroundedCollection<{ _id: string }>(removalsDBName);
  }

  async groundWithLiveCollection(collection: Mongo.Collection<T>[]): Promise<void> {
    await this.groundedServerRecords.waitUntilLoaded();
    this.groundedServerRecords.observeSource(collection.find());
    this.groundedServerRecords.keep(collection.find());
    await Promise.all([
      this.recordsToInsert.waitUntilLoaded(),
      this.recordsToUpdate.waitUntilLoaded(),
      this.recordsToRemove.waitUntilLoaded(),
    ]);
    this.initUserChangesProcessingAutorun();
    await waitUntilReactive(() => this.groundedServerRecords.pendingWrites.isZero());
    this._offlineReady.set(true);
  }

  private _offlineReady = new ReactiveVar(false);
  offlineReady(): boolean {
    return this._offlineReady.get();
  }

  pendingChanges(): number {
    return (
      this.recordsToInsert.find().count() +
      this.recordsToUpdate.find().count() +
      this.recordsToRemove.find().count()
    );
  }

  async insert(record: Mongo.OptionalId<T>): Promise<string> {
    record.unsynced = 'new';
    record.createdAt = record.updatedAt = new Date();
    const newId = this.recordsToInsert.insert(record);
    const newRecord = this.recordsToInsert.findOne(newId);
    if (!newRecord) {
      throw new Error('Could not find record that was just inserted offline');
    }
    this.groundedServerRecords.insert(newRecord as Mongo.OptionalId<T>);
    return newId;
  }

  async update(record: T): Promise<T> {
    record.updatedAt = new Date();
    if (!record.unsynced) record.unsynced = true;
    const unsyncedRecord = this.recordsToInsert.findOne(record._id);
    if (unsyncedRecord) {
      this.recordsToInsert.upsert(record._id, record);
    } else {
      this.recordsToUpdate.upsert(record._id, record);
    }
    this.groundedServerRecords.upsert(record._id, record);
    return unsyncedRecord || record;
  }

  async remove(recordId: string): Promise<void> {
    if (this.recordsToInsert.findOne(recordId)) {
      this.recordsToInsert.remove(recordId);
    } else {
      this.recordsToRemove.upsert(recordId, { _id: recordId });
    }
    this.recordsToUpdate.remove(recordId);
    this.groundedServerRecords.remove(recordId);
  }

  protected preventSync = new ReactiveVar(false);

  private initUserChangesProcessingAutorun(): void {
    const syncInProgress = new ReactiveVar(false);
    Tracker.autorun(async () => {
      if (syncInProgress.get()) return;
      if (this.preventSync.get()) return;
      if (!Meteor.status().connected) return; // can't do this offline!
      if (!Meteor.userId()) return; // don't do this if not logged in
      const toInsert = this.recordsToInsert.find().fetch();
      const toUpdate = this.recordsToUpdate.find().fetch();
      const toRemove = this.recordsToRemove.find().fetch();
      if ([ toInsert, toUpdate, toRemove ].every((array) => array.length === 0)) return;
      syncInProgress.set(true);
      try {
        await Promise.all([
          ...toInsert.map(async (record) => {
            await this.doServerInsert(record);
            this.recordsToInsert.remove(record._id);
          }),
          ...toUpdate.map(async (record) => {
            await this.doServerUpdate(record);
            this.recordsToUpdate.remove(record._id);
          }),
          ...toRemove.map(async (record) => {
            await this.doServerRemove(record);
            this.recordsToRemove.remove(record._id);
          }),
        ]);
        Meteor.setTimeout(() => syncInProgress.set(false), 1000);
      } catch (error) {
        console.error(error); // eslint-disable-line no-console
        Sentry.captureException(error);
        // Bert.alert('Error while syncing data to the cloud. Will retry in 20 seconds.');
        Meteor.setTimeout(() => syncInProgress.set(false), 20000);
      }
    });
  }

  protected abstract doServerInsert(record: T): Promise<string>;
  protected abstract doServerUpdate(record: T): Promise<void>;
  protected abstract doServerRemove(record: { _id: string }): Promise<void>;
  protected abstract getRecordsFromServer(recordIds: string[]): Promise<T[]>;
};

// ---------------------------------

export class OfflineWidgetsBackend extends OfflineCollectionBackendBase<any> {
  constructor() {
    super({
      indexDBName: 'widgetsIndex',
      insertionsDBName: 'widgetsInserted',
      updatesDBname: 'widgetsUpdated',
      removalsDBName: 'widgetsRemoved',
    });
  }

  protected async doServerInsert(record: WidgetRecord): Promise<string> {
    const oldId = record._id;
    const newId = await callServerMethodWithoutRetry('widgets.insert', record);
    if (oldId && oldId !== newId) {
      // updateRouteId(oldId, newId);
      this.groundedServerRecords.remove(oldId); // this will be resynced automatically from the publication
    }
    return newId;
  }

  protected async doServerUpdate(record: WidgetRecord): Promise<void> {
    return await callServerMethodWithoutRetry('widgets.update', record);
  }

  protected async doServerRemove({ _id: songId }: { _id: string }): Promise<void> {
    return await callServerMethodWithoutRetry('widgets.delete', songId);
  }

  protected async getRecordsFromServer(recordIds: string[]): Promise<WidgetRecord[]> {
    if (recordIds.length === 0) return [];
    const result = await callServerMethodWithoutRetry('widgets.fetchMultiple', recordIds);
    return result.songs;
  }
}

// ---------------------------------

export function callServerMethodWithoutRetry(methodName: string, ...args: any[]): Promise<any> {
  return new Promise((resolve, reject) => {
    Meteor.apply(
      methodName,
      args,
      //@ts-expect-error Mistake in Meteor's typings
      { noRetry: true },
      (error, result) => {
        if (error) return reject(error);
        resolve(result);
      }
    );
  });
}

[truedon]

Damn bro this is godlike. I guess this should be in the base package for offline? Thanks for sharing this

[bratelefant]

@banjerluke thanks for this beautiful piece of code and all your explanations. Offline first is crucial to apps imho. Currently testing your GroundedCollection in react hooks to fetch data. I‘d really like seeing stuff like this in the base someday

[jam]

Curious if you made any headway on the demo app . Would be great to see an example.

[cmoore]

woah, really late reply. But, no, I haven’t made any progress for an offline storage sample. I’ve had way too much on my plate for a long time. Sorry about that.

[bratelefant]

I’d also like to bring this up again; I’m wondering whether there’s a canonical way to implement offline first support for Meteor.user() and Roles…

In combination with Grounded Collections I’d like to display offline-data (possibly already available on the client from offline caching, so no policy issues here) if Meteors connection status is offline. But often views (in my case react components) depend on Meteor.user() or Roles…

[paulishca]

Hi @bratelefant would you be interested to do a 1 to 1 on this one of these days, maybe next week.

[bratelefant]

Hi, really sorry to get back so late. I decided to give it a try with redux + redux persist in a react native client for my app, following this article with some modifications: Offline First React Native + Meteor Apps | by Spencer Carli | HackerNoon.com | Medium

This also works with Meteor.users collection. Maybe this could also do the trick in non-native react using a indexeddb adapter for instance…

[paulishca]

I went this way for all those fixtures that you only need to send on a client once like app settings, data collections in the form of JSON etc. I even fetch a larger json and store it in Persist.
This is the way I wanted to suggest if we went in a 1:1. I am glad you went this path and hopefully will be getting more and more of those nice things popularized here.

[paulishca]

I am at a Mongo Conference right now :). Offline and sync + full reactivity available when Atlas as a DB host https://www.mongodb.com/docs/realm/

[cmoore]

From what I can see, sync is only available for non web apps. Unless you have learned something new at the conference… which would be awesome!

From https://www.mongodb.com/docs/realm/web/ …

Unlike the other Realm SDKs, the Web SDK does not support creating a local database or using Atlas Device Sync.

and, https://www.mongodb.com/docs/realm/sdk/node/ …

Use the Node.js SDK to build for server applications, desktop applications, CLIs, IoT programs, and more.

So, creating a desktop or mobile app from meteor could potentially work.

The Node.js SDK does not support applications written for web browsers. For Web apps, use the Realm Web SDK.

[paulishca]

You are 100% right. I am digging in now to see the difference between their Javascript React Native SDK and Javascript Web SDK but I suspect this has something to do with the browser technology missing something.