Is GroundDB the best for offline storage?

I can replicate on real device too.

I wonder if it’s something caused by an update to prevent tracking or something that’s all that comes to mind

I’ll discover later in which specific version of meteor this problem starts… I’m keeping my mobile app working with meteor 1.11.1

1 Like

I found that the internal API in LocalCollection that ground:db uses switched from an object to a Map sometime around Meteor 2.4. I didn’t think to look at any of the forks. Rather, I did a from-the-ground-up (pun unintended) rewrite, which I’m calling GroundedCollection. I switched to Typescript, added the ability to specify compression/decompression functions for writes/reads to IndexedDB, and switched from localforage to Dexie, but otherwise tried to keep the API similar.

As a bonus, Dexie allowed me to use bulk reads/writes, which decreased load times by an enormous amount – I can’t remember the multiplier but it was in the tens if not hundreds of times.

It’s not packaged or anything, but it’s been running in production for a couple months now with no issues, so I will share it here in case anyone would like to use it:

11 Likes

Neat! I had been thinking of trying a ground up rewrite myself - glad someone beat me to it!

Awesome stuff man thanks for sharing!!

The problem of persistency I mentioned earlier starts in Meteor 2.3

This is awesome! Thank you very much for sharing this! :raised_hands:

Hello @banjerluke! Sorry to bother you, but could you give an example of how to use your code?

First, since this isn’t a package or anything, you’ll have to drop the files into your codebase somewhere and do an import { GroundedCollection } from 'path/to/GroundedCollection.ts';. You’ll also need to install Dexie and underscore: npm install --save dexie underscore. (You can swap out the throttle implementation if you don’t want to use underscore.)

I kept the API more or less compatible with ground:db, so start with the ground:db README and whatever else you find about ground:db – GroundedCollection.ts is more or less a drop-in replacement in terms of API. I did rename some of the more “internal” functions to more self-explanatory names (i.e., saveDocumentToMemory, etc).

The basic idea is that there’s an in-memory collection that you can access via minimongo, and a localstorage-backed collection that is loaded initially and then kept in sync.

I cannot stress enough how helpful it was to go through the source of ground:db line by line, which is what I did years ago when I first implemented offline data storage (well before my rewrite). It was intimidating at first, but when it was a “black box” my implementation really suffered. Once I understood what was happening under the hood, I was able to make my app much better. So I would definitely recommend doing the same with my code.

That said here’s some actual usage code pulled from my app so you can get an idea of basic usage:

    this.subscription = Meteor.subscribe(options.publicationName);
    this.groundedCollection = new GroundedCollection(storageDBName);
    await this.groundedCollection.waitUntilLoaded();
    await waitUntilReactive(() => this.subscription.ready());
    this.groundedCollection.observeSource(this.liveCollection.find());
    this.groundedCollection.keep(this.liveCollection.find());
    this._offlineReady.set(true);

    // then interact with this.groundedCollection like a normal Meteor collection
    this.groundedCollection.findOne();

Btw, the waitUntilReactive call is to a super-useful utility function I made that converts a Tracker autorun into a promise that’s called when the function returns a truthy value:

export const waitUntilReactive = (condition: () => any): Promise<any> => {
  return new Promise((resolve) => {
    Tracker.autorun((comp: Tracker.Computation) => {
      const result = condition();
      if (result) {
        comp.stop();
        resolve(result);
      }
    });
  });
};
4 Likes

Hello @banjerluke ! Thanks for this excellent explanation. I’ve never tried Dexie before, good to know there is such a tool for IndexedDB. You did a nice job creating the bulk operation and keeping the usage similar to grounddb, I’ll test it for sure.

About grounddb, there is an effort to bring it to the meteor community packages, according to this post: https://github.com/GroundMeteor/db/issues/212. No reply by creator yet.

@banjerluke , I got your code to work to save data to indexedDB. It works great! I can read that data as well. But I seem to be missing something? I’m not sure how to sync this up to MongoDB? Is this only a pure offline solution that doesn’t sync with MongoDB?

@cmoore I updated the demo code above with some comments to explain a bit about what’s going on. Meteor does the syncing with the server, GroundedCollection syncs with the in-browser copy that Meteor provides. It’s pretty automatic for read-only access. If you want to make changes back to the database, you should use Meteor methods as usual. If you want to be able to make changes offline, this means queueing up a list of methods to call when online, and it all gets rather complicated and situation-specific; I do this for a few collections but I use slightly different approaches depending on the nature of the collection. The general approach, though, is that I have additional GroundedCollections that store a list of records to be inserted, updated, and removed, and then when Meteor.status().connected == true I loop through them and call the relevant insert/update/remove methods on the server.

The original version of ground:db did automatic method-queueing, I believe, but they dropped all that for version 2 because it was problematic. I do think this is a situation where it’s good to build your own solution so that you deeply understand what’s going on; too many tricky edge cases to rely on a “magic” solution (which I don’t even think exists here).

@banjerluke , thanks for explaining more. I’m going to deep dive into this because I want an app to be able to work offline and then sync up when online. This is a great start! From what I can tell, I can build out a Meteor app the “normal way”, then easily adjust it with GroundedCollection to make the data available offline, and then put in an effort to que the changed data and sync it up when Meteor is connected again.

ok, I’ve got my data available offline and coordinating with the Meteor data. Now it’s time to figure out queing the data when offline and syncing when Meteor is connected again.

When that is done, I will create a demo app and publish it to github for all to view. :sunglasses:

2 Likes

This thread might be of some use. In summary, if you are offline, use functions, when online, use methods and understand the security risks and some scenarios (e.g. user has access to write “something”. User goes offline. Meanwhile, user looses right to write “something”. User writes that “something” offline. Then when online users syncs and there is a mismatch in user writes which leads to data inconsistency.

1 Like

I do all access control on the server, which is probably where it should be happening in any case. I never thought about the situation where a method gets called that requires a permission that the user loses while offline – I guess it would fail repeatedly, which is something I catch (using Meteor.apply() instead of Meteor.call()). Anyway, hasn’t come up for me yet but that’s a good point.

For what it’s worth, my offline implementation got a lot less messy when I moved everything into a wrapper class (one for each collection) that handles the data fetching and manipulation, and another class that specifically handles the offline stuff. For the most part, the rest of the app interacts with the wrapper class without caring whether it’s actually online or offline. The class figures out what to do with each function call based on whether we’re online or offline. Hence the references to this.groundedCollection in the code above, for example. No other class on the client interacts with the collection or minimongo directly.

Here’s an example of one of those inner classes that deals with offline stuff. It’s a generic class that gets subclassed (further down in the paste) for specific collections. In reality I have a bit of other app-specific code in here which I’ve removed, so as written this code is UNTESTED and is given “as is”. But it is at least pretty close to what has been working well for me.

interface StorableType {
  _id: string;
  unsynced?: boolean | string;
  createdAt: Date;
  updatedAt: Date;
}

export abstract class OfflineCollectionBackendBase<T extends StorableType> {
  groundedServerRecords: GroundedCollection<T>;

  protected recordsToInsert: GroundedCollection<T>;
  protected recordsToUpdate: GroundedCollection<T>;
  protected recordsToRemove: GroundedCollection<{ _id: string }>;

  constructor({
    indexDBName,
    insertionsDBName,
    updatesDBname,
    removalsDBName,
  }: {
    indexDBName: string;
    insertionsDBName: string;
    updatesDBname: string;
    removalsDBName: string;
  }) {
    this.groundedServerRecords = new GroundedCollection<T>(indexDBName);
    this.recordsToInsert = new GroundedCollection<T>(insertionsDBName);
    this.recordsToUpdate = new GroundedCollection<T>(updatesDBname);
    this.recordsToRemove = new GroundedCollection<{ _id: string }>(removalsDBName);
  }

  async groundWithLiveCollection(collection: Mongo.Collection<T>[]): Promise<void> {
    await this.groundedServerRecords.waitUntilLoaded();
    this.groundedServerRecords.observeSource(collection.find());
    this.groundedServerRecords.keep(collection.find());
    await Promise.all([
      this.recordsToInsert.waitUntilLoaded(),
      this.recordsToUpdate.waitUntilLoaded(),
      this.recordsToRemove.waitUntilLoaded(),
    ]);
    this.initUserChangesProcessingAutorun();
    await waitUntilReactive(() => this.groundedServerRecords.pendingWrites.isZero());
    this._offlineReady.set(true);
  }

  private _offlineReady = new ReactiveVar(false);
  offlineReady(): boolean {
    return this._offlineReady.get();
  }

  pendingChanges(): number {
    return (
      this.recordsToInsert.find().count() +
      this.recordsToUpdate.find().count() +
      this.recordsToRemove.find().count()
    );
  }

  async insert(record: Mongo.OptionalId<T>): Promise<string> {
    record.unsynced = 'new';
    record.createdAt = record.updatedAt = new Date();
    const newId = this.recordsToInsert.insert(record);
    const newRecord = this.recordsToInsert.findOne(newId);
    if (!newRecord) {
      throw new Error('Could not find record that was just inserted offline');
    }
    this.groundedServerRecords.insert(newRecord as Mongo.OptionalId<T>);
    return newId;
  }

  async update(record: T): Promise<T> {
    record.updatedAt = new Date();
    if (!record.unsynced) record.unsynced = true;
    const unsyncedRecord = this.recordsToInsert.findOne(record._id);
    if (unsyncedRecord) {
      this.recordsToInsert.upsert(record._id, record);
    } else {
      this.recordsToUpdate.upsert(record._id, record);
    }
    this.groundedServerRecords.upsert(record._id, record);
    return unsyncedRecord || record;
  }

  async remove(recordId: string): Promise<void> {
    if (this.recordsToInsert.findOne(recordId)) {
      this.recordsToInsert.remove(recordId);
    } else {
      this.recordsToRemove.upsert(recordId, { _id: recordId });
    }
    this.recordsToUpdate.remove(recordId);
    this.groundedServerRecords.remove(recordId);
  }

  protected preventSync = new ReactiveVar(false);

  private initUserChangesProcessingAutorun(): void {
    const syncInProgress = new ReactiveVar(false);
    Tracker.autorun(async () => {
      if (syncInProgress.get()) return;
      if (this.preventSync.get()) return;
      if (!Meteor.status().connected) return; // can't do this offline!
      if (!Meteor.userId()) return; // don't do this if not logged in
      const toInsert = this.recordsToInsert.find().fetch();
      const toUpdate = this.recordsToUpdate.find().fetch();
      const toRemove = this.recordsToRemove.find().fetch();
      if ([ toInsert, toUpdate, toRemove ].every((array) => array.length === 0)) return;
      syncInProgress.set(true);
      try {
        await Promise.all([
          ...toInsert.map(async (record) => {
            await this.doServerInsert(record);
            this.recordsToInsert.remove(record._id);
          }),
          ...toUpdate.map(async (record) => {
            await this.doServerUpdate(record);
            this.recordsToUpdate.remove(record._id);
          }),
          ...toRemove.map(async (record) => {
            await this.doServerRemove(record);
            this.recordsToRemove.remove(record._id);
          }),
        ]);
        Meteor.setTimeout(() => syncInProgress.set(false), 1000);
      } catch (error) {
        console.error(error); // eslint-disable-line no-console
        Sentry.captureException(error);
        // Bert.alert('Error while syncing data to the cloud. Will retry in 20 seconds.');
        Meteor.setTimeout(() => syncInProgress.set(false), 20000);
      }
    });
  }

  protected abstract doServerInsert(record: T): Promise<string>;
  protected abstract doServerUpdate(record: T): Promise<void>;
  protected abstract doServerRemove(record: { _id: string }): Promise<void>;
  protected abstract getRecordsFromServer(recordIds: string[]): Promise<T[]>;
};

// ---------------------------------

export class OfflineWidgetsBackend extends OfflineCollectionBackendBase<any> {
  constructor() {
    super({
      indexDBName: 'widgetsIndex',
      insertionsDBName: 'widgetsInserted',
      updatesDBname: 'widgetsUpdated',
      removalsDBName: 'widgetsRemoved',
    });
  }

  protected async doServerInsert(record: WidgetRecord): Promise<string> {
    const oldId = record._id;
    const newId = await callServerMethodWithoutRetry('widgets.insert', record);
    if (oldId && oldId !== newId) {
      // updateRouteId(oldId, newId);
      this.groundedServerRecords.remove(oldId); // this will be resynced automatically from the publication
    }
    return newId;
  }

  protected async doServerUpdate(record: WidgetRecord): Promise<void> {
    return await callServerMethodWithoutRetry('widgets.update', record);
  }

  protected async doServerRemove({ _id: songId }: { _id: string }): Promise<void> {
    return await callServerMethodWithoutRetry('widgets.delete', songId);
  }

  protected async getRecordsFromServer(recordIds: string[]): Promise<WidgetRecord[]> {
    if (recordIds.length === 0) return [];
    const result = await callServerMethodWithoutRetry('widgets.fetchMultiple', recordIds);
    return result.songs;
  }
}

// ---------------------------------

export function callServerMethodWithoutRetry(methodName: string, ...args: any[]): Promise<any> {
  return new Promise((resolve, reject) => {
    Meteor.apply(
      methodName,
      args,
      //@ts-expect-error Mistake in Meteor's typings
      { noRetry: true },
      (error, result) => {
        if (error) return reject(error);
        resolve(result);
      }
    );
  });
}

2 Likes

Damn bro this is godlike. I guess this should be in the base package for offline? Thanks for sharing this

1 Like

@banjerluke thanks for this beautiful piece of code and all your explanations. Offline first is crucial to apps imho. Currently testing your GroundedCollection in react hooks to fetch data. I‘d really like seeing stuff like this in the base someday

2 Likes

Curious if you made any headway on the demo app :slightly_smiling_face:. Would be great to see an example.