New Article Advocating for Sync Engines (a la Meteor Live Data)

meta
1

Linked today on HackerNews: “Sync Engines are the Future”

…getting data is a continuous process: data changes over time and becomes stale, requests fail, updates arrive later than you might’ve wanted, or out of order. Errors will happen. Occasional if (!response.ok) will not get you very far.

fetch(new Request('/user/update', { method: 'POST' })).then((response) => {
  if (!response.ok) {
    // Do what? Pretend it never happened?
    // Stop the entire application?
    // Retry? What if user already issued another update that invalidates this one?
    // What if update actually got through?
  }
});

And you can’t just give up and declare everything invalid. You have to keep working. You need a system. You can’t solve this problem at the level of single request.

Meteor Live Data is relevant here. Is Meteor the premiere sync-engine framework?

See previous discussion on this topic on this thread.

2 Likes
2

Let’s be honest, the Meteor approach is a bit dated and could use an upgrade, but in general I think it would be a good start.

1 Like
3

Go on…

What don’t you like about it? Where do you see room for improvement?

4

A few from the top of my head:

  1. Batching of the DDP messages (it would require a new DDP protocol version + backward compatibility layer).
  2. Support for merging of nested fields, i.e., being able to send only the deeply nested fields once they change (instead of the entire top-level field).
  3. Remove the updated DDP message as it can be inferred from result (again, a new protocol version).
  4. Add “streaming methods”, i.e., a method that can return multiple times (think: generators in code, progress bar in the UI).
  5. Document the cleared field in the added DDP message (it’s not in the docs but Meteor does send it).
4 Likes
5

Not an expert (and honestly should look more into it, but @radekmie sums it up well), but in beyond that in general it needs a bit of cleaning of the dust and review & documentation. Potentially to spin it out into its own independent NPM package (though that might be difficult), if possible to attract more attention to it. But I think the underlying principles are solid.

1 Like
6

This one seems like a very fun task – how have you imagined it? I think using updated would be an option, Meteor.stream(“method”) → yield on the backend until get a result message

7

From the top of my head, I see two options here: ambient and generator (both could use a better name). Both would work like current Meteor methods, i.e., method/result DDP messages with an additional progress message.

Ambient would be sent using the new progress method on method context. Example:

Meteor.methods({
  summarizeInvoices(userId) {
    let total = 0;
    for await (const invoice of Invoices.find({ userId })) {
      total += invoice.getTotal();
      this.progress(total); // New API.
    }
    return total;
  },
});

Generator would use generators instead.

Meteor.methods({
  *summarizeInvoices(userId) {
    let total = 0;
    for await (const invoice of Invoices.find({ userId })) {
      total += invoice.getTotal();
      yield total; // New API.
    }
    return total;
  },
});

On the client it could use an additional callback:

Meteor.callPromise('summarizeInvoices', userId, {
  onProgress(total) { // New API.
    render({ total, status: 'progress' });
  },
}).then(total => {
  render({ total, status: 'done' });
});

And yes, it could be wrapped into an async iterator:

for await (const total of Meteor.callStream('summarizeInvoices', userId)) {
  render({ total }); // Using `for..of` does not tell us if it's the last element.
}

On top of that, both could make the server part resumable:

  • this.progress could return a Promise<boolean> indicating whether the client wants to continue.
  • yield could return boolean directly (i.e., no Promise needed).

This would require the client to send something, e.g., resume with the method ID. It could be fallible, e.g., require the user to reply within some time frame.

3 Likes
8

That sounds cool af – I’ll check if I have time to play with this idea and try coming up with a PR

1 Like