Looking for collaboration on a modular meteor server/client

community
1

Hello everyone,

TLDR; Planning to create a standalone meteor-rpc package so you can migrate to a modern node app structure. Looking for interested parties so the result is something that can benefit the Meteor community.

For over a decade now, I’ve been involved in a writing & maintaining a Meteor-based application (since Meteor’s v0.5.5 days). Through the years, I’ve observed how the trio of Meteor’s core components – the build tool, data synchronization, and accounts – can lead to a certain lock-in effect. Moving away from one component becomes challenging due to dependencies on the others.

In light of the rich array of build tools within the Node.js ecosystem, there’s immense value for us in crafting a modular version of Meteor’s data synchronization, encompassing subscriptions and RPC.

In fact, we’ve already taken the initiative in our own application. By replacing Meteor’s build tool with vite for the frontend and rollup for the server, we’ve made significant strides. However, there are several design choices in the structure of meteor’s codebase that hold this approach back.

Our goal involves the creation of an independent RPC and subscriptions library. Drawing inspiration from Meteor packages like ddp, minimongo, mongo, and redis-oplog, we’re aiming to deliver a solution that isn’t solely focused on api compatibility, but rather on upholding Meteor’s subscription and method RPC call contracts (Optimistic cache updating).

I’m putting this out to the community to see if there’s enough interest to establish a working group. Perhaps the final design can be useful to more than just one application.

As a point of conversation, this is my “shopping list” for writing a new meteor-rpc client/server:

Keeping:

  • MongoDB
  • Redis-Oplog style change notifications
  • Client side method-stubs with Optimistic cache updates

Changing:

  • Typescript / es modules
  • No Fibers or AsyncLocalStorage
  • No oplog driver
  • No globals or import-time side-effects
  • BYO websocket server & client
  • BYO Auth
7 Likes
2

Hi @nathan_muir ,

I’m writing because I think this looks like something potentially interesting, but it’s hard for me to get a clear picture what would be left of Meteor when using your modular server/client.

I’m very gruk-brained, could you draw / write me an example, maybe as a few → word → diagram or somesuch, as to how this could be used / what possible use-cases could look like?

I think it might be good to give a full example or two so others can also grok / gruk easily whether this would be something potentially useful / valuable for them and / or the community & because I think your Idea should probably have the chance to get a bigger audience this way.

3

Hi Daniel,

This isn’t the final API i’m going for, but just to illustrate.

The first point of change, would be ensuring that nothing happens at import time, and you have to start up the Meteor client & server somewhere in your codebase.

If you were looking for a blueprint, it would be something along the lines of tRPC. (we could even build meteor-rpc atop tRPC). It would also take inspiration from successful libraries like TanStack Query

In your app, startup your own meteor server which handles your publications & methods; You then configure whatever you like on top.

import http from 'node:http';
import { createServer } from '@meteor/server';
import { createAccounts } from '@meteor/accounts';
import { myMethods, myPublications } from './some-path';

const MyServer = createServer({
  mongoUrl: process.env.MONGO_URL,
  redisUrl: process.env.REDIS_URL,
});
const Accounts = createAccounts(MyServer);

MyServer.publish(myPublications);
MyServer.methods(myMethods);

const httpServer = http.createServer();
MyServer.attachTo(httpServer);

On the client, you can think of Meteor’s data layer as two parts:

  1. A regular RPC api, eg. const response = await Meteor.call('someMethod')
  2. Populating a client-side cache (via Meteor.subscribe('someData'); which we optimistically update before an RPC call.

You could shape however you like, eg, integrating with something like TanStack Query.

Your components that need to read / update data from meteor could look something like this:

const meteor = useMeteor(); // starts the client / connects to server

// equiv. Meteor.subscribe(), but could work with SSR
const subscription = useSubscription(meteor, 'myData', prop.id);
const records = useCursor(
  meteor.db.myCollection.find({ complete: false }, { fields: { name: true } }),
);
const mutation = useMutation(args => meteor.rpc.myCollection.update(args));

function onSubmit() {
  mutation.mutate(/*...*/);
}
2 Likes
4

Why not build it on top of Simpleddp, there is also library to handle the sever. I have this this setup on one project, running on express, webpack. I was looking into porting it to Deno.

5

Hi @alawi

I tried searching for a server library for simpleddp but came up short.

Can you link to the github / npm package?

6

There nothing mature, I basically forked this ddp-server - npm. Then used passport for authentication and exposed methods, they were just JS functions handling DDP events. But I guess that is what you to create.

7

The question is how will this be version controlled and derived from the meteor codebase? I’ve worked on something just like this before, basically extracting the meteor client bundle into a js file for use in non meteor projects such as react native projects. I was thinking it sure would be great if I just made this a package on npm but have yet to do it. Official meteor client in react native

1 Like
8

You’d look after Grapher Documentation in order not to reinvent the wheel…

9

@leonardoventurini Can helene be of any help?

10

Yes @harry97 @nathan_muir it’s fully event driven client-server/server-server real-time, you can build anything Meteor has on it; I even added a “minimongo” equivalent, we could even build a live query pub/sub mechanism now. I could demo it to you privately if you want.

1 Like
11