šŸš€ Meteor 3.5 (RC Available): Change Streams & Performance improvements

@nachocodoner

I fixed login bug by adding router fallback:

1 Like

This is the purpose of the beta: to test and find bugs in real projects before an official release. I’m on vacation until the 19th, so I’ll review it soon.

btw thank you very much wekan for using that beta. We appreciate this kind of contribution a lot.

1 Like

Previous change did not fix Login bug, so this one did fix it:

My two cents on this is that Wekan had lots of lingering bugs that weren’t really attended to and were brewing underneath. Once the migration was done, it got exposed so some of these aren’t really the beta problems.

Nonetheless, the enthusiasm and having a big project like Wekan test out these betas is an amazing benefit to Wekan itself and the broader Meteor community!

4 Likes

I had some problems with Snap, so I migrated to Docker. Here is how I run Meteor 3.5-beta, Node.js 24.x, MongoDB 7.x with changeStreams and uws at Hetzner bare metal server that has 64 GB RAM, 2x1TB NVME RAID, Ubuntu 26.04, Linux kernel 7,x, and over 14k users. (MongoDB 8.x did not work, it causes some attachment upload and disappearance bugs).

Next interesting bug. At my WeKan hosting, every browser page reload does load webpage of different customer. This is the free hosting login page, but sometimes there is different custom logo login page:

https://boards.wekan.team/sign-in

I tried to make to not cache at Caddy config, but it did not help:

Now I turned off CloudFlare caching with Development Mode. I’ll check does it help.

It looks like it did not help. Well, I’ll think of something else.

I fixed this multitenancy bug by changing from changeStreams uws to oplog sockjs and some other settings.

1 Like

@xet7 can you please provide some guidance of how to reproduce it, please? so I can investigate what is happening

1 Like

@italojs

Bug description:

Current working config. If it is changed to changeStreams uws, it breaks:

It was a little bit hard to reproduce and debug but now I have a propose fix.

The root cause come from the interaction of two bugs in the ddp-server/transports/ package of Meteor 3.5-beta.10 when multiple processes (or tenants) share the same network namespace. First, the uws internal port configuration silently fails because the code attempts to read settings from a client-side object (__meteor_runtime_config__.meteorSettings) that is never populated on the server, thus ignoring user configurations and always forcing the default port (5001). Second, since the uWebSockets.js library uses the SO_REUSEPORT option by default, multiple different Meteor instances can successfully bind to the exact same port (127.0.0.1:5001) without throwing any collision errors. As a result, the Linux kernel randomly load-balances and routes incoming WebSocket connections across the different processes, inadvertently mixing one tenant’s requests into another tenant’s database.

Here you can find a PR with the fix where I will land in the next beta, where you will be able to test it again.

4 Likes

Reading this code is a little bit confusing. Meteor already uses Express under the hood. But it seems like this code is creating a ā€œnewā€ express with the express() call.

Does this mean this code is telling Meteor’s existing express (which we extend with Meteor.handlers) to also use the new Express server, so we have two? Does it replace Meteor’s with a new one? Or what exactly?

Btw the docs are very lacking on these features, and because of it AI is making wrong assumptions or hallucinating.

I would greatly appreciate for all Meteor features to be released with complete docs, so we don’t have to dig into source code, and so AI won’t create random junk when we ask it to do something.

I think documentation is super pertinent in the AI era, otherwise we end up with super slop, ultra crazy unnecessary workarounds, or stuff that just doesn’t work.

It is also not clear what the difference is between the above example and the following which does authentication checks without a new express() being called:

import { WebApp } from 'meteor/webapp';
import { createAuthMiddleware } from 'meteor/accounts-express';

// Apply auth middleware to /api routes
WebApp.handlers.use('/api', createAuthMiddleware());

// Protected endpoint under /api
WebApp.handlers.use('/api/users', async (req, res) => {
  if (!(await Meteor.userAsync()).profile.isAllowedToDoSomething) {
    // ...block action, return 500 error to client...
    return
  }

  // ... proceed with authenticated user ...

})

Have you had a chance to check the original post from @italojs? It already included the link to the docs (press ā€œDocumentationā€ link): https://github.com/meteor/meteor/blob/release-3.5/v3-docs/docs/packages/accounts-express.md. This Markdown file will be officially available in Meteor docs once Meteor 3.5 final is released, and it includes what is needed to understand and use the package.

It is also accessible in the Meteor 3.5 beta preview docs: accounts-express | Docs.
More info in the dedicated forum post: Expose Meteor user context to Express endpoints, where we also cared to update the first post to describe the final approach and usage.

Remember this is still a beta. While we have provided the docs needed to use the feature, they are not available yet in the official Meteor 3 docs. So AI tools will not always have that context by default. For new beta features, it helps to provide the links and information shared in the Meteor 3.5 highlights and in the dedicated forum post for this addition.

It is also not clear what the difference is between the above example and the following which does authentication checks without a new express() being called

That’s right. The example above is a high-level one, meant to show the general idea of getting an Express instance to use the new endpoints. But Meteor can also reuse its own Express instance.

In the docs we prepared and linked, and in the forum post, there is also an example focused on Meteor’s built-in Express instance. That is referenced as the default documented approach.

Of course, docs need visibility from our side, and we tried to share all the materials we prepared. After a re-read focused on them, do you think they cover the main cases?

Where do you think we could improve?

:rocket: Meteor 3.5-beta.12: Bug fixes only.

Hello everyone! A new beta of Meteor 3.5 is out, 3.5-beta.12. This is mostly a stability/maintenance beta with a few useful improvements and an important alignment of the Change Streams MongoDB requirement.

:test_tube: Experimenting with 3.5-beta.12

Create a New App


meteor create my-app --release 3.5-beta.12

Update Your App


meteor update --release 3.5-beta.12

:hammer: MongoDB 6+ requirement is now enforced for Change Streams

Since the very first beta we documented that Change Streams need MongoDB 3.6+ on a replica set or sharded cluster. The code, however, we detected actually it requires MongoDB 6+.

In beta.12 the check now matches the docs: only MongoDB 6+ is accepted for Change Streams.

What this means for you:

  • :green_circle: MongoDB 6+ → nothing changes, you get all the Change Streams benefits

  • :yellow_circle: MongoDB < 6 → Meteor automatically falls back to oplog (or polling). No action required, your app keeps working — you just won’t be on the Change Streams driver

  • :page_with_curl: Updated docs

If you want to explicitly pin to oplog regardless of Mongo version, drop this in your settings.json:


{

"packages": {

"mongo": {

"reactivity": ["oplog", "polling"]

}

}

}

:sparkles: Highlights since beta.11

:green_circle: Node.js 24.15.0 & NPM 11.12.1 (#14399)

Beta.12 ships Node.js 24.15.0 and NPM 11.12.1, picking up the latest stability and security patches in the Node 24.x line. Big thanks again to @StorytellerCZ for keeping the runtime current. Unfortunately we will include 24.16+ at 3.5.1 due red CI when bump the node version.

:rocket: uWebSockets.js v20.66.0 in ddp-server (#14330)

For users on the new uws DDP transport, we bumped uWebSockets.js from v20.58.0 to v20.66.0 — improved stability and a few upstream fixes baked in. Thanks to @dupontbertrand for the bump.

:hammer_and_wrench: Fix uws transport settings & port collisions (#14425)

Fixed a bug where the uws DDP transport would mishandle settings and could cause port collisions in some setups (especially CI / test-packages). If you hit weird ā€œaddress in useā€ errors with uws, this beta should clear them up.

:memo: TypeScript type declarations for accounts-express (#14433)

Added proper TypeScript type declarations for the new accounts-express package — much nicer DX when wiring up authenticated REST endpoints from a TS app. Thanks to @nachocodoner.

:e-mail: email package dependency refresh (#14028)

Refreshed the email package’s npm dependencies to the latest stable versions — keeps the transport layer current and patches a couple of transitive CVEs. Thanks again to @StorytellerCZ.

Other PRs

Everything that’s landed on the 3.5 line so far is in the Release 3.5 milestone.

:building_construction: Coming next

We’re closing in on RC. A few items still in motion:

  • More uws transport battle-testing — please report anything weird

  • Continued docs polishing for the new Change Streams / DDP transport pages

:handshake: Big Thanks to Our Contributors

Featured in beta.12:

Plus everyone who tested previous betas and reported issues — your feedback is what’s getting us to a stable 3.5.0. :pray:

Your Feedback

If you’re already on a 3.5 beta, just run meteor update --release 3.5-beta.12. Areas where I’d especially love feedback for this beta:

  • :yellow_circle: MongoDB < 6 users: please confirm you transparently fall back to oplog/polling (no errors, no panic, your app keeps reactivity)

  • :rocket: uws transport users: please retest reconnection and port-handling — that’s the area that got the most fixes

  • :memo: TypeScript + accounts-express users: try out the new types and let us know if anything is missing

9 Likes

With 3.5-beta.12 I did not get working changeStreams uws. So I’m back to using oplog sockjs.

Could you tell me exactly what you tried to set up? I’m going to give it a try today :+1: Just setup Uws with changeStreams ? Blaze ?

My app still works, do you know why it stops working?

cc: @harry97

Just for documenting proposal:

@xet7 as explained here, you should configure the multitenacy for uws first.

Your services use network_mode: ā€œhostā€, so every container shares the host’s network namespace. You’re giving each one a distinct PORT (3039, 3040, …), but that’s only the main HTTP por, the uws transport still tries to bind 127.0.0.1:5001 in every container, so the second one fails with address already in use. The listener uses an exclusive-port bind on purpose, so instead of silently splitting WebSocket traffic between unrelated apps it stops with a loud error.

The uws port can only be set via Meteor.settings, so in Wekan you pass it through METEOR_SETTINGS with a distinct port per container:

  customer1:
    environment:
      - DDP_TRANSPORT=uws
      - PORT=3039
      - METEOR_SETTINGS={"packages":{"ddp-server":{"uws":{"port":5001}}}}

  customer2:
    environment:
      - DDP_TRANSPORT=uws
      - PORT=3040
      - METEOR_SETTINGS={"packages":{"ddp-server":{"uws":{"port":5002}}}}
      

today the uws port is read only by settings.json, at 3.5.1 I’ll include a env for it

So: distinct PORT and distinct uws.port, paired per instance. This wasn’t documented, will include it soon

here is my local test docker-compose

# =============================================================================
# Multitenancy bug reproduction stack
# =============================================================================
# Purpose: reproduce the uws + changeStreams cross-tenant data leakage bug
# described in docs/Platforms/FOSS/Docker/Meteor3/multitenancy.md
#
# Two Wekan instances share the LinuxKit VM kernel network namespace
# (network_mode: "host"), both connected to the same MongoDB replica-set,
# both using DDP_TRANSPORT=uws + METEOR_REACTIVITY_ORDER=changeStreams.
#
# Usage:
#   docker compose -f docker-compose.multitenancy.yml up -d
#   open http://localhost:8081   # tenant 1
#   open http://localhost:8082   # tenant 2
#
# (Different ports = different browser origins, so cookies/localStorage stay
# isolated between tenants — no /etc/hosts entries needed.)
#
# Tear down:
#   docker compose -f docker-compose.multitenancy.yml down -v
# =============================================================================

services:

  # ---------------------------------------------------------------------------
  # Shared MongoDB 7 with replica set (required for changeStreams)
  # Bound only to 127.0.0.1:27018 on the LinuxKit VM host netns.
  # ---------------------------------------------------------------------------
  wekan-db-mt:
    image: mongo:7
    container_name: wekan-db-mt
    restart: unless-stopped
    network_mode: "host"
    entrypoint: ["sh", "-c"]
    command:
      - |
        mongod --storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --timeZoneInfo /usr/share/zoneinfo --oplogSize 2048 --replSet rs0 --bind_ip 127.0.0.1 --port 27018 --quiet &
        until mongosh --host 127.0.0.1 --port 27018 --quiet --eval 'try { const s = rs.status(); quit(s.ok === 1 ? 0 : 1); } catch (e) { if (e.codeName === "NotYetInitialized" || /no replset config has been received|not yet initialized/.test(e.message)) { rs.initiate({_id:"rs0", members:[{_id:0, host:"127.0.0.1:27018"}]}); quit(1); } quit(1); }' >/dev/null 2>&1; do
          sleep 2
        done
        wait
    volumes:
      - wekan-db-mt:/data/db

  # ---------------------------------------------------------------------------
  # Tenant 1: ROOT_URL http://tenant1.local:8081
  # ---------------------------------------------------------------------------
  wekan-tenant1:
    image: wekan-beta12-fix:latest
    container_name: wekan-tenant1
    restart: unless-stopped
    network_mode: "host"
    depends_on:
      - wekan-db-mt
    environment:
      - PORT=8081
      - ROOT_URL=http://localhost:8081
      - MONGO_URL=mongodb://127.0.0.1:27018/wekan_tenant1?replicaSet=rs0&appName=wekan-tenant1
      - MONGO_OPLOG_URL=mongodb://127.0.0.1:27018/local?replicaSet=rs0&appName=wekan-tenant1-oplog
      - METEOR_REACTIVITY_ORDER=changeStreams,oplog,polling
      - DDP_TRANSPORT=uws
      # Distinct uws internal port per tenant — required since the
      # 3.5-beta.12 fix (PR #14425) binds with LIBUS_LISTEN_EXCLUSIVE_PORT.
      - METEOR_SETTINGS={"packages":{"ddp-server":{"uws":{"port":5001,"host":"127.0.0.1"}}}}
      - WRITABLE_PATH=/data
      - WITH_API=true
      - RICHER_CARD_COMMENT_EDITOR=false
      - CARD_OPENED_WEBHOOK_ENABLED=false
      - BIGEVENTS_PATTERN=NONE
      - BROWSER_POLICY_ENABLED=true
      - TRUSTED_URL=
      - WEBHOOKS_ATTRIBUTES=
      - OAUTH2_ENABLED=false
      - DEFAULT_AUTHENTICATION_METHOD=password
      - PASSWORD_LOGIN_ENABLED=true
      - WAIT_SPINNER=Bounce
    volumes:
      - wekan-tenant1-data:/data

  # ---------------------------------------------------------------------------
  # Tenant 2: ROOT_URL http://tenant2.local:8082
  # ---------------------------------------------------------------------------
  wekan-tenant2:
    image: wekan-beta12-fix:latest
    container_name: wekan-tenant2
    restart: unless-stopped
    network_mode: "host"
    depends_on:
      - wekan-db-mt
    environment:
      - PORT=8082
      - ROOT_URL=http://localhost:8082
      - MONGO_URL=mongodb://127.0.0.1:27018/wekan_tenant2?replicaSet=rs0&appName=wekan-tenant2
      - MONGO_OPLOG_URL=mongodb://127.0.0.1:27018/local?replicaSet=rs0&appName=wekan-tenant2-oplog
      - METEOR_REACTIVITY_ORDER=changeStreams,oplog,polling
      - DDP_TRANSPORT=uws
      # Distinct uws internal port per tenant — required since the
      # 3.5-beta.12 fix (PR #14425) binds with LIBUS_LISTEN_EXCLUSIVE_PORT.
      - METEOR_SETTINGS={"packages":{"ddp-server":{"uws":{"port":5002,"host":"127.0.0.1"}}}}
      - WRITABLE_PATH=/data
      - WITH_API=true
      - RICHER_CARD_COMMENT_EDITOR=false
      - CARD_OPENED_WEBHOOK_ENABLED=false
      - BIGEVENTS_PATTERN=NONE
      - BROWSER_POLICY_ENABLED=true
      - TRUSTED_URL=
      - WEBHOOKS_ATTRIBUTES=
      - OAUTH2_ENABLED=false
      - DEFAULT_AUTHENTICATION_METHOD=password
      - PASSWORD_LOGIN_ENABLED=true
      - WAIT_SPINNER=Bounce
    volumes:
      - wekan-tenant2-data:/data

volumes:
  wekan-db-mt:
  wekan-tenant1-data:
  wekan-tenant2-data:

is it requires separate web socket port for Caddy for each tenant?

Caddy doesn’t need a separate WebSocket port per tenant. The uws.port in the new docker-compose-multitenancy.yml example is internal to each Wekan process, caddy never connects to it.

There are two distinct ports involved per tenant, and they’re easy to confuse:

Port Scope Who connects to it Distinct per tenant?
PORT (e.g. 3039, 3040, …) Public HTTP port the Meteor process binds The browser, via Caddy’s reverse_proxy 127.0.0.1:PORT Yes (you already do this, reverse_proxy 127.0.0.1:3039 for tenant1, :3040 for tenant2, etc.)
uws.port in METEOR_SETTINGS (e.g. 5001, 5002, …) Internal proxy port uWebSockets.js binds inside the Wekan process on 127.0.0.1 Meteor’s own main HTTP server, locally, via net.createConnection — never reachable from outside the container Yes (new requirement in Meteor 3.5 multi-process deployments)

The flow for an incoming wss://wekan.customer.com/websocket upgrade:

Browser ──HTTPS WS upgrade──▶ Caddy ──HTTP /websocket──▶ 127.0.0.1:3039
                                                              │  (tenant1's PORT)
                                                              ā–¼
                                                       Wekan tenant1
                                                          (Meteor main http.Server
                                                           on :3039)
                                                              │
                                                              │  internal proxy
                                                              ā–¼
                                                       127.0.0.1:5001
                                                          (tenant1's uws.port — 
                                                           same process, loopback only)

Caddy only ever sees 127.0.0.1:3039. The 5001 is an implementation detail of how Meteor’s uws transport bridges its own HTTP server to the µWebSockets.js engine in the same process. Each tenant’s uws.port lives inside that tenant’s container and is bound to 127.0.0.1 of the LinuxKit / host netns, never published outside.

So your existing Caddyfile pattern is correct as-is. The only thing that needs to be distinct per tenant on the Caddy side is the upstream reverse_proxy target, which is the PORT env var
(3039, 3040, …), and that was already distinct in your example.

so…

Same pattern you already have, just one upstream port per tenant:

wekan.customer1.com {
    # …CSP and headers…
    handle {
        reverse_proxy 127.0.0.1:3039     # tenant1's PORT
    }
}

wekan.customer2.com {
    # …CSP and headers…
    handle {
        reverse_proxy 127.0.0.1:3040     # tenant2's PORT
    }
}

The 5001 / 5002 / etc. uws.port values stay inside the
respective containers’ METEOR_SETTINGS. Caddy is unaware of them.

1 Like

:loudspeaker: Meteor 3.5-rc.1 is out!

Just a heads-up for everyone following this thread: the Release Candidate is live and feature-complete. :tada:

All the details: Change Streams on by default, the new pluggable DDP transport (uWebSockets), DDP Session Resumption, accounts-express, async DDPRateLimiter matchers, MongoDB Collation, the EJSON/DDP performance batch, Node.js 24.15.0, and the full list of fixes, are already covered in the main post above. :arrow_up:

:point_right: Give it a try:

meteor update --release 3.5-rc.1

This is the final checkpoint before 3.5 official, so the most valuable thing you can do right now is test your apps against 3.5-rc.1 and report anything unusual here in the thread. Your feedback is what gets us to a solid stable release. :pray:

Thanks again to everyone who contributed and tested along the way! :green_heart: :comet:

6 Likes