Pub/sub bypassing MongoDB?

I’m only finding very old threads on this subject, but I’m wondering the best way to go about this? For example, for a chat application? Reactivity with the database involved would only slow it down and there’s no reason for us to save communication between users.

I also saw some mention that Redis-Oplog should be capable of doing this. I’d assume this method would be optimal. But couldn’t find information as to how or through what feature that would work?

Hi @Spyridon,

there used to be a package, now most probably outdated tech, but I still find the documentation relevant: Meteor Streams - Realtime Messaging for Meteor

Another way to go about it is to subscribe to 1 record and use device storage or a client DB extensively.
For example, in a chat window, let’s say you have 20 initial messages (the newest). You only subscribe to the date of the last message and if it changes you pull the difference with a method and drop it in the local storage. Things like Contacts you may persist in the cache for longer together with a date of the last contacts update. You subscribe to the date of the last changes. In general, you don’t subscribe to your data that goes in Views but to flags, triggers that show you that some data has changed. On trigger that I use extensively is the number of new notifications. I keep that on the User and I pull that data within one of the very few subscriptions I use.
In general, I personally use subscriptions to see if data has changed (you can use Observers for that too) and not to pull data. Data comes via methods.

2 Likes

Are you referring to Synthetic Mutation?

redis-oplog/finetuning.md at master · cult-of-coders/redis-oplog · GitHub (bottom of the page)

You can use any driver for pub/sub although it’s really working best with Mongo it seems. I bypass it because it’s slow and use memcache and server-render packages more

For chat specifically you would find it easiest using a protocol like webrtc which then you can have video and audio chat also - the beauty of meteor is it’s all javascript and can run npm packages easily so you can just take anything from npm

I think rocket chat does this.

They use GitHub - RocketChat/meteor-streamer: 2 way communication over DDP with better performance.

Thank you for the reply!

I saw the streams package before, but I’m honestly a bit reluctant to add a package that isn’t really maintained anymore.

Thank you for the suggestion of that strategy using subscriptions! I had considered something similar, but not using a date, and that sounds like it would work nicely for optimizing subscriptions!

Possibly! To be honest, I only saw others discussing it, so I don’t know exactly what the feature is called, lol.

One concern I have is since this is through the redis-oplog, it may not work through the development server, which may make testing/implementation difficult.

@truedon and @alawi - Thank you for the suggestions! My only hesitance is same as I mentioned earlier, the packages seem a bit outdated, and I’m doing some cleanup with the newer version of our software, so I’m a little reluctant to bring in any packages that may not be maintained.

Not sure what you meant by this? We are developing with redis-oplog in our respective machines and test servers with an instance of redis running using docker

“but I’m honestly a bit reluctant to add a package” - I really hope you can’t :slight_smile: given its age. However, Streams is something I am planning to get more into. This is a for-free technology that can save many thousands a month by not needing to involve DB or DB layers (like Redis) in a large infrastructure.

I really don’t think it would be an issue, assuming rocket chat is using it.

Beside DDP has been very stable, I don’t think any update were required for those packages. But I don’t use it so you have to check for yourself.

I have packages that are years old and still working perfectly, and as long as I understand the code and can fork it, I keep using it.

Here is an example of how you can use publish and subscribe without interacting with a database.
In this case I will also ignore the need to have a client-side collection, a simple object will be used instead, a collection name and an id are informed because of the API rules, don’t create it.

An EventEmitter came up by chance, use whatever font you think best.
It would be better to use Meteor.call(‘newMessage’ {to, message}) and use Meteor’s internal routines to access the active connections, but I don’t remember now, but they do exist.

Some adjustment to this code needs to be done, it was a memory I had along with something ready-made I had when I read your question here on the forum, English is not my language and I’m not on a computer that allows me to test, but the basic idea are you there

/server/publish.js

import EventEmitter from 'events';
import { Meteor } from 'meteor/meteor'

const chats = {};

const chatEvents = new EventEmitter();

chatEvents.on('newMessage', ({from, to, message}) => {
  chats[to].added(
  'chatMessages', // a collection name
  someUniqueId, // id
  {from, message}
  );
})

Meteor.publish('chat', function(name) {
  chats[name] = this;
  this.ready();
});

/client/subscribe.js

import {Meteor} from 'meteor/meteor'

const obj = [
  'update', //gotcha!
  'beginUpdate',
  'endUpdate',
  'saveOriginals',
  'retrieveOriginals',
  'getDoc',
  '_getCollection',
].reduce((a, d) => {
  a[d] = (...args) => {
    console.log(d, args);
  };
  return a;
}, {});

obj.update = ({ msg, collection, id, fields }) => {
  switch (msg) {
    case 'added':
      console.log('A new message!', fields) // {from, message}
    break;
    default:
      console.log('other event', { msg, collection, id, fields });
  }
};

Meteor.subscribe('chat', MyUniqueName, function(){
  this.connection.registerStore('chatMessages', obj);
});

2 Likes