Solid chat solution

help
1

I’m planning on moving chat functionality from our core app to a microservice avoiding future oplog issues and better scaling. I have some options in mind but not sure what would be easier/safer/faster:

  • Streams: It’s what RocketChat uses but doesn’t seem to fit my plan as I need storage. I want to avoid mongodb + oplog tailing + meteor issues. Also, Streams is dead so it’s a bit risky to create a fork.
  • RethinkDB + Meteor: Sounds fine as I’m already using Meteor and RehinkDB support seems fine for such a simple funcionality (It will just store conversations and messages).
  • RethinkDB + Phoenix: Never used Phoenix so this is really crazy but seems like a good option to get an stable solution for the future. Hard but fast.

Any “external” app either meteor or Phoenix would require a way to send that data to the main meteor app. Is there any way to feed a publication from another source?. How can I make that source reactive?

I’m now I’m asking “big” questions but just want to share some ideas and know what other options I have. I’m scared that we could hit the oplog tailing cpu issue fast due to chat functionality.

2

Creating microservices with Meteor is easy!

http://docs.meteor.com/#/full/ddp_connect

var remoteConnection = DDP.connect("http://my.url.com")

remoteConnection.subscribe("messages");

When you have the chat on a separate db, you would get double the headroom for oplog scaling, as you say. When / if you hit that limit, there are ways to mitigate the issue, but it would naturally require more engineering. It would of course depend on the scale we are talking here. With proper homework done, it is possible to scale to tens of thousands of concurrent users. More than that though, and you would probably need to look at alternatives, or do even more crazy engineering.

1 Like
3

Maybe the easiest plan is creating more microservices and then oplog limit would be really far. With this ddp microservice approach it would be easy to observe that dpp events (added,updated,removed) and send them to client inside a publish function no?

4

Look into RabbitMQ or MQTT for your solution. There are packages on atmosphere for both.

1 Like
5

But why should I use RabbitMQ or MQTT if I already have a ddp connection? Or this reply is to communicate Phoenix with Meteor?

thanks for your help :smile:

6

DDP is a protocol tailored towards reactive updates of databases.

Messaging frameworks are tailored towards messaging. MQTT is in fact designed for IOT telemetry. ie, lightweight.

You won’t as @jorgeer says need to worry about mitigating or doing crazy engineering to scale it as there is no pesky oplog tailing involved. Using a database as a message broker is just asking for pain.

Right tool for the job.

7

Hmm, and how can I publish mqtt messages over ddp? I use ddp for web and iOS apps.

8

https://atmospherejs.com/perak/mqtt-collection

That package appears to integrate well, speaking to an MQTT message broker via a collection. There are 4 other MQTT packages when I last looked.

Other than that it should be easy to engineer yourself. DDP sits on top of TCP. HTTP sits on top of TCP. MQTT sits on top of TCP.

Just open a socket.

1 Like
9

Thanks @jacobin!

That package seems to be perfect for what I need :smile: