Guide: How to use Meteor with Postgres 😮

[satya]

I’ve been using Meteor for many years, but recently had a need to use Postgres for a new project. So I decided to make it work with Meteor. It’s actually quite easy. Here’s the full guide

https://medium.com/@satyavh/using-meteor-with-postgresql-1385ff630b20

[davideonmeteor]

Hi,
really congratulations on your project, it’s really very interesting.
Do you have planned to manage offline data for a Cordova based app ? Minimongo permit it, not in a full mode, but with some trick you can keep data also without network.

[perumalkuk]

Great and shows how meteor can be extended. Just wondering about your opinion about using Postgres in parallel without removing mongodb/minimongo(using it only for Accounts) …is it an overhead…or does it complicate things or does it cause other issues.

Thanks

[vikr00001]

It would be great to see PostGres-based accounts become an officially-supported Meteor option!

[vlasky]

Hello @satya,

I congratulate you on your successful use of PostgreSQL with Meteor. Although I don’t use PostgreSQL myself, I have much respect for it and take an interest in projects that combine it with Meteor.

I am the defacto maintainer of the Meteor MySQL integration (vlasky:mysql). Instead of using the Mongo oplog, the Meteor MySQL integration implements reactive queries and publications using the MySQL binary log as the event source, in a manner compatible with Mongo on the client side.

I strongly disagree with parts of your article disparaging the use of reactive publications and subscriptions in Meteor. I have quoted them at the end.

As I write this post, customers across Australia are tracking their food and alcohol deliveries in real-time using Meteor webapps developed by my businesses that leverage reactive MySQL publications to update maps and tables with order status updates as well exchanging instant messages between drivers and dispatchers.

We have found using publications to be more efficient and performant than having the client needlessly poll the server. They greatly reduce latency and the amount of data sent over the wire.

So why are some people having difficulty scaling Meteor apps that use publications?

1. Meteor’s reactive Mongo driver is missing a critical throttling feature - a minimum interval between successive query result set fetches.

Publications are most often used to update a UI component on the client side. If a collection is receiving many updates per second, it is pointless and wasteful to push data to the client for each one.

Instead, what if you could specify a minimum interval of say 1 second between updates? This would greatly reduce bandwidth use as well as CPU/IO overhead on the client and server.

This key performance-enabling feature has long been present in the Meteor MySQL integration and I have urged for it to be implemented in Meteor’s Mongo driver and also in redis-oplog.

2. Out of the box, Meteor’s server-side merge box stores a copy of the collection data for each user who subscribes to the publication.

As long as your server has enough RAM and your Node.js process is configured to use it by passing the --max-old-space-size argument on the node command line, you should not get into trouble.

Development efforts are already underway to provide finer control over the behaviour of the mergebox that I expect to alleviate this problem for good.

PR#11368 - Support for publication strategies in livedata server
PR#11151 - Initial support for publication strategies in livedata server

This is a technology that was very cool when Meteor launched, and it has its uses. But it’s also the main bottleneck for Meteor’s performance. It adds load to the server, and to the database. And honestly, real-time can be solved in many ways these days.

As a rule of thumb: don’t use Meteor’s publications, they will always cause you headaches. Simply use websockets (or Meteor’s DDP) to inform the client of changes, and then have the client fetch data by API.

Don’t use Meteor’s publications, it’s useless by default due to performance implications and will always cause you headaches.

In other words, Meteor must stay tightly coupled to Mongo, because of its real-time publications offering. But that technology is quite useless by default due to its performance implications, and therefor should not be used.

[vikr00001]

@satya, the first part of your article, describing the benefits of Meteor, could be also be published independently. It would be a great resource for those looking for the latest info.

Also, have you considered possibly doing a pull request containing your PostGres-based accounts as an optional account system for Meteor? That way the community could take a look at it and vet it for potential official inclusion. Just a thought!

[satya]

First thanks and great to get your input/feedback!

These 2 were serious issues with Publications that we ourselves ran into multiple times, so we decided never to use Publications again. At a certain point in time servers will run to 100% CPU or 100% memory.

Memory Leak - App crash every day

Hi all, I could use some help on this memory leak issue. I’m struggling to find out where the memory leaks come from in my app. My containers (standard size on galaxy) are crashing every day… I took screenshots of the past 24 hours for one instance, (same instance for all screenshots) and hopefully someone here will see something obvious that I’ve unfortunately missed so far. [05] [55] [43] [39] [47] I’ve hunted global variables and set them to local, but it’s possible that I missed…

Memory Utilization is running high

Perhaps this issue:

But, perhaps the issue is fixed - although git still reports it

[1.6.1] 100% CPU on Ubuntu server, caused by client?

[mrrk]

I’m curious, if you’re not interested in the Mongo integration and reactive data layer, which parts of Meteor do you like?

[mrrk]

@vlasky

We have found using publications to be more efficient and performant than having the client needlessly poll the server. They greatly reduce latency and the amount of data sent over the wire.

But this isn’t publications per se, right? It’s just a benefit of using websockets. In fact if your client doesn’t support websockets you are needlessly polling the server.

My understanding of the OP’s criticism is that it’s not of the “push changes over the wire” model, but the way that Mongo detects those changes and pushes them to the right clients (oplog tailing, mergebox) which makes them inefficient and hard to scale.

[satya]

The answer is in my article

[vlasky]

We are comparing the performance impact of polling for updates by performing periodic Meteor method calls with the performance impact of sending updates using Meteor publications/subscriptions.

Both operate over WebSockets unless you are in running an an environment where WebSockets are unavailable and long polling mode is then used as a fallback.

The first advantage with publications is that the server can send to the client at any time of its choosing, ensuring minimal latency, set by the reactivity response time. In contrast, when performing periodic Meteor method calls, the latency is determined by the polling period plus the method completion time.

The second advantage with publications is that they avoid all that recurring Meteor method-related data going over the wire and the server-side CPU/IO load required to service the method call - an unavoidable consequence of polling.

The disadvantages with publications are the need for the server-side code to consume RAM for the merge box and the overhead of following the MongoDB oplog or MySQL binlog. I have already discussed the remedies for these two issues.

[mrrk]

Oh my apologies, it’s literally in a list of bullet points.