Guide: How to use Meteor with Postgres 😮

I’ve been using Meteor for many years, but recently had a need to use Postgres for a new project. So I decided to make it work with Meteor. It’s actually quite easy. Here’s the full guide

https://medium.com/@satyavh/using-meteor-with-postgresql-1385ff630b20

14 Likes

Hi,
really congratulations on your project, it’s really very interesting.
Do you have planned to manage offline data for a Cordova based app ? Minimongo permit it, not in a full mode, but with some trick you can keep data also without network.

2 Likes

Great and shows how meteor can be extended. Just wondering about your opinion about using Postgres in parallel without removing mongodb/minimongo(using it only for Accounts) …is it an overhead…or does it complicate things or does it cause other issues.

Thanks

1 Like

It would be great to see PostGres-based accounts become an officially-supported Meteor option!

3 Likes

Hello @satya,

I congratulate you on your successful use of PostgreSQL with Meteor. Although I don’t use PostgreSQL myself, I have much respect for it and take an interest in projects that combine it with Meteor.

I am the defacto maintainer of the Meteor MySQL integration (vlasky:mysql). Instead of using the Mongo oplog, the Meteor MySQL integration implements reactive queries and publications using the MySQL binary log as the event source, in a manner compatible with Mongo on the client side.

I strongly disagree with parts of your article disparaging the use of reactive publications and subscriptions in Meteor. I have quoted them at the end.

As I write this post, customers across Australia are tracking their food and alcohol deliveries in real-time using Meteor webapps developed by my businesses that leverage reactive MySQL publications to update maps and tables with order status updates as well exchanging instant messages between drivers and dispatchers.

We have found using publications to be more efficient and performant than having the client needlessly poll the server. They greatly reduce latency and the amount of data sent over the wire.

So why are some people having difficulty scaling Meteor apps that use publications?

1. Meteor’s reactive Mongo driver is missing a critical throttling feature - a minimum interval between successive query result set fetches.

Publications are most often used to update a UI component on the client side. If a collection is receiving many updates per second, it is pointless and wasteful to push data to the client for each one.

Instead, what if you could specify a minimum interval of say 1 second between updates? This would greatly reduce bandwidth use as well as CPU/IO overhead on the client and server.

This key performance-enabling feature has long been present in the Meteor MySQL integration and I have urged for it to be implemented in Meteor’s Mongo driver and also in redis-oplog.

2. Out of the box, Meteor’s server-side merge box stores a copy of the collection data for each user who subscribes to the publication.

As long as your server has enough RAM and your Node.js process is configured to use it by passing the --max-old-space-size argument on the node command line, you should not get into trouble.

Development efforts are already underway to provide finer control over the behaviour of the mergebox that I expect to alleviate this problem for good.

PR#11368 - Support for publication strategies in livedata server
PR#11151 - Initial support for publication strategies in livedata server

This is a technology that was very cool when Meteor launched, and it has its uses. But it’s also the main bottleneck for Meteor’s performance. It adds load to the server, and to the database. And honestly, real-time can be solved in many ways these days.

As a rule of thumb: don’t use Meteor’s publications, they will always cause you headaches. Simply use websockets (or Meteor’s DDP) to inform the client of changes, and then have the client fetch data by API.

Don’t use Meteor’s publications, it’s useless by default due to performance implications and will always cause you headaches.

In other words, Meteor must stay tightly coupled to Mongo, because of its real-time publications offering. But that technology is quite useless by default due to its performance implications, and therefor should not be used.

13 Likes

@satya, the first part of your article, describing the benefits of Meteor, could be also be published independently. It would be a great resource for those looking for the latest info.

Also, have you considered possibly doing a pull request containing your PostGres-based accounts as an optional account system for Meteor? That way the community could take a look at it and vet it for potential official inclusion. Just a thought!

2 Likes

First thanks and great to get your input/feedback!

These 2 were serious issues with Publications that we ourselves ran into multiple times, so we decided never to use Publications again. At a certain point in time servers will run to 100% CPU or 100% memory.

But, perhaps the issue is fixed - although git still reports it

I’m curious, if you’re not interested in the Mongo integration and reactive data layer, which parts of Meteor do you like?

But this isn’t publications per se, right? It’s just a benefit of using websockets. In fact if your client doesn’t support websockets you are needlessly polling the server.

My understanding of the OP’s criticism is that it’s not of the ā€œpush changes over the wireā€ model, but the way that Mongo detects those changes and pushes them to the right clients (oplog tailing, mergebox) which makes them inefficient and hard to scale.

The answer is in my article :slight_smile:

We are comparing the performance impact of polling for updates by performing periodic Meteor method calls with the performance impact of sending updates using Meteor publications/subscriptions.

Both operate over WebSockets unless you are in running an an environment where WebSockets are unavailable and long polling mode is then used as a fallback.

The first advantage with publications is that the server can send to the client at any time of its choosing, ensuring minimal latency, set by the reactivity response time. In contrast, when performing periodic Meteor method calls, the latency is determined by the polling period plus the method completion time.

The second advantage with publications is that they avoid all that recurring Meteor method-related data going over the wire and the server-side CPU/IO load required to service the method call - an unavoidable consequence of polling.

The disadvantages with publications are the need for the server-side code to consume RAM for the merge box and the overhead of following the MongoDB oplog or MySQL binlog. I have already discussed the remedies for these two issues.

4 Likes

Oh my apologies, it’s literally in a list of bullet points. :grimacing:

Hi @satya

I’ve recently been concerned about the MongoDB license, and wondered if Meteor.js is still the right solution for my new projects because of the very limiting SSPL license, which makes hosters very scarce …

Are you still using your accounts implementation with Meteor after these years? Is it running well?

I also like that there’s a MySQL implementation, but my reason to consider other DBs is not that I love SQL, it’s just the license of MongoDB is concerning, and that of MySQL isn’t much better, as far as I know. That’s why Postgres is interesting.

Anyway, would love to hear people’s experience with using Postgres or another truly open source DB with Meteor in a production setting!

1 Like

Try FerretDB 2.0 with PostgreSQL. I will also try when I have some time.

1 Like

No, from my perspective Meteor has become irrelevant technology. All good things come to an end :wink:

1 Like

This sounds and looks amazing! Thanks a lot for sharing this.

Do you have this running for a production app? How’s the performance? Any compatibility issues?

I will definitely try this out myself also.

1 Like

That’s a pretty extreme statement - which isn’t a problem in my book, but it makes me very curious why you think Meteor has become an irrelevant technology; and what are you using now as an alternative to get an MVP up fast?

3 Likes

Do you have this running for a production app? How’s the performance? Any compatibility issues?

Not yet, because I need install newest version and test.

1 Like

No, from my perspective Meteor has become irrelevant technology.

Hehe, according to someone at Internet, everything is already dead.

Even when Meteor is using newest Node.js, and there are many large production apps running Meteor 3, like RocketChat.

I would also like to know, what you currently use instead. I like to try new stuff.

2 Likes

Arlight, I tried out FerretDB 2.2 (latest) locally (with a Meteor repo without much functionality, but more than a fresh installation).

It’s very easy to get started locally with Docker: GitHub - FerretDB/FerretDB: A truly Open Source MongoDB alternative

I instantly got Deadlock errors caused by the accounts-password package trying to create sparse indexes, which are not supported yet in FerretDB.

However, I was able to create a shim/patch for the functions that create indexes and published that as package here: The trusted source for JavaScript packages, Meteor.js resources and tools | Atmosphere
(follow the instructions in the README; especially, add the package before the accounts packages in .meteor/packages).

Will keep testing out FerretDB. Would be great to see other test it out as well.

3 Likes