What is the plan for RDBMS access, particularly postgres-packages?

1

I want to use Meteor to develop applications against Postgres databases. I am experimenting with postgres-packages, which a pretty good start but has some way to go to be usable in production.

There are a number of things I’d like to see:

  • lower overheads on notifications;
  • the ability to use my own PK fields (not just called id), including concatenated keys;
  • the ability to build collections from all relations, not just tables;
  • better update support ($inc, etc);
  • the ability to connect to multiple data sources (not just a single database that is opened when the library is loaded);

So, what level of commitment exists from Meteor and/or the wider community?

I am prepared to do a lot of this work myself, and to contribute it back but I’m not going to do this in isolation.

Maybe we should set up a new topic category for this, at least if there is any interest in Meteor on RDBMS.

2 Likes
2

Have you checked out this thread? An early look at SQL in Meteor

3

Before of all, please let me express my deep appreciation to anybody who involved in the development of Meteor, especially because it is Open Source project and it’s very kind and generous of you to develop a so useful free framework. Please don’t accept my words as a criticism, but only as my try to advance Meteor further.

I’m not the original poster, but from reading his message, I’m sure that he read that thread.
That thread looks quite dead, and the last activity was 5 days ago by Ben Green, who asked:
“Any news regarding this feature? Cheers.”.
Nobody replied.
If even the original developer of the PostgreSQL support is ignored when he asks about the status of his baby, then why bloodnok would be answered?!
So bloodnok started a new thread, and in several hours there are already two responses!

I feel that the community misses the point:
The number of developers who need SQL is huge, enormous.
And many of them are coming from the industry and from companies which are willing to pay a lot for development, the classical potential customers of Galaxy.
However, the community contains only the minority who don’t need SQL, or can live without it; Otherwise, they would not adopt Meteor.
And yet, even within the Meteor community, this is the #1 requested feature (935 votes in Trello).

Till MDG “adopted” Ben’s project, the progress was nice, and people expected a stable release soon. But now there are concerns that the adoption killed the project. The last fix (in the GitHub of MDG) happened about 2 weeks ago, and this fix was only a deletion of one line of dead code; it doesn’t look as the flagship project which aims to attract the zillions to adopt Meteor.

I haven’t jumped yet on the bandwagon of Meteor, and wait only for PostgreSQL support, but there are thousands in the same status. We hoped to see it in 1.2, and disappointed. We will be happy to get an answer.

Thank in advance for your answers, and thanks for developing Meteor!

4 Likes
4

I would just like to +1 on this…

I have two Meteor projects in production, a small one and a decent sized app which sees around 2-10 DB updates per second around the clock. The second one has not been without issues in terms of DB performance, and we have had to cluster and make the code jump through some hoops to get it to perform.

Now I am trying to select the stack for a new project, which will need to scale to a much wider audience. And I am slightly hesitant to commit to Meteor because of fears of how this will work out using MongoDB only.

Knowing that there is a PG option would ease those fears.

2 Likes
5

I’d like to follow-up on my original post, and also echo netmask’s thanks to the community for all that they have done so far. Like netmask, I do not want to criticise but rather help to advance the Meteor framework.

I have only lately begun to learn JS and Meteor, so I don’t yet have the expertise to deal with issues touching the core architecture. Attempting to debug the interaction between server and client, for instance, has been for me an adventure in exploration with neither map nor compass.

If I am to spend significant time working with Meteor I need to know that there are people out there who have the expertise, willingness and commitment to deal with such issues, or at least provide ongoing guidance to the rest of us.

For myself, my expertise is with Postgres, and database and server-side architectures. I am going to try solve some of the issues I originally outlined and am prepared to commit a chunk of personal time to do this but if there is no-one able, available and committed to tackle those areas outside of my expertise, then my efforts will go to waste.

Again, this is not to critcise anyone. I know that there are people working on some of the issues I have raised, and I am grateful to them but I get the impression that the time they have for this project is very limited.

I guess what I’m looking for here is for someone from MDG to talk about their policy/priorities/roadmap as far as it applies to Postgres.

1 Like
6

In the recent AMA with 4 core MDG members there were 2 questions about SQL\Postgres support. None of them were answered. Same as with the dead trello board.

I think you get the point and possibility of having sql db support from MDG in the near future.

2 Likes
7

I am one of the two people who worked on that project for 2 weeks and I am no longer an employee at MDG but can provide the info that I have:

  • the project has achieved the desired level of usability to be hackable by anyone and somewhat useful to the community (i.e. it has a decent organization and structure, and most of the big questions are figured out)
  • we (Sashko and I) are actively responding to issues and PRs for this repo
  • neither of us (Sashko nor I) have time to work on this project at the moment. I no longer work for MDG so I am not actively developing new features for it

So for this project: it achieved its initial goals. It is in a form that contributions to it are relatively easy to make. We maintain the bug tracker and contributions.

2 Likes
8

Hey, sorry we didn’t get to those questions in the AMA. It turns out that writing good answers there took longer than we expected. We tried to get to the most upvoted questions, but sorting didn’t seem to be working correctly on crater so we could have missed something.

I think the answer right now is that it is quite unfortunate that Meteor’s core data story requires a lot of effort to integrate a new data source. As you can see in the postgres experiment, there is a lot of complicated code to make basic Meteor features work, and we didn’t even get to stuff like optimistic UI, I think.

We’re looking at solutions that will make it easy to add any data source to Meteor, be it a REST API, SQL database, or anything else, without having to build a realtime driver for it. It might require changing some core things about how data is handled to, for example, make the realtime component optional (a REST API is never going to be realtime anyway!).

2 Likes
9

As you can see in the postgres experiment, there is a lot of complicated code to make basic Meteor features work, and we didn’t even get to stuff like optimistic UI, I think.

I think we got Optimistic UI working. But only for updates implemented in our knex over minimongo.

10

Could you expand on this? What sort of updates would not work optimistically?

11

I would like to see Meteor meet VoltDB!

12

Any update would work but some Knex queries will just not compile into a valid mongo syntax. This is important if you want to run the same query code on client and server. You can always have if (Meteor.isClient) block to run a different code on client simulations.

13

What would it take for MDG to make this a priority?
It seems that you are underestimating the impact of a SQL capable Meteor.
It would be huge. After the blog entry i was thinking that this will be taken seriously, and by seriously I mean that you have a internal team working on it full time. I’m sorry to hear that it’s not the case.
I hope MDG takes attention to this.
Best regards.

1 Like
14

This is so true!

Don’t be mistaken by the fact that it’s the #1 requested feature in Trello (almost 1000 votes).
It’s much much more than that!
That survey checked the wish list of the community.
Since Meteor has never supported SQL, its community contains only the minority who could live without SQL, otherwise they would not adopt Meteor.
People I speak with about Meteor, ignore it because it doesn’t support SQL.
If the survey would be done among non-users of Meteor, the number could be thousands of percents higher, but even within the Meteor community, it achieved 10 times the number of people who want React (actually much more, because the numbers of React includes people who want Angular). So if I would be MDG, and I didn’t have enough developers for SQL, I would move developers from the React team to the SQL team (especially when so many people prefer Blaze over React, there are zillion apps which were developed using Blaze, and Blaze was one of the main reasons for choosing Meteor).

And the enormous need for SQL is strong mainly in the target group of Galaxy: the industry and companies which are willing to pay a lot for development. Especially companies which already have huge data stored in SQL DBMS.
I will not be surprised to hear that Galaxy has had disappointing sales so far; its success depends on good SQL support, and the interest of both - MDG and the community - is to have great SQL support as soon as possible.

Thanks in any case for developing Meteor!

1 Like
15

A Meteor framework that can read and write any data source would be revolutionary indeed. Right now, @justinsb is full time on investigating the right direction to make this happen.

3 Likes
16

He’s the guy for that. Can you spin up a couple more copies of him so he can work round the clock?

:wink:

2 Likes
17

That’s what we’re working on for Galaxy 2.0 - containerized developers!

2 Likes
18

Wow, this is huge - and awesome - news!

19

I think the move to decouple mongo package from the meteor base package is already a good start. I think if we need a sql package or a specific rdbms like mysql or postgres package, we need to wait…

Or we can study the mongo package then mimic it i guess?

Or we can throw our free time and help MDG in dev

20

I am very pleased to hear this. I had given up on Meteor. As it is, I won’t be attempting to use it again until progress is made on this.

That said, I am willing to help on the postgres side of things, so @justinb please email me (marc@bloodnok.com) if there is anything I can do for you - I am between jobs right now and have free time available.

Here is what I need from a postgres implementation (much of which I expected to have to imlement myself), and which was lacking in the proof of concept:

  • no restrictions on keys
    I need to be able to define my own PKs. It would be good if they could be discovered by default, but at least let me define them manually. Requiring a field called id does not cut it. Also concatenated keys need to be allowed. Mapping the actual pk to an internal _id field should not be a big deal. User-definable mechanisms for this would be cool

  • schema awareness
    Allow database objects to be prefixed with a schema name.

  • The ability to deal with views as well as tables
    My application will not be allowed to directly query from tables. It would be good to allow user-defined schema queries to be used for object discovery.

  • Connections to multiple databases
    The current postgres-packages connects by default to a single database instance on server startup. This is unnecessarily restrictive.

  • Hooks for pool acquire/release
    This is to allow database code to be executed before and after each transaction. This can be used, for example, to exchange authentication details with a Virtual Private Database implementation (I have an implementation of this but it is brittle).

  • No DDL run from the application
    The meteor application should be able to use a minimally privileged database account. It should not have the rights to create or modify database objects.

  • Low database overhead
    The current implementatiion does a lot of work on the notification side of things. It should not normally be necessary for the server to perform any extra queries on receipt of a notification. That said, this is a tricky area and some user-definable bevaviour might be necessary. For my application, which uses a VPD, there would need to be a user-based visibility check performed on rows identified as updated (ie just because a row of interest to the client has been touched, it does not automatically follow that the client can see it). The simplest way to implement this would be to tell the client to refresh itself. A more sophisticated mechanism would be to allow the server to make this assessment, but then the server would have to be able to authenticate as the VPD user.

I am probably unusual in that I have no need for an ORM. I don’t object to it but please don’t make it the only way to get at the database.

1 Like