Meteor Needs Easier Data Fetching

core
1

I wanted to bring up a discussion/proposal/feature request. Looking at Netflix’s Falcor and Facebook’s Graphql, it’s very apparent that these were thought through with the front-end in mind as the consumer.

The current data fetching in Meteor is good but what would make it great is the ability to just write what JSON data we want declaratively. Meteor should just handle getting it for the template.

This is going to be exacerbated once we get SQL support. I mean who really wants to write SQL on the front-end to fetch data??

The back-end can also be declarative. Specify what a user is allowed to fetch and deny everything else (this could help solve the lack of a schema). Our publication is already the same concept of a custom endpoint, usually per resource (user, posts, comments).

Both Falcor and GraphQL are platform/DB agnostic so I think these ideas can be used with Meteor fairly easily. If we can bake this (or something similar) into core it would make it much easier to be the consumer. How nice would it be to do a meteor create myapp and meteor add graphql and be done?!?

If interested checkout these videos:
Exploring GraphQL at react-europe 2015
Netflix JavaScript Talks - Falcor

Here’s a sample request and response with GraphQL syntax:

Template.post.queryData({
  user(_id: '1234') {
    _id,
    name,
    isViewerFriend,
    profilePicture(size: 50)  {
      uri,
      width,
      height
    }
  }
})

and the template would receive this data: (it would handle the subscribe and fetching)

{
  "user" : {
    "_id": '1234',
    "name": "Jane Doe",
    "isViewerFriend": true,
    "profilePicture": {
      "uri": "http://someurl.cdn/pic.jpg",
      "width": 50,
      "height": 50
    }
  }
}
3 Likes
2

Doesn’t it go against the Meteor concept of client side database ?
Doesn’t it go against isomorphism, with most of the work done on the server ?
It seems to me that GraphQL is a replacement for that, but hasn’t Meteor already gone way past that ?

3

Are you familiar with JSONiq?
http://www.jsoniq.org/

It was developed by ADP, the billion dollar payroll company, which apparently was an early investor in Mongo/Node stacks, and has an in-house alternative to Meteor. Or, rather, Meteor is something of an open-source version of what folks like ADP and Bloomberg and a few other companies have been able to piece together with their warchests.

Anywho, ADP opensourced JSONiq last year during the Node Roadshow, and I think it might be something close to what you’re looking for… a declarative json data structure query. I’m rather surprised there hasn’t been more interest in it yet. But lots of folks get caught up by front-end shinyness, rather than geeking out on data transform libraries. Ah well.

4

This is a good point. Perhaps. It is still data over the wire. Also I still think that a clientside cache is necessary. You should still be able to query it on the client/server if needed.

I’m thinking GraphQL more less provides an alternative to the subscription interface.


[quote="vjau, post:2, topic:7517"] It seems to me that GraphQL is a replacement for that, but hasn't Meteor already gone way past that ? [/quote]

I don’t think it’s a 1:1 replacement. In fact you could use the GraphQL JS library and babel to transpile the GraphQL to JSON and pass that to the subscription param. However setting up the backend to publish the right data for GraphQL is non-trivial. I think a lot of people are experimenting with this now on many types of platforms.

Relay does overlap a lot (FB’s clientside cache, pub/sub, etc…) but doesn’t have to be used with GraphQL.

Do you feel that SQL in the browser will be a good thing? (not trolling just interested) Perhaps it’s just that i’m not as familiar diving in with SQL queries.


[quote="awatson1978, post:3, topic:7517"] Are you familiar with JSONiq? [/quote]

Oh wow i’ve never seen that! I wonder if it’s because their docs are so bad. A quick look through the docs and example and i’m still not entirely sure how their syntax works

Perhaps the interest in GraphQL is that to query it you basically put in the shape of data you’re expecting and done (more front-end shinyness :laughing:)

5

Well, it was written by accountants. What do you expect? :stuck_out_tongue:

But if you dig into it, it’s doing a similar thing… you specify the shape of the data you’re expecting, and there are a bunch of JSON mapping functions that produce it. Seeing it in action was pretty slick, and everybody was like ‘oooh, I want JQL’… (which was what they were calling it before JSONiq).

The problem with doing these ORM technologies, of course, is the underlying data stores differ in their topologies. SQL has normalized tables; Graphs have graph objects; Mongo has JSON tree objects. Better have a copy of Knuth if you’re going to be tackling those kinds of problems.

Joking aside, the reason that the GraphQL can work so cleanly is because the underlying Graph API uses normalized nodes and edges and vertices. Its not normalized against 1NF, 2NF, 3NF, or BCNF, but it’s got something along the lines of a Property Graph Model, so call it PGM normalization for lack of a better term. Put another way, to get that clean GraphQL query syntax, they’ve compromised a lot with regard to their database structure, and are saying ‘okay, in order to get this clean API, we’re only going to program apps that can store functionality in a particular type of normalized graph database’. Works great if you’re only really worried about programming a social network. But if you’re programming other apps? That’s a serious architectural compromise.

The JSONiq is way less flashy, but it can produce some similar results as GraphQL for Mongo and perhaps SQL. Once one says, ‘well, some of our data isn’t going to conform to PGM normalization’, then GraphQL no longer is sufficient, and one starts needing aggregation functions, grouping, filtering, sorting, etc. JSONiq query syntax is a bit more like SQL than GraphQL, but that’s because it’s dealing with messier denormalized data.

6

Why should it be worse to put it in the client than in the server ?
If you are the one coding the frontend and there is another one coding the backend, then i understand, but i’m not sure that is the point of Meteor.
I don’t know what MDG has in mind for SQL integration, if they intend to create a mini-sql on the client or not.

2 Likes
7

Yea that’s a good point, I guess a lot of Meteor users will be doing this full stack. Hopefully we’ll get some ORM support for SQL databases… i’m not looking forward to using this to get a record: :laughing:

SELECT Shippers.ShipperName,COUNT(Orders.OrderID) AS NumberOfOrders FROM Orders
LEFT JOIN Shippers
ON Orders.ShipperID=Shippers.ShipperID
GROUP BY ShipperName;
1 Like
8

I know that PostreSQL and probably other are able to output JSON, i don’t know how this work though.

9

I believe developers with SQL background work either one of

  • SQL
  • An ORM layer
  • Plain business objects that decouple CRUD operations (basically mini-orm’s)

Furthermore, fetching/mutating data should be considered by (at least) two aspects

  • The method used
  • Its format

Its format can be JSON, XML, or some (probably delimited) text format and of course, considering the evolution of data representation formats for web, JSON sounds like the better choice.

On the other hand, I don’t agree, not even the slightest, that SQL should be abstracted away. If we are talking SQL support, than SQL should be the method (language) of interaction, both on the server and on the client, if we want to stick to the isomorphism.

Then of course, either MDG, or perhaps preferably the community can come up with the abstractions.

There also will be need for an extensible API since almost all SQL database vendors provide their own proprietary (open or closed) dialects to SQL, so that the community can also develop adaptors for vendor-specific data types, functions and operators.

So to sum up, I don’t think Meteor needs any easier data fetching or any meta-abstractions. Meteor needs consistent API’s that are aligned with its host data storage providers. If I’m doing a mongo query, I want to be able to test a query on the mongo console and copy it over to the app with minimal editing. And if I’m doing an SQL query, I’d like to be able to do the same with that as well. No need to go through a translator where I already know the language. No need to worry about potential problems or shortcomings that (may/will) come with the translator.

3 Likes