Grapher - Collection Relationship Manager + GraphQL Like Data Fetcher

announce
1

A short introduction to Grapher

Context

I personally worked on creating over 10 projects (small and big) written in Meteor (Blaze and React), and in every project with complex db relationships we hit the NoSQL walls. When we had to fetch the data, we had to create a lot of boilerplate code, which was prone to many security leaks. This is why we had to re-invent the way data is fetched and linked in Meteor.

The story of Grapher goes back to Meteor 1.2, before the module approach of Meteor 1.3. We invented a plugin-based approach: http://www.quantum-framework.com/ + Package oriented architecture for dealing with Meteor. Enabling packages to work together in a common namespace: The Quantum.

The quantum project is solid and battle tested, and we will continue to maintain it, however for various reasons we stopped iterating on it. It’s still LTS until 2018.

However, these two important “plugins”:
http://www.quantum-framework.com/plugins/collection-links
http://www.quantum-framework.com/plugins/query

…have been merged into Grapher.

Creating this was an adventure of it’s own. Roughly 80% was invested in design of the api 20% in actual coding work.

We licensed this M.I.T. because we believe in Open-Source, and we really want Meteor to continue conquering the JavaScript market. We aren’t yet sold on Apollo (Sorry MDG). Meteor is defined by beauty and simplicity, something that Apollo lacks in my opinion. + The performance doesn’t even come near Grapher.

You can play with it here:

Key Features of Grapher:

  1. Full JavaSript + Meteor.
  2. Makes linking collections and fetching links in Meteor dead simple. And it works with all types of “NoSQL” relationships.
  3. It has a Grapher-Live package which let’s you test your query live right from the browser. Much like “graphiql”
  4. You have the same API for retrieving data reactively and non-reactively.
  5. Highly performant.
  6. Like apollo you have the ability of “linking” to external data-sources.

About performance

Imagine this request:

users: {
    posts: {
         comments: {
              author: {}
         }
    }
}

No matter how many users, posts, comments you’ve got. The data you need for this will be fetched in just 4 DB queries (in the case above, becase you have 4 collection nodes).

Previously we approached this naively and for a small chunk of data we reached about 2000 DB requests. We had to find a solution, this is how the “hypernova” module was born which merges filters and assembles them after receving, this resulted into 40x performance boost for a medium query. And exponentially more for larger queries.

There is a lot to it, please check it out, try to break it ( if you can :wink: ) and submit an issue, we’ll be more than happy to resolve it.

We will continue working on this. Our next milestone will be to have a dedicated website for a top-notch clear documentation.

Thanks for the time.


37 Likes
2

GraphQL-like

This is a great idea, but I’m a little bummed because it won’t work with GraphQL-based developer tools.

IMO there would be a big benefit from using “actual” GraphQL for the query language, and having a schema to enable dev tools like GraphiQL.

3

@sashko thanks for your input. It won’t work, however we will be building the right tools.

The current “live debugger” tool is:

The “docs” for it can be created from the links and the actual SimpleSchema attached to the collections. Easily do-able.

We believe that Grapher Live currently offers the basic debug tool for Grapher, and it will be improved in the future. For sure!

However, our next step will be to create a proper documentation, after that we iterate on security, which is going to be the key feature in this. Then we can offer much better dev-tools.

2 Likes
4

Any reason? have you tried simply converting the SimpleSchema to a GraphQL schema? Basically, it just sounds like you are reinventing some wheels that have already been solved in the GraphQL community, and it would save you a lot of work to use those solutions instead. I’m not trying to convince you to use Apollo or whatever, just saying it might be easier for you.

6 Likes
6

@sahko here are some reasons:

  1. This is Meteor. Meaning you do “meteor add cultofcoders:grapher”. You can configure your links with your EXISTING project with your EXISTING database and EXISTING RELATIONS and Schema. You do not have to rewrite anything. You have almost all the advantages in GraphQL in your Meteor App. You don’t have to rewrite anything! Just add it as a plugin.

  2. Performance for graphing a lot of data doesn’t even come close. I would like to see how GraphQL retrieves 100,000 objects in 5 nesting levels. We focus on 2 things only: Meteor and MongoDB and with the help of aggregate framework we achieve ultra-fast responses. And as “leaf nodes” we allow “resolver links” which can let you connect to any type of DB/REST API/etc.

  3. You benefit from Meteor’s reactivity in the data graph. You can make it reactive or non-reactive. It’s the same API.

  4. Regarding saving time. After a deep analysis of Apollo, we believe it’s an amazing tool. However, it tries to satisfy everyone this will slow down it’s progress, but it trully is the 2nd best data-system out there… besides Meteor :slight_smile:

15 Likes
7

You don’t have to reinvent or rewrite anything here. It’s just a plugin, works out of the box with your app.

8

This is pretty great, I agree!

1 Like
9

Why did I have to start another project now ~_~

9 Likes
10

haha, we’re in the JavaScript explosion era my friend, natural selection will decide the ultimate framework.

Just choose something and stick with it. That’s the recipe for success. Banks still use cobol internally from 30 years ago. Did they switch because it’s a shiny new thing ? No. they stuck with it because it just works.

5 Likes
11

Glad you like it. The main BENEFIT is:

  • It works with Blaze & Meteor existing collections. You can create the links even if you already have them, it already supports most types of collection-linking in MongoDB.
  • It can simplify a lot Blaze & React views, we separate the concept of data retrieval as a “query”. That query can be reactive or not, depends how you configure it. They are viewed as “Data Providers”
  • They already have abilities to link via “resolver links” to any type of Data-Source (REST-apis/MySQL databases/Other MongoDB databases/Any type of API that communicates via socket)
5 Likes
12

I think it would be pretty simple to hack this to use a “real” GraphQL schema and query language, while maintaining all of those benefits. Then you could use graphiql, native mobile GraphQL clients, etc.

Perhaps if I have some time I’ll whip together a proof of concept. But it would be really dope - a fully-reactive GraphQL implementation is something a lot of people would be extremely excited for, which could bring in more contributors, ideas, etc.

Not saying you need to do this - just from my perspective it would be cool.

2 Likes
13

Ty for the feedback. I see your point, the API is already there, it needs to generate “Queries” and Schemas based on Links and SimpleSchema. I also think it’s simple to hack for someone that knows GraphQL and how the Relationship Manager works.

However,
The only need to do this is to be a part of GraphQL community and use their tools, but you only seem to mention GraphiQL as a benefit. But GraphiQL doesn’t offer anything useful besides the data. In our grapher-live package we actually measure DB request times and response times (you could even expose grapher-live on an actual live environment, and check live which queries are heavy, you can even test it logged-in or non-logged in to check the actual exposed data, it already has options for add),

And unlike graphiql, we’ll add a documentation that it’s truly useful (generated based on simple-schema and links vizualized as an actual drawn graph with links displaying information on click, displaying information about the links on hover/click).

Plus, you can also use “meteor debug” and debug every step of graph generation and data retrieval. So I really think GraphiQL is inferior.

And regarding native mobile GraphQL clients, this “graph” can be exposed as an HTTP API since it can easily accept “json” graph requests.

6 Likes
14

Yeah one concrete example is allowing you to use something like apollo-ios if you have a native app: https://github.com/apollostack/apollo-ios

Really cool project btw. I’d love to see it take off, it seems to solve the issue of having to write a lot of publications.

15

Hi a lot has changed since 10d ago. We introduced Named Queries, the missing link to have complete security.

Now our next iteration is making documentation perfect, and fixing the bugs.

Documentation: http://grapher.cultofcoders.com/

Live Test: http://grapher-live.cultofcoders.com/ and Go To “Grapher Live”

(This is basically the https://github.com/cult-of-coders/grapher-boilerplate deployed)

9 Likes
16

This looks amazing, congratulations on putting it together.

Are there any examples of using it with Blaze?
Also looking for a tutorial on getting started.

Cheers,
Praney

17

Thanks! There is so much more to it. We went to great lengths to make it performant, and we are now battling with security: Checkout Named Queries and Exposure Body. Basically the struggle now is: How do we make it simple to use and understand. And frankly when dealing with graphs as data, there is no simple way, because there are a lot of concepts involved, but we will continue our research.

Regarding Blaze, Good point. I will add some examples for blaze in the Docs.

For now:

import userListQuery from '/imports/api/user/query/list.js';

onCreated() {
    this.query = userListQuery.clone()
    this.query.subscribe()
}
helpers: {
   isReady() {
       return Template.instance().query.subscriptionHandle.ready(),
  }
  users() {
       return Template.instance().query.fetch() 
  }
}

By method call example:

onCreated() {
    this.query = userListQuery.clone()
    this.ready = new ReactiveVar(false);
    this.query.fetch((err, res) => {
          this.queryData = res;
          this.ready.set(true);
    })
}
helpers: {
  isReady() {
       return Template.instance().ready.get()
  }
  users() {
       return Template.instance().queryData;
  }
}

Cheers!

4 Likes
18

Cheers mate, appreciate it.

19

Mate just another quick question, you are referring to “/imports/api/user/query/list.js” is there an example you are referring to?

20

Really cool project :smiley:

What are some ideas as to how one could use graph queries, such as ‘market basket analysis’, with Grapher? E.g.

Given a product in our store
What are the products most often purchased in the same cart as this product?
So we can add a ‘recommended products’ list to this product page

Given a person with a favorite books list
What are the books most related to this persons reading interests?
So we can add a ‘recommended reading’ list to the user homepage.

Source: Extending market basket analysis with graph miningtechniques: A real case

Given a patient diagnosis
What are likely diseases/disorders that the patient may also experience?
So that we can plan proper treatment for the patient.

680×675 37.3 KB

Patient population #1. Note how Chronic Heart Disease has a strong
co-occurrence rate with Hyperlipidemia, but actually occurs less
frequently overall than a number of different diagnoses.
Source: Revisiting the Analysis of Diagnoses in Populations

See also:

1 Like
21

It’s just a query exported in that file using: createQuery or Collection.createQuery.