Grapher - Collection Relationship Manager + GraphQL Like Data Fetcher


#22

Grapher may be what you are looking for for fetching the data. However for providing that type of analysis you may need to use a deep learning library, or write the code yourself.

Retrieve the baskets that contain the product id.
Perform a function to retrieve the top products…

Grapher is going to keep focus on data-fetching. So it acts like a DB-driver. Databases should not have that sort of intelligence of providing insight.


#23

This sounds amazing. Looking forward to trying it out on my Meteor-classic-Blaze project if it’s just a drop in.


#24

Honestly this is very very clever and elegant solution.
You are making use of Mongo Aggregations framework very smartly.
As sashko said, too bad it is not compatible with graphQL because it is a bit of a risk to tie our app on this libs for now…
But again, they are many many great ideas here :thumbsup: :clap:


#25

Thanks for the kind words.

Regarding risk, if I was in your position, yes I would see it as a risk, however I am in my position and I can tell you for sure, it’s a risk if you don’t do this. I can’t imagine right now exposing a database without grapher, it’s like going back in time.

Grapher can be easily integrated with GraphQL, but we believe having the “data request” as a JS object makes it more powerful, and elegant, you don’t have to re-invent new concepts to use “mixins”, functions, etc. If you read about $filter() and the fact that we allow $filter() $filters under each collection-node, you’ll understand why it is more powerful.

You see, the reason why this relatively small lib, works so well is that it has a laser-focus: Meteor, MongoDB, JavaScript. That’s it. No satisfy “every language”, “every db” and “every developer” from around the world. It can be easily decoupled from Meteor (without the pub/sub & methods system), but again, why bother at this stage ?

We are using this in production apps, it just works perfectly for us. The only reason we would use GraphQL in a Meteor app is to have joins and what “Grapher” does for us. GraphQL mutations ? In our opinion you can’t get more elegant than RPC. And that’s what Meteor methods are exactly.

Regarding long-term support. We will soon release version 1.2, which contains more tests, more speed improvements, better code, few new cool features. And we are going to make it LTS 2020. To give people even more trust.


#26

One of the reasons you’re getting flack on the GraphQL issue is of course because MDG is going full blast on Apollo (GraphQL). Heck that’s where they’re going to try to make $.

Don’t worry so much about pleasing everyone here – accept you’re not going to, stay true to your original mission or ideals and if they’re right, you’ll get traction.


#27

Hey man, I’m just trying to give helpful advice here. There are legitimate reasons to join a larger community with great tools rather than working with something very similar but only different in syntax. But I think Grapher is a great idea.


#28

I don’t think he meant to be rude. You are right regarding joining the community rather than reinventing the wheel, except this is not reinventing anything.

@aadams: I really hope Apollo will bring some heavy $ to MDG so they can invest it in Meteor. They should use the money to find a way to clone @benjamn and @abernix - 5 clones of them and Meteor will have no rival.


#29

What I said was not rude at all. It’s natural that if a company is going to make money on a product, they want as many people as possible to use it, right? This forum is one method to help with that no? How do we think they justify keeping the lights on in this place? – its a line item under cost of doing business.

Are we suppose to pretend that everyone’s motivations on here are 100% altruistic?

There’s nothing wrong with MDG employees trying to convince others to use their products and services – specially when they’re the awesomeness. There’s nothing wrong with them trying to encourage others to use their tools instead of reinventing something else and/or using alternatives – and to call others out from time-to-time.

.

On top of this their products so far have been really awesome – so part of the motivation is to share this awesomeness with others – I do this as well with software I really like, enjoy using, and if it enriches someone’s development experience. I certainly believe this is a big motivation MDG and specially @sashko has when they promote their software and services. And I do think @sashko motivations and reputation has been unimpeachable so far.

.

I too really hope Apollo is successful and brings in the big $ for MDG because I want them to continue making great software – is this also a bad thing to say? Are we not suppose to know they’re motivated to make money and are a real business?

.

Where is the disconnect coming from? I agree with @sashko on this, the way I’m reading it, you seem to taking a different path to solving a similar problem – one in which MDG is trying to solve. Why is what you’re doing not duplicating efforts in some respects?

.

Haha, do they really have a rival today? Most of their man power is hunkered down burning the midnight old banging away on Apollo and not Meteor, let us see what they produce with in 1.5.

.

Let’s try to stop being so sensitive and keep-it-real on here.

.

Having said all this, Grapher does look pretty-fing-awesome.


#30

Yes, you can say efforts are duplicated, but Apollo is huge compared to Grapher, we just solved MongoDB relational mapping and data fetching. We solved the problem within Meteor not outside it. This is why, yes, even if we may have common goals, the paths we took are completely different.

:smiley: Let’s see. Curious also.

Cheers man!

There’s something big coming to grapher 1.2. We finally managed to find a way to solve the problem we had for “Exposures”, I’m really really excited about it. Because I thought it would be an unsolvable issue for over 2 months, however a random guy @damonmaria gave me a spark that helped me arrive to the solution, by posting an issue in GitHub. (Thanks!)

That will complete the security aspect, giving developer full flexibility.
Along with that there are some other neat features coming for Meta Relationships.

After this we’ll put a lockdown on API and features, and make it our LTS version.
In which we will continue focusing on: solving any bugs, improving performance, security, memory usage and commenting and documenting the code better, but no new features.

The next phase of grapher is a bit foggy, we don’t have a Roadmap, because we did not think further than 1.2

We could go towards the direction Apollo is going, by abstracting entry points, giving the abiltiy of creating VirtualCollections that communicate with any kind of API. We are basically reinventing the wheel of what GraphQL does. And GraphQL does it well enough. An idea will be to have an abstraction layer over graphql and apollo in order to solve this. But again, very foggy and highly unlikely. Better to have an integration of grapher for apollo than the other way around.

Another great thing that may come is aggregation:

  • Give ability to use the aggregate framework for your links.
  • Provide an abstraction layer for aggregate-framework for the most common things. ($count, $sum, $countIf etc)
  • Allow aggregate framework to work reactively

#31

1.2.1 is out. Contains ability to filter linked items by metadata using $meta in $filters. Stores $metadata as object in child links. Both work for “inversed” and “direct links”.

And the “godfather” feature: Exposure body, which allows functions as body, and self referencing functions that are computed on-demand. Allowing stuff like

const userBody = function (userId) {
    return {
          profile: 1,
          friends: userBody
    }  
}

Users.expose({body: userBody});

// and the request like:
createQuery({
    users: {
         profile: 1,
         friends: {
               profile: 1,
               friends: {
                     friends: { ... you get the idea ... }
               }
         }
    }
})

Locking the features for now, made it LTS. Next focus is on making the code better, more performant, better tested, better documented, better docs.

Now shifting focus to the tutorials :smiley:


#32

I think there’s a lot of potential for this project, so I know you didn’t ask for it, but here’s a bit of friendly advice:

  • Take your time and make thoughtful decisions.
  • It’s better to think things through and get it right the first time.
  • Focus on the long term vision for the tool – not short term wins.
  • Don’t try to make your tool everything to everyone – stay focused.
  • Think about the needs of the developer (api, feature set, performance, etc.).
  • Think about and ask your target audience what their real needs/issues/pain-points are.
  • Be willing to take advice, and make a venue to solicit advice from developers.
  • Be willing and open to changing your mind based upon feedback from your developers.
  • Make great documentation and tutorials if you want real adoption.
  • Think about if and how you’d like to monetize your efforts (for example a patreon campaign).

#33

Your guide covers the one-to-one and the one-to-many relationships.

But I don’t see how one would define a many-to-many relationship.


#34

You should reconsider using mongodb(and any nosql) for many-to-many relationship at all.
What grapher does is optimizing(minimizing) amount of queries. Its fine with hierarchically structured data.

What many-to-many relationship leads to is exact opposite. It populates queries.
it actually isn’t m-t-m relationship inside db, we just use an abstraction to operate this data as many of o-t-m

So, in short: there’s no way.
If you dive into mongodb official docs, there’s no guide on this data model as well.
You have to use abstraction of data model or graph database(if that’s a case of MUCH data).


#35

I know that. In fact, if you had read any of my previous posts regarding NoSQL databases you’d know that I find them moderately idiotic because they lead to the re-invention of the wheel…

… just like Grapher nicely demonstrates. Instead of having relationships (and the assurance of their integrity) defined at the db level, we instead have to muck around at the application level.


#36

Well, Grapher isn’t a database to blame it… Its just a meteor package that simplifies use of related data.
So its mostly about removing overhead with SQL-style data for meteor, rather than reinventing mongodb.

P.S. I actually recall reading your comments, sorry for unflattering assumptions, its night after all :grin:


#37

I’m certainly not blaming Grapher, in fact I’m deeply impressed by what is achieved by it.

However, Grapher is just a symptom of how “new and shiny” doesn’t automatically mean “better suited to the task” - because quite a lot of what Grapher does is built into SQL databases right from the start.


#38

Well, its great when you can get best from the both worlds along with meteor reactive binding.

Best abstractional many-to-many-of-something database.
We’ll see it comming in 2099, but unlikely make use of it ourselves :smiley:


#39

Ok @rhywden , the feedback I get from you is that I need to write in the docs a “SQL relationship comparison” because any type of relationship is achievable. But people are very familiar with those and they need a point of reference :slight_smile:

So let me explain we have 4 types of relationships “One”, “One-Meta”, “Many”, “Many-Meta”, the name refers to how the data is stored: How many links we store. The overall relationship status is defined by you. However we support One-To-Many by having the storage in the “Many” side like in the SQL. You can also have One-To-One which is a “One” relationship with “unique: true”.

@gothicmage there is a way ! :smiley:

A Many-to-Many relationship can be a simple “many” relationship example:

Users.addLinks({
    groups: {
          type: 'many',
          collection: Groups,
          field: 'groupIds'
    } 
})
Groups.addLinks({
     users: {
         collection: Users,
         inversedBy: 'groups'
     }
})

This can also act as One-To-Many indeed, if all your groupIds from all users are unique. But if it’s truly One-to-Many keep the storage in the “Many” side.

That’s it. Here’s your Many-To-Many relationship :slight_smile:

Performance is not an issue with Grapher. The queries are very fast. Because of the aggregation of filters and re-assembly. I’m not sure if it’s faster than SQL joins, someone can test this :slight_smile:

Plus if you have a M2M and expect having a lot of fields 1000+. It is not a problem. Grapher forces you to take only the fields you need. And if you want the linked elements, you would have taken those 1000 either way.

Now coming back on “SQL reationships”. MongoDB is something new, the relationships in SQL can be easily modeled in MongoDB with Grapher, but the way we think about relationships needs to change, this is why in the API of Grapher there is no such thing as “One-To-One” … Because there is so much more to it. For example, instead of having separate tables for Many-To-Many, you store it directly as array. If you want to put additional data about your relationship (like User has many Groups but he is admin to some), you specify metadata. This is superior to SQL in my opinion, and far easier. It was tricky for me at first to shift from SQL relationships.


#40

You have raised very good points here:

But I don’t see how one would define a many-to-many relationship.

You can have a “real” m-t-m. By creating a new collection “UsersGroups” and link with user and group (Ensure groupId and userId as unique pair). Each Users and Groups will link to UserGroups using “inversedBy” and fetch the data like:

{
    users: {
         usersGroups: {
                group: {
                      name: 1
                }
         }
    }
}

Doesn’t feel to good. But it is a “true” m2m.

quite a lot of what Grapher does is built into SQL databases right from the start.

Grapher has 3 ingredients:

  • Collection Relationship Manager
  • Query as Graph
  • Safely expose data module

SQL has only the first one :smiley: So “a lot” may not be the propper term. Not wanting to pick on semantics but, in real life scenarios grapher covers cases 95%+ cases. If you want complex actions like, count, groupBy, I think it’s weird to give so much intelligence to the Database. In my opinion the central point of intelligence should be in the app and controlled by the app itself.

The reason for this is that your storage layer can be abstracted easily later. So computing things like total calculations like users that bought at least 3 donuts on “odd” dates and they have from at least 3 different donut providers (intentionally stupid example).

This can be modeled into multiple queries and a service that applies the logic to give you the data you need. That service is completely abstracted from the storage layer and it has the advantages of:

  • You can test it in your app, without having to touch the database at all.
  • You can shift the storage layer later without having to rewrite your complex queries

In fact, I believe it is sometimes faster to code it in JS in OO style, rather than have a 500 line long query. (I’ve seen bad things in my life.)

To summarize: MongoDB is dumber than SQL DB but it’s faster + it’s “blended” with JS. One language one love. Cheers.

.


#41

Well, it may be faster but it also comes with quite some drawbacks.

Transactions. Relationships guaranteed by the database. Guaranteed type safety. Guaranteed uniqueness.

Those are not small things.