Grapher - Collection Relationship Manager + GraphQL Like Data Fetcher

One of the reasons you’re getting flack on the GraphQL issue is of course because MDG is going full blast on Apollo (GraphQL). Heck that’s where they’re going to try to make $.

Don’t worry so much about pleasing everyone here – accept you’re not going to, stay true to your original mission or ideals and if they’re right, you’ll get traction.

Hey man, I’m just trying to give helpful advice here. There are legitimate reasons to join a larger community with great tools rather than working with something very similar but only different in syntax. But I think Grapher is a great idea.

4 Likes

I don’t think he meant to be rude. You are right regarding joining the community rather than reinventing the wheel, except this is not reinventing anything.

@aadams: I really hope Apollo will bring some heavy $ to MDG so they can invest it in Meteor. They should use the money to find a way to clone @benjamn and @abernix - 5 clones of them and Meteor will have no rival.

4 Likes

What I said was not rude at all. It’s natural that if a company is going to make money on a product, they want as many people as possible to use it, right? This forum is one method to help with that no? How do we think they justify keeping the lights on in this place? – its a line item under cost of doing business.

Are we suppose to pretend that everyone’s motivations on here are 100% altruistic?

There’s nothing wrong with MDG employees trying to convince others to use their products and services – specially when they’re the awesomeness. There’s nothing wrong with them trying to encourage others to use their tools instead of reinventing something else and/or using alternatives – and to call others out from time-to-time.

.

On top of this their products so far have been really awesome – so part of the motivation is to share this awesomeness with others – I do this as well with software I really like, enjoy using, and if it enriches someone’s development experience. I certainly believe this is a big motivation MDG and specially @sashko has when they promote their software and services. And I do think @sashko motivations and reputation has been unimpeachable so far.

.

I too really hope Apollo is successful and brings in the big $ for MDG because I want them to continue making great software – is this also a bad thing to say? Are we not suppose to know they’re motivated to make money and are a real business?

.

Where is the disconnect coming from? I agree with @sashko on this, the way I’m reading it, you seem to taking a different path to solving a similar problem – one in which MDG is trying to solve. Why is what you’re doing not duplicating efforts in some respects?

.

Haha, do they really have a rival today? Most of their man power is hunkered down burning the midnight old banging away on Apollo and not Meteor, let us see what they produce with in 1.5.

.

Let’s try to stop being so sensitive and keep-it-real on here.

.

Having said all this, Grapher does look pretty-fing-awesome.

Yes, you can say efforts are duplicated, but Apollo is huge compared to Grapher, we just solved MongoDB relational mapping and data fetching. We solved the problem within Meteor not outside it. This is why, yes, even if we may have common goals, the paths we took are completely different.

:smiley: Let’s see. Curious also.

Cheers man!

There’s something big coming to grapher 1.2. We finally managed to find a way to solve the problem we had for “Exposures”, I’m really really excited about it. Because I thought it would be an unsolvable issue for over 2 months, however a random guy @damonmaria gave me a spark that helped me arrive to the solution, by posting an issue in GitHub. (Thanks!)

That will complete the security aspect, giving developer full flexibility.
Along with that there are some other neat features coming for Meta Relationships.

After this we’ll put a lockdown on API and features, and make it our LTS version.
In which we will continue focusing on: solving any bugs, improving performance, security, memory usage and commenting and documenting the code better, but no new features.

The next phase of grapher is a bit foggy, we don’t have a Roadmap, because we did not think further than 1.2

We could go towards the direction Apollo is going, by abstracting entry points, giving the abiltiy of creating VirtualCollections that communicate with any kind of API. We are basically reinventing the wheel of what GraphQL does. And GraphQL does it well enough. An idea will be to have an abstraction layer over graphql and apollo in order to solve this. But again, very foggy and highly unlikely. Better to have an integration of grapher for apollo than the other way around.

Another great thing that may come is aggregation:

  • Give ability to use the aggregate framework for your links.
  • Provide an abstraction layer for aggregate-framework for the most common things. ($count, $sum, $countIf etc)
  • Allow aggregate framework to work reactively
3 Likes

1.2.1 is out. Contains ability to filter linked items by metadata using $meta in $filters. Stores $metadata as object in child links. Both work for “inversed” and “direct links”.

And the “godfather” feature: Exposure body, which allows functions as body, and self referencing functions that are computed on-demand. Allowing stuff like

const userBody = function (userId) {
    return {
          profile: 1,
          friends: userBody
    }  
}

Users.expose({body: userBody});

// and the request like:
createQuery({
    users: {
         profile: 1,
         friends: {
               profile: 1,
               friends: {
                     friends: { ... you get the idea ... }
               }
         }
    }
})

Locking the features for now, made it LTS. Next focus is on making the code better, more performant, better tested, better documented, better docs.

Now shifting focus to the tutorials :smiley:

5 Likes

I think there’s a lot of potential for this project, so I know you didn’t ask for it, but here’s a bit of friendly advice:

  • Take your time and make thoughtful decisions.
  • It’s better to think things through and get it right the first time.
  • Focus on the long term vision for the tool – not short term wins.
  • Don’t try to make your tool everything to everyone – stay focused.
  • Think about the needs of the developer (api, feature set, performance, etc.).
  • Think about and ask your target audience what their real needs/issues/pain-points are.
  • Be willing to take advice, and make a venue to solicit advice from developers.
  • Be willing and open to changing your mind based upon feedback from your developers.
  • Make great documentation and tutorials if you want real adoption.
  • Think about if and how you’d like to monetize your efforts (for example a patreon campaign).
4 Likes

Your guide covers the one-to-one and the one-to-many relationships.

But I don’t see how one would define a many-to-many relationship.

You should reconsider using mongodb(and any nosql) for many-to-many relationship at all.
What grapher does is optimizing(minimizing) amount of queries. Its fine with hierarchically structured data.

What many-to-many relationship leads to is exact opposite. It populates queries.
it actually isn’t m-t-m relationship inside db, we just use an abstraction to operate this data as many of o-t-m

So, in short: there’s no way.
If you dive into mongodb official docs, there’s no guide on this data model as well.
You have to use abstraction of data model or graph database(if that’s a case of MUCH data).

I know that. In fact, if you had read any of my previous posts regarding NoSQL databases you’d know that I find them moderately idiotic because they lead to the re-invention of the wheel…

… just like Grapher nicely demonstrates. Instead of having relationships (and the assurance of their integrity) defined at the db level, we instead have to muck around at the application level.

Well, Grapher isn’t a database to blame it… Its just a meteor package that simplifies use of related data.
So its mostly about removing overhead with SQL-style data for meteor, rather than reinventing mongodb.

P.S. I actually recall reading your comments, sorry for unflattering assumptions, its night after all :grin:

I’m certainly not blaming Grapher, in fact I’m deeply impressed by what is achieved by it.

However, Grapher is just a symptom of how “new and shiny” doesn’t automatically mean “better suited to the task” - because quite a lot of what Grapher does is built into SQL databases right from the start.

Well, its great when you can get best from the both worlds along with meteor reactive binding.

Best abstractional many-to-many-of-something database.
We’ll see it comming in 2099, but unlikely make use of it ourselves :smiley:

1 Like

Ok @rhywden , the feedback I get from you is that I need to write in the docs a “SQL relationship comparison” because any type of relationship is achievable. But people are very familiar with those and they need a point of reference :slight_smile:

So let me explain we have 4 types of relationships “One”, “One-Meta”, “Many”, “Many-Meta”, the name refers to how the data is stored: How many links we store. The overall relationship status is defined by you. However we support One-To-Many by having the storage in the “Many” side like in the SQL. You can also have One-To-One which is a “One” relationship with “unique: true”.

@gothicmage there is a way ! :smiley:

A Many-to-Many relationship can be a simple “many” relationship example:

Users.addLinks({
    groups: {
          type: 'many',
          collection: Groups,
          field: 'groupIds'
    } 
})
Groups.addLinks({
     users: {
         collection: Users,
         inversedBy: 'groups'
     }
})

This can also act as One-To-Many indeed, if all your groupIds from all users are unique. But if it’s truly One-to-Many keep the storage in the “Many” side.

That’s it. Here’s your Many-To-Many relationship :slight_smile:

Performance is not an issue with Grapher. The queries are very fast. Because of the aggregation of filters and re-assembly. I’m not sure if it’s faster than SQL joins, someone can test this :slight_smile:

Plus if you have a M2M and expect having a lot of fields 1000+. It is not a problem. Grapher forces you to take only the fields you need. And if you want the linked elements, you would have taken those 1000 either way.

Now coming back on “SQL reationships”. MongoDB is something new, the relationships in SQL can be easily modeled in MongoDB with Grapher, but the way we think about relationships needs to change, this is why in the API of Grapher there is no such thing as “One-To-One” … Because there is so much more to it. For example, instead of having separate tables for Many-To-Many, you store it directly as array. If you want to put additional data about your relationship (like User has many Groups but he is admin to some), you specify metadata. This is superior to SQL in my opinion, and far easier. It was tricky for me at first to shift from SQL relationships.

You have raised very good points here:

But I don’t see how one would define a many-to-many relationship.

You can have a “real” m-t-m. By creating a new collection “UsersGroups” and link with user and group (Ensure groupId and userId as unique pair). Each Users and Groups will link to UserGroups using “inversedBy” and fetch the data like:

{
    users: {
         usersGroups: {
                group: {
                      name: 1
                }
         }
    }
}

Doesn’t feel to good. But it is a “true” m2m.

quite a lot of what Grapher does is built into SQL databases right from the start.

Grapher has 3 ingredients:

  • Collection Relationship Manager
  • Query as Graph
  • Safely expose data module

SQL has only the first one :smiley: So “a lot” may not be the propper term. Not wanting to pick on semantics but, in real life scenarios grapher covers cases 95%+ cases. If you want complex actions like, count, groupBy, I think it’s weird to give so much intelligence to the Database. In my opinion the central point of intelligence should be in the app and controlled by the app itself.

The reason for this is that your storage layer can be abstracted easily later. So computing things like total calculations like users that bought at least 3 donuts on “odd” dates and they have from at least 3 different donut providers (intentionally stupid example).

This can be modeled into multiple queries and a service that applies the logic to give you the data you need. That service is completely abstracted from the storage layer and it has the advantages of:

  • You can test it in your app, without having to touch the database at all.
  • You can shift the storage layer later without having to rewrite your complex queries

In fact, I believe it is sometimes faster to code it in JS in OO style, rather than have a 500 line long query. (I’ve seen bad things in my life.)

To summarize: MongoDB is dumber than SQL DB but it’s faster + it’s “blended” with JS. One language one love. Cheers.

.

8 Likes

Well, it may be faster but it also comes with quite some drawbacks.

Transactions. Relationships guaranteed by the database. Guaranteed type safety. Guaranteed uniqueness.

Those are not small things.

Transactions:


https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/

Guaranteed uniqueness.
https://docs.mongodb.com/v3.2/core/index-unique/

Guaranteed type safety:
Collection2 + Simple Schema Package

Relationships guaranteed by database:
Why is this a problem? Makes it harder to make mistakes ? I doubt it :slight_smile:

MongoDB gives a lot of flexibility and makes coding much faster. Ability to store hash objects in which you do not know the form without recuring to additional forms of serialization or deserialization. + ability to search and index those “potentialy existing” fields. This is what I call a true database, a db that allows me to do anything I want with it :slight_smile:

I think a mental shift is required. Just like you go from any other template engine to React.

Maybe you are stuck in the SQL era. Not saying SQL is bad, but in my opinion it has way more drawbacks than MongoDB. I’ve worked with both large scale enterprise. And the majority of big apps in PHP tend to go to NoSQL data storages like MongoDB or Redis or DynamoDB because the SQL simply can’t handle it.

At this stage the only advantage I personally see in SQL is that it is more mature and better battle-tested. This will change in 5 years.

Cheers :slight_smile:

7 Likes

The transactions thing is bolted on and doesn’t actually do what a real SQL transaction manages to do.
I mean, your mongodb example of “two-phase commits” reminds me of a Rube Goldberg machine.

Type safety, again, this is application level.

And as for the relationship guarantee: Well, with a foreign key constraint properly set you literally cannot create a link to a non-existent document or, even more importantly, cannot unilaterally delete a document bound by such a restraint.

And this whaffling about how easier it is to store objects “where you don’t know the form” - seriously, if you don’t know what you’re writing to your database then you’re doing it wrong.

I’m also not surprised by “big app in PHP” switching to NoSQL - that’s kind of like the one-eyed leading the blind.

I think grapher is really great and it would be cool to actually talk about that, can we not drag the whole SQL vs Mongo thing into this thread again? If you want, start a new thread.

3 Likes

Well, I’m actually currently trying to try out the boilerplate example.

But the cloned repo is now stuck at Downloading meteor-tool@1.4.2-beta.7

This is on a Ubuntu VM, by the way.