Grapher 1.3 has been released

announce
1

Hello Meteor community,

After a lot of work, we managed to release Grapher 1.3. I chose to make a separate post out of it, since I want this focused solely on this 1.3 release.

I believe that currently, what Grapher offers is jaw-breaking in terms of performance. Click here to learn more about Hypernova. It outmatches relational SQL databases in terms of bandwidth (fact!) and it will also outmatch it in certain scenarios also.

My guess is that it’s faster than a traditional SQL+ORM (Let’s say Doctrine) on top of it. However, I know that big claims need to be backed up by big evidence, and if someone wants to step up and prove me wrong (so I can focus on coding and other stuff) I’ll be very happy. If not, I will eventually test this hypothesis.

If you are already using Grapher 1.2, read the MIGRATION.md file. It should take you less than one hour to adapt your code base.

We’ve introduced denormalization, result caching, resolver queries which will replace all your Meteor methods that fetch data, we’ve cleared all the bugs, and we improved performance even more. Alongside that, we created a new documentation aimed to walk you through Grapher in a natural way, and showing all the goodies it has.

We’ve opened to path to use Grapher outside Meteor in React Native apps or any other language (PHP, Python, Ruby, etc)

We cleaned the design of Grapher Live, making it more pretty to look at, and made Grapher React use the HOC pattern recommended by Facebook + included with nice displayNames for easy debugging. As a side note, Grapher works with Redux.

Grapher can work easily with Redis Oplog, giving you absolute power when building Meteor Apps.

Special thanks to @herteby who made this awesome package herteby:denormalize which enables Grapher to become to most efficient data layer for MongoDB yet. And also for his integration with Vue (https://github.com/Herteby/grapher-vue) and thanks to you all for the encouragements and nice words.

I firmly believe that this is the start of a revolution in the way you think about data. I also think Apollo will have it’s place and it cannot be replaced by Grapher, because Grapher has it’s roots deep in MongoDB, but you can have reducers (leaves) or resolver queries, that communicate with any API.

The next probable steps:

  • Achieve a test coverage of 80%
  • Invest in security (maybe hire a consulting agency)
  • Move Grapher-Client to NPM
  • Abstract Hypernova so it can be used by Apollo, other clients
  • Some other features that will be requested by the community

Enjoy!

PS: We’ve also opened https://opencollective.com/grapher/ . All contributions will be redirected to Grapher in form of bounty hunts.

37 Likes
2

I have made some shocking discoveries.

MongoDB 3.6 has been recently released, and if you don’t know, it offers a an $aggregation pipeline for lookup. Meaning you can do joins as you need them, and nested, basically what Grapher already does using Hypernova, however, I figured using native “joins” it would make it much faster.

As soon as I got the binary I set it up on my machine and tried it, because I wanted Grapher to be on the bleeding edge, and give people ability to opt-in into this if they don’t have Mongo 3.6…

This is the query I tried to get users, posts, and comments of the posts. (Post has ‘ownerId’ => User, Comment has ‘postId’ => Post)

db.getCollection('users').aggregate([
    {
        $project: { _id: 1 }
    },
    {
        $lookup: {
            from: "posts",
            as: "posts",
            let: { id: "$_id" },
            pipeline: [
                {
                    $project: { _id: 1, ownerId: 1 }
                },
                {
                    $match: {
                        $expr: {
                            $eq: ["$ownerId", "$$id"]
                        }
                    },
                },
                {
                    $lookup: {
                        from: 'comments',
                        as: 'comments',
                        let: { id: "$_id"},
                        pipeline: [
                            {
                                $project: { _id: 1, postId: 1 }
                            },
                            {
                                $match: {
                                   $expr: {
                                       $eq: ["$postId", "$$id"]
                                   }
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
]);

The execution time for this is 411ms.

The same query {users: { posts: { comments: {} } } }… from Grapher’s Hypernova took 19-23ms. WTF! If someone can shed some light into this I’ll be very happy. 20x speed is flabbergasting, something may be wrong with their release, I’ll be waiting for some patches or 3.6.1 to re-test this, or maybe some of you can help me understand this.

If you want to test this for yourself: https://github.com/cult-of-coders/grapher-boilerplate it already contains fixtures for the linked data, once you start it, it will populate your db and you can run the query above. And with Grapher Live (which is also installed) you can inspect that query.

Note that Grapher and the aggregation above were run on the same database with the same indexes.

Any ideas what I might be doing wrong? 20x faster is shocking.

10 Likes
3

Made a mistake in the code above (fixed it now), the performance is actually 20x faster with Grapher.

6 Likes
4

This is really cool! 20x faster!!

5

As said many time on the other thread, I will move Orderlion over to Grapher very soon! 20x performance is impressive! backed!

1 Like
6

Thank you @klabauter . In my vision Grapher’s performance is a secondary feature, the main feature is ease of use and development speed. That’s the main focus.

7

Indeed I agree - I know Mongo is a NoSQL db, so no relations, no joins and so on, but still - I think in every proper app one needs joins and things like that to have a proper db structure. So Grapher comes in very handy right there. I am curious how long it will take me to transfer all my vanilla withTracker() react code to Grapher, but I hope in the end a lot of my calls will be much easier. :slight_smile:

8

I don’t suppose you’d document your migration?

9

Just one question @diaconutheodor, can I use the Astonomy package and your Grapher package at the same time ?
Or theses two packages doing the same things ?

10

This is a pretty cool discovery. Two thoughts on it:

  1. It could be that MongoDB rushed to implement this feature for marketing purposes, and plans to improve it at a later date. This would be in tune with how they operate.
  2. While Grapher provided better performance, I’m not sure this would apply to a production use case. Grapher has to make multiple requests, whereas using aggregate lets you do it in one request.

Considering network latency, the performance of a database under heavy load, and MongoDB potential to improve, it might be better to align with the aggregate feature.

Have you thought about opening a support ticket to let MongoDB know about this issue?

1 Like
11

Yes you can. You can use grapher alongside any library.

12

Agree. This is why Grapher can harness the benefits of aggregation pipeline. Currently its too much, 20x is huge even if network latency comes into picture, the roundtrips are not heavy bc connection to db is via socket. I will keep an eye on mongo updates, i am sure they can make it more performant so i am open to progress so we can use this nice feature in grapher.

13

i think, this is a known issue.
https://docs.mongodb.com/manual/release-notes/3.6/#known-issues-in-3-6-0
https://jira.mongodb.org/browse/SERVER-31760

14

I just read through the documentation for grapher… very cool and great documentation. Thank you very much for offering this to the community. :+1: I have a quick question for clarity.

I understand that you setup links in your collections to define how collections link to each other. The question I have is related to the Linker Engine - Settings Links section of the documentation… are we to understand that after defining the links in your collections, then after inserting a doc, that you then must also set the link each time you add a document? So that it links correctly?

I was assuming that after adding a “link definition” in your collections that when inserting documents, grapher would automatically take care of actually setting the links and populating the internal link collections.

Maybe I’m missing something, but this section of the documentation confused me. If we do have to add the links upon document insertion… it seems like this could be something done automatically within grapher with an internal hook or something no?

Also, what did you use to generate the grapher documentation static pages… very cool and easy to read. Is that a Github Pages Jekyll theme?

15

Linker Engine is what powers Grapher behind the scenes. You can live without it and you can store links

Comments.insert({ postId: 'XXX', text: 'Hello' });
// equivallent, provided there's an inversed link to comments
Posts.getLink(postId, 'comments').add({text: 'Hello'});

It just offers an API for storing links inside the database, you can still do it manually if you prefer, but you will notice in time that after you used it 2 or 3 times it will be much more natural.

Nope, it does not matter.

It can help you with that, but ultimately it’s your choice how and when you store these links. Grapher is a fetching layer, but it offers this small API (Linker Engine) to play with links.

This is thanks to @herteby, he coded it in Vue, look in docs/index.html

1 Like
16

There are no “internal link collections”.

If you for example want to link your Posts to their authors, all you have to do is:

Posts.addLinks({
  author:{
    type:'one',
    collection:Meteor.users,
    field:'authorId'
  }
})

And then, if a Post has a user id in its authorId field, and you make a query that includes author, you’ll get that user :slight_smile: If the field is empty or the id doesn’t match any user, then the post in the result won’t have an author.

2 Likes
17

Great job on this package, will definitely use it for Sounds Social!

1 Like
18

Guys, can you comment on this PR : https://github.com/meteor/guide/pull/713 so we can get Grapher into official docs ? Maybe if some of the community members can tune in, there’s a higher chance of it getting merged. Thank you

19

Clearly the community support is lacking, hence no comment posted in https://github.com/meteor/guide/pull/713

GraphQL Bridge has been released: https://github.com/cult-of-coders/grapher/blob/master/docs/graphql.md

You can now benefit of Hypernova on GraphQL queries, wonderfully blended. Cheers.

2 Likes
20

I think the issue with the community support is because the package is targeting a niche within a niche. By that I mean, from all the Meteor apps being developed, a small subset of them will reach enough use cases that would require the need for Grapher and out of those smaller will adopt Grapher. I think the meteor guide should keep things minimal to minimize the number of concepts new comers need to learn, yet with that said, I also see your point of getting people aware of it’s existence for those who needs it.

Cult of coders are developing multiple great libraries and solutions which are targeting advanced use cases within the Meteor community perhaps you can have a landing page of all those packages and their use cases?

1 Like