[Idea] Fetch and Carry: A Database-Agnostic Query Tool, and a Data Layer for Meteor


#1

Meteor remains a great way to build applications but has two major limitations out of the box: rendering issues on the client and hard scaling limitations on the server.

On the client, the main problem lies in the data management tools and how they inform Blaze/etc about the changes. On the server, the main problem lies to how closely Meteor tools are coupled with MongoDB and LiveQuery.

While Meteor Development Group is solving the latter problem with GraphQL/Apollo, I believe the initial solution could be far simpler, and most of the parts for it are already with-in Meteor.

Fetch and Carry are a concept of how that solution might look. The two reinforce what Meteor is best known for, which is a fast, simple and practical way to build applications.

Fetch: a Database Agnostic Query Tool

The idea behind fetch is to create a simple way for you to query any database and return the results to the client.

On the server, you would define your queries, along with a permission enforcer and processor that would run each time you fetch a query:

Fetcher.register({
	admin: {
		permission: function () {
			if (Meteor.user().isAdmin) {
				return true;	
			}
		},
		processor: function (query) {
			data = mutateYourQueryAsYoudLike(query);			return data;		
		},
		queries: {
			adminAccounts: function () {
				Database.query({...});
			},
			normalAccounts: function () {
				Database.query({...});
			},
			billingHistory: function (userId) {
				Database.query({...});
			},
			incomeHistory: function (from, until) {
				Database.query({...});
			},
		}
	}
})

On the client, you would run the respective query and get back the data, which you would then store it in your data layer. Here’s one idea of how it could work:

myQuery = Fetch({
	from: "admin",
	query: "incomeHistory",
	parameters: ["June", "August"],
	callback: function () {
		// ...
	}
})

myQuery.state() // returns ["notLoaded","loading","loaded","reloading"];
myQuery.reload() // reloads the query
myQuery.lastUpdate() // returns the timestamp

Compared to Pub/Sub, you would be run be able to run multiple, similar queries but they would all be separated. Compared to GraphQL, you would have to walk a lot less miles to get information from your database into your client.

Perhaps Fetch can be designed in a way that makes it easy to transition to Apollo/GraphQL when the time is right?

Carry: a Versatile Data Layer

Currently, Meteor’s data layer is a state of flux (pun intended). The situation is:

  • minimongo has nice filtering and sorting abilities, but it too closely coupled to MongoDB and LiveQuery
  • reactive-var and reactive-dict hav performance issues with objects
  • reactive-var and reactive-dict do not support sorting or filtering arrays of objects

I’ll go on a leap and say what we all love about Meteor was the reactive programming style. Tracker that makes it possible and not the libraries built on top of it. What Meteor needs next is a new Tracker-based data layer that leapfrogs the limitations of the current libraries.

If Fetch and Carry were integrated, we could perhaps pick up our Fetch query in the client like this:

Template.incomeHistory.helpers({
	data: function () {
		myQuery.get().sort({...}).filter({...});
	}
});

In that case, Fetch would automatically create a new instance Carry. We’d also be able to do it ourselves, perhaps with an API similar to this:

MyData = new Carry("myNewData");

MyData.set({
	toys: [{...},{...},{...}],
	lastUpdated: "04:45 PM..",
	otherItems: {
		idk: ["Chicken"] 
	}
});

MyData.get("toys").sort({age: -1});

The Overall Concept

The overall concept comes two problems:

First, we now have three ways to obtain data with-in Meteor:
1. Using Pub/Sub with the classic Mongo stack
2. Using Apollo
3. Using Method calls

Second, we have a data layer that is becoming outdated:

  1. it is not well adapted to how we retrieve data in Meteor
  2. it causes poor rendering in some cases

The big takeaway is that we need a new data layer built on Tracker that can perform and adapt to all these new situations. If we do that right, it would also solve all the rendering issues that are associated with Blaze.

What do you think of this idea? Are you focused on using Apollo next, or are you looking for alternatives? It would be great to hear what you think.


#2

As far as implementation, I suppose this would be reasonably easy to build on top of Methods and Tracker, thus not requiring much maintenance. I suspect it could be a great way to swap out the Mongo/Minimongo/LiveQuery packages while keeping the Meteor development experience fast and easy.


#3

Yes, with Apollo round the corner, I suppose there would also be a replacement for minimongo.

(BTW, why has MDG not adopted http://www.html5rocks.com/en/features/storage, or https://pouchdb.com/?
Why not just use https://www.w3.org/TR/webstorage/ or https://www.w3.org/TR/IndexedDB/?)


#4

I would make this

And then use the ‘meteor/apollo’ package :grin:


#5

I though @miro is working on blaze apollo integration here: https://github.com/apollostack/blaze-apollo


#7

This sounds nice on the surface @msavin, and you have street cred, but unless/until someone writes the tool, we actually see it in action, a project that we can plug into our existing Meteor classic projects – it’s hard to tell how much merit to give your idea. But I’m open to trying it out if its an easy plugin in play experience for Meteor classic devs like myself.

We’ve yet to see Blaze + Apollo integration and I haven’t migrated to React. I don’t know enough about how Apollo works or the ‘real’ benefits and I don’t know if Reactivity is fully baked. So, I’m on the fence right now with Apollo.

If there was ever a time to introduce something like what you proposed, now, before Apollo really hits the streets, while the community is in “flux”, is the right time. If you wait too long to bring it to market, after Apollo takes hold here in this community, I’m not sure how well it will be adopted – unless of course their is a clear benefit to using it over Apollo – but inertia is a powerful thing.


#8

Added the link to the issue, thanks :slight_smile:


#9

@msavin Max, I feel your concept is very close to Apollo. Could you please explain more how does it differ? As for Apollo I use it together with MobX on the client side and do not use Tracker at all - completely happy with this combination :slight_smile: Actually, the data polled with Apollo is delivered into the observable structures, which could be uses any way you want. I use React for rendering, but the same approach is applicable to Blaze and everything else.


#10

The problem with GraphQL is that you have to define multiple, very sophisticated layers of database queries. In contrast, when you work with a database you just have run a query to get the result.

GraphQL solves some problems for huge applications and organizations; but it’s not for free.

Recently, I’ve been looking at building a classifieds application for a client. In that scenario, MongoDB would be more than enough. I would be crazy to put myself through the work of building it with Apollo.

IMO they are just two different ways of solving a different problems. I lean on the side of simple apps, where something real time and a static database covers what I need and beyond. For “monster” products like Facebook, Salesforce, etc, I could see why GraphQL is big.


#11

The problem with GraphQL is that you have to define multiple, very sophisticated layers of database queries. In contrast, when you work with a database you just have run a query to get the result.

Do you talk about the necessity to write the queries twice, on the client and on the server instead of doing that just on the client side like in simple pub/sub apps?

Recently I had one very trivial project (book store) for my client. There are books, people (authors, designers, illustrators, editors, etc.), and book series entities. For some historical reasons MongoDB was used despite the fact, that the relational database would be more suitable for this type of data. On the client the data from different collections (actually tables) is merged to show book info. Books are organized and sorted by different fields.

You can see that the task is really trivial and the level of difficulty is maybe the same as for everyone’s favorite ToDo List. I have used Apollo both on back- and front-end (mostly to see how good is it for such small tasks). I have found that the major part of the boilerplate is not the GraphQL queries, but the database schema. I am lazy by nature, but here I see no huge difference in size of the boilerplate in comparison to the pub/sub approach. So I would definitely use Apollo and not pub/sub knowing that I have the possibility to change the underlying database without touching the client.

I have to add that Apollo (read GraphQL) helped me to keep the client code very simple, avoiding merging of structures there. The other way to reach the same result was denormalization of the data in the database and keeping it in actual state using hooks, but for me it is more difficult than adding a few simple read queries on the server side.


#12

Use grapher: