Database drivers without client side databases


#1

So I’ve hit the limit with Mongo and need a relational database. The biggest problem with integrating a new database is building a client side version from scratch. This seems infeasible if we want to support a range of databases, so I’ve been doing some thinking about how we could use a database without a client-side mini-version.

Most people building larger production Meteor apps end up using Meteor.methods as opposed to client-side insert, update, and remove with allow/deny rules. Using this pattern, they also end up repeating the same search queries in their Blaze helpers as in their publications. So here’s my idea:

A pub/sub is no different from a reactive query. When you subscribe to the database, you’re returned a subscription handle that has a reactive ready function and a stop function, just like a normal subscription. In addition, it also has a reactive variable (or cursor) for results.

On the server, a publication sends all the relevant data to the client. When the data changes, the server pushes the changes to the client and the results are updated.

From the client, you can use sub.results.get() in Blaze helper or in any reactive function.

In the most simple implementation, results is just a collection of results from the query. In a most sophisticated implementation, results would be a Cursor object that manages added, changed, and removed for each document returned from the query.

The beauty of this sort of implementation is the simplicity. The only challenge is monitoring changes to the database, but this should eventually be the responsibility of the database, and we are seeing that.

That said, the only real issue here is latency compensation. I don’t this it would be too hard to handle this manually. Typically when you are performing some action, you know exactly what its going to change. Thus, you can update the sub.results as you see fit, and let the server update the change.

Anyways, just some ideas. Let me know what you think…


#2

You are kind of looking at a ORM kind of thing and make the minimongo as the client side API. I see that as good point.

Some more discussions: https://gist.github.com/mquandalle/32162e20a87c391771ed
But we couldn’t make it a real.

Also, as a side note there is a WIP rethink driver from @slava. See: https://gist.github.com/mquandalle/32162e20a87c391771ed


#3

Hmm. The whole mini-everything just seems like quite a pain… I’m curious how this will be handled…


#4

I’m currently thinking of connecting my Meteor back-end to a SQL Server instance.

Now, for my purposes that connection doesn’t need to be reactive in and of itself. But I was hoping I could just have a middle man SQL-Mongo translation module or SSIS package of sorts, with SQL Server writing its transactions into Mongo and Meteor doing the same reversed.

It would mean it’s not quite as reactive as one would like, but I wonder how difficult this could be?


#5

Well, @batjko, today is your lucky day! What timing:


#6

@ccorcos That looks interesting.
I’m only a beginner with Meteor, but I’ll definitely play with this thing.
Do you have any suggested mssql drivers I should use this with?


#7

I haven’t built one yet but it shouldn’t be too hard. Check out my package ccorcos:neo4j as an example of what you’d need to do. numtel:pg and numtel:mysql are pretty close to working with this as well. I just haven’t had time do that yet.

I think what will be really exciting is integrating with rethinkdb and redis with database-supported reactivity rather than how Mteor does it in Node