Hyperdex, hyperdex-livedata, minidex?


#1

As of version 1.6, HyperDex has a Mongo veneer and is ACID compliant. Lets talk about what would be needed to get it into the Meteor ecosystem, and if it makes sense to do some experimentation.

We have three core packages which are involved in implementing Mongo:
mongo
minimongo
mongo-livedata

And we have an additional two packages from the Redis implement that can be used for compare/contrast.
redis-livedata
miniredis

In theory, we should be able to clone the mongo packages, do a hand-wavy search-and-replace of ‘mongo’ with ‘hyderdex’ and be about 80% of the way done to adding an ACID compliant datastore that’s compatible with the Mongo API.

Thoughts?


#2

AFAICT you wouldn’t get oplog tailing, right? So livequeries would become pretty inefficient. On the other hand, maybe your app doesn’t need them, so YMMV.


#3

Looks interesting. Why HyperDex in particular?

Have you looked at TokuMX? They apparently provide MVCC for the MongoAPI. We’d only need to re-write mongo-livedata to support their different oplog format.

Also, it looks like RethinkDB is next on the horizon - as it has a built in “livequery” engine.


#4

We did some tests with TokuMX. It’s good and gives better performance than 2.6. But, mongodb 3.0 with WiredTiger is just like the same in performance.

Oplog issue is still there.


#5

I proposed Hyperdex here Why MongoDB is unreliable so glad you like it.

From another comment:
TokuMX only fixes the data loss problems in the mongo replication design (by tying acknowledgements to election terms a la raft). TokuMX doesn’t fix the stale/dirty reads problem.

Mongo clients just don’t expect to verify that the server they’re talking to is actually the primary; without changing clients or adding a lot of server-side logic to proxy for clients, I don’t think this is fixable in the replication design mongo uses and TokuMX inherited.


#6

No oplog tailing, as far as I can tell. Or replica sets or fault-tolerance, for that matter. But I’m thinking of specific use cases along the lines of storing aggregation data and rolling averages. If Mongo is the bookshelfs of the library, then using HyperDex would be a card-catalog system of sorts.

@nathan_muir - thank you very much for sharing TokuMX. I wasn’t aware of them, but that looks like a very interesting solution also.

Why HyperDex in particular? No exact reason; but lots of small reasons… the Mongo veneer API and ACID compliance being the two primary ones (and which makes TokuMX also suitable for my needs).

Also, over the next few months, I’m going to be moving into shard testing of various bioinformatics datasets; and will probably be hacking on the mongo, minimongo, and mongo-livequery packages. Writing a hyperdex, hyperdex-livequery, and minidex package would be the kind of exercise that would require me reading through and understanding the existing mongo drivers, and getting my knowledge of livequery up to the point where I could then hack on the mongo packages themselves.


#7

The source for the entire core, which includes everything except the transactions, is available for free.

HyperDex licensing


#8

More from HackerNews:

Hyperdex is the result of their ongoing research at Cornell University. They have a commercial spinoff which sells Hyperdex Warp which adds full ACID transactions on hyperdex. So if you don’t need full acid transactions, you can use the OSS version, if you want the extra services you have to pay.


#9

Well, there goes the enthusiasm for that. Hrm.

(Thanks for digging out that licensing info, dandv. You may have just saved me a day or two worth of work.)


#10

Update: HyperDex licensing discussion on GitHub. Thanks @awatson1978 for jumping in.