I’ve been using iterations of this package in production for over two years and it’s worked pretty well. That said, it still needs a lot of work. Just throwing it out there to see what people think of the API and the general idea.
Very nice idea, but Mongo isn’t designed for transactions. If you find yourself in need of constant transactions, then your collection design is likely flawed. I do see the occasional need for it, but not something I would rely on for stability and atomicity, unless you fulfill the requirements of true atomicity.
How does this transaction guarantee atomicity upon failure mid-transaction without added logic? Connection loss, server dropout at worst possible time etc. Is this a guaranteed atomic transaction? Or does it simply store it on a stack and undo/redo when these commands are called?
Given a classic bank transaction example.
Funds are withdrawn from A
Funds are added to B
Bank transaction log created.
What happens if the DDP connection drops between 1 and 2, or the server goes down completely? Are the transactions stored when the server reboots and then cancelled? If they are not, I’d definitely look in to such functionality!
Yep. I’ve had it running with a 4-instance app for a couple of years and it’s been fine. But, as the docs acknowledge, it’s all last write wins, so there might be occasions when there are near-simultaneous writes to the same document(s) by two users and neither user can successfully undo their transaction because theirs is no longer the last write for one of the documents in that transaction. Like I said, mongo isn’t really designed for multi-document ACID transactions and this package definitely does not guarantee data integrity, although it does aim to give it a fighting chance (most of the time).
Can you tell if this lib offers some real advantage? It is strange nobody is using or talking about considering all the features it promoves:
Distributed Transactional In-Memory Database
Performance and Scalable
Fast in memory data access, up to 25,000 ops (single doc read/write) per shard (each shard take one CPU core).
System capacity is horizontally scalable, performance grows linearly by adding more shards.
No single point bottleneck, all part of system is scalable, unlimited capability potential.
True Distributed ACID Transaction
True ACID (Stands for Atomicity, Consistency, Isolation, Durability) transaction support on distributed environment.
MemDB brings ACID transaction support for MongoDB, on distributed environment! You can get full transaction support of traditional SQL database (like MySQL), while not losing the scalability of NoSQL database (like MongoDB).
MongoDB and Mongoose Compatible
It’s just a ‘MongoDB’ with a cache layer which support distributed transaction.
Directly use of MongoDB’s query API.
Built-in Mongoose support, easy to port existing Mongoose project to MemDB.
High Availability
Each shard is backed by one or more slaves, no single point of failure.
It’s only a bad idea if your database already has transactions. If the database doesn’t support transactions, application level transactions is a time-honored tradition.
This is a fantastic addition to the ecosystem! Thank you so much!
There are no guarantees with this package, but it does give a much better chance of atomicity than just making a bunch of writes would. As it stands, it makes twice as many writes as normal (doing something like a mongo 2-phase commit). I believe I could get it closer to guaranteeing atomicity if the package made 50% more writes but, for the moment, it’s already putting more pressure on the oplog/mergebox than I’d like.
The package was initially designed to bundle up a bunch of writes in a stack and give undo/redo options (method calls) – I just wanted undo/redo buttons for the UX and sometimes this involved writes to multiple documents. After a while, I was encouraged to expand the scope of the package to shoot for atomicity, so in answer to the question: if the server goes down halfway through the writes, the database state is often (not always) recoverable. That’s why I say in the docs “Don’t use it for banking transactions. Seriously, don’t.”. It’s use-case is more for something like: “I remove a post and all its associated comments in one transaction then decide to undo that” – the world’s not going to end if one comment goes missing due to a server crash.
As for collection design flaw … I’d be more inclined to say its a problem with database choice. I don’t want to take a Comments collection and redistribute all of those into arrays on their individual posts in the Posts collection (for a whole lot of reasons), but I do want to be able to remove a post and all its comments and then be able to undo that later.
This might be good – I don’t know enough about it. But unless it’s got a livequery connector, it’s going to be hard to hook up to Meteor and get the same joy we get from mongo.
https://github.com/rain1017/memdb had its first commit in Feb this year. I wrote the first iteration of the babrahams:transactions package back in July 2013, which is why it didn’t inform my research.
It looks like a pretty interesting solution to me (and certainly more robust and scalable than the package I wrote), but it also looks like it would be a significant challenge to make it as easy to use as meteor add membd:memdb.
Useful disclaimer I doubt Mongo or Meteor will be used for a banking app any time soon anyways. Just a few questions you gave a very good answer to! I’m still interested as to how this package scales. Its’ functionality is of course something you won’t do often, but since you worry the oplog is already being pressured, will it work for larger apps? Regardless I’m definitely keeping an eye on this package! Good job!
Also, a good use for this I believe would be a small shift towards system backup. Mongodump and mongorestore are all well and good, but a versioning/timestamp system would without a doubt be beneficial in a development setting. I usually deploy to a demo server for clients a little while before completion because there are always specs to be changed prior to release. Often the client gets carried away with inserting data and you’d like to return it to a previous state. There are migratory tools between versions, but they do not store the data. They just allow you to use some versions so you can mix the collections and their properties up given a schema change.
A daily transaction if you will, in a development stage, or even as a production daily backup (obviously with export) might prove highly useful to many!
Interesting use case - using this package to restore database state after over-keen clients have messed it up a bit. I wish I could say “Yes. No problem.” (I too have a demo site that I want people to play with, but then be able to clean up afterwards.) Unfortunately, that’s not the case.
The way the package maintains a reasonable semblance of database integrity is by not allowing an undo in certain situations - e.g. a document is inserted in one transaction and updated in another transaction - the first transaction cannot now be undone (as then the document would be removed from the database, meaning the second transaction couldn’t be undone). This means that certain transactions have ‘expired’ and are no longer undo or redo-able.
There are certainly be better ways to manage this and, in response to your suggestion, I’ve begun thinking about how this might be done, but I can’t promise big changes anytime soon.
With regard to scaling, I have to give that perennially annoying answer – it depends on the app. If you don’t subscribe to the transactions collection, it’s fairly low impact.
However, if you use babrahams:undo-redo for an easy undo/redo widget, you’ll have a unique subscription to the transactions collection per user, and because this collection gets belted with writes on every transaction, the oplog gets a lot of activity, which then puts massive burn on the cpu as the mergebox tries to update the subscriptions for each of the individual users. For apps with a heavy write load and a lot of connected users, the babrahams:undo-redo package is going to need to use something a bit smarter than the current pub-sub model. Not too hard to implement, I think, just not implemented yet.
will sure use in some oportunity, thanks.
I doubt Mongo or Meteor will be used for a banking app any time soon anyways. Just a few questions you gave a very good answer to! I’m still interested as to how this package scales. Its’ functionality is of course something you won’t do often, but since you worry the oplog is already being pressured, will it work for larger apps? Regardless I’m definitely keeping an eye on this package! Good job! 