I have been using Meteor for over a year now but I always feel like I am “doing it wrong” when it comes to error handling inserting to multiple collections with Meteor Methods. I can’t seem to find anything in the guide or on the forums that truly clears this up for me, but maybe that’s because I am misunderstanding it at some fundamental level? Apologies if I have missed something obvious.
Imagine a (presumably very common) scenario where the client calls a Meteor Method that must insert an entry into Collection A. If Collection A succeeds the next step is to insert a related entry into Collection B. If the Collection A insert fails, then the Collection B insert should NOT happen (easy enough). If the Collection B insert fails, then the new entry in Collection A is considered in a bad state and we need to do something about it. One option is to remove it manually I suppose. Is that the standard approach in Meteor? I suppose, if this scenario only happens once or twice in the app, it’s okay to just manually remove it, but ideally, we don’t want the overhead of doing inserts and removes to the database as part of the error handling (not to mention it just feels wrong - what if the remove fails for example?). What we really want is transactions where the db changes only happen at commit time at the end of the method, and any errors during the method will cause a rollback, right? This is just taken for granted where I come from in Java Enterprise Land, but from what I can figure out transactions aren’t supported by MongoDB, or at least not the version used by Meteor? I’m not really 100% sure, it’s all very confusing. Please correct me if I’m wrong about Mongo and transactions.
Any help/opinions/general feedback on how the above scenario is typically dealt with in Meteor would be greatly appreciated, thanks.
Hmm, now I’m wondering if the lack of responses is because this is a bad question somehow? If so, can someone please let me know and I’ll try to improve it or come up with a better example.
Even if the answer is as simple as “Meteor doesn’t support transactions” or “there is a package for that”, that’s fine. It would help to know either way. The whole reason for posting was to specifically discuss how other developers deal with this kind of scenario, and maybe talk about adding something about this to the guide to improve the experience for newcomers to Meteor.
NoSQL, joins and atomicity. Not the kind of love story that everyone likes to dig in. Fortunately, there are patterns for that since a long time ago. Before SQL was even a thing, actually.
To keep it straight, if you need a transaction pattern, you have to emulate it.
NoSQL obtains great performance and easy sharding because it doesn’t use transaction when not required. Unfortunately, it comes with a price: it’s up to the developper to create the transaction pattern when its data model requires it.
Thanks for the response and the link @pierreeric - I’ll definitely check out that book.
I am not interested in bashing on Meteor/Mongo for its lack of transactions or rollback. I appreciate that NoSQL comes with it’s pros and cons just like everything else and I’m fine with that.
I’m just curious what Meteor developers typically do in a situation like this as I think it’s very common, even for trivial apps, right? Do they actually remove bad entries if there is an error or does everyone have to go to the effort of simulating rollback? Or maybe the standard approach is to allow the data to go bad, then have background admin processes that find and remove them at a later stage? That sounds like a good solution to me actually, anyone else agree?
For me, this is not about supporting “enterprise” or dealing with an edge-case scaling issue that only happens if your app takes off and gets thousands of users. I stumble across this scenario in nearly every single non-trivial app I write in Meteor, so it’s a general problem IMO.
I think many developers coming from a Java (etc…) background will be surprised to find no discussion of this at all in the guide and it might put them off. A clear explanation of it and maybe a link to that book would be far better than just ignoring the issue and hoping people won’t ask about it, right?
On my apps, I strongly favor decoupled models with as few joints as possible because that’s where NoSQL shines.
But, hell, yes, sooner or later, you need a bit of normalization wether it be for easing the queries or the display or being DRY. In this case, it all depends on the rate of changes of these coupled data. Most of the time, I’m lazy. I don’t even bother with atomicity. A shameless truth. I favor async idempotent post treatment.
I’m surprised that you mention that theses discussions are treated by Java developers. I’ve been one of them (and in many other types of team). For me, these questions and discussions are not tied to the language nor the framework. As you will even see it, the example in the book that I’ve mentioned are in Python. Discussions on data model should be agnostic to the used language and more related to the persistence tiers that has been chosen for a project. We had these discussions with my teammates whatever was the chosen application tiers or the language(s).
I agree that decoupled models with as few joints as possible is a good thing to aim for but I would argue that this discussion isn’t about model structure or relationships at all. It’s about any Meteor Method that performs more than one database update within the same method and I am sure that happens very often in most Meteor apps, right, regardless of collection relationships?
For most of my apps, I honestly don’t know how I would split the Meteor Method so it only does one DB update - the code I am writing demands more than one update - nothing I can do to change that fact. Either I do the two updates in the method, or I split it into two methods and get the client to call both, but then I still have the same problem - potential for a bad entry to exist in the database if the second method fails.
I should have mentioned The Spring Framework when I mentioned Java. In my professional experience, most of my Java-based clients have used Spring or some similar framework and they all provide transaction support out of the box. Whenever I am coding in the data layer, I know I can make multiple database calls and throw exceptions whenever I need to without thinking too much about it. Of course, in some specific cases, you need to tweak transaction boundaries and settings, but in general, it’s got your back.
Now, it’s important to note that I am not complaining about the lack of this feature in Meteor. I get it - MongoDB doesn’t do transactions, and I’m okay with that. I knew that before I started using Meteor and it was my decision to invest in it regardless. I started this discussion to find out more about how this kind of thing is commonly handled in Meteor, given the lack of transaction support and because I always feel like “I am doing it wrong”. I suppose - in a way - I just wanted some reassurance that I haven’t missed some fancy approach that everyone knows about except me!
I think the fact this isn’t discussed much (if at all) in the guide is one of the main reason why I felt like I must be missing something obvious. If anyone from MDG is reading, what are your thoughts on adding a section to the guide to specifically cover this?
I am so glad you are asking about this. I have exactly the same question. How common is it for one-to-many and many-to-many relationships get out of sync in MongoDB – and what techniques are used to prevent that from happening?
I use this excellent package in my Meteor application (full disclosure : I have contributed a few PRs to this package in the past).
The README is very informative, but , as ever, caveat emptor.
There is a performance penalty to pay to use something like this in increased number of reads before each write, so you will just have to evaluate if the value of pseudo-transaction protection outweighs the additional database hit.
Thanks @vikr00001 for getting more views on this and thanks for the link @rsbatech.
I did take a look at meteor-transactions a while back and to be fair it does look like a well made package. However, not to take anything away from the library itself (I appreciate the author taking the time to point out the warnings and issues with transactions in Mongo), but I did get the general impression from the slightly negative tone about Meteor transactions in the documentation that this isn’t a general solution and deciding whether to use it or not shouldn’t be taken lightly.
But even if this package is a good solution for some scenarios, I would still like to know the answer to this question:
Is it generally acceptable to make multiple MongoDB calls within a single Meteor Method or do we all have to implement two-phase commit (either ourselves or by using a library such as meteor-transactions)?
And I’d just like to reiterate, it’s not a problem for me if the answer is: “You can’t make multiple database calls in a single meteor method”. That’s fine - I am just saying it should be made clear in the guide.
To get an answer to this once and for all, the next step is to start analysing code in GitHub to find out for myself how others do it. I plan to do that - but haven’t had the time so far. I’ll update this thread with results when I finally get round to it.
You can definitely make multiple MongoDB calls inside a single Meteor method. There’s nothing wrong with that.
Have you considered using a library such as Astronomy to solve the issue of one-to-many relationships? e.g.:
const transfer = new Transfer();
// some loop where several transactions are fetched or created:
try {
transactions.forEach(tx => {
// do things. if we hit an error, we bail out and no harm done, because
// transfer.addTransaction() doesn't actually save anything to the DB
transfer.addTransaction(tx);
});
transfer.save(); // now we're good, and we save to the DB
} catch (e) {
console.error('stuff went horribly wrong');
}
I’ve heard of Astronomy but never really looked into it - thanks for the link, I’ll take a look. But when you say “nothing wrong with that”, do you mean - “nothing wrong with that, as long as you use something like Astronomy to handle error scenarios using their transaction support” or are you just saying Astronomy is one option, but there is nothing wrong with multiple db updates even if you don’t use it?
I just don’t see how it’s possible to make two database writes in a single Meteor Method without using some kind of transaction based commit/rollback mechanism to handle errors. It’s not limited to a one-to-many relationship issue for me - the method might be dealing with two completely different collections in many cases. Or are we saying that a method shouldn’t do a write to unrelated collections? Is that bad form or something?
I feel like I’m missing something obvious with this. A simple example of how to handle 2 db writes in a method without using any kind of 3rd package would be extremely helpful to help clear up any I maybe have on this.
Off the top of my head, this might be one approach to a 2-stage DB save in a method:
Meteor.methods({
/*
* Saves file info & metadata into Uploads, and writes AWS S3 information in
* a separate collection.
*/
saveUpload(uploadData) {
const {
filename,
mimeType,
...s3Data,
} = uploadData;
let uploadId = null, s3RecordId = null;
try {
uploadId = Uploads.insert({ filename, mimeType });
// some other things that might fail
// ....
s3RecordId = S3Records.insert(s3Data);
} catch (e) {
if (uploadId) {
// We inserted an upload, but failure occurred afterwards - roll it back
Uploads.remove({ _id: uploadId });
}
if (s3RecordId) {
// Same for this
S3Records.remove({ _id: s3RecordId });
}
throw new Meteor.Error('upload-save-fail', 'Upload save failed');
}
// Success!
return uploadId;
},
});
Again, I just threw this together without much thought and no actual testing. Someone can feel free to point out any mistakes or improve upon it!
Thanks for the code example @ffxsam. So this is the “manual remove” option that I asked about in my original post. I’ve done exactly this in my code but I couldn’t shake the feeling that I am doing it wrong, hence the reason I posted in the first place.
I have two issues with this style of code. Firstly, the removes are not protected so they could still potentially fail. Secondly, your code demonstrates recovering from an insert, but what if the first database call is an update to an existing record? Do we try to restore the old value in the catch block using another update? And what happens if that fails?