Hi!
I will be running a nightly cron job to query a collection and then send results to another system.
I need to sync this collection between two systems.
Documents can be removed from the host and this deletion needs to be reflected on the client system.
So - my question is, is there a way to query for documents that have been recently deleted?
I’m looking for something like db.Collection.find({RECORDS_THAT_WERE_DELETED_YESTERDAY});
I was reading about parsing the oplog. However, I don’t have one setup yet. Is that something you can introduce into an existing DB?
I dont believe there is a “magic” way of doing this. You could do it with the oplog, which you can easily enable (any replicaset will have an oplog - if you’re running it with a standlone, you can google for tutorials for your specific version, but it’s easy).
If you do it with an oplog, you must make sure that your oplog is large enough that the data you want to find will exist in the oplog (it’s a capped collection). Your query will look something like db.oplog.rs.find({ ns: "dbName.collectionName", op: "r" }).
This isn’t really how the oplog is designed to be used though - so it will be a quite inefficient query. Instead, you could use the collection-hooks package, and insert a document into a different collection as part of a remove hook. You could also update your documents to make them inactive (or flagged for deletion, or whatever, you would need to modify your queries to ignore the inactive documents, but your nightly cron could clean these up) rather than removing them.
1 Like
I have a similar need; I simply use…
$set: { deletedAt: new Date() }
…to mark the documents as being logically deleted. The task that cares for them can easily query those, perform whatever needs to be done, and subsequently permanently remove them.
2 Likes
“Soft-delete” option seems to be the easiest/cleanest soln. Not overly concerned with storage atm and I’m sure at some point I can actually remove and archive the soft-deleted records.
1 Like