Update many records with a cronjob service

#1

Hello everyone,

I have an issue, and I want to ask everyone to have the right solution.

Situation

I have a projects collection. When I change name of any project in this collection, I need to change all data refer to this entity in other collections (eg. this project name in collection user, user_activities, etc.). The number of records need to be updated may be very large and it takes a lot of time to update.

My Solution

I need an asynchronous service running on server side to update all these collections later. I also look for some packages to execute this task and meteor-synced-cron seem to be fit the requirement.

My Question

  • Is my solution right in this situation?
  • I also want to build (and run) this service independently with my current Meteor project. How to design this service?

I really appreciate any help you can provide.

#2

You might want to look into Change Streams, MongoDB has introduced this feature in version 3.6. I ended-up using it for similar use case.

In a nutshell, you can open a change stream on any collection and listen for specific events (insert, update, delete, replace) in real-time.

In your case you can you can open a change stream on projects collection and listen for the changes and propagate those changes to other collections (users, user_activities ).

This could be part of a meteor app or it could also run as a separate app.

Also, I should mention that Change Streams only works with replica-sets. So for development and testing one also has to setup a replica-set.

1 Like
#3

Hello @distalx, thank you for your suggestion, I am researching about Change Streams now.

1 Like
#4

TBH, I’d change the situation. If you used the project’s _id wherever it was referenced, instead of the name, you’d only need to change it in one place. There are several packages/techniques you could use to manage this sort of relationship (grapher, aggregations, publish-composite, etc.).

Large-scale database mutations of the sort you’re suggesting can cause problems for oplog tailing and dramatically hit app performance.

7 Likes
#5

Thank you very much for helping me @robfallows. I am researching about the packages you mentioned. I will be back when having the result.

1 Like