Scheduling Emails

help
1

Hi, I’m working on on this app and wanted to get some thoughts on if my approach is sound or if another option is more viable.

The goal is to let users schedule emails to send out at a later specified date/time…

My basic idea on tackling this problem is to:

  • Create a collection for the “scheduled emails”.
  • When an email is scheduled by a user, it would call a server method to:
    • Add the email object (to, from, etc…) to the collection
    • start a Meteor.setTimeout() ?
  • When timeout completes, it would send the mail and remove the item from collection.

There would also be a piece that would look if there are items in the collection on start up ( in case of crash or whatever ) and create new timeouts for each.

I’ve never done anything like this so I’m not sure if timeout is the right approach or how to accomplish that particular piece. Setting day long timeouts seems… I dunno bad? Thoughts? Thanks!

2

That’s what I’m doing right now. Although in my case the emails are sent 5 mins after the event. Day long timeouts does seem excessive.

I’m also curious to see what other people come up with.

3

You could also add dates and check if the time matches

if (setDate == Date.now() ) {
sendMail()

}

4

And when will that run?

5

Sorry, new to JS.
Aren’t there watchers to check when something becomes true?

edit: just checked StackOverflow and it seems the only way to do it is with setTimeout, but what if you have hundreds of those that run for days??
How is this solved in JS?

6

Thanks for your input - this sort of check will definitely exist to send the email but it would still need to be triggered at the appropriate time.

7

The more I think on it the more I think that it would be safer to just implement a polling mechanism.

i.e. have a single interval running every X ( X = arbitrary ) duration that is set to look at the collection’s results that are expiring by next interval tick and register those to an array of results to “process”. Processing would require to validate that the email send date/time wasn’t updated in the duration, but that’s trivial.

Delaying the emails X minutes might be a premature optimization, but would be less error prone on start up and more maintainable in the long run. It would also allow easier control over this piece of functionality depending on server load/scaling need… ( i.e. setting a threshold and slowing down the interval if server load is high )

If there are any other ideas let me know, still in charted territory :wink:

Best route forward might to just get started and see what sticks and what doesn’t…

1 Like
8

In your case I’d resort to polling. We implemented something like that in a project and it worked fine (there’s something to be said about simple solutions that work). On server startup we fired up a setInterval to run every X minutes and check if there was anything to process.

9

I’d look at implementing a more feature complete queue solution such as https://github.com/vsivsi/meteor-job-collection

1 Like
10

I use percolate:synced-cron for a similar task. Something along the lines of this would work:

// on the server
SyncedCron.add({
  name: 'Query the scheduled emails and send if necessary.',
  schedule: function(parser) {
    return parser.text('every 10 mins');
  },
  job: function() {
    Meteor.call('sendScheduledEmails');
  }
});

// on the server
Meteor.methods({
  sendScheduledEmails: function() {
    ScheduledEmailsCollection.find( { sent: false, scheduledTime: { $lte: new Date() } } ).forEach( function(email) {
      // send the emails here and mark as sent
    });
  }
});

If you don’t want each email send operation to block, you can further refactor the code so that you can use this.unblock() within your methods and fire off emails almost concurrently.

6 Likes
11

Awesome thanks, I’ll check it out.

12

Thanks for linking, I wasn’t aware of this package. Looks very promising!

13

oh and by the way I got curious about what would happen if the email load grew and how I could offload it somewhere else.

It turns out, there is a great solution: http://blog.mailgun.com/tips-tricks-scheduling-email-delivery/

14

With percolate:synced-cron you’re still using timeouts :smile:

15

You are most definitely correct.

Yet, compared to handcrafting a quick timeout, this has the added benefit of

  • running and persisting gracefully across restarts
  • making sure that only one process runs a given task when there are multiple meteor server processes
  • keeping a readable log of the jobs’ statuses
  • reporting back to you over the standard collections interface
  • bonus: I love the later.js syntax :smile:
16

The general solution for executing work in a scalable way is to use a job queue as @rhyslbw mentioned. Probably overkill for your current use case but at some point you may need a better guarantee that the job is actually executed. In that case, I’d recommend reading up on message queues and exploring HA architecture a bit (keeping in mind that pragmatism is king). :slight_smile:

RabbitMQ and Amazon SQS are two popular, highly-available, durable queues which can handle delayed messages. Here’s some relevant links for SQS:

2 Likes
17

have you seen this http://richsilv.github.io/meteor/scheduling-events-in-the-future-with-meteor/

18

If I understand correctly, all the talk up to this point has been about sending emails in Meteor (albeit after a delay). I will try to be brief.

Although percolate:synced-cron solves what you are trying to do, it is not a good idea to have Meteor send the emails in the first place.

You should move sending emails to a separate process. It can be Meteor, but it should not be a user-facing process.

Here’s why. You know how it’s always said Node is non-blocking? That’s a big fat lie (well, let’s soften it up, more of a marketing thing). Give Nodejs or Meteor one big fat calculation that takes five seconds, neither would be able to do anything (that includes fetching any data from anywhere, answering incoming connections, anything). It has to finish that calculation before anything else.

What this translates to is this: Even when you are using multiple cores with Meteor, every time you are sending and email from inside Meteor, you are blocking the Meteor process. You should not do that. That’s a bad idea. You should send the emails from another process.

It can be written in any programming language, including Meteor. You could start a separate Meteor process that does not listen in on incoming HTTP or DDP connections (I remember reading about this but I couldn’t find the link) and you would still be using Meteor, although not the same Meteor app.

But the main idea is to not send the emails from the web server.

It would be the same even if you were using PHP. Do not send the emails from the client-facing web app. Use a separate process.

19

You are correct, but we are already advised to wrap long-running code within server methods and use this.unblock() so that meteor’s runner does not hold up the queue.

That means, your email code can be fired up and subsequent methods can begin running instantly, without blocking the server process.

Also, meteor’s use of fibers does allow for a more relaxed architecture than a traditional nodejs server thread.

20

I would do the email sending in a separate process anyway. I would not take the risk when I am sure a separate process would work no matter what. My two cents.