Scheduling Emails

help
4

And when will that run?

5

Sorry, new to JS.
Aren’t there watchers to check when something becomes true?

edit: just checked StackOverflow and it seems the only way to do it is with setTimeout, but what if you have hundreds of those that run for days??
How is this solved in JS?

6

Thanks for your input - this sort of check will definitely exist to send the email but it would still need to be triggered at the appropriate time.

7

The more I think on it the more I think that it would be safer to just implement a polling mechanism.

i.e. have a single interval running every X ( X = arbitrary ) duration that is set to look at the collection’s results that are expiring by next interval tick and register those to an array of results to “process”. Processing would require to validate that the email send date/time wasn’t updated in the duration, but that’s trivial.

Delaying the emails X minutes might be a premature optimization, but would be less error prone on start up and more maintainable in the long run. It would also allow easier control over this piece of functionality depending on server load/scaling need… ( i.e. setting a threshold and slowing down the interval if server load is high )

If there are any other ideas let me know, still in charted territory :wink:

Best route forward might to just get started and see what sticks and what doesn’t…

1 Like
8

In your case I’d resort to polling. We implemented something like that in a project and it worked fine (there’s something to be said about simple solutions that work). On server startup we fired up a setInterval to run every X minutes and check if there was anything to process.

9

I’d look at implementing a more feature complete queue solution such as https://github.com/vsivsi/meteor-job-collection

1 Like
10

I use percolate:synced-cron for a similar task. Something along the lines of this would work:

// on the server
SyncedCron.add({
  name: 'Query the scheduled emails and send if necessary.',
  schedule: function(parser) {
    return parser.text('every 10 mins');
  },
  job: function() {
    Meteor.call('sendScheduledEmails');
  }
});

// on the server
Meteor.methods({
  sendScheduledEmails: function() {
    ScheduledEmailsCollection.find( { sent: false, scheduledTime: { $lte: new Date() } } ).forEach( function(email) {
      // send the emails here and mark as sent
    });
  }
});

If you don’t want each email send operation to block, you can further refactor the code so that you can use this.unblock() within your methods and fire off emails almost concurrently.

6 Likes
11

Awesome thanks, I’ll check it out.

12

Thanks for linking, I wasn’t aware of this package. Looks very promising!

13

oh and by the way I got curious about what would happen if the email load grew and how I could offload it somewhere else.

It turns out, there is a great solution: http://blog.mailgun.com/tips-tricks-scheduling-email-delivery/

14

With percolate:synced-cron you’re still using timeouts :smile:

15

You are most definitely correct.

Yet, compared to handcrafting a quick timeout, this has the added benefit of

  • running and persisting gracefully across restarts
  • making sure that only one process runs a given task when there are multiple meteor server processes
  • keeping a readable log of the jobs’ statuses
  • reporting back to you over the standard collections interface
  • bonus: I love the later.js syntax :smile:
16

The general solution for executing work in a scalable way is to use a job queue as @rhyslbw mentioned. Probably overkill for your current use case but at some point you may need a better guarantee that the job is actually executed. In that case, I’d recommend reading up on message queues and exploring HA architecture a bit (keeping in mind that pragmatism is king). :slight_smile:

RabbitMQ and Amazon SQS are two popular, highly-available, durable queues which can handle delayed messages. Here’s some relevant links for SQS:

2 Likes
17

have you seen this http://richsilv.github.io/meteor/scheduling-events-in-the-future-with-meteor/

18

If I understand correctly, all the talk up to this point has been about sending emails in Meteor (albeit after a delay). I will try to be brief.

Although percolate:synced-cron solves what you are trying to do, it is not a good idea to have Meteor send the emails in the first place.

You should move sending emails to a separate process. It can be Meteor, but it should not be a user-facing process.

Here’s why. You know how it’s always said Node is non-blocking? That’s a big fat lie (well, let’s soften it up, more of a marketing thing). Give Nodejs or Meteor one big fat calculation that takes five seconds, neither would be able to do anything (that includes fetching any data from anywhere, answering incoming connections, anything). It has to finish that calculation before anything else.

What this translates to is this: Even when you are using multiple cores with Meteor, every time you are sending and email from inside Meteor, you are blocking the Meteor process. You should not do that. That’s a bad idea. You should send the emails from another process.

It can be written in any programming language, including Meteor. You could start a separate Meteor process that does not listen in on incoming HTTP or DDP connections (I remember reading about this but I couldn’t find the link) and you would still be using Meteor, although not the same Meteor app.

But the main idea is to not send the emails from the web server.

It would be the same even if you were using PHP. Do not send the emails from the client-facing web app. Use a separate process.

19

You are correct, but we are already advised to wrap long-running code within server methods and use this.unblock() so that meteor’s runner does not hold up the queue.

That means, your email code can be fired up and subsequent methods can begin running instantly, without blocking the server process.

Also, meteor’s use of fibers does allow for a more relaxed architecture than a traditional nodejs server thread.

20

I would do the email sending in a separate process anyway. I would not take the risk when I am sure a separate process would work no matter what. My two cents.

21

I do agree with you with marketing/bulk emails where no home-grown
solution be it meteor or else would provide any benefit.

But for transactional emails,

a) offloading a message to another thing requires you to compile the
building blocks of the message and pass on to that thing using its API.

b) sending a message directly from with meteor requires you to compile
the building blocks of the message and pass on to that thing using SMTP.

And both would be non-blocking. So what would be the benefit, in your
opinion, to take emailing out and create a separate stack?

The way I see it is unnecessary complication.

But I do sincerely not want to sound like a wise-ass here and I do want
to know your reasons.

Thanks

22

In either A or B, I would only pass the fields and their values to the other process, which may easily be an external service like Mandrill or an internal cron job.

Anything that uses CPU cycles is a blocking operation, no matter how small. Anything that is put into the ‘work queue’ so to speak is a blocking operation. Because the main thread works by going through that work queue. Whether it is ten emails a day or 100 emails a day, something that goes into the main work queue is extra load. You may think this over-optimization, but having seen so many projects implementing sending of transactional emails as part of the core web app and failing, I would prefer to be cautious and spend an extra day implementing a separate worker for emails to sending them from the main app to gain a few hours of development time.

A web application should not be responsible for sending emails, unless we are talking about a very small application. This has nothing to do with Meteor’s superior work model or sending emails being a non-blocking operation. Meteor’s only responsibility should be serving clients on the web front. (Logins with an external service is a different subject altogether.)

Here is how I would implement it.

In Meteor main app, any email to be sent is written to ‘email_queue’. Only the fields, recipients _id, and which template to use. Not the HTML itself, just the _id of the template.

The other thread has access to user info and the templates. And works on that data. If it’s not using an external service like Mandrill, it generates the HTML and sends the email itself. It takes into account all possibilities like mailbox full, recipient mailbox does not exist.

I have a quick question that I don’t know the answer to. Let’s say the recipient server returned mailbox full. How do you proceed? Or the recipient server is down. What happens in case of a timeout? How do you handle those? I’m not trying to prove anything, just curious as to the actual process of sending the emails from Meteor. Because, you know, there may be problems in virtually every step.

23

Regarding the question, I do:

  1. this.unblock() so that I don’t actually wait for the email method to return.
  2. I use the email method’s callback to look for an error object and reschedule as necessary

Regarding your overall answer; first things first, thank you for taking the time to explain it so thoroughly!

I agree with you on a broader basis where if the app is one that you are looking to scale, then your overall premise should be applied to all parts of the application in terms of a (micro)service architecture where all and any part of your application is a confined piece of code that takes an input and reports back to a backbone service orchestrator.

But hey, I’m coming from a javaee world where I’ve seen such architecture overapplied and utterly abused one too many times.

The “it should be able to scale” pseudo-requirement causes hello world apps to take two weeks to print out the world without the hello.

So my take is, start out with fully-confined apps (except when you have to compartmentalize) and iterate as you go along. I think of taking out email sending code into a separate worker as a feature that should be prioritiezed among a broader list of features including that shiny call to action button on the top left corner that the marketing guy asked for.

So in essence, we actually agree :smile: