Honestly, you probably shouldn’t be building this out yourself. Bulk email platforms, for the most part, are commodity products and have prices hovering around the bottom of what is sustainable. If your application idea doesn’t work with these prices, you may need to rethink your idea/profit/etc…?
BUT, thinking about how to build this is a fun exercise. When building out system that directly communicate with people, always plan for failure and how you will respond. Failures will happen. This is a guarantee.
Let’s call each email you want to send out a Campaign
. Each Campaign
might have some metadata (a template, who its from, etc…), and a list of receivers. A naive way (on a worker process, ignoring timeouts, etc…) of delivering a Campaign
might look something like…
let campaign = Campaigns.findOne(...);
campaign.receivers.map((receiver) => {
Email.send({
to: receiver,
...
});
});
But what if Email.send
throws an exception? That will interrupt your map
and prevent all subsequent sends from happening.
Let’s say we’ve fixed the problem (if it was ours to fix), and now if we want to retry sending the campaign. Unfortunately, we have no idea who’s already been sent the email. Our retry will resend the email to lots of users. Not good!
You need some way of handling errors as they happen, and tracking who’s been successfully sent the campaign.
function deliverCampaign(campaignId) {
let campaign = Campaigns.findOne(campaignId);
let receivers = _.without(campaign.receivers, campaign.delivered);
let delivered = [];
receivers.map((receiver) => {
try {
Email.send({
to: receiver,
...
});
delivered.push(receiver);
} catch (e) {
console.error(`Problem sending ${campaignId} to ${receiver}`, e);
}
});
Campaigns.update(campaignId, {
$push: {
delivered
}
});
}
But what if problems happen elsewhere? What if the process is killed before we can update delivered
on Campaigns
? In theory we should update delivered
after every email sent, but now we’re looking at millions of sequential Email.send
calls followed by Campaigns.update
. Not good.
A better architecture could be to use some kind of job queue. Each delivery could be a job pushed onto the queue with data about the Campaign
being sent, the receiver, and information about how the job should be handled (retries, backoff, etc…). Each job handles sending one email to one receiver, and handles individual failures on its own. Whether or not a receiver was sent the email can either be inferred from the job’s completion status, or can be stored back in the Campaign
.
What’s really nice is that we could potentially scale this solution to use many worker processes (across many servers) in parallel. Each process pulls a job off the queue and sends an email.
But again, there will be unforseen problems with this solution too. Will those problems result in double-sending emails and spamming users? You’d better hope not.
I’m rambling… In a very round-about way, I’m just trying to say that there are many things to consider when building out these kinds of solutions. I would happily pay $XX/mo for the peace of mind of handling it off to a SaaS solution like Mailchimp, etc…