Hi all,
I’m thinking of using amazon ses https://github.com/dwyl/sendemail to send close to a million emails
Are there any best practices to follow ? Like I dont want to crash my servers sending all my emails at once. Maybe break the job in to pieces ? I’m not too sure how to get around this issue.
Any help would be really appreciated.
Thanks
Depending on your normal server load, you may want to do this as a separate process, such as a simple Node script, instead of inside of your Meteor app. This is especially a good idea if your server has multiple cores, since your Meteor app can run on core while the script runs on another. Of course, if you’re using multiple servers to host your app this is less of an issue (but still not a bad idea).
You’re unlikely to crash your servers unless you have a memory issue (like continuing to hold onto each email message until your process runs out of RAM). If that’s an issue, implement a design where only X number of messages can be in your mail sending queue at once; when you have fewer than X messages, retrigger the process that adds more messages. If you can get by with processing and sending each message one at a time, then this shouldn’t be an issue.
Another option would be to use something like setTimeout() to add a small delay between sending each message. Definitely don’t do it in a loop in your main Meteor app, as your site would be unresponsive until all the messages are sent.
I’d also recommend storing the progress in a database, so if the job is interrupted for any reason, you can pick up where you left off.
Depending on how you plan to trigger sending these updates, you might find vsivsi:job-collection to be helpful for scheduling the jobs. You can use it from either Meteor or pure node.js.
Hmm… ok… let me dive in to this and see what happens
I do not want to make it too complicated… Like to keep things simple. I will use the setTimeout and Meteor.defer() and this.unblock() function too
Simple is good! 
For standard email sending, that should do the trick.
I’m not sure what your particular situation looks like, so I just tried to give a good overview.
My situation is fariy simple…
Lets me explain.
I have clients… lets say 100
My clients have customers lets say 1000 each
So one of my clients wants to send out a newsletter to their 1000 customers using my app.
And the app has to be able to track the email open/click and so and so… So this is what I’m building… Basically MailChimp in Meteorjs
Do you have any updates to your answer after reading my situation ?
Thanks
Sounds like a pretty standard emailing situation: a static email that is sent to a bunch of people. In that case, there’s no real processing so overhead is low (by processing I mean stuff like generating a PDF attachment on the fly for each user).
How likely is it that all your clients will be sending emails at the same time? If it’s at all likely, then make sure you do some tests sending out 100,000+ emails at once and see how your site responds; hopefully you will find that by using Meteor.defer or Meteor.setTimeout the app performs just fine.
I would recommend tracking progress in case something does go wrong at say user 300, you don’t want to send the email to the first 299 people again. Of course, you probably need to track the number of sends in addition to open / click, so if you update a counter in your database each time you send the message, you’re already tracking progress.
As a side note, make sure you look into spam laws and enforce things like an opt-out option. I’m definitely not a lawyer and it’s been a while since I had to look into the legal side of things, but from what I recall since you are essentially sending emails on behalf of your clients, you can be held liable if you don’t ensure that the emails include appropriate opt-out information.
Wow. thanks so much. your answers are really informative.
just a quick question. what do you do for a job ? and how long have you been doing that job. because you know what you’re talking about 
just curious…
Just to add to @nathantreid’s excellent feedback: the bad news is that emailing (mailshot emailing) is hard and there’s a whole bunch of stuff you’ll need to be on top of. The good news is that emailing is hard - so there’s always going to be an opportunity for providing this service .
So just to come up with a non-exhaustive list of things you should be aware of in no particular order. Some of these may not apply as you are using SES:
Finally, you’ve chosen Meteor for a reason. Nothing discussed so far requires that Meteor magic - so give your service an edge. How about a drag and drop mailshot designer which allows you and your client to collaborate remotely?
Thanks Rob, Lots of information here.
The email functionality is a part of a complete app I’d say the email functionality is 30% of the pie.
So it is important but not so much. But has to be super simple… I like simple.
I do not think I will manage campaigns with SES. I will solely use it as a email transport service. thats it. again keeping it simple and all of the data in my app.
Thanks again.
One issue I think you’ll bump into is forEach loops timing out.
Correct me if I’m wrong, but I think there’s a 10MB limit as well as a 10 minute limit in place.
If you did something like:
EmailsToSend.find().forEach(email => send(email));
If you’re trying to send a large quantity of emails, it will probably timeout before it finishes.
Would be nice if there were a nice way around this problem that I’m unaware of.
As @nathantreid indicated, this needs to be in a separate worker process(es). I would go further and say a separate non-meteor worker process(e). Meteor has safeguards in place which really don’t belong when pumping out data in long-running methods like this.
Thanks! I’m a senior software developer at an advertising agency, and have implemented quite a few emailing applications in the 7 years I’ve been here.
Yes. this is one of my big concerns. Not too sure how to test this either. I don’t have any testing experience. 
Why not just mailchimp?
Personally I would use GitHub - normanmaurer/niosmtp: NIO implementation of a SMTP/LMTP Client to use NIO’s non-blocking IO to do the mail sending in Java along with http://akka.io/ to manage the concurrency while providing a simple JAX-RS front end to take the data for mailing or just a command line. This would be on Akka which is not run on Java EE but just plain JDK.
The tracking and capturing can be done with JAX-RS API services on Java EE as well and make it write to a mongodb database. You can do that with Meteor as well, I just prefer a Java EE app because I would have more options available such as bluemix.
The mongodb database will then be accessed through a Meteor app which will allow you to build a nice admin UI with more ease than a Java solution, this would likely be the most annoying part because you’d have to adjust the UI to what your business wants to see the rest is just server side stuff that they won’t care about.
If you want to do it all on Meteor you may have to deal with performance and what not plus you cannot distribute the work as easily if it is all one app. By breaking it down to a few architectural components each with a defined purpose you can scale more.
I dont have money to spend on mailchimp just yet. Although I’m saying million people… I dont have nay yet… I’m planning for the worst case scenario.
So I need the best cheap and efficient solution.
I’m sure there is a solution out there… its just finding it
Postmark might be a great option for you. It’s very affordable. Also look at SendGrid. I definitely wouldn’t do mass emailings right from Meteor. Use a third-party service like the aforementioned—that’s what they’re there for!
niosmtp and akka are free. You just need to put them together. You’re going to have more problem finding an SMTP provider that would allow you to bulk mail like that I think.
For tracking you just need a URL that would get triggered when they click on a link, most mail apps will filter out loading external resources so you won’t see them unless the user agrees to show images etc. That would be a simple redirect link that captures some encoded ID to link them to your campaign. Part of the handler is to store the tracking to a mongo database. You can probably do it with the http-server that comes with node rather than meteor if you don’t want to use Java.
Another tool you can use is http://www.maxprog.com/site/software/internet-marketing/mass-bulk-emailer_sheet_us.php MaxBulkMailer available for Windows and Mac to do the sends for you. However that’s not free, costs $60USD, but gives a nice UI to help send the messages.
However, in the end I still recommend you to use meteor to provide a UI to the results that are stored in the MongoDB database.
Everything is so expensive. Dont have that much money.
I need to find a way to do this.
Lets see what happens
Dude. Postmark is $1.50 per thousand emails sent. Your first 25,000 emails are free, and if you set up DMARC properly, they give you 100,000 credits. Not to be blunt, but you can’t afford $1.50 once your free 25,000 emails are sent?