Spring cleaning

Over the time I have released a lot of open source code, mostly stuff around email but also about ePub, gettext, SSL certificates, RSS/Atom and so on. Some of the major ones I have gave away to other maintainers but there’s still a lot of projects where I’m the owner and that are not actually maintained. These projects generate a lot of issues in Github that I never respond to, there are actual unfixed bugs and huge pile of PRs that never get pulled.

So, as I’m not able to actually maintain these projects I’m starting to mark these as deprecated urging users to move over to alternatives.

In short, if the project is not related to Nodemailer, WildDuck or ZoneMTA (eg. it’s not a dependency for the mentioned projects) then I’m not supporting it.

Reasons: I can’t afford it. Nodemailer has its own revenue streams (it is somewhat sponsored and has ads), I get paid to work on ZoneMTA and WildDuck is something that might have revenue streams in the future, so there’s incentives for making sure these projects get all needed attention. Other projects are just a time drain, using up my limited free time without giving anything back.

Building fault tolerant email storage for $30 a month

Ethereal.email is not a big news anymore. It’s a simple service that allows you to generate email accounts using an API call from Nodemailer or by clicking a button on the Ethereal homepage. If you try to send mail using that account, then all messages are caught and stored in the INBOX of your account where you can then access the messages through a web interface or by using your favorite IMAP client.

What might be still interesting though is that one of the main reasons I created it was to test out some of my newer email projects. The mail store happens to be based on Wild Duck Mail Server and unlike regular mail servers it is designed to be fault tolerant by default. Having lots of people sending all kind of messages to that system allows to see how it works in practice.

Whenever you connect to a Ethereal.email related URL or service, for example to the Ethereal homepage or the MX server or try to send mail through the Ethereal MSA then actually you hit an HAProxy instance first. This HAProxy instance then sends your request to one of three available application servers. Wild Duck is stateless, so all requests are balanced with the most simplest round robin algorithm.

All three servers make up a MongoDB replica set and also a Redis Sentinel set. Applications (IMAP, POP3, MSA, MX, WWW) connect to these replica sets to get their data. MongoDB is used as the mail store and Redis is for caching, counters, pubsub and such.

So to the fault tolerant part. If one of the application server dies for whatever reason, then the service should keep working unaffected. HAProxy detects that the specific backend is down and removes it from backend list, so no request is routed to that specific instance. As Redis and MongoDB are both set up in a replicated setup then applications either keep using the old Primary instance of MongoDB and Redis or wait until MongoDB replica set and Redis Sentinel have elected a new Primary and seamlessly switch over.

If your long lived IMAP connection happens to target the instance that goes down then connection is obviously lost. Usually IMAP clients resume the connection immediately and this time HAProxy routes the connection to a healthy instance, so as an user you probably don’t even notice that your mail client reconnected.

Application servers have 50GB of storage each which is not much but as Wild Duck is a lot more efficient than normal mail stores when storing messages to disk, then in reality it should be able to store a lot more messages than 50GB. All messages expire in 7 days, so even if the storage gets full, it’s a temporary problem.

On the left is the combined size of maildir folders for ~2000 users. On the right are the same messages imported to Wild Duck using all available optimization options

“Normal” email servers usually dedicate a specific folder on a specific disk to email users, so your IMAP connections must always end up in the same specific instance. Otherwise your mailbox becomes unavailable. In case of Wild Duck it does not matter to which host you connect to as all data is stored to replicated document storage.

And now to the costs. I host the application in OVH, with each instance using a €2.99 VPS (2GB RAM, 1 CPU). Application instances have an additional 50GB disk attached (€5 each). TLS certificates are handled by Let’s Encrypt (and acme.sh) which are free. So the total costs being 4*2.99 + 3*5 = 26.96€ which roughly translates to $30.

My benchmarks show that this $30 cluster is able to process 30 messages per second (10Mb/s). For testing I used the messages accumulated over the years to my main email address, so the processed messages should reflect everyday usage (different messages, different kind of attachments etc). Not quite enough to run a large scale email hosting but good enough for that money.

Ethereal email testing

Today I’m really excited to announce the release of Nodemailer v4.1.0. If you have everything already set up regarding email sending then you probably do not care too much about this release. If you are currently building something or plan to build an app that, amongst other things, sends email, then this new release might interest you.

Nodemailer v4.1.0 includes 2 new API methods




where the first one generates an actually working email account out of the blue and the other one returns extra information about a delivery.

These autogenerated accounts are not too real though, these are test accounts from Ethereal.email mail testing service.

When an Ethereal test account is used then Nodemailer establishes a normal SMTP connection against Ethereal SMTP server, authenticates with actual credentials and the server accepts message for delivery, so nothing unusual about it. What is a bit unusual though is that the Ethereal server never does the actual delivery, it stores the message to the account of the authenticated user and that’s it. You don’t have to worry about unexpected deliveries where mail is delivered to actual recipients. Ethereal never sends any messages.

Finally, the getTestMessageUrl(info) method returns a web URL that can be used to preview the sent message in a browser.  You can preview the message HTML, download the RFC822 source of the message or just check the message headers.

You can store the autogenerated credentials and start using these as development credentials instead of spamming a real email account. Or if you do not want to then you can generate fresh credentials for every new test email, it’s your own choice.  If you want to use IMAP to preview the sent messages then you probably want to use pre-generated credentials, otherwise it wouldn’t make much sense.

v4.0.0 – back to MIT

Nodemailer v4.0.0 is now released and is using the MIT license. This version does not bring any other changes, it is a republished v3.1.8


v3.1.0 is a non-critical feature release. The main change is first-class support for AWS SES. If you are sending through SES then you do not need any plugins besides the aws-sdk module to use Nodemailer.

The built-in SES support is an improvement over the previous nodemailer-ses-transport plugin. You can instantiate the aws-sdk object as you wish yourself instead of letting Nodemailer handle the setup. Rate limiting is vastly improved, Nodemailer guarantees the most optimal sending speed, without hitting the actual limits. There’s also an option to specify SES related mail options like Tags or ConfigurationSetName. If you use DKIM signing then Message-ID and Date fields are automatically suppressed from the signature, no special config needed.

You can find all the details required to send mail with SES from the Nodemailer homepage.

Nodemailer v3.0.0

This post is mostly about the newest release of Nodemailer. For the license changing rant, scroll down past the feature list.

It’s been a while since a major release of Nodemailer happened and now it’s finally here, the initial version of Nodemailer 3! This update brings a lot of breaking changes. I hope to write a separate blog post for each of these but for now, here’s what changed:

  • License changed from MIT to EUPL-1.1. This was possible as the new version of Nodemailer is a major rewrite. The features I don’t have ownership for, were removed or reimplemented. If there’s still some snippets in the code that have vague ownership then notify me at [email protected] about the conflicting code and I’ll fix it
  • Requires Node.js v6+ as a lot of ES6 specific features are used. Some make life easier (eg. classes versus Function.prototype), some make the code better (using let instead of var eliminates several possible memory leak scenarios)
  • All templating is gone. It was too confusing to use and to be really universal a huge list of different renderers would be required. Nodemailer is about email, not about parsing different template syntaxes
  • No NTLM authentication. It was too difficult to re-implement. If you still need it then it would be possible to introduce a pluggable SASL interface where you could load the NTLM module in your own code and pass it to Nodemailer. Currently this is not possible.
  • OAuth2 authentication is built in and has a different configuration. You can use both user (3LO) and service (2LO) accounts to generate access tokens from Nodemailer. Additionally there’s a new feature to authenticate differently for every message – useful if your application sends on behalf of different users instead of a single sender.
  • Improved Calendaring. Provide an ical file to Nodemailer to send out calendar events.

There’s also some non-breaking changes:

  • All dependencies were dropped. There is exactly 0 dependencies needed to use Nodemailer. This brings the installation time of Nodemailer from NPM down to less than 2 seconds
  • Delivery status notifications added to Nodemailer
  • Improved and built-in DKIM signing of messages. Previously you needed an external module for this and it did quite a lousy job with larger messages. The updated version processes messages as streams and also is able to cache parts of the data to disk if the message is very large
  • Stream transport to return a RFC822 formatted message as a stream. Useful if you want to use Nodemailer as a preprocessor and not for actual delivery.
  • Sendmail transport built-in, no need for external transport plugin

See Nodemailer.com for full documentation. I hope you’ll enjoy the new features!

PS. Several people have asked me about the license change. Why use such an esoteric license like EUPL (European Union Public Licence – EUPL v.1.1)? And why a copyleft license, whats wrong with good old MIT/BSD?

EUPL is very similar to GPLv2 as these licenses are more or less compatible. EUPL has the advantage over GPL of having an official translation of the license text in the Estonian language as all EU documents  are translated into all EU languages. I don’t know any other open source license that has such translation. Additionally GPL is more US specific while EUPL is EU specific and Nodemailer is built in EU (in Estonia), so its a natural choice if copyleft license is preferred.

So, why copyleft?

I’ve been building Nodemailer for the last 6 or 7 years and while it has been a source of great mental satisfaction (you know, solving some really difficult problems, squashing hard to find bugs, studying a myriad of email related RFC’s and so on) then it’s been a lousy project in financial terms. I’ve been collecting donations since day one and the grand total I’ve received during this time is around $500 (the sum would be double of that if I hadn’t spent the bitcoin I received long before the price surge). And these donations were made mostly by fellow developers like myself (huge props to all who have ever donated to any of my projects, I really appreciate it!).

Nodemailer, at least the core of it, is not a community project. Obviously I’ve received a lot of help during the years but nevertheless I consider it a solo project, every feature and every change directly translates back to the time I’ve spent building this project.

The problem how I see it, is that Nodemailer is not used only by fellow developers like myself, it is also used by larger companies that do not tend to give anything back. I’m aware that no company is successful because of using Nodemailer, I’m also aware that using Nodemailer saves a lot of developer hours because of the easy to use API and the Just Works approach. A lot of the value is under the hood – for example Nodemailer gives its best to build as compliant messages as possible to avoid being marked as spam (there’s a ton of different compliancy tests spam filters usually check for and messages generated by Nodemailer should pass them all). Nodemailer also tries to be as resource friendly as possible – this in turn means faster sending speed and better performance. To arrive into such state has taken huge efforts on my side and I see companies getting a lot of value out of it but I don’t see any gains for myself.

In short, that’s why I decided to change the license. I don’t feel like I should be sponsoring big companies with my unpaid time. I can’t (and I don’t want to) revert already published versions but for the new stuff I go with copyleft. For most people this should be just as fine as anything else. If, for whatever reason, copyleft is not acceptable, then there’s always the option to purchase a commercial license without the limitations of copyleft or use something else to send the emails, for example vendor provided API clients or some other SMTP client.

The mess that is attachment filenames

If we wanted to send an email with some attachments then we can do this, nothing easier. If we wanted to use emojis in the filename of an attachment, then, in most cases, we can do this as well. But how did we end up here, has this been always possible? Let’s dig in!

Email attachments are one of the most used email features but this hasn’t always been the case. In the dark ages of email, messages were only plain text ASCII, so no-no to attachments. Today we assume that attachments became possible with the addition of Multipurpose Internet Mail Extensions (in short MIME) that was described in RFC1341 back in 1992. This is not quite accurate as well. The first attachments were actually uuencoded files included in the plaintext message body that pre-dates MIME. And in fact modern email applications often still support these kind of attachments.

Unsurprisingly the uuencoded attachments in non-MIME messages have a limited range of characters that can be used in the filename. Basically just plain ASCII (No emoji. Total loser). File naming was much simpler back in the day.

An RFC822 formatted email message with a text part and an attachment called test.txt would look like this (↵ marks newlines):

From: ...↵
Subject: This is a test message↵
Test content↵
begin 644 test.txt↵

And when opening such message with a modern email application we can indeed see that there is an attachment called test.txt attached.

These prehistoric attachments are not too interesting though (unless you’re a phisher wanting to bypass some attachment related security checks), so let’s move on.

As already mentioned, 1992 gave us MIME and a standardized way to include non-text parts in a message. This is also the attachment system as we know it today – you have a multipart data tree where each node has its own separate header and body block. If the body block needs to include non-ASCII text then this is also possible, the content is encoded and the encoding scheme is noted in the header of that node. So the same uuencoded message could be converted to MIME like this:

Subject: This is a test message↵
MIME-Version: 1.0↵
Content-Type: multipart/mixed; boundary=abc↵
Content-Type: text/plain↵
Test content↵
Content-Type: application/octet-stream;↵
Content-Transfer-Encoding: base64↵

And when opening such email message with an email client we see this:

Looks pretty much the same, doesn’t it?

So where did the filename test.txt in the screenshot came from? If we look at the source then we see that the Content-Type header has an extra argument called name that defines the filename. This is already pretty awesome, we can use quoted parameters in MIME and inside the quotes we can use almost any (but no all) printable ASCII characters we like. So no filenames that have special characters like quotes and still lightyears away from emojis but we already can have words separated by spaces as filename. If we would like it so.

At this point the non-english speaking countries are starting to tag along and this means a new challenge: non-ASCII characters in message header. Message bodies already can use it, you can set a “charset” argument to the Content-Type header, use some encoding (usually either Base64 or Quoted-Printable) and you’re ready to send some mail, assuming you’re all right with using only latin characters in the subject line and also in the attachment filenames.

So in parallel to the RFC1341 a new standard was proposed, RFC1342 that defined something called “encoded-words”. It’s a construct that using only ASCII symbols presents sequences of encoded characters in any character set. If a word in our subject line includes a non-ASCII character, for example an umlaut “test messäge, awesome!” then the Subject header could be formatted like this

Subject: test =?ISO-8869-1?Q?mess=E4ge,?= awesome!

Unfortunately, encoded-words come with a restriction: these can only be used as an atom and not inside quotes as anything that is quoted is meant to be kept the way it is presented.

If we would try to use encoded-words as an attachment filename without quotes it would look obviously wrong

Content-Type: application/octet-stream;↵

Notice the double equal sign? This doesn’t seem right at all.

So even if we gained support for non-latin characters in the Subject line then we are still limited in our options when using attachment names.

We needed an alternative solution and even if it took some time we finally got there.

In 1996 RFC2045 re-defined the Content-Type header but this time there was no explicit “name” parameter mentioned. In 1997 RFC2183 fixed this by adding a new header Content-Disposition that has a parameter called filename, suitable for, as the name suggests, defining attachment filenames.

In parallel to RFC2813 another standard was released, RFC2814 that together with the newly defined filename parameter finally fixed the long standing issue of non-latin attachment filenames. This was done by introducing yet another encoding scheme, Parameter Values and Parameter Value Continuations. When encoded-words used equals sign to indicate encoded characters and question mark to separate different parts of the scheme then Parameter Values use percent sign for encoding and single quote for separation.

Content-Disposition: attachment;↵

Asterisk in the end of the parameter name indicates that the value uses Parameter Value encoding.

The same mechanism allows splitting long values into multiple chunks (thats the Continuation part) but this is not super important, so we will not cover it here. What is super important, is that the changes done in 1997 allow us to finally use emojis in filenames even if emojis as characters weren’t invented yet.

If we want to use a filename like “😁😂’.txt” then this is totally doable with the Parameter Value scheme. We need to provide correct charset and also encode the bytes in the value. It would look something like this:

Content-Disposition: attachment;↵

On a good day this would end our journey because this is exactly how emojis in attachment filenames are supposed to work. In the real world things are not as easy.

Let’s send a message with an attachment name formatted this way to for example an email address hosted by QQ, a huge chinese email provider. How does the QQ interface display us the attachment? Assuming that the chinese are accustomed to using non-latin characters then this should work as intended? No?

Apparently no. It seems that QQ does not handle these characters at all, as this is how the attachment is shown in the web interface

Do you really have to keep using only latin characters when sending to a chinese service? Fortunately not. QQ webmail does support non-latin characters, including in attachment names, they just do not follow the standards for that.

I guess that they haven’t yet catched up with the latest trends of 1997

It appears that they still go with the Content-Type header with name parameter where they allow to use encoded-words in quotes.

So what you really need to do in this case would be to add the same filename to attachment headers twice. Once to Content-Disposition, using the standard Parameter Value encoding and once to Content-Type, using encoded-word encoding.

Content-Type: application/octet-stream;↵
Content-Transfer-Encoding: base64↵
Content-Disposition: attachment;↵

And the result is exactly what we wanted, an attachment named with emojis.

More standard obeying email clients would use the name shown in Content-Disposition. QQ and some other less standards-savvy clients would use the Content-Type header.

Nodemailer obviously handles all of this out of the box.