v3.1.0

v3.1.0 is a non-critical feature release. The main change is first-class support for AWS SES. If you are sending through SES then you do not need any plugins besides the aws-sdk module to use Nodemailer.

The built-in SES support is an improvement over the previous nodemailer-ses-transport plugin. You can instantiate the aws-sdk object as you wish yourself instead of letting Nodemailer handle the setup. Rate limiting is vastly improved, Nodemailer guarantees the most optimal sending speed, without hitting the actual limits. There’s also an option to specify SES related mail options like Tags or ConfigurationSetName. If you use DKIM signing then Message-ID and Date fields are automatically suppressed from the signature, no special config needed.

You can find all the details required to send mail with SES from the Nodemailer homepage.

Nodemailer v3.0.0

This post is mostly about the newest release of Nodemailer. For the license changing rant, scroll down past the feature list.

It’s been a while since a major release of Nodemailer happened and now it’s finally here, the initial version of Nodemailer 3! This update brings a lot of breaking changes. I hope to write a separate blog post for each of these but for now, here’s what changed:

There’s also some non-breaking changes:

  • All dependencies were dropped. There is exactly 0 dependencies needed to use Nodemailer. This brings the installation time of Nodemailer from NPM down to less than 2 seconds
  • Delivery status notifications added to Nodemailer
  • Improved and built-in DKIM signing of messages. Previously you needed an external module for this and it did quite a lousy job with larger messages. The updated version processes messages as streams and also is able to cache parts of the data to disk if the message is very large
  • Stream transport to return a RFC822 formatted message as a stream. Useful if you want to use Nodemailer as a preprocessor and not for actual delivery.
  • Sendmail transport built-in, no need for external transport plugin

See Nodemailer.com for full documentation. I hope you’ll enjoy the new features!


PS. Several people have asked me about the license change. Why use such an esoteric license like EUPL (European Union Public Licence – EUPL v.1.1)? And why a copyleft license, whats wrong with good old MIT/BSD?

EUPL is very similar to GPLv2 as these licenses are more or less compatible. EUPL has the advantage over GPL of having an official translation of the license text in the Estonian language as all EU documents  are translated into all EU languages. I don’t know any other open source license that has such translation. Additionally GPL is more US specific while EUPL is EU specific and Nodemailer is built in EU (in Estonia), so its a natural choice if copyleft license is preferred.

So, why copyleft?

I’ve been building Nodemailer for the last 6 or 7 years and while it has been a source of great mental satisfaction (you know, solving some really difficult problems, squashing hard to find bugs, studying a myriad of email related RFC’s and so on) then it’s been a lousy project in financial terms. I’ve been collecting donations since day one and the grand total I’ve received during this time is around $500 (the sum would be double of that if I hadn’t spent the bitcoin I received long before the price surge). And these donations were made mostly by fellow developers like myself (huge props to all who have ever donated to any of my projects, I really appreciate it!).

Nodemailer, at least the core of it, is not a community project. Obviously I’ve received a lot of help during the years but nevertheless I consider it a solo project, every feature and every change directly translates back to the time I’ve spent building this project.

The problem how I see it, is that Nodemailer is not used only by fellow developers like myself, it is also used by larger companies that do not tend to give anything back. I’m aware that no company is successful because of using Nodemailer, I’m also aware that using Nodemailer saves a lot of developer hours because of the easy to use API and the Just Works approach. A lot of the value is under the hood – for example Nodemailer gives its best to build as compliant messages as possible to avoid being marked as spam (there’s a ton of different compliancy tests spam filters usually check for and messages generated by Nodemailer should pass them all). Nodemailer also tries to be as resource friendly as possible – this in turn means faster sending speed and better performance. To arrive into such state has taken huge efforts on my side and I see companies getting a lot of value out of it but I don’t see any gains for myself.

In short, that’s why I decided to change the license. I don’t feel like I should be sponsoring big companies with my unpaid time. I can’t (and I don’t want to) revert already published versions but for the new stuff I go with copyleft. For most people this should be just as fine as anything else. If, for whatever reason, copyleft is not acceptable, then there’s always the option to purchase a commercial license without the limitations of copyleft or use something else to send the emails, for example vendor provided API clients or some other SMTP client.

The mess that is attachment filenames

If we wanted to send an email with some attachments then we can do this, nothing easier. If we wanted to use emojis in the filename of an attachment, then, in most cases, we can do this as well. But how did we end up here, has this been always possible? Let’s dig in!

Email attachments are one of the most used email features but this hasn’t always been the case. In the dark ages of email, messages were only plain text ASCII, so no-no to attachments. Today we assume that attachments became possible with the addition of Multipurpose Internet Mail Extensions (in short MIME) that was described in RFC1341 back in 1992. This is not quite accurate as well. The first attachments were actually uuencoded files included in the plaintext message body that pre-dates MIME. And in fact modern email applications often still support these kind of attachments.

Unsurprisingly the uuencoded attachments in non-MIME messages have a limited range of characters that can be used in the filename. Basically just plain ASCII (No emoji. Total loser). File naming was much simpler back in the day.

An RFC822 formatted email message with a text part and an attachment called test.txt would look like this (↵ marks newlines):

From: ...↵
To:...↵
Subject: This is a test message↵
↵
Test content↵
↵
begin 644 test.txt↵
#0V%T↵
`↵
end↵

And when opening such message with a modern email application we can indeed see that there is an attachment called test.txt attached.

These prehistoric attachments are not too interesting though (unless you’re a phisher wanting to bypass some attachment related security checks), so let’s move on.

As already mentioned, 1992 gave us MIME and a standardized way to include non-text parts in a message. This is also the attachment system as we know it today – you have a multipart data tree where each node has its own separate header and body block. If the body block needs to include non-ASCII text then this is also possible, the content is encoded and the encoding scheme is noted in the header of that node. So the same uuencoded message could be converted to MIME like this:

Subject: This is a test message↵
MIME-Version: 1.0↵
Content-Type: multipart/mixed; boundary=abc↵
↵
--abc↵
Content-Type: text/plain↵
↵
Test content↵
↵
--abc↵
Content-Type: application/octet-stream;↵
    name="test.txt"↵
Content-Transfer-Encoding: base64↵
↵
Q2F0↵
--abc--↵

And when opening such email message with an email client we see this:

Looks pretty much the same, doesn’t it?

So where did the filename test.txt in the screenshot came from? If we look at the source then we see that the Content-Type header has an extra argument called name that defines the filename. This is already pretty awesome, we can use quoted parameters in MIME and inside the quotes we can use almost any (but no all) printable ASCII characters we like. So no filenames that have special characters like quotes and still lightyears away from emojis but we already can have words separated by spaces as filename. If we would like it so.

At this point the non-english speaking countries are starting to tag along and this means a new challenge: non-ASCII characters in message header. Message bodies already can use it, you can set a “charset” argument to the Content-Type header, use some encoding (usually either Base64 or Quoted-Printable) and you’re ready to send some mail, assuming you’re all right with using only latin characters in the subject line and also in the attachment filenames.

So in parallel to the RFC1341 a new standard was proposed, RFC1342 that defined something called “encoded-words”. It’s a construct that using only ASCII symbols presents sequences of encoded characters in any character set. If a word in our subject line includes a non-ASCII character, for example an umlaut “test messäge, awesome!” then the Subject header could be formatted like this

Subject: test =?ISO-8869-1?Q?mess=E4ge,?= awesome!

Unfortunately, encoded-words come with a restriction: these can only be used as an atom and not inside quotes as anything that is quoted is meant to be kept the way it is presented.

If we would try to use encoded-words as an attachment filename without quotes it would look obviously wrong

Content-Type: application/octet-stream;↵
    name==?ISO-8859-1?Q?mess=E4ge.txt?=

Notice the double equal sign? This doesn’t seem right at all.

So even if we gained support for non-latin characters in the Subject line then we are still limited in our options when using attachment names.

We needed an alternative solution and even if it took some time we finally got there.

In 1996 RFC2045 re-defined the Content-Type header but this time there was no explicit “name” parameter mentioned. In 1997 RFC2183 fixed this by adding a new header Content-Disposition that has a parameter called filename, suitable for, as the name suggests, defining attachment filenames.

In parallel to RFC2813 another standard was released, RFC2814 that together with the newly defined filename parameter finally fixed the long standing issue of non-latin attachment filenames. This was done by introducing yet another encoding scheme, Parameter Values and Parameter Value Continuations. When encoded-words used equals sign to indicate encoded characters and question mark to separate different parts of the scheme then Parameter Values use percent sign for encoding and single quote for separation.

Content-Disposition: attachment;↵
    filename*=iso-8859-1''mess%4Ege.txt

Asterisk in the end of the parameter name indicates that the value uses Parameter Value encoding.

The same mechanism allows splitting long values into multiple chunks (thats the Continuation part) but this is not super important, so we will not cover it here. What is super important, is that the changes done in 1997 allow us to finally use emojis in filenames even if emojis as characters weren’t invented yet.

If we want to use a filename like “😁😂’.txt” then this is totally doable with the Parameter Value scheme. We need to provide correct charset and also encode the bytes in the value. It would look something like this:

Content-Disposition: attachment;↵
    filename*=utf-8''%F0%9F%98%81%F0%9F%98%82.txt

On a good day this would end our journey because this is exactly how emojis in attachment filenames are supposed to work. In the real world things are not as easy.

Let’s send a message with an attachment name formatted this way to for example an email address hosted by QQ, a huge chinese email provider. How does the QQ interface display us the attachment? Assuming that the chinese are accustomed to using non-latin characters then this should work as intended? No?

Apparently no. It seems that QQ does not handle these characters at all, as this is how the attachment is shown in the web interface

Do you really have to keep using only latin characters when sending to a chinese service? Fortunately not. QQ webmail does support non-latin characters, including in attachment names, they just do not follow the standards for that.

I guess that they haven’t yet catched up with the latest trends of 1997

It appears that they still go with the Content-Type header with name parameter where they allow to use encoded-words in quotes.

So what you really need to do in this case would be to add the same filename to attachment headers twice. Once to Content-Disposition, using the standard Parameter Value encoding and once to Content-Type, using encoded-word encoding.

Content-Type: application/octet-stream;↵
    name="=?utf-8?Q?=F0=9F=98=81=F0=9F=98=82.txt?="↵
Content-Transfer-Encoding: base64↵
Content-Disposition: attachment;↵
    filename*0*=utf-8''%F0%9F%98%81%F0%9F%98%82.txt↵

And the result is exactly what we wanted, an attachment named with emojis.

More standard obeying email clients would use the name shown in Content-Disposition. QQ and some other less standards-savvy clients would use the Content-Type header.

Nodemailer obviously handles all of this out of the box.