Posted on

Tracking deleted messages on an IMAP account

Accessing IMAP accounts usually is handled by two approaches – either we want to show the snapshot of current mailbox state, that is usually what happens when you log in to a webmail application to see your emails. Or we want to track changes on an account and somehow either display or record these, eg. when doing incremental backups or some kind of synchronisation.

While it is fairly easy to show the current snapshot and it is not hard to track new emails coming in then there is one particular change that is usually quite difficult to track. That is message deletions. Sure we can enter into IDLE state or NOOP-loop and start listening for those EXPUNGE notifications but consider the following:

  • IDLE is for a single folder only. You only see what is happening in the currently opened folder
  • IMAP connection counts are usually limited, you can’t open a separate connection for each folder
  • Reconnects happen, both because of network issues and also forced logouts. When logged in again you don’t get notifications for the events that happened when you were disconnected
  • Even if you get EXPUNGE notifications, these are against sequence numbers not against ID’s, so you must be super sure that the sequence number on your end corresponds to what the server thinks it is

So what to do to overcome these nuisances?

Was something even deleted?

First is to check is if something has even been deleted. No need to do anything if all the messages are still there.

IMAP servers expose a value called UIDNEXT which is the predicted UID value of the next message that is going to be stored in that folder. Usually it is the latest UID + 1. If we store the combo of the count of stored messages and UIDNEXT and then come back and both values are still the same (or both values have grown equally without any skips) then we can be fairly certain that no message was deleted in between.

Some servers have the CONDSTORE extension enabled that exposes MODSEQ values. We can also store that value for the folder and if it hasn’t changed then no messages were deleted from the folder. Though MODSEQ does not track only deletions but any changes, including stuff like Seen/Unseen flag changes, so the value actually changes quite often.

Sequence numbers

When receiving those EXPUNGE notifications we get a sequence number of a deleted message in currently opened mailbox. This means that we have to have the correct sequence pointers of messages on our end as well. The easiest (not the best though) is to issue a UID SEARCH ALL command after a folder is opened, store the returned list in sorted order and use it as the base of our sequencing.

If we then get an EXPUNGE notification then we treat the list as 1-based array, resolve the UID value from the sequence position and then remove that element from the list. This way we know the UID of the deleted message and also keep the sequencing in sync.

Diff

If we have detected an anomaly between the expected message count and actual message count when opening a folder then we can assume that something was deleted. We do not know what exactly was deleted, we only see a list of messages that are still there but no indication of what happened in between. The easiest approach is to compare the UID sequence list that we discussed in the previous point against UID SEARCH ALL command result. This approach assumes that we actually store the UID sequence list somewhere and keep it updated. Anything listed in our stored list but missing in the server provided list was deleted and anything listed in the server response but missing in our stored list, is a new message.

And that’s it, this is how “easy” it is to track deleted messages on an IMAP account. Indeed, SEARCH ALL command is a bad approach when used against a folder that contains a lot of messages but it is the easiest to implement, and you can always move on from there to better approaches. Or alternatively if you do not want to implement IMAP tracking logic on your own then you can use something like IMAP API that does the tracking itself and notifies you on every change on the account via web-hooks.

Posted on

Tracking email replies with IMAP API

One of the spin-off projects from Nodemailer is IMAP API, a daemon application to access IMAP accounts over REST. In this blog post I would like to describe how to use IMAP API to track email replies.

Reply tracking is useful when building integrations with users’ email accounts. Let’s say our service sends out emails as if these were sent by the user, eg. automated sales emails, and now the recipient replies to such message. Reply signal is usually a strong indicator that might change the status of a cold lead to a hot lead, so in most cases we would want to catch it for automated lead processing.

Tracking replies is basically a filtered use case of tracking all new messages. You get a notification that a message was added to INBOX (or Junk if you care about potential false-positive spam messages). Then perform basic checks to detect if this message is a “normal” message or a reply. If it is a reply then validate if it is a reply to a message we actually care about. And finally, check if it is not an automated out of office reply because there is no value in these (besides validating that the address still exists).

Send a message to track replies for

Sending a message to track replies for should be rather simple. Use any kind of message settings you like and in addition do the following:

  • Use a predefined (but otherwise unique) Message-ID header. This is important as the replies will reference this value. Store Message-ID to your database so it would be possible to later check which message was replied to.
  • Try to suppress automated replies by using X-Auto-Response-Suppress header. This does not guarantee that you would not get any OOF emails but it is at least some help.

In IMAP API you can use the following config when submitting messages for delivery:

{
  "messageId": "<[email protected]>",
  "headers": {
    "X-Auto-Response-Suppress": "OOF"
  },
  "from": "....
}

Make sure you store "<[email protected]>" to database as a reference to the sent message.

I guess that’s about it when sending messages. Next start waiting for replies.

Process email notifications

IMAP API sends webhook notifications for every received message. When processing such notifications there is an important distinction with “regular” email servers and GSuite/Gmail-like servers to be aware of.

For all “normal” email servers IMAP API reports the folder name the message was found in (eg. “INBOX”) in the webhook data but for Gmail accounts IMAP API reports folder names only for “All Mail”, “Junk” and “Trash”.

In such case you would have to check the “labels” value (an array of strings) for actual message placement. Eg. if a message has label \Inbox then this message was found from the INBOX. Labels array includes special-use tags for special folders (\Inbox, \Sent, \Important etc.) and folder name strings for regular folders. So if you see a labels array like the following:

{
  …,
  "path": "[Google Mail]/All Mail",
  "specialUse": "\\All",
  "event": "messageNew",
  "data": {
    …,
    "labels": [
      "\\Inbox",
      "My Stuff"
    ]
  }
}

Then it means the message was stored both in INBOX and in a folder called “My Stuff”. Regular IMAP servers do not allow storing the same message in multiple folders, so for these you would get 2 notifications, one for INBOX and one for “My Stuff”.

{
  …,
  "path": "INBOX",
  "specialUse": "\\Inbox",
  "event": "messageNew",
  "data": {
    …
  }
}

Detect replies

In general, reply detection is quite straightforward. Just check that the message is placed in INBOX (which means that either the webhook specialUser value is \Inbox or labels value, if set, includes the tag \Inbox) and look for the In-Reply-To header which is included in message data under the inReplyTo value. If this value is the same as any Message-ID we previously used to send out a message then this is a reply to this message.

{
  …,
  "path": "[Gmail]/All Mail",
  "specialUse": "\\All",
  "event": "messageNew",
  "data": {
    "id": "AAAAAQAAMqo",
    …,
    "inReplyTo": "<[email protected]>",
    "labels": [
      "\\Important",
      "\\Inbox"
    ]
  }
}

In this example we can look up the previously sent unique Message-ID value "<[email protected]>" so the received message must be a reply to this stored message.

Not done yet

At this point we know that the message is a reply but we do not know if it’s an actual reply or an automated response. We did use the OOF suppression header but it only filters out some of the automated responses, not all. To be able to determine reply status more precisely we have to fetch the extended message information from IMAP API using the “id” value in the webhook: 

/v1/account/account_id/message/AAAAAQAAMqo

Extended message information includes several important properties. We are mostly interested in the “headers” object that includes parsed message headers. IMAP API parses headers into a structure where lowercase header key is the object key and all header values for this key are presented as an array of strings. For most headers you would only have an array with a single string value but for some (eg. “received”) there might be several values.

The first thing to look for from the headers is Return-Path value. If it is empty (and in this case "<>" equals empty) then we can ignore this message as empty Return-Path usually means a bounce. An actual email address means that we can continue processing.

{
  "id": "AAAAAQAAMqo",
   …,
  "headers": {
    …,
    "return-path": [
      "<[email protected]>"
    ],
    "in-reply-to": [
      "<[email protected]>"
    ]
    …
  }
}

Next we should look for standard auto-reply headers like Auto-Submitted. If this header is set and it has any other value than “no” then you can ignore this message. We could also check headers like List-ID or List-Unsubscribe. Even if it is a reply then usually replies from mailing lists are not meant specifically for a single user, so such replies might not mean that the recipient is a hot sales lead.

Another thing to look out for is checking if the email subject has a prefix like “Out of Office:” or “Auto:”, these messages should be discarded as well.

At last…

If the messages passed all these checks then at last we could declare this message as a reply to our previous message. Depending on the nature of the automation we may or may not care about the actual reply content. For example if we only use the fact of replying as a signal for calculating lead “hotness” then we probably do not need any additional information. If we do care about the reply contents then we can use the /message endpoint to fetch message text or relevant attachments for whatever processing we need to perform.

Posted on

Testing AMP4Email with Nodemailer

Even though AMP4Email was supposed to be The Next Big Thing™ in email, it is still supported only by a handful of email service providers and the tools for development are kind of lacking.

From positive side, Gmail is one the email service providers that support AMP4Email and Gmail alone makes up a huge portion of current email ecosystem. From negative side – you need to get verified by Gmail before your dynamic emails are accepted.

Anyhow, let’s forget about the hurdles of getting your mail actually rendered and let’s look at the tooling. And specifically, how to use Nodemailer to develop those dynamic emails.

It just happens that Nodemailer can get you quite far on that side as Nodemailer, the email sending library, has AMP4Email support built in. And it can also be used for testing, as NodemailerApp, the email debugging app, is capable of rendering not just regular HTML emails but AMP4Email emails as well.

NodemailerApp rendering an AMP4Email email

NodemailerApp

To test AMP4Email emails you need to somehow display the messages. Download NodemailerApp for Windows or OSX from here or if you are using Linux, then you can get it from the Snapcraft Store.

Once you have the app, start it and create a new email project.

Next open local server preferences (“Server” -> “Configure”) and make sure that SMTP server configuration looks about right. In this example we use the default port 1025 for receiving emails through SMTP. For extra convenience enable checkbox “Run server on application start” and hit “OK”.

In the “Server” menu, click on “Start server” if server is not already started. This gives us a running SMTP server where to send out AMP4Email messages for preview.

Next open “Local server” view in the email project to get the Nodemailer configuration settings. If everything is OK then you should see a green marker with “server is currently running” message.

Nodemailer

At this point we have everything set up on the receiving side. Now we have to send some AMP4Emails to test it out. We’ll use a simple Node.js script for it and Nodemailer specifically.

$ npm install nodemailer

Once we have nodemailer installed, let’s create a new script and copy the nodemailer configuration from NodemailerApp:

const nodemailer = require('nodemailer');
const transporter = nodemailer.createTransport({
     host: 'localhost',
     port: 1025,
     auth: {
         user: 'project.12',
         pass: 'secret.12'
     }
 });

Next we need to send an actual email using this configuration.

AMP4Email configuration looks pretty much the same as any regular email. You can specify AMP4Email content with the amp configuration key but remember that AMP4Email requires either plaintext or HTML content present as well. This is because it is highly probable that your dynamic content is not rendered, thus you need alternative content to fall back to if this happens.

let message = {
   from: 'Nodemailer <[email protected]>',
   to: 'Nodemailer <[email protected]>',
   subject: 'AMP4EMAIL message',
   text: 'For clients with plaintext support only',
   html: '<p>For clients that do not support AMP4EMAIL or amp content is not valid</p>',
   amp: `<!doctype html>
   <html ⚡4email>
     <head>
       <meta charset="utf-8">
       <style amp4email-boilerplate>body{visibility:hidden}</style>
       <script async src="https://cdn.ampproject.org/v0.js"></script>
       <script async custom-element="amp-anim" src="https://cdn.ampproject.org/v0/amp-anim-0.1.js"></script>
     </head>
     <body>
       <p>Image: <amp-img src="https://cldup.com/P0b1bUmEet.png" width="16" height="16"/></p>
       <p>GIF (requires "amp-anim" script in header):<br/>
         <amp-anim src="https://cldup.com/D72zpdwI-i.gif" width="500" height="350"/></p>
     </body>
   </html>`
 }

Now we have everything we need, we just have to use the previously defined transporter to send out the configured message object.

transporter.sendMail(message, (err, info) => {
  console.log(err || info);
});

Let’s see the result

Running this script should return us information about sent message

{
   accepted: [ '[email protected]' ],
   response: '250 Message imported as 1',
   envelope: { from: '[email protected]', to: [ '[email protected]' ] },
   messageId: '<[email protected]>'
}

And the NodemailerApp should show us an animated cat image from the AMP4Email

If something went wrong or the AMP4Email message was not valid, then we should see an error message instead.

If you have already got a working AMP4Email email working in NodemailerApp then you can try to send the same message to a real service as well.

Remember that there is not much point in sending invalid AMP4Emails to Gmail as Gmail never tells you what went wrong, you just see the fallback HTML or plaintext and that’s it. So if NodemailerApp is not able to show you your message then other services most probably would not render it as well.

Posted on

Spring cleaning

Over the time I have released a lot of open source code, mostly stuff around email but also about ePub, gettext, SSL certificates, RSS/Atom and so on. Some of the major ones I have gave away to other maintainers but there’s still a lot of projects where I’m the owner and that are not actually maintained. These projects generate a lot of issues in Github that I never respond to, there are actual unfixed bugs and huge pile of PRs that never get pulled.

So, as I’m not able to actually maintain these projects I’m starting to mark these as deprecated urging users to move over to alternatives.

In short, if the project is not related to Nodemailer, WildDuck or ZoneMTA (eg. it’s not a dependency for the mentioned projects) then I’m not supporting it.

Reasons: I can’t afford it. Nodemailer has its own revenue streams (it is somewhat sponsored and has ads), I get paid to work on ZoneMTA and WildDuck is something that might have revenue streams in the future, so there’s incentives for making sure these projects get all needed attention. Other projects are just a time drain, using up my limited free time without giving anything back.

Posted on

Building fault tolerant email storage for $30 a month

Ethereal.email is not a big news anymore. It’s a simple service that allows you to generate email accounts using an API call from Nodemailer or by clicking a button on the Ethereal homepage. If you try to send mail using that account, then all messages are caught and stored in the INBOX of your account where you can then access the messages through a web interface or by using your favorite IMAP client.

What might be still interesting though is that one of the main reasons I created it was to test out some of my newer email projects. The mail store happens to be based on Wild Duck Mail Server and unlike regular mail servers it is designed to be fault tolerant by default. Having lots of people sending all kind of messages to that system allows to see how it works in practice.

Whenever you connect to a Ethereal.email related URL or service, for example to the Ethereal homepage or the MX server or try to send mail through the Ethereal MSA then actually you hit an HAProxy instance first. This HAProxy instance then sends your request to one of three available application servers. Wild Duck is stateless, so all requests are balanced with the most simplest round robin algorithm.

All three servers make up a MongoDB replica set and also a Redis Sentinel set. Applications (IMAP, POP3, MSA, MX, WWW) connect to these replica sets to get their data. MongoDB is used as the mail store and Redis is for caching, counters, pubsub and such.

So to the fault tolerant part. If one of the application server dies for whatever reason, then the service should keep working unaffected. HAProxy detects that the specific backend is down and removes it from backend list, so no request is routed to that specific instance. As Redis and MongoDB are both set up in a replicated setup then applications either keep using the old Primary instance of MongoDB and Redis or wait until MongoDB replica set and Redis Sentinel have elected a new Primary and seamlessly switch over.

If your long lived IMAP connection happens to target the instance that goes down then connection is obviously lost. Usually IMAP clients resume the connection immediately and this time HAProxy routes the connection to a healthy instance, so as an user you probably don’t even notice that your mail client reconnected.

Application servers have 50GB of storage each which is not much but as Wild Duck is a lot more efficient than normal mail stores when storing messages to disk, then in reality it should be able to store a lot more messages than 50GB. All messages expire in 7 days, so even if the storage gets full, it’s a temporary problem.

On the left is the combined size of maildir folders for ~2000 users. On the right are the same messages imported to Wild Duck using all available optimization options

“Normal” email servers usually dedicate a specific folder on a specific disk to email users, so your IMAP connections must always end up in the same specific instance. Otherwise your mailbox becomes unavailable. In case of Wild Duck it does not matter to which host you connect to as all data is stored to replicated document storage.

And now to the costs. I host the application in OVH, with each instance using a €2.99 VPS (2GB RAM, 1 CPU). Application instances have an additional 50GB disk attached (€5 each). TLS certificates are handled by Let’s Encrypt (and acme.sh) which are free. So the total costs being 4*2.99 + 3*5 = 26.96€ which roughly translates to $30.

My benchmarks show that this $30 cluster is able to process 30 messages per second (10Mb/s). For testing I used the messages accumulated over the years to my main email address, so the processed messages should reflect everyday usage (different messages, different kind of attachments etc). Not quite enough to run a large scale email hosting but good enough for that money.

Posted on

Ethereal email testing

Today I’m really excited to announce the release of Nodemailer v4.1.0. If you have everything already set up regarding email sending then you probably do not care too much about this release. If you are currently building something or plan to build an app that, amongst other things, sends email, then this new release might interest you.

Nodemailer v4.1.0 includes 2 new API methods

nodemailer.createTestAccount(callback)

and

nodemailer.getTestMessageUrl(info)

where the first one generates an actually working email account out of the blue and the other one returns extra information about a delivery.

These autogenerated accounts are not too real though, these are test accounts from Ethereal.email mail testing service.

When an Ethereal test account is used then Nodemailer establishes a normal SMTP connection against Ethereal SMTP server, authenticates with actual credentials and the server accepts message for delivery, so nothing unusual about it. What is a bit unusual though is that the Ethereal server never does the actual delivery, it stores the message to the account of the authenticated user and that’s it. You don’t have to worry about unexpected deliveries where mail is delivered to actual recipients. Ethereal never sends any messages.

Finally, the getTestMessageUrl(info) method returns a web URL that can be used to preview the sent message in a browser.  You can preview the message HTML, download the RFC822 source of the message or just check the message headers.

You can store the autogenerated credentials and start using these as development credentials instead of spamming a real email account. Or if you do not want to then you can generate fresh credentials for every new test email, it’s your own choice.  If you want to use IMAP to preview the sent messages then you probably want to use pre-generated credentials, otherwise it wouldn’t make much sense.

Posted on

v4.0.0 – back to MIT

Nodemailer v4.0.0 is now released and is using the MIT license. This version does not bring any other changes, it is a republished v3.1.8

Posted on

v3.1.0

v3.1.0 is a non-critical feature release. The main change is first-class support for AWS SES. If you are sending through SES then you do not need any plugins besides the aws-sdk module to use Nodemailer.

The built-in SES support is an improvement over the previous nodemailer-ses-transport plugin. You can instantiate the aws-sdk object as you wish yourself instead of letting Nodemailer handle the setup. Rate limiting is vastly improved, Nodemailer guarantees the most optimal sending speed, without hitting the actual limits. There’s also an option to specify SES related mail options like Tags or ConfigurationSetName. If you use DKIM signing then Message-ID and Date fields are automatically suppressed from the signature, no special config needed.

You can find all the details required to send mail with SES from the Nodemailer homepage.

Posted on

Nodemailer v3.0.0

This post is mostly about the newest release of Nodemailer. For the license changing rant, scroll down past the feature list.

It’s been a while since a major release of Nodemailer happened and now it’s finally here, the initial version of Nodemailer 3! This update brings a lot of breaking changes. I hope to write a separate blog post for each of these but for now, here’s what changed:

  • License changed from MIT to EUPL-1.1. This was possible as the new version of Nodemailer is a major rewrite. The features I don’t have ownership for, were removed or reimplemented. If there’s still some snippets in the code that have vague ownership then notify me at [email protected] about the conflicting code and I’ll fix it
  • Requires Node.js v6+ as a lot of ES6 specific features are used. Some make life easier (eg. classes versus Function.prototype), some make the code better (using let instead of var eliminates several possible memory leak scenarios)
  • All templating is gone. It was too confusing to use and to be really universal a huge list of different renderers would be required. Nodemailer is about email, not about parsing different template syntaxes
  • No NTLM authentication. It was too difficult to re-implement. If you still need it then it would be possible to introduce a pluggable SASL interface where you could load the NTLM module in your own code and pass it to Nodemailer. Currently this is not possible.
  • OAuth2 authentication is built in and has a different configuration. You can use both user (3LO) and service (2LO) accounts to generate access tokens from Nodemailer. Additionally there’s a new feature to authenticate differently for every message – useful if your application sends on behalf of different users instead of a single sender.
  • Improved Calendaring. Provide an ical file to Nodemailer to send out calendar events.

There’s also some non-breaking changes:

  • All dependencies were dropped. There is exactly 0 dependencies needed to use Nodemailer. This brings the installation time of Nodemailer from NPM down to less than 2 seconds
  • Delivery status notifications added to Nodemailer
  • Improved and built-in DKIM signing of messages. Previously you needed an external module for this and it did quite a lousy job with larger messages. The updated version processes messages as streams and also is able to cache parts of the data to disk if the message is very large
  • Stream transport to return a RFC822 formatted message as a stream. Useful if you want to use Nodemailer as a preprocessor and not for actual delivery.
  • Sendmail transport built-in, no need for external transport plugin

See Nodemailer.com for full documentation. I hope you’ll enjoy the new features!

PS. Several people have asked me about the license change. Why use such an esoteric license like EUPL (European Union Public Licence – EUPL v.1.1)? And why a copyleft license, whats wrong with good old MIT/BSD?

EUPL is very similar to GPLv2 as these licenses are more or less compatible. EUPL has the advantage over GPL of having an official translation of the license text in the Estonian language as all EU documents  are translated into all EU languages. I don’t know any other open source license that has such translation. Additionally GPL is more US specific while EUPL is EU specific and Nodemailer is built in EU (in Estonia), so its a natural choice if copyleft license is preferred.

So, why copyleft?

I’ve been building Nodemailer for the last 6 or 7 years and while it has been a source of great mental satisfaction (you know, solving some really difficult problems, squashing hard to find bugs, studying a myriad of email related RFC’s and so on) then it’s been a lousy project in financial terms. I’ve been collecting donations since day one and the grand total I’ve received during this time is around $500 (the sum would be double of that if I hadn’t spent the bitcoin I received long before the price surge). And these donations were made mostly by fellow developers like myself (huge props to all who have ever donated to any of my projects, I really appreciate it!).

Nodemailer, at least the core of it, is not a community project. Obviously I’ve received a lot of help during the years but nevertheless I consider it a solo project, every feature and every change directly translates back to the time I’ve spent building this project.

The problem how I see it, is that Nodemailer is not used only by fellow developers like myself, it is also used by larger companies that do not tend to give anything back. I’m aware that no company is successful because of using Nodemailer, I’m also aware that using Nodemailer saves a lot of developer hours because of the easy to use API and the Just Works approach. A lot of the value is under the hood – for example Nodemailer gives its best to build as compliant messages as possible to avoid being marked as spam (there’s a ton of different compliancy tests spam filters usually check for and messages generated by Nodemailer should pass them all). Nodemailer also tries to be as resource friendly as possible – this in turn means faster sending speed and better performance. To arrive into such state has taken huge efforts on my side and I see companies getting a lot of value out of it but I don’t see any gains for myself.

In short, that’s why I decided to change the license. I don’t feel like I should be sponsoring big companies with my unpaid time. I can’t (and I don’t want to) revert already published versions but for the new stuff I go with copyleft. For most people this should be just as fine as anything else. If, for whatever reason, copyleft is not acceptable, then there’s always the option to purchase a commercial license without the limitations of copyleft or use something else to send the emails, for example vendor provided API clients or some other SMTP client.

Posted on

The mess that is attachment filenames

If we wanted to send an email with some attachments then we can do this, nothing easier. If we wanted to use emojis in the filename of an attachment, then, in most cases, we can do this as well. But how did we end up here, has this been always possible? Let’s dig in!

Email attachments are one of the most used email features but this hasn’t always been the case. In the dark ages of email, messages were only plain text ASCII, so no-no to attachments. Today we assume that attachments became possible with the addition of Multipurpose Internet Mail Extensions (in short MIME) that was described in RFC1341 back in 1992. This is not quite accurate as well. The first attachments were actually uuencoded files included in the plaintext message body that pre-dates MIME. And in fact modern email applications often still support these kind of attachments.

Unsurprisingly the uuencoded attachments in non-MIME messages have a limited range of characters that can be used in the filename. Basically just plain ASCII (No emoji. Total loser). File naming was much simpler back in the day.

An RFC822 formatted email message with a text part and an attachment called test.txt would look like this (↵ marks newlines):

From: ...↵
To:...↵
Subject: This is a test message↵
↵
Test content↵
↵
begin 644 test.txt↵
#0V%T↵
`↵
end↵

And when opening such message with a modern email application we can indeed see that there is an attachment called test.txt attached.

These prehistoric attachments are not too interesting though (unless you’re a phisher wanting to bypass some attachment related security checks), so let’s move on.

As already mentioned, 1992 gave us MIME and a standardized way to include non-text parts in a message. This is also the attachment system as we know it today – you have a multipart data tree where each node has its own separate header and body block. If the body block needs to include non-ASCII text then this is also possible, the content is encoded and the encoding scheme is noted in the header of that node. So the same uuencoded message could be converted to MIME like this:

Subject: This is a test message↵
MIME-Version: 1.0↵
Content-Type: multipart/mixed; boundary=abc↵
↵
--abc↵
Content-Type: text/plain↵
↵
Test content↵
↵
--abc↵
Content-Type: application/octet-stream;↵
    name="test.txt"↵
Content-Transfer-Encoding: base64↵
↵
Q2F0↵
--abc--↵

And when opening such email message with an email client we see this:

Looks pretty much the same, doesn’t it?

So where did the filename test.txt in the screenshot came from? If we look at the source then we see that the Content-Type header has an extra argument called name that defines the filename. This is already pretty awesome, we can use quoted parameters in MIME and inside the quotes we can use almost any (but no all) printable ASCII characters we like. So no filenames that have special characters like quotes and still lightyears away from emojis but we already can have words separated by spaces as filename. If we would like it so.

At this point the non-english speaking countries are starting to tag along and this means a new challenge: non-ASCII characters in message header. Message bodies already can use it, you can set a “charset” argument to the Content-Type header, use some encoding (usually either Base64 or Quoted-Printable) and you’re ready to send some mail, assuming you’re all right with using only latin characters in the subject line and also in the attachment filenames.

So in parallel to the RFC1341 a new standard was proposed, RFC1342 that defined something called “encoded-words”. It’s a construct that using only ASCII symbols presents sequences of encoded characters in any character set. If a word in our subject line includes a non-ASCII character, for example an umlaut “test messäge, awesome!” then the Subject header could be formatted like this

Subject: test =?ISO-8869-1?Q?mess=E4ge,?= awesome!

Unfortunately, encoded-words come with a restriction: these can only be used as an atom and not inside quotes as anything that is quoted is meant to be kept the way it is presented.

If we would try to use encoded-words as an attachment filename without quotes it would look obviously wrong

Content-Type: application/octet-stream;↵
    name==?ISO-8859-1?Q?mess=E4ge.txt?=

Notice the double equal sign? This doesn’t seem right at all.

So even if we gained support for non-latin characters in the Subject line then we are still limited in our options when using attachment names.

We needed an alternative solution and even if it took some time we finally got there.

In 1996 RFC2045 re-defined the Content-Type header but this time there was no explicit “name” parameter mentioned. In 1997 RFC2183 fixed this by adding a new header Content-Disposition that has a parameter called filename, suitable for, as the name suggests, defining attachment filenames.

In parallel to RFC2813 another standard was released, RFC2814 that together with the newly defined filename parameter finally fixed the long standing issue of non-latin attachment filenames. This was done by introducing yet another encoding scheme, Parameter Values and Parameter Value Continuations. When encoded-words used equals sign to indicate encoded characters and question mark to separate different parts of the scheme then Parameter Values use percent sign for encoding and single quote for separation.

Content-Disposition: attachment;↵
    filename*=iso-8859-1''mess%4Ege.txt

Asterisk in the end of the parameter name indicates that the value uses Parameter Value encoding.

The same mechanism allows splitting long values into multiple chunks (thats the Continuation part) but this is not super important, so we will not cover it here. What is super important, is that the changes done in 1997 allow us to finally use emojis in filenames even if emojis as characters weren’t invented yet.

If we want to use a filename like “😁😂’.txt” then this is totally doable with the Parameter Value scheme. We need to provide correct charset and also encode the bytes in the value. It would look something like this:

Content-Disposition: attachment;↵
    filename*=utf-8''%F0%9F%98%81%F0%9F%98%82.txt

On a good day this would end our journey because this is exactly how emojis in attachment filenames are supposed to work. In the real world things are not as easy.

Let’s send a message with an attachment name formatted this way to for example an email address hosted by QQ, a huge chinese email provider. How does the QQ interface display us the attachment? Assuming that the chinese are accustomed to using non-latin characters then this should work as intended? No?

Apparently no. It seems that QQ does not handle these characters at all, as this is how the attachment is shown in the web interface

Do you really have to keep using only latin characters when sending to a chinese service? Fortunately not. QQ webmail does support non-latin characters, including in attachment names, they just do not follow the standards for that.

I guess that they haven’t yet catched up with the latest trends of 1997

It appears that they still go with the Content-Type header with name parameter where they allow to use encoded-words in quotes.

So what you really need to do in this case would be to add the same filename to attachment headers twice. Once to Content-Disposition, using the standard Parameter Value encoding and once to Content-Type, using encoded-word encoding.

Content-Type: application/octet-stream;↵
    name="=?utf-8?Q?=F0=9F=98=81=F0=9F=98=82.txt?="↵
Content-Transfer-Encoding: base64↵
Content-Disposition: attachment;↵
    filename*0*=utf-8''%F0%9F%98%81%F0%9F%98%82.txt↵

And the result is exactly what we wanted, an attachment named with emojis.

More standard obeying email clients would use the name shown in Content-Disposition. QQ and some other less standards-savvy clients would use the Content-Type header.

Nodemailer obviously handles all of this out of the box.