Accessing IMAP accounts usually is handled by two approaches – either we want to show the snapshot of current mailbox state, that is usually what happens when you log in to a webmail application to see your emails. Or we want to track changes on an account and somehow either display or record these, eg. when doing incremental backups or some kind of synchronisation.
While it is fairly easy to show the current snapshot and it is not hard to track new emails coming in then there is one particular change that is usually quite difficult to track. That is message deletions. Sure we can enter into IDLE state or NOOP-loop and start listening for those EXPUNGE notifications but consider the following:
- IDLE is for a single folder only. You only see what is happening in the currently opened folder
- IMAP connection counts are usually limited, you can’t open a separate connection for each folder
- Reconnects happen, both because of network issues and also forced logouts. When logged in again you don’t get notifications for the events that happened when you were disconnected
- Even if you get EXPUNGE notifications, these are against sequence numbers not against ID’s, so you must be super sure that the sequence number on your end corresponds to what the server thinks it is
So what to do to overcome these nuisances?
Was something even deleted?
First is to check is if something has even been deleted. No need to do anything if all the messages are still there.
IMAP servers expose a value called UIDNEXT which is the predicted UID value of the next message that is going to be stored in that folder. Usually it is the latest UID + 1. If we store the combo of the count of stored messages and UIDNEXT and then come back and both values are still the same (or both values have grown equally without any skips) then we can be fairly certain that no message was deleted in between.
Some servers have the CONDSTORE extension enabled that exposes MODSEQ values. We can also store that value for the folder and if it hasn’t changed then no messages were deleted from the folder. Though MODSEQ does not track only deletions but any changes, including stuff like Seen/Unseen flag changes, so the value actually changes quite often.
Sequence numbers
When receiving those EXPUNGE notifications we get a sequence number of a deleted message in currently opened mailbox. This means that we have to have the correct sequence pointers of messages on our end as well. The easiest (not the best though) is to issue a UID SEARCH ALL command after a folder is opened, store the returned list in sorted order and use it as the base of our sequencing.
If we then get an EXPUNGE notification then we treat the list as 1-based array, resolve the UID value from the sequence position and then remove that element from the list. This way we know the UID of the deleted message and also keep the sequencing in sync.
Diff
If we have detected an anomaly between the expected message count and actual message count when opening a folder then we can assume that something was deleted. We do not know what exactly was deleted, we only see a list of messages that are still there but no indication of what happened in between. The easiest approach is to compare the UID sequence list that we discussed in the previous point against UID SEARCH ALL command result. This approach assumes that we actually store the UID sequence list somewhere and keep it updated. Anything listed in our stored list but missing in the server provided list was deleted and anything listed in the server response but missing in our stored list, is a new message.
And that’s it, this is how “easy” it is to track deleted messages on an IMAP account. Indeed, SEARCH ALL command is a bad approach when used against a folder that contains a lot of messages but it is the easiest to implement, and you can always move on from there to better approaches. Or alternatively if you do not want to implement IMAP tracking logic on your own then you can use something like IMAP API that does the tracking itself and notifies you on every change on the account via web-hooks.