Removing Duplicate E-mail Messages From A Mailbox

Occasionally your mail delivery scheme might hiccup, leaving you with duplicate copies of email messages sitting in your mailboxes. I find this happens occasionally if something goes wrong with fetchmail – you kill the fetchmail process before it has expunged the deleted email from the remote POP3 server, so the next time you run fetchmail it downloads a second copy of each email. This is a simple process that I came up with to remove duplicate email messages from a maildir format mailbox.

As a bit of background, a maildir mailbox is a small directory tree:

$ du .boxes.xml-dev
4       .boxes.xml-dev/tmp
124     .boxes.xml-dev/new
52340   .boxes.xml-dev/cur
52920   .boxes.xml-dev
$

Hierarchy is represented by components of the mailbox name separated by dots, so the mailbox above is called xml-dev and it is in the boxes mailbox. Messages are files in either the new or cur directories. Transport agents place messages into the new directory. When a user agent opens a mailbox it moves all the messages from new to cur. If you’re accessing your mail through an IMAP server like Courier-IMAP the IMAP server will deal with this for you.

1. Make sure there’s nothing sitting in the new subdirectory.
   $ ls new
$
   
   If there are messages in the new subdirectory, open the mailbox in a user agent to get it to move them into cur.
2. See how many messages you have:
   $ ls cur | wc -l 842
$
3. Check they all have Message-IDs:
   $ for i in cur/*; do reformail -x Message-ID: <$i; done | wc -l
842
$
4. See how many you have if you filter out duplicate Message-IDs:
   $ for i in cur/*; do reformail -x Message-ID: <$i; done | sort -u | wc -l 698
$
5. See how many we’re going to delete:
   $ rm /tmp/dups $ for i in cur/*; do reformail -D 20000 /tmp/dups <$i && echo $i; done | wc -l
144
$
expr 698 + 144
842
$
   If this total doesn’t match you should increase the 20000 – reformail isn’t remembering enough Message-IDs to spot all the duplicates.
6. Delete the messages and check things look right afterwards:
   $ rm /tmp/dups $ for i in cur/*; do reformail -D 20000 /tmp/dups <$i && rm $i; done
$ ls cur | wc -l
698
$