Colin.Guthr.ie Illegitimi non carborundum

26Feb/093

Migrating your old IMAP to Google Mail

Some years ago, my mum needed email access from several locations but didn't wand to use a webmail system as her primary interface. Well using an IMAP system made sense. It could provide a webmail frontend when it was needed, but use a proper email client when at her workstation. At the time her ISP didn't offer an IMAP account, so I setup a simple box at her house to handle her incoming mail and give her a Horde IMP webmail interface. Her broadband connection was fast enough that she could connect to it when she was away from home and do the essential tasks.

Over the years, maintaining this box has become more and more hassle, not to mention that the box is an ugly biege thing that makes a lot of noise!

Thankfully, Google now offer IMAP access to their email system and also allow you to register with your own domain name. Perfect!! But with over 2.5gigs worth of data in several hundred thousand emails, migrating to the new system and still keeping the messages looked like it could be a challenge. Well it was! But it wasn't all that bad thanks to imapsync, although there were still some problems I encountered that I'm documenting here so that others will hopefully benefit!

The first problem I ran into was with the "folders" system on GMail. Firstly Google implements this as a "label" but you can get the same basic functionality of folders and when accessing via IMAP it doesn't really look much different. The main problem is that these labels are limited to 40 characters! Some of the folders my mum had used were ultimately longer than 40 characters (the name of the parent folder(s) is used in the label as is the separating /). So I had to find all her folders that were longer than 40 characters and rename them. As the IMAP server I was migrating from was Cyrus-IMAP, I was able to list all her folders via the cyradm system. I then passed the output through some sed magic to strip off the unnecessary guff and then fired it through a little bash loop and wc -c to print the length of each folder. I only printed out the ones that were greater than 40 and then got me a list. Sadly the only thing to do here is to think up new names, but after doing this and with a bit more shell script magic I was able to generate a list of: rename "from" "to" commands that I could dump into cyradm.

The next problem tuned out to be that Google doesn't like labels that end with whitespace. That's fair enough but due to various typos in the past, a few of the folders turned out to have spaces at the end. So again a bit more sed magic and some renames and I'm good to go.

Getting imapsync to run was pretty easy, but you have to handle the sent mail specially due to the [Google Mail] namespacey prefixy thing when accessing GMail via IMAP.

The first command I issued was:

./imapsync --syncinternaldates --useheader 'Message-Id' --skipsize \
--host1 myoldimap.server.com --user1 myolduser --passfile1 pass1.txt --ssl1 \
--host2 imap.googlemail.com --port2 993 --user2 myemail(at)myhosteddomain.org --passfile2 passfile2.txt --ssl2 \
--exclude Sent

The options "--syncinternaldates", "--userheader 'Message-Id'" and "--skipsize" are all recommended by the imapsync FAQ in some capacity. You need to use SSL at the GMail side and as you can see above, I also enalbe SSL on the source side too - safety first!! I also exclude the "Sent" folder as I will handle this separately due to the namespace issue I mentioned above.

This (for me) took about 25 hours to run!! Even then it didn't work flawlessly (if truth be told, the problems I mentioned above were only discovered at this stage and I ran this above command several times over throught the course of this migration!). it's quite safe to run this multiple times, but with >2Gigs of mail it can take serveral hours to go through each run.

Once I was happy with the results from the initial upload, I transferred the Sent mail:

./imapsync --syncinternaldates --useheader 'Message-Id' --skipsize \
--host1 myoldimap.server.com --user1 myolduser --passfile1 pass1.txt --ssl1 \
--host2 imap.googlemail.com --port2 993 --user2 myemail(at)myhosteddomain.org --passfile2 passfile2.txt --ssl2 \
--folder "Sent" --prefix2 '[Google Mail]/' --regextrans2 's/Sent/Sent Mail/'

 

That's all there is to it really! I appreciate I've been a bit hand-wavey with the cyradm tweaks I did, but if truth be told I only used this to save me time. Everything I did there would be possible via any run of the mill IMAP client app, although it may be harder to spot which folders will translate to labels larger than 40 characters.

Anyway, I hope this helps some people in the same situation.

Share and Enjoy:
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks
  • Identi.ca
  • Slashdot
  • I’ve used imapsync too when migrating 30 IMAP accounts on a rented server, totalling 45GB, to another server at another ISP, and it worked great for me. It took half the night.

    The main advantage of using this over drag and drop in a client is that you can use it from the hosted server, which has a much much better bandwidth than typical DSL connections — 100MBps upload versus 1MBps makes a huge difference.

  • Licia

    This is very useful. I need to move all my mail out of my old IMAP and I want to use google mail (so I will not have to deal with incomprehensible instructions from sys admins on how to manage my email). I have tried to copy emails over folder by folder (drag and drop) but when folders contain more than say 10 emails the copy is incomplete (%^&$^$!!!) and only few random emails get copied over (why?)
    For an uninitiated like me who does not have access to the server hosting the email,(which does not support POP), what can I do to copy my email over to gmail?
    Do I have to install imapsync first? and where? would any machine do? then can I type that long command from a slow internet connection? what happens if I have to log out (or connection dies) before the transfer is finished?
    Thanks a lot!

    • Colin

      Hi Licia,

      First of all you don’t need access to either server hosting the mail (well you almost certainly can’t get access to the Google Mail servers anyway!!). You just need access to some machine somewhere that can run the imapsync process. Imapsync is just a regular imap client (like your current mail program – i.e Thunderbird, Apple Mail, Evolution or (if you are really unlucky) Lookout ExtrastressOutlook Express.

      You tell imapsync your mail passwords for both your current IMAP and your Google Mail and it simply logs into both and then copies the mails across for you creating the folders etc as needed.

      As imapsync is a console app, if you have any kind of console access (i.e. ssh) to a unix/linux/OSX machine (not tried it on windows but as it’s perl it should work there too I guess?) you can run the “screen” app and then run imapsync inside screen, allowing you to detach and login again later.

      If the server’s connection dies then imapsync will terminate, but as it’s a sync program is should be safe to run it several times and it should fill in the missing mails.

      Hope this helps.

      Col

  • Pingback: IMAP su Gmail e AOL « Carlo Mazza Home()