Backing Up Flickr Photos with Amazon S3

I love that I now have an Amazon S3 billing page that reads like a really cheap phone or water bill. I think that they’re silently changing the game (again) without telling anyone else. I really like the implications of this magepiebrain post and decided to start using S3 “for real” myself last night.

The first ingredient was James Clarke’s flickr.py. Getting a list of my photos is pretty simple:

import flickr
me = flickr.people_findByUsername("postneo")
photos = flickr.people_getPublicPhotos(me.id)

The second ingredient for getting the job done was a pythonic wrapper around the Amazon example python libraries by Mitch Garnaat called BitBucket. Because it builds on the example libraries, there’s very little error checking, so be careful. Check out Mitch’s site for some example BitBucket usage, it’s pretty slick.

Once I was familiar with both libraries, I put together a little script that finds all of my photos and uploads the original quality image to S3, using the flickr photo ID as the key. Here’s the complete code for flickrbackup.py, all 25 lines of it.

After uploading 160 or so photos to Amazon, I owe them about a penny.

Getting photos back out is really easy too:

>>> import BitBucket
>>> bucket = BitBucket.BitBucket("postneo-flickr")
>>> bucket.keys()
>>> bits = bucket[u'116201243']
>>> bits.to_file("photo.jpg")

56 Responses to “Backing Up Flickr Photos with Amazon S3”


  1. 1 Jeff Milner Mar 22nd, 2006 at 4:17 pm

    This seems like a pretty nifty thing to do, but other than the novelty of it, why would you want to? In case you decide you don’t want to renew your pro account or something?

  2. 2 Thomas Hawk Mar 22nd, 2006 at 5:06 pm

    If you use the Slickr screensaver it automatically downloads high res versions of all of your photos to your hard drive for free. Given that 10,000 photos is only about 5 gigs of storage, seems easy.

  3. 3 Franck Arnulfo Mar 22nd, 2006 at 5:12 pm

    Don’t need Slickr to backup your photos, use Flickr Backup : http://sunkencity.org/flickrbackup instead.

  4. 4 Andy Mar 22nd, 2006 at 5:30 pm

    I’ve been trying to get all my Flickr photos ~3,000 on my hard drive for a couple of days. FlickrBackup doesn’t seem to work for me. It usually crashes or hangs when trying to get all 3,000 and if i try to download a set I just get the public ones, even though I’m logged in and authenticated.

  5. 5 Andy Mar 22nd, 2006 at 5:39 pm

    Anyone know of any other Flickr Backup options? (to local hard drive…eventually)

  6. 6 Armaan Mar 22nd, 2006 at 8:25 pm

    I have an honest question… why would one ever need to copy all their photos from flickr _back_ onto their hard drive again? That’s where they all started… why not just save the original copies?

  7. 7 Matt Croydon Mar 22nd, 2006 at 8:48 pm

    Armaan:

    Actually most of my photos originate and only exist on an RS-MMC card on my mobile phone. I just had my previous card crap out on me, so an extra backup gives me that warm fuzzy feeling.

    –Matt

  8. 8 Ajay Mar 23rd, 2006 at 12:12 am

    You can try more cool flickr stuff with these third party tools
    http://www.quickonlinetips.com/archives/2005/03/great-flickr-tools-collection/

  9. 9 Jason Mar 23rd, 2006 at 5:46 pm

    This is nice, but how about a script that will back up any files you want to the S3 service?

  10. 10 Mike Mar 23rd, 2006 at 7:24 pm

    I enjoyed your post and the thought process behind what you’re doing. I work for a company called Streamload. We recently launched our own web services API for our online storage platform. For non-commercial uses it’s totally free and any user can have 25 GB of online storage free. In the coming weeks, we’ll be exposing other really cool features via the API (like digital video transcoding, photo re-sizing and rotating and music bitrate conversion), so that whatever you’re building can go well beyond just tapping into an existing storage and bandwidth infrastructure. In fact, we built our new consumer online media center service, Streamload MediaMax, using it. You can find more information on it at http://webservices.streamload.com.

  11. 11 Andy Mar 23rd, 2006 at 7:56 pm

    At one time they were all on my computer, but i’ve had a lot of hard drive crashes and computer changes over the last couple of years. Eventually I just figured they were safe on Flickr so I’d just upload new ones from whereever and keep them their, but I’d like to have them all locally again so I could upload them to Riya.com and give it a try.

  12. 12 Daniel Drucker May 7th, 2006 at 8:37 pm

    I’d like to see this include the all-important metadata as well - I spend many many hours tagging my photos and putting them into proper sets (and in the case of older scanned photos, properly dating them).

    I’d like to see this utility save the metadata via the Flickr API to an XML file, and upload that information to S3 too. A backup isn’t really a backup if you can’t restore from it - and you can’t restore from your utility as it stands now, because you don’t have the metadata saved.

  13. 13 Iain Hindshaw Jun 18th, 2006 at 6:17 am

    Just found your home page its great, it looks like you folks do great service keep up the good work.

  14. 14 Priyadarshan Jul 25th, 2006 at 8:24 pm

    Unfortunately it seems the tags, the description and all that kind of metadata will not be backed up. Does the Flickr API not allow that? What about the comments?

  15. 15 Daniel Drucker Jul 26th, 2006 at 9:05 am

    This is really NOT a good way to back up your Flickr photos if you care about your tags, metadata, etc.

    The Flickr API absolutely DOES allow all that stuff to be backed up - I do it once a month via a perl script, using Net::Flickr::Backup. Using that, all my Flickr photos are backed up to files, and all the metadata for each photo is backed up to an associated RDF file that contains tags, notes, comments, exif data - absolutely everything Flickr knows about the photo.

    There is nothing preventing this script from doing the same!

  16. 16 kalsdi2ko2 Aug 5th, 2006 at 5:13 am
  17. 17 Smith Aug 27th, 2006 at 10:17 am

    Thanks so very much for taking your time to create this very useful and informative site. I have learned a lot from your site. Thanks!!

  18. 18 Michel Bohn Mar 6th, 2007 at 9:03 am

    I found this one better for windows. Based on .net. http://greggman.com/pages/flickrdown.htm

    Very easy to use and stable.

    Mike

  19. 19 inside Mar 18th, 2007 at 4:39 pm

    Hi, this site is very nice…

  20. 20 AFRif Mar 28th, 2007 at 8:26 pm

    Not bad, it really can occur

  21. 21 MaryJames May 23rd, 2007 at 5:31 pm

    Hello all

    How I can change avatar in this forum?

  22. 22 MaryJames May 23rd, 2007 at 6:00 pm

    Hola all

    How I can change avatar in this forum?

  23. 23 MaryJames May 23rd, 2007 at 6:03 pm

    Wazzup all

    How I can change avatar in this forum?

  24. 24 cbmeeks May 24th, 2007 at 4:15 pm

    I just recently destroyed years worth of work (well, doodle work but still) the other day. Totally forgot about it. It was mostly pictures, old OLD programs, etc.

    Not too terribly important. But I formatted the drive without thinking (or caring).

    Only to suddenly remember all of the stuff I lost. My bone-head mistake.

    So, I took what I do have and started backing it up everywhere. The more the better.

    So for me, automated python scripts that take stuff from my server and put them everywhere is really cool.

    http://www.signaldev.com

  25. 25 roma Jun 8th, 2007 at 9:03 pm
  26. 26 yura Jun 9th, 2007 at 1:49 am
  27. 27 vasiliy Jun 14th, 2007 at 6:18 am
  28. 28 dimon Jun 14th, 2007 at 6:49 am
  29. 29 eemwclfgth Jun 19th, 2007 at 10:08 am

    Hello! Good Site! Thanks you! folbtedrkxpbg

  30. 30 datindkrn Aug 22nd, 2007 at 1:28 pm

    My name is Elena. To me of 20 years.
    I would like to get acquainted with you if you not against.
    I Look forward to hear you soon with impatience. Dating[url=http://toplop.com/?idAff=59]Dating[/url]

  31. 31 datindkjg Aug 23rd, 2007 at 2:58 pm

    My name is Elena. To me of 20 years.
    I would like to get acquainted with you if you not against.
    I Look forward to hear you soon with impatience. Dating[url=http://toplop.com/?idAff=59]Dating[/url]

  32. 32 datindkjg Aug 23rd, 2007 at 2:58 pm

    My name is Elena. To me of 20 years.
    I would like to get acquainted with you if you not against.
    I Look forward to hear you soon with impatience. Dating[url=http://toplop.com/?idAff=59]Dating[/url]

  33. 33 careybagsbon Nov 18th, 2007 at 1:27 pm

    Greetings to all.

    Prompt the best online shop on sale of Books.

  34. 34 Katy Jan 8th, 2008 at 5:41 pm

    Hi,

    As for a software to download images from Flickr (to your hard drive first) and then to other online services, I found Migratr very useful.

    I also have Slickr, and find it a neat way to share photos with my Dad. I installed it on his computer, and now whenever I tag a photo in Flickr as “fordad” it shows up in his screensaver.

  35. 35 bob Jan 17th, 2008 at 3:59 am

    EXSTOY hi nice site thx http://peace.com

  36. 36 download file mp3 Feb 2nd, 2008 at 2:08 am

    adobe insurance nc insurance renewal notification state mandate
    [url=http://www.conectarnos.com/search.php?query=site:greatlistquotes.com&pr=&image=&www=true&start=20]selective auto insurance company of new jersey[/url]
    state of florida legislation auto insurance

  37. 37 zithromax for urinary tract infection Feb 6th, 2008 at 10:09 pm

    levaquin allegra
    [url=http://forum.bestechvideos.com/users/67/posts ]zyvox[/url]

    linezolid mrsa treatment

  38. 38 bob Feb 20th, 2008 at 11:01 am

    CHC9xD hi good site thx http://peace.com

  39. 39 bob Feb 20th, 2008 at 11:57 am

    JnQUV5 hi nice site thx http://peace.com

  40. 40 bob Mar 2nd, 2008 at 12:24 pm

    vySA75 nice site thx http://peace.com

  41. 41 wefolp Mar 3rd, 2008 at 3:31 pm
  42. 42 mona Mar 6th, 2008 at 4:38 am

    hi great site 10x

  43. 43 ben Mar 6th, 2008 at 10:04 am

    interesting site man

  44. 44 bang Mar 24th, 2008 at 11:54 am

    ARubHo great site man thx http://peace.com

  45. 45 bob Mar 24th, 2008 at 11:54 am

    V08Qel hi nice site thx http://peace.com

  46. 46 Face Lift Apr 29th, 2008 at 12:18 pm

    thx for the information

  47. 47 zhgrudkxew May 3rd, 2008 at 9:06 am

    Wow, cool man, big thanks! http://lkensxtoujtn.com

  48. 48 Christopher Sinnott May 22nd, 2008 at 9:30 am

    Tampa,

  49. 49 miky79 Jun 24th, 2008 at 3:53 am

    I found this Guide to online backup on Wikipedia! I thought it was extremely helpful so I put it here to share! (http://memopal.clickmeter.com/891931.html)! I just discovered online backup and I think it’s a good way to protect data! Can anyone confirm this???

  1. 1 Tiernans Comms Closet Trackback on Mar 22nd, 2006 at 4:33 pm
  2. 2 townx Trackback on Mar 31st, 2006 at 7:05 pm
  3. 3 Jennie Finch Photo Trackback on Jul 28th, 2007 at 9:03 am
  4. 4 does antivert work effectively Trackback on Aug 12th, 2007 at 12:31 am
  5. 5 stream video Trackback on Aug 23rd, 2007 at 1:26 am
  6. 6 women and men difference Trackback on Sep 13th, 2007 at 12:31 am
  7. 7 penis enlargement Trackback on Sep 22nd, 2007 at 4:25 am

Leave a Reply