Removing duplicates from your iPhoto library
While I'm frequently called upon by friends and family to assist in cleaning up their iTunes duplicates, I was recently asked by a friend to help clean up duplicates of a different sort: an iPhoto library. At first glance, I knew this was going to be a tough job: duplicates were littered throughout my friend's library, and there was no visible pattern to it. So, rather than manually sifting through what was, at the time, a 10GB iPhoto library -- which would've tested the limits and accuracy of my eyes, as well as taking me through my AARP years to complete -- I decided to think different -- 'cause that's what us Mac users do. After doing some research, I stumbled upon Brattoo Propaganda's Duplicate Annihilator (link). The end result was a 10GB iPhoto library trimmed down to 6GB.
Using a host of algorithms, Duplicate Annihilator scans your entire iPhoto library (or selected photos) to detect for duplicates. By default, it does this by comparing the photos' MD5 checksum; however, the photos' CRC32, Exif, file name, and photo dimension information can be enabled for comparison as well.
After finding duplicates, Duplicate Annihilator can perform a host of actions. One is to add a comment to the photos' "description" field. And it can do this to originals as well as duplicates. While users can choose to replace existing content in the field, the safer method is to append to the field, rather than replace it. So, say you have a picture with "Mary getting Carl Weathers' autograph" in the description field, the duplicate of it would have "Mary getting Carl Weathers' autograph-duplicate" instead.
And then there's the Popeil Showtime Rotisserie oven option, which allows for a "just set it and forget it" means of ridding yourself of duplicates: you can move duplicates directly to the trash. In addition you can also set Duplicate Annihilator to empty your trash. That's right, in just several installments, your duplicates will be completely gone. But, unless you're the daring type, you really shouldn't select this option.
As in most situations like this, user discretion is advised. In other words, backup your library before doing anything of this sort (you'll find it in ~/Username/Pictures). Because, as the old saying goes, a picture is worth a thousand words. And this being the case, permanently deleting pictures may result in several thousand unwanted words hurled your way.
Duplicate Annihilator is shareware and is available for $7.95.
![TUAW [Cafepress]](http://www.blogsmithmedia.com/www.tuaw.com/media/tuaw-cafepress-promo.png)
Reader Comments (Page 1 of 2)
Jerry said 3:36PM on 7-06-2009
What i wish it could do was replace duplicate pictures in albums with links to the original.
Pre iPhoto, I used Picassa on the PC and it used folders instead of albums to organize photos. That meant to make more than one album, I ended up duplicating the picture. That's how I ended up with duplicates.
Reply
DJFriar said 4:30PM on 7-06-2009
I don't see the need for this if you use keywords extensively. For example, in my library if I have a picture of my wife and I, its tagged with both our names. It may be located in her event, my event or something like "christmas 2008" event, but a simple keyword search pulls it up, or the smart album will already have it.
Essentially, it doesn't matter where the picture is, since the keywords are how I am going to find it anyway.
And if you do use keywords a lot, you NEED Keyword Manager (http://www.bullstorm.se). I can't even use iPhoto without it now, its awesome.
But yeah, using keywords I don't understand the need for a "link" to anywhere.
RicoSuave said 3:45PM on 7-06-2009
Jerry,
Will this help?
http://dougscripts.com/itunes/scripts/searchTheScripts.php
Reply
TroLoo said 4:00PM on 7-06-2009
What about photos converted/edited/changed that remain as original and copy? Is there any way to remove originals that we no longer need because we have a tuned copy?
Reply
jamkam said 4:01PM on 7-06-2009
Isn't time Aperture users had something like this?
Reply
Josh M said 4:05PM on 7-06-2009
Tidyup also has an algorithm where it can do this AND add links to the original file (can't remember the term for those links)
Reply
jamkam said 4:29PM on 7-06-2009
I've looked at Tidyup. A plug-in that's Aperture-specific would be much nicer and make my hair hurt a lot less, tho.
Reply
ADT2 said 4:19PM on 7-06-2009
Unfortunately, Duplicate Annihilator leaves a bit to be desired. After running it on my library, which I know for a fact contains hundreds, if not thousands, of duplicates, I am still left with duplicates.
Seems like I remember somewhere an application (or maybe a service?) that grouped images based on their actual visual similarity; if I could sort my photos that way, it'd be a lot easier to sift through them and remove the offending dupes.
Reply
Brattoo Propaganda Software said 4:38PM on 7-06-2009
Dear ADT2,
sometimes the default settings of Duplicate Annihilator won't catch all of your duplicates and you might therefor have to experiment with different combinations of detection algorithms. There are a few examples in our documentation (available online) and there is the "compare images" tool within Duplicate Annihilator that is useful when trying to determine which settings to use. If you still encounter problems with duplicates please feel free to contact us.
Elektrozzo said 4:53PM on 7-06-2009
My library is full of duplicates. After a data recovery of a broken hdd my library is full of same images in different sizes. From thumbnails to high res photos. I have no idea how to grind through all these data (around 14gb). Because of that they have all the same date and so on... Dont think that this program does work on my problem
Reply
David Hildreth said 5:19PM on 7-06-2009
How do people manage to get so many duplicates?
Reply
Ben Y said 5:27PM on 7-06-2009
I think the real issue here is: Why the **** hasn't Apple put this feature in iPhoto yet? It has facial recognition built in, puts your pictures on a ****ing map for you, etc. etc. But it can't clean up DUPLICATE PICTURES?
FAIL
Reply
hansning said 9:53PM on 7-06-2009
well, my lib went from 8.7 to 8.5, so it removed about 200mb, but that was using the shareware, and only using MD5. Not bad i suppose, since it couldn't scan all 5000 at once.
Reply
Brattoo Propaganda Software said 10:31PM on 7-06-2009
Dear hansing,
Duplicate Annihilator has no limitations on how big iPhoto Libraries it can process. We have had customers who have processed libraries containing over 250.000 images. Send us an email and we'll figure out what's causing your problems.
hansning said 11:39PM on 7-06-2009
Sorry, I think I wasn't clear. I was just trying out the shareware with the 500 photo limitation. I'm assuming that the 500 scanned aren't compared to the rest of the library.
Joseph said 11:52PM on 7-06-2009
it always sketches me out when someone comes into the blog to sell their software. yikes. I have to applaud the effort and ability to reach their customers where they are at. But it sketches me out too.
Anyone else feel the same?
hansning said 12:12AM on 7-07-2009
i dunno, i kinda like it actually.
artifex said 3:14PM on 7-07-2009
He or she's not pushing it, just giving support.
If I had a product and I heard people were discussing it, I'd try to be there in case people had questions, too.
corcoranm said 8:44PM on 7-06-2009
Aside from duplicates hogging space- i have another issue, perhaps you have a solution. I have a TON of photos that i've gotten from other people throughout the years that are their original size, around 1MB. Is there a way to batch compress or batch shrink these like the photoshop 'save for web' feature?
Reply
karl said 8:59PM on 7-06-2009
The problem I have with removing duplicates from either iTunes or iPhoto is that I usually play music from playlists and view photos in albums. When I have duplicates, I don't know which is in the playlist or album. I wish the program(s) would put the non-annihilated tune and/or photo in the playlist and/or album.
Reply