I am regularly asked about how I keep and process my images. Also regularly I experiment with new techniques, new tips I get from browsing the web or hear from other fellow photographers. Sharing knowledge is as much important in photography as it is to create great images, all great photographers do love to share with other photographers beginners and advanced alike. Today I’ll talk about general Digital Asset Management and some basic principles…
Digital Asset Management
Let’s start off by telling you: I am a Mac user, many would consider me as an Apple fanboy… and I am one!
After I started digital photography, one thing came quickly clear was that files did accumulate fast creating the need to organise them well and for the future. I applied and still continue today a scheme with a folder structure based on shooting date: year/month/day.
At the time post-processing meant using photoshop (maybe 7 or so) which was not user friendly and hard to use (it’s still hard today but much less). Looking for photos was difficult but at least they were not all stored in a big single folder with thousands of photos… Also there was relatively limited means to properly process pictures, in particular photoshop was not able to process RAW files (Adobe Camera Raw did not yet exist) nor able to handle 16-bit images (Yes not only I am a geek, but I am also a data hoarder, playing with a 16-bit image is not useful unless you print super large formats but I just can’t stand loosing any data… good news is that I still have those old photos some of which are great and we have now the necessary quality in post processing to make them even greater… more to come… ).
When Adobe Lightroom came out I jumped on its first version and I just loved it. It was possible to do much processing, in particular non-destructive editing, and image tagging, etc. A great software. 6 years ago, as photography was becoming more important to me I moved away from Wintel and bought a Mac Pro. The great thing was that Adobe Lightroom was also available on Mac so I could seamlessly transfer the family library to Lightroom 2.0. Which was fitting our need quite well. At the time, the big competitor Apple Aperture was tempting for its better interface (from my point of view) and its editing capabilities, but performance wise Aperture 1 and 2 were just not cutting it! Then early 2010, Aperture 3.0 came out and I fell in love with it: great functionalities with great performance. Amongst other things I liked the idea that Aperture could manage itself the storage of my photos and that I would only have to deal with the logical organisation of the photos. The pain was that I lost all the tags and editions I made with Lightroom, but I felt that was a low price to pay (at the same time Lightroom 3.0 was released but I was no longer convinced). Looking back I wouldn’t change anything.
So I’m now using Aperture 3.4 which has evolved quite a bit since version 3.0 especially from an interface point of view. The only complaint I would have with it is that it lacks some of the modern functionalities such as geometric corrections or layers, but more on this later. It is a great way to catalog photos, to give them rating or tags, to compare them and selecting the best version, best of all: it makes it much easier to find photos in huge collection.
Should everyone use Aperture? In my book, yes if you’re not already using Lightroom and happy with it, you can just give it a try and decide if you prefer it or not (It is pretty much like Canon or Nikon: both are great brand, they differ mostly in their “user interface” ). Aperture is however only available on Mac…
I’m not going to dwell too much on hardware (again Wintel or Apple is a matter of personal taste) but I really like my setup: I’m using a Mac Pro as main machine (can’t wait to try the new Mac Pro announced a few days ago) and a 15” MacBook Pro with retina display which has amazing power. The MacBook gives me great mobility and has power to run apps such as Aperture, Photoshop with capacity to spare, and is secure (no moving part as storage is on SDD rather than a spinning hard drive).
Other variants can of course be quite adequate, one of today’s best in my view is owning a new 15” MacBook retina: using an Apple thunderbolt display at home and having external storage on thunderbolt (typically a disk array). This gives both mobility with a light laptop with a beautiful screen and a powerful desktop with the comfort of fast and large capacity, big screen and external keyboard. For those who’d like a lighter option for mobility, I could recommend the 13” MacBook Air which has the power to run photo software as well and is super light, however I’m not sure I would use it as main computer but rather a 27” iMac or a Mac Pro.
Based on my experience, the things I am looking for when it comes to Digital Asset Management are:
- Organisational capabilities (logical catalog, image rating, tagging, geolocation, etc)
- Library wide management (e.g. splitting or merging libraries)
- Easily transfer new data from a library to another (e.g. a main library and a mobile library)
- Responsive image viewing (including zooming) and image comparison
- Non-destructive photo editing
- Secure storage
The first 5 points are well covered with Aperture and Lightroom whilst the last two are independent from the software you’ll be using. So let’s talk about some of these points and why they are important to me.
Whether you want to manage your storage yourself or let your asset manager software do it for you, I recommend that you keep a file structure based on date and time as it is quite logical: you will add new photos with time, so naturally you’ll create new folders for new shooting sessions. I name my top folder with the year numeral “2012”, “2013”, and the subfolders with the month numeral and name, “01 – January”, “02 – February”, etc (I include the numeral so that folder list remain in the right order, otherwise “April” would come first and “September” last in an alphanumerically sorted list… ), within the month folder each photo shoot gets its own folder with a name format with the full date and some comment such as “20130628 Sunbird at Botanic Garden” for 28th June 2013. Again here I’m using a year/month/day format rather than a day/month/year or month/day/year for sorting reason to ensure that dates are properly sorted. I could have gone with just the numeral of the day but I felt the full date to be easier, also in some system if a file or folder named “11 Chinese Garden” will come before “2 Sungei Buloh” because computers are dumb and they compare only characters, so “2” goes after “1” and not “2” goes before “11” as human would think…
I leave Aperture to deal with all other meta information such as flags, ratings and tags. Which can be later used for searches and prioritisation, also when merging my mobile library after a trip all meta information are transferred along the photos in the master library.
Recently I have come with an alternative organisation for Photoanimalium.com, photos that I select for the site from the main library are exported and treated in a dedicated Photoanimalium.com library. This allows me to keep the original, a master photo after treatment and the web version. Thus I centralise all versions in one place which is handy when someone order a print. Also I keep a large “inbox” serving of pipeline of photo selected for publishing and then a “Production” section containing all published photo (original, master and web version). It is quite handy because I can keep this smaller library in a cloud based service and thus can work on the website whether I am at home or on the move.
Whilst on the move, I keep a mobile library which in my case is fully managed by Aperture and I let Aperture import photos from the memory card. It allows me to compare, select photos and start the post-processing while being mobile. As you might accumulate several thousands pictures a day in a photo trip, you better be able to at least make a first selection before coming home. Every time I have been lazy, my photos have stayed months or years in storage without being treated… So you might want to do the same. Once back, I merge the mobile library to the main library. I used to transfer via a firewire cable from the laptop to the desktop, now I simply merge via WiFi by accessing wirelessly the laptop hard drive from the desktop. With a good network it is not much more slower than a cable and much more convenient.
How many know that if you open a JPEG file and re-save it without making any changes, the new image will be degraded even if using the highest quality setting?
If you don’t, you should try to take a JPEG photo, open it with your favourite image editor, re-save it (using a different name) and keep re-opening the new file and re-saving it a few times, then compare the original file and the new file.
The quality degradation is due to the fact that even at its highest quality setting, the JPEG format is based on a destructive compression. The compression that occurs attempt to eliminate from the image all details that are “not important” to save space. To determine whether a detail is important or not is simply whether two neighbouring pixels have a very close colour: if they do the algorithm is assigning the same colour to both pixels. The quality setting that is set in the algorithm simply establish whether pixels must have be of very close or only close colour to be considered with the same colour. This is great for web content but less so for photography…
Most DSLR camera, and some high end compact, can produce RAW images. RAW images are images which are native from the camera sensor: basically what the camera really “sees” when the photography is created. For simplicity, this is the equivalent of the negative of the old days of film camera and, same as negative, RAW files have to be processed to produce a usable image. Aperture and Lightroom include a processor to convert RAW files to an actual image, display it on the screen, edit it and print it.
There are a number of advantages of keeping RAW files over JPEG, amongst them having the cleanest possible file to start with and having access to a wider dynamic range (typically 12-14 bits with modern camera vs. 8 bits with JPEG). In the past computer power made conversion from RAW a slow process but since then computing power has dramatically increased and is mostly not an issue anymore, it remains that handling RAW files properly requires more time and knowledge than having JPEG files directly out of the camera. Some of the complexity of RAW treatment can be avoided as most modern DSLR offer a number of creative features such as realistic film rendering, customised rendering for specific scenes such as landscape or portrait, or even more creative filters. These effects, and more, can be done in post-processing from RAW files but it requires work so here is the trade-off.
As a true data hoarder, I keep my photo in RAW format. One of the side benefits I discovered is the fact that as software technology keep improving, I can re-process old photos with new tools and get results I couldn’t have dreamed for when I created the exposure in the first place. Also as nature photographer, I often heavily crop my photos and I want to make sure that I keep the maximum quality of every few remaining pixels, especially to keep a reasonable chance to have acceptable larger prints.
Note that I have met with a number of pro photographer who shoot in JPEG and even not in full size. In particular photo journalists will often shoot in Medium or even Small JPEG files so that they can send the files quickly to newspaper editors.
Non-destructive photo editing
An important benefit that true Digital Asset Manager software have brought is the ability to work non-destructively on photos. Non-destructive editing is a concept where changes you may apply on the photos are kept separately from the actual original image file. As such you always can come back to the original version or make different version of the original image without ever modifying it, thus you can return later to your image and make yet another version. It is useful for photograph of all levels, for beginners it lets them try and experiment post processing techniques on their picture without risking to lose their image, for beginners and intermediate it allows them to return to their initial treatment when they learn new methods for better results. Also as I mentioned just before, all can benefits from keeping untouched their original images to re-process them when new technologies become available.
Of course there is another way to do this: open the original image, treat it and save it in a different file. However this tends to increase dramatically the storage space necessary to keep the original and the multiple versions. Especially when working with TIFF or PSD files which are generally several times bigger than the original RAW files. Software such as Aperture or Lightroom will keep a very low overhead when storing the changes which they will re-apply to the image each time it is displayed or printed, with modern computer this is quite fluid and does not require too much time to refresh. Thus it allows to keep the best of both worlds and simplify quite dramatically the post-processing workflow.
iPhoto and Photoshop Elements
There are consumer entry level software available, iPhoto available with every new Macs and Photoshop Elements given with most new camera. I’m not using either software but they both have functions equivalent to their big brothers Aperture and Lightroom. They are easy and a good start to manage your digital assets and for some limited non-destructive editing capabilities. One good thing about iPhoto is that it is now compatible forward and backward with Aperture libraries (the same library can be opened by either software without having to import/export the photos in their respective managed library). This is great as Apple offer a photo book printing service of high quality directly from either iPhoto or Aperture, however photo books styles available in iPhoto are different from those available in Aperture and it is great to have access to both styles.
Secure Storage and Backup
One last thing about asset management is about how to ensure to safeguard your digital asset management. Over the years I’ve gone through various aspects of storage and backup. Accumulation increases quickly, I’ve filled about 2Tb of hard drive in 7 years, and the next 2Tb in less than 4 years…
At this level, backup is simply no longer practical. Especially with the cost of hard drive as low as it is today. My preferred solution for storage is a disk array: a box containing 4 or more hard drives, once plugged to the computer it sees only 1 drive. This technology was available in the past only for data centres and they are since a few years available for consumers as well at reasonable price. Typically I use array of 4 drives formatted in RAID 5 which allows one drive to fail whilst the data is not lost until another drive fails. When one of the unit fails, it can be replaced and the disk array rebuild the missing disk with the information available on the remaining 3 disks and thus restoring the fault tolerance capability (of course if more than one drive fails, all the data is lost, often with high availability servers, another disk could fail during the rebuild phase of the first unit, destroying the whole array, thus high availability servers have moved toward RAID 6 format where up to 2 disk can fail while the data remain secure). Also, as I am sharing photo libraries with my wife, we have two physical copies of the library thus if one full unit would fail we still have a full copy available (actually I even have a 3rd version of all the data as my MacPro uses Time Machine for backups which include my photo libraries).
This is a pretty strong setup already but still based on physical devices which can be destroyed or stolen. There are people who either buy a fireproof safe or use a bank safe to physically protect a full copy of their library, however they are updated as regularly as the owner is prepared to make updates. I am rather opting for on-line backups which are keeping my about 4Tb photo library backed up and constantly updated, it initially takes several weeks or even months to set up but once it is done, only updates are uploaded which does not require a huge internet band width. To err on the safe side I have even 2 online backups in case one of the backup company would fail I can still rely on the second one.
You can also consider having the same approach with other digital asset you might have: such as multimedia library (our iTunes is 6Tb in size with all music, movie, TV series, etc.) which you might want to protect if you no longer keep physical CDs or DVDs and which I kinda hope to be able to pass on to the next generation when the time comes… At the moment, I am considering consolidating the family storage on a single NAS (Network Attached Storage) unit which is nothing else than a big disk array connected to the network rather than directly to a single computer. There are few options of big arrays of 8 to 16 drives configurable in RAID 6 for massive storage which can be shared in a small office or for a house hold (with my daughter doing more and more photos we are now 3 big data producers).
This is a long post, on a topic which is not sexy for most photographers but nonetheless an important one, which I would summarise like this: if you are serious about photography you should consider
- shooting RAW rather than JPEG
- moving to a robust Digital Asset Manager with either Adobe Lightroom for Mac and Wintel or Aperture for Mac
- consider secure and expandable storage considering contingency scenarios: disk arrays with either offsite physical duplicate and/or on-line backups
- having a good mobile configuration either with a very small laptop with sufficient computing power or a compact high power laptop with keyboard/screen/storage in your home-office
I’ll be also happy to see your comments or answer your questions if you have any.