How to store photos on your disks

As a photographer you collect many, many photos over the years. Back, in film days, there were prints, slides, mounted slides and developed films to store carefully. They had to be stored dark, dry and preferably dust-free.

Today, when we have digital photos, the problems changed a bit. We don’t need that large cabinets anymore, because a digital photo need much less space. But, because it’s so cheap to make a photo, we have much more of them.

In film days, we have had folders for the developed films and hand-written index sheets or card boxes for the slide trays. This was done carefully, to have chance to find a certain photo without much effort.

Today I want to introduce you to my digital filing system.

First of all, I have my SD card containing the photos. On trips I usually transfer them every evening from my card to my computer and check them. The check is for making sure there is no corrupt file. I also delete bad photos: unintentionally shutter releases, obvious mismatched focus or unrepairable over- or under-exposed photos. Then I make a second copy to an external USB disk and clear the SD cards for the next day.

I create a folder for the trip like: “20140629 – Iceland”
Below this folder I create separate folders for each part of the trip. (note: photowalks and roundtables usually don’t have this additional level)

20140629 – arrival
20140630 – 01 – Gullfoss
20140630 – 02 – Geysir
20140630 – 03 – Brúaráfoss
20140701 – 01 – Pseudo crate field
20140701 – 02 – Viking house
20140701 – 03 – Gjáin Foss

This scheme makes it much easier to find a certain photo later. (keywording is not subject to this post)

As I wrote in my last ‘tech‘ post, I’m a raw shooter. Thus, I have to do my developments, when I’m back at home (on a trip I only develop very few, but really fantastic, photos).

Therefore I create a sub-folder called “edit” inside the folder of certain trip part. Each developed photo is stored inside this edit folder. When I’m ready with all development of a folder, I start a batch process for exporting all developments to JPG-Files. Subsequently I start another batch job, that creates some further sub-folders for the original (printable) JPG and a watermarked file in web resolution.

When I’m finished with a project, I move all JPGs to my file server, all developed files go to my NAS and all untouched raws to an external USB disk. On both storage devices, I have folders named by the year to hold the folders mentioned above. My file server and my NAS are backed up every day (night) to separate, independent USB disks. The external USB disk with all the raws isn’t backed up separately. Now, the pristine second copy isn’t needed anymore and thus cleared for a new use.

2014
 ⤷ 2014 – Iceland
    ⤷ 20140629 - arrival
       ⤷ big
       ⤷ org
       ⤷ web
    ⤷ 20140630 - 01 - Gullfoss
       ⤷ big
       ⤷ org
       ⤷ web
    ⤷ 20140630 - 02 – Geysir
       ⤷ big
       ⤷ org
       ⤷ web
    ⤷ 20140630 - 03 – Brúaráfoss
    ⤷ 20140701 - 01 - Pseudo crate field
    ⤷ 20140701 - 02 - Viking house
    ⤷ 20140701 - 03 - Gjáin Foss

Both, file server and NAS, are built from regular computer parts and set up with Linux. Thus, I’m able to adapt them easily, in case of new requirements.

My desktop computer is also backed up every night. All those backups are incremental backups. This is copying only the modified files, instead of the whole disk. Thus a backup is finished usually in less than half an hour, depending on the amount of modified files.

Take care for you valuable files. Have a backup!

Happy snapping 🙂

 

Why should I do the extra work for using raw instead of jpg out-of-cam?

Most, if not all, digital cameras come preconfigured by the manufacturer to save the photos in jpg format. This format is standardized by the Joint Photographic Experts Group, usually written as an acronym JPEG. You can recognize files following this standard by their extension .JPG or .JPEG.

The jpg format has a lossy compression algorithm to shrink the file size. The compression level is usually leveled by fine, good, moderate, web, small size or in percent levels. Regardless of the notation, the higher the quality, the bigger the files. Or to change the angle of view: the higher the compression, the worse the quality.

JPG has also another design problem. It only uses 8 bit for coding the colors. Each color is assembled by leveling the three color channels red, green and blue (RGB). I.e. a white point is saved a 100% red, 100% green and 100% blue. Thus you need 3 byte (1 for each color channel) for storing 1 pixel. Assuming, your camera sensor has a resolution of 3000×2000 pixel, you can easily calculate the size for a photo: 3000 x 2000 x 3 = 18000000 = 18000 kilo-byte = 18 mega-byte.
The compression algorithm tries to find neighboring pixel of similar (not same!!) levels in all of their 3 color channels and sets them as equal to save space. Especially in nice color gradients like skies at sunset you will see serious problems: the colors don’t blend smooth, but change in bigger steps instead. Very ugly. Have a look in the attached screenshot below.

Higher-quality and semi-professional cameras support more than 8 bit in their sensors. They use 12, 14 or even 16 bit for each of the three color channels. This results in 4096, 16383 or 65535 color nuances per channel instead of 256 when using 8 bit. Do you see the difference?
So, your camera is recording a scene with a color depth of 14 bit (thats 16383 color nuances), just like mine and JPG only uses 8 of them (for 256 nuances). How can this work? It works by compressing similar tones to only one. The result is, you’re loosing much of your details in the mid-tones, in the darks and in the highlights. While the highlights will tend to turn to white, the darks will tend to turn black.

One of the loopholes might be, using TIFF instead of JPG. This format preserves all information, because it uses 16 bit for each color channel and does not use any compression. The downside of this quality is the size. Using the same values as above a TIFF file would eat up 36 mega-byte. Or, by using values from my camera: while a JPG usually uses between 10 an 12 MB, a TIFF file has 155 MB.

Many cameras have another option: raw files! These files are called raw files, because they are stored without any touching inside the camera. They preserve all bits recorded by the sensor and use a lossless compression. But, these formats are camera dependant. Each manufacturer uses his own format and to make this even worse, it is different for each camera model. When using this format, each of my photos needs about 20-30 MB of storage space. And these raw files aren’t ready for use. They have to be developed in post-production, just like the films in analog times.

So, why should you use this option? There are already TIFF and JPG in place? Why shout I invest much more time for post-production? And, why should I invest more money in software, when I can get ready to use photos right out of my camera? My print shop ony takes JPG anyway.

That depends! Think of the color depth I mentioned earlier. You could get better quality prints by spreading the tones, lighten the darks and darken the highlights in post-production. So, you can get a more natural ambience without huge black and / or white areas without details. Another option is, you can change a few basic settings like the white-balance or slightly shifting the aperture lossless. And higher quality print shops also accept TIFF-file, and they know, why 🙂 Look at the web site of your print shop for accepted file formats.

So, you only can win by using raw-files. There is software available for Linux, Mac OS X and Windows for doing the necessary work called “the digital darkroom”. This name derived from the darkroom necessary in film days for developing the film and to process the film negatives to prints.

There’s a huge variety in products available. You don’t need to use Adobe’s Lightroom. There’s even software available as open-source for free and maybe even from your camera manufacturer using optimized setting for your camera and the lenses.

I attached a 100% crop from one of my photos taken last month in Belgium to demonstrate the (bad) effect. I shrinked the quality level to 40% before clipping out this part of the upper left corner of the sky. Keep in mind, depending on the dye distribution in a certain photo, the effect becomes visual earlier or later. Sometimes this effect is already there, when using a quality level of 80% on other photos it appears not until 50%. For my photos, I don’t use any compression.

Click on the photo, to see the effect better. Below I attached the original photo. Feel free, to drop me a note, a question or a suggestion in the box below.

Take care!