editing a raw file

Wow, what a long time! I started this post back in March 2015! 😳😲 Unbelievable!
Recently, I got again a question on my workflow, so I decided to finally finish this post. Hint: This post contains some links to software manufacturers. No one paid me anything for getting a link or had any influence on my opinion.

Besides creating jpg files directly in-camera many digital cameras are able to create raw files instead of the JPG or in addition. Raw file means saving the bare information captured by the sensor and not processing it in-camera. This has a couple of advantages. A few years ago, I already published a post on these advantages. Many years ago, I decided to capture only raw files and process them myself afterward to have more control over the process and the final look. No, don’t get me wrong, I don’t use any filters. In only do, what was necessary back in film days.

In the past, I already published some posts covering how I store my images and a bit about, why I’m doing the extra work for raw development. Some of the benefits of doing it this way are in this post. I really encourage you to click on the links and read a bit about the background. Although most recent digital cameras are really great in creating good-looking JPGs, you still have a lot more in raw.

I’m using dedicated raw-processing software and it’s not Lightroom, because this software needs so much unnecessary work which stands in the way where I’d have to work around. The workflow is complicated and not straightforward. And, you’re locked into the software when you want to keep access to your own work.

The DAM (Digital Asset Management) would be nice for the final images, but it’s absolutely the wrong way to import all undeveloped images only for deleting a huge portion right after the import. A database makes additional problems when being used that way and slows down the computer over time. You can google for these problems and find gazillions of people suffering from them. An often recommended solution of having a separate catalog for each job on the other hand leads the whole idea ad absurdum. (btw. the same is true for the recent versions of Luminar, and that’s the reason why I left their affiliate program). Not being able to save all your edits outside of a catalog also hinders you to get your image edited by someone else and get the edits back for learning from the outcome.

These are my steps after coming back from a job or a trip:

1. copy all images in a dedicated folder on my internal disk named with the date and a tiny description of the contents (i.e. 20210507 – garden birds)
2. create a first backup of the whole bunch of images, which will hopefully never be used
3. import the GPS data into the image files (only for trips)
4. normalize the file names when having used more than one camera body to get them in the right order again. My naming scheme is YYYYMMDD_hhmmss-XXX_xxx.NEF. I guess the first and second part of date and time is easily understandable. I take this information from the metadata of each file: the time of releasing the shutter. The XXX stands for a 3 digit code of the used camera body followed by 4 numbers. These third and fourth parts are given to the file by the camera at creation time. The metadata are pieces of information stored in the images by the camera at the time when the image is captured.
5. create a second backup on a second external disk. This one is my backup in case of an emergency.
6. open up a digital light table to check the images and select the ones to get processed.
7. create a sub-folder called “edit” and copy all selected images in this folder, direct the raw processor to the edit folder and process the images
8. the final steps are already in the post, I mentioned above.

In this post, I want to describe the actions I do in my raw processor. You know, I started photography more than 40 years ago and run my own lab in those days.  And the options I had in those days are still the steps I do in digital photography:

1. remove some blemishes and dust-spots
2. brighten the shadows and darken the lights to get a better balance in the image if necessary
3. balance the horizon if necessary
4. correct the with balance if necessary
5. boost the vibrance or increase the saturation if necessary
6. crop, if necessary
7. post-sharpening
8. export to jpg

You see, my workflow is quite easy and straightforward. It costs me a maximum of approximately 2 minutes per image. Some of the actions can be bundled and applied to a couple of images at the same time to speed up the work. But, in general, I’m not a big fan of such bundlings, besides the export to jpg.

You might ask, what kind of software you can use. There are a couple of commercial products around. All I know, offer trial versions for a couple of days/weeks: DxO Photolab, Capture One Pro (free versions for some Fuji and Sony cameras available), OnOne Photo Raw, NX Studio (free, Nikon only – afaik Canon has something similar), and some open-source products, which are also free of charge: Rawtherapee, Lightzone, ufraw, digikam, Darktable. Affinity Photo is more a replacement for Photoshop or Gimp, although it also has a good raw development module. Photoshop brings kind of a lite-version of Lightroom for raw development called Adobe Camera Raw (ACR) and Gimp incorporates either ufraw or Rawtherapee, depending on the operating system you’re using. There might be some more products in the market, which I didn’t mention here because I don’t know them. So, this list isn’t complete. Sorry!

Although the open-source products usually have a background in the Linux community, they are also available for Windows and macOS. I worked with all of them a bit and would recommend either Lightzone or Rawtherapee. ufraw is a bit like Adobe Camera Raw for Photoshop. Darktable has the same mechanisms as Lightroom. So, it’s unusable for me. I don’t want to import the raw files into a database and stick with this single database because all of my edits are in that database. A corrupt database could make me lose all ever-done edits. DigiKam is an all-in-one solution: fat but mighty. I like the organizing module very much: DAM = Digital Assets Management. For some time, I’m using Excire Foto for DAM. You see, the linked post is quite old. I have to write a new one. This product is amazing.

For modern lenses, it is extremely important to have software being able to adjust and correct lens failures: barrel or cushion distortion and achromatic abbreviations. Back in the film days, the good lenses were designed for not having them. Nowadays it’s easier (and cheaper for the manufacturer) to create a piece of software to correct the failures. All the software I mentioned above is IMHO able to work that way. On the other hand, I have a lot of old lenses, which don’t need such corrections.

In case, you want to start developing your photos, I’d recommend either DigiKam, because it is an all-in-one solution, or Lightzone / Rawthereapee. In case, you need some image manipulation tools, try gimp. You can find tutorials on the relevant homepage I mentioned above or on YouTube. Although there are rumors of Gimp would be complicated, that’s not completely true. Also, Photoshop is very complicated, but there are more talkative called-by-themselves experts telling it otherwise because they are kind of experienced because of extensive usage. Each software is complicated on the first try. But, open-source software usually has a very active community willing to help when you’re investing at least a bit of time reading or watching (YouTube) tutorials. The other option is, to give one of the commercial software a try and download the trial version. Also, commercial software needs you to learn how to handle them. So, giving open-source a try first, won’t cost you any money. When talking about commercial software, I like DxO Photolab the most followed by Capture One Pro. When it comes to image manipulation software, I’m using Gimp and Affinity Photo. In the past, I used Photoshop CS6, but it’s not necessary anymore. First, I need such software only for approximately 10-20 images a year, on the other hand, Gimp has everything, I need. A few weeks ago, I published an article on one of my use cases for image manipulation software on NikonRumors and here in my blog.

I tried all the products I mentioned above last year when I had to investigate a replacement for my raw development software. Although the last update was in 2012 it run well and gave me the flexibility, quality, and tools I needed. Unfortunately, the developer decided to retire the software. For some years I was unable to re-install it if there were a reason to do so, but the recent updates of the operating software of my computer made the tool even completely unusable. So, I needed to find a replacement and I did. Recommending software to someone else is not easy because everyone has her own workflow, her own requirements, and her own wishes. So, you have to try on your own and find the software fulfilling all of your requirements to the best.

Below, you can see two screenshots from while developing a raw image in LightZone. The final image is on the top of this post.

browsing a folder with raw files with the folder structure on the left and the metadata of the selected image to the right

 

a couple of presets on the left, first developing step on the right and seeing the distribution of light in the image (Zone Model of Anselm Adams) in the upper right corner

 

Read the steps on the right from bottom to top. The last step is missing: sharpening

In this image, it was necessary to darken the highlights to recover the fine structures in the petals. In-camera development would have left only completely white spaces. From the raw file, I was able to recover the fine lines in the petals as well as the stamens. The erected twig was removed afterward with image manipulation software (Gimp) as well as the cropping.

Take care and happy snapping 😊

Lens-Artists Photo Challange 134: ” From Forgettable to Favorite”

…. or the benefits of raw!

Fortunately, the subject for this week’s Lens-Artists Photo Challenge was published ahead. So, I had enough time to prepare this post. Each photographer struggles sometimes over the limitations of the photographic gear and gets images (far) away from the expected result. Fortunately, nowadays in digital photography, you have the option to increase the image in the digital darkroom. As our host this week, Tina asks to show such images as examples.

developed from RAW

You know, I’m taking my photographs in raw format instead of getting the JPGs processed automatically straight from the camera. Although this takes additional time and work, I’m usually getting better images. The link above directs you to a post I wrote a couple of years ago. Despite the abilities of the sensors increased over time, I still don’t want to throw away quality.

Usually, I don’t do much post-processing. I only sharpen my images and balance the exposure by subtly brightening the darks and shading the lights a bit, if necessary. So, the additional effort is very small and I can use batch-processing quite often.

But, every now and then I have images demanding a bit more work, just like the image I’m showing you today.

This image is taken on Helgoland in January 2016. My idea was to capture the light beam of the lighthouse. So I went out during the blue hour because I wanted to have a slight dark-blue sky instead of a black sky. The slight snow and rain that night didn’t disturb much. When I arrived at the planned location, I noticed a family walking towards the lighthouse. So, a quite short exposure was necessary to get a sharp family and get nice light beams. The exposure was set to fit the lights of the window: full-frame, ISO 2000, f4, 1/25, handheld (no time for setting up the tripod because of the family).

out-of-camera JPG

As expected from that scene, the captured image was very dark. Thanks to the raw format, this wasn’t a loss! These steps were taken to get the final image:

1. increased exposure compensation in my raw developing software by +2
2. decreased the lights a little bit
3. increased the darks a little bit further
4. un-sharp mask
5. export to JPG

In the other image, I integrated the OoC for comparison. Click on the image to enlarge it. It’s also taken on Helgoland in January 2016. It shows the remains of an old pier.

This long exposure is also taken in raw and the exposure is aligned to the bright areas to avoid burnt-out areas. Besides a tripod, I used a gray-filter and a graduated gray filter.

full-frame, 24mm, ISO 50, f16, 10 seconds

Developing steps:

1. remove dust spots in the sky
2. balancing the horizon
3. lighten the darks
4. increased the warm tones in the clouds from the setting sun
5. slightly cropped
6. un-sharp mask
7. export to JPG

These dust spots are almost always in your images when using a camera with interchangeable lenses because they are in the air and when changing the lens they can come into your camera. The same is true when you using nun-sealed lenses. When dust is inside your camera, it’s easy for the particles to settle on the sensor. You can recognize them as dark mostly round spots in the image. Most easily you can see them in a bright sky or on homogeneous areas in your image. The other possible source for the spots the lens itself. Either you might have spots on the back lens of your interchangeable lens or on your front lens. And, although the front lens is quite easy to keep clean, spots will appear. When now taking your final image in JPG format to do the corrections, you’re losing quality because the image will always be compressed with a lossy algorithm when storing it. So, it’s much better to do all the necessary work on top of a raw file and export the finished image. I’m recommending reading the post, I linked further up in this article.

For the next 2 images, I also embedded the original image into the final one.

1. lighten the darks in the face to reveal the eyes a bit
2. slightly cropped
3. un-sharp mask
4. coverted to monochrome by using software that emulates monochrome film instead of desaturating the colors
5. adding a subtle dark vignette
6. export to JPG

This is a wildlife image. Despite using a 400mm lens, I was too far away from the seals for my planned composition. You know, gray seals are raptors and you have to stay at least 30 meters away from them. They are much faster as you might think. So, you better respect the recommended distance.

1. lighten the darks
2. increased the warm tones a little bit
3. slightly cropped
4. un-sharp mask
5. export to JPG

Take care!

How to store photos on your disks

As a photographer you collect many, many photos over the years. Back, in film days, there were prints, slides, mounted slides and developed films to store carefully. They had to be stored dark, dry and preferably dust-free.

Today, when we have digital photos, the problems changed a bit. We don’t need that large cabinets anymore, because a digital photo need much less space. But, because it’s so cheap to make a photo, we have much more of them.

In film days, we have had folders for the developed films and hand-written index sheets or card boxes for the slide trays. This was done carefully, to have chance to find a certain photo without much effort.

Today I want to introduce you to my digital filing system.

First of all, I have my SD card containing the photos. On trips I usually transfer them every evening from my card to my computer and check them. The check is for making sure there is no corrupt file. I also delete bad photos: unintentionally shutter releases, obvious mismatched focus or unrepairable over- or under-exposed photos. Then I make a second copy to an external USB disk and clear the SD cards for the next day.

I create a folder for the trip like: “20140629 – Iceland”
Below this folder I create separate folders for each part of the trip. (note: photowalks and roundtables usually don’t have this additional level)

20140629 – arrival
20140630 – 01 – Gullfoss
20140630 – 02 – Geysir
20140630 – 03 – Brúaráfoss
20140701 – 01 – Pseudo crate field
20140701 – 02 – Viking house
20140701 – 03 – Gjáin Foss

This scheme makes it much easier to find a certain photo later. (keywording is not subject to this post)

As I wrote in my last ‘tech‘ post, I’m a raw shooter. Thus, I have to do my developments, when I’m back at home (on a trip I only develop very few, but really fantastic, photos).

Therefore I create a sub-folder called “edit” inside the folder of certain trip part. Each developed photo is stored inside this edit folder. When I’m ready with all development of a folder, I start a batch process for exporting all developments to JPG-Files. Subsequently I start another batch job, that creates some further sub-folders for the original (printable) JPG and a watermarked file in web resolution.

When I’m finished with a project, I move all JPGs to my file server, all developed files go to my NAS and all untouched raws to an external USB disk. On both storage devices, I have folders named by the year to hold the folders mentioned above. My file server and my NAS are backed up every day (night) to separate, independent USB disks. The external USB disk with all the raws isn’t backed up separately. Now, the pristine second copy isn’t needed anymore and thus cleared for a new use.

2014
 ⤷ 2014 – Iceland
    ⤷ 20140629 - arrival
       ⤷ big
       ⤷ org
       ⤷ web
    ⤷ 20140630 - 01 - Gullfoss
       ⤷ big
       ⤷ org
       ⤷ web
    ⤷ 20140630 - 02 – Geysir
       ⤷ big
       ⤷ org
       ⤷ web
    ⤷ 20140630 - 03 – Brúaráfoss
    ⤷ 20140701 - 01 - Pseudo crate field
    ⤷ 20140701 - 02 - Viking house
    ⤷ 20140701 - 03 - Gjáin Foss

Both, file server and NAS, are built from regular computer parts and set up with Linux. Thus, I’m able to adapt them easily, in case of new requirements.

My desktop computer is also backed up every night. All those backups are incremental backups. This is copying only the modified files, instead of the whole disk. Thus a backup is finished usually in less than half an hour, depending on the amount of modified files.

Take care for you valuable files. Have a backup!

Happy snapping 🙂

 

Why should I do the extra work for using raw instead of jpg out-of-cam?

Most, if not all, digital cameras come preconfigured by the manufacturer to save the photos in jpg format. This format is standardized by the Joint Photographic Experts Group, usually written as an acronym JPEG. You can recognize files following this standard by their extension .JPG or .JPEG.

The jpg format has a lossy compression algorithm to shrink the file size. The compression level is usually leveled by fine, good, moderate, web, small size or in percent levels. Regardless of the notation, the higher the quality, the bigger the files. Or to change the angle of view: the higher the compression, the worse the quality.

JPG has also another design problem. It only uses 8 bit for coding the colors. Each color is assembled by leveling the three color channels red, green and blue (RGB). I.e. a white point is saved a 100% red, 100% green and 100% blue. Thus you need 3 byte (1 for each color channel) for storing 1 pixel. Assuming, your camera sensor has a resolution of 3000×2000 pixel, you can easily calculate the size for a photo: 3000 x 2000 x 3 = 18000000 = 18000 kilo-byte = 18 mega-byte.
The compression algorithm tries to find neighboring pixel of similar (not same!!) levels in all of their 3 color channels and sets them as equal to save space. Especially in nice color gradients like skies at sunset you will see serious problems: the colors don’t blend smooth, but change in bigger steps instead. Very ugly. Have a look in the attached screenshot below.

Higher-quality and semi-professional cameras support more than 8 bit in their sensors. They use 12, 14 or even 16 bit for each of the three color channels. This results in 4096, 16383 or 65535 color nuances per channel instead of 256 when using 8 bit. Do you see the difference?
So, your camera is recording a scene with a color depth of 14 bit (thats 16383 color nuances), just like mine and JPG only uses 8 of them (for 256 nuances). How can this work? It works by compressing similar tones to only one. The result is, you’re loosing much of your details in the mid-tones, in the darks and in the highlights. While the highlights will tend to turn to white, the darks will tend to turn black.

One of the loopholes might be, using TIFF instead of JPG. This format preserves all information, because it uses 16 bit for each color channel and does not use any compression. The downside of this quality is the size. Using the same values as above a TIFF file would eat up 36 mega-byte. Or, by using values from my camera: while a JPG usually uses between 10 an 12 MB, a TIFF file has 155 MB.

Many cameras have another option: raw files! These files are called raw files, because they are stored without any touching inside the camera. They preserve all bits recorded by the sensor and use a lossless compression. But, these formats are camera dependant. Each manufacturer uses his own format and to make this even worse, it is different for each camera model. When using this format, each of my photos needs about 20-30 MB of storage space. And these raw files aren’t ready for use. They have to be developed in post-production, just like the films in analog times.

So, why should you use this option? There are already TIFF and JPG in place? Why shout I invest much more time for post-production? And, why should I invest more money in software, when I can get ready to use photos right out of my camera? My print shop ony takes JPG anyway.

That depends! Think of the color depth I mentioned earlier. You could get better quality prints by spreading the tones, lighten the darks and darken the highlights in post-production. So, you can get a more natural ambience without huge black and / or white areas without details. Another option is, you can change a few basic settings like the white-balance or slightly shifting the aperture lossless. And higher quality print shops also accept TIFF-file, and they know, why 🙂 Look at the web site of your print shop for accepted file formats.

So, you only can win by using raw-files. There is software available for Linux, Mac OS X and Windows for doing the necessary work called “the digital darkroom”. This name derived from the darkroom necessary in film days for developing the film and to process the film negatives to prints.

There’s a huge variety in products available. You don’t need to use Adobe’s Lightroom. There’s even software available as open-source for free and maybe even from your camera manufacturer using optimized setting for your camera and the lenses.

I attached a 100% crop from one of my photos taken last month in Belgium to demonstrate the (bad) effect. I shrinked the quality level to 40% before clipping out this part of the upper left corner of the sky. Keep in mind, depending on the dye distribution in a certain photo, the effect becomes visual earlier or later. Sometimes this effect is already there, when using a quality level of 80% on other photos it appears not until 50%. For my photos, I don’t use any compression.

Click on the photo, to see the effect better. Below I attached the original photo. Feel free, to drop me a note, a question or a suggestion in the box below.

Take care!