Digital Noise and Photography

The Problem
Noise is not only a problem for getting good analytical results in the laboratory, but is an issue for the photographer.  In my previous three blogs I presented a theoretical basis for dealing with analytical noise that applies to any measurement process including photography.  For the photographer, digital noise is most bothersome in the shadow details.  This is because there is less digital information in the shadows than the mid-tones and highlights.  Less light means fewer photons, fewer photons mean fewer pixels, and fewer pixels is less digital information.  Increasing the sensitivity to boost the signal for low light does not solve the problem, it just increases the noise.

[If you missed the earlier blogs that covered basic theory, don't panic.  You can still read them starting with What is Detection and What Does it Mean for Photography. ]

An Approach from the Lab
In earlier blogs I have noted that while an individual measurement near the detection limit has limited value because of the large associated error (i.e., noise is large relative to the measurement), in the context of a data set a collection of measurements has statistical utility even when all of the measurements are near or even below detection.  That is because the errors, which are random in nature, average out.  The same is true in photography.  Consider that each pixel in your image is the consequence of photons striking the sensor.  But sometimes the sensor will create more than one pixel for an in-coming photon and sometimes it will create no pixels for in-coming photons.  This is a random process, meaning that sometimes the effect will be positive and sometimes negative.  If you can average enough random events in a series of nearly identical events, the random noise will be averaged out and canceled.

An Example
Consider the following photograph titled It's Really Very Underexposed!  The only visible detail is in the sky, where the highlights are.  For this shot near sundown, I exposed it so that the highlights in the sky would not be blown and set the f/stop to maximize depth-of-field.  (Note a double-click on any of the images will open it at a larger resolution.)

It's Really Very Underexposed!  

The problem is that the shadow areas have dramatically less light than the sky and so are totally blocked up (i.e., very few photons).  We might conclude that there is nothing of value in the shadows, but to make sure I applied a curves adjustment to get Curves Adjusted to Open Shadows.

Curves Adjusted to Open Shadows

So now it should be obvious that there was light in the shadow areas, but not very much.  As a consequence, just like measurements in the lab, when the measurement is near the limit of detection, there is a lot of noise that makes accurate results difficult - but not impossible.  See the next image as a crop of the one above to convince yourself that there is a lot of noise in this version (mouse-click the image for a larger more viewable version). 

Crop of Curves Adjusted to Open Shadows

So how do you get rid of the noise?

Calculating Detection Limits - Some Math Required

See Detection What is it and What Does it Mean for Photography for an introduction and Detection in the Testing Laboratory for a discussion of the concepts of detection, quantification, false positives, and false negatives.




Detection Concepts Chart: The blue curve is a distribution of measurements on a sample with nothing in it (a very clean blank).  The red horizontal line represents the detection level.  Only 1% of the measurements of an uncontaminated blank will exceed the detection level.  The 1% level protects against false positives 99% of the time.  The orange line represents a level that should be detected all of the time if the detection level is truly achievable.  It protects against false negatives.  

MDL Defined by Code
The standard for the calculation of detection limits (MDLs) is the US EPA Code of Federal Regulations (40 CFR 136).    MDLs are calculated from the variance of the replicate analysis of either a ultra-clean sample (blank) or a quality control sample made from a blank by adding the substance(s) of interest at a level above but near detection. Replicates can be analyzed either in a single batch (usually with 7 replicates) or over time (with 20 or more replicates). If the former approach is used, the calculated detection limit ignores the contribution of errors of time-variance. While both approaches are allowed, there are differences in the results. Single-batch limits tend to be less than comparable time-variance determined limits. Some, but not all, laboratories, include one or more low level quality control samples in each day's batch for the purpose of controlling against false negatives and to generate a data base for the calculation of detection limits and/or confirmation of detection capability.

Problems with the Traditional Approach and a Robust Alternative
The analysis of seven replicates in a single batch generates a detection limit that is artificially low and not achievable on a routine basis.  The MDL as does not account for false negatives: an analytical result below detection has a 50% chance of being a false negative.  Performing a single batch detection limit study requires reassigning staff out of the routine production stream.  The aperiodic approach to confirming detection limit capability on a more-or-less annual basis sidelines detection limits from the routine batch based Quality Control.  By adding a routine QC sample to the analytical stream, it is possible to avoid these problems and develop detection limits that are robust, achievable on a routine basis, account for false negatives, and do not require to reassign staff off of production. 

False Negatives
The inability to detect an analyte at two to three times the stated detection limit indicates a false negative. Such a failure may be due to chemical interference, instrument malfunction, or analyst error. In addition to a determination that the LCS is routinely detectable with each preparatory batch, accuracy is calculated and reported as an indicator of agreement between the known (i.e., true) and measured concentrations on an on-going basis and is used to generate control charts for LCS results.

Detection and Quantification in the Testing Laboratory

For a previous discussion on the meaning of detection and quantification see Detection - What is it and What Does it Mean for Photography?.

Your Drinking Water
A drinking water laboratory tests water samples to determine what is in them. The ultimate objective is to ensure the safety and reliability of drinking water supplies. But every analytical measurement has error. To control error, laboratories must include check samples to distinguish the known from the unknown. Sometimes the quantity of a substance is too low for the method to distinguish from a sample with nothing in it (a blank). The lowest threshold at which a method can determine the presence of a substance versus its absence is called the detection limit. In other words, detection occurs when a test method confirms the presence of a substance (analyte).

Detection and Quantification
A method detection limit (aka MDL in United States and Lc in most other countries) represents the detection capability of a method including all analytical and preparatory steps. While meaningful in a statistical context, an analytical result at or slightly above detection is not quantified and by itself cannot generally be used to make decisions about the quantity of analyte present in the analyzed sample other than the fact that it is present. The IUPAC (International Union of Pure and Applied Chemistry) term for detection is the Critical Value (Lc).

Quantification is when the method establishes that the amount of the substance is sufficiently large that noise is a small fraction enabling scientific, legal, and process decisions. A quantified result is one that can be assigned a number with a known precision and accuracy while a detected result is one where it is only possible to state that a substance exists at some quantity above zero but the exact number cannot be known.

A quantified result is one with an acceptably low error so that a decision can be made about the quantity of analyte. Reporting Limit (RL), Minimum Level (ML), Detection Limit for Reporting (DLR), FNQS (False Negative Quality Control Sample), and Practical Quantitation Limit (PQL) are some of the terms used to describe the minimum concentration necessary to make a quantified decsion. The IUPAC term for quantification is the Limit of Quantification (Lq). Another term, Limit of Detection (LOD) is an IUPAC quantity between the Lc and Lq that minimizes false negatives.

Measurement Error and False Decisions
All analytical results include measurement error so that any result has a specified probability of being either falsely positive or falsely negative. An substance detected when it is not present is a false positive; a substance not detected when it is present is a false negative. The detection limit (MDL/Lc) is a statistic that is defined to limit the odds of a false positive to 1% while a quantification limit (FNQS, ML, DLR, Lq) is generally defined to limit the odds of a false negative to 5%. Thus a measurement that barely exceeds the MDL could be in error 1 time out of 100 and a measurement that is non-detect at the quantification level could be in error 5 times out of 100. In the former case, the the analyzed substance would be falsely concluded as being present and in the latter it would be falsely concluded as being absent.

Next
In my next blog, I'll show how to calculate detection limits in a way that avoids taking staff out of routine production.

What is Detection? What's it Mean for Photography?

Noise, Observations, Detection
Noise in our daily observations, in analytical measurements, and in photography obscures informational content.  That information can be in a conversation, a test result on a water sample, or detail in a photograph.  Whether we can extract information from a measurement or observation depends upon the sensitivity and selectivity of the measurement method otherwise known as detection and quantitation capabilities.  In this series on detection and quantitation, post 1 will introduce the concepts, post 2 will demonstrate the use of detection and quantitation in a laboratory setting, and post 3 will develop the mathematical concepts.  Posts 4-6 will provide examples as applied to photography.

Detection and Quantitation in an Everyday Experience
Let me start with an example based on this scene from the bazaar in Cairo.  As the two women walk through the gate, the woman to the right turns and asks her friend about the wares of the merchant ahead of them.  The other woman knows her friend is talking to her but cannot hear her above the crowd so asks her to speak up.  She can now hear her friend but cannot understand her words. This is detection.  An observation or measurement distinguishable from the noise is a detection.  She asks her to speak up some more and finally can hear her words and understands her question.  This is quantification; an observation or measurement sufficiently different from the noise to allow decisions to be made.

In the next two blogs I will discuss more of the theoretical concepts before presenting some applications to photography.  I hope to show how the concepts of detection and quantitation relate to photography and are the theoretical basis for dealing with digital noise, optimization of shadow details, and building models to distinguish unaltered vs modified digital images.


The Promise for Photography
Here's the proposal: a photograph with maximum DOF (i.e., minimum aperture), great shadow detail, shot in low light, at low ISO, and exposed for the low end of the highlights (zone 8).  Using the theories on digital noise and detection, I will show you how it can be done.

Next
Detection in the Testing Laboratory for a discussion of detection and quantitation concepts in the testing laboratory and how laboratories can check to ensure that the test results are valid.

Footnote
For those curious about the photo. It is a composite of five photographs all taken within a one hour period in the Cairo souk (market place). Western elements in the main image were replaced with more traditional subjects: two men in jeans walking under the arch were replaced with the two women; the apartments in the background were replaced with a scene from a nearby street; the women talking to the merchant were from a nearby street talking to another merchant and replaced a group of tourists. The turbaned merchant and man entering from the right are as they appeared in the primary photo.