Random Events, Trends, and Hot Temperatures

Random Events, Trends, and Hot Temperatures

(c) Ken Osborn Jan 17, 2015

Today's papers (Jan 17, 2015) say 2014 was the hottest on record since 1880. An engineering friend says the trend may be just a random thing. Could that be? Though the data do show a reasonable closeness to an independently produced set of random values with the same average and variability (standard deviation), I decided to test it by comparing the first half of 1880-2014 to the second half.

If random, dividing the data into two halves and ordering the values from lowest value to highest within each half would produce two superimposed sets that would be indistinguishable. If they are part of a trend of increasing temperatures, the sets would not superimpose and the second set would be shifted to the right on the chart. So which is it? Check out the graphs.

 [Data source: http://data.giss.nasa.gov/]


First, let me state that I do understand that random events can generate what appears to be a trend, as in this example:


Chart 1: Example of a 'trend' from a random generation of values 


The Random Walk chart displays what appears to be a strong trend with a correlation coefficient of 0.96, sufficient to earn an award for superior performance by an inebriated soul staggering along a mostly straight path.  I should note that it took over 50 runs of the program to get this result and doubt I could reproduce this particular set again.  


But if we can get a randomly generated collection of results to look like a trend, how do we know that our trend was not generated randomly?  

One way, of course, would be to rerun the events and see if we get the same trend.  However, since I'm looking at data collected from 1880 to 2014 that is not a realistic approach.  Another way would be to take the data, split it into two halves, and compare the two halves.  If the data are a random distribution around a central value, the first half of the data should match the second half of the data.  But first, let's try that with some randomly generated data.


Chart 2: Plot of 100 random values generated by a Monte Carlo simulation 




In chart 2, 100 random values ordered from low to high are plotted against their rank (probability). Half of the values exceed and half are below the average of zero and are symmetrically distributed around the center.  In other words, these data exhibit a nice Normal type distribution.  Note that the average and standard deviation for these data are the same as the temperature data to follow.  




So now if those data are split into two halves and each half is ordered independently from lowest value to highest value and plotted against its rank, what do we get?  


Chart 3:  Comparison of two halves of a randomly generated set of values after ranking each half




The values in blue represent the first half of 100 random values and the ones in red the second half.  Each half was then ranked from lowest to highest value and plotted against the ranking (1 to 50).  The superimposition is not exact, but these are randomly generated results so one should not expect an exact agreement between individual values.  But maybe a real trend would also show something like this.  Let's try it.  


Chart 4: A trend (Y= 10X+5) of 100 values partitioned into two ordered halves 




The trended data set of 100 values was generated from the formula Y = 10X+ 5.  The set was divided into two halves and each half was ordered from lowest value to highest value then plotted against its rank from 1 to 50.  Unlike in chart 3 with conformable data sets, these two sets show no overlap at all.  So how will this work with real data?  


 

Chart 5: Comparing the ranked temperature anomalies from 1880 to 2014 with a random data set  (Source: http://data.giss.nasa.gov/)

The values in green in chart 5 represent the temperature anomalies from 1880 to 2014 ranked from lowest to highest value.  Each value represents the deviation from the average for the 20th century.  The values in blue were randomly generated using the mean and standard deviation from the temperature anomaly set.  They do look as close as two separate runs of a random number generator.  But remember, the real test is to see if the first half of the data (1880 to 1946) matches the second half (1947 to 2014).  Any guesses?  

Chart 6: Comparison of two halves of the 1880-2014 temperature data anomalies


In chart 6, the values in red are for the years 1880 to 1946 and the values in green for 1947 to 2014.  Each set is ordered from its lowest to highest value and plotted against corresponding year.  They do not match and are clearly two separate distributions.  I leave the conclusion to you as to whether these data have been generated by random events.    

The Outlier Paradox and Global Warming

The Outlier Paradox and Global Warming

© Ken Osborn

Feb 2012

Hot winters ahead?  I can’t say for sure, but this last January seemed a lot warmer than a couple years ago when the winter was mild.  I checked the records for the city of Oakland (airport) and the daily highs for Jan 2012 averaged 2.3 degrees Fahrenheit (dF) warmer than Jan 2010.  That doesn’t seem like a lot.  But when I checked for the warmest days in 2010 and compared them to 2012 I saw that my impression that it seemed warmer this year than in 2010 was correct.  In 2010 two of the Jan daily high temperatures exceeded 60 dF and in 2012 there were 7 days in Jan exceeding 60 dF. 

So my impression that this winter was warmer was based not on the average of the daily high temperatures but rather the number of warmer than typical days.  This I call the Outlier Paradox: increasing means accelerates extremes.  Small changes in a data set's average are associated with large changes in the frequency of values exceeding some threshold value: these extreme values are also referred to as outliers.  This is a statistical property of numerical distributions, as I will try to explain.

Measurements repeatedly performed on any item and collected into a data set are frequently randomly distributed.  Given a large enough data set, if the individual results are randomly distributed, the shape of the plotted numbers will be symmetrical: values exceeding the average will balance nicely against numbers below the average in a mirror image fashion.  This is called a Normal distribution.  So are temperature records Normally distributed? 

Chart 1 is a plot of the Jan 2012 daily high temperature readings for the City of Oakland.  The data are ranked from low to high and plotted against the probability that a given measurement will exceed all other measurements in the set.  Except for the four highest values, the plotted temperature readings (red) nicely fit the curve for a Normal distribution (blue).  The curve is symmetrical around the center with roughly the same number of readings on either side.  Measurements close to the center have a higher probability and measurements far from the center a lower probability.  This is visualized by the flattening of the curve at the extreme ends. 

Chart 1: Testing temperature readings for Normality

An interesting property of Normal distributions is that even large changes in the extreme values have a smaller effect on the average.  When examining changes over time, averages only may be a poor predictor of environmental impacts if the outliers are ignored.  

As an example, see Chart 2.  Data set 1 represents a collection of measurements with an average of 100 and a standard deviation of 10%.  Approximately 4 out of 1000 measurements will exceed a threshold of 125.  Change the average to 105 and the number of measurements exceeding 125 increases to 16 out of 1000.  A 5% change in the average becomes a 400% change in the values exceeding a threshold just 25% above the average (100*16/4 = 400%).  An increase in the standard deviation will magnify the spread of extreme values even more. 

Chart 2: A small change in the mean is associated with large changes in the frequency of outliers

This property holds for any set of measurements taken over time when the distribution is Normal.  Records of rainfall, temperature, atmospheric pressure, and number of cars per hour passing a given point on the freeway can all be treated as Normal distributions.  Even though there are causal factors associated with each of these, any given measurement is randomly determined relative to the measurements that immediately precede or follow.  Though we know it may rain tomorrow, the exact number of inches of rain that will fall is an unknown until after the event.  

Returning to the records for the City of Oakland for Jan of 2010 and 2012, let’s take a statistical look at the distribution of daily high readings (Chart 3).  The average high temperature for Jan 2010 was 55.7 degrees Fahrenheit (dF) and the variance as measured by the standard deviation was 2.6 dF.  For 2012 the Jan average high temperature was 58 dF and the standard deviation 4.9.  Using the lower 2010 variance for both years, the fitted Normal curves predict 5% of days exceeding 62 dF for 2012 and 0.8% for 2010.

Chart 3: Comparison of Jan High Temperatures for 2010 and 2012 using 2010 Variance

Chart 3 demonstrates the effect of changing the mean of a distribution of data but not the standard deviation.  The curve shifts horizontally to the right for an increase in the mean with each individual point moving the same amount so that the two curves are parallel to each other. 

Of course the variance was not the same for the two years, and when the change in variance is considered (Chart 4), the differences are even greater with a prediction of 21.8% of the days exceeding 62 dF for 2012 compared to the 0.8% for 2010. 

Chart 4: Comparison of Jan High Temperatures for 2010 and 2012

When both the standard deviation and mean are changed, the curve not only shifts laterally but also rotates.  Chart 5 using a hypothetical set of temperature demonstrates what happens when the mean is fixed mean but the standard deviation changes.  Here the curve rotates around the center but the center does not move horizontally.   Thus if two sets of temperature records (or any measurement records) have the same average but different standard deviations, the set with the higher standard deviation will have more extreme values at both the high and low temperatures.   

Chart 5:  Changing only the standard deviation rotates the curve around the mid-point

  

The extra warm days of Jan 2012 should not be taken in isolation to determine whether global warming is a reality.  These data represent a narrow temporal and spatial snapshot.  Next winter may bring even warmer winter days or it could bring winter lows that are the lowest of the decade.  While we might take note of unusually extreme temperatures, it is the preponderance of data that must answer the question of whether an apparent trend is merely a statistical excursion or a real trend.  

Ref: http://www.wunderground.com/history/airport/KOAK/2010/1/1/MonthlyHistory.html?req_city=NA&req_state=NA&req_statename=NA