The right way to calculate process limits
The utility of process behavior charts rests in their ability to discriminate between the two types of variation: common causes of routine variation and assignable causes of exceptional variation. The process behavior charts ability to discriminate between these two types of variation is a function of Shewhart's generic three-sigma limits:
Here, X-Bar is the mean and Sigma(X) is a placeholder for one of several within-subgroup measures of dispersion. While you may be inclined to take these generic three-sigma limits literally, and as such add and subtract the product of three times the global standard deviation statistic to a datasets average, the calculation of process limits using the global standard deviation will yield limits that are inflated. This can produce situations where assignable causes of exceptional variation go undetected. To avoid this, the Upper Process Limit (UPL), Lower Process Limit (LPL), and Upper Range Limit (URL) for a dataset composed of logically comparable individual values should be calculated using following formulas:
Here, the X-Bar is the mean of the individual values in the dataset, 2.660 and 3.268 are both scaling factors, and the R-bar is the average moving range.
The 2.660 scaling factor is required in the formulas for the UPL and LPL to convert the average moving range into the appropriate amount of spread for the individual values. When a dataset consists of logically comparable individual values, the value of this scaling factor will always be 2.660. Like 2.660, the 3.268 scaling factor is used to convert the average moving range into the appropriate amount of spread relative to the moving range values. For subgroups of size n=2, as is the case when calculating the difference between subsequent values in a dataset composed of individual values, the value of this second scaling factor will always be 3.268.
Before the average moving range can be calculated, the values of the moving range must be calculated. The values of the moving range are the absolute value of the difference between subsequent values in a dataset. For instance, if the first value in a dataset of individual values is 3 and the second value is 2, the absolute value of the difference between these values is 1. If the second value in a dataset is 2 and the third value is 4, the second moving range is 2. Continuing this process for all subsequent pairs of values in a dataset produces the moving range.
“The proper use of data requires that you have simple and effective methods of analysis which will properly separate potential signals from probable noise.”
— Donald J. Wheeler
If Shewhart’s generic formula for the calculation of process limits capable of discriminating between the two types of variation is Average +/- 3 Sigma(X) why then can we not use the global standard deviation statistic to calculate the process limits. Is it not the case that in statistics Sigma represents the standard deviation of a dataset?
Standard deviation is a measure of dispersion that relates the distribution of all the values in a dataset to the mean. Its calculation assumes that “the data can be logically considered to be one large homogeneous collection of values, all obtained from the same underlying and unchanging process” (Donald Wheeler, Making Sense of Data). This assumption sits at odds with the intent of the XmR Chart which “examines a collection of values to see if they might have come from one underlying or unchanging process, or if they show evidence of process changes” (Donald Wheeler, Making Sense of Data). Put more simply, the XmR Chart examines process data for evidence of non-homogeneity while the standard deviation statistic assumes a dataset is homogeneous. This mismatch in purpose makes standard deviation ill suited for the calculation of process limits, especially for a dataset that exhibits nonhomogeneous behavior like that shown in Figure 1.
Why global standard deviation doesn’t work
Fig 1: Comparison of process limits calculated using different methods
Moreover, process limits that use the standard deviation statistic will be inflated. These limits will be wider than they would be using the within-subgroup measure of dispersion. As a result, the sensitivity of the process behavior chart is reduced. This can lead to misleading interpretations and results. Figure 2 serves as an example.
Fig 2: Comparison of process limits calculated using different methods
In Figure 2, the process limits for the manufacturing process from the prior section are calculated using the formula Average +/- 3 Sigma on the left and Average +/- 2.660*R-Bar on the right. While the Lower Process Limit in this case remains the same for both sets of process limits, the value of the Upper Process Limit that results from using the global standard deviation is 0.1 unit larger. While in this particular case, the inflated value of the Upper Process Limit does not change the characterization of this process from predictable to unpredictable, in instances where values hover near the process limits this could be the difference between identifying an assignable cause of exceptional variation and missing it.
In order to get robust process limits, as articulated by Donald J. Wheeler on page 162 of Making Sense of Data, you must “use the moving ranges to characterize the short-term, point-to-point variation. The formulas then use this short-term variation to place limits on the long-term variation.” The interplay between short and long term variation is what makes calculating process limits using the scaling factors and average moving range an effective approach for identifying assignable causes. Without this interplay, the fidelity of Shewhart’s method is lost and the possibility of mistakenly characterizing an unpredictable process as predictable increases.