Evaluation of measurement uncertainty – Monte Carlo method

Advantages of Monte Carlo method are presented and compared with A and B type method of measurement uncertainty evaluation. Problem of uncertainty determination, in case of two or more dominant components, is discussed. Results of experiment to evaluate impact of probing strategy on measurement uncertainty of roundness deviation are presented. Issue of ‘systematic error’ in evaluation of coordinate measurement uncertainty is analyzed.

When estimating the measurement uncertainty, apart from methods of type A and B [1][2][3], the simulation technique, called Monte Carlo (MC) [4], is becoming more and more widely used.The analysis of the literature shows that it is the most commonly used alone.In the area related to the product geometry specification, for example, software for estimating the uncertainty of coordinate measurements is known [5].
The aim of this publication is to draw attention to the fact that the MC method can also be used to estimate individual components of measurement uncertainty and, above all, to analyze measurement uncertainty.
The MC method is, by its nature, universal.It allows to determine the distribution of any function (not only the sum) of any number of random variables with arbitrary probability distributions.The direct result of the MC simulation is a significant number (in [4], there is even a million) resultant random variables, based on which it is possible (depending on the need) to determine the empirical distribution (histogram) or calculate any characteristics of the random variable as: average, standard deviation, higher order moments or quantiles.
An important advantage of the MC method is intuitiveness, which will be shown in three examples: estimation of measurement uncertainty in caliper calibration, evaluation of the impact of sampling strategy on the uncertainty of roundness deviation measurement and discussion of the "systematic error" problem in the estimation of uncertainty of coordinate measurements with reference objects.
If the purpose of the simulation is to determine the expanded uncertainty of measurement U -and this is the ___________________ * Dr inż.Wojciech Płowucha (wplowucha@ ath.bielsko.pl),dr hab.inż.Władysław Jakubiec prof.ATH (wjakubiec@ath.bielsko.pl),mgr inż.Paweł Rosner (prosneri@ath.kielce.pl)-Wydział Budowy Maszyn i Informatyki Akademii Techniczno-Humanistycznej w Bielsku-Białej case in most cases -it is not necessary to determine the probability distribution in the strict sense of the termneither the form nor the distribution parameters are valid.Uncertainty U may be calculated directly from the empirical distribution of the error (random variable X) after rejection of 5% of the extreme values (both positive and negative), which corresponds to the direct application of the formula:

Random number generators
Commonly available software -used both in industry and at universities -is MS Excel.It contains a random number generator with a uniform distribution on the segment (0, 1), that is in the form of the function "= los()", in addition, the "Generating of pseudo-random numbers" tool that allows to generate random numbers is available in the appendix "Data analysis" for two probability distributions of a continuous random variable: uniform (uniform, rectangular) and normal.If we assume that in the MC method, the same distributions will be used as the error models, as in the B method, at least two random number generators are missing: the triangular distribution and the U distribution.In both cases, the cumulative distribution method can be used or the other, simpler solutions:  random numbers with a triangular distribution T(-a, a) can be obtained when two independently generated random numbers with the same uniform distributions R(a/2, a/2) are added,  random numbers with the distribution U(-a, a), understood as the distribution of the arcsine of the sine, can be generated using the fact that the random variable X defined as has a U(-a, a) distribution, if the random variable Y has the uniform distribution R(0, 2π).
Minitab [6] software is better suited for conducting simulations, which has much more generators, although only three are typical for this application (monotonous, normal and triangular).Significant advantages of this software include a maximum of 10,000,000 runs (the number of rows of a sheet) and friendly graphical tools (e.g.easy drawing of a histogram with the possibility of applying a graph of the density function).
Simulation using the MC method is also possible in the software for estimating the uncertainty of measurement -GUM Workbench [7].There are generators of uniform, triangular, normal, U and trapezoidal distributions.This software also enables simulation for correlated components.

Analysis of the caliper calibration example
The basic document used in the calibration laboratories to estimate the measurement uncertainty in the calibration is EA-4/02 [8].According to it, the uncertainty of measurement U (according to [8] "obtained from multiplying the standard uncertainty by the expansion coefficient") and the expansion coefficient k, should be given on the calibration certificates.The requirement given in brackets contradicts the practice, because often the extended uncertainty is determined first.
A good example of showing the advantages of simulation technology is the estimation of the uncertainty of measurement when calipers are calibrated.The document assumes that two dominant uncertainty components have uniform distributions: R(-50, 50) and R(-25, 25) (in μm).It can be calculated theoretically (using the probability calculation) that their sum has the following trapezoidal distribution (fig.1):  In an MS Excel spreadsheet, it can be generated up to approximately 32,000 random numbers in one column.If such a number of repetitions is adopted, estimates of the expanded uncertainty of measurement from the range (58.87÷59.40)μm are obtained.Since the uncertainty of measurement is given with accuracy to one or two significant places, the accuracy obtained is quite sufficient.
According to the requirements of document EA-4/02 [8], value of the extension coefficient k has yet to be found on the calibration certificate.In the example given in EA-4/02, there are formulas, the origin of which is known or can be reproduced only by experts in the field of account probability.If the simulation technique is used, the task is simple: standard uncertainty is the standard deviation of the results of the simulation experiment, and the expansion coefficient is the quotient of U uncertainty and standard uncertainty u.In the simulation example u = 32 μm and k factor = 1.85 were obtained, while the exact (analytically calculated) values are 33 μm and 1.83, respectively.
It is worth noting that the example with a trapezoidal distribution as the sum of two uniform distributions is the only task solved by analytical tools.

Analysis of the example -influence of the number of sampling points on the uncertainty of roundness deviation measurement
The essence of the MC method is also explained by the example regarding the evaluation of the probability distribution of errors in measuring the roundness deviation (hole or shaft) given in [9].It refers to the recommendation that, if it is necessary to use a small number of sampling points, it should be 7 points distributed evenly around the circumference of the circle.It was assumed that the deviation of roundness of the measured object has the form of triangularity (this is the least favorable case), and sampling takes place in 6, 7 or 8 evenly spaced points.
The equation of the circle with the deviation in the form of triangularity in the polar system (r, φ) has the form where: R -nominal radius of circle, A -amplitude of roundness deviation.It was assumed that R = 100 mm and A = 0.05 mm (the simulation can be performed only for specific data).
In the example given below -in contrast to [9] -the error of measuring this deviation is calculated instead of the deviation value.The simulation experiment in the MS Excel worksheet consists of:  generating from the uniform distribution on the distance (0, 2π/7) of the angular position of the first sampling point:  calculation of angular positions of the remaining 6 sampling points:  calculation of radius values (second coordinate):  converting the coordinates of points to the Cartesian system:  calculation of coordinates of the center and radius of the circle (the algorithm given in [10] was used, allowing to write patterns in one row of the MS Excel spreadsheet),  calculating the roundness deviation Δ as the maximum difference in the distance of the sampling points from the calculated center of the circle:  calculation of the measurement error of roundness deviation δ (correct result is: 2A = 0.1 mm):  repeating these calculations a significant number of times and building the obtained histogram results.
In the example, instead of generating the angular position of the first sampling point, it can be systematically divided the segment (0, 2π/7) into n parts and perform calculations n times, changing the position with a step of 2π/7/n.Number of repetitions does not have to be large; already for n = 50 very good results are obtained.However, if the results of the simulations are to be developed in the form of a histogram, it is worth assuming, for example, n = 1000.In each case, the measurement errors are negative.With 6 sampling points (fig.2a), it is possible to make both an error equal to 0 and -100 μm (the value of the measured deviation is 100 μm).After rejection of 5% of the worst simulation results, the uncertainty U = 92 μm is obtained.In the case of 7 sampling points (fig.2b), the measurement errors range from -2.5 to -5 μm, which is only 2.5% to 5% of the measured value, although they are never less than -2.5 μm.95% of errors do not exceed U value = 4.6 μm.In the case of 8 sampling points (fig.2b), measurement errors range from -7,6 μm to 0. 95% of errors do not exceed U = 6.9 μm.
Similarly, it can be analyzed the effect of other sampling strategies on the results of the diameter measurement and/or roundness deviation.

Discussion of the problem with "systematic error" in the ISO 15530-3 standard
Despite the publication of JCGM 100 and 101 [1,4] documents, there is still no full agreement on the use of various methods for estimating uncertainty.In 2011, the ISO 15530-3 standard [11] was published, providing a procedure for estimating the uncertainty of coordinate measurement using the experiment.Before it, there was the technical specification ISO/TS 15530-3 [12] from 2004.Significant differences between these documents led to a discussion, in which the authors of this publication also participate [e.g.13].
In both versions, the experiment consists in performing a 20-fold measurement of the calibrated artifact.Measurements are spread over a longer period (in the example attached to the standard, it is 7 days).They are performed at different times of the day and by different operators.The document from 2004 [12] provides for the estimation of the expanded uncertainty of measurement based on a formula that, after omitting the details that are not relevant for this publication, is: where: x -mean value, s -standard deviation of the observed differences with the results of the calibration.
During the work upon the newer document, it was considered a systematic error, which should be -in accordance with the recommendations of the guide [1]corrected.
The problem lies in the fact that the average value mentioned is not the result of any known systematic influence and should not be treated as a systematic error.This can be confirmed by performing a simulation experiment consisting in generating 20 numbers from different distributions and demonstrating that the results obtained according to the formula from the document [12] do not give grounds to question its correctness.
The experiment considered four probability distributions for measurement errors (normal, uniform, triangular and U) and it was assumed that in all cases, the expanded uncertainty of measurement U corresponding to the probability of 0.95 is the same and amounts to 10 μm.The results of the experiment defined in this way are listed in the Table.

Conclusions
The MC method -due to its versatility as well as the ease of design and implementation of the experiment -is worth recommending to supplement the methods of type A and B.
The examples indicate many possible applications.In addition, the MC method has didactic advantages.
A spreadsheet containing calculations can be found on the ATH Laboratory of Metrology (www.lm.ath.bielsko.pl) in the "Download" tab.

( 3 which
allows to calculate the expanded uncertainty of measurement U as a quantile of 0.975 of this distribution.The value U = 59.19 μm is calculated in this way.

Fig. 1 .
Fig. 1.Graph of the probability function of the trapezoidal distribution as the sum of two uniform distributions R(-50, 50) and R(-25, 25), with the quantiles 0.025 and 0.975 marked However, value of U can be determined much more easily (and without knowledge of probability) by the MC method.Just:  generate two sequences of random numbers with the aforementioned uniform distributions and calculate the sum of pairs of random numbers,  reject 5% of the extreme values of the random numbers obtained in this way (it is more convenient to calculate the absolute values of the sum of pairs of random numbers and reject 5% of the largest values or calculate the quantile value of 0.95).In an MS Excel spreadsheet, it can be generated up to approximately 32,000 random numbers in one column.If such a number of repetitions is adopted, estimates of the expanded uncertainty of measurement from the range (58.87÷59.40)μm are obtained.Since the uncertainty of measurement is given with accuracy to one or two

Fig. 2 .
Fig. 2. Histogram of roundness deviation error evaluation depending on the number of sampling points: a) 6 points, b) 7 and 8 points Results in the form of histograms are shown in fig.2.In each case, the measurement errors are negative.With 6 sampling points (fig.2a), it is possible to make both an