Microsoft Excel Numeric Precision

Despite the use of 15-figure precision, Excel can display many more figures (up to thirty) upon user request. But the displayed figures are not those actually used in its computations, and so, for example, the difference of two numbers may differ from the difference of their displayed values. Although such departures are usually beyond the 15th decimal, exceptions do occur, especially for very large or very small numbers. Serious errors can occur if decisions are made based upon automated comparisons of numbers (for example, using the Excel If function), as equality of two numbers can be unpredictable.

In the figure the fraction 1/9000 is displayed in Excel. Although this number has a decimal representation that is an infinite string of ones, Excel displays only the leading 15 figures. In the second line, the number one is added to the fraction, and again Excel displays only 15 figures. In the third line, one is subtracted from the sum using Excel. Because the sum in the second line has only eleven 1’s after the decimal, the difference when 1 is subtracted from this displayed value is three 0’s followed by a string of eleven 1’s. However, the difference reported by Excel in the third line is three 0’s followed by a string of thirteen 1’s and two extra erroneous digits. Thus, the numbers Excel calculates with to obtain the third line are not the numbers that it displays in the first two lines. Moreover, the error in Excel’s answer is not just round-off error.

Excel works with a modified 1985 version of the IEEE 754 specification. Excel’s implementation involves an amalgam of truncations and conversions between binary and decimal representations, leading to accuracy that sometimes is better than one would expect from simple fifteen digit precision, and sometimes much worse. See the main article for details.

Besides accuracy in user computations, the question of accuracy in Excel-provided functions may be raised. Particularly in the arena of statistical functions, Excel has been criticized for sacrificing accuracy for speed of calculation.

As many calculations in Excel are executed using VBA, an additional issue is the accuracy of VBA, which varies with variable type and user-requested precision.