A round-off error, also called rounding error, is the difference between the calculated approximation of a number and its exact mathematical value. Numerical analysis specifically tries to estimate this error when using approximation equations and/or algorithms, especially when using finitely many digits to represent real numbers (which in theory have infinitely many digits). This is a form of quantization error.
When a sequence of calculations subject to rounding error are made, errors may accumulate in certain cases known as ill-conditioned, sometimes to such an extent as to dominate the calculation and make the result meaningless.
The error introduced by attempting to represent a number on the computer is called representation error. Some examples:
| Notation | Represent | Approximate | Error |
|---|---|---|---|
| 1/7 | 0.142 857 | 0.142 857 | 0.000 000 142 857 |
| ln 2 | 0.693 147 180 559 945 309 41... | 0.693 147 | 0.000 000 180 559 945 309 41... |
| log10 2 | 0.301 029 995 663 981 195 21... | 0.3010 | 0.000 029 995 663 981 195 21... |
| ∛ 2 | 1.259 921 049 894 873 164 76... | 1.25992 | 0.000 001 049 894 873 164 76... |
| √ 2 | 1.414 213 562 373 095 048 80... | 1.41421 | 0.000 003 562 373 095 048 80... |
| e | 2.718 281 828 459 045 235 36... | 2.718 281 828 459 045 | 0.000 000 000 000 000 235 36... |
| π | 3.141 592 653 589 793 238 46... | 3.141 592 653 589 793 | 0.000 000 000 000 000 238 46... |
Increasing the number of digits allowed in a representation reduces the magnitude of possible round-off errors, but any representation limited to finitely many digits will still cause some degree of round-off error for uncountably many real numbers. This kind of error is unavoidable for conventional representations of numbers, but can be reduced by the use of guard digits.
Double-rounding can increase the round-off error. For example, if the numeral 9.945309 is rounded to two decimal places (9.95) for data entry purposes, and then rounded again to one decimal place (10.0) for display purposes, the apparent round-off error is 0.054691. If the original number was rounded to one decimal place in one step (9.9), the round-off error is only 0.045309.
There are five standard ways of performing the rounding in IEEE standard arithmetic:
If the final digit of a decimal number is 5, then it doesn't matter whether it is rounded up or down because a rounding error of the same magnitude will occur. A common rule is to round toward the nearest even number; since 5 is odd, this rule prevents cascading of rounding errors.