Page: 13 prev next


Cancellation is probably the most serious type of floating-point problem. However, cancellation is not a type of error per se.

Cancellation occurs when two numbers with identical leading digits are subtracted. The subtraction leaves behind the difference of the remaining digits, which is stored as a new double precision number.

Cancellation does not degrade precision if there is no rounding error in the numbers but magnifies rounding error if the numbers are imprecise.