Floating arithmetic

Addition of large and small floating point numbers can lead to loss of information. Again, we're using base 10 with 5 significant digits. Suppose we decide to add 3.4725 $\times$ $10^0$ and 1.1203 $\times$ $10^{-3}$. Our floating point arithmetic unit aligns these two numbers (somewhat as you would line them up to add them on paper) by expressing them with the same exponent: 3.4725 $\times$ $10^0$ plus 0.0011 $\times$ $10^0$ -- the second mantissa lost the significant digits 203 by being squashed into 5 digits! The sum is 3.4736 $\times$ $10^0$.

Things would be even worse if the exponent of the second number were smaller. Suppose we added 3.4725 $\times$ $10^0$ plus 1.1203 $\times$ $10^{-5}$. To align the two numbers, the second one is re-normalized as, well, 0.0000 $\times$ $10^0$ -- all the significant digits fell off the right hand side. So the sum is 3.4725 $\times$ $10^0$. Care is required when adding numbers of greatly different magnitudes.

Stranger still, floating point addition is no longer associative. The following equation says we can group addition according to taste, and the results should be the same:

$$((((1.0000 \times 10^2 + 3.0000 \times 10^{-3}) + 3.0000 \times 10^{-3}) + 3.0000 \times 10^{-3}) + 3.0000 \times 10^{-3})$$

$$\qquad = (1.0000 \times 10^2 + (3.0000 \times 10^{-3} + (3.0000 \times 10^{-3} + (3.0000 \times 10^{-3} + 3.0000 \times 10^{-3}))))$$

The floating point calculation on the right is 1.0000 $\times$ $10^2$, the one on the right is 1.0001 $\times$ $10^2$ -- not quite the same.

Subtraction can have catastrophic cancelling when the numbers being subtracted are very close. Suppose the true values we were subtracting were 4.57235000 - 4.57234999. We can only keep 5 digits, and the rules say to round one up and the other down, so we end up with 4.5724 - 4.5723 = 0.0001. However, the true difference is 0.00000001, yielding a relative error of 9999, or 999,900%