Decimal Floating Point in Scientific Calculators

Back in the 1980's when I first studied numerical analysis, different computers used different storage schemes for floating point arithmetic.  Although binary floating point was widespread, there were significant differences between the binary floating point systems on machines from different manufacturers.  IBM made things even more interesting with the hexadecimal (base-16) floating point used on the System-360/370 computers.  The IEEE-754 standard was just being developed.  The variety of floating point standards was a useful source of examples for teaching the basic concepts of floating point arithmetic.

Today, every computer in wide use, along with smartphones, tablets, and higher-end graphing calculators all use IEEE-754 binary floating point arithmetic.  There really aren't any other systems in widespread use (one recent exception is the use of 16 bit "half precision" in neural network computations.)

Inexpensive scientific calculators from Casio, HP, Sharp, and Texas Instruments are one exception to this trend.  These calculators typically use decimal floating point arithmetic.  It's nearly universal in these calculators to have an exponent range from 10 to the -99 power to 10 to the +99 power, with 10 significant digits displayed.

The 10 digit results shown on the display should be properly rounded.  You can check how rounding is done with

1.123456788 + 6E-10

which should round up to

1.123456789

and

1.123456788 + 5E-10

which might either round up (ending in "89") or round to nearest even (ending in "88".)  In practice, nearly every calculator I've seen rounds 5's up (R5UP in the table below.)

Often times, the calculator performs internal computations with more significant digits.  You can check this on a calculator by performing a calculation like

1.123456789+1.23456789E-10

and then subtracting

ANS-1.123456789

A result of 1.2E-10 indicates that the calculator maintains 12 digits internally.

If 15 digits were being kept internally, then you'd get back 1.2345E-10, or perhaps 1.2346E-10, depending on whether the calculator was properly rounding the fifteenth digit or truncating.

A feature that some newer calculators have is the ability to convert from decimals to nearby exact fractions with small numerators and denominators.  Typically, the user just presses a button to switch back and forth from decimal to fractional form. In the case of the Casio FX-115ES PLUS described in the table below, you can also get the answer in repeating decimal form with a bar over the repeating digits.  The Casio limits fractions to a total of 10 digits in the numerator and denominator.  The conversion to fractional form seems to be quite resistant to small round-off errors. 

The conversion to fractional form can be done by forming the continued fraction expansion of the decimal and truncating after a small number of terms in the expansion.

I grabbed the scientific calculators in my desk and ran the tests described above.  The results are shown in the following table. Here R5UP refers to conventional rounding with 5's rounded up, TRUNC refers to truncation with no rounding.  I'd appreciate information about any other scientific calculators that you might have in your desk drawer.