[TxMt] Re: rounding bug in Math bundle...

[Martin Kühl]

On Tue, Dec 1, 2009 at 17:31, Ale Muñoz <bomberstudios at gmail.com> wrote:
> I expected something akin to what every decent tool out there produces:
>
> $ ruby -e "puts (72/2.54)*2.54"
> 72.0
>
> $ php -r "echo((72/2.54)*2.54);"
> 72
>
> $ perl -e "print ((72/2.54)*2.54)"
> 72
>
> heck, even JavaScript gets it right...

The fact that the reduced precision those tools use masks their use of
imprecise numbers results in an error which matches your expectations
doesn’t actually make them decent.

    > ruby -e 'puts 72/2.54'
    28.3464566929134

    > perl -e "print 72/2.54"
    28.3464566929134%

    > php -r "echo((72/2.54);"
    28.346456692913%

Contrast with the tool TextMate uses underneath:

    > bc -l <<<'72 / 2.54'
    28.34645669291338582677

> Funnily enough, the "Send Line to Google Calculator" on the same
> bundle gets it right...

…as demonstrated in this expression:

72 / 2.54 = 28.3464567


I’m afraid I don’t think I understand your criteria of either decency
/or/ correctness…
Have you heard of “What Every Computer Scientist Should Know About
Floating-Point Arithmetic”[1]? Maybe it could help establish some
common ground here…

[1] http://docs.sun.com/source/806-3568/ncg_goldberg.html