[TxMt] Re: rounding bug in Math bundle...

[King, Steven]

On 1 Dec 2009, at 10:54, Timothy Bates wrote:

> What else did you expect? It's using a calculator not a symbolic math  
> package, so 71.99999999999999999999 is spot on.


That was my initial reaction, too. It's just normal floating-point imprecision. But I tried it in C with single-precision floats, and got 72.0 as a result. Hmmm...

Okay, so what's the bundle doing? Here's the code:

    printf " = %s" `{ tr -d ,; echo; } | bc -l`| perl -pe 's/(\.[^0]+)0+$|\.0+$/$1/'

Yikes! Good lord, that's ugly! The culprit is the 'bc' command, which is doing the actual calculation. All the rest is just for formatting.  Getting rid of all the rest of the cruft,

    macduff:/tmp$ echo '(72 / 2.54) * 2.54' | bc -l
    71.99999999999999999999

You could take it up with the Free Software Foundation, though I suspect they have a fanatical academic reason why they're doing it the One True Way.

To be fair, I don't think bc is the right tool for this job anyway. It uses an arbitrary-precision library rather than the floating-point that everything else uses. Both methods have their advantages and disadvantages, but for the precision most people tend to deal with I think the standard floating-point libraries do a better job. At the very least, we're more accustomed to dealing with their idiosyncrasies.

Personally, I'd re-write the bundle to avoid bc. Since perl is being used anyway it wouldn't be hard to write the command such that perl does the whole thing, eliminating that ugly pipeline altogether.

-- 
Steve King
Sr. Software Engineer
Arbor Networks
+1 734 821 1461
www.arbornetworks.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macromates.com/textmate/attachments/20091201/55c8347c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2463 bytes
Desc: not available
URL: <http://lists.macromates.com/textmate/attachments/20091201/55c8347c/attachment.p7s>