Friday, 29 October 2010

ActionScript Precision discrepancy problems

mmmm, this is a bit of nightmare, especially for those writing financial software... check this, take a perfectly good number (AS3 Number) such as 11995, then divide it by 1000000, then multiply it by 1000. you would expect to get 11.995, but no, you get 11.995000000000001. then if you multiply by ten, the discrepancy gets slightly worse so each time you apply calculations, you are dealing with the wrong number.

to sort this out, you could roll your own string manipulation functions or there's a neat little static function built into the Number object called toFixed, see the docs:

you use it like this...

var num : Number = ( 11995 / 1000000 );
var num2 : Number = num * 1000;
var str2 : String = num2.toFixed( 6 );
var num3 : Number = new Number( str2 );


No comments:

Post a Comment

The internet is a crazy place so please help us keep it under control by leaving only relevant messages and comments, thanks a bunch! :m)