calca

Open full view…

Problem with modulo and floating point numbers

voptiplex
Wed, 03 Feb 2016 12:03:29 GMT

Hi, I do have experienced a problem with floating point numbers and modulo. I just copied the calca sheet into this forum entry: # Problem: Modulo with floating point numbers The following calculations are correct (as expected): 0,6 / 0,2 => 3 4,3 mod 3,0 => 1,3 The following modulo calculations are incorrect: 0,6 mod 0,2 => 0,2 1,2 mod 0,4 => 0,4 2,4 mod 0,8 => 0,8 4,8 mod 1,6 => 1,6 9,6 mod 3,2 => 3,2 0,5999999999999999 mod 0,2 => 0,2 # Web discussion I found a similar problem which existed in Google spreadsheets and the discussion is quite informative: https://productforums.google.com/forum/#!topic/docs/DwfSwlWby7Q ## Quote from the website: I think I see a pattern here: for every x = n*0.2 (with n = 1, 2, 3, ....) and y = 3 * x -> =MOD( y ; x ) -> x (instead of the expected 0) It may very well be this is another issue with floating point precision and display rounding. E.g. in MOD( 0.6 ; 0.2 ) the value of 0.6 may become encoded as floating point 0.5999999999999999 and therefore the result of the function call may become 0.1999999999999999 . As a result of the display rounding routines this - in essence valid result of the modulo operation - would be rounded up to 0.2 which looks as if it is an invalid result. The issue therefore may lie a bit deeper than with just the MOD function - it may be an issue with the floating point encoding of numbers and how these numbers in turn are displayed in the spreadsheet. When the actual encoding and what we see in the spreadsheet differs we are bound to misinterpret the results that we see

onlywebmail
Fri, 30 Sep 2016 20:56:25 GMT

Yes, the output is just wrong - it seems to do it´s own rounding, which is bad.