Skip to content

float math issues in 1.2.5 #33

@gMakaan

Description

@gMakaan

Try this formulas without setting <<formulaPrecision>> and <<formulaFixed>>:

  • 0.99999 = 1
  • 0.999999 = 0.1 (wrong result at 6+ digits after point, but using <<formulaPrecision>> and <<formulaFixed>> can fix it)
  • 0.999999/1 = 0.1
  • trunc(1,1) = 0.9
  • trunc(1,5) = 0.99999
  • trunc(1,6) = 0.10 (with parameter 7 and more it returns same result 0.10)
  • trunc(-1,1) = -0.9000000000000001
  • trunc(-1,2) = -0.9900000000000005 (others in similar manner)
  • round(1,19) = 0.10
  • round(1,37) = 0.10 (here may be more such "wrong numbers")
  • round(-1,1) = -1.0000000000000002
  • round(-1,2) = -1.0000000000000004 (with parameter 3 and more last numbers changed via floating point precision errors)
  • round(-1,14) = -0.9999999999999958
  • round(-1,15) = -1.0000000000000002 (16, 17, 18 param value return -1 as it must)
  • round(-1,19) = -0.9999999999999999 (but it already out of bounds)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions