Topic: Misunderstanding rem function?

[gskelly]

Hi folks...

I've been lurking around here for a while... reading and learning. I am getting my feet under me so to speak but I am a bit confused with some results I am seeing when using the REM function.

Example: an angle in the format DD.MMSSsss converted to decimal degrees.

Code: [Select]

(defun g (dms / tmp)
    (setq tmp (rem dms 0.01))
    (+
        (fix dms)
        (/ (- (rem dms 1.0) tmp) 0.6)
        (/ tmp 0.36)
    )
)

(g 1) gives 1.0 (good)
(g 31.01) gives 31.01666... (yeah!)

Now if I have a quadrant bearing in the format QDD.MMSSsss I figured I should be able to knock the quadrant Q off using REM:
(g (rem 131.01 100.0)) gives 31.02777... !!! this is not right !!!

this look to me like the (rem 131.01 100.0) must be returning 31.014, however,

(rtos (rem 131.01 100.0) 2 9) gives "31.01" which is what I expected.

Well, now I feel a bit dense. Am I doing something obvious wrong, like completely misusing REM?

Thanks.

Greg

[Kerry]

Long story short.

The implementation of REM is affected by the inherent inaccuracies of floating point calculations and the associated rounding anomalies.

[Kerry]

Links
http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm
http://docs.sun.com/source/806-3568/ncg_goldberg.html
http://en.wikipedia.org/wiki/IEEE_754-1985

[gskelly]

Long story short.

The implementation of REM is affected by the inherent inaccuracies of floating point calculations and the associated rounding anomalies.

Ok... I had given that a passing thought but with the magnitude of the numbers I was dealing with I did not expect to see such significant roundoff errors due to floating point representation. I was also confused by the output from (rtos (rem 131.01 100.0) 2 9) yielding the expected "131.01"... Lesson learned.

Do you know if this high level of introduced error is limited to REM or should I be wary of other functions like angle and distance?

Thanks!

[Kerry]

Do you know if this high level of introduced error is limited to REM or should I be wary of other functions like angle and distance?

yes, I do know.
no, it's not limited to REM.
yes, be wary.

[gskelly]

Oops! I just found a recent post about REM I missed or forgot about... 

[gskelly]

Well, I was just being a little dense last night... for whatever reason I was only using "9" for the precision in the rots calls I made  :ugly:

Need more sleep I guess...

Anyway it is very obvious with more precision displayed:

Code: [Select]


(g 31.01)
31.01
(rem dms 0.01) = 0.0000000000000009
(rem dms 0.01) = 0.0000000000000009
     (fix dms) = 31
 (rem dms 1.0) = 0.0100000000000016
31.01666666666667


(g (rem 131.01 1.0))
31.00999999999999
(rem dms 0.01) = 0.0099999999999903
(rem dms 0.01) = 0.0099999999999903
     (fix dms) = 31
 (rem dms 1.0) = 0.0099999999999909
31.02777777777775

Thanks Kerry.

[vladimirzm]

i don't know but for me it doesn't sound like a "precision" problem:

(rtos (- 200 0.1) 2 20)  ---> "199.9"
(rtos (- 1000 0.1) 2 20) ---> "999.9"

but
(- 100 0.1)  ---> 99.9 ok?
(rtos (- 100 0.1) 2 20) ---> "99.90000000000001"

so the problem is not REM itself. try to build a new version of REM and you will find the same problem.

[gskelly]

so the problem is not REM itself. try to build a new version of REM and you will find the same problem.

Yes, you are correct, I was misusing the function and not alert enough to realize it.

Thanks.