Variable Rounding Affects Logic?

[EX-wccprogrammer]

Hey guys, I am stumped on this one. I am using X6, doing some custom post work. Trying to compare the value of cabs with prv_cabs. When cabs is formatted with 3 decimal places, the value of cabs = 335. and prv_cabs = 335. However, the logic cabs <> prv_cabs is true.

 

If I change the formatting of cabs to 7 decimal places, the value of cabs = 335.0000002 and prv_cabs = 335.0000002. However, the logic cabs <> prv_cabs is false.

 

I really need this to work properly. Has anyone else had a problem like this?

[EX-wccprogrammer]

OK, I found the answer. I needed to use the fmtrnd(x) function to evaluate the rounded value of the variable. Not really sure why, but it works.

[crazy^millman]

Can you change the formatting for that process and then change the output for the code to go back to 3 places? Might have to make place holder variables to help with that.

[crazy^millman]

okay scratch that idea.

[Colin Gilchrist]

For anyone that runs across this again, I've got some information that will help. The setting of 'bug4$' controls the output when using the tilde '~' character in front of a variable. To see the "raw" value, set 'bug4$' to '-1'. Then when you output using the debug modifier (~), the raw values are output.

 

Whenever a variable is output, it will be output as the formatted (rounded) value. So the previous (prv_) values will always hold the rounded value. That is why you should use the fmtrnd function when doing a comparison. The fmtrnd function also does not change the raw value of the current variable. It creates a local copy of the variable and rounds that local copy when doing the boolean comparison.

 

Hope that helps,

 

Colin