Masking a number

[richard]

if a and b are int type variables and a=0 b=0 then If the statement a==b does not return a true result then that is a bug in Oshonsoft basic {unless Oshonsoft basic is weird}
however if a and or b are of a float type then a true result is not guaranteed ( at least in c or c++ ) for equals type comparisons .
digit masking of float data types is an invalid action , only int data types can be meaningfully masked

since this is a pbp forum (where Oshonsoft basic vagaries not really known) you need to provide more info about your available variable types for a more detailed answer

[camerart]

Hi Richard,

I tried it in Oshonsoft, using a calculation where 1 then = 0.995. when a and b (both = 1) are tried, the variable keeps rounding up and down, so I'll have to find how to get over the Oshonsoft problem.

I'm not good at maths, so can you explain what 'digit masking of float types is an invalid action' please. I think that INT = a whole number with nothing after the decimal point??

C.

[richard]

pbp has no floating point type vars so a discussion about floating point arithmetic on this forum is probably inappropriate

a value like 0.995 is a float type .
google "comparing floating point numbers" for an explanation

[richard]

for your second question

I'm not good at maths, so can you explain what 'digit masking of float types is an invalid action' please

I think you mean
I'm not good at maths, so can you explain why 'digit masking of float types is an invalid action' please


see this link for how a computer stores a floating point number (I would assume Oshonsoft uses ieee754 32 bit floats) masking floats is entirely inappropriate

http://www.h-schmidt.net/FloatConverter/IEEE754.html

[camerart]

for your second question

I think you mean
I'm not good at maths, so can you explain why 'digit masking of float types is an invalid action' please


see this link for how a computer stores a floating point number (I would assume Oshonsoft uses ieee754 32 bit floats) masking floats is entirely inappropriate

http://www.h-schmidt.net/FloatConverter/IEEE754.html

Hi R,

It appears that I'm not good at English either

Thanks, I'll check the link.

C.

[camerart]

Ok, thanks. C.