Masking a number

[camerart]

Hi Richard,

I tried it in Oshonsoft, using a calculation where 1 then = 0.995. when a and b (both = 1) are tried, the variable keeps rounding up and down, so I'll have to find how to get over the Oshonsoft problem.

I'm not good at maths, so can you explain what 'digit masking of float types is an invalid action' please. I think that INT = a whole number with nothing after the decimal point??

C.

[richard]

pbp has no floating point type vars so a discussion about floating point arithmetic on this forum is probably inappropriate

a value like 0.995 is a float type .
google "comparing floating point numbers" for an explanation

[richard]

for your second question

I'm not good at maths, so can you explain what 'digit masking of float types is an invalid action' please

I think you mean
I'm not good at maths, so can you explain why 'digit masking of float types is an invalid action' please


see this link for how a computer stores a floating point number (I would assume Oshonsoft uses ieee754 32 bit floats) masking floats is entirely inappropriate

http://www.h-schmidt.net/FloatConverter/IEEE754.html

[camerart]

for your second question

I think you mean
I'm not good at maths, so can you explain why 'digit masking of float types is an invalid action' please


see this link for how a computer stores a floating point number (I would assume Oshonsoft uses ieee754 32 bit floats) masking floats is entirely inappropriate

http://www.h-schmidt.net/FloatConverter/IEEE754.html

Hi R,

It appears that I'm not good at English either

Thanks, I'll check the link.

C.

[camerart]

Ok, thanks. C.