question for math wizards

[aajgss]

Hi Acetronics,

Here is some brief details of what I did and why I did it.

ADC Counts              ADC Volts           mmH2O         Knots

1                       0.0003              0.041         1.56 

2                       0.0006              0.081         2.21 

3                       0.0009              0.122         2.71



now using

knots = ((SQR(counts))*1563)

and looking at the integer value that I would expect would be returned

ADC Counts     ADC Volts     mmH2O      SQR(counts)       Integer valuegives me     Knots
1             0.0003        0.041         1                       1                  1.56

2             0.0006        0.081         1.414                   1                  1.56

3             0.0009        0.122         1.73                    1                  1.56



now using

knots = ((SQR(counts*100))*1563)/10000



ADC Counts     ADC Volts     mmH2O      SQR(counts*100)       Integer valuegives me     Knots

1              0.0003        0.041             10                    10               1.56
2              0.0006        0.081             14.14                 14               2.188
3              0.0009        0.122             17.32                 17               2.657

So I guess what I am saying is that it was the only way I could see of getting close to the resolution I needed.

As for the remainder, that is the .56 at 1 count, .188 at 2 counts etc.

Anyway, that was my train of thought. I am sure there must be a simpler way, i'm just not able to see it. Well I'm off to find that section of the forum for ideas to improve PBP "Floating Point Math"

regards
aajgss

[Demon]

Yup, I tried three counts in Open Office Calc using 2 versions of the formula and got identical results:

((SQRT(cell))*1563)/1000
((SQRT(cell*100))*1563)/10000

1  1.563
2  2.210
3  2.707

Are those above the expected results?


Or are these?

1  1.560
2  2.188
3  2.657

[aajgss]

Hi Robert,

I think the difference comes from the multiply by 100. This gives the result of for example 3 counts 17 and not 1.732.
But as I said one decimal place is close enough for me.

aajgss

[aajgss]

Thanks to all for you help to this stage.

Now I'm off to create a new thread for the next bit. I have the hardware setup and producing data, just a bit of trouble reading it.

aajgss