Averaging AtoD samples

[HankMcSpank]

Hi Darrel,

Thanks!! (that all sounds too simple...even for someone as clueless as me)

I'll try that soonest ...but I'll be sure to keep the chair nearby to get right back up upon!

(PS I wasn't saying your routine was a hammer to crack a nut per se, but simply a hammer to crack a nut with my simple requirements!)

PPS why do I keep getting that Random Question "missing letter from the word cof_ee" wrong? It's an 'f' damnit.

[HankMcSpank]

This is somewhat related to my ongoing pitch detection quest (in that it's what's driven me to ask - I'm seeing occasional glitches in the input signal, which is causing a hiccup in the 'time measured between comparator interrupts, oncde in a while), but it could relate to any situation...not just pitch detection, so hence coming back to this thread.

My question relates to 'intelligent' averaging (& rejecting spurious values)

Let's say you have a series of data (counts, samples, pulses....whatever) , but within the data occurs the odd 'sample' that's way out of whack. For example.....

2, 2, 2, 3,2,2,3,2,2,2,8,2,2,2,3,3,2

What I'd like to do is average all those samples, but reject any one occurence that's way out of whack (eg if 1 sample per 16 is out of whack, ignore it). so in that example all the 2s & 3s get averaged, but the 8 is ignored....else it'll really dick the average!

Ok, if it were that simple I could probably kludge something together - BUT, lets say that the above series of data is then immediately followed on by by say...

10, 10, 11, 10, 11, 10, 10, 11, 10

therefore giving a stream of values like thus.... 2, 2, 2, 3,2,2,3,2,2,2,8,2,2,2,3,3,2,10, 10, 11, 10, 11, 10, 10, 11, 10

There clearly becomes a problem, because the first couple of 10s of the second series would appear out of whack to any 'moving' simple check that was applied to the first series.

Ideally, I'd like to put in place some form of intelligent averaging routine, where only samples that are close together (say percentage terms) are averaged, any spurious samples out of that range (say 1 in 8 samples) are rejected ..... but if a threshold is breached, then the spurious values aren't treated as spurious anymore, but as new valid data to be then averaged.

Clear as mud?

[mackrackit]

I can not think of a "fast" way of doing it but you could check the difference between each pair and if the difference is more than X toss the value...
maybe...

[HankMcSpank]

I can not think of a "fast" way of doing it but you could check the difference between each pair and if the difference is more than X toss the value...
maybe...

but in that example above, it would mean when the series 10s & 11s arrived, there'd be a problem? If comparing in pairs & throwing the second one a way (ie if it deiviates too much), then the 10s would get rejected cos they're way out of whack wrt to the previous series of 2 & 3s (or have I misunderstood your line of thinking)

[mackrackit]

The first 10 would be tossed but the second 10 would not because now the pairs are 10,10,11...

[HankMcSpank]

The first 10 would be tossed but the second 10 would not because now the pairs are 10,10,11...

Aah...ok, gotcha.

Actually my example 'data series' wasn't a good one, as the data values would actually be in the hundreds (vs units which I used), which allows percentages to be used.

So, assuming a series of data coming into a variable called 'sample'...

' compare a pair called value & value2 from the incoming 'sample'

value = sample     '

upper_window = value2+(value2*4/100) 'create a window 4% above previous sample
lower_window = value2-(value2*4/100)  'create a value 4% below previous sample

if (value>= lower_window) AND (value <= upper_window) then incoming 'sample is within 'allowable error window'?
gosub average 'goto averaging routine
endif
value2 = value1

etc

would something like the above work?#

Edit: Just seen your post/link scalerobotics - off to have reader's digestion! (& yes, I did a search but I guess my keywords weren't erhm key)

[HankMcSpank]

ok, so scalerobotic's link to Melanie's 'sorting numbers' article, is exactly what I need, so I'm trying to get the thing to work.

i've basically got a small array that a prepopulate with some random numbers.

@ __CONFIG _FCMEN_OFF & _HS_OSC & _WDT_OFF & _MCLRE_OFF & _CP_OFF & _IESO_OFF & _BOR_OFF & _PWRTE_OFF

DEFINE  OSC 20          ' set Oscillator at 20Mhz.
DEFINE  NO_CLRWDT 1   ' PBP doesn't clear WDT automatically
DEFINE HSER_CLROERR 1
DEFINE HSER_TXSTA 24h ' Enable transmit
DEFINE HSER_SPBRG 129 ' set USART to 9600 baud  @20Mhz

    CounterA var Byte
    DataA var Byte
    RawData var Byte [6]
    Averaged var byte
    hserout [27,91,72]          'home cursor
    hserout [27,91,50,74]       'clear screen
    
    RawData[0]  = 25
    RawData[1]  = 27
    RawData[2]  = 32
    RawData[3]  = 30
    RawData[4]  = 34
    RawData[5]  = 33

    
    hserout ["before....", 13, 10]
    hserout [dec rawdata[0],13,10]
    hserout [dec rawdata[1],13,10]
    hserout [dec rawdata[2],13,10]
    hserout [dec rawdata[3],13,10]
    hserout [dec rawdata[4],13,10]
    hserout [dec rawdata[5],13,10]
    hserout [13,10]
    hserout [13,10]


    CounterA =0
    gosub SortArray
    gosub Average_data
    hserout ["after....", 13, 10]
    hserout [dec rawdata[0],13,10]
    hserout [dec rawdata[1],13,10]
    hserout [dec rawdata[2],13,10]
    hserout [dec rawdata[3],13,10]
    hserout [dec rawdata[4],13,10]
    hserout [dec rawdata[5],13,10]

    hserout ["averaged = ", dec averaged,13,10]
    goto end1
    
    
Average_data:
    hserout [dec rawData[2]," + ", dec rawData[3], 13,10]
    Averaged = (rawData[2]+rawData[3])/2    ' average the middle two of 6 ongoing samples
    return

SortArray:
SortLoop:
    If RawData(CounterA+1) < RawData(CounterA) then
        DataA=RawData(CounterA)
        RawData(CounterA)=RawData(CounterA+1)
        RawData(CounterA+1+0)=DataA
        If CounterA > 0 then CounterA=CounterA-2
        endif
    CounterA=CounterA+1
    If CounterA < 5 then goto SortLoop
    Return

    end1:
    end

here's my screen output (I average the middle two numbers Rawdata[2] & RawData[3] )....

before....
25
27
32
30
34
33

after....
6
25
27
30
32
33
averaged = 31

It all works (ie sorts & averages correctly) - but where on earth has the number 6 in the sorted numbers come from? (there was no number six in the original array - also, whatever number I place in that array position RawData[4] comes out sorted as 6!?!!)

Edit: Ok, think I've stumbled on the issue - it seems to be related to my array declaration ....because when I use RawData var Byte [8] ...everything works.

So why can't I declare an array 5 bytes 'deep'? ie RawData var byte [5] ???