New software filter for adc reading

[aratti]

To overcome the ADC reading instability, several type of software filters have been devised and
among these the average, or mean value, is the simpler and most widely used.

I came to the conclusion that a better method is the MEDIAN and not the MEAN since the average
incorporate into the final result (the mean) also any anomalous reading due to spikes or other
electrical noisy phenomenon could occur during the sampling.

The MEDIAN which is the central value of the frequency distribution of our ADC reading, coincide
with the mean if the frequency distribution is normal, while the MEAN drift from the MEDIAN
all the time that anomalous reading are taken.

Here the contradiction, we make the average to get rid of anomalous reading but we just incorporate
them, in other words, the mean value is a way to dilute anomalous reading while, the MEDIAN is a mean to
get rid of them.

Rock stable results have been obtained extracting the MEDIAN out of 15 consecutive readings, with
the INCLUDE module "Al_ADC_Median.PBP", which I am glad to post here and share it with the community.

The use is rather simple, just assign the ADC channel to the channel variable and call the module
with "GOSUB GetADC" the ADC_Value (word variable) returned, will contain the ADC MEDIAN of 15 reading.

I have also posted a working code using PIC 12F683 as an example on how to use the include module.

Place the "Al_ADC_Median.PBP in the PBP folder, add the include to your code and you are done.

I hope the folks working with analog can find it useful. Comments on the subject and on the stability
of the MEDIAN filter are welcome.

Cheer and All the best wishes for the new year.

Al.

PS. You need to rename the extention form .TXT to .PBP since I was anable to upload the files with the original extention

[mackrackit]

Thanks Al, I like the simplicity of the solution.

[Charlie]

I came to the conclusion that a better method is the MEDIAN and not the MEAN since the average
incorporate into the final result (the mean) also any anomalous reading due to spikes or other
electrical noisy phenomenon could occur during the sampling.

Here the contradiction, we make the average to get rid of anomalous reading but we just incorporate
them, in other words, the mean value is a way to dilute anomalous reading while, the MEDIAN is a mean to
get rid of them.

I have not looked at your implementation, but I've been mulling over your logic, and I don't think I can agree.

First, it sounds like you are really interested in the mode, rather than the median. i.e. take a number of readings and report the one that comes up most often.

But I guess I have a problem with the fundamental assertion that using average is somehow flawed. The value fluctuates because of quantization distortion - basically the real value falls between the ADC steps. You can actually get additional resolution by "oversampling" and averaging the results. There's lots of detail on the issue (or opportunity depending on you point of view), and an excellent solution here http://www.darreltaylor.com/DT_Analog/ It's discussed at length on the MicroChip site too.

The bottom line, though, is that the fluctuation is not noise - it's additional information that you should WANT to incorporate, and hence the almost universal use of average.

Great discussion topic - too often we do things without thinking about why we do them.

[aratti]

Dave thank you for your positive comment.

Charlie if you read more carefully my introduction, you will note that I am refering to UNWANTED NOISE, some thing you want to remove from your measurement.
The algorithm posted will return the true MEDIAN of 15 ADC reading, nothing to do with the mode, even if in some cases MEDIAN; MODE and MEAN can coincide.

If you are interested in oversampling than the algorithm posted is not sutable since it will remove all the noise you are after, and very likely the sampling time will not respect the Nyquist theorem requirement

Thank you for your comment

Cheers

Al

[Charlie]

Hi again Al,

Unwanted noise should be eliminated long before it gets quantized. With a clean signal, using the MEAN does not seem more valid than using the AVERAGE to me - quite the opposite.
This is not a statistical problem, but rather an encoder accuracy issue.

I'm also missing something, I guess, because I'm also not sure how your reference to the outdated Nyquist sample rate is relevant - Nyquist / Shannon's famous algorithm relates to information theory and the proposal that you need to sample at least twice as fast as the highest frequency in the signal you are looking at. I say outdated, because far more efficient methods are in use today to cram more information into less bandwidth. When we started digital telephony, it took 8 KHz to carry 4 KHz bandwidth. (Nyquist rate) Now my cell phone / SIP phone both take considerably less than 1/2 of that thanks to new coding methods.

At any rate you do put forward an interesting idea and it may be better in a noisy environment. It would be great to have a large collection of use cases executed to determine which is best in which situation. Still, I think I'll rely on an analog filter, oversampling, averaging, and then drop bits if I'm worried about least significant digit stability.

[Ioannis]

I agree that the design engineer should do the best to eliminate every possible source of noise by good design and filtering.

This does not make sure though that noise, spikes or whatever cannot enter the signal path.

So,after sampling, oversampling or whatever, a digital method of cleaning the samples is desired.

But, for every job there is the appropriate tool. You have now more tools to select.

Select the best that suits your project.

Thanks Al for sharring one more idea.

Ioannis

[Darrel Taylor]

Comments on the subject and on the stability of the MEDIAN filter are welcome.

It sounded like an interesting idea, so I thought I'd see what it could do.

I created a circuit with 2.50 volts going to AN0 of a 12F683.
The A/D converter reads ~511 with 10-bit results
Using another 12F683, I generated Pink Noise using the SOUND command Note 255, and injected ~40mV of that noise into the analog signal.
I realize that's more noise than you should ever have on an analog input. But if you want to test a filter ... you gotta have noise.

The "Median Filter" routine then takes 14 samples (even though it says it takes 15), which looks like this ...
512 510 512 510 509 509 509 509 509 509 509 513 515 512

It then does a bubble sort on the array and takes the 8th value as the result, in this case 510.
509 509 509 509 509 509 509 510 510 512 512 512 513 515

This method is very similar to what Melanie posted several years ago, with the exception that after she did the bubble sort, she threw out the high and low readings, then averaged what was left.
This "New Software Filter" skips the step of averaging the middle results and just takes the value in the center. But there's no guarantee that the value in the center of the array is the value in the center of the noise.

Another set of samples from the test above looked like this ...

511 514 516 516 516 516 515 515 515 515 513 511 510 508
508 510 511 511 513 514 515 515 515 515 516 516 516 516

The value in the middle ended up much higher simply because the set of samples had more high values in it.
_____________________________ ___________________________ _______________________________

In this graph, I'm taking samples 5 different ways.
1. Raw A/D readings with no filter
2. The "Median" filter described in this thread
3. Melanie's Bubble Sort and Average
4. Standard averaging, summing the values then dividing by the sample count
5. And finally using 14-bit oversampling which the "Median Filter" is supposed to better than (according to this thread).

The results were output via RS232 and plotted in Excel.

The first thing I noticed was that the "Median" results are much higher.
This turned out to be an incorrect voltage calculation in the "AL_ADC_12F_8_Mhz.pbp" program.
I left it like that since it separates the plot from the other data making it easier to see.

The second thing you'll notice, is that the Median routines add no "filtering" to the signal.
Well, very little filtering. The noise is only slightly less than the original waveform.

The third thing to notice is that Oversampling (Red line) is the clear winner.



Now in all fairness, the number of samples in the dataset can make a huge difference when filtering.
So for the second plot I increased the sample size to the largest WORD array I could fit in the 12F683 (36 words).
All of the method's sample size were increased, except for Oversampling which by definition has more samples anyway.
I did add some rounding of the least significant digit of the Oversampling since its resolution is better than 1mV.

All methods have better results, but they are still all over the place as far as accuracy.




And one final plot where the only thing I changed was Oversampling to 15-bits.
Can you possibly get a flatter red line?



I did notice that if there is NO NOISE in the signal, the Median routine does give a "Rock Solid" reading, in other words .. it doesn't change as much.
But that's not the point of a filter. And the other methods work just as well or better with NO NOISE.
A filter needs to remove noise to be called a "Filter".

So I'm sorry Al, I don't think the "New software filter for adc reading" has any benefits over even the worst of the other filtering methods.

[Ioannis]

Excellent analysis Darrel!

It would be interesting (at least to me) on your same setup to test the moving or running average too.

Also, how did you collect the samples? From Excel side I mean. By text file?

Ioannis

[aratti]

Thank you Darrel for your interesting post.
I need to correct your statement, that the routine takes only 14 samples instead of 15. I have re-checked and found no error. From raw data 0 to raw data 14 makes a total of 15 samples! If I am in error please let me know where the mistake is, since the true MEDIAN can be taken only from an odd number of samples.

In your test you have injected audio signal into a DC line, well this is rather different from a non cyclic random noise that you can see in harsh environment.

Performing DC reading, I hate to see the decimal part of the display behaving like a dancer, hence, I consider the MEDIAN approach interesting, since it make the reading rock stable.

Let see other reports, you will be counted as against. Thank you again for your preciuos contribution.

Cheers.

Al.

[aratti]

Adding part to my previous post.

Ok found the error the sorting part of the routine was stating:

If Index < Sample then goto SDH_Loop

Now since Sample = 14 then the statement should be:

If Index < (Sample + 1) then goto SDH_Loop

in order to sort the whole series of data.

Thank you Darrel for pointing it out

Cheers

Al.

[Darrel Taylor]

Well no.

The sorting routine was OK.
It needs that statement because the sorting uses Raw_Data[Index + 1]
If you make your suggested change, it will overrun the array.

The problem is in the sampling portion.

If Index < sample then goto Read_Again

Should be ...

If Index <= sample then goto Read_Again
____________________ _______________________ ____________________

Ioannis: Thanks!

If you have a particular running average routine (I've seen several), I can throw it in the test.

And the data was formatted like this ... using a total of 300 samples

Ready
2.502, 2.534, 2.497, 2.497, 2.500
2.507, 2.529, 2.497, 2.497, 2.500
2.487, 2.529, 2.492, 2.497, 2.500
2.502, 2.534, 2.502, 2.502, 2.500
2.502, 2.534, 2.502, 2.502, 2.500
2.507, 2.529, 2.492, 2.492, 2.500
2.517, 2.529, 2.497, 2.497, 2.500
2.492, 2.529, 2.497, 2.497, 2.500
2.492, 2.534, 2.502, 2.502, 2.500
2.507, 2.534, 2.502, 2.502, 2.500
2.517, 2.529, 2.497, 2.497, 2.500
2.497, 2.529, 2.492, 2.497, 2.500
2.502, 2.529, 2.497, 2.497, 2.500
2.512, 2.534, 2.497, 2.497, 2.500
2.497, 2.534, 2.502, 2.502, 2.500

The order is ... Raw, Median, Melanie, Average, Oversample.

I cut&paste the data from the terminal program into Excel and used the "Text to Columns" command.

We could create another thread and test out various averaging routines.
Figure out the best sample size and measurement techniques. But I doubt anything could ever be better than Oversampling.

[Ioannis]

Hi Darrel.I too agree that nothing may beat the oversampling. It is obvious.

The average I have in mind would be average=average*15/16+sample. Not many, but not too little and /16 can be done fast with shifts.

Cut and paste? Well, I was thinking too complex I guess...

Ioannis

[Jerson]

The average I have in mind would be average=average*15/16+sample. Not many, but not too little and /16 can be done fast with shifts.
Ioannis

Ioannis, I think you must have meant

Average = (Average*15 + sample)/16

[Ioannis]

Yes Jerson. That was it. It was too late here, so...

Thanks for correcting.

Ioannis

[aratti]

Darrel thank you for pointing out where the error was. So obvious that I could not see it!

I have an improvment (mark2) and I will be glad to post it, if you could test it under the same condition as for mark1.

I am asking that since you have suggested to create another thread for testing the various statistical manipolation of the raw data. So please let me know if you are going to create this thread or if I can post it here.

Thank you again for your work.

Cheers

Al.

[Darrel Taylor]

Sure, post it here. This thread is about your routines.

My stuff is at the office, so I'll run it and the moving average Monday (if I don't get snowed in).

I'm still thinking about starting the other thread, but it needs to use a bigger chip, have different noise types and frequencies, use pre-sampled datasets so they're each working with the same samples, maybe some software and circuits to automate the process, etc. .... it may be awhile.

[Charlie]

Ioannis, I think you must have meant

Average = (Average*15 + sample)/16

One of the issues with so many samples is that you may run into the maximum size of numbers. You may need to play with the order of operations or break it into several steps to make sure you don't accidentally wrap. You also likely will be dropping significant digits which may or may not be an issue.

[aratti]

Hi Darrel here the version 2. In this version, I am using the mean instead of single data and then I select the median. Let see how it perform.

Charlie:

With a 12 bits ADC there is the possibility of overflow, with a 10 bits ADC you are ways within the word size, even if you sample 1023 which is the maximum value.

Cheers

Al.

[Darrel Taylor]

Sorry for the delay, I was getting unexpected results from the moving average and I wanted to make sure it wasn't something I was doing wrong.
But I 'll get to that in a minute.

Al, here's the results of "Mark2" under the same conditions as before.
I haven't changed the voltage calculation for it, so it's still reading higher than normal.
It looks a little different, but still jumps +/- 2 counts fairly frequently.



Now notice the green line in the above plot.
What, you don't see it ... me neither.
It turns out that the Running average is so slow to respond that it's reading is not even on the chart.
Zooming out a bit shows this ...



This was the formula used ...

RunAverage  VAR WORD

ADCIN  0, ADval
RunAverage = (RunAverage*15 + ADval)/16

I let it run for over an hour and it was getting close, but it still hadn't reached 2.5 volts.
I don't think I'd want to use that formula.

[Ioannis]

Thanks Darrel. Is that on the X-axis ... time? I guess it is and the time it takes is forever.

OK, if we use 3/4 will sure be faster, but how accurate?

On an experiment with touch sensors I can confirm that it could not get close to the real sample value though...

Ioannis

[Charlie]

Hey Darrel - I'm sure you don't want to make a career out of this mini-project, but can you explain that result? It's not obvious (to me at least) why it takes so long and the green line never reaches ADval. Playing with this in a spreadsheet, it can take between 100 & 150 samples to get there, but you have WAY more than that, and it's not even close! Do you think this the accumulated result of dropping LSBs, and effectively always rounding down?

Also, to address the ramp time, when I've done something similar in practice, the first time through the loop I make RunAverage = ADval to effectively eliminate the ramp, and at least start in the right ballpark. (I'm sure you typically do as well). Wouldn't it be more fair to add that curve instead (or as well)?

[Ioannis]

I think with the running average will never reach the final ADC value because of rounding errors.

E.g. for 10 bit ADC and say a RunAverage value of 1008, on next iteration 0.9375 should be added, but since PBP does only integer numbers, the RunAverage will sit at 1008 for ever. This would be the final value of the filter.

I suppose this kind of running average routine is for floating point maths only.

Ioannis

[aratti]

Thank you Darrel for such a quick answer. I will keep thinking what else I could do for further improvment.

As far as the running average is concerned, a way to have a faster answer could be:

ADCIN chan,ADVal

If RAv = 0 then
RAv = ADVal
ELSE
RAv = (RAv*15 + ADVal)/16
ENDIF

In this way you load the variable with the raw value and then you smooth it. Just thinking!

Cheers

Al.

[Jerson]

Another running average that may be useful to all. This one will work even if you do not have floating point. However, you need to be able to handle large numbers or size the buffer appropriately that it doesn't overflow 'sum' while summing. Of course, you need way more ram than the Avg = (Avg*(n-1)+adcval)/n formula which is good for Ram Constrained systems.

Pseudo code.

NumOfSamples   equ   16

AdcVal             Word                               ' adc reading is placed here
Samples           Word [NumOfSamples]     ' rolling buffer having NumOfSamples of AdcReading (must retain value between calls to R_Average)
Index              var     byte                       ' next position where the sample will be put (must retain value between calls to R_Average)
Sum                var     long                      ' gross sum of all the samples (must retain value between calls to R_Average)

RA_Init:  ' initialize the running average
      Index = 0
      Sum  =0
      return

' enter with A
R_Average:
     sum = sum - Samples[Index]                    ' remove the oldest sample from the sum
     sum = sum + AdcVal                                ' add in the latest sample             to sum
     Samples[Index] = AdcVal                          ' and save it in the buffer too

     Index = Index+1                                     ' move the index
     If Index > NumOfSamples then Index = 0  ' wrap around if needed

     return sum / NumOfSamples                     ' the running average

Caller has to initiate with RA_Init before using the R_Average routine. I hope you will excuse my pseudo code since I am a bit rusty on my PBP at this moment.

[Darrel Taylor]

I agree!
That Avg = (Avg*15 + ADval)/16 running average is only going to work with floating point math.
It's surprising how many times we've seen it used around here in an integer only language.

Charlie, I don't think setting it to ADval first will help, because it is likely to still be a short distance from the real value.
And it's when it's close to the real value that it doesn't move anymore, so it would never get any closer than that first reading.

[Ioannis]

Ehh, hmm, I think it is Al, Darrel

It is indeed surprising how many have used it, but who thought that it needed floating? Also Microchip use it in various AN like the cap-sense routines but of course C does support floats.

Anyway, Fast Average and Oversampling I think is the best Darrel-Soft as usual.

Ioannis

[Charlie]

And it's when it's close to the real value then it doesn't move anymore.

Well, that's sort of the objective of filtering, right?

But I think I'm getting my head around this - it's an issue of the SIZE of change needed to make an impact on the reading. So if there is no averaging, a change is recorded when the reading changes by a whole number value of 1. If you are averaging 4 readings, then the change must be at least a whole number value of 4 before it gets included. With 16 samples, the reading has to change by at least 16 before it registers, and so on. In effect you are not integrating small changes, you are tossing them out... which is sort of where this thread started.

I think I'd better revisit several projects where I've used a rolling average, although I don't believe I've ever used more than 4 samples.

[Ioannis]

You need to use floats whatever the sample are.

Ioannis

[HenrikOlsson]

Interesting thread!
A while back (2+ years as it turns out) I did some work on a "general purpose" low pass filter routine which, like many projects, never really made it to the finish line.
I based it on one of Tracy Allens filter routines and it's basically that 15/16 type filter disussed in later posts of the thread - with some adjustments. With my version you're able to select one out of 9 different filter constants and it handles negative values (two's complement). Seeing the dissusions here on that type of filter not converging properly, I tool a look at the old plots I did while testing and the output of my filter does seem to converge to the actual input value:



It's a bit hard to see in the plot but all "traces" does end up at the actual input value when given enough time, here's a detailed view of the first step of signal:



Finally here's a plot that shows an "ideal" sinewave and that same sine with another sine (f=10x) on top. Then it shows what the filter output looks like at a couple of different filter constants:



I should probably note that all of the above tests was performed with "simulated data" ie the waveform fed to the filter was stored in the PIC so no ADC was involved. I don't think that actually makes any difference though.

If there's any interest in it I can clean it up a bit and post it. It will not be the smallest and most likely not the fastest but it may have other benefits.

/Henrik.

[Demon]

Moved posts pertaining to my jitter issue over to here:

https://www.picbasic.co.uk/forum/sho...ng-ADC-capture