New software filter for adc reading

[aratti]

To overcome the ADC reading instability, several type of software filters have been devised and
among these the average, or mean value, is the simpler and most widely used.

I came to the conclusion that a better method is the MEDIAN and not the MEAN since the average
incorporate into the final result (the mean) also any anomalous reading due to spikes or other
electrical noisy phenomenon could occur during the sampling.

The MEDIAN which is the central value of the frequency distribution of our ADC reading, coincide
with the mean if the frequency distribution is normal, while the MEAN drift from the MEDIAN
all the time that anomalous reading are taken.

Here the contradiction, we make the average to get rid of anomalous reading but we just incorporate
them, in other words, the mean value is a way to dilute anomalous reading while, the MEDIAN is a mean to
get rid of them.

Rock stable results have been obtained extracting the MEDIAN out of 15 consecutive readings, with
the INCLUDE module "Al_ADC_Median.PBP", which I am glad to post here and share it with the community.

The use is rather simple, just assign the ADC channel to the channel variable and call the module
with "GOSUB GetADC" the ADC_Value (word variable) returned, will contain the ADC MEDIAN of 15 reading.

I have also posted a working code using PIC 12F683 as an example on how to use the include module.

Place the "Al_ADC_Median.PBP in the PBP folder, add the include to your code and you are done.

I hope the folks working with analog can find it useful. Comments on the subject and on the stability
of the MEDIAN filter are welcome.

Cheer and All the best wishes for the new year.

Al.

PS. You need to rename the extention form .TXT to .PBP since I was anable to upload the files with the original extention

[mackrackit]

Thanks Al, I like the simplicity of the solution.

[Charlie]

I came to the conclusion that a better method is the MEDIAN and not the MEAN since the average
incorporate into the final result (the mean) also any anomalous reading due to spikes or other
electrical noisy phenomenon could occur during the sampling.

Here the contradiction, we make the average to get rid of anomalous reading but we just incorporate
them, in other words, the mean value is a way to dilute anomalous reading while, the MEDIAN is a mean to
get rid of them.

I have not looked at your implementation, but I've been mulling over your logic, and I don't think I can agree.

First, it sounds like you are really interested in the mode, rather than the median. i.e. take a number of readings and report the one that comes up most often.

But I guess I have a problem with the fundamental assertion that using average is somehow flawed. The value fluctuates because of quantization distortion - basically the real value falls between the ADC steps. You can actually get additional resolution by "oversampling" and averaging the results. There's lots of detail on the issue (or opportunity depending on you point of view), and an excellent solution here http://www.darreltaylor.com/DT_Analog/ It's discussed at length on the MicroChip site too.

The bottom line, though, is that the fluctuation is not noise - it's additional information that you should WANT to incorporate, and hence the almost universal use of average.

Great discussion topic - too often we do things without thinking about why we do them.

[aratti]

Dave thank you for your positive comment.

Charlie if you read more carefully my introduction, you will note that I am refering to UNWANTED NOISE, some thing you want to remove from your measurement.
The algorithm posted will return the true MEDIAN of 15 ADC reading, nothing to do with the mode, even if in some cases MEDIAN; MODE and MEAN can coincide.

If you are interested in oversampling than the algorithm posted is not sutable since it will remove all the noise you are after, and very likely the sampling time will not respect the Nyquist theorem requirement

Thank you for your comment

Cheers

Al

[Charlie]

Hi again Al,

Unwanted noise should be eliminated long before it gets quantized. With a clean signal, using the MEAN does not seem more valid than using the AVERAGE to me - quite the opposite.
This is not a statistical problem, but rather an encoder accuracy issue.

I'm also missing something, I guess, because I'm also not sure how your reference to the outdated Nyquist sample rate is relevant - Nyquist / Shannon's famous algorithm relates to information theory and the proposal that you need to sample at least twice as fast as the highest frequency in the signal you are looking at. I say outdated, because far more efficient methods are in use today to cram more information into less bandwidth. When we started digital telephony, it took 8 KHz to carry 4 KHz bandwidth. (Nyquist rate) Now my cell phone / SIP phone both take considerably less than 1/2 of that thanks to new coding methods.

At any rate you do put forward an interesting idea and it may be better in a noisy environment. It would be great to have a large collection of use cases executed to determine which is best in which situation. Still, I think I'll rely on an analog filter, oversampling, averaging, and then drop bits if I'm worried about least significant digit stability.

[Ioannis]

I agree that the design engineer should do the best to eliminate every possible source of noise by good design and filtering.

This does not make sure though that noise, spikes or whatever cannot enter the signal path.

So,after sampling, oversampling or whatever, a digital method of cleaning the samples is desired.

But, for every job there is the appropriate tool. You have now more tools to select.

Select the best that suits your project.

Thanks Al for sharring one more idea.

Ioannis

[Darrel Taylor]

Comments on the subject and on the stability of the MEDIAN filter are welcome.

It sounded like an interesting idea, so I thought I'd see what it could do.

I created a circuit with 2.50 volts going to AN0 of a 12F683.
The A/D converter reads ~511 with 10-bit results
Using another 12F683, I generated Pink Noise using the SOUND command Note 255, and injected ~40mV of that noise into the analog signal.
I realize that's more noise than you should ever have on an analog input. But if you want to test a filter ... you gotta have noise.

The "Median Filter" routine then takes 14 samples (even though it says it takes 15), which looks like this ...
512 510 512 510 509 509 509 509 509 509 509 513 515 512

It then does a bubble sort on the array and takes the 8th value as the result, in this case 510.
509 509 509 509 509 509 509 510 510 512 512 512 513 515

This method is very similar to what Melanie posted several years ago, with the exception that after she did the bubble sort, she threw out the high and low readings, then averaged what was left.
This "New Software Filter" skips the step of averaging the middle results and just takes the value in the center. But there's no guarantee that the value in the center of the array is the value in the center of the noise.

Another set of samples from the test above looked like this ...

511 514 516 516 516 516 515 515 515 515 513 511 510 508
508 510 511 511 513 514 515 515 515 515 516 516 516 516

The value in the middle ended up much higher simply because the set of samples had more high values in it.
_____________________________ ___________________________ _______________________________

In this graph, I'm taking samples 5 different ways.
1. Raw A/D readings with no filter
2. The "Median" filter described in this thread
3. Melanie's Bubble Sort and Average
4. Standard averaging, summing the values then dividing by the sample count
5. And finally using 14-bit oversampling which the "Median Filter" is supposed to better than (according to this thread).

The results were output via RS232 and plotted in Excel.

The first thing I noticed was that the "Median" results are much higher.
This turned out to be an incorrect voltage calculation in the "AL_ADC_12F_8_Mhz.pbp" program.
I left it like that since it separates the plot from the other data making it easier to see.

The second thing you'll notice, is that the Median routines add no "filtering" to the signal.
Well, very little filtering. The noise is only slightly less than the original waveform.

The third thing to notice is that Oversampling (Red line) is the clear winner.



Now in all fairness, the number of samples in the dataset can make a huge difference when filtering.
So for the second plot I increased the sample size to the largest WORD array I could fit in the 12F683 (36 words).
All of the method's sample size were increased, except for Oversampling which by definition has more samples anyway.
I did add some rounding of the least significant digit of the Oversampling since its resolution is better than 1mV.

All methods have better results, but they are still all over the place as far as accuracy.




And one final plot where the only thing I changed was Oversampling to 15-bits.
Can you possibly get a flatter red line?



I did notice that if there is NO NOISE in the signal, the Median routine does give a "Rock Solid" reading, in other words .. it doesn't change as much.
But that's not the point of a filter. And the other methods work just as well or better with NO NOISE.
A filter needs to remove noise to be called a "Filter".

So I'm sorry Al, I don't think the "New software filter for adc reading" has any benefits over even the worst of the other filtering methods.