New software filter for adc reading

[Ioannis]

I agree that the design engineer should do the best to eliminate every possible source of noise by good design and filtering.

This does not make sure though that noise, spikes or whatever cannot enter the signal path.

So,after sampling, oversampling or whatever, a digital method of cleaning the samples is desired.

But, for every job there is the appropriate tool. You have now more tools to select.

Select the best that suits your project.

Thanks Al for sharring one more idea.

Ioannis

[Darrel Taylor]

Comments on the subject and on the stability of the MEDIAN filter are welcome.

It sounded like an interesting idea, so I thought I'd see what it could do.

I created a circuit with 2.50 volts going to AN0 of a 12F683.
The A/D converter reads ~511 with 10-bit results
Using another 12F683, I generated Pink Noise using the SOUND command Note 255, and injected ~40mV of that noise into the analog signal.
I realize that's more noise than you should ever have on an analog input. But if you want to test a filter ... you gotta have noise.

The "Median Filter" routine then takes 14 samples (even though it says it takes 15), which looks like this ...
512 510 512 510 509 509 509 509 509 509 509 513 515 512

It then does a bubble sort on the array and takes the 8th value as the result, in this case 510.
509 509 509 509 509 509 509 510 510 512 512 512 513 515

This method is very similar to what Melanie posted several years ago, with the exception that after she did the bubble sort, she threw out the high and low readings, then averaged what was left.
This "New Software Filter" skips the step of averaging the middle results and just takes the value in the center. But there's no guarantee that the value in the center of the array is the value in the center of the noise.

Another set of samples from the test above looked like this ...

511 514 516 516 516 516 515 515 515 515 513 511 510 508
508 510 511 511 513 514 515 515 515 515 516 516 516 516

The value in the middle ended up much higher simply because the set of samples had more high values in it.
_____________________________ ___________________________ _______________________________

In this graph, I'm taking samples 5 different ways.
1. Raw A/D readings with no filter
2. The "Median" filter described in this thread
3. Melanie's Bubble Sort and Average
4. Standard averaging, summing the values then dividing by the sample count
5. And finally using 14-bit oversampling which the "Median Filter" is supposed to better than (according to this thread).

The results were output via RS232 and plotted in Excel.

The first thing I noticed was that the "Median" results are much higher.
This turned out to be an incorrect voltage calculation in the "AL_ADC_12F_8_Mhz.pbp" program.
I left it like that since it separates the plot from the other data making it easier to see.

The second thing you'll notice, is that the Median routines add no "filtering" to the signal.
Well, very little filtering. The noise is only slightly less than the original waveform.

The third thing to notice is that Oversampling (Red line) is the clear winner.



Now in all fairness, the number of samples in the dataset can make a huge difference when filtering.
So for the second plot I increased the sample size to the largest WORD array I could fit in the 12F683 (36 words).
All of the method's sample size were increased, except for Oversampling which by definition has more samples anyway.
I did add some rounding of the least significant digit of the Oversampling since its resolution is better than 1mV.

All methods have better results, but they are still all over the place as far as accuracy.




And one final plot where the only thing I changed was Oversampling to 15-bits.
Can you possibly get a flatter red line?



I did notice that if there is NO NOISE in the signal, the Median routine does give a "Rock Solid" reading, in other words .. it doesn't change as much.
But that's not the point of a filter. And the other methods work just as well or better with NO NOISE.
A filter needs to remove noise to be called a "Filter".

So I'm sorry Al, I don't think the "New software filter for adc reading" has any benefits over even the worst of the other filtering methods.

[Ioannis]

Excellent analysis Darrel!

It would be interesting (at least to me) on your same setup to test the moving or running average too.

Also, how did you collect the samples? From Excel side I mean. By text file?

Ioannis

[aratti]

Thank you Darrel for your interesting post.
I need to correct your statement, that the routine takes only 14 samples instead of 15. I have re-checked and found no error. From raw data 0 to raw data 14 makes a total of 15 samples! If I am in error please let me know where the mistake is, since the true MEDIAN can be taken only from an odd number of samples.

In your test you have injected audio signal into a DC line, well this is rather different from a non cyclic random noise that you can see in harsh environment.

Performing DC reading, I hate to see the decimal part of the display behaving like a dancer, hence, I consider the MEDIAN approach interesting, since it make the reading rock stable.

Let see other reports, you will be counted as against. Thank you again for your preciuos contribution.

Cheers.

Al.

[aratti]

Adding part to my previous post.

Ok found the error the sorting part of the routine was stating:

If Index < Sample then goto SDH_Loop

Now since Sample = 14 then the statement should be:

If Index < (Sample + 1) then goto SDH_Loop

in order to sort the whole series of data.

Thank you Darrel for pointing it out

Cheers

Al.

[Darrel Taylor]

Well no.

The sorting routine was OK.
It needs that statement because the sorting uses Raw_Data[Index + 1]
If you make your suggested change, it will overrun the array.

The problem is in the sampling portion.

If Index < sample then goto Read_Again

Should be ...

If Index <= sample then goto Read_Again
____________________ _______________________ ____________________

Ioannis: Thanks!

If you have a particular running average routine (I've seen several), I can throw it in the test.

And the data was formatted like this ... using a total of 300 samples

Ready
2.502, 2.534, 2.497, 2.497, 2.500
2.507, 2.529, 2.497, 2.497, 2.500
2.487, 2.529, 2.492, 2.497, 2.500
2.502, 2.534, 2.502, 2.502, 2.500
2.502, 2.534, 2.502, 2.502, 2.500
2.507, 2.529, 2.492, 2.492, 2.500
2.517, 2.529, 2.497, 2.497, 2.500
2.492, 2.529, 2.497, 2.497, 2.500
2.492, 2.534, 2.502, 2.502, 2.500
2.507, 2.534, 2.502, 2.502, 2.500
2.517, 2.529, 2.497, 2.497, 2.500
2.497, 2.529, 2.492, 2.497, 2.500
2.502, 2.529, 2.497, 2.497, 2.500
2.512, 2.534, 2.497, 2.497, 2.500
2.497, 2.534, 2.502, 2.502, 2.500

The order is ... Raw, Median, Melanie, Average, Oversample.

I cut&paste the data from the terminal program into Excel and used the "Text to Columns" command.

We could create another thread and test out various averaging routines.
Figure out the best sample size and measurement techniques. But I doubt anything could ever be better than Oversampling.

[Ioannis]

Hi Darrel.I too agree that nothing may beat the oversampling. It is obvious.

The average I have in mind would be average=average*15/16+sample. Not many, but not too little and /16 can be done fast with shifts.

Cut and paste? Well, I was thinking too complex I guess...

Ioannis