byte compression

[Norbert]

First post from a newbie!

I'm working on a logger (16f873a). To save external EEPROM space, the 10 bytes data from RTC, sensors etc are compressed to an 8-byte array that is sent (byte by byte) to the EEPROM at the end of each logged event. The byte compression is done by culling the unused bits in the data bytes and combining the essential data from 2 bytes into one. The following code snippet works, but there must be a more elegant and economical way of doing it. Any suggestions?

Thanks!
Month var byte 'formatted data from RTC, only 4 bits neccessary
Day var byte 'formatted data from RTC, only 5 bits neccesary
Hours var byte 'formatted data from RTC
Mins var byte 'formatted data from RTC
Secs var byte 'formatted data from RTC
logdata var byte [8] '8 byte long array, containing data from RTC, incident direction,
' 'incident duration, temperature and humidity.
This array is sent to the external EEPROM.
'
logdir var byte 'incident direction byte, only 2 bits neccessary
modir var byte 'temporary byte used to compress Month and Direction data to
' 'same byte in array to save eeprom space
daytra var byte ''temporary byte used to compress Day and part of Duration data
' 'to same byte in array to save eeprom space
transwrite var byte 'temporary byte for part of flittime data
flittime var word 'Event duration word, 11 bits neccessary
thermo var byte 'Temperature byte
humi var byte 'Humidity byte
'
logdata[1] = Hours 'the full 8 bits of each of these 3 data bytes is transferred
' 'to the corresponding byte in the array
logdata[2] = Mins
logdata[3] = Secs
'
modir.0=Month.0 'here we strip the bottom 4 bits from the Month byte and put them
' 'in the corresponding bits of the modir byte
modir.1=Month.1
modir.2=Month.2
modir.3=Month.3
'
daytra.0=Day.0 'and the same with the bottom 5 bits from the Day variable,
' 'going to the daytra byte
daytra.1=Day.1
daytra.2=Day.2
daytra.3=Day.3
daytra.4=Day.4
modir.4=logdir.0 'here we strip the bottom four bits from the logdir variable
' 'and put them in the top 4 bits of the modir byte
modir.5=logdir.1
modir.6=logdir.2
modir.7=logdir.3
'
logdata[4] = modir 'so this byte in the array contains both Month and Direction data
'
transwrite=flittime.lowbyte 'Since flittime is a word variable (with 11 bits of valid data)
' 'we need to distribute it over 2 bytes.The low byte goes to transwrite.
logdata[5] = transwrite
'
daytra.5=flittime.8 'The 3 unused bits of the daytra byte are filled with 3 more bits from
' 'the flittime variable.
daytra.6=flittime.9
daytra.7=flittime.10
'
logdata[0] = daytra 'Byte 0 of the logdata string contains both the 3 high bits of the transition
' 'time count and 5 bits Day data
logdata[6] = thermo 'puts Temperature data in byte 7 of the array
logdata[7] = humi 'puts humidity data in byte 8 of the array

[skimask]

Month var byte:Day var byte:Hours var byte:Mins var byte:Secs var byte
logdata var byte [8]:logdir var byte:modir var byte:daytra var byte
transwrite var byte:flittime var word:thermo var byte:humi var byte
logdata[1]=Hours:logdata[2]=Mins:logdata[3]=Secs

logdata[4]=(month & $f)+(logdir << 4):logdata[5]=flittime.lowbyte
logdata[0]=(day & $1f)+((flttime & $0700)>>3)
logdata[6]=thermo:logdata[7]=humi


No idea if this will work...it's just simple boolean logic and combining terms...

[SteveB]

Another idea, which can be implemented together with the bit-packing, depending on the exact specification/requirements of your application:

If your data is aquired and logged at consistant intervals (say every 20 sec) you can log the data as a group. Start by saving a time/date. Then every 10 (or whatever interval is approariate) events, save another time/date. Since you know the interval of the datapoints, and have the time stamps, the rest can be determined later in whatever application does the post processing of the information. You can reduce a lot of overhead this way.

If the events are not periodic, you could implement this as an offset from the start time. Start with the time/date, then each data event contains an offset from the original time/date. Depending on the expected max interval, this could save you a lot of overhead as well.

SteveB

[skimask]

If your data is aquired and logged at consistant intervals (say every 20 sec) you can log the data as a group. Start by saving a time/date. Then every 10 (or whatever interval is approariate) events, save another time/date. Since you know the interval of the datapoints, and have the time stamps, the rest can be determined later in whatever application does the post processing of the information. You can reduce a lot of overhead this way.

If the events are not periodic, you could implement this as an offset from the start time. Start with the time/date, then each data event contains an offset from the original time/date. Depending on the expected max interval, this could save you a lot of overhead as well.
SteveB

Just like the 'sliding window' method used in most compression methods...
(Wait a minute...was that example out in left field or what?)
Maybe something like the julian calender, but extended down to date/time offset from a certain start date/time, but then you'd get less than one day's worth of seconds in a word...

[SteveB]

Maybe something like the julian calender, but extended down to date/time offset from a certain start date/time, but then you'd get less than one day's worth of seconds in a word...

Yea, like a julian calander, except instead of restarting the count of days every year, you can restart the count of seconds every 18.2 hours (if you use a word). But, once an event occurs, you can restart the counter, and the new offset begins from that last offset. Sort of like restarting the Julian Calandar every time it rains in Del Rio! :D Or better yet, SNOWS!!

[Norbert]

Thanks for the suggestions, Skimask and SteveB.
I'll try your code, Skimask, and see if it works. Would it save any code space on the PIC (compared to my original code?)

The application logs data at very irregular intervals, from seconds to days. So SteveB's suggestion would require three bytes (or at least 21 bits anyway): two bytes for the seconds since the last incident, and 5 bits for the number of days since the last incident (and I think I would only count seconds up to 12 hours, then increment a half-day counter). A couple of bytes saved, but two different time data formats in the EEPROM: the initial time stamp, and then the incremented offsets, and post-processing required for the data.
I think I'll stick with time stamps on each event.

Norbert

[skimask]

Thanks for the suggestions, Skimask and SteveB.
I'll try your code, Skimask, and see if it works. Would it save any code space on the PIC (compared to my original code?)
I'm not sure if it'll save any code space, and it does make it a bit harder to read. About the only thing it will save for sure it space on the screen while you're looking at it. :)


I think I'll stick with time stamps on each event.
Norbert
There ya go. Sometimes it's a lot easier to stick with something a bit wasteful and make the hardware take up the slack (i.e. a larger eeprom or more of them)...