Where should I discuss SD/MMC FAT issues?

[JD123]

Hey there guys and gals, new poster here, long time tinker of PBP.

Where (what forum) should I post questions and issues revolving around SD/MMC FAT16 and FAT32? My project could use the memory card as a bit-bucket and then 232 the data out, but I'd rather have the cards PC readable.

For the first time ever, I've bit-banged an MMC card into submission using a 16F876A. I'm finally in and reading data after about 5 hours of chit (not including the hardware), but working with FAT's is giving me a migraine. I guess I don't really know what questions to ask right now as I'm still learning the FAT structure. I'm still trying to sort out the strait-line math 2^x AND (2^z(y+1) >> 7) ... etc. ... yadi, yadi, yadi... just to find the next location of the next math nightmare.

Just would like to know where would be the best place for this topic?

Thanks,

Gary

[BrianT]

Hi JD123,

I am also trying to use a MMC card as a large data store that MUST be PC readable. I bought some development hardware from Ron Diol at Compsys a year back but the project got pushed aside by the urgent driving out the important. Ron was very helpful and there are some good links there.

http://www.compsys1.com/workbench/On...c_project.html

http://elm-chan.org/docs/mmc/mmc_e.html


I am thinking along the lines of preformatting the SD/MMC card in FAT 16 or 32 and loading two files of known names on the card. The SPI mode does not require any CRC calculations (I think) and is plenty fast enough for my needs.

The PIC code would open one file and append the latest data. It would close that file and open the second file, write the same code a second time and close. The two file are so if the card is removed while a file is still open, the data should be recoverable from the other file. No doubt there are better ways.

I want to keep the PIC side code as lean as possible and I can always preformat the cards but the final data must be directly readable by a PC.


Please keep this going in any forum as I think it will be a pretty interesting thread as you get it all going.

Cheers
BrianT

[JD123]

Hi Brian,

This is where I'm at. I can format the MMC on the PC, make a singel file using the PC, find the directory listing for the file using the PIC, find the file using the PIC, read and modify the file on the PIC and read the results on the PC via the MMC card reader.

What I now need to do is modify the directory listing to reflect the changes made to the file, so that the PC can understand the file information correctly. For example, I originally made the file "TEST123.TXT" 10 bytes long with the string "1234567890" so that it was easy to search the MMC to locate the file. As I made changes to the file using the PIC, if I went over the original 10 byte length, the data gets recorded, but the PC only sees the original first 10 bytes.

I found the file size information in the directory listing at offset 28 and this falls into what I was expecting based on that I've read. Problem is that the file size information is a 4 byte Dword and FAT specs call for the 4 bytes to START at offset 28. In reality, I would have thought that the "10" for the file size would have been in offset 31 (LSByte of the 4 bytes). I'm trying to figure out where I place the additional 3 bytes of the file size information. I can't find anything to support that the file size should be written low byte first and that wouldn't make sense anyway. So, I'm thinking that there's something wrong with my understanding of where the bytes are stored.

Anyway, this is where I was last night. I was tickled-pink just to see that I could get the PIC to make changes to the MMC that the PC could read. One thing that became real clear last night was that I'll have no choice but to add RAM to my hardware. I've got to be able to work on a sector's individual bytes and that's only going to happen if I have it in RAM and then send it back to the MMC when I'm done tinkering with the sector. I happen to have some 64KBit I2C EEPROMS handy, so tonight I'll press one of them into service on the proto board. This alone will put the breaks on FAT work for a few days (I only get a few hours a day to 'play' with PICs) while I deal with the hardware and I2C code.

I'll post updates as they come. If anyone can help me understand the offset location and data style (MSB or LSB first) of the file size data I'd appreciate reading your thoughts.

Gary

Edit: Brian, I forgot to say thanks for the links. I've been to both before and the second one was very helpful in getting me started.
--

[skimask]

I found the file size information in the directory listing at offset 28 and this falls into what I was expecting based on that I've read. Problem is that the file size information is a 4 byte Dword and FAT specs call for the 4 bytes to START at offset 28. In reality, I would have thought that the "10" for the file size would have been in offset 31 (LSByte of the 4 bytes). I'm trying to figure out where I place the additional 3 bytes of the file size information. I can't find anything to support that the file size should be written low byte first and that wouldn't make sense anyway. So, I'm thinking that there's something wrong with my understanding of where the bytes are stored.

Isn't the file size in FAT32 stored at 2 different places? And both of them have to be identical? I forget...it's been awhile since I read the FAT32 documentation...

I'll have no choice but to add RAM to my hardware.

I built an MP3 player awhile back based off an 18F4620 reading from CF cards in IDE mode. Had enough ram to buffer 6 complete sectors with room left over for a load of variables.
Also, if that's not enough, look into the F-RAM options. Serial EEPROM just like a 24C256 type, but it's ram, read/write just as fast, not slow writes/fast reads like eeprom, and practically unlimited lifetime.

understand the offset location and data style (MSB or LSB first) of the file size data I'd appreciate reading your thoughts.

Keep track of your little and big ENDIAN formats according to the FAT32 documentation. I don't know how many times that sort of thing tripped me up bad...

EDIT: Just dug out the FAT32 doc file.
Is the file you're basing your FAT32 code from called 'fatgen103.doc'?
If so, that file has a few issues in itself, wrong places for some values, some values omitted, and so on...
gain, it's been awhile. I'll see if I can't dig up the specifics.

[JD123]

Hi Skimask,

Isn't the file size in FAT32 stored at 2 different places? And both of them have to be identical? I forget...it's been awhile since I read the FAT32 documentation...

Sorry, I forgot to say that I'm using FAT16 when I format the MMC. It's the native format that the MMC's came with. I don't know what version of FAT16 I'm using. I understand there at least several versions or revisions to the FAT16 system. The MMC's I'm using are 128MB in size.

...look into the F-RAM options.

Yes, for a final in-field application this would be the way to go. As it is, I'm cheap and have the EEPROM's on hand. My application is data recording at about 100 sectors a second. For writing the data it will be sent directly to the card. The EEPROM use in my ap would only be at the first and end of the recording events (about 10-15 minutes long) and this would be just to address the FAT issues.

I'm back off to some of my web haunts where I've been reading about FAT16. Maybe I'll find the answer by going back over these sites. I do have this odd recollection of there being some part of the FAT where data is written LSByte first. Maybe it was the file size?

Gary
--

[skimask]

Sorry, I forgot to say that I'm using FAT16 when I format the MMC.

That fat32 document has a load of info in it, and specifies where it differ's from fat16...not that it matters much

My application is data recording at about 100 sectors a second.

I was able to shove a bit over 600K/sec into the CF card on my player, but that's using the IDE mode, parallel data transfer. 600K was as fast as the PIC could go @ 40Mhz.
You'll be limited to the max SPI speed, and I think you'll be lucky to be able to handle 100 sectors/second (100 sectors = 512000 bytes = 5.12mbps) and still have any time leftover to do any sort of interrupt processing.

I do have this odd recollection of there being some part of the FAT where data is written LSByte first. Maybe it was the file size?

Do a search for fatgen103.doc and you'll see what I'm looking at.
And yes, offset 28 is the file size, and yes, it is LSB first...little endian...

[JD123]

YES! I came back here to say I found this, only to see you found it too. The size IS written LSByte first. I can't tell you how many places on the web fail to state this. This is what I just now found:

data length ( =4 bytes ) 

the minimum can be 00 00 00 00 - 0 and the maximum ff ff ff ff -
 4,294,967,295
gb, mb, kb, by

this entry will state the length of data to read in.
eg, AC 3B 96 00
flip these values: 00 96 3B AC
convert to decimal: 9,845,676 bytes or 9.38MB
folders have a value of: 00 00 00 00

Thanks guys for getting me out of this ditch. It's amazing how much time I waste on some of the 'simplest' problems.

skimask, thanks for letting me know what you were able to get for speed. I'm shooting for 100, 8 byte recordings per second, but if I can make 60 I'll be happy. The recordable rate will be selectable and I was picking 100 as a top-shelf goal.

Edit: skimask, sorry... I see what caused the issue about speed. I originally ment to say ONE sector per second, not 600. 600 is the byte rate, not the sector rate. I guess I really WAS pushing the limits, the way I originally stated it!!!

[skimask]

skimask, thanks for letting me know what you were able to get for speed. I'm shooting for 100, 8 byte recordings per second, but if I can make 60 I'll be happy. The recordable rate will be selectable and I was picking 100 as a top-shelf goal.

100 8 byte recordings per second? That's only 800 bytes... I said 600KB per second, as in, 614,400 bytes per second, 6,144,000 bits per second.
But again...it was a parallel operation.

Now are you talking 100 8 bytes recordings...
or 100 512 byte (sector) recordings

[JD123]

100 8 byte recordings per second? That's only 800 bytes... I said 600KB per second, as in, 614,400 bytes per second, 6,144,000 bits per second.
But again...it was a parallel operation.

Now are you talking 100 8 bytes recordings...
or 100 512 byte (sector) recordings

Yes, I messed up. See my edit.

That should be quite reachable. The real issue is reading the PWM outputs from the accelerometers and an airspeed pressure sensor. My co-worker on this project wants to read the PWM as filtered analog resolved from a 1ms PWM frame rate. I'm arguing that I can get everything done and more in the 16.66ms that we have at 60 recordings per second. Also, using the CCP and MUX'ing the inputs I can get rock solid data with a 5000 FS range. On a +/-10G accelerometer that will be .4mG resolution - well enough for what we are doing.

Gary
--

[JD123]

I'm not going to turn this into a thread of me talking to myself, but I wanted to say that I put a 24LC16 on the pic and I'm now reading the directory table into memory, working on it and writing it back to the MMC without any problems. The PC now accepts the modified files with the correct file information from the MMC.

Reading out of the MMC to the I2C takes ~3 seconds for a 512 sector. I'll be getting it faster by coding for loading the 16 byte buffer on the I2C.

[skimask]

The PC now accepts the modified files with the correct file information from the MMC.

So it was the file size issue then?

Reading out of the MMC to the I2C takes ~3 seconds for a 512 sector. I'll be getting it faster by coding for loading the 16 byte buffer on the I2C.

Post some code. I'm sure there's a way to make it faster... It only took my code about 7-ish seconds to erase a full 24LC256 (32Kbytes), while displaying the current address on an LCD.

[JD123]

I'll post the code if you promise not to laugh. When I'm in the R&D phase my goal is to understand, not to write pretty code. All I'm doing right now is building a reference for dealing with the MMC when I make a final application.

I'm at work (work has nothing to do with PICs) and the files are at home. No internet at home (yes, I live in a cave... next-door to Osama) so it will be a day or two before I get it up. Most likely I'll have the code faster by then as tonight I'll probably change it to write the page buffer on the I2C memory. I started with the recommended 10ms delay after a write and once things were stable, I moved it to 5ms. Next step will be polling for busy for the shortest delays.

[skimask]

I'll post the code if you promise not to laugh.

The difference between R&D code and pretty code is....who cares...

Most likely I'll have the code faster by then as tonight I'll probably change it to write the page buffer on the I2C memory. I started with the recommended 10ms delay after a write and once things were stable, I moved it to 5ms. Next step will be polling for busy for the shortest delays.

5ms is max. write cycle time for the 24LC16 (byte or page).
You start writing 16 byte pages, and it should speed up by a factor of about 16, with the same 5ms delay after the 16 byte packet is sent out.

[JD123]

Oh, and yes, the issue about the PC not 'seeing' the complete file was the file size information at offset 28 in the directory table. What you said about flipping the size to LSB first works well.

For giggles and grins I formatted the MMC as FAT32 and tried to work with the FAT. Let's just say... NOT! I'll stick with FAT16 as long as I can. Does Vista allow FAT16 and or format media cards in FAT16? Thought I heard somewhere that FAT16 support was dropped.

[Acetronics2]

Hi, JD ( Pffffff !!!!)

If you're interested ... I've such a project with a 18F452 ... but Written In MikroC ...
not so far from Basic.

I do not know why, but I think you'll be Ok ...

Alain

PM me a mailBox address if needed ...

[JD123]

Hi Alain,

Thanks for the offer. I'll take you up on that. My next step is getting some intelligent information about adding files and making sure things like overwrites and PC compatibility are handled correctly.

BTW, I recognize you from RCG.

Gary
[email protected]
--

[skimask]

For giggles and grins I formatted the MMC as FAT32 and tried to work with the FAT. Let's just say... NOT! I'll stick with FAT16 as long as I can. Does Vista allow FAT16 and or format media cards in FAT16? Thought I heard somewhere that FAT16 support was dropped.

In that document, it specifies the cluster count cutoff for determining if something is formatted in FAT12/16/32.
If FAT16 was dropped, I would infer that FAT12 would've been dropped, which means no more floppies at all.
If you read that document, FAT32 is easy to work with, just have to take heed of the differences. The thing that's a real pain is the long filename's that come with it. But if you ignore those and stick with 8.3, FAT32 is a piece of cake.

[BrianT]

http://www.mmca.org/compliance/buy_s...EDSD84-A43.pdf

Cheers
BrianT

[JD123]

http://www.mmca.org/compliance/buy_s...EDSD84-A43.pdf

Cheers
BrianT

Thanks Brian. I have a copy of that. It reads like the USA budget passed by Congress :0 I couldn't find any information about FAT in the doc, though it may be in there.

Last night I decided to see if I could find a file armed only with the file name. I was able to do it. I started to write a post that cuts to the chase in explaining how I find a file but decided if I was going to get up in the morning, I'd have to finish it later. I'll post it when it's finished. It will be a real time saver for the first-time MMC user.

In short, I only need to read a few items from the MBR, Partition Record and the Directory Table (I think that's all I was reading - it was late). It's not hard once I know what to look for and stay within the structure of the FAT.

[skimask]

Thanks Brian. I have a copy of that. It reads like the USA budget passed by Congress :0 I couldn't find any information about FAT in the doc, though it may be in there.

It's only an MMC document.
I forget, do you have the FAT spec's document? (fatgen103?)

[JD123]

Skimask, yes. I have the one you were talking about and about 10 other pages found on the net. Together it's concise.

Thanks again for the LSB information. As I worked last night, I come to find out that all Word and Dword data that points to an address or its offset are written this way in FAT. I never have used the SWAP function of PBP before, but it's coming in very handy right now.

{4 data bytes, LSB first} D,C,B,A

swap A, D
swap B, C

{4 data bytes, MSB first} A,B,C,D

I understand this is just a work-around for the fact that "I" work with Word & Dword addresses as MSB first. I'm stuck in my ways... what can I say!

[JD123]

Oh, and for what it's worth, I just ordered some FRAM chips from Mouser. That should speed up sector reading a bit. I got 64K's so I can have up to 16 sectors or 4 clusters in RAM at one time. Since it will be so fast (up to 1mhz), I might use it to hold other data too. Having bigger and faster memory is like buying a Ford Excursion after owning a VW Rabbit.

[JD123]

Okay, the F-RAM chip is in place and working. Taking all delays out of the code I can get a sector (517 bytes) of the MMC onto the F-RAM in about 13.5ms (MMC > PIC > I2C). That's much faster than the old I2C. It's nice not having to stop and write the buffer page!

I'm still looking for a faster way to move the I2C data line, so I'm writing my own I2C handler routines. My whole program cycle has to fit into 16.67ms. It doesn't look like I'll be able to load the I2C with sample data then move it to the MMC in this window unless I can get the program cycle, minus the data captures, down to about 6ms. It's going to be close.

[skimask]

I'm still looking for a faster way to move the I2C data line, so I'm writing my own I2C handler routines. My whole program cycle has to fit into 16.67ms. It doesn't look like I'll be able to load the I2C with sample data then move it to the MMC in this window unless I can get the program cycle, minus the data captures, down to about 6ms.

What speed is your osc running at?
I know you can get 500-ish bytes inside of a few ms...
Show some code!

[JD123]

What speed is your osc running at?
I know you can get 500-ish bytes inside of a few ms...
Show some code!

Codes at home. The proto board is running at 16mhz and the final version will run at 20mhz.

I know what's slowing it down. In my rush to write the code (I wanted to see if this F-RAM really was a drop-in replacement for the 24C64's), I simply re-addressed the I2C for every write and this is sending way too many bits per write. It's kicking out 48 bits per byte write yet my I2C handler will send out 9 bits per byte (after a one-time command/address string) and at about twice the clocking speed. That will bring the data transfer rate up to about 10 times faster overall.

I was about to go ahead and write a variable string of say 16 bytes per I2C_WRITE to speed things up, but decided not to waste my time, knowing I'd sooner or later bit-bang the I2C for the fastest results. I'm writing the I2C handler to run at the 20mhz speeds, so what I get at 16mhz will be off of top possible speeds by 20%.

I should get the simple use of my I2C handler finished tonight. It would have been done last night, but I went to bed early. I've been puting in some long days the past few weeks thanks to this project. I'll see if I can remember to bring some code to work to post. I think it's about 500 lines now (text editor lines, not instruction lines) but should be easy to follow because I tab, space and sub block in logical places, except when I'm debugging fast. The code is still full of unnecessary debug lines.

[JD123]

This is an example of my code style. It's not from this project.

'*************** MODE 4 ****************

IF MODE = 4 THEN
    
    TIME = TIME - 5

    IF TIME = 1005 THEN 
        FOR T = 1 TO 50
            GOSUB SENDOUT
            GOSUB DELAY
        NEXT T
    ENDIF

    IF TIME = 1995 THEN 
        FOR T = 1 TO 50
            GOSUB SENDOUT
            GOSUB DELAY
        NEXT T
    ENDIF

    IF TIME < 900 THEN 
        TIME = 900
    ENDIF
    RETURN
 ENDIF

[JD123]

Well, I half-ass got the I2C running last night. I knew I'd have to play with the NOP's to get the fastest speeds, but ran out of time. I spent over 2 hours debugging the I2C only to find that I was clocking out using:

SDA and SCL are port pins

SDA = 1
@ NOP
@ NOP
@ NOP
@ NOP
@ NOP
SCL = 0
@ NOP
etc..

I spent so much time coding SDA followed by SCL that I guess I couldn't stop! With that fixed I was out of time. I did run the Tcy for MMC to I2C transfers and I'm only saving about 1000 Tcy, far less than what I was hoping for. I know the issue is in the number and placement of @ NOP's. Something is eating tons of time with Tcy up over 54,000, but it's not all going to be in the I2C. The MMC runs without any delays in a serial read, so I'm not sure what's going on.

Skimask, PM me with an email address and I'll send you a copy of the code. It's just short of 900 text lines and won't fit in a post here. Note that for debug reasons, the I2C reading has been reverted back to the PBP I2CREAD handler.

[skimask]

Skimask, PM me with an email address and I'll send you a copy of the code. It's just short of 900 text lines and won't fit in a post here. Note that for debug reasons, the I2C reading has been reverted back to the PBP I2CREAD handler.

Just attach it here as a .txt file.

[JD123]

Okay, I didn't see the attachments in posting.

Keep in mind that this is still a FORWARD process and I've left a lot of trash in my wake. When it's working the way I want I'll go clean up the trash.

Also, if it will help stop comments like "well, that was stupid", this is only my 7th PIC program to write, though I bought my first PIC in 1993. No one has ever seen my code before, other than bits and pieces. This kind of feels like getting undressed in public... on stage... at the Opera!

What I've made:

An LED blink in asm
A/D > LCD in asm
AoA (Angle of Attack) controller for model airplanes in CCS-C
Automatic slip/skid (yaw) controller for model airplanes in PBP and asm
A servo driver in PBP and asm
A proof’er w/LCD for calibrating the servo drivers in PBP and asm
And now this MMC reader and soon to be data logger in PBP

This is really only my 4th PBP program, though I have a number of Quick Basic 4.5 PC oriented programs under my belt. Be kind.

[skimask]

1st thought right off the bat...
You're using an '876A. It's easy to replace that with an 18F2620.
Practically the same pinout, load more code space, lots more ram, more internal functions...
Having a look at the rest of the code right now...

[JD123]

I remember looking at 18F datasheets and felt like I was in a different world. Since the 876 was at hand that seemed the PIC to use for now. The I2C and SPI connections are set on the correct pins and ports, so I can move to those functions later. For now, I just needed to take baby steps in learning how to run the SPI and I2C without adding learning the hardware too. When I (if I) move to those hardware functions I should have a better idea of what they expect and how to use them.

[JD123]

I do like one thing about that chip... enough SRAM to load the sectors directly on the PIC.

[skimask]

I remember looking at 18F datasheets and felt like I was in a different world. Since the 876 was at hand that seemed the PIC to use for now.

Ah...that's the beauty of it...they're practically identical as far as PBP goes...except for minor changes in registers and such. This type of program should almost port right over, as long as you compare any/all registers you mess with, with the ones on both datasheets to make sure they match well enough.

If you're looking for all out speed, you might want to take some of your CLOCK_OUT type subroutines (the ones where you only flip a bit back and forth) and put those inline.
For each Gosub and Return, especially in a PIC16Fxxx, you use at least 2 cycles to GOSUB, 2 cycles to RETURN, and a cycle here and there just to set the page of the bit you want to flip. So that 2 cycle operation, turns into 6 or 8 depending.

Somewhere up towards the top, you've got a sector loop that runs 0-512 bytes. should be 0-511???

Figure out which byte variables you use the most (not pins assigned to a variable, but a RAM variable), and assign those to BANK0.
A variable assigned to BANK0 only needs a single instruction to set the PICs page register to ZERO. A variable assigned to other banks usually needs 2 instructions to set the PICs page register.
For instance, you've got:
w_byte var byte
try:
w_byte var byte BANK0
If there is no difference in code space, then you didn't save anything because it was already set in BANK0. If there is a different, then you did save because it was originally in BANK1,2 or 3, depending on total ram usage...

(I could really go nuts with the colons here! )

[JD123]

Somewhere up towards the top, you've got a sector loop that runs 0-512 bytes. should be 0-511???

That's an error. I'll fix it.

Figure out which byte variables you use the most (not pins assigned to a variable, but a RAM variable), and assign those to BANK0.
A variable assigned to BANK0 only needs a single instruction to set the PICs page register to ZERO. A variable assigned to other banks usually needs 2 instructions to set the PICs page register.
For instance, you've got:
w_byte var byte
try:
w_byte var byte BANK0

That's good to know - I had no idea. I'll give this a try. Thanks!

(I could really go nuts with the colons here!)

You mean in the text format? I've been known to make the text file shorter doing this, but for 'me' it's easier read the logic without hunting down colons.

[skimask]

That's good to know - I had no idea. I'll give this a try. Thanks!

Somewhere on these forums is writeup talking all about moving variables around to save both code space and increase execution speed a bit...don't remember what it is though...

You mean in the text format?

It's all good...I've been known to take a perfectly readable file of source code, and knock it down to fit a very small space and make it perfectly unreadable to anyone but me by using loads of colons and removing extraneous spacing.
And I forgot to mention...
I just like piling as much information on the screen as possible, which is why I run 1600x1200, sometimes more...on more than one screen at a time...

[JD123]

Am I reading this right? The 18F2620 doesn't use banks in memory locations? If so, cool! Looks like I'll be moving to this chip soon, if for no other reason than the onboard SRAM. Anyone want to buy a 'gently' used F-RAM? (just kidding)

[JD123]

I've been known to take a perfectly readable file of source code, and knock it down to fit a very small space and make it perfectly unreadable to anyone but me

AKA Job Security!

I just like piling as much information on the screen as possible, which is why I run 1600x1200

See first quote.

[skimask]

Am I reading this right? The 18F2620 doesn't use banks in memory locations? If so, cool! Looks like I'll be moving to this chip soon, if for no other reason than the onboard SRAM. Anyone want to buy a 'gently' used F-RAM? (just kidding)

You got it... AND the 18Fxxx's run 40Mhz (although I think they should be rated at 55Mhz, 'cause every one of mine goes that fast, at least at 4.5-5v).

[JD123]

I've got to get away from this pooter and get some real work done. Since I found the attachments button (duoh) here a screen capture of the F_ind file option.

[skimask]

AKA Job Security!
See first quote.

Yeah...but everybody else seems to hate it.
(Refer to this thread: http://www.picbasic.co.uk/forum/show...t=optimization )
Check the file attached to post #29...

And just for grins...if I'd rewrite that program it would look like this: