Where should I discuss SD/MMC FAT issues? [Archive]

JD123

- 11th March 2008, 22:35

Hey there guys and gals, new poster here, long time tinker of PBP.

Where (what forum) should I post questions and issues revolving around SD/MMC FAT16 and FAT32? My project could use the memory card as a bit-bucket and then 232 the data out, but I'd rather have the cards PC readable.

For the first time ever, I've bit-banged an MMC card into submission using a 16F876A. I'm finally in and reading data after about 5 hours of chit (not including the hardware), but working with FAT's is giving me a migraine. I guess I don't really know what questions to ask right now as I'm still learning the FAT structure. I'm still trying to sort out the strait-line math 2^x AND (2^z(y+1) >> 7) ... etc. ... yadi, yadi, yadi... just to find the next location of the next math nightmare.

Just would like to know where would be the best place for this topic?

Thanks,

Gary

BrianT

- 12th March 2008, 04:58

Hi JD123,

I am also trying to use a MMC card as a large data store that MUST be PC readable. I bought some development hardware from Ron Diol at Compsys a year back but the project got pushed aside by the urgent driving out the important. Ron was very helpful and there are some good links there.

http://www.compsys1.com/workbench/On_top_of_the_Bench/MMC_Project/mmc_project.html

http://elm-chan.org/docs/mmc/mmc_e.html

I am thinking along the lines of preformatting the SD/MMC card in FAT 16 or 32 and loading two files of known names on the card. The SPI mode does not require any CRC calculations (I think) and is plenty fast enough for my needs.

The PIC code would open one file and append the latest data. It would close that file and open the second file, write the same code a second time and close. The two file are so if the card is removed while a file is still open, the data should be recoverable from the other file. No doubt there are better ways.

I want to keep the PIC side code as lean as possible and I can always preformat the cards but the final data must be directly readable by a PC.

Please keep this going in any forum as I think it will be a pretty interesting thread as you get it all going.

Cheers
BrianT

JD123

- 12th March 2008, 16:53

Hi Brian,

This is where I'm at. I can format the MMC on the PC, make a singel file using the PC, find the directory listing for the file using the PIC, find the file using the PIC, read and modify the file on the PIC and read the results on the PC via the MMC card reader.

What I now need to do is modify the directory listing to reflect the changes made to the file, so that the PC can understand the file information correctly. For example, I originally made the file "TEST123.TXT" 10 bytes long with the string "1234567890" so that it was easy to search the MMC to locate the file. As I made changes to the file using the PIC, if I went over the original 10 byte length, the data gets recorded, but the PC only sees the original first 10 bytes.

I found the file size information in the directory listing at offset 28 and this falls into what I was expecting based on that I've read. Problem is that the file size information is a 4 byte Dword and FAT specs call for the 4 bytes to START at offset 28. In reality, I would have thought that the "10" for the file size would have been in offset 31 (LSByte of the 4 bytes). I'm trying to figure out where I place the additional 3 bytes of the file size information. I can't find anything to support that the file size should be written low byte first and that wouldn't make sense anyway. So, I'm thinking that there's something wrong with my understanding of where the bytes are stored.

Anyway, this is where I was last night. I was tickled-pink just to see that I could get the PIC to make changes to the MMC that the PC could read. One thing that became real clear last night was that I'll have no choice but to add RAM to my hardware. I've got to be able to work on a sector's individual bytes and that's only going to happen if I have it in RAM and then send it back to the MMC when I'm done tinkering with the sector. I happen to have some 64KBit I2C EEPROMS handy, so tonight I'll press one of them into service on the proto board. This alone will put the breaks on FAT work for a few days (I only get a few hours a day to 'play' with PICs) while I deal with the hardware and I2C code.

I'll post updates as they come. If anyone can help me understand the offset location and data style (MSB or LSB first) of the file size data I'd appreciate reading your thoughts.

Gary

Edit: Brian, I forgot to say thanks for the links. I've been to both before and the second one was very helpful in getting me started.
--

skimask

- 12th March 2008, 17:00

I found the file size information in the directory listing at offset 28 and this falls into what I was expecting based on that I've read. Problem is that the file size information is a 4 byte Dword and FAT specs call for the 4 bytes to START at offset 28. In reality, I would have thought that the "10" for the file size would have been in offset 31 (LSByte of the 4 bytes). I'm trying to figure out where I place the additional 3 bytes of the file size information. I can't find anything to support that the file size should be written low byte first and that wouldn't make sense anyway. So, I'm thinking that there's something wrong with my understanding of where the bytes are stored.
Isn't the file size in FAT32 stored at 2 different places? And both of them have to be identical? I forget...it's been awhile since I read the FAT32 documentation...

I'll have no choice but to add RAM to my hardware.
I built an MP3 player awhile back based off an 18F4620 reading from CF cards in IDE mode. Had enough ram to buffer 6 complete sectors with room left over for a load of variables.
Also, if that's not enough, look into the F-RAM options. Serial EEPROM just like a 24C256 type, but it's ram, read/write just as fast, not slow writes/fast reads like eeprom, and practically unlimited lifetime.

understand the offset location and data style (MSB or LSB first) of the file size data I'd appreciate reading your thoughts.
Keep track of your little and big ENDIAN formats according to the FAT32 documentation. I don't know how many times that sort of thing tripped me up bad...

EDIT: Just dug out the FAT32 doc file.
Is the file you're basing your FAT32 code from called 'fatgen103.doc'?
If so, that file has a few issues in itself, wrong places for some values, some values omitted, and so on...
gain, it's been awhile. I'll see if I can't dig up the specifics.

JD123

- 12th March 2008, 17:29

Hi Skimask,

Isn't the file size in FAT32 stored at 2 different places? And both of them have to be identical? I forget...it's been awhile since I read the FAT32 documentation...

Sorry, I forgot to say that I'm using FAT16 when I format the MMC. It's the native format that the MMC's came with. I don't know what version of FAT16 I'm using. I understand there at least several versions or revisions to the FAT16 system. The MMC's I'm using are 128MB in size.

...look into the F-RAM options.
Yes, for a final in-field application this would be the way to go. As it is, I'm cheap and have the EEPROM's on hand. My application is data recording at about 100 sectors a second. For writing the data it will be sent directly to the card. The EEPROM use in my ap would only be at the first and end of the recording events (about 10-15 minutes long) and this would be just to address the FAT issues.

I'm back off to some of my web haunts where I've been reading about FAT16. Maybe I'll find the answer by going back over these sites. I do have this odd recollection of there being some part of the FAT where data is written LSByte first. Maybe it was the file size?

Gary
--

skimask

- 12th March 2008, 17:50

Sorry, I forgot to say that I'm using FAT16 when I format the MMC.
That fat32 document has a load of info in it, and specifies where it differ's from fat16...not that it matters much :)

My application is data recording at about 100 sectors a second.
I was able to shove a bit over 600K/sec into the CF card on my player, but that's using the IDE mode, parallel data transfer. 600K was as fast as the PIC could go @ 40Mhz.
You'll be limited to the max SPI speed, and I think you'll be lucky to be able to handle 100 sectors/second (100 sectors = 512000 bytes = 5.12mbps) and still have any time leftover to do any sort of interrupt processing.

I do have this odd recollection of there being some part of the FAT where data is written LSByte first. Maybe it was the file size?
Do a search for fatgen103.doc and you'll see what I'm looking at.
And yes, offset 28 is the file size, and yes, it is LSB first...little endian...

JD123

- 12th March 2008, 18:00

YES! I came back here to say I found this, only to see you found it too. The size IS written LSByte first. I can't tell you how many places on the web fail to state this. This is what I just now found:

data length ( =4 bytes )

the minimum can be 00 00 00 00 - 0 and the maximum ff ff ff ff -
4,294,967,295
gb, mb, kb, by

this entry will state the length of data to read in.
eg, AC 3B 96 00
flip these values: 00 96 3B AC
convert to decimal: 9,845,676 bytes or 9.38MB
folders have a value of: 00 00 00 00

Thanks guys for getting me out of this ditch. It's amazing how much time I waste on some of the 'simplest' problems.

skimask, thanks for letting me know what you were able to get for speed. I'm shooting for 100, 8 byte recordings per second, but if I can make 60 I'll be happy. The recordable rate will be selectable and I was picking 100 as a top-shelf goal.

Edit: skimask, sorry... I see what caused the issue about speed. I originally ment to say ONE sector per second, not 600. 600 is the byte rate, not the sector rate. I guess I really WAS pushing the limits, the way I originally stated it!!!

skimask

- 12th March 2008, 18:03

skimask, thanks for letting me know what you were able to get for speed. I'm shooting for 100, 8 byte recordings per second, but if I can make 60 I'll be happy. The recordable rate will be selectable and I was picking 100 as a top-shelf goal.

100 8 byte recordings per second? That's only 800 bytes... I said 600KB per second, as in, 614,400 bytes per second, 6,144,000 bits per second.
But again...it was a parallel operation.

Now are you talking 100 8 bytes recordings...
or 100 512 byte (sector) recordings

JD123

- 12th March 2008, 18:17

100 8 byte recordings per second? That's only 800 bytes... I said 600KB per second, as in, 614,400 bytes per second, 6,144,000 bits per second.
But again...it was a parallel operation.

Now are you talking 100 8 bytes recordings...
or 100 512 byte (sector) recordings

Yes, I messed up. See my edit.

That should be quite reachable. The real issue is reading the PWM outputs from the accelerometers and an airspeed pressure sensor. My co-worker on this project wants to read the PWM as filtered analog resolved from a 1ms PWM frame rate. I'm arguing that I can get everything done and more in the 16.66ms that we have at 60 recordings per second. Also, using the CCP and MUX'ing the inputs I can get rock solid data with a 5000 FS range. On a +/-10G accelerometer that will be .4mG resolution - well enough for what we are doing.

Gary
--

JD123

- 13th March 2008, 15:32

I'm not going to turn this into a thread of me talking to myself, but I wanted to say that I put a 24LC16 on the pic and I'm now reading the directory table into memory, working on it and writing it back to the MMC without any problems. The PC now accepts the modified files with the correct file information from the MMC.

Reading out of the MMC to the I2C takes ~3 seconds for a 512 sector. I'll be getting it faster by coding for loading the 16 byte buffer on the I2C.

skimask

- 13th March 2008, 15:37

The PC now accepts the modified files with the correct file information from the MMC.
So it was the file size issue then?

Reading out of the MMC to the I2C takes ~3 seconds for a 512 sector. I'll be getting it faster by coding for loading the 16 byte buffer on the I2C.
Post some code. I'm sure there's a way to make it faster... It only took my code about 7-ish seconds to erase a full 24LC256 (32Kbytes), while displaying the current address on an LCD.

JD123

- 13th March 2008, 15:55

I'll post the code if you promise not to laugh. When I'm in the R&D phase my goal is to understand, not to write pretty code. All I'm doing right now is building a reference for dealing with the MMC when I make a final application.

I'm at work (work has nothing to do with PICs) and the files are at home. No internet at home (yes, I live in a cave... next-door to Osama) so it will be a day or two before I get it up. Most likely I'll have the code faster by then as tonight I'll probably change it to write the page buffer on the I2C memory. I started with the recommended 10ms delay after a write and once things were stable, I moved it to 5ms. Next step will be polling for busy for the shortest delays.

skimask

- 13th March 2008, 16:06

I'll post the code if you promise not to laugh.
The difference between R&D code and pretty code is....who cares...

Most likely I'll have the code faster by then as tonight I'll probably change it to write the page buffer on the I2C memory. I started with the recommended 10ms delay after a write and once things were stable, I moved it to 5ms. Next step will be polling for busy for the shortest delays.
5ms is max. write cycle time for the 24LC16 (byte or page).
You start writing 16 byte pages, and it should speed up by a factor of about 16, with the same 5ms delay after the 16 byte packet is sent out.

JD123

- 13th March 2008, 16:39

Oh, and yes, the issue about the PC not 'seeing' the complete file was the file size information at offset 28 in the directory table. What you said about flipping the size to LSB first works well.

For giggles and grins I formatted the MMC as FAT32 and tried to work with the FAT. Let's just say... NOT! I'll stick with FAT16 as long as I can. Does Vista allow FAT16 and or format media cards in FAT16? Thought I heard somewhere that FAT16 support was dropped.

Acetronics2

- 13th March 2008, 16:40

Hi, JD ( Pffffff !!!!)

If you're interested ... I've such a project with a 18F452 ... but Written In MikroC ...
not so far from Basic.

I do not know why, but I think you'll be Ok ...

Alain

PM me a mailBox address if needed ...

JD123

- 13th March 2008, 16:50

Hi Alain,

Thanks for the offer. I'll take you up on that. My next step is getting some intelligent information about adding files and making sure things like overwrites and PC compatibility are handled correctly.

BTW, I recognize you from RCG.

Gary
[email protected]
--

skimask

- 13th March 2008, 17:45

For giggles and grins I formatted the MMC as FAT32 and tried to work with the FAT. Let's just say... NOT! I'll stick with FAT16 as long as I can. Does Vista allow FAT16 and or format media cards in FAT16? Thought I heard somewhere that FAT16 support was dropped.
In that document, it specifies the cluster count cutoff for determining if something is formatted in FAT12/16/32.
If FAT16 was dropped, I would infer that FAT12 would've been dropped, which means no more floppies at all.
If you read that document, FAT32 is easy to work with, just have to take heed of the differences. The thing that's a real pain is the long filename's that come with it. But if you ignore those and stick with 8.3, FAT32 is a piece of cake.

BrianT

- 14th March 2008, 21:55

http://www.mmca.org/compliance/buy_spec/mmc_spec_v4_3/JEDSD84-A43.pdf

Cheers
BrianT

JD123

- 14th March 2008, 22:13

http://www.mmca.org/compliance/buy_spec/mmc_spec_v4_3/JEDSD84-A43.pdf

Cheers
BrianT
Thanks Brian. I have a copy of that. It reads like the USA budget passed by Congress :0 I couldn't find any information about FAT in the doc, though it may be in there.

Last night I decided to see if I could find a file armed only with the file name. I was able to do it. I started to write a post that cuts to the chase in explaining how I find a file but decided if I was going to get up in the morning, I'd have to finish it later. I'll post it when it's finished. It will be a real time saver for the first-time MMC user.

In short, I only need to read a few items from the MBR, Partition Record and the Directory Table (I think that's all I was reading - it was late). It's not hard once I know what to look for and stay within the structure of the FAT.

skimask

- 14th March 2008, 22:15

Thanks Brian. I have a copy of that. It reads like the USA budget passed by Congress :0 I couldn't find any information about FAT in the doc, though it may be in there.
It's only an MMC document.
I forget, do you have the FAT spec's document? (fatgen103?)

JD123

- 14th March 2008, 22:30

Skimask, yes. I have the one you were talking about and about 10 other pages found on the net. Together it's concise.

Thanks again for the LSB information. As I worked last night, I come to find out that all Word and Dword data that points to an address or its offset are written this way in FAT. I never have used the SWAP function of PBP before, but it's coming in very handy right now.

{4 data bytes, LSB first} D,C,B,A

swap A, D
swap B, C

{4 data bytes, MSB first} A,B,C,D

I understand this is just a work-around for the fact that "I" work with Word & Dword addresses as MSB first. I'm stuck in my ways... what can I say!

JD123

- 14th March 2008, 23:04

Oh, and for what it's worth, I just ordered some FRAM chips from Mouser. That should speed up sector reading a bit. I got 64K's so I can have up to 16 sectors or 4 clusters in RAM at one time. Since it will be so fast (up to 1mhz), I might use it to hold other data too. Having bigger and faster memory is like buying a Ford Excursion after owning a VW Rabbit.

JD123

- 19th March 2008, 18:51

Okay, the F-RAM chip is in place and working. Taking all delays out of the code I can get a sector (517 bytes) of the MMC onto the F-RAM in about 13.5ms (MMC > PIC > I2C). That's much faster than the old I2C. It's nice not having to stop and write the buffer page!

I'm still looking for a faster way to move the I2C data line, so I'm writing my own I2C handler routines. My whole program cycle has to fit into 16.67ms. It doesn't look like I'll be able to load the I2C with sample data then move it to the MMC in this window unless I can get the program cycle, minus the data captures, down to about 6ms. It's going to be close.

skimask

- 19th March 2008, 19:47

I'm still looking for a faster way to move the I2C data line, so I'm writing my own I2C handler routines. My whole program cycle has to fit into 16.67ms. It doesn't look like I'll be able to load the I2C with sample data then move it to the MMC in this window unless I can get the program cycle, minus the data captures, down to about 6ms.
What speed is your osc running at?
I know you can get 500-ish bytes inside of a few ms...
Show some code!

JD123

- 19th March 2008, 20:32

What speed is your osc running at?
I know you can get 500-ish bytes inside of a few ms...
Show some code!

Codes at home. The proto board is running at 16mhz and the final version will run at 20mhz.

I know what's slowing it down. In my rush to write the code (I wanted to see if this F-RAM really was a drop-in replacement for the 24C64's), I simply re-addressed the I2C for every write and this is sending way too many bits per write. It's kicking out 48 bits per byte write yet my I2C handler will send out 9 bits per byte (after a one-time command/address string) and at about twice the clocking speed. That will bring the data transfer rate up to about 10 times faster overall.

I was about to go ahead and write a variable string of say 16 bytes per I2C_WRITE to speed things up, but decided not to waste my time, knowing I'd sooner or later bit-bang the I2C for the fastest results. I'm writing the I2C handler to run at the 20mhz speeds, so what I get at 16mhz will be off of top possible speeds by 20%.

I should get the simple use of my I2C handler finished tonight. It would have been done last night, but I went to bed early. I've been puting in some long days the past few weeks thanks to this project. I'll see if I can remember to bring some code to work to post. I think it's about 500 lines now (text editor lines, not instruction lines) but should be easy to follow because I tab, space and sub block in logical places, except when I'm debugging fast. The code is still full of unnecessary debug lines.

JD123

- 19th March 2008, 21:08

This is an example of my code style. It's not from this project.

'*************** MODE 4 ****************

IF MODE = 4 THEN

TIME = TIME - 5

IF TIME = 1005 THEN
FOR T = 1 TO 50
GOSUB SENDOUT
GOSUB DELAY
NEXT T
ENDIF

IF TIME = 1995 THEN
FOR T = 1 TO 50
GOSUB SENDOUT
GOSUB DELAY
NEXT T
ENDIF

IF TIME < 900 THEN
TIME = 900
ENDIF
RETURN
ENDIF

JD123

- 20th March 2008, 15:32

Well, I half-ass got the I2C running last night. I knew I'd have to play with the NOP's to get the fastest speeds, but ran out of time. I spent over 2 hours debugging the I2C only to find that I was clocking out using:

SDA and SCL are port pins

SDA = 1
@ NOP
@ NOP
@ NOP
@ NOP
@ NOP
SCL = 0
@ NOP
etc..

I spent so much time coding SDA followed by SCL that I guess I couldn't stop! With that fixed I was out of time. I did run the Tcy for MMC to I2C transfers and I'm only saving about 1000 Tcy, far less than what I was hoping for. I know the issue is in the number and placement of @ NOP's. Something is eating tons of time with Tcy up over 54,000, but it's not all going to be in the I2C. The MMC runs without any delays in a serial read, so I'm not sure what's going on.

Skimask, PM me with an email address and I'll send you a copy of the code. It's just short of 900 text lines and won't fit in a post here. Note that for debug reasons, the I2C reading has been reverted back to the PBP I2CREAD handler.

skimask

- 20th March 2008, 15:44

Skimask, PM me with an email address and I'll send you a copy of the code. It's just short of 900 text lines and won't fit in a post here. Note that for debug reasons, the I2C reading has been reverted back to the PBP I2CREAD handler.
Just attach it here as a .txt file.

JD123

- 20th March 2008, 17:46

Okay, I didn't see the attachments in posting.

Keep in mind that this is still a FORWARD process and I've left a lot of trash in my wake. When it's working the way I want I'll go clean up the trash.

Also, if it will help stop comments like "well, that was stupid", this is only my 7th PIC program to write, though I bought my first PIC in 1993. No one has ever seen my code before, other than bits and pieces. This kind of feels like getting undressed in public... on stage... at the Opera!

What I've made:

An LED blink in asm
A/D > LCD in asm
AoA (Angle of Attack) controller for model airplanes in CCS-C
Automatic slip/skid (yaw) controller for model airplanes in PBP and asm
A servo driver in PBP and asm
A proof’er w/LCD for calibrating the servo drivers in PBP and asm
And now this MMC reader and soon to be data logger in PBP

This is really only my 4th PBP program, though I have a number of Quick Basic 4.5 PC oriented programs under my belt. Be kind.

skimask

- 20th March 2008, 17:54

1st thought right off the bat...
You're using an '876A. It's easy to replace that with an 18F2620.
Practically the same pinout, load more code space, lots more ram, more internal functions...
Having a look at the rest of the code right now...

JD123

- 20th March 2008, 18:08

I remember looking at 18F datasheets and felt like I was in a different world. Since the 876 was at hand that seemed the PIC to use for now. The I2C and SPI connections are set on the correct pins and ports, so I can move to those functions later. For now, I just needed to take baby steps in learning how to run the SPI and I2C without adding learning the hardware too. When I (if I) move to those hardware functions I should have a better idea of what they expect and how to use them.

JD123

- 20th March 2008, 18:14

I do like one thing about that chip... enough SRAM to load the sectors directly on the PIC.

skimask

- 20th March 2008, 18:18

I remember looking at 18F datasheets and felt like I was in a different world. Since the 876 was at hand that seemed the PIC to use for now.
Ah...that's the beauty of it...they're practically identical as far as PBP goes...except for minor changes in registers and such. This type of program should almost port right over, as long as you compare any/all registers you mess with, with the ones on both datasheets to make sure they match well enough.

If you're looking for all out speed, you might want to take some of your CLOCK_OUT type subroutines (the ones where you only flip a bit back and forth) and put those inline.
For each Gosub and Return, especially in a PIC16Fxxx, you use at least 2 cycles to GOSUB, 2 cycles to RETURN, and a cycle here and there just to set the page of the bit you want to flip. So that 2 cycle operation, turns into 6 or 8 depending.

Somewhere up towards the top, you've got a sector loop that runs 0-512 bytes. should be 0-511???

Figure out which byte variables you use the most (not pins assigned to a variable, but a RAM variable), and assign those to BANK0.
A variable assigned to BANK0 only needs a single instruction to set the PICs page register to ZERO. A variable assigned to other banks usually needs 2 instructions to set the PICs page register.
For instance, you've got:
w_byte var byte
try:
w_byte var byte BANK0
If there is no difference in code space, then you didn't save anything because it was already set in BANK0. If there is a different, then you did save because it was originally in BANK1,2 or 3, depending on total ram usage...

(I could really go nuts with the colons here! :D )

JD123

- 20th March 2008, 18:34

Somewhere up towards the top, you've got a sector loop that runs 0-512 bytes. should be 0-511???

That's an error. I'll fix it.

Figure out which byte variables you use the most (not pins assigned to a variable, but a RAM variable), and assign those to BANK0.
A variable assigned to BANK0 only needs a single instruction to set the PICs page register to ZERO. A variable assigned to other banks usually needs 2 instructions to set the PICs page register.
For instance, you've got:
w_byte var byte
try:
w_byte var byte BANK0

That's good to know - I had no idea. I'll give this a try. Thanks!

(I could really go nuts with the colons here!) You mean in the text format? I've been known to make the text file shorter doing this, but for 'me' it's easier read the logic without hunting down colons.

skimask

- 20th March 2008, 19:18

That's good to know - I had no idea. I'll give this a try. Thanks!
Somewhere on these forums is writeup talking all about moving variables around to save both code space and increase execution speed a bit...don't remember what it is though...

You mean in the text format?
It's all good...I've been known to take a perfectly readable file of source code, and knock it down to fit a very small space and make it perfectly unreadable to anyone but me by using loads of colons and removing extraneous spacing.
And I forgot to mention...
I just like piling as much information on the screen as possible, which is why I run 1600x1200, sometimes more...on more than one screen at a time...

JD123

- 20th March 2008, 19:25

JD123

- 20th March 2008, 19:28

I've been known to take a perfectly readable file of source code, and knock it down to fit a very small space and make it perfectly unreadable to anyone but me AKA Job Security!

I just like piling as much information on the screen as possible, which is why I run 1600x1200 See first quote.

skimask

- 20th March 2008, 19:29

Am I reading this right? The 18F2620 doesn't use banks in memory locations? If so, cool! Looks like I'll be moving to this chip soon, if for no other reason than the onboard SRAM. Anyone want to buy a 'gently' used F-RAM? (just kidding)

You got it... AND the 18Fxxx's run 40Mhz (although I think they should be rated at 55Mhz, 'cause every one of mine goes that fast, at least at 4.5-5v).

JD123

- 20th March 2008, 19:51

I've got to get away from this pooter and get some real work done. Since I found the attachments button (duoh) here a screen capture of the F_ind file option.

skimask

- 21st March 2008, 00:46

AKA Job Security!
See first quote.
Yeah...but everybody else seems to hate it.
(Refer to this thread: http://www.picbasic.co.uk/forum/showthread.php?t=5445&highlight=optimization )
Check the file attached to post #29... :D

And just for grins...if I'd rewrite that program it would look like this: :D

skimask

- 21st March 2008, 00:48

I've got to get away from this pooter and get some real work done. Since I found the attachments button (duoh) here a screen capture of the F_ind file option.
Actually, you're one of only a couple of other people I've seen to get PBP to work with FAT on an MMC. I got a PIC/PBP to talk to an MMC, but not talking FAT.
I'm going to have a good hard look at your code and see what (if any) major changes would be needed for FAT32 compatibility.

JD123

- 21st March 2008, 14:31

Actually, you're one of only a couple of other people I've seen to get PBP to work with FAT on an MMC. I got a PIC/PBP to talk to an MMC, but not talking FAT.
I'm going to have a good hard look at your code and see what (if any) major changes would be needed for FAT32 compatibility.

Two things. First, I have a treat for you and anyone else who's trying to understand what I've done. It was a post I put together about how to simply find a file using a file's name and FAT. I wasn't happy with the limit of the scope, but now that I've put code out there, I think some people can benefit from what I've found.

Second, man you hit the nail on the head! I went from about 55,000 Tcy to 5098 Tcy by forcing the bank locations to 0 for the variables used in the read/write loops. I was so shocked, I thought that I didn't get the data, but with each read of the F-RAM to look at it's content, sure enough all my data is there. That's amazing how much difference that made. Do you think by pushing the loop subs up to the top of the program I could see even faster speeds?

(Oops... wrong file ;))

JD123

- 21st March 2008, 14:38

Yeah...but everybody else seems to hate it.
(Refer to this thread: http://www.picbasic.co.uk/forum/showthread.php?t=5445&highlight=optimization )
Check the file attached to post #29... :D

And just for grins...if I'd rewrite that program it would look like this: :D

LOL! Yep, that would be like a Ford truck I once owned. It drove so bad (hard to keep it in a strait line) that when a friend of mine tried to drive it he said "this is a one person truck". That code would also be a one person truck!

skimask

- 21st March 2008, 15:25

Second, man you hit the nail on the head! I went from about 55,000 Tcy to 5098 Tcy by forcing the bank locations to 0 for the variables used in the read/write loops
SWEET!

Do you think by pushing the loop subs up to the top of the program I could see even faster speeds?
Probably not...BUT...if you take all of the variables in that tight loop, and make them all BANK0, you might see an incremental speed increase. I can't remember if the compiler will complain once it's out of BANK0 space or not, or if you'll even know it. One thing to do would be to check the .lst file and see where they're being put.
(looking at the new doc file now)

JD123

- 21st March 2008, 15:34

SWEET!

Probably not...BUT...if you take all of the variables in that tight loop, and make them all BANK0, you might see an incremental speed increase. I can't remember if the compiler will complain once it's out of BANK0 space or not, or if you'll even know it. One thing to do would be to check the .lst file and see where they're being put.
(looking at the new doc file now)

That's what I did - ALL variables used in the loops.

I re-read the PBP manual this morning and it does say that it will issue a warning if a 'requested' bank assignment won't fix.

skimask

- 21st March 2008, 15:39

That's what I did - ALL variables used in the loops.
I re-read the PBP manual this morning and it does say that it will issue a warning if a 'requested' bank assignment won't fix.
Ok, then moving that loop to the beginning of the program might have a small-ish impact on speed for the same reason, keeping it in BANK0. And if the whole loop won't fit up there, just try to put the tightest, smallest chunk of it up there. As I found out when I built my MP3 player, one cycle inside of a tight 512 count loop, inside of a tight 32 count loop (16K counts total) makes a big difference...especially when that loop was only 12 cycles long in the first place.

JD123

- 21st March 2008, 22:04

Last night I tried coding for the READ_I2C function (my sub) and ran into a fair amount of trouble. I’m not sure, but I think it revolved around the repeated hits to the ACK sub, where I originally wrote it for writes only (see where I toggle the port looking for the ACK from the SDA on the I2C) and my not understanding (while I was head-banging the problem) that the PIC issues the ACK if it is the receiver (reading the I2C).

It was late and I didn't give it a fair chance. I'm sure I get it figured out. Still, I'm up for suggestions as to what 'should' be the I2C functions that I make. I think I've got things so fractured that it's both getting hard to follow the loops and I maybe re-writing code unnecessarily. I was thinking that I should write the functions (subs) like this:

OPEN_I2C with a R/W bit set for read/write keeping all aspects inside the sub, and I2C stays open after address until...

CLOSE_I2C will send a stop command to end single or serial byte reads and writes.

WRITE_I2C simply sends out a byte with ACK back from the I2C, I2C stays open for next WRIET_I2C.

READ_I2C simply reads in a byte with ACK from PIC, I2C stays open for next READ_I2C.

Does this sound like a logical way to breakup the I2C subs so that I can read/write both bytes and serial strings?

This would be easier if PBP had an open-ended I2C function (like CCS's C does) for long reads and writes. Not all EEPROMS have page write buffers and all EEPROMS have unlimited serial reads (less you roll it over). Think I should put a note in the suggestion box?

skimask

- 21st March 2008, 22:11

This would be easier if PBP had an open-ended I2C function (like CCS's C does) for long reads and writes. Not all EEPROMS have page write buffers and all EEPROMS have unlimited serial reads (less you roll it over). Think I should put a note in the suggestion box?
Quite frankly, I've never tried making my own I2C subroutines.
And besides, I believe PBP's I2C routines let you write multiple bytes just by the fact that you have multiple bytes in a statement (see the str modifier in the manual, Page 85 of the PBP2.50a manual, 2nd paragraph from the bottom). That should cover reading and/or writing pages at a time.
PBP is pretty tight as it is and you might not be gaining a whole heck of a lot by rolling your own subs...just banging your head....or is that 'bit-banging' your head :)
The suggestion might not be needed...

JD123

- 21st March 2008, 22:53

skimask

- 21st March 2008, 22:56

I'm trying to avoid this huge 512 byte array, which won't fit on this PIC anyway.

Speaking of the PIC, I was at the store today and picked up an 18F2050. It's not a 2025, but it should do. I loose one CCP but gain a USB. Past that I can't tell them apart.

2050? Maybe a 2550...
Should do just fine. Combine that PIC with the examples from Mr.E's USB demo in another thread (or the config bit hassle that WILL follow...just seems to happen that way), and you'll be in great shape.

And the 512 byte array won't fit, but the 32 byte or 64 byte array of the EEPROM page will...

JD123

- 21st March 2008, 23:07

{getting out my reading glasses...} Yup, it's a 2550.

I don't see a "MrE, Mr. E, Mr.E or Mr E" in the member list???

skimask

- 21st March 2008, 23:13

I don't see a "MrE, Mr. E, Mr.E or Mr E" in the member list???

http://www.picbasic.co.uk/forum/showthread.php?t=5418&highlight=mister

I got this one in my hotlinks...

JD123

- 22nd March 2008, 13:57

Thanks. That's a thread I wish I was around for back in 06.

JD123

- 22nd March 2008, 15:33

Something is wrong with my Tcy counter. It must be rolling over. I didn't make provisions for it rolling over because I couldn't imagin it taking more than 65K Tcy to move the data off the MMC and onto the F-RAM. I guess I was wrong... very wrong.

Best condition Tcy's to move data:

gosub to MMC read - 2Tcy

get byte - 24Tcy (3Tcy per bit)

return - 2Tcy

gosub to write F-RAM - 2Tcy

write F-RAM - 117Tcy (13Tcy per bit (min. clock delay 5Tcy), 9 bits w/ack)

Ack overhead - 9Tcy if tested, 3Tcy if blind read

return - 2Tcy

total for single byte loops - 158Tcy

repeat 512 times - 80,896Tcy

I must be moving data WAY slower than I thought. Time to detect TMR1 rollover and see what it really is.

skimask

- 22nd March 2008, 16:17

Something is wrong with my Tcy counter. It must be rolling over. I didn't make provisions for it rolling over because I couldn't imagin it taking more than 65K Tcy to move the data off the MMC and onto the F-RAM. I guess I was wrong... very wrong.
Make an interrupt for the overflow interrupt and increment another word based on that.
Also, I was looking thru your code again.
Whenever you multiply or divide by a power of 2, you can easily use a shift to accomplish the same thing. Saves a bunch of cycles, and some code space.
For instance, you've got:
OP = OP * 512 (at least 9 shifts, 9 double word adds + carry's, probably 100-ish cycles)
That can be rewritten:
OP = OP << 9 (9+ cycles)

repeat 512 times - 91,136Tcy
Put some of your shorter Gosub/Returns in-line with the loop that runs 512 times. For each Gosub/Return you eliminate in the loop, you save at least 2,048 cycles (about 2% for each one).

JD123

- 22nd March 2008, 17:04

Noted. I'll spend a bit more time on this, but with 80K+ Tcy's, I won't come even close to using this function while recording the 8 byte data every 16.67ms. Since this function will come into play only when starting and ending a session, time won't be an issue.

By the first of next week I need to get back on track with the core of the project.

JD123

- 28th March 2008, 21:09

We'll, I've done my best and yes, about 84K Tcy is as fast as this thing is going to read 512 bytes from the MMC and write the 512 bytes to the FRAM. It's 83,968 Tcy to be exact. Last night I thought I'd give it one more go.

At the start of the evening I got my latest code loaded and checked for timer1 overflows. Yep, it was running over. The true transfer times were 166,837 Tcy. I started dissecting the code bit by bit and embedding timers to catch cycle times. Turns out I had a very inefficient MMC routine. I strait-lined the code and got it back up to speed. I tweaked the I2C as well as moving the two function subs WRITE_I2C and GET_BYTE to the top of the program. Within an hour or two I got it down to 129,837 Tcy. After doing some math, I figured I was eating 45 Tcy with every loop and couldn't account for where it was coming from. The last part of the routine to be counted was the time needed to simply perform the FOR-NEXT loop.

Bingo! Turns out using WORD sized variables in a FOR-NEXT loop can be a huge Tcy eater. The solution I used was to break up the 512 count in the for-next loop to two 256 count loops. Boom! The cycle times fell to 83,986 Tcy. So, in a few hours, I saved 82,869 Tcy, saved 400 lines of complied code and gained about twice the speeds. I'm currently at 16mhz ripping the MMC sector and writing it in a total of just under 21ms. Still not fast enough to use it as I would like, but still... that's fast I2C.

Here's the updated code:

skimask

- 29th March 2008, 07:58

We'll, I've done my best and yes, about 84K Tcy is as fast as this thing is going to read 512 bytes from the MMC and write the 512 bytes to the FRAM. It's 83,968 Tcy to be exact. Last night I thought I'd give it one more go.
Add
DEFINE NO_CLRWDT 1 'no extra clear watchdog timer instructions
to the top of your program.
Will probably save a few more cycles there.

Also, in your 'big' loop, this might help somewhat. Remove the 2 for/next loops, replace with this:
loop1:
x1 = 2 : x = 0
loop2:
gosub get_byte : gosub write_i2c : x = x - 1 : if x = 0 then loop2
x1 = x1 - 1 : if x1 = 0 then loop1

Rewriting the 2 gosubs as inline code would save you about 2K cycles at most, might get you down below 20ms, but would use up code space. No real savings there.
Also, go back and try to BANK0 as much as you can, the commonly used variables first, until something doesn't end up in BANK0 (check the .lst file).

JD123

- 29th March 2008, 15:04

Add
DEFINE NO_CLRWDT 1 'no extra clear watchdog timer instructions
to the top of your program.
Will probably save a few more cycles there.

Interesting... I didn't think I called any routines that used the WDT. I'll check the .lst and see if there are any in there.

Also, in your 'big' loop, this might help somewhat. Remove the 2 for/next loops, replace with this:
loop1:
x1 = 2 : x = 0
loop2:
gosub get_byte : gosub write_i2c : x = x - 1 : if x = 0 then loop2
x1 = x1 - 1 : if x1 = 0 then loop1
Rewriting the 2 gosubs as inline code would save you about 2K cycles at most, might get you down below 20ms, but would use up code space. No real savings there.

On the outside, this looks to take about the same or more Tcy's to run as my 2 serial loops. I'll compile the two different ways and look at the .lst to count the Tcy's and see which is faster.

Also, go back and try to BANK0 as much as you can, the commonly used variables first, until something doesn't end up in BANK0 (check the .lst file).
I've made sure that all variables used in this loop process are in bank0. Can variables not used in the loop slow the loop down?

skimask

- 29th March 2008, 23:49

Interesting... I didn't think I called any routines that used the WDT. I'll check the .lst and see if there are any in there.
PBP inserts CLEARWDT instructions whenever it thinks there's a possibility of the WDT timing out during execution. But I don't know, and I've never bothered checking, if PBP even takes a look at the config fuses to see if the WDT is enabled and/or being used. I'm assuming that PBP ALWAYS inserts the instruction whether you are using the WDT or not. Better safe than sorry type thing. At any rate, I know from my experience that in projects where I'm not using the WDT timer at all, the DEFINE saves a load of program space...or at least a fair percentage.

On the outside, this looks to take about the same or more Tcy's to run as my 2 serial loops. I'll compile the two different ways and look at the .lst to count the Tcy's and see which is faster.
And they'll probably end up being close if not identical. About the only thing I know for sure, is the nested 'manual' for/next loop will save a few bytes of program space.

I've made sure that all variables used in this loop process are in bank0. Can variables not used in the loop slow the loop down?
I wouldn't think so, but...again, keeping everything you can in BANK0 can only speed things up a bit and save a few bytes here and there.

mister_e

- 30th March 2008, 10:21

But I don't know, and I've never bothered checking, if PBP even takes a look at the config fuses to see if the WDT is enabled and/or being used.
NOPE, unless they wouldn't add this DEFINE NO_CLRWDT 1 in their list.. and there's still 2 different way to set fuses... and not a lot of user seems to use 'em anyways...

Repeat-until will execute faster than a For-Next and need less codespace. This remind me the following thread...
http://www.picbasic.co.uk/forum/showthread.php?t=6419&highlight=sharing&page=2

Post 8 and++

JD123

- 31st March 2008, 16:04

Repeat-until will execute faster than a For-Next and need less codespace. This remind me the following thread...
http://www.picbasic.co.uk/forum/showthread.php?t=6419&highlight=sharing&page=2

Post 8 and++

Thanks. That a good read. I've written some short programs to do the same kind of benchmarking. I didn't consider the WHILE like loops, because though I know the loop functions are faster, adding back in a counter instruction seemed to offset any gains in total Tcy's per cycler. If checking for a match (end of loop condition) does not depend on a sequential counting, the WHILE type statments would be faster, for sure. Is my understanding about this correct?

skimask

- 31st March 2008, 17:45

Thanks. That a good read.
Just so I can remember where you're going with this...
You're getting some data, saving it in an FRAM, then dumping it out to an MMC, right?
But...you need it to dump inside of 16ms...

We could go on optimizing this all day (week? month? :) ), saving a couple of cycles here and there, trying to shave off that last couple of uSec.
I think in the end, you'd be much better off switching over to an 18F4620 (or whatever suits you, 18Fxxxx), kick up the osc. speed a bit, and take advantage of the extra ram in the 18F. Just going to 40Mhz will get you an extra 2.5x theoretical improvement ('cause you'll probably have to add NOPs at various points to keep the timings in spec), no BANKing issues will save another bunch of cycles...and so on and so on... Jist of the story, that 20-ish ms could theoretically drop to a bit over 8ms per chunk.

JD123

- 31st March 2008, 18:30

Just so I can remember where you're going with this...
You're getting some data, saving it in an FRAM, then dumping it out to an MMC, right?
But...you need it to dump inside of 16ms...

We could go on optimizing this all day (week? month? :) ), saving a couple of cycles here and there, trying to shave off that last couple of uSec.
I think in the end, you'd be much better off switching over to an 18F4620 (or whatever suits you, 18Fxxxx), kick up the osc. speed a bit, and take advantage of the extra ram in the 18F. Just going to 40Mhz will get you an extra 2.5x theoretical improvement ('cause you'll probably have to add NOPs at various points to keep the timings in spec), no BANKing issues will save another bunch of cycles...and so on and so on... Jist of the story, that 20-ish ms could theoretically drop to a bit over 8ms per chunk.

You're right about moving up in the PIC. I've started reading about the I2C port on the PIC I have. It's a bit daunting for the first-time user of this port function.

So far as the function of the memory, up to this point, it's MMC to PIC to FRAM. If I could get it down to a few ms then I was considering using it as the write buffer for the collected data instead of directly dumping the collected data onto the MMC. I don't think this is going to work unless I use both the SPI and I2C ports on my PIC. Even still, I don't think I have the time in the data collection cycle to do this. The advantages of using the hardware ports is a near perfect 1mhz to the FRAM and max speed to the MMC (mmc takes up to 20mhz). I was reading that the ports clock out at Fosc/2 when it's all done and said, I think. Like I said, I'm still reading... and reading... and reading. In theory, I could move 512 bytes from the MMC in a touch over 2ms and the same data to the FRAM at 4ms, but this does not include any code overhead - it's just the bit rates.

Back to the MMC for a minute:

The only way I've found to stop a sector read that allows for the next command to be another sector read has been to either finish the original sector read or to initialize the MMC. Am I missing a command that allows aborting a sector read followed by another sector read command? I did find the STOP command, but found out that sending another sector read command following the STOP command fails to be accepted by the MMC. Seems that the STOP command would be better described as a "PAUSE" command. Even the datasheet's say it's to free up the SPI bus for other SPI functions. It doesn't say anything about aborting the read command. The only way to do all this has been by initializing the MMC, which can take quite some time to do.

One other thing skimask, I was going to post this question in the PBP forums, but since you brought it up here... How come when I use your ":" colon text editing to condense my I2C routines I get an error surrounding the "@ NOP" lines?

I'm using text lines that look like:

SDA=1:@ NOP:@ NOP:@ NOP:SCL=1:@ NOP:@ NOP:@ NOP:SCL=0:{...ect.}

The "warning" listed is #255 "Expected op code instead of 'NOP :@' ". at the end of many warnings it "fails" and aborts the assembly process.

It looks like you can not nest any lines using the colon after an ASM (@) insertion. I checked the edited text lines to make sure I didn't remove the needed space before the "NOP". Since the code is now stable, I'm just looking to compact it for ease of moving around in the program.

mister_e

- 31st March 2008, 18:51

Not a colonic master, but yeah, i believe you can't stack any ASM on the same line.

Note, a GOTO $+1 = 2 X NOP. For the 18F you have to use GOTO $+2

skimask

- 31st March 2008, 18:54

Not a colonic master, but yeah, i believe you can't stack any ASM on the same line.
Nope, you can't...and that really bummed me out...I could've saved close to 300 lines if I was able to stack 'em up....

skimask

- 31st March 2008, 18:57

The only way I've found to stop a sector read that allows for the next command to be another sector read has been to either finish the original sector read or to initialize the MMC.
I know there's an ABORT sequence of some sort in there somewhere. I'll do some digging and get back to you...

when I use your ":" colon text editing to condense my I2C routines I get an error surrounding the "@ NOP" lines?Previous post... One @ per line, nothing else...

JD123

- 31st March 2008, 19:06

Thanks guys. This is what I thought too - only one "@" item per line. Maybe I'll just move the I2C files into a "Include I2C.bas". I'll just have to keep track of file locations as I progress from one folder (part of the project) to another.

If I include the full path name of the file, can I get PBP to find it, regardless of where the main program exists? Stupid question, I know it should be ok, just asking.

JD123

- 31st March 2008, 19:09

Not a colonic master, but yeah, i believe you can't stack any ASM on the same line.

Note, a GOTO $+1 = 2 X NOP. For the 18F you have to use GOTO $+2

Is that GOTO literal or is there a value to "$"?

mister_e

- 31st March 2008, 19:13

in asm $ represent the current address (line). using GOTO $+1 will jump on the next address, $-1 to the previous. It avoid to have to use/add tons of label when you don't really need them.

JD123

- 31st March 2008, 19:49

in asm $ represent the current address (line). using GOTO $+1 will jump on the next address, $-1 to the previous. It avoid to have to use/add tons of label when you don't really need them.
Thanks for the answer MrE.

mister_e

- 31st March 2008, 21:20

no problem, you're welcome. Once you have a newer version, just post it here, maybe we could reduce it again and again.

skimask

- 1st April 2008, 14:19

Maybe I'll just move the I2C files into a "Include I2C.bas".
You could turn all those multi-line @ NOP's into macros.
Put this near the top of your code:

ASM
nop5 macro
NOP
NOP
NOP
NOP
NOP
endm
ENDASM

To use it:

@ nop5

JD123

- 1st April 2008, 16:25

You could turn all those multi-line @ NOP's into macros.
Put this near the top of your code:

ASM
nop5 macro
NOP
NOP
NOP
NOP
NOP
endm
ENDASM

To use it:

@ nop5

That would be 9 Tcy, right?

mister_e

- 1st April 2008, 16:30

JD123

- 1st April 2008, 17:37

Shouldn't be more than the amount of NOP. When you call a macro it just paste the macro code, but use another name.

While it's not the best code efficient, You could also use some self expanding macro by adding an argument, so if you need 2,3 or 10 NOP, it will do it for you automatically. Not sure if you need it here.

something like

<font color="#000080">ASM
NOP?C macro Cin
local a
a=0
while a<Cin
nop
a++=1
endw
endm
ENDASM
</font>

If you use @ NOP?C 1 it will paste 1 NOP and so on.
Thanks! I've written ASM programs, but never explored the macro functions. When I said 9Tcy I was looking at the added overhead of jumping in and out of the ASM sub. I was thinking that just to jump-to and return-from takes 4Tcy.

mister_e

- 1st April 2008, 17:43

Sometime knowing a bit of ASM and MPASM directive is handy. I learned it through Darrel's codeS and postS... it's all it's fault :D

JD123

- 1st April 2008, 17:50

Sometime knowing a bit of ASM and MPASM directive is handy. I learned it through Darrel's codeS and postS... it's all it's fault :D
If you have to be corrupted by someone, Darrel seems to be a good choice! :)

skimask

- 2nd April 2008, 14:39

If you have to be corrupted by someone, Darrel seems to be a good choice! :)

Ya, no kidding, and of course, you're absolutely right about the 9 Tcyc.
A guy should never post 'from the hip', but I do it anyways...

If you use @ NOP?C 1 it will paste 1 NOP and so on.
And this little tidbit right here could be the solution to one of my ongoing 'issues' on another project...

mister_e

- 2nd April 2008, 14:50

Great! But you where wrong about the 9Tcy... ;) Read previous post again... or build an ASM project then watch the Program Memory window.

skimask

- 2nd April 2008, 15:08

Great! But you where wrong about the 9Tcy... ;) Read previous post again... or build an ASM project then watch the Program Memory window.
And yet again, that's what I get for posting off the hip... Since when is a macro...a subroutine? Must forgive my blatent stupidity...the house payment is due today :)
So, does that little @ NOP?C trick work any other little way that I'm not knowing of yet with anything else?

BTW...Self-Expanding Macro is my new buzzword...

mister_e

- 2nd April 2008, 15:19

So, does that little @ NOP?C trick work any other little way that I'm not knowing of yet with anything else?
Well i don't know.. i'm not in your own short :D

When you call a Macro, the compiler just copy/paste the whole macro code where you called the macro.

let see...

OneMacro macro
MOVE?CT 1,PORTA,0
MOVE?BB _MyVar,_YourVar
endm

EACH TIME you call the macro like this...

@ OneMacro

This will paste

MOVE?CT 1,PORTA,0
MOVE?BB _MyVar,_YourVar

so...

@ OneMacro
@ OneMacro

will obviously generate

MOVE?CT 1,PORTA,0
MOVE?BB _MyVar,_YourVar
MOVE?CT 1,PORTA,0
MOVE?BB _MyVar,_YourVar

Hence why we often suggest to reduce your macro size.

in NOP?C, The Conditional While (not sure if conditional or MPASM directive.. but anyway), just copy the amount of NOP you asked in the argument parameter

NOP?C 5

will juste paste
@ NOP
@ NOP
@ NOP
@ NOP
@ NOP

that's it. No code overhead, no nothing.... self expanding... or expandOnDemand ... huh?