bootloader that xfers code from I2C into program memory

[picster]

This question will undoubtedly seem rudimentary and simple to the bootload gurus here...

I've seen a few posts here by people looking to transfer firmware upgrades from I2C to program memory.

This would be very handy, since the code to get the update from a PC and store it temporarily in I2C EEPROM prior to the actual "firmware update" can be done with PBP without using the hardware UART, and furthermore, encryption can be done (to the hex file before the end user downloads it).

As I understand it, the problem is that the instructions to do the actual writecode would have to be in high memory so it's not overwritten by the update process, and all the libraries are sitting in low memory so there's really no way to "stick" a subroutine written with PBP up there in high memory.

Is there any way to combine two .hex files so that one can sit in upper memory (with the necessary libraries included at that location) and somehow just be called by the "normal" program residing in LOW memory (at which point it's overwritten)? I know it's not very efficient from a codespace standpoint, but if the extra memory is just sitting there....

Thanks,

Picster

[pedja089]

I think that I posted somewhere code for that.
But here is my code that copy from external I2C mem to FLASH

    DEFINE RESET_ORG 1EC00h 'Move all library sub and code to start from location 125952
    EraseStr        VAR BYTE[128]
    Adr             VAR LONG
    MAdr            VAR WORD
    MCtrl           VAR BYTE
    FlashByte1      VAR BYTE
    FlashByte2      VAR BYTE
    FlashByte3      VAR BYTE
    Err             VAR BYTE
    BlockSize       VAR BYTE BANKA SYSTEM

    @ MovLW BLOCK_SIZE
    @ MovWF BlockSize

    Mem_Vcc=1
    PAUSE 6
    FOR Adr=0 TO 79999 STEP BlockSize
        ERASECODE Adr
    NEXT Adr
    FOR Adr=0 TO 79999
        MAdr=Adr.WORD0
        MCtrl=$A0 + ((Adr.BYTE2 & %00000011)<<1)
        Err=0
    BootI2CR:
        Err=Err+1
        IF Err>1 THEN PAUSE 1
        I2CREAD Mem_Sda,Mem_Scl,MCtrl,MAdr,[FlashByte1],BootI2CR
        IF Err>1 THEN PAUSE 1
        I2CREAD Mem_Sda,Mem_Scl,MCtrl,MAdr,[FlashByte2],BootI2CR
        IF Err>1 THEN PAUSE 1
        I2CREAD Mem_Sda,Mem_Scl,MCtrl,MAdr,[FlashByte3],BootI2CR
        IF FlashByte1<>FlashByte2 THEN GOTO BootI2CR
        IF FlashByte1<>FlashByte3 THEN GOTO BootI2CR
        WRITECODE Adr, FlashByte1
    NEXT Adr
    'Optional Erase I2C EEPROM
    FOR Adr=0 TO 79999 STEP 128
        MAdr=Adr.WORD0
        MCtrl=$A0 + ((Adr.BYTE2 & %00000011)<<1)
     BootI2CW:
        I2CWRITE Mem_Sda,Mem_Scl,MCtrl,MAdr,[STR EraseStr\128],BootI2CW
        PAUSE 5
    NEXT Adr
    INPUT Mem_Sda
    INPUT Mem_Scl
    Mem_Vcc=0
    @ RESET

For first programming I joint two hex files manually.
First compile bootloader, then open hex in MPLAB X, and copy code from program memory window from 1EC00h to end of bootloader and then format it and put at end of main app.

@ ORG 1EC00h
StartBootloader:
ASM
    dw 06A01h, 0EFEBh, 0F0F6h, 08A01h, 0AA01h, 09A01h, 0B601h, 0D003h, 08601h, 0D851h, 0E234h, 00E08h, 06E18h, 0D861h, 03617h, 02E18h
    dw 0D7FCh, 0AA01h, 0D83Bh, 0D869h, 0D862h, 0BA01h, 0D828h, 0D82Eh, 05017h, 090D8h, 0EFE8h, 0F0F6h, 08C01h, 0AC01h, 09C01h, 06ED9h
    dw 00603h, 0E203h, 0BC01h, 0D01Bh, 0D018h, 050DEh, 0D804h, 0E215h, 0D7F7h, 08A01h, 0AA01h, 09A01h, 0B401h, 0D005h, 00BFEh, 06E0Ch
    dw 08401h, 090D8h, 0D00Ah, 0B801h, 0D005h, 06E1Fh, 08801h, 0D823h, 0E206h, 0501Fh, 0D827h, 0E203h, 0AA01h, 0EFE8h, 0F0F6
    ...........
ENDASM

In Main application I use GOTO StartBootloader to run bootloader.
In last project I had lot of free space in pic, and use Flash to Flash bootloader.

DEFINE RESET_ORG .65024

Adr VAR WORD
Tmp VAR BYTE
BlockSize       VAR BYTE BANKA SYSTEM
@ MovLW BLOCK_SIZE
@ MovWF BlockSize

FOR Adr=0 TO 32255 STEP BlockSize
    ERASECODE Adr
NEXT Adr

FOR Adr=0 TO 32255 
    Adr.15=1
    READCODE Adr,Tmp
    Adr.15=0
    WRITECODE Adr,Tmp
NEXT Adr
@ RESET

EDIT:
I didn't point out that using this method all library for bootloader are stored after RESET_ORG vector.
This way it uses little bit more code space(because you probably have same library in main application), but this makes implementation of bootloader and main app as simple as possible.

[picster]

Thanks pedja089, that's awesome - is that using an 18F PIC? Do you know if this can be done using the 16F1xxx series? From what I've read, the DEFINE RESET ORG option is only functional on the 18's - does that sound right?

Can anyone else confirm?

[pedja089]

I use 18F series for all my project.
You can just compile some code for 16F, something like
DEFINE RESET_ORG 16
pause 1
And you should get 16 bytes FF then some other values. Also to view content of hex file you could use any programmer software, but I prefer MPLAB X, because I can export HEX values of just piece of hex to text file, and format it to ASM sintax.
Then let us know
I don't know what 16F part are able to do flash write. And also flash size is critical.
So I think any way, better to go with some 18F that have twice flash you need, so you could use flash to flash bootloader. It's really neat, small, simple and doesn't require any external parts!
All this can be done in ASM to get even smaller bootloaders, but I just didn't bother with that.
By the way all 18F can do code erase, write and read eaven if it is locked. Code protect bit affect only ICSP.

[picster]

This is definitely a great starting point. I'll test with some simple stuff and then see how it works out on the PIC I'm using. Might be just what I needed.

picster

[picster]

Ok so looking at the code, I had a couple of questions - I'm hoping you can clarify for me.

1) Flashbyte2 and Flashbyte3 - redundant copies, with the same data 3 times in i2c eeprom?

2) What is the purpose of EraseStr VAR BYTE[128]? I don't see it elsewhere in the pasted code.

3) Lastly, in the ASM portion, you start each line with dw - is this just to "define words" (hex format little-endian) that will simply sit there in memory until you jump to the label?

I really appreciate your posting this, it's hugely helpful.

Picster

[pedja089]

1) Redundant read of same memory location. Just to be sure that there is no error in communication. Same at writing, write 32 bytes, then read them, and compare. Just to on safe side.
2) Scroll little bit down

'Optional Erase I2C EEPROM
    FOR Adr=0 TO 79999 STEP 128
        MAdr=Adr.WORD0
        MCtrl=$A0 + ((Adr.BYTE2 & %00000011)<<1)
     BootI2CW:
        I2CWRITE Mem_Sda,Mem_Scl,MCtrl,MAdr,[STR EraseStr\128],BootI2CW
        PAUSE 5

Just easy way to dump 128 zeros in memory.
3) Not sure what you are asking, but that is code for bootloader in main app. That code will sit there forever. And execute when jump to label.
Output of first 32 line from random project in MPLAB X program windows(right click output to file)

       Line      Address       Opcode           Label                        DisAssy                 
        1    00000         EF30                             GOTO 0x260                               
        2    00002         F001                             NOP                                      
        3    00004         FFFF                             NOP                                      
        4    00006         FFFF                             NOP                                      
        5    00008         0012                             RETURN 0                                 
        6    0000A         FFFF                             NOP                                      
        7    0000C         FFFF                             NOP                                      
        8    0000E         FFFF                             NOP                                      
        9    00010         FFFF                             NOP                                      
       10    00012         FFFF                             NOP                                      
       11    00014         FFFF                             NOP                                      
       12    00016         FFFF                             NOP                                      
       13    00018         0012                             RETURN 0                                 
       14    0001A         010F         HSEROUT             MOVLB 0xF                                
       15    0001C         0004         hseroutloop         CLRWDT                                   
       16    0001E         A89E                             BTFSS PIR1, 4, ACCESS                    
       17    00020         D7FD                             BRA hseroutloop

Similar thing you will get when compile bootloader, but at different start address. And you need to copy opcode column to to your main program and address must lineup.
I'm not sure that I answered your question.

[picster]

Got it, thanks.

Makes sense

picster

[pedja089]

Great!
Did you try to compile code for 16F with RESET_ORG?

[picster]

YUP, and it worked fine Had to download MPLABX before continuing...

I'll be writing a small program for Windows to xfer the code via RS-232 INTO the I2C EEPROM (which the Pic's main program will manage), in order to test it properly. Weekend stuff.

Will let you know how it works out, but this looks really promising, thanks again!

[pedja089]

Take look at microchip AN851. That is VB6 application for bootloader.
I used that as starting point for my VB .net app.
Other solution is to export hex file as text, eg from MPLAB X program windows or some programmer software.

[picster]

Will have a look - I use VB6 a fair bit so that should work out well. Thx!

[picster]

Ok, so far so good with a little test program for calling code that's pre-compiled in upper program memory, so I think I'm on track.

Item of note for the 16F series (I'm using the 16F1788):

DEFINE RESET_ORG 36B0h
*****also needs*****
DEFINE USE_LINKER 1

Both lines must be present when generating the "upper memory code" above 800h.
(otherwise there's a compile error re: library exceeding address 800h - found this little gem here:
http://www.picbasic.co.uk/forum/show...+cannot+exceed )

I generated the complied hex, then copied it out of MPLAB, pasted in a text file, did a little search/replace and editing, and pasted the result at the end of the program as ASM as you demonstrated...

used a goto to jump to it at the beginning of my program, and it worked like a charm.

Looking at the resulting compiled hex shows the "upper space subroutine" right where it should be.

I'll let you know how the xfer from i2c goes, but I can't foresee any issues at this rate.

[pedja089]

Great!
I started with led blink, and worked from there.
Good luck, and keep us posted

[picster]

Got an LED flasher working from I2Cread->writecode with the program that performs the update in upper memory per the above.

One subtle difference:

Writecode on a PIC16Fxxxx is word length, so has to be transferred 2 bytes at a time from I2C to ONE codespace address

this works fine, just reading data out of the i2c sequentially as WORDS from every 2nd location:

' Now get data from I2C and write it to codespace
for CodeAddr=0 to 13919
I2CAddr=2*CodeAddr
i2cread PORTB.7, PORTB.6, I2CControl, I2CAddr, [Instruction], I2Cerror
writecode CodeAddr, Instruction
next CodeAddr

Thanks again!

picster

[pedja089]

Great!
Did you try to play with VB?
I didn't leave copy of translated code, and I don't want to post whole code, because of encryption method I implemented on file and communication.
And now it would be too much to do to remove all code for that, and just recreate code for hex import.

[picster]

Yeah I already had a VB program I'd written for dumping data to the PIC to store in I2CEEPROM for lookup, so this is just more of the same. Encryption will be pretty simple to add.

Upper memory code to reprogram from I2C is sitting at less than 1k in size, and that's including LCD user prompts.

Nice. Very nice.

picster

[pedja089]

I think other users would appreciate if you share your work.
Thanks!

[picster]

I think you mean YOUR work!

Without posting a VB "updater" (which is going to be very application specific for this case), there's really not much to post.

The idea here is to create a "firmware update file" from a master, so it can be distributed in the field or wherever.

Suffice to say that if your main program has a hidden "dump all my code out via serial" subroutine, then you don't have to muck about with interpreting intel hex files, you just get raw code (instructions) from memory location A to memory location B from your "master" that you've updated with your PIC programmer (the upper memory program is always included with this).

Once you have that "master" file captured by a PC, you can send it back out its serial port to be intercepted by the "main program" on any unit to be UPDATED, and stored on I2C. After checking the checksum, then you call the "reprogram" code sitting in upper memory to rewrite all the lower code, stopping shy of the "tucked" upper subroutine location. Encryption is just a matter of manipulation at the PC end after receiving the original dump, and reverse-manipulation at the PIC end as you pull the words back out of I2C (before doing the codewrite).

[pedja089]

It didn't get to me that you can easily dump PIC code from programed PIC if you don't want to muck with hex.
That is great idea!

[picster]

See clarification below....

I think you mean YOUR work!

Without posting a VB "updater" (which is going to be very application specific for this case), there's really not much to post.

The idea here is to create a "firmware update file" from a master, so it can be distributed in the field or wherever.

Suffice to say that if your main program has a hidden "dump all my code out via serial" subroutine, then you don't have to muck about with interpreting intel hex files, you just get raw code (instructions) from memory location A to memory location B from your "master" that you've updated with your PIC programmer (the upper memory program is always included with this) -clarification: the MASTER chip always has the upper memory subroutine sent to it via the PIC programmer - however, the only part you "dump" (and codewrite in the field) is the part PRIOR to that subroutine!!).

Once you have that "master" file captured by a PC, you can send it back out its serial port to be intercepted by the "main program" on any unit to be UPDATED, and stored on I2C. After checking the checksum, then you call the "reprogram" code sitting in upper memory to rewrite all the lower code, stopping shy of the "tucked" upper subroutine location. Encryption is just a matter of manipulation at the PC end after receiving the original dump, and reverse-manipulation at the PIC end as you pull the words back out of I2C (before doing the codewrite).

[picster]

Yeah well, at least I contributed SOMETHING! LOL

This is a major game-changer for field updates. The approach should also work fine for PIC16F88's.

I'll try to create a somewhat "generic" windows firmware updater based on this approach, with VB6 in the next little while and post it for others to use. It will only be able to Xfer a file IN (PIC readcode dump-->PC), and send a file OUT (PC-->PIC firmware update). It will NOT have encryption built into it, since that's also critically dependent upon a PIC decryption subroutine.

Did you manage to successfully pass variables from your "main program" to your "bootloader" section? If so, did you declare them in BOTH, or was that the purpose of your BLOCKSIZE declaration as a system variable, so it would be accessible for the upper memory AND the resident program?

Thanks,

picster

[pedja089]

I did, but via EEPROM and other way. So main app knows when it is first run, so it can configure I2C memory, etc...
But you can define variables at specific addresses.
From manual:
wsave VAR BYTE $70 'Creates "wsave" at RAM address 0x70 (hex)
Just create variable at address $70 both in main app and bootloader. And load it in main app before jumping to bootloader, and use it in bootloader. But you must be careful not to clear it, before use.

BLOCK_SIZE is ASM constant for each pic. And it tell how long is one block or page in FLASH.
So to get this information to PBP, I used this code:

@ MovLW BLOCK_SIZE ;move literal to W
@ MovWF BlockSize ;Move W to File(variable)

If variable is not defined as system

    BlockSize       VAR BYTE BANKA

    @ MovLW BLOCK_SIZE
    @ MovWF _BlockSize

BANKA must be used, or in ASM you need to set correct bank.
This can be done in few ways, but i don't like that. All my ASM routines use variables in BANKA, so I doesn't have to deal with selecting bank.
SYSTEM just removes underscore in ASM variable name. Default behavior of PBP is to put _ in front of name. Eg if you declare A var byte, in ASM variable is called _A. With modifiers SYSTEM PBP will create same name for ASM.
Just habit that I have to put all ASM variables to BANKA SYSTEM...
[EDIT]This also could be done with EXT modifier.[/EDIT]

FLASH is erased and write ONE PAGE at TIME. PBP will track address that you are writing, and when it align with page it will dump all data to FLASH.
Data are temporally stored to TBL. So if you just put one WRITECODE instruction, your byte or word wont be written to FLASH. It took me some while to get it. At least for PIC18F.
If you measure how much time it takes to do instruction WRITECODE, you will see eg 63 very fast(10-20 uS, depending on oscillator speed), and 64th will take about 5mS(doesn't depend on oscillator). This is if your page size is 64 byte. Don't hold me to exact figures, I can remember just ballpark value.

[picster]

Ok, thanks - so variables can be transferred either via EEPROM (which is bidirectional [main program<-->updater] since it stays put after reset) or via variables assigned in BANKA (which is uni-directional, since it's cleared upon reboot after the lower code is rewritten).

You determined that BLOCK_SIZE was a constant by looking at the INCLUDE file for the pic, right?

I haven't tried it, but do you think it should be possible to use:

Blocksize VAR BYTE $70
Blocksize=BLOCK_SIZE

and then pass it that way, since BLOCK_SIZE is pre-defined in the device's include file?

[pedja089]

You get something wrong, probably because my bad English
Constant BLOCK_SIZE is defined in microchip MPLAB, and doesn't have anything with PBP. So PBP compiler won't be aware of BLOCK_SIZE and will throw an error.
(And I can't remember where I found it, I think it was used in some PBP lib file)

[EDIT]BLOCK_SIZE is defined in includes that are located at PBP3\DEVICES. But this is ASM level include as far as I know. It is just added to .lst file after compiling. So compiler isn't aware of this file.[/EDIT]

But if you use it in ASM, compiler will just forward your ASM line to .LST file. And .LST file will be assembled into ASM and then in HEX file.
So to be clear(or try to be), BLOCK_SIZE is in assembler include, BlockSize is any variable in PBP preferably in BANKA.
There is two way, one I used is to move constant to variable in ASM, and other is similar to yours, but you must use EXT to tell PBP compiler that your constant is declared somewhere else, but assembler can reach it.
That should look like this
BLOCK_SIZE CON EXT
BlockSize VAR BYTE
BlockSize=BLOCK_SIZE
But then you don't need variable BlockSize, because you can use constant BLOCK_SIZE.
I didn't try this, I just know what I learned from Darrel's post about EXT.

To pass variable from main app to bootloader, variable doesn't have to be in any specific bank, it must be at specific address.
In main app:
VarAtAdr50 VAR BYTE 50 'This is variable declared at decimal address 50
....
VarAtAdr50=10
GOTO StartBootloader

In Bootloader:
VarAtAdr50 VAR BYTE 50
If VarAtAdr50=10 THEN ....
I hope that this clears things, if not can somebody else try to explain it better? Thanks, in advance.

[picster]

Ok pedja089, so to summarize...


1) Declaring a variable at the same RAM location for BOTH the "bootloader" AND the main program will result in that variable's contents being "preserved" and accessible by the "bootloader".

MySharedVar1 var byte $40 'assigns the byte variable at RAM location $40 - this line would be included in both the bootloader (prior to compiling/copying) and the main program

Setting the RAM location doesn't CLEAR the variable, so when it's subsequently set at the same RAM location (again) when the bootloader is started, the contents remain the same (they're preserved).


2) An ASM level variable (such as BLOCK_SIZE, the value of which is defined in the .INC file for the device) can only be accessed via ASM, but it can be passed to a "preserved" variable like in #1.

BlockSize var byte $40
@ MovLW BLOCK_SIZE
@ MovWF _BlockSize

This loads the assigned value (from the INCLUDE file) into the designated location for the PBP variable "BlockSize".

Anything I'm missing here?

[pedja089]

1) Yes, but write to that address in RAM can corrupt value(eg CLEAR instruction in bootloader). Be careful.
2) Yes and No.
Yes, it can be accessed via ASM. But that isn't ONLY way. Other is via modifier EXT. More about EXT: http://www.picbasic.co.uk/forum/showthread.php?t=3891
BLOCK_SIZE is constant, not variable. And BlockSize is best to leave to compiler to allocate variable as all other "regular variables", but in BANKA. BlockSize VAR BYTE BANKA.
I think, but not sure, that is best to leave to compiler to allocate variable. If you access lot to that variable then it should be in BANKA, or to simplify ASM code. And if you must, then you specify location.

[picster]

Without getting into the EXT modifier,

Yes, I agree - BLOCK_SIZE is a constant, but still treated "like" a variable when passed from one side to the other in this manner. The point here being, it will change and be assigned the value according to the INCLUDE file for the PIC.

So if I just do:

BlockSize VAR BYTE BANKA

in both the bootloader and the main program, will they both allocate to the same RAM location automatically? I know they'll do the same BANK, but the same actual location? Wouldn't I need to specify the same location in BANKA for both to ensure I'm grabbing the same byte from RAM?

I'm really not so concerned about doing it with BLOCK_SIZE because it IS a constant that I can just declare in my bootloader as 32 - but I'm thinking about other variables that might come in handy to "pass" from the lower program to higher program.

[pedja089]

I think that now I understand what you didn't understand about BLOCK_SIZE.
First main app usually doesn't have to know what is block size. Only if you use FLASH2FLASH bootloader you must know what is block size.
Bootloader should know what block size is, so you can erase pic page by page.

Because BLOCK_SIZE is used in internally PBP library for CODE WRITE/ERASE, you MUST compile bootloader for each PIC.
So there is no need(also there is no way to pass constant at runtime in LIB) to pass BLOCK_SIZE from main app to bootloader app.
You can't use same HEX for different pic's, for some you can, but for others you can't. Because register map isn't same for all pic.
So no to take any chance, just compile bootloader for different pics and don't worry about PAGE_SIZE.

Conclusion is that you can use same code for almost all PICs, but you can't use same HEX. You must compile, extract and edit bootloader HEX for each PIC you want to use.

[picster]

Agreed - just the bootloader needs it.

If you're passing OTHER stuff (variables), then you'd want to assign the same location in BOTH files, correct?

[pedja089]

Yes. It must be same location and same size.