HSEROUT of array of data with termination on a null

[jimbostlawrence]

Hi all,

Having an issue with outputting data from a pic to PC via a USB-serial COM port.

I have an array of 256 bytes (1048574 of these pages of bytes in fact!)

What I'd like to do is simply:

HSEROUT [STR page_array\256]

Can't do that as limited to 255 char's.

HSEROUT [STR page_array\128] works, but terminates if any of the bytes in the array are 0 (null). Just my luck!

Is there a way of combining HEX2 in this and stopping the null termination?

i.e. HSEROUT [HEX2 STR page_array\128]

Additionally, is there a way to output the second half of the page_array using an offset? i.e. I was wondering if there's a way to access the address of page_array and assign this+128 bytes to a different variable:

page_array2 = {address of}page_array + 128 {bytes}
HSEROUT [HEX2 STR page_array2\128]

?

Essentially, trying to offload up to 256 MB of data from a flash chip to VB on laptop over 115.2k BAUD connection. I know this will take some time. the more efficient the transfer the better. If anyone has experience in this, I'd love some help!

Many thanks in advance.

Jimbo

[jimbostlawrence]

Update:

Currently, as a short term solution, outputting the array as 16x HSEROUTs of 16x pairs of hex digits. Crude, but functional, and VB is able to convert this data back to its integer format at the other end. Not sure how much this slows things down compared to using a single HSEROUT [page_array\256] would be, or similar, were it possible to output a full 256 Byte array.

[tumbleweed]

If you're looking for more efficient then drop the formatting entirely and just send the binary data.
If you format 256MB of data using HEX2 you end up with twice the number of bytes to transfer (512MB), and it's slower because of the conversions.

At 115K each byte takes about 86us to send, so as long as your code doesn't add more than that in overhead it won't matter. The fastest way would be something like this:

i  var     byte
b  var     byte[256]

for i = 0 to 255
    ' wait for TRMT == 1
    while (TXSTA.1 = 0)
    wend
    ' send byte from array
    TXREG = b[i]
next

I think you also need to think about using handshaking (RTS/CTS). On the PC side I doubt that it can keep up with a 256MB stream of data.

[richard]

or setup printing as a background task and dump your buffer into the queue when its empty

'*****************************************************************
'*  Name    : PRINT_18_TX_ASM_INT.pbp                            *
'*  Author  : richard                                            *
'*  Notice  :                                                    *
'*          :                                                    *
'*  Date    :                                                    *
'*  Version :    18f26K22                                        *
'*  Notes   :                                                    *
'*          :                                                    *
'*                                                               *
'*****************************************************************  
  
#CONFIG
  CONFIG  FOSC=INTIO67, PLLCFG=OFF, PRICLKEN=OFF, FCMEN=OFF, IESO=OFF
  CONFIG  PWRTEN=OFF, BOREN=SBORDIS, BORV=190, WDTEN=ON, WDTPS=32768
  CONFIG  CCP2MX=PORTC1, PBADEN=OFF, CCP3MX=PORTB5, HFOFST=ON, T3CMX=PORTC0
  CONFIG  P2BMX=PORTB5, MCLRE=EXTMCLR, STVREN=ON, LVP=OFF, XINST=OFF, DEBUG=OFF
  CONFIG  CP0=OFF, CP1=OFF, CP2=OFF, CP3=OFF, CPB=OFF, CPD=OFF, WRT0=OFF
  CONFIG  WRT1=OFF, WRT2=OFF, WRT3=OFF, WRTC=OFF, WRTB=OFF, WRTD=OFF, EBTR0=OFF
  CONFIG  EBTR1=OFF, EBTR2=OFF, EBTR3=OFF, EBTRB=OFF
#ENDCONFIG

DEFINE OSC 64
DEFINE INTHAND do_tx
 
RCSTA = $90   ' Enable serial port & continuous receive
TXSTA = $24   ' Enable transmit, BRGH = 1
SPBRG = 130   ' 9600 Baud @ 64MHz, -0.02%
SPBRGH = 6
BAUDCON.3 = 1 ' Enable 16 bit baudrate generator

prnb var byte[256]
rpos VAR BYTE
empty VAR BYTE

TRISC=%11111111
OSCCON    = $70   ; 64Mhz
OSCTUNE.6 = 1     ; Enable 4x PLL 
while ! osccon2.7 :WEND ; to make sure the pll has stabilised before you run any other code 
ANSELA=0
ANSELC=0
intcon=$c0
empty=1

pause 1000
hserout ["ready",13,10 ] ;  PROOF OF LIFE
pause 1000

mainloop:
    if empty then
        ;hserout [13,10]
        arraywrite prnb[240],[13,10,0,"RTIOP12345678"]
        gosub print
    ENDIF
    pause 2000
goto   mainloop
 
'  SAMPLE OF RAW BINARY OUTPUT FROM TX PIN IN HEX
'  0D 0A 00 52 54 49 4F 50 31 32 33 34 35 36 37 38
'  0D 0A 00 52 54 49 4F 50 31 32 33 34 35 36 37 38
'  ..............


print: 
    rpos = 240   ;FOR TEST START AT INX 240 IE PRINT CHRS  240-255  IN BUFFER
    empty=0
    PIE1.4=1
return 
 
asm   
do_tx
    LFSR 0, _prnb   ; Store buffer ADDR to FSR0H     
    BANKSEL _rpos 
    MOVF   _rpos,W  ; Add rpos to pointer 
    ADDWF   FSR0L,F ; Store Low byte  in FSR0
    BNC  NOV
    INCF    FSR0H ,F
NOV 
    movf    INDF0,W
    movwf  TXREG   
    INCFSZ  _rpos,F ; Increment  rpos
    BRA    exP      ; if buffer empty turn ofF interrupt
    bcf    PIE1,TXIF
    BANKSEL _empty  ; AND SET FLAG
    BSF   _empty ,0
exP 
    BANKSEL 0
    RETFIE 
endasm

[tumbleweed]

richard's code has some merits, but in almost all cases interrupts just add overhead... they rarely speed things up if that's what you're looking for.

Here's another version of the code I posted with the array access time buried in the byte transmit time:

i  var     byte
b  var     byte[256]
txd var byte

for i = 0 to 255
    ' get byte from the array
    txd = b[i]
    ' wait for TRMT == 1
    while (TXSTA.1 = 0)
    wend
    ' send byte
    TXREG = txd
next

[Dave]

That's great but, at 9600 baud it will be in this routine for 26.5 milliseconds. That's OK if you can afford it...

[tumbleweed]

...trying to offload up to 256 MB of data from a flash chip to VB on laptop over 115.2k BAUD connection.

At 115K it takes ~22ms to send 256 bytes assuming back to back transfers. At 9600 it's over 10x longer than that.

You could use background serial transfers while you do something to read that flash chip, but you're going to have to know what you're doing to get all that working.

[amgen]

you could...... forget hserout, then you can make your data buffer bigger, use longs if your pic can do it.... 18f's, then write data to ram or program mem location that can store more than 256 bytes, then access data knowing starting location and loop through data with TX routine number of chars to send. I have sent 10K bytes at a time serially at 115K over 8266esp
I would dig up some code if interested in that technique.

[amgen]

on second thought, using longs, again requires an 18F device, you could just do one big loop......(up to 4,000,000 at a time) index is the long 32bit.
then using index, get byte, send it, get next byte send it,,,,,, up to about 4 million..... in send loop, check for exit char if needed.
pretty straight forward

[jimbostlawrence]

To all,

MANY thanks for all your help with this guys. I have some data recovery to do on another project but will be back on this either tonight or over the next few days.

I'm running this on an 18LF26J53 hooked up to a 256MB Micron NOR FLASH chip, together with a crystal running at the magical 22.1184 MHz, so 115k BAUD is nice and clean. Will give the binary / TXREG a shot and get back to you. I used to be into assember back in the 80's (6502 / BBC B, and 68000 / Amiga) but not gone that deep in a long time. I think for now, I'll stick high-level and only go that deep if needs must.

Thanks again. Will see what happens.

Much appreciation,

Jimbo

[jimbostlawrence]

Hi all,

using:

          
for cnt = 0 to 255
                
    while (TXSTA.1 = 0)
    wend         
                
    TXREG = page_buffer[cnt]
                
next

Worked very nicely.

Initially, as I was working with VB as well, that I'm not that familiar with, I decided to initially stick with the transfer of double hex chars per byte, working on the multiple 256-page byte capture and export to file. I then brought in the above code (thanks Tumbleweed!) on the PIC side to stream the data as binary, halving the number of transmitted bits. It was then a fun job to adapt the threaded side of things in VB, but finally got it working.

As hex char's, I calculated it to be about 23.5 hrs (!) to transfer the entire 256MB of data from the Micron chip. That's now down to around 14.5 hrs. Still a long time, but transfer is stable; no errors.

I could probably bring this down some more by buffering the multiple pages, rather than "file.Write(byte_array, 0, 256)" each page to file. I expected that with the number of char's halving, that the overall time would halve, so assume this is now the bottleneck.

Perseverance, sweat, copious amounts of coffee, n munch, goes a long way!

Many thanks again all.

Best,

Jimbo

[HenrikOlsson]

I'd try increasing the baudrate even further.
According to Wikipedia 256kbits/second is a predefined serial port speed in Windows so you "should" have that available without doing anything "special". If you put a RS485 tranceiver at each end you get up into the Mbits/seconds without much problem but then something like DMA on the PIC-side would start to make sense.

[tumbleweed]

Most PC serial ports will run at 230400 baud.
The J53 will... just set:
BRG16=1
BRGH=0
SPBRGH=0
SPBRG=5

That might shave it down to under 4 hrs.
Make sure the MAX232 (or whatever you're using) supports 230K baud... some transceivers don't.

If I were sending that much data I'd be inclined to use a transfer protocol like XMODEM or something.
Are you just blasting 256MB out the pic uart and hoping the PC will keep up?

[jimbostlawrence]

Hi all again,

I'll give that a shot Tumbleweed. Was wondering about possibly taking it higher. I'll give those parameters a shot. I'm hooking up to a FT231X chip on a small custom board. The laptop has kept up without a hitch at 115.2k BAUD no problem. Not running any ECC but will if need be.

Yup, blasting out of the PIC EUSART thingy! Running all at 2.0V. It's actually the 18LF26J53 (not sure if I mentioned that above). The FT231X is good down to 1.8 V I believe.

Best,

Jimbo

[jimbostlawrence]

Quick update - thanks again Tumbleweed. Now running at 230.4k BAUD.

Time for 256 MB download now at an estimated ~ 11.2 hrs.

The bottleneck will be the handshake as I'm "requesting" each page (256 Bytes) by sending a short stream of characters $$$8xxxxxxx where xxxxxxx is the page number. The PIC waits for the $$$ and then reads the page request value, transfers that page from memory to the PICs RAM and replies with the data. This way it's a very ordered delivery; perhaps too ordered!!!

I think I'll try having it send each block (or whatever its correct term is) of 65536 pages in one long stream (16 of them in all on that memory chip, making 1,048,576 pages x 256 Bytes = 256 MB). Just need to buffer for longer on the PC side. If I get any errors, I can just reduce the number to something more manageable and request corrupted pages.

Will let you know how it goes, and many thanks again!

Best,

Jimbo

[jimbostlawrence]

Ok, didn't go for the full 65,536 pages.

Set initially to transfer 16 pages which brought it down to something considerably less.

Then upped it to transfer 64 pages which only improved it by about 20 minutes.

** Final result: a few minutes over 5 hrs

This is probably slightly higher than your original estimate (@Tumbleweed) due to having to transfer each page (very brief but does add up I guess).

Thank you!!!

[tumbleweed]

Final result: a few minutes over 5 hrs

That's probably not too bad, considering you have to send packet requests to get data.

Doing that's a good idea... I was worried you were just going to stream 256MB out the pic and expect the PC to keep up.

[jimbostlawrence]

Hi Tumbleweed,

Looking at the DS (Pg 342 / Ex 21-1), the calc, using the crystal on this board:

(((22,118,400 / 230,400) / 64) - 1) = 0.5

When sent to the SPBRG, this will result in just 0

Reversing:

22,118,400 / (64 (0 + 1)) = 345,600 BAUD

Is that correct?

I've checked the output file on the PC side and it's all correct - I just substituted '65' instead of the data from the on-board memory, and the file is filled with 'A''s...

The fact that there is no errors - is this because it's asynchronous and the systems are just managing, or am I missing something here?

Edit - just found a site that says switching on the BRGH = 1 with SPBRG = 5, BRG16 = 1, I can achieve 921,600 BAUD. I vaguely remember doing this about 6 years ago with a really bespoke strip board setup! Well, just tested it and the data continues to stream through to file seamlessly.

2.46 hrs for 256 MB.

Will have another read of the DS as I'm definitely missing something here.

Edit!

Ok, just realised my ref for Ex 21-1 on Pg 342 of the DS was for 8-bit BRG!

The correct formula (for anyone that's following this path), with BRGH = 1 with SPBRG = 5, BRG16 = 1, and using a 22.1184 MHz Xtal, should be:

Fosc/(4 * (n + 1)) where n is the SPBRG value (5), so 22,118,400 / (4 * (5 + 1)) = 921,600 BAUD.

Perfect.

Best,

Jimbo

[tumbleweed]

Jimbo,

The pic can transmit at that rate but you might run into trouble receiving.
921K is only 10us/byte, so you'll have to be able to handle incoming data at that rate.

I've run 460K baud with FOSC=64MHz, but it gets pretty tight much faster than that.
YMMV