HSEROUT of array of data with termination on a null


+ Reply to Thread
Results 1 to 19 of 19
  1. #1
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default HSEROUT of array of data with termination on a null

    Hi all,

    Having an issue with outputting data from a pic to PC via a USB-serial COM port.

    I have an array of 256 bytes (1048574 of these pages of bytes in fact!)

    What I'd like to do is simply:

    HSEROUT [STR page_array\256]

    Can't do that as limited to 255 char's.

    HSEROUT [STR page_array\128] works, but terminates if any of the bytes in the array are 0 (null). Just my luck!

    Is there a way of combining HEX2 in this and stopping the null termination?

    i.e. HSEROUT [HEX2 STR page_array\128]

    Additionally, is there a way to output the second half of the page_array using an offset? i.e. I was wondering if there's a way to access the address of page_array and assign this+128 bytes to a different variable:

    page_array2 = {address of}page_array + 128 {bytes}
    HSEROUT [HEX2 STR page_array2\128]

    ?

    Essentially, trying to offload up to 256 MB of data from a flash chip to VB on laptop over 115.2k BAUD connection. I know this will take some time. the more efficient the transfer the better. If anyone has experience in this, I'd love some help!

    Many thanks in advance.

    Jimbo

  2. #2
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default Re: HSEROUT of array of data with termination on a null

    Update:

    Currently, as a short term solution, outputting the array as 16x HSEROUTs of 16x pairs of hex digits. Crude, but functional, and VB is able to convert this data back to its integer format at the other end. Not sure how much this slows things down compared to using a single HSEROUT [page_array\256] would be, or similar, were it possible to output a full 256 Byte array.

  3. #3

    Default Re: HSEROUT of array of data with termination on a null

    If you're looking for more efficient then drop the formatting entirely and just send the binary data.
    If you format 256MB of data using HEX2 you end up with twice the number of bytes to transfer (512MB), and it's slower because of the conversions.

    At 115K each byte takes about 86us to send, so as long as your code doesn't add more than that in overhead it won't matter. The fastest way would be something like this:
    Code:
    i  var     byte
    b  var     byte[256]
    
    for i = 0 to 255
        ' wait for TRMT == 1
        while (TXSTA.1 = 0)
        wend
        ' send byte from array
        TXREG = b[i]
    next
    I think you also need to think about using handshaking (RTS/CTS). On the PC side I doubt that it can keep up with a 256MB stream of data.

  4. #4
    Join Date
    May 2013
    Location
    australia
    Posts
    1,731

    Default Re: HSEROUT of array of data with termination on a null

    or setup printing as a background task and dump your buffer into the queue when its empty

    Code:
    '*****************************************************************
    '*  Name    : PRINT_18_TX_ASM_INT.pbp                            *
    '*  Author  : richard                                            *
    '*  Notice  :                                                    *
    '*          :                                                    *
    '*  Date    :                                                    *
    '*  Version :    18f26K22                                        *
    '*  Notes   :                                                    *
    '*          :                                                    *
    '*                                                               *
    '*****************************************************************  
      
    #CONFIG
      CONFIG  FOSC=INTIO67, PLLCFG=OFF, PRICLKEN=OFF, FCMEN=OFF, IESO=OFF
      CONFIG  PWRTEN=OFF, BOREN=SBORDIS, BORV=190, WDTEN=ON, WDTPS=32768
      CONFIG  CCP2MX=PORTC1, PBADEN=OFF, CCP3MX=PORTB5, HFOFST=ON, T3CMX=PORTC0
      CONFIG  P2BMX=PORTB5, MCLRE=EXTMCLR, STVREN=ON, LVP=OFF, XINST=OFF, DEBUG=OFF
      CONFIG  CP0=OFF, CP1=OFF, CP2=OFF, CP3=OFF, CPB=OFF, CPD=OFF, WRT0=OFF
      CONFIG  WRT1=OFF, WRT2=OFF, WRT3=OFF, WRTC=OFF, WRTB=OFF, WRTD=OFF, EBTR0=OFF
      CONFIG  EBTR1=OFF, EBTR2=OFF, EBTR3=OFF, EBTRB=OFF
    #ENDCONFIG
    
    DEFINE OSC 64
    DEFINE INTHAND do_tx
     
    RCSTA = $90   ' Enable serial port & continuous receive
    TXSTA = $24   ' Enable transmit, BRGH = 1
    SPBRG = 130   ' 9600 Baud @ 64MHz, -0.02%
    SPBRGH = 6
    BAUDCON.3 = 1 ' Enable 16 bit baudrate generator
    
    prnb var byte[256]
    rpos VAR BYTE
    empty VAR BYTE
    
    TRISC=%11111111
    OSCCON    = $70   ; 64Mhz
    OSCTUNE.6 = 1     ; Enable 4x PLL 
    while ! osccon2.7 :WEND ; to make sure the pll has stabilised before you run any other code 
    ANSELA=0
    ANSELC=0
    intcon=$c0
    empty=1
    
    pause 1000
    hserout ["ready",13,10 ] ;  PROOF OF LIFE
    pause 1000
    
    mainloop:
        if empty then
            ;hserout [13,10]
            arraywrite prnb[240],[13,10,0,"RTIOP12345678"]
            gosub print
        ENDIF
        pause 2000
    goto   mainloop
     
    '  SAMPLE OF RAW BINARY OUTPUT FROM TX PIN IN HEX
    '  0D 0A 00 52 54 49 4F 50 31 32 33 34 35 36 37 38
    '  0D 0A 00 52 54 49 4F 50 31 32 33 34 35 36 37 38
    '  ..............
    
    
    print: 
        rpos = 240   ;FOR TEST START AT INX 240 IE PRINT CHRS  240-255  IN BUFFER
        empty=0
        PIE1.4=1
    return 
     
    asm   
    do_tx
        LFSR 0, _prnb   ; Store buffer ADDR to FSR0H     
        BANKSEL _rpos 
        MOVF   _rpos,W  ; Add rpos to pointer 
        ADDWF   FSR0L,F ; Store Low byte  in FSR0
        BNC  NOV
        INCF    FSR0H ,F
    NOV 
        movf    INDF0,W
        movwf  TXREG   
        INCFSZ  _rpos,F ; Increment  rpos
        BRA    exP      ; if buffer empty turn ofF interrupt
        bcf    PIE1,TXIF
        BANKSEL _empty  ; AND SET FLAG
        BSF   _empty ,0
    exP 
        BANKSEL 0
        RETFIE 
    endasm
    This is more entertaining than Free to Air TV

  5. #5

    Default Re: HSEROUT of array of data with termination on a null

    richard's code has some merits, but in almost all cases interrupts just add overhead... they rarely speed things up if that's what you're looking for.

    Here's another version of the code I posted with the array access time buried in the byte transmit time:
    Code:
    i  var     byte
    b  var     byte[256]
    txd var byte
    
    for i = 0 to 255
        ' get byte from the array
        txd = b[i]
        ' wait for TRMT == 1
        while (TXSTA.1 = 0)
        wend
        ' send byte
        TXREG = txd
    next

  6. #6
    Join Date
    Mar 2003
    Location
    Commerce Michigan USA
    Posts
    1,162

    Default Re: HSEROUT of array of data with termination on a null

    That's great but, at 9600 baud it will be in this routine for 26.5 milliseconds. That's OK if you can afford it...
    Dave Purola,
    N8NTA
    EN82fn

  7. #7

    Default Re: HSEROUT of array of data with termination on a null

    ...trying to offload up to 256 MB of data from a flash chip to VB on laptop over 115.2k BAUD connection.
    At 115K it takes ~22ms to send 256 bytes assuming back to back transfers. At 9600 it's over 10x longer than that.

    You could use background serial transfers while you do something to read that flash chip, but you're going to have to know what you're doing to get all that working.

  8. #8

    Default Re: HSEROUT of array of data with termination on a null

    you could...... forget hserout, then you can make your data buffer bigger, use longs if your pic can do it.... 18f's, then write data to ram or program mem location that can store more than 256 bytes, then access data knowing starting location and loop through data with TX routine number of chars to send. I have sent 10K bytes at a time serially at 115K over 8266esp
    I would dig up some code if interested in that technique.

  9. #9

    Default Re: HSEROUT of array of data with termination on a null

    on second thought, using longs, again requires an 18F device, you could just do one big loop......(up to 4,000,000 at a time) index is the long 32bit.
    then using index, get byte, send it, get next byte send it,,,,,, up to about 4 million..... in send loop, check for exit char if needed.
    pretty straight forward

  10. #10
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default Re: HSEROUT of array of data with termination on a null

    To all,

    MANY thanks for all your help with this guys. I have some data recovery to do on another project but will be back on this either tonight or over the next few days.

    I'm running this on an 18LF26J53 hooked up to a 256MB Micron NOR FLASH chip, together with a crystal running at the magical 22.1184 MHz, so 115k BAUD is nice and clean. Will give the binary / TXREG a shot and get back to you. I used to be into assember back in the 80's (6502 / BBC B, and 68000 / Amiga) but not gone that deep in a long time. I think for now, I'll stick high-level and only go that deep if needs must.

    Thanks again. Will see what happens.

    Much appreciation,

    Jimbo

  11. #11
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Cool Re: HSEROUT of array of data with termination on a null

    Hi all,

    using:

    Code:
              
    for cnt = 0 to 255
                    
        while (TXSTA.1 = 0)
        wend         
                    
        TXREG = page_buffer[cnt]
                    
    next
    Worked very nicely.

    Initially, as I was working with VB as well, that I'm not that familiar with, I decided to initially stick with the transfer of double hex chars per byte, working on the multiple 256-page byte capture and export to file. I then brought in the above code (thanks Tumbleweed!) on the PIC side to stream the data as binary, halving the number of transmitted bits. It was then a fun job to adapt the threaded side of things in VB, but finally got it working.

    As hex char's, I calculated it to be about 23.5 hrs (!) to transfer the entire 256MB of data from the Micron chip. That's now down to around 14.5 hrs. Still a long time, but transfer is stable; no errors.

    I could probably bring this down some more by buffering the multiple pages, rather than "file.Write(byte_array, 0, 256)" each page to file. I expected that with the number of char's halving, that the overall time would halve, so assume this is now the bottleneck.

    Perseverance, sweat, copious amounts of coffee, n munch, goes a long way!

    Many thanks again all.

    Best,

    Jimbo

  12. #12
    Join Date
    Oct 2005
    Location
    Sweden
    Posts
    3,283

    Default Re: HSEROUT of array of data with termination on a null

    I'd try increasing the baudrate even further.
    According to Wikipedia 256kbits/second is a predefined serial port speed in Windows so you "should" have that available without doing anything "special". If you put a RS485 tranceiver at each end you get up into the Mbits/seconds without much problem but then something like DMA on the PIC-side would start to make sense.

  13. #13

    Default Re: HSEROUT of array of data with termination on a null

    Most PC serial ports will run at 230400 baud.
    The J53 will... just set:
    BRG16=1
    BRGH=0
    SPBRGH=0
    SPBRG=5

    That might shave it down to under 4 hrs.
    Make sure the MAX232 (or whatever you're using) supports 230K baud... some transceivers don't.

    If I were sending that much data I'd be inclined to use a transfer protocol like XMODEM or something.
    Are you just blasting 256MB out the pic uart and hoping the PC will keep up?

  14. #14
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default Re: HSEROUT of array of data with termination on a null

    Hi all again,

    I'll give that a shot Tumbleweed. Was wondering about possibly taking it higher. I'll give those parameters a shot. I'm hooking up to a FT231X chip on a small custom board. The laptop has kept up without a hitch at 115.2k BAUD no problem. Not running any ECC but will if need be.

    Yup, blasting out of the PIC EUSART thingy! Running all at 2.0V. It's actually the 18LF26J53 (not sure if I mentioned that above). The FT231X is good down to 1.8 V I believe.

    Best,

    Jimbo

  15. #15
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default Re: HSEROUT of array of data with termination on a null

    Quick update - thanks again Tumbleweed. Now running at 230.4k BAUD.

    Time for 256 MB download now at an estimated ~ 11.2 hrs.

    The bottleneck will be the handshake as I'm "requesting" each page (256 Bytes) by sending a short stream of characters $$$8xxxxxxx where xxxxxxx is the page number. The PIC waits for the $$$ and then reads the page request value, transfers that page from memory to the PICs RAM and replies with the data. This way it's a very ordered delivery; perhaps too ordered!!!

    I think I'll try having it send each block (or whatever its correct term is) of 65536 pages in one long stream (16 of them in all on that memory chip, making 1,048,576 pages x 256 Bytes = 256 MB). Just need to buffer for longer on the PC side. If I get any errors, I can just reduce the number to something more manageable and request corrupted pages.

    Will let you know how it goes, and many thanks again!

    Best,

    Jimbo

  16. #16
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default Re: HSEROUT of array of data with termination on a null

    Ok, didn't go for the full 65,536 pages.

    Set initially to transfer 16 pages which brought it down to something considerably less.

    Then upped it to transfer 64 pages which only improved it by about 20 minutes.

    ** Final result: a few minutes over 5 hrs

    This is probably slightly higher than your original estimate (@Tumbleweed) due to having to transfer each page (very brief but does add up I guess).

    Thank you!!!

  17. #17

    Default Re: HSEROUT of array of data with termination on a null

    Final result: a few minutes over 5 hrs
    That's probably not too bad, considering you have to send packet requests to get data.

    Doing that's a good idea... I was worried you were just going to stream 256MB out the pic and expect the PC to keep up.

  18. #18
    Join Date
    Sep 2009
    Location
    South Wales (UK)
    Posts
    62

    Default Re: HSEROUT of array of data with termination on a null

    Hi Tumbleweed,

    Looking at the DS (Pg 342 / Ex 21-1), the calc, using the crystal on this board:

    (((22,118,400 / 230,400) / 64) - 1) = 0.5

    When sent to the SPBRG, this will result in just 0

    Reversing:

    22,118,400 / (64 (0 + 1)) = 345,600 BAUD

    Is that correct?

    I've checked the output file on the PC side and it's all correct - I just substituted '65' instead of the data from the on-board memory, and the file is filled with 'A''s...

    The fact that there is no errors - is this because it's asynchronous and the systems are just managing, or am I missing something here?

    Edit - just found a site that says switching on the BRGH = 1 with SPBRG = 5, BRG16 = 1, I can achieve 921,600 BAUD. I vaguely remember doing this about 6 years ago with a really bespoke strip board setup! Well, just tested it and the data continues to stream through to file seamlessly.

    2.46 hrs for 256 MB.

    Will have another read of the DS as I'm definitely missing something here.

    Edit!

    Ok, just realised my ref for Ex 21-1 on Pg 342 of the DS was for 8-bit BRG!

    The correct formula (for anyone that's following this path), with BRGH = 1 with SPBRG = 5, BRG16 = 1, and using a 22.1184 MHz Xtal, should be:

    Fosc/(4 * (n + 1)) where n is the SPBRG value (5), so 22,118,400 / (4 * (5 + 1)) = 921,600 BAUD.

    Perfect.

    Best,

    Jimbo
    Last edited by jimbostlawrence; - 12th October 2020 at 19:11.

  19. #19

    Default Re: HSEROUT of array of data with termination on a null

    Jimbo,

    The pic can transmit at that rate but you might run into trouble receiving.
    921K is only 10us/byte, so you'll have to be able to handle incoming data at that rate.

    I've run 460K baud with FOSC=64MHz, but it gets pretty tight much faster than that.
    YMMV

Similar Threads

  1. Null statement (NOP or CONTINUE)
    By RussMartin in forum General
    Replies: 8
    Last Post: - 4th February 2009, 22:28
  2. How to get ASCII data in array.
    By Ioannis in forum Serial
    Replies: 8
    Last Post: - 6th November 2008, 21:14
  3. Bit/Byte array for Hserin/Hserout
    By mrx23 in forum mel PIC BASIC Pro
    Replies: 5
    Last Post: - 1st September 2006, 23:07
  4. 8 data and parity using HSERIN, HSEROUT
    By Smity42 in forum Serial
    Replies: 2
    Last Post: - 3rd April 2006, 01:45
  5. Array data being corrupted
    By caverman in forum mel PIC BASIC Pro
    Replies: 1
    Last Post: - 17th March 2006, 05:22

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts