Using the Arduino Ethernet shield with AMICUS18 (Joint forum project?)

[HenrikOlsson]

Hello,
Now that we know how to setup the W5100 to listen for TCP trafic on the correct port, how to determine if there's data to be read in the sockets RX memory buffer and where in the buffer that data is it's time to look at how we can parse the received data and act upon it. After all, simply "resending" the received data out the USART may be cool (for 5 minutes) but it doesn't really do anything for us, though it does serve as a nice debugging tool.

So today I'll try to cover how we can load the recevied data from the W5100 into a local buffer, do some (for now very) simple parsing of that data in order to determine what the client wants and then build and send a response. The end result will be a simple web-page displayed in the browser window.

Trying to parse the data directly in the W5100 proved to be a bit tricky so I decided to move it to RAM in the PIC and do the parsing there. To do this we first need somewhere to put the data, an array called Buffer of some, for now arbitrary, length:

BufferSize  CON     128
Buffer      VAR     BYTE[BufferSize]' Local buffer for loading and unloading data from the W5100

Then I modified the GetData routine posted earlier to not only "print" the received bytes on the serial terminal but to also stuff them into our previously declared array. The GetData routine now looks like this:

'*****************************************************************************************************
GetData:
    HSEROUT [REP "-"\20, " Data read from Rx buffer follows ", REP "-"\20, 13]
 
    Counter = 0                                 ' Start loading the local array buffer at the beginning.

    ' Do we have a buffer wrap around condition?
    If WizEnd < WizStart THEN
 
    ' When that happends we need to start reading at the first byte in the received
    ' package and go on until the end of the sockets RX memory....
        For WizAdress = WizStart to W5100_S0_RX_END
            Gosub Read_W5100
            HSEROUT[WizData]

            ' Make sure we don't write outside of the allocated buffer.
            If Counter <= BufferSize THEN
                Buffer[Counter] = WizData
            ENDIF
 
            Counter = Counter + 1                ' Keep count

        NEXT
 
        ' ...then we need to start over at the beginning of the buffer and keep reading
        ' until we've got all the bytes in the buffer.
 
        For WizAdress = W5100_S0_RX_START to WizEnd
            Gosub Read_W5100
            HSEROUT[WizData]

            ' Make sure we don't write outside of the allocated buffer.            
            If Counter <= BufferSize THEN
                Buffer[Counter] = WizData
            ENDIF
 
            Counter = Counter + 1                ' Keep count
        NEXT
 
    ELSE                                         ' There's no buffer wrap-around.
 
        ' WizStart now contains the physical adress of the first byte to read.
        For WizAdress = WizStart to WizEnd
            Gosub Read_W5100
            HSEROUT[WizData]
 
            ' Make sure we don't write outside of the allocated buffer.
            If Counter <= BufferSize THEN
                Buffer[Counter] = WizData
            ENDIF

            Counter = Counter + 1                ' Keep count

        NEXT
 
    ENDIF
 
    HSEROUT [REP "-"\23, " End of data from Rx buffer ", REP "-"\23, 13]
    HSEROUT ["Total number of bytes read:", #Counter,13]
    HSEROUT [REP "-"\76, 13, 13]
 
    ' The variable RxPtr contains the "raw value" read from the sockets Rx Read Pointer Register.
    ' Now we need to be update that so it points one "step" ahead of the last location we just read.
    ' This is done to let the W5100 how "far" in the buffer we've read and that it can use any space
    ' "before" this location for new data. Note that this is not the ACTUAL adress of the data,
    ' just the unmasked index.
 
    WizPtr = WizPtr + WizSize
    WizAdress = W5100_S0_RX_RD0 : WizData = WizPtr.HighByte : GOSUB Write_W5100
    WizAdress = W5100_S0_RX_RD1 : WizData = WizPtr.LowByte  : GOSUB Write_W5100

    ' Then we send the RECV command to the sockets command register to let it know that
    ' we've handled all the data we currently are prepared to handle. This updates the
    ' pointers in the sockets memory so that it knows where to put new data. It is not
    ' until after issuing the RECV command that the sockets RX pointer is actually updated
    ' even though we write to it above.

    WizAdress = W5100_S0_CR : WizData = W5100_Sn_RECV : GOSUB Write_W5100
 
RETURN
'*****************************************************************************************************

As you can see we are no longer disconnecting and closing the connection at the end of the GetData routine, the reason is obviously because we intend to send a response to the client so we need to keep the connection open until we've done that.

Now that we have the data in our local array Buffer (or at least the first 128 bytes of it) we are ready to start analysing it in order to determine what the client wants. Looking back at the screenshot of the received data being printed to the serial terminal we can see that simply browsing to the W5100's IP adress results in the first couple of bytes in the buffer looking like this:

GET / HTTP/1.1

What we're interested in here is the GET part, that is what the client wants to do - it wants to GET something from our server. The client could have specified a specific resource to get after the first forward slash, like

GET /thatpage.html HTTP/1.1

But for now we won't look at serving more than a single resource so as soon as we see the word GET we know we should respond with the webpage we've got in the PIC. Here's a very simple routine to parse the buffer, starting at the beginning:

'******************************************************************************************
' Very simple routine to parse the data in the local buffer.
' If the first word is GET we set a semaphore signaling the main program that we've got
' a GET request.
'******************************************************************************************
GetRequest VAR BIT

ParseData:

If Buffer[0] = "G" THEN
    IF Buffer[1] = "E" THEN
        IF Buffer[2] = "T" THEN
            IF Buffer[3] = " " THEN
                GetRequest = 1
                HSEROUT["Found GET request",13]
             ENDIF
        ENDIF
    ENDIF
ENDIF

RETURN

Once we've found the word GET at the very beginning of the received data we're going to respond with a simple web-page. To do that the first order of business is to determine the amount of free space in the socket transmit buffer and where it wants us to start loading the data. This is done in a similar way to checking the RX-buffer, in fact we're reusing the same variables (WizPtr, WisStart and WizSize) for this purpose.

'*****************************************************************************************************
' This routine checks the amount of free space in the sockets transmit register.
' It also gets the pointer of where to start loading data and calculates the
' physical adress for us. It returns the amount of free space in WizSize and
' the adress of the first byte in WizStart. WizPtr is the "raw" pointer value
' and is used to update the socket transmit pointer after writing to the buffer.
'*****************************************************************************************************
GetFreeTXSize:
 
    ' Get the total amount of free memory in the sockets transmit buffer.
    WizAdress = W5100_S0_TX_FSR0 : GOSUB Read_W5100 : WizSize.HighByte = WizData
    WizAdress = W5100_S0_TX_FSR1 : GOSUB Read_W5100 : WizSize.LowByte = WizData
 
    ' Get the pointer adress of where to put new data.
    WizAdress = W5100_S0_TX_WR0 : GOSUB Read_W5100 : WizPtr.HighByte = WizData
    WizAdress = W5100_S0_TX_WR1 : GOSUB Read_W5100 : WizPtr.LowByte = WizData
 
    'Calculate physical adress of first byte
    WizStart = W5100_S0_TX_START + (WizPtr & W5100_S0_TX_MASK)
 
RETURN

When we return from this subroutine the amount of free space is available to us in WizSize, the raw pointer value (working exactly as the pointer into the RX memory buffer) is avaliable to us in WizPtr and the actual, physical, memory adress of the first free location is available to us inWizStart.

In order to somewhat effeciently load HTML data into the buffer I came up with the following routine. What it does is loading null-terminated strings from the local array Buffer to the sockets TX memory, it handles the circular buffer and it keeps track of the total number of bytes loaded to the buffer so we can tell the W5100 how "far" into the buffer it should send.

'******************************************************************************************
' Subroutine to load a string from the local array Buffer to the W5100 socket 0's
' TX memory buffer. The string can be any length but must end with NULL.
'******************************************************************************************
PutStringInTXBuffer:
 
    Counter = 0                                 ' Counter for indexing the local array we're about to load into the TX buffer.

    While Buffer[Counter] <> 0                  ' Last byte in each part is null.

    ' Check if we are about to write outside the sockets memory area.
    ' If so we wrap around to the start of the buffer.

        If WizStart + TotalCount > W5100_S0_TX_END then 
            WizStart = W5100_S0_TX_Start - TotalCount
        ENDIF

    ' Write byte to TX memory buffer.
        WizAdress = WizStart + TotalCount : WizData = Buffer[Counter] : GOSUB Write_W5100
        Counter = Counter + 1             ' Point at the next byte in the array.
        TotalCount = TotalCount + 1             ' Count total number of bytes for the response.
    WEND

RETURN

We can then "feed" the PutStringInTxBuffer routine with data like in the first half of the HandleGetRequest routine:

'******************************************************************************************
' Subroutine used to build a simple web-page in response to a GET request.
'******************************************************************************************
HandleGetRequest:

    RequestCount = RequestCount + 1

    GOSUB GetFreeTXSize

    ' Reset the total byte count. This will be incremented in the PutStringInTXBuffer routine
    ' and used to keep track of the total number of bytes loaded to the TX memory buffer.
    TotalCount = 0

    HSEROUT ["Loading buffer...",13]

    ArrayWrite Buffer,["HTTP/1.0 200 OK",13,10,0]
    GOSUB PutStringInTXBuffer

    ArrayWrite Buffer,["Content-Type: text/html",13,10,13,10,0]
    GOSUB PutStringInTXBuffer

    ArrayWrite Buffer,["<html><body><title>AMICUS18</title>",13,10,0]
    Gosub PutStringInTXBuffer

    ArrayWrite Buffer,["<h1>Embedded Web Server</h1>",13,10,0]
    Gosub PutStringInTXBuffer

    ArrayWrite Buffer,["<h3>PIC18F25K20 / W5100 Ethernet chip</h3>",13,10,0]
    GOSUB PutStringInTXBuffer

    ArrayWrite Buffer,["<p>Henrik Olsson 2011</p>",13,10,0]
    GOSUB PutStringInTXBuffer

    ArrayWrite Buffer,["<a href=",34,"http://www.picbasic.co.uk/forum",34,">Visit the PBP Forum</a>",13,10,0]
    Gosub PutStringInTXBuffer

    ArrayWrite Buffer,["<p>This page has been requested ", DEC RequestCount, " times since powerup/reset</p>",13,10,0]
    GOSUB PutStringInTXBuffer

    ArrayWrite Buffer,["</body></html>",13,10,13,10,0]
    GOSUB PutStringInTXBuffer

    HSEROUT[13, "done...",DEC TotalCount, " bytes.."]

The above is the actual HTML response that we're sending as a result of the GET request. What's important here is that we must reset the TotalCount variable before starting to load data and then not touch it until we're done. Each "part" of the response must end with NULL as can be seen in the ArrayWrite operations. There's no real requirement to split the response up in a certain way, it's more for the purpose of readabillity.

Using ArrayWrite like this is a simple aproach but it seems very (VERY) inefficient because each and every character/byte of adds 6 bytes(!) to the footprint of the code. This isn't really a problem for this type of simple web-page but for more complicated stuff it'll add up quickly. We'll have to look into getting the SD card working as a storage "container" for the HMTL pages but that's another topic.

Once we've got all the data loaded to the TX buffer we're ready to send it and this is handled by the second half of the HandleGetRequest subroutine:

    ' Now recalculate the transmit pointer so the W5100 knows where to stop sending. WizPtr currently contains
    ' the adress of the first free byte and TotalCount is the number of bytes of byte we've loaded to the buffer.
    WizPtr = WizPtr + TotalCount

    ' Write the updated pointer value to the W5100
    WizAdress = W5100_S0_TX_WR0  : WizData = WizPtr.HighByte : Gosub Write_W5100
    WizAdress = W5100_S0_TX_WR1  : WizData = WizPtr.LowByte : Gosub Write_W5100

    ' Issue a SEND command.
    WizAdress = W5100_S0_CR : WizData = W5100_Sn_SEND : GOSUB Write_W5100

WaitForCompletion:
    GOSUB Read_W5100
    If WizData <> 0 then Goto WaitForCompletion

    HSEROUT[".sent.",13]

    ' Then disconnect from the client.
    WizAdress = W5100_S0_CR : WizData = W5100_Sn_DISCON : GOSUB Write_W5100
 
    ' And close the socket. We'll reopen the socket again in the main routine.
    WizAdress = W5100_S0_CR : WizData = W5100_Sn_CLOSE : GOSUB Write_W5100
    GetRequest = 0
 
RETURN

Note that I don't actually verify if the data I'm about to load will fit in the amount of free space provided in the sockets TX memory. Not doing that should be considered bad practice but for now I'll do it anyway since I know that the size of the buffer is more than 6 times the size of the data I'm writing to it.

Now all we need to do in our main routine is to:
1) Check the received size register of the socket, if it's >0 we'll handle the received data using the above routines.
2) Since we're closing the socket after sedning the data it's the main routines job to re-open it and set it to listen mode.

HSEROUT ["Program start",13]

GOSUB Init_W5100   ' Reset W5100 and set MAC, IP, NetMask and Gateway adress, 

Main:
    ' The socket 0 status register automatically switches to SOCK_ESTABLISHED 
    ' when a valid request to connect comes in from a client.
    WizAdress = W5100_S0_SR : GOSUB Read_W5100
 
    If WizData = W5100_SOCK_ESTABLISHED then
        Gosub CheckBufferForData
        If WizSize > 0 then
            GOSUB GetData
            GOSUB ParseData
            If GetRequest = 1 then GOSUB HandleGetRequest
        ENDIF
    ENDIF
 
    ' If the socket is closed (we're closing it in the GetData routine) then
    ' we must re-open it so we are prepared for the next client request.    
    WizAdress = W5100_S0_SR : GOSUB Read_W5100
    If WizData = Socket_CLOSED THEN
        GOSUB Init_Socket_0
    ENDIF
 
Pause 250
Goto Main

To this post I'm attaching the source file (remove the .txt extension and you'll need the W5100_defs.pbp file as well) I used to arrive at the following results:

It compiles to 5275 bytes out of which the ArrayWrite statements (with a total of ~340 bytes of HTML) occupies 2038 bytes - yikes. So, without the HTML we're looking at less than 3.5k for the actual server code - not bad.

/Henrik.