Using the Arduino Ethernet shield with AMICUS18 (Joint forum project?)

[HenrikOlsson]

Hi,
With this post I hope to spark the interest in what could develop into a joint forum project. The purpose of the project is to get the Arduino Ethernet shield working with the AMICUS18 board and, of course PBP.


Unfortunately the Ethernet shield isn't very directly compatible with the AMICUS18. The reason for this is that the shield picks up the SPI signals used to communicate with the W5100 ethernet chip and the onboard SD-card from the ICSP programming header on the Arduino. For apparent reasons this header does not exist on the AMICUS18. This means that Ethernet shield needs to be modifed in order to use it and this first post of mine is going to cover what and how I did to make it work with the AMICUS18.


Before I go on I'd like to point out a couple of things:
1) I'm using Ethernet shield v.6 which I got from NKC electronics, so what I describe here applies to that particular version. I do not know how it'll work with previous (or upcomming) versions of the shield.
2) Doing these modification most likely voids any warranty you might have on the Ethernet shield.
3) You do it at your own risk, don't blame me if you mess it up.




5521

As have been said the Ethernet shield uses a SPI interface to communicate with the host CPU. Since these signals are brought to the ICSP header on the shield we need to "move" them to pins that suits us better. The 18F25K22 that is used on the AMICUS18 has its SPI signals on RC3, 4 & 5 which are the same as pins 4, 3 & 2 on the Ethernet shield. Now, moving the SPI signals from the ICSP header to RC3, 4 & 5 wouldn't be too hard but unfortunately two out of three pins (RC3 & RC5) are already in use by the shield so we need to move those signals too - before we can connect the SPI signals to these I/O's.

Pin 2 on the Ethernet shield (RC5 on the AMICUS18) carries the interrups signal from the W5100 chip. Since we might want to use this signal later on it would make sense to move it to an I/O-pin of the 18F25K22 capable of generating an interrupt. I've selected RB0 which on the Ethernet shield is labeled AREF. Fortunately AREF is not connected to anything on the shield so we're good to go there. By default the W5100 interrupt signal isn't actually connected to Pin2, instead there are two small solder pads on the back of the board which, when bridged with solder, connects the signal to Pin2. We'll use one of these pads to "hijack" the signal and re-route it to RB0 instead.

Pin4 on the Ethernet shield carries the chip select signal for the onboard micro-SD card socket. This signal is an ordinary I/O so any pin will do really. I've selected RB2 which, on the Ethernet shield, is labeled 13.

Why not RB1 you may think. It's because RB1 on the shield is GND and it would be hard to gain access to that pin without serious board surgury. This also brings up an important point: On the AMICUS18 board there's jumper that allows you the select if RB1 should "be" RB1 or if it's GND. This jumper should be in the GND location.

When these two signals are moved we can connect the SPI interface signals from the 2X3 ICSP header to pins 2, 3 and 4 on the shield. Let's start with the trickiest one, Pin4, which carries the chip select signal for the SD-Card. First we need to "disconnect" the signal from Pin4, this is done by cutting the PCB trace just where it comes out from under the Arduino logo:


5524

With the trace cut we can turn the board over and find a good spot to "tap into" the signal:


5520

Running a wire from the via marked SD_SC to RB2 and a wire from the pad marked W5100_INT to RB0 is all it takes to "free up" the AMICU18's SPI pins so we can use them.

In the next photo I've marked which pins in the ICSP header that we're interested in. SCK goes to RC3, MISO goes to RC4 and MOSI goes to RC5. I opted to remove the header altogether because it has a slight interference with the socket for the 18F25K20 on the AMICUS18 board.



5525


Here's the the back of the board with the modification complete. One trace cut and five jumper wires and we should be good to go. (I guess you can't really call that "Arduino shield compatible"....)


5523


/Henrik.

[mackrackit]

Thank you Henrik,

I promoted this to an article.
http://www.picbasic.co.uk/forum/content.php?r=346-amicus18

[Heckler]

good write up on the modification Henrik,

I have always dreamed of setting up a little web connected interface for home automation type stuff. This seems like it would be a great place to start... especially if it was based on PICbasic pro.

I guess I better get me one of those amicus's and an ethernet shield.

[HenrikOlsson]

You can definitely use the EM202 or one of Tibbos other modules, or you can use theXPort (http://www.lantronix.com/device-networking/embedded-device-servers/xport.html) or perhaps something from Netburner (http://www.netburner.com/products/core_modules.html), or maby the EZWebLynx (http://www.ezweblynx.com/) or the DigiConnectMe (http://www.digi.com/products/wireless-wired-embedded-solutions/solutions-on-module/digi-connect/digiconnectme.jsp#overview). Or you might want to go "low level / hardcore" and use the ENC28J60 or ENC424J600 or some other ethernet control and write the TCP/IP stack yourself. There are many ways of doing it.

But this thread wasn't supposed to be about other ways of doing it, it was meant to be about how to do it with the Wiznet W5100 which is what is being used on the Ethernet shield.

By the way, I've got PING working....and I've got to a point where I can see the HTTP Get request in the W5100 Receive memory buffer :-)

/Henrik.

[ScaleRobotics]

For those wanting to follow along, but maybe not wanting to invest in the shield and Amicus18 right now, there is a $25 solution that uses the same W5100 ethernet chip. http://www.sparkfun.com/products/9473

[lester]

My applogies to Henrick, quite right, this thread was (is) about using Arduino shields on the Amicus18. My post was out of order, promoting an alternative , was incorrect.

Henrick, i stand corrected, I hope you accept my appology, thread hijacking is unacceptable. i've removed my post and those related to it.

[Heckler]

Henrick,

Keep us posted on your progress on this... I have absolutely no experience getting into the "nuts and volts" of ethernet protocol... but, as I mentioned earlier, would love to have a little web interface to some home automation capability.

Thanks for your efforts.

[HenrikOlsson]

Hi,
I don't know what the "best" way to aproach this actually is. I don't think me posting hundreds or thousands of lines of code is going to actually help others that much and I don't think I'll get much out of it either.

Instead I think I'll post snippets and routines, explaining what they do (or at least intend to do) and why. This way I think it's easier to tag along, one step at the time and, which is important, we can discuss suggestions on improvements and alternative ways of doing the same thing. In the end we will hopefully have a set of routines which can be used to build a working application.

Before we go on I just have to point out that this is new territory for me too. I don't know much about ethernet, UDP, TCP, sockets, HTTP, HTML etc so this is very much a learning experience for me as well. Please feel free to correct any statements I make in error and please do bring suggestions on how to do things differently and, hopefully, better.


Right, the Wiznet W5100 ethernet chip (which is used on the Ethernet shield) is controlled by reading and writing to its internal registers. There are quite a few registers and instead of using a bunch of 'magic numbers' (ie the adress of the register in the W5100's memory) I've put together a file (W5100_defs.pbp) containing the name and memory adress for the registers as well as a couple of other constants we might need. Attached to this post is v1.0 of that file (remove .txt extenstion), there might be newer versions in the posts to come.

I've set up my AMICUS18 board to use the hardware USART at 115200 baud for debugging purposes. Then I set up the TRISB and TRISC registers to match the pinout of the modified Ethernet shield:

' By default the AMICUS18 board uses the PIC18F25K20, make sure to
' set up MCSP or whatever IDE is being used to compile for the
' correct device.

DEFINE OSC 64
DEFINE LOADER_USED 1 ' We're using a bootloader.
DEFINE HSER_RCSTA 90h ' Enable serial port & continuous receive
DEFINE HSER_TXSTA 24h ' Enable transmit, BRGH = 1
DEFINE HSER_CLROERR 1 ' Clear overflow automatically
DEFINE HSER_SPBRG 138 ' 115200 Baud @ 64MHz, -0,08%

SPBRGH = 0
BAUDCON.3 = 1 ' Enable 16 bit baudrate generator

TRISC = %10010111
' RC7: USART RX
' RC6: USART TX
' RC5: ETHERNET SPI Data out (SDO/MOSI)
' RC4: ETHERNET SPI Data in (SDI/MISO)
' RC3: ETHERNET SPI clock (SCK) set to INPUT according to datasheet....
' RC2: N/U - currently
' RC2: N/U - currently
' RC1: N/U - currently

TRISB = %11011011
' RB7: N/U - currently
' RB6: N/U - currently
' RB5: ETHERNET Chip select line to W5100
' RB4: N/U - currently
' RB3: N/U - currently
' RB2: ETHERNET Chip select line to SD-Card
' RB1: N/U and can not be used with the Ethernet shield
' RB0: ETHERNET Interrupt pin from W5100


Then I pretty much cut'n'pasted the register definition/aliases for the MSSP module from the SPI master example (http://melabs.com/samples/PBP-mixed/spimast.htm) on MELABS web site. However, I had to change the CKE bit from 0 to 1 in order to get reliable communications with the W5100. This feels a bit awkward but since doing that it has been rock solid so it must be right - right?


' Aliases for the MSSP module status and control bits.
SSPEN VAR SSPCON1.5 ' SSP Enable bit
CKP VAR SSPCON1.4 ' Clock Polarity Select
SMP VAR SSPSTAT.7 ' Data input sample phase
CKE VAR SSPSTAT.6 ' Clock Edge Select bit
SSPIF VAR PIR1.3 ' SPI interrupt flag

'Set up the MSSP module.
CKP = 0 ' Clock idles low
CKE = 1 ' Transmit on active to idle transition.
SSPIF = 0 ' Clear buffer full status
SMP = 0 ' Sample in middle of data
SSPEN = 1 ' Enable SPI pins
SSPCON1.0 = 1 'Slow down SPI clock to Fosc/16 for now.
I also had to slow down the SPI clock from the default (Fosc/4) to Fosc/16 in order to get reliable communication. The AMICUS18 defaults to 64MHz so with Fosc/16 the SPI clock is running at 4Mhz which isn't directly slow but well below what the W5100 is capable of (if I read the datasheet correctly that would be ~14Mhz). However, the modification done to the shield and the fact that W5100 is on a separate board from the 18F25K20 may have an impact on the maximum speed achievable. Anyway, I'll leave it at Fosc/16 for now.

Finally I included the file with all the register definitions for the W5100:

INCLUDE "W5100_defs.pbp" ' Register definitions and constants for the W5100.

That's all for this post.

/Henrik.

[pedja089]

I am interested in this project. For now, I have no hardware, no time to make it. So for now I'll watch and learn:)

[HenrikOlsson]

Hi,
In todays post I'll try to cover the basic input and output routines I use to communicate with the W5100.

The W5100 can communicate with the main controller either over a parallel data bus featuring 14 adress lines, 8 data lines and a couple of control lines or it can use a serial interface, SPI. On the Ethernet shield the W5100 is setup to use the SPI interface which is fortunate because we simply don't have enough I/O lines on the AMICUS18.

Every transaction to and from the W5100, when using SPI, requires 4 bytes. First an OP code telling the W5100 if we're doing a read or a write operation, then two bytes containing the adress we are writing or reading and finally the actual data. When writing to the W5100 WE put the data in the forth byte, when reading from the W5100 IT puts the data in the forth byte - basically. So for each transaction we pull the Chip Select line of the W5100, send 4 bytes and release the Chip Select line (pull high). Here's the code for my read and write subroutines:



WizAdress VAR WORD ' Adress in the W5100 to be read or written
WizData VAR BYTE ' Data to be written or data read from the W5100
WizResponse VAR BYTE[4] ' Respose to the four bytes in each SPI transaction. Should be 0,1,2,3 or 0,1,2,data

W5100_Select VAR PortB.5 ' Chip select line of W5100 connected to RB.5
W5100_Select = 1

'************************************************* *****************************************
Read_W5100:
WizResponse[0] = SSPBUF ' Dummy read

W5100_Select = 0 ' Select W5100

SSPBUF = W5100_READ_OP ' Send OP code for read operation
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[0] = SSPBUF ' Store result, should be 0

SSPBUF = WizAdress.HighByte ' Send high byte of adress
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[1] = SSPBUF ' Store result, should be 1
SSPBUF = WizAdress.LowByte ' Send low byte of adress
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[2] = SSPBUF ' Store result, should be 2

SSPBUF = 0 ' Send data
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[3] = SSPBUF ' Store result, will be the actual data read at the specified adress

WizData = WizResponse[3] ' Copy databyte from array to WizData.
W5100_Select = 1 ' Deselect the W5100
RETURN
'************************************************* *****************************************

'************************************************* *****************************************
Write_W5100:
WizResponse[0] = SSPBUF ' Dummy read.

W5100_Select = 0 ' Pull chip select line low to select W5100

SSPBUF = W5100_WRITE_OP ' Send OP code for write operation
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[0] = SSPBUF ' Store result, should be 0

SSPBUF = WizAdress.HighByte ' Send high byte of adress
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[1] = SSPBUF ' Store result, should be 1

SSPBUF = WizAdress.LowByte ' Send low byte of adress
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[2] = SSPBUF ' Store result, should be 2

SSPBUF = WizData ' Send the data byte
While SSPSTAT.0 = 0 : WEND ' Wait for operation to complete
WizResponse[3] = SSPBUF ' Store the result, should be 3

W5100_Select = 1 ' Deselect the W5100

RETURN
'************************************************* *****************************************


I've declared two variables of specific interest here, WizAdress and WizData. WizAdress is the adress in the W5100 we want to access and WizData is, in the case of a write operation, the data we want to write to that adress OR, in the case of a read operation, the data read FROM that adress. In the W5100_defs.pbp file attached to an earlier post the OP-codes for read and write operation as well as the names and adresses of all the registers are defined. Looking at the datasheet for the W5100 we can see that setting bit 7 of the Mode register will perform a reset of the device after which bit 7 is automatically cleared again. Doing this with help from the above subroutines is as simple as:


WizAdress = W5100_Mode : WizData = %10000000 : GOSUB Write_W5100


In a similar manner we can read, for example, the interrupt register at location $0015 and output its content over the USART like this:


WizAdress = W5100_IR : GOSUB Read_W5100
HSEROUT ["The status of the Interrupt register is: ", BIN8 WizData, 13]


As ilustrated here, when writing TO the W5100 WE put the data in the WizData variable and when reading FROM the W5100 we GET the data in the WizData variable.


To get the Ethernet shield to respond to a Ping all we need to do is set its MAC-adress, IP-adress and NetMask. Doing that is just a series of write operations:



SetUp:
HSEROUT["W5100_Init",13]

'Send Reset command to W5100.
WizAdress = W5100_Mode : WizData = 128 : Gosub Write_W5100

' Set MAC-adress, see sticker on the back of the Ethernet shield.
WizAdress = W5100_SHAR0 : WizData = $05 : GOSUB Write_W5100
WizAdress = W5100_SHAR1 : WizData = $06 : GOSUB Write_W5100
WizAdress = W5100_SHAR2 : WizData = $07 : GOSUB Write_W5100
WizAdress = W5100_SHAR3 : WizData = $08 : GOSUB Write_W5100
WizAdress = W5100_SHAR4 : WizData = $09 : GOSUB Write_W5100
WizAdress = W5100_SHAR5 : WizData = $0A : GOSUB Write_W5100

'Set IP adress of W5100
WizAdress = W5100_SIPR0 : WizData = 192 : GOSUB Write_W5100
WizAdress = W5100_SIPR1 : WizData = 168 : GOSUB Write_W5100
WizAdress = W5100_SIPR2 : WizData = 1 : GOSUB Write_W5100
WizAdress = W5100_SIPR3 : WizData = 50 : GOSUB Write_W5100
'Set net-mask
WizAdress = W5100_SUBR0 : WizData = 255 : GOSUB Write_W5100
WizAdress = W5100_SUBR1 : WizData = 255 : GOSUB Write_W5100
WizAdress = W5100_SUBR2 : WizData = 255 : GOSUB Write_W5100
WizAdress = W5100_SUBR3 : WizData = 0 : GOSUB Write_W5100
HSEROUT ["Basic setup done",13]

Main:
@ NOP
GOTO MAIN



5539
(Sorry about the Swedish screenshot but you'll get the idea...it works...)

Without the HSEROUT statements we're looking at a just above 400 bytes (out of the 18F25K20's 16k) worth of program memory. Granted, only being able to respond to a ping isn't really worth much from a practical standpoint but it IS a start.


Finally a word or two about the WizResponse array. When using SPI, data is shifted in and out of the PIC at the same time. Basically as one bit goes out at the "low end" of the MSSP modules shift register one bit goes at the "high end" so after 8 clock cycles the byte we loaded into SSPBUF have been exchanged with a byte from the W5100.

When writing data to the W5100 it responds with 0, 1, 2, 3 for the four bytes it receives and I'm storing this response in the array WizResponse in order to potentionally use it to verify that the operation was successfull. When reading from the W5100 it responds with 0, 1, 2 and then the actual data in the 4th byte so all I'm doing is copying the data from the 4th byte of the array to the WizData variable before returning from the subroutine.

/Henrik.

[HenrikOlsson]

Hi,
In todays post(s) I thought I'd try demonstrate how to set up one of the W5100's four sockets in TCP mode and how to read data that is received. In the end we'll be able to see the data that a web browser sends when it tries to reach the W5100.

The W5100 can have four connections, or sockets as it called, open simultanously. What this means is that you can have one socket setup to listen for TCP trafic on port 80 (which is the standard port for HTTP) thus acting as a webserver. At the same time you can have another socket connect TO a server and send an email while a third socket connects to a NTP server and retrieves the correct time.

For now we'll concentrate on a single socket though.

First order of business is to initialize the W5100, like before this is done by performing a soft reset and then write to its SHAR, SIPR, SUBR and GAR registers to set the MAC-adress, IP-adress, net-mask and gateway adress. Setting the gateway adress isn't strictly needed for this to work but I did that anyway. For now I kept all the settings "hardcoded" but in a real device these would obviously have to be changable thru some kind of userinterface and stored in EEPROM or whatever.


'************************************************* *****************************************
Init_W5100:

'Perform a soft reset of the W5100
WizAdress = W5100_Mode : WizData = 128 : Gosub Write_W5100

' Set W5100 MAC adress
WizAdress = W5100_SHAR0 : WizData = $90 : GOSUB Write_W5100
WizAdress = W5100_SHAR1 : WizData = $A2 : GOSUB Write_W5100
WizAdress = W5100_SHAR2 : WizData = $DA : GOSUB Write_W5100
WizAdress = W5100_SHAR3 : WizData = $00 : GOSUB Write_W5100
WizAdress = W5100_SHAR4 : WizData = $48 : GOSUB Write_W5100
WizAdress = W5100_SHAR5 : WizData = $03 : GOSUB Write_W5100

' Set W5100 IP adress
WizAdress = W5100_SIPR0 : WizData = 192 : Gosub Write_W5100
WizAdress = W5100_SIPR1 : WizData = 168 : Gosub Write_W5100
WizAdress = W5100_SIPR2 : WizData = 1 : Gosub Write_W5100
WizAdress = W5100_SIPR3 : WizData = 50 : Gosub Write_W5100

' Set W5100 Net mask
WizAdress = W5100_SUBR0 : WizData = 255 : GOSUB Write_W5100
WizAdress = W5100_SUBR1 : WizData = 255 : GOSUB Write_W5100
WizAdress = W5100_SUBR2 : WizData = 255 : GOSUB Write_W5100
WizAdress = W5100_SUBR3 : WizData = 0 : GOSUB Write_W5100

' Set W5100 gateway adress
WizAdress = W5100_GAR0 : WizData = 192 : GOSUB Write_W5100
WizAdress = W5100_GAR1 : WizData = 168 : GOSUB Write_W5100
WizAdress = W5100_GAR2 : WizData = 1 : GOSUB Write_W5100
WizAdress = W5100_GAR3 : WizData = 254 : GOSUB Write_W5100
RETURN
'************************************************* *****************************************


Next task is to setup socket 0 to listen for TCP trafic on port 80. To do this we first write the value 1 to the sockets Mode register, this sets the protocol for the socket to TCP. Then we write the value 80 to the sockets Source Port register and then tell the W5100 to open the socket. Once the socket is open we can finally switch it into Listen mode which is what we want when acting as a server.


'************************************************* *****************************************
Init_Socket_0:
' Set socket 0 protocol to TCP by assigning the sockets mode register the value 1.
WizAdress = W5100_S0_MR : WizData = 1 : GOSUB Write_W5100

' Set the port register of socket 0.
WizAdress = W5100_S0_PORT0 : WizData = 0 : Gosub Write_W5100
WizAdress = W5100_S0_PORT1 : WizData = 80 : GOSUB Write_W5100

Open_Socket_0:
' Set the Command Register of socket 0 to OPEN - this inits the socket.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_OPEN : GOSUB Write_W5100

' Now switch Socket 0 into Listen Mode.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_LISTEN : GOSUB Write_W5100
RETURN
'************************************************* *****************************************

Once the socket is in Listen mode it will automatically switch to Established mode when it receives a connection request from a client. We can check in which mode the socket is by reading the sockets Status register.


HSEROUT ["Program Start", 13]
Main:
WizAdress = W5100_S0_SR : GOSUB Read_W5100
HSEROUT["Socket 0 status = $", HEX WizData,13]
Pause 500
Goto Main


If you run this code with a serial terminal open and type in the W5100's IP-adress in a browser you should see the sockets status register changing from $14 (Listen) to $17 (Established):


5613

...to be continued in the next post.

/Henrik.

[HenrikOlsson]

...continued from previous post...

In the W5100 there's a total of 8k of RAM reserved for the sockets receive memory buffer as well as 8k for the sockets transmit memory buffer. The memory can be assigned to each socket in various configuration by writing to the RMSR and TMSR registers. By default both these registers are set to $55 which assigns 2k of RX and TX memory to each of the four sockets. Since we're only about to use one socket we COULD assign then full 8k of each buffer to socket 0 but for this exercise we'll just leave it at the default.

Once we've established a connection with the client we can determine if any data has been received by reading the sockets Received Size register. The value returned tells us the number of "new" bytes in the sockets receive buffer. What it doesn't tell us is where in the buffer these bytes are....

The sockets receive (and transmit) buffers are circular buffers meaning once they hit the highest memory adress they wrap around and start over at the beginning. The first byte we're interested in might be somewhere in the middle of the buffer and the last byte might be just before it even though it intuitively should be after the first byte. And, because all four sockets are assigned a "segment" of memory out of a total of 8k the actual start and end adresses of each sockets buffer can "move around" depending on the amount of memory assigned to each socket. If we take a look at the bottom of the W5100_defs.pbp file posted earlier we'll find this:


'************************************************* *************************************
' ----------- THESE VALUES MAY NEED TO BE CHANGED BY THE USER OF THIS FILE ------------
'************************************************* *************************************
' Socket 0-3 RX memory START & END adresses and MASK values for calculating pointer.
' Make sure to change these if the RMSR value is changed from its default value of $55

W5100_S0_RX_START CON $6000 'Start adress of Socket 0 (not changeable)
W5100_S0_RX_END CON $67FF
W5100_S0_RX_MASK CON $07FF 'Mask value for 2k

W5100_S1_RX_START CON $6800 'Start adress of Socket 1 (depends on the size of socket 0)
W5100_S1_RX_END CON $6FFF
W5100_S1_RX_MASK CON $07FF 'Mask value for 2k

W5100_S2_RX_START CON $7000 'Start adress of socket 2 (depends on the size of sockets 0 & 1)
W5100_S2_RX_END CON $77FF
W5100_S2_RX_MASK CON $07FF 'Mask value for 2k

W5100_S3_RX_START CON $8000 'Start adress of socket 3 (depends on the size of sockets 0, 1 & 2)
W5100_S3_RX_END CON $8FFF
W5100_S3_RX_MASK CON $07FF 'Mask value for 2k
'************************************************* *************************************


These are the physical start and end adresses for each sockets RX memory buffer when they are each assigned 2k of memory. If the memory allocation between sockets are changed these constants must be updated to reflect the start and end adress of each sockets memory area.

OK, so how do we know where in the buffer our new data is? This is done by reading the sockets RX Read Pointer register but unfortunately this doesn't give us the real adress of the data, instead it gives us an offset into the sockets memory area. If the value of the pointer is 0 then the first byte of our new data is located at the start-adress of the sockets RX memory ($6000 in this case), if the value is $123 then the first byte is located at adress $6123. Not to bad.... But to make it a bit more complicated this pointer doesn't wrap around "in sync" with the circular buffer. The Rx Read Pointer register is a 16bit register meaning it counts from 0 to 65535 even though we only have 2048 bytes assigned to our buffer - so where is our data when the pointer value is $1234?

This is where the sockets RX_MASK constant defined in the W5100_defs.pbp file comes in. When the raw pointer value is bitwise AND'ed with the RX_MASK value we'll get the correct offset into the sockets buffer. If we then add this offset to the physical start adress of sockets buffer we finally have the physical adress of the first byte of new data in the buffer.

So now we know where the first byte of our new data is as well as how many byte there is to read, now we only need to figure out where the last byte is. Given the above this shouldn't be too hard to do... If the adress of the first byte plus the total number of bytes does not "extend" beyond the end of the buffer (no buffer wrap-around) we're done - just read from first byte to first byte + number of bytes. But if it does "extend" beyond the end of the buffer we need to first read up to the end of the buffer, then start over at the beginning, subtracting the number of bytes already accounted for and continue reading. I've wrapped all of this into, yet another, subroutine:


'************************************************* *****************************************
' Variables WizPtr, WizStart, WizEnd and WizSize (all WORDs) declared elsewhere.

CheckBufferForData:

' Read the sockets receive size register.
WizAdress = W5100_S0_RX_RSR0 : GOSUB Read_W5100 : WizSize.HighByte = WizData
WizAdress = W5100_S0_RX_RSR1 : GOSUB Read_W5100 : WizSize.LowByte = WizData

' If received size is more than 0 we do indeed have new data in the buffer.
' In that case we read the socket 0 receive pointer register.
IF WizSize > 0 THEN

WizAdress = W5100_S0_RX_RD0 : Gosub Read_W5100 : WizPtr.HighByte = WizData
WizAdress = W5100_S0_RX_RD1 : GOSUB Read_W5100 : WizPtr.LowByte = WizData

' Calculate the physical adress of the first "fresh" byte in the buffer.
WizStart = W5100_S0_RX_START + (WizPtr & W5100_S0_RX_MASK)

' Is the buffer about to wrap around?
IF WizStart + WizSize > W5100_S0_RX_END THEN
WizEnd = W5100_S0_RX_START + ((WizStart + WizSize) - W5100_S0_RX_END)
WizEnd = WizEnd - 2 ' Account for the "zero indexing"

ELSE 'No wrap around.

WizEnd = WizStart + WizSize - 1
ENDIF

ENDIF
RETURN
'************************************************* *****************************************

When returning from this subroutine the physical adress of the first byte as well as that of the last byte of new data are available to us in the WizStart and WizEnd variables. If WizEnd is less than WizStart the buffer has wrapped around and we need to account for that when actually "extracting" the data. If WizEnd is bigger than [/i]WizStart[/i] then it's as simple as reading from the WizStart to WizEnd.


'************************************************* ************************************************** **
GetData:
HSEROUT [REP "-"\20, " Data read from Rx buffer follows ", REP "-"\20, 13]

Counter = 0 ' Temporary as a dubug aid

' Do we have a buffer wrap around condition?
If WizEnd < WizStart THEN

' When that happends we need to start reading at the first byte in the received
' package and go on until the end of the sockets RX memory....
For WizAdress = WizStart to W5100_S0_RX_END
Gosub Read_W5100
HSEROUT[WizData]
Counter = Counter + 1 ' Keep count
NEXT

' ...then we need to start over at the beginning of the buffer and keep reading
' until we've got all the bytes in the buffer.
For WizAdress = W5100_S0_RX_START to WizEnd
Gosub Read_W5100
HSEROUT[WizData]
Counter = Counter + 1 ' Keep count
Next

ELSE ' There's no buffer wrap-around.

' WizStart now contains the physical adress of the first byte to read.
For WizAdress = WizStart to WizEnd
Gosub Read_W5100
HSEROUT[WizData]
Counter = Counter + 1 ' Keep count
NEXT

ENDIF

HSEROUT [REP "-"\23, " End of data from Rx buffer ", REP "-"\23, 13]
HSEROUT ["Total number of bytes read:", #Counter,13]
HSEROUT [REP "-"\76, 13, 13]


When all data have been "extracted" from the buffer we have to tell the W5100 that we're done so it can recycle that memory for new data that comes in. This is done by first updating the RX Read pointer register so it points to where we left off and finally issuing the RECV command to the sockets command register. Issuing the RECV command tells the W5100 that we're done with the recevied data up to the "adress" of the newly written RX Read pointer register.

Finally, since we're currently not responding to the request from the client, we simply disconnect and switch back into Listen mode. Not very polite but it'll look the browser as if no one is at the other end so it's OK for now.


WizPtr = WizPtr + WizSize

WizAdress = W5100_S0_RX_RD0 : WizData = WizPtr.HighByte : GOSUB Write_W5100
WizAdress = W5100_S0_RX_RD1 : WizData = WizPtr.LowByte : GOSUB Write_W5100

WizAdress = W5100_S0_CR : WizData = W5100_Sn_RECV : GOSUB Write_W5100
WizAdress = W5100_S0_CR : WizData = W5100_Sn_LISTEN : GOSUB Write_W5100

Using the above code you'll be able to "see" the HTTP GET request issued by a web-browser pointing at the W5100's IP-adress:


5615

To this post I've attached the exact file I compiled to arrive at the screenshot above (remove .txt extension), you'll also need the W5100_defs.pbp posted earlier in the thread.

/Henrik.

[HenrikOlsson]

Hi,
Anyone knows what needs to be done to have a device listen to a "name" as well as its IP-adress?
For example, I have a SOHO server which has a web-interface, I can access that by either typing in its IP-adress or its name in the adress field of the browser.

I don't know if it's even possible to do with the W5100 but without knowing how the name is mapped to the IP-adress I don't know how to find out if it can be done. I'm guessing the W5100 needs to "present itself" to the network somehow but how does that mechanism work? Anyone knows or have a reference you can point me at?

/Henrik.

[dhouston]

Anyone knows what needs to be done to have a device listen to a "name" as well as its IP-adress?
For example, I have a SOHO server which has a web-interface, I can access that by either typing in its IP-adress or its name in the adress field of the browser.I think your SOHO server (or router) has a DNS/DHCP server which maintains a list of names and corresponding IP addresses. This page (http://www.wiznet.co.kr/Sub_Modules/en/technical/q_a.asp) might help.

[HenrikOlsson]

Thanks a lot Dave!
Simply knowing it's the DNS servers job pairing up names with IP-numbers narrows the search for me. I feel I should've know that but for some reason I was only thinking of DHCP which I knew it wasn't.
Wiznet has an appnote on DNS but I'm not sure it pertains to the W5100, but again, now I know what to look for.

Thanks!
/Henrik.

[HenrikOlsson]

Hi,
I now have a working DHCP client. This means I can just plug in the network cable and the thing gets an IP-adress from the DHCP server (my router in this case). It also passes a host name to the DHCP-server which then somehow passes that on to the DNS server that apparently runs on the router. In the end it means that I can now get to it by browsing to http://amicus or whatever name I wish the unit to have - good times :-)

But, this has lead me to the unavoidable...
The routines previously posted are more or less hardcoded to use socket 0. This is of course not optimal and now is the time to do something about that. Each socket has its own control and status registers etc at a fixed offset of $100 between them so it's pretty easy to calculate the correct adress based on the socket we want the routine to access. For example:

WizSocket VAR BYTE
WizSocket = 2

WizAdress = W5100_S0_MR + (WizSocket * $100) : WizData = 1 : GOSUB Write_W5100
This will access socket 2's mode register - pretty straight forward.

The problem comes with the resizable circular receive and transmitt buffers. The only fixed point here is the start of socket 0's RX and TX buffer. The end of it as well as the start AND end of the other sockets buffers can move around based on how we set it up. I'm looking for ideas on how to handle this effeciently and user friendly.

Currently I've done this:

GetSocketRXAdress:
' Complete routine executes in 174 cycles.
WizAdressOffset = WizSocket * $100

' These 3 lookup statements takes 259 bytes of program memory
Lookup2 WizSocket, [W5100_S0_RX_Start, W5100_S1_RX_Start, W5100_S2_RX_Start, W5100_S3_RX_Start], Socket_RX_Start
Lookup2 WizSocket, [W5100_S0_RX_END, W5100_S1_RX_END, W5100_S1_RX_END, W5100_S1_RX_END], Socket_RX_End
Lookup2 WizSocket, [W5100_S0_RX_MASK, W5100_S1_RX_MASK, W5100_S2_RX_MASK, W5100_S3_RX_MASK], Socket_RX_MASK

RETURN
And of course a similar routine for the TX buffers. Instead of using W5100_S0_RX_Start etc the routines that accesses the RX buffer now uses Socket_RX_Start etc instead.

These routines are then called when "changing sockets" meaning that we don't have to calculate the adresses every time we access a register we only do it when changing sockets. The routine to check the amount of free space in any sockets TX buffer then looks like this:

GetFreeTXSize:
If WizSocket <> WizOldSocket THEN ' No need to get offsets etc if we're using the same socket as last time.
GOSUB GetSocketTXAdress
ENDIF

' Get the total amount of free memory in the sockets transmit buffer.
WizAdress = W5100_S0_TX_FSR0 + WizAdressOffset : GOSUB Read_W5100 : WizSize.HighByte = WizData
WizAdress = W5100_S0_TX_FSR1 + WizAdressOffset : GOSUB Read_W5100 : WizSize.LowByte = WizData

' Get the pointer adress of where to put new data.
WizAdress = W5100_S0_TX_WR0 + WizAdressOffset : GOSUB Read_W5100 : WizPtr.HighByte = WizData
WizAdress = W5100_S0_TX_WR1 + WizAdressOffset : GOSUB Read_W5100 : WizPtr.LowByte = WizData

'Calculate physical adress of first byte
WizStart = Socket_TX_START + (WizPtr & Socket_TX_MASK)

WizOldSocket = WizSocket ' Remember which socket we used.
RETURN

I think this is a workable solution to allowing access to any socket without duplicating all the routines with hardcoded adresses AND without having to calculate/retreive the adresses every time we access the W5100.

What I'm looking for is ideas on how to make it more efficient, I particullarly dislike the fact that the lookup routines (for both TX and RX buffers) takes well over 500 bytes (that's something like 10% of a my complete working web-server including a couple of hundred bytes HTML). Me think there must be a slicker way...

Thank you in advance!

/Henrik.

[pedja089]

Try something like this

Socket_RX_Start var word[4]
Socket_RX_Start[0]=W5100_S0_RX_Start
Socket_RX_Start[1]=W5100_S1_RX_Start
Socket_RX_Start[2]=W5100_S2_RX_Start
Socket_RX_Start[3]=W5100_S3_RX_Start
Then you won't need GetSocketRXAdress. In routines just use
WizStart = Socket_RX_Start[WizSocket] + (WizPtr & Socket_TX_MASK)

[HenrikOlsson]

Hello again,
I guess you can all see the bug in the Lookup code - the line where it looks up the sockets end adress currently returns the wrong adress for socket 2 and 3. As usual I can't edit the post :-(

/Henrik.

[HenrikOlsson]

Thanks!
That's one solution which I actually have thought of but it comes at a cost of 48 vs 12 bytes of RAM. But in the end it might be worth it though, I'll give it second thought, thanks!

/Henrik.

[pedja089]

http://www.picbasic.co.uk/forum/showthread.php?t=14923
Look at post #5.
You can store data in eeprom, but then you must use READ.

[HenrikOlsson]

Hello,
Now that we know how to setup the W5100 to listen for TCP trafic on the correct port, how to determine if there's data to be read in the sockets RX memory buffer and where in the buffer that data is it's time to look at how we can parse the received data and act upon it. After all, simply "resending" the received data out the USART may be cool (for 5 minutes) but it doesn't really do anything for us, though it does serve as a nice debugging tool.

So today I'll try to cover how we can load the recevied data from the W5100 into a local buffer, do some (for now very) simple parsing of that data in order to determine what the client wants and then build and send a response. The end result will be a simple web-page displayed in the browser window.

Trying to parse the data directly in the W5100 proved to be a bit tricky so I decided to move it to RAM in the PIC and do the parsing there. To do this we first need somewhere to put the data, an array called Buffer of some, for now arbitrary, length:

BufferSize CON 128
Buffer VAR BYTE[BufferSize]' Local buffer for loading and unloading data from the W5100
Then I modified the GetData routine posted earlier to not only "print" the received bytes on the serial terminal but to also stuff them into our previously declared array. The GetData routine now looks like this:

'************************************************* ************************************************** **
GetData:
HSEROUT [REP "-"\20, " Data read from Rx buffer follows ", REP "-"\20, 13]

Counter = 0 ' Start loading the local array buffer at the beginning.

' Do we have a buffer wrap around condition?
If WizEnd < WizStart THEN

' When that happends we need to start reading at the first byte in the received
' package and go on until the end of the sockets RX memory....
For WizAdress = WizStart to W5100_S0_RX_END
Gosub Read_W5100
HSEROUT[WizData]

' Make sure we don't write outside of the allocated buffer.
If Counter <= BufferSize THEN
Buffer[Counter] = WizData
ENDIF

Counter = Counter + 1 ' Keep count

NEXT

' ...then we need to start over at the beginning of the buffer and keep reading
' until we've got all the bytes in the buffer.

For WizAdress = W5100_S0_RX_START to WizEnd
Gosub Read_W5100
HSEROUT[WizData]

' Make sure we don't write outside of the allocated buffer.
If Counter <= BufferSize THEN
Buffer[Counter] = WizData
ENDIF

Counter = Counter + 1 ' Keep count
NEXT

ELSE ' There's no buffer wrap-around.

' WizStart now contains the physical adress of the first byte to read.
For WizAdress = WizStart to WizEnd
Gosub Read_W5100
HSEROUT[WizData]

' Make sure we don't write outside of the allocated buffer.
If Counter <= BufferSize THEN
Buffer[Counter] = WizData
ENDIF

Counter = Counter + 1 ' Keep count

NEXT

ENDIF

HSEROUT [REP "-"\23, " End of data from Rx buffer ", REP "-"\23, 13]
HSEROUT ["Total number of bytes read:", #Counter,13]
HSEROUT [REP "-"\76, 13, 13]

' The variable RxPtr contains the "raw value" read from the sockets Rx Read Pointer Register.
' Now we need to be update that so it points one "step" ahead of the last location we just read.
' This is done to let the W5100 how "far" in the buffer we've read and that it can use any space
' "before" this location for new data. Note that this is not the ACTUAL adress of the data,
' just the unmasked index.

WizPtr = WizPtr + WizSize
WizAdress = W5100_S0_RX_RD0 : WizData = WizPtr.HighByte : GOSUB Write_W5100
WizAdress = W5100_S0_RX_RD1 : WizData = WizPtr.LowByte : GOSUB Write_W5100

' Then we send the RECV command to the sockets command register to let it know that
' we've handled all the data we currently are prepared to handle. This updates the
' pointers in the sockets memory so that it knows where to put new data. It is not
' until after issuing the RECV command that the sockets RX pointer is actually updated
' even though we write to it above.

WizAdress = W5100_S0_CR : WizData = W5100_Sn_RECV : GOSUB Write_W5100

RETURN
'************************************************* ************************************************** **
As you can see we are no longer disconnecting and closing the connection at the end of the GetData routine, the reason is obviously because we intend to send a response to the client so we need to keep the connection open until we've done that.

Now that we have the data in our local array Buffer (or at least the first 128 bytes of it) we are ready to start analysing it in order to determine what the client wants. Looking back at the screenshot of the received data being printed to the serial terminal we can see that simply browsing to the W5100's IP adress results in the first couple of bytes in the buffer looking like this:

GET / HTTP/1.1
What we're interested in here is the GET part, that is what the client wants to do - it wants to GET something from our server. The client could have specified a specific resource to get after the first forward slash, like

GET /thatpage.html HTTP/1.1
But for now we won't look at serving more than a single resource so as soon as we see the word GET we know we should respond with the webpage we've got in the PIC. Here's a very simple routine to parse the buffer, starting at the beginning:

'************************************************* *****************************************
' Very simple routine to parse the data in the local buffer.
' If the first word is GET we set a semaphore signaling the main program that we've got
' a GET request.
'************************************************* *****************************************
GetRequest VAR BIT

ParseData:

If Buffer[0] = "G" THEN
IF Buffer[1] = "E" THEN
IF Buffer[2] = "T" THEN
IF Buffer[3] = " " THEN
GetRequest = 1
HSEROUT["Found GET request",13]
ENDIF
ENDIF
ENDIF
ENDIF

RETURN
Once we've found the word GET at the very beginning of the received data we're going to respond with a simple web-page. To do that the first order of business is to determine the amount of free space in the socket transmit buffer and where it wants us to start loading the data. This is done in a similar way to checking the RX-buffer, in fact we're reusing the same variables (WizPtr, WisStart and WizSize) for this purpose.

'************************************************* ************************************************** **
' This routine checks the amount of free space in the sockets transmit register.
' It also gets the pointer of where to start loading data and calculates the
' physical adress for us. It returns the amount of free space in WizSize and
' the adress of the first byte in WizStart. WizPtr is the "raw" pointer value
' and is used to update the socket transmit pointer after writing to the buffer.
'************************************************* ************************************************** **
GetFreeTXSize:

' Get the total amount of free memory in the sockets transmit buffer.
WizAdress = W5100_S0_TX_FSR0 : GOSUB Read_W5100 : WizSize.HighByte = WizData
WizAdress = W5100_S0_TX_FSR1 : GOSUB Read_W5100 : WizSize.LowByte = WizData

' Get the pointer adress of where to put new data.
WizAdress = W5100_S0_TX_WR0 : GOSUB Read_W5100 : WizPtr.HighByte = WizData
WizAdress = W5100_S0_TX_WR1 : GOSUB Read_W5100 : WizPtr.LowByte = WizData

'Calculate physical adress of first byte
WizStart = W5100_S0_TX_START + (WizPtr & W5100_S0_TX_MASK)

RETURN
When we return from this subroutine the amount of free space is available to us in WizSize, the raw pointer value (working exactly as the pointer into the RX memory buffer) is avaliable to us in WizPtr and the actual, physical, memory adress of the first free location is available to us inWizStart.

In order to somewhat effeciently load HTML data into the buffer I came up with the following routine. What it does is loading null-terminated strings from the local array Buffer to the sockets TX memory, it handles the circular buffer and it keeps track of the total number of bytes loaded to the buffer so we can tell the W5100 how "far" into the buffer it should send.

'************************************************* *****************************************
' Subroutine to load a string from the local array Buffer to the W5100 socket 0's
' TX memory buffer. The string can be any length but must end with NULL.
'************************************************* *****************************************
PutStringInTXBuffer:

Counter = 0 ' Counter for indexing the local array we're about to load into the TX buffer.

While Buffer[Counter] <> 0 ' Last byte in each part is null.

' Check if we are about to write outside the sockets memory area.
' If so we wrap around to the start of the buffer.

If WizStart + TotalCount > W5100_S0_TX_END then
WizStart = W5100_S0_TX_Start - TotalCount
ENDIF

' Write byte to TX memory buffer.
WizAdress = WizStart + TotalCount : WizData = Buffer[Counter] : GOSUB Write_W5100
Counter = Counter + 1 ' Point at the next byte in the array.
TotalCount = TotalCount + 1 ' Count total number of bytes for the response.
WEND

RETURN
We can then "feed" the PutStringInTxBuffer routine with data like in the first half of the HandleGetRequest routine:

'************************************************* *****************************************
' Subroutine used to build a simple web-page in response to a GET request.
'************************************************* *****************************************
HandleGetRequest:

RequestCount = RequestCount + 1

GOSUB GetFreeTXSize

' Reset the total byte count. This will be incremented in the PutStringInTXBuffer routine
' and used to keep track of the total number of bytes loaded to the TX memory buffer.
TotalCount = 0

HSEROUT ["Loading buffer...",13]

ArrayWrite Buffer,["HTTP/1.0 200 OK",13,10,0]
GOSUB PutStringInTXBuffer

ArrayWrite Buffer,["Content-Type: text/html",13,10,13,10,0]
GOSUB PutStringInTXBuffer

ArrayWrite Buffer,["<html><body><title>AMICUS18</title>",13,10,0]
Gosub PutStringInTXBuffer

ArrayWrite Buffer,["<h1>Embedded Web Server</h1>",13,10,0]
Gosub PutStringInTXBuffer

ArrayWrite Buffer,["<h3>PIC18F25K20 / W5100 Ethernet chip</h3>",13,10,0]
GOSUB PutStringInTXBuffer

ArrayWrite Buffer,["<p>Henrik Olsson 2011</p>",13,10,0]
GOSUB PutStringInTXBuffer

ArrayWrite Buffer,["<a href=",34,"http://www.picbasic.co.uk/forum",34,">Visit the PBP Forum</a>",13,10,0]
Gosub PutStringInTXBuffer

ArrayWrite Buffer,["<p>This page has been requested ", DEC RequestCount, " times since powerup/reset</p>",13,10,0]
GOSUB PutStringInTXBuffer

ArrayWrite Buffer,["</body></html>",13,10,13,10,0]
GOSUB PutStringInTXBuffer

HSEROUT[13, "done...",DEC TotalCount, " bytes.."]
The above is the actual HTML response that we're sending as a result of the GET request. What's important here is that we must reset the TotalCount variable before starting to load data and then not touch it until we're done. Each "part" of the response must end with NULL as can be seen in the ArrayWrite operations. There's no real requirement to split the response up in a certain way, it's more for the purpose of readabillity.

Using ArrayWrite like this is a simple aproach but it seems very (VERY) inefficient because each and every character/byte of adds 6 bytes(!) to the footprint of the code. This isn't really a problem for this type of simple web-page but for more complicated stuff it'll add up quickly. We'll have to look into getting the SD card working as a storage "container" for the HMTL pages but that's another topic.

Once we've got all the data loaded to the TX buffer we're ready to send it and this is handled by the second half of the HandleGetRequest subroutine:

' Now recalculate the transmit pointer so the W5100 knows where to stop sending. WizPtr currently contains
' the adress of the first free byte and TotalCount is the number of bytes of byte we've loaded to the buffer.
WizPtr = WizPtr + TotalCount

' Write the updated pointer value to the W5100
WizAdress = W5100_S0_TX_WR0 : WizData = WizPtr.HighByte : Gosub Write_W5100
WizAdress = W5100_S0_TX_WR1 : WizData = WizPtr.LowByte : Gosub Write_W5100

' Issue a SEND command.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_SEND : GOSUB Write_W5100

WaitForCompletion:
GOSUB Read_W5100
If WizData <> 0 then Goto WaitForCompletion

HSEROUT[".sent.",13]

' Then disconnect from the client.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_DISCON : GOSUB Write_W5100

' And close the socket. We'll reopen the socket again in the main routine.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_CLOSE : GOSUB Write_W5100
GetRequest = 0

RETURN
Note that I don't actually verify if the data I'm about to load will fit in the amount of free space provided in the sockets TX memory. Not doing that should be considered bad practice but for now I'll do it anyway since I know that the size of the buffer is more than 6 times the size of the data I'm writing to it.

Now all we need to do in our main routine is to:
1) Check the received size register of the socket, if it's >0 we'll handle the received data using the above routines.
2) Since we're closing the socket after sedning the data it's the main routines job to re-open it and set it to listen mode.

HSEROUT ["Program start",13]

GOSUB Init_W5100 ' Reset W5100 and set MAC, IP, NetMask and Gateway adress,

Main:
' The socket 0 status register automatically switches to SOCK_ESTABLISHED
' when a valid request to connect comes in from a client.
WizAdress = W5100_S0_SR : GOSUB Read_W5100

If WizData = W5100_SOCK_ESTABLISHED then
Gosub CheckBufferForData
If WizSize > 0 then
GOSUB GetData
GOSUB ParseData
If GetRequest = 1 then GOSUB HandleGetRequest
ENDIF
ENDIF

' If the socket is closed (we're closing it in the GetData routine) then
' we must re-open it so we are prepared for the next client request.
WizAdress = W5100_S0_SR : GOSUB Read_W5100
If WizData = Socket_CLOSED THEN
GOSUB Init_Socket_0
ENDIF

Pause 250
Goto Main
To this post I'm attaching the source file (remove the .txt extension and you'll need the W5100_defs.pbp file as well) I used to arrive at the following results:

5708

It compiles to 5275 bytes out of which the ArrayWrite statements (with a total of ~340 bytes of HTML) occupies 2038 bytes - yikes. So, without the HTML we're looking at less than 3.5k for the actual server code - not bad.


/Henrik.

[ScaleRobotics]

Henrik, I just wanted to thank you for the excellent job you are doing. Very exciting! You have made it so easy for the rest of us. I just received my Wiznet module, the WIZ811MJ, which contains a W5100. So I hope to join the fun when time permits.

Thanks again.

[HenrikOlsson]

Thanks Walter!
And please do join the fun!

Perhaps we should emphasise, in case that isn't 100% clear, that it's not needed to have an AMICUS18 board and Ethernet shield, it just happens to be my personal development setup. As long as you have a W5100 connected to a PIC via SPI you're all set - like a WIZ811MJ (http://www.sparkfun.com/search/results?term=WIZ811MJ&what=products) and your favourite PIC on a piece of perfboard.

You can even re-write the Read_W5100 and Write_W5100 subroutines if you want to use shiftout/shiftin instead (it'll be kind of slow though....) or if you want to go to the more traditional adress/data interface (requires a boatload of I/O). That's all you'd need to change, all the other routines go "thru" those two so should work just the same.

What I'm not happy with so far is that all posted code has been hardcoded for socket 0. I have changed it, as discussed in an earlier post, but decided to keep these initial examples hardcoded to keep it easier. Also, I'm NOT happy with the way ArrayWrite literally EATS the codespace, we'd have to come up with a SD card routine or perhaps one of the strings in codespace routines mixed with a couple of ArrayWrite for the dynamic data.

Oh well, one step at a time...

/Henrik.

PS. DHCP is now working nicely, with proper handling of leasetime and renewal of lease. Very cool but it adds about 3kB and 42 bytes of RAM to the total footprint when included, I'll try to get that down a bit further.

[HenrikOlsson]

Hi everyone,
Time for an update. As I wrote in the previous post I've had DHCP working nicely and today I thought it was time to show you how it can be used. However, for this to work nicely, there had to be some changes made to the overall structure of the package of files used. It gets easier to USE but a tad more complicated to maintain. I won't go into specific details, just give you the overall picture.


First of all, the code now requires PBP3. This is beacuse I've added a bunch of conditional debug statements in the code making use of the wonderful conditional compilation feature of PBP3. For example, you can add #W5100_DEBUG_LEVEL 1 to the top of the code and you'll get a whole bunch of data sent over the USART, if you want even more details do #W5100_DEBUG_LEVEL 2. Do note though that these takes up a lot of program space so if the application grows they might not fit when "activated".


The main include file for using the W5100 is called W5100.pbp and that's what you should included from the main program/application. It is also in THIS file that you may need to make some changes to match your project:

1) BufferLenght CON 256 - This is the length of the local buffer (array) that gets declared, you can change this to match your needs. Don't make it too small though, the DHCP routines - if you're going to use those, requires a lenght of ~48 minimum.

2) MAC-adress - This is the hardware adress of the device, change as needed - and please DO change it, don't use mine! As can be seen the MAC adress is defined as constants and can therefor currently not be changed at runtime - this MAY change or you can change it yourself.

3) W5100_Select VAR PortB.5 - this is the PIC-pin to which ChipSelect pin of the W5100 is connected. My modified Arduino Ethernet Shield uses PortB.5

This file (W5100.pbp) then includes two more files - W5100_defs.pbp (which we've looked at before) and W5100_subs.pbp which contains several of the routines discussed in earlier posts as well as others. Neither of these files 'should' need to be edited.


In previous "exercises" we've hardcoded the IP-adress, netmask and gateway address of the W5100 - for DHCP to work properly this can't be done. So I've declared three arrays (in W5100_subs.pbp) called W5100_IP, W5100_Netmask and W5100_Gateway - all 4 bytes long. To set the network parameters of the W5100 we can now do something like:

W5100_IP[0] = 192
W5100_IP[1] = 168
W5100_IP[2] = 1
W5100_IP[3] = 100

W5100_NetMask[0] = 255
W5100_NetMask[1] = 255
W5100_NetMask[2] = 255
W5100_NetMask[3] = 0

W5100_Gateway[0] = 192
W5100_Gateway[1] = 168
W5100_Gateway[2] = 1
W5100_Gateway[3] = 254

GOSUB Init_W5100
The Init_W5100 subroutine uses the above arrays and sets the parameters of the W5100 accordingly.

To be continued in the next post....

[HenrikOlsson]

...continued from previous post.

OK, Dynamic Host Configuration Protocol - DHCP.
DHCP is used to automatically configure a networkdevice without actually knowing anything about the network it's attached to. There are hundreds and hundreds of pages to read about how it works so I won't cover it in much detials, just give a basic overview.


The process starts with the client sending out what's called a DHCP Discover message. This is sent as a brodcast message meaning it's addressed to any and all DHCP servers on the current subnet. One, or more, servers(s) will respond with a DHCP Offer message containing an offer for a lease of a specific IP adress for a specific amount of time. The client can then selects one of these offers and sends a DHCP request message, still as a broadcast message but it has now filled in the server it wants to use in a specific field in the message. If everything goes well the server responds with a DHCP Ack message meaning that it has now reserved the IP-adress for the amount of time specified and the client is free to use it. The client is NOT allowed to start using the IP-adress untill a valid DHCP Ack message is received. Along with the IP adress the server also provides other important setting such as the netmask and the gateway adress.

Like I said, this is just the basic outline of the process - there are alot of details to get into, which we don't have to do right now.


The DHCP implementation for the W5100/PBP3 is supplied as three files. The main file to include in your project is DHCP.pbp It's also in this file that a change or two may be needed. At the top, you'll find this:

HostNameLength CON 6 ' <- Change this to match the length of the name.
HostName VAR BYTE[HostNameLength]
HostName[0] = "A"
HostName[1] = "m"
HostName[2] = "i"
HostName[3] = "c"
HostName[4] = "u"
HostName[5] = "s"
This is the host name that is supplied to DHCP server. It then (if it has the functionallity) passes it on the DNS server and the end-result is that the client device can be accessed by name, ie you can browse to http://amicus instead of trying to remeber the IP-adress. What you need to change is the HostNameLenght so that it matches exactly the lenght of the array containing the name you want to use. As in the example above the name is 6 bytes long therefor HostNameLenght is 6.

The DHCP.pbp file then includes two more files, DHCP_subs.pbp and DHCP_defs.pbp - neither of which 'should' need any editing for normal use.

As with the main "driver" files for the W5100 I've added a while bunch of debug-messages to the DHCP code. These can be enabled by adding a #DEFINE DHCP_DEBUG_LEVEL 1 or #DEFINE DHCP_DEBUG_LEVEL 2 to the application. Again, these adds quite a bit of code.

When the DHCP process is complete we can get the network setting to use from the arrays DHCP_IP, DHCP_NetMask and DHCP_GateWay and use these to initilise the W5100.

To be continued in the next post....

[HenrikOlsson]

...continued from previous post...

OK, lets write some code....(finally)....

The first thing to do, as always is to set up the hardware. I'm using the AMICUS18 and the modifed Arduino Ethernet Shield, here's the first part of the code:

DEFINE OSC 64 ' Default oscillator speed on the AMICUS18 is 16*4MHz
DEFINE LOADER_USED 1 ' We're using a bootloader.
DEFINE HSER_RCSTA 90h ' Enable serial port & continuous receive
DEFINE HSER_TXSTA 24h ' Enable transmit, BRGH = 1
DEFINE HSER_CLROERR 1 ' Clear overflow automatically
DEFINE HSER_SPBRG 138 ' 115200 Baud @ 64MHz, -0,08%

SPBRGH = 0
BAUDCON.3 = 1 ' Enable 16 bit baudrate generator

TRISC = %10010110 ' RX, TX, MOSI, MISO, SCK, N/U, N/U, LED
TRISB = %11011011 ' N/U, N/U, W5100_CS, N/U, N/U, SD_CS, N/U, W5100_Interrupt

' Aliases for the MSSP module status and control bits.
SSPEN VAR SSPCON1.5 ' SSP Enable bit
CKP VAR SSPCON1.4 ' Clock Polarity Select
SMP VAR SSPSTAT.7 ' Data input sample phase
CKE VAR SSPSTAT.6 ' Clock Edge Select bit
SSPIF VAR PIR1.3 ' SPI interrupt flag

milliseconds VAR WORD ' Timer variable, used in the ISR.
TMR1_Temp VAR WORD ' Used when reloading the timer in the ISR
i VAR BYTE ' General purpose.

CLEAR

' Set up the MSSP module.
CKP = 0 ' Clock idles low
CKE = 1 ' Transmit on active to idle transition.
SSPIF = 0 ' Clear buffer full status
SMP = 0 ' Sample in middle of data
SSPEN = 1 ' Enable SPI pins
SSPCON1.0 = 1 ' Slow down SPI clock to Fosc/16 for now.

Then we need some what to keep track of time. The DHCP server leases us an IP-adress for a finite (usually) amount of time and we need to keep track of that time so we can renew the lease before the time runs out. For this purpose I'm using Darrels DT-Ints so I'm including his files and while I'm at it I'll also include the W5100.pbp and the DHCP.pbp files and the set up an interrupt to trip when TMR1 overflows:

INCLUDE "DT_Ints-18.bas" ' <- Darrel Taylores interupt engine
INCLUDE "ReEnterPBP-18.bas" ' <- Part of Darrels interrupt engine.
INCLUDE "W5100.pbp" ' <- Main W5100 "driver" and auxiallary routines.
INCLUDE "DHCP.pbp" ' <- Main file for the DHCP engine.

'--- Set up TMR1 interupt.
ASM
INT_LIST macro ; IntSource, Label, Type, ResetFlag?
INT_Handler TMR1_INT, _Tick, PBP, yes
endm
INT_CREATE ; Creates the interrupt processor
ENDASM

TMR1_Reload CON 25539 ' Reload value for 200Hz @64MHz, 1:2 prescaler
T1CON.0 = %00010001 ' Start timer 1, 1:2 prescaler
@ INT_ENABLE TMR1_INT

I've put my TMR1 Interrupt service routine at the end of the program but to keep the disussion on track I'll show it here:

' This is the TMR1 interrupt service routine.
Tick:
milliseconds = milliseconds + 5

If milliseconds => 1000 THEN
DHCP_Seconds = DHCP_Seconds + 1
milliseconds = 0
ENDIF

T1CON.0 = 0

TMR1_Temp.HighByte = TMR1H
TMR1_Temp.LowByte = TMR1L
TMR1_Temp = TMR1_Temp + TMR1_Reload
TMR1H = TMR1_Temp.HighByte
TMR1L = TMR1_Temp:LowByte

T1CON.0 = 1
@ INT_RETURN

This runs at ~200Hz, each time it increments the millisecond variable by 5 and when it reaches 1000 (1 second) the variable DHCP_Seconds is incremented - and this is the important thing. The DHCP code must be called periodically, even after an IP-adress had been recieved. This is because it has to keep track of the lease time which it does by subtracting the number of seconds on DHCP_Seconds from the total lease time given to us by the server. It then resets DHCP_Seconds to 0.

So, "we" must make sure that DHCP_Seconds contains "true time" and GOSUB DHCP periodically. We don't have to increment DHCP_Seconds at exactly 1Hz, we can increment it by 10 every 10 seconds or whatever but it's impotant that it's fairly close to real time. Likewise, we don't have to GOSUB DHCP every second or even every minute but because DHCP_Seconds is a BYTE variable we must atleast do it once per 4 minutes or there abouts.

Next we initilise the W5100 and set the network parameters to "all zeros":

Begin:
HSEROUT["Program start...", 13]
' A substantial pause is needed for the W5100 to start up and establish
' a link with the network.
Pause 2000

' We don't know any network parameters so these should all be set to 0
' when using DHCP.
HSEROUT["Clearing IP, Netmask and Gateway adress",13]
For i = 0 to 3
W5100_IP[i] = 0 ' Set our IP adress to 0.0.0.0
W5100_Netmask[i] = 0 ' Set our netmask to 0.0.0.0
W5100_GateWay[i] = 0 ' Set gatway adress to 0.0.0.0
NEXT
' Now initilise the W5100 with the above parameters.
GOSUB Init_W5100

There's a bit variable called DHCP_Enabled which should be set to 1 if DHCP is to be used. There might be ocations where we want to have the abbility to use DHCP but also to turn it off, this flag makes sure that the DHCP code doesn't execute, even if called, if not specifically "enabled". Then we must tell the DHCP code which of the W5100's four sockets to use, this is done thru the DHCP_Socket variable. Finally we set the state of the DHCP state machine (it's all implemented as a big state machine by the way) to DHCP_State_Start.

Now, all we need to to is keep GOSUB'ing DHCP periodically and if everything goes well we'll have all the network parameters we need within a second or three:

DHCP_Enabled = 1 ' Enable DHCP.
DHCP_Socket = 2 ' Use socket 2 for DHCP.
DHCP_State = DHCP_State_Start ' Reset the DHCP state machine.

' Loop around here until we recieve a valid IP adress.
HSEROUT["Entering DHCP",13]
While DHCP_State <> DHCP_State_Bound
GOSUB DHCP
Pause 500
WEND

The key to detect when the process is finished is that the state variable (DHCP_State) is set to DHCP_State_Bound and that's what weäre using here. We keep calling DHCP untill we see the correct state. As long as we also updates DHCP_Seconds in the background the DHCP state machine will automatically issue retransmissions at specific intervals (as per the DHCP specification).

Once we see the bound state we can get the parameters from the DHCP_IP, DHCO_NetMask and DHCP_GateWay arrays and use those to re-initilise the W5100:

' At this point we should have received a set of valid network paramters from
' the DHCP server on the network. Let's start by simply printing them out.
HSEROUT["The offered IP adress is: "]
For i = 0 to 3
HSEROUT[#DHCP_IP[i], "."]
NEXT

HSEROUT[13,"The gateway is: "]
For i = 0 to 3
HSEROUT[#DHCP_Gateway_IP[i], "."]
NEXT

HSEROUT[13, "The netmask is: "]
For i = 0 to 3
HSEROUT[#DHCP_NetMask[i], "."]
NEXT

' The DHCP engine does NOT actually SET the IP adress for us, it just TELLS
' us what to use. We must make the changes ourselves:
HSEROUT[13,"Setting the W5100 parameters..."]
For i = 0 to 3
W5100_IP[i] = DHCP_IP[i]
W5100_GateWay[i] = DHCP_GateWay_IP[i]
W5100_NetMask[i] = DHCP_NetMask[i]
NEXT


' Now re-initilise the W5100 with the new network parameters.
GOSUB Init_W5100
HSEROUT["DONE!",13]
HSEROUT["Try pingning by IP or name!",13]


Don't forget that we still need to have DHCP_Seconds updated in the ISR and we still have to GOSUB DHCP on a regular basis so that it can automatically renew the IP adress lease when it's about to expire:

Main:
Toggle PortC.0
Pause 500
GOSUB DHCP
Goto Main


6020

6021
Attached to this post are all the updated files (W5100.pbp, W5100_defs.pbp, W5100_subs.pbp, DHCP.pbp, DHCP_defs.pbp, DHCP_subs.pbp and the DHCP_Demo application used to get the above results.

/Henrik.

[HenrikOlsson]

In post #24 above, second paragraph regarding the conditional debug output. Those two statements should of course read:
#DEFINE W5100_DEBUG_LEVEL 1 or #DEFINE W5100_DEBUG_LEVEL 2

As usual we're not allowd to correct such mistakes if not imediatley detected :-(

[mackrackit]

WOW!!!
Nice work Henrik.

[pedja089]

Hi,
Nice work. But i think that you have some mistake.
You wrote


W5100_Gateway[0] = 192
W5100_Gateway[0] = 168
W5100_Gateway[0] = 1
W5100_Gateway[0] = 254
If i understand it should be

W5100_Gateway[0] = 192
W5100_Gateway[1] = 168
W5100_Gateway[2] = 1
W5100_Gateway[3] = 254

[HenrikOlsson]

Ah, yes of course, thanks for spotting that. It should be OK in the example project attached to previous message.
I wish I could go back and fix it in the post but I can't :-(

/Henrik.

[pedja089]

Maybe you should be moderator on AMICUS18 section...

[mackrackit]

The code in post #24 is fixed.

[Demon]

Henrik, Bravo!

Robert

[Heckler]

Hey Henrik (and others),
I just received my Amicus18 and the Arduino Ethernet Shield...

I have not completed the modifications yet... but am starting that tonight.

I assume your example code is intended to overwrite the Amicus boot loader? (I did read the amicus with my PicKit2 and save the hex file, so I think I can go back to the boot loader if I want.)

How has your progress been?? Where are you hoping to go with your experiments?

I would like to end up with a device that...
I can log into and read or view values like temperature, voltage status bits etc.
That could possible email me or text me if a given value changes beyond a set point.
Be able to remotely change values and bits.
?? other things yet un thought up.

I do not have any experience programing ethernet values via a PIC.

I have plenty of experience Pinging... seting static IP... subnet mask... Telnet... etc. From a Windows XP machine.

Thanks for any help, direction and code examples you (and others) can give.

[Heckler]

Henrik,

My Amicus board came with a 18F25K20 NOT the 18F25K22 that you mention in your article. I see the difference is 3.3v vs 5v. Am I going to need to get the K22 chip?

Thanks

[HenrikOlsson]

Hi Dwight,
Wonderful, please join the fun!

No, I wrote 25K22 in error, it's meant to say 25K20 so you're fine - just use the AMICUS18 as it is. (You CAN definitely use the 25K22 if you want but I think you'd need to run it on 3.3V, I don't remember if the W5100 has 5V tolerant inputs). I'd like to stress, again, that you don't need an AMICUS18 at all, the routines will work with ANY PIC that has a MSSP module.

No, don't overwrite/replace the bootloader. Add DEFINE LOADER_USED 1 at the top and set MCSP (or whatever you use) up to use the AMICUS Loader. It'll be a lot faster than re-flashing with the PicKit2.

Progress is good but a bit slow. This is all new territory for me, researching protocols, reading RFC's, trying, failing, trying again.

The goal is basically a set of routines, lets call it a "framework", which you can use to build the application you need. It won't do everything but it'll provide the base functionallity upon which we can write the actual application. At the moment I have the basic routines used to communicate with the W5100 and a couple of support routines to make using sockets and moving data in and out of easier. This is allready presented and published in the thread.

I have DHCP (Dynamic Host Configuration Protocol) working - also presented in the thread.
I have SNTP (Simple Net Time Protocol) working, (derived from Charles really accurate clock code but without using LONGs)
I have SMTP (Simple Mail Transfer Protocol) (with authentication) basically working (it sends mails) but it needs more work to increase the robustness and improve the interface to the main application.

When I'm happy with the SMTP code I think I'll take on DNS and if/when that works I think I'll try to make some kind of HTTP server "building block" but I'm not sure it'll work since each and every application will be different. Perhaps it's just better to write that from scratch - I don't know. The biggest hurdle, as I see it, is how to handle a "filesystem" of some sort on the PIC. A single, very simple web-page is not a problem but it gets complicated when you want multiple pages, perhaps images etc. Actually storing them (SD-card, SPI-flash, I2C-EEPROM whatever) is one part, keeping track of them (ie. the filesystem) is another and how to get them INTO the memory device is a third.

The application you envision is pretty much what I had in mind when I started but I'm trying to write the code as generic as possible. That way you can included the functionallity you need and leave the rest out. With the code and examples already presented you should be able to get DHCP and a simple web-server going. Perhaps you'll get a general HTTP server building block going before I get to it.

Keep us posted on your progress!

/Henrik.

[HenrikOlsson]

Hi,
Just thought I'd give those with interest in this project a short update of what I've been doing.

Take a look at the following screenshot:

6085

The screenshot itself is pretty self explainatory but I'll do a short explanation of what is happening (it's quite a bit).

At bootup the DHCP code is executed to retrieve the network parameters from the DHCP server, then the W5100 is configured with those parameters. Then the SNTP code is executed to retrieve current time from an SNTP server on the net. Then a series of DNS queries are run, looking up the IP-adress of some well known URL's. The last URL that DNS queries is the SMTP server for my email account. Once it gets that IP-adress it uses it in the SMTP code to login to my account, using authentication (no encryption just AUTH LOGIN (BASE64 encoded username and password)), and sends an email to my account - which you can see next to the serial terminal.


There is one thing I don't get though. Browsing to to the IP-adress retrieved by the DNS code gets me to the correct places in all but ONE instances, namely THIS SITE. If I PING www.picbasic.co.uk (http://www.picbasic.co.uk) I get the same IP as my DNS code and if I run Wireshark and browse to www.picbasic.co.uk (http://www.picbasic.co.uk) its DNS-records shows the same IP-adress but if I browse to that very same IP I do not end up here. I'm pretty sure it's not an error in my code since every other URL I've tried seems to work. It has to be something specific with where and how this site is hosted. Anyone know, and can explain, what's going on?

Anyway the DNS code currently "only" handles normal queries, not reversed or any other type, like MX etc. It does handle both direct type A responses and CNAME responses, either single or "nested". For example, the query for microsoft results in a response with no less than three CNAME records (aliases) before finally arriving at the IP-adress shown.

That's it, just wanted to show you what's going on at my end. If anyone else is doing anything cool, please tell!

/Henrik.

[mackrackit]

It has to be something specific with where and how this site is hosted. Anyone know, and can explain, what's going on?
Shared/virtual machine/site.

[Bluecoal]

Henrik, have you been able to work on this project any more?

john

[HenrikOlsson]

Hi John,
Yes, a little bit. I've been working on external SPI flash routines for storing HTML-data and use "dynamic variables" in the HTML. This now works fairly well and I'm working on a way to use MPFS2 images. As it stands I can read the FAT table and access the files alright but I'm trying to figure out the correct/best way to have more than one file open at a time. I tend to be a bit (perhaps a bit too) conservative when it comes to using RAM and I try to make the most out of it. Having multiple file "open" seems to be quite a hog in that respect but perhaps I'll just have to accept that and just get it working.

(It would be nice to have proper functions and local variables in PBP)

Is there anything in particular you're trying/want to do or thinking about?

/Henrik.

[Bluecoal]

I have done a couple small projects for fun but always wanted to do that self made home security/automation system. I know, I know, there are plenty of companies selling them and a few home brew projects. The general operation of the pieces are the easy part. The missing part is web interface for setting up the system and monitoring remotely.

I purchased a Netburner kit but decided it would be nicer if it could be a complete PBP project and not a mix. Then my 2 and 4 year old boys decided to put the project on hold a year ago....

I have been following this thread closely. Your work is very impressive.

Thank John

[Bluecoal]

Henrik,

Would you mind sharing your code through your Oct 25th 2011 post (or after) that included SMTP and SNTP. I would like to start working on sending simple variables and I/O status between two modules and your code around that point really would be a great starting point for me.

John

[HenrikOlsson]

Hi John,
Sorry for the late reply. It's not really ready for "release", it needs some proper documentation and I've found making posts here is not the way. I'll need to put in a pdf and host it where I've got control over it - not enough time but I'm getting there.

Are you going to email the variables etc between the two units or why do you want/need SMTP? The core functionallity is already posted.

Finally, which version of PBP do you have? I'm asking because the code requires PBP3 due the conditional compilation statements thru out.

/Henrik.

[Bluecoal]

Henrik,

For now I really do not need the SMTP or SNTP part but was thinking that that version would have more information on making connections to other devices. I am hoping in the end to have it working but only emailing alarm updates. I have been playing with HTTP_part_ 1 and 2 waiting on the second shield to come in (came yesterday). I get to use my wire wrap tool again..

I did have to change (version 2)

If WizData = Socket_CLOSED THEN
GOSUB Init_Socket_0
ENDIF

to

If WizData = W5100_SOCK_CLOSED THEN
GOSUB Init_Socket_0
ENDIF

is that right..

To send my variables back and forth I guess I will parse the data like what you did for the get. First part of the data is the var name with some type of marker after with the var data right after. Does that make since or am I offtrack.

I am using PBP3. Thanks John

[HenrikOlsson]

Hi John,
Looking at the W5100_defs.pbp file both SOCKET_CLOSED and W5100_SOCK_CLOSED are defined as the value 0 so it should have worked in both cases. Are you saying it didn't? (I'm not at home so I looked at the file in DHCP_Demo package.)

If the operation prior to the IF statement you show is to read the sockets status register then W5100_SOCK_CLOSED 'looks' more right but again, there really should be no difference since they are both effectively replaced with IF WizData = 0 THEN...

Your approach sounds OK to me, build a packet at one end, load it to W5100 buffer and send it off, at the other end retrieve it from the buffer and parse it. If the package is always "the same" you could create aliases pointing into the buffer and just read out the data as normal variables.

There's no real changes to any of the core socket interface. SNTP and SMPT are both extra and separate "modules" and I don't think they'll do you any good at this stage but if still want it you can PM me your email adress and I'll send it to you.

/Henrik.

[Bluecoal]

I was using the original W5100_defs.pbp post #8 way back in May 2011, that may have been the difference.

Thanks John

[VaGyver]

Hello Henrik!

I would like to congratulate you for your very nice work here!
You made me, buy a shield, modify the code and get it worked!
Thank you that. I prefer that to a ready solution.

What i did is i translated your code into Proton+ Compiler and made:
a) a server, a simple web page with dynamic variables and
b) a client, which loads values into cosm database (ex pachube).

So, i would like to ask two questions:
a) Do you use multisocket operation? Or only one socket? If yes, what is the basic concept? Opening all 4 sockets, set to listen and wait by looping pediocally to sockets wait for a "GET"request?
b) There is a thought of opening a thread at Proton+ Compiler's forum with OF COURSE reference to you (your name and this thread's address) with some translation of your code into Proton. But i have to have your permission first.
Take a look here:
http://www.protonbasic.co.uk/showthread.php/66100-Help-choosing-serial-to-ethernet-module/page2?p=476797#post476797

[HenrikOlsson]

Hi,
Thank you, nice job on porting the code!

I'm interested in how you implemented the dynamic variables. It's been quite some time since I worked on this code but last time I did work on it (back in January) I went of on a tangent starting to implement a subset of the Microchip File System for storing the webpage content in external flash instead of in the PIC. Geez, can't even remember where I left off....

So, your two questions:
a) The code should work for multiple sockets. I've not used it for multiple connections to a web-server but I've had one socket to DHCP, one socket DNS, one socket SNTP and one socket SMTP - at the "same" time.
I don't think you should open all socket in listen mode at the same time though. If your device is acting as a server to serve up the web-page (or webpages) then you open a socket (say socket 0) and wait for a connection. When you get a connection on socket 0 you can open a second socket and wait for a connection on IT. That's probably how I'd try it as a start.

b) Sure, go ahead! (Thank you for asking, I appreciate that!) If you build upon the code and post in the Proton forum perhaps there are ideas and implementations I can "convert back" into the PBP code and so on. That was kind of the idea of this thread from the start - to build a base or "framework" (popular word) for the W5100 which others could build "applications" around (web-server, emailer, twitter client etc). Obviously I would've preferred if you did it in PBP but you can't have it all ;-)

If you do put something up please send me a link!

Damn, now you resparked my interest in this... I even got a couple of WIZ820IO with the (now no longer brand new) W5200 chip that I was going to implement in the same code. Too many projects...

/Henrik.

[VaGyver]

- About dynamic variaables:
What i did is not effective, but works. I converted the variables to strings and then i just added to the rest string, meaning the Str Buffer.
Fore example:
StrN Buffer = Str$(Dec variable_to_publish)
GoSub PutStringInTXBuffer

- About multisocket:
I am also searching this. The issue here is that i need to also pass dynamic variables as well. So, this means i must control all sockets simultanously (and my microchip chip also does other task too). Anyway, i will make some test on this and tell you the results.

- Publish your (modified for Proton+ compiler) code:
Of course i will do that (inform you). By the way, i intent, for start, only to publish a simple code that makes a web page with dynamic variables only (without DHCP). And i found out that these two compilers have very little differences.


- I would like to ask a little help for a snippet. Why you are doing this in DHCP code, because i couldn't convert it with easy way?

' Get second Bit in DHCP_LeaseTime And copy it To first Bit of Buffer And so On.

For DHCP_i = 0 To 30
Buffer.0[DHCP_i] = DHCP_LeaseTime.0[DHCP_i + 1]
Next

Thank you in advance.


Vangelis

[HenrikOlsson]

Hi,
Thanks! Actually I was more aiming at what format you used to "place" a variable in the actual HTML and how you detect that when the program parses the HTML page to replace the "dynamic variable identifier" with the actual variable value. Does that make sense?

Anyway, regarding the DHCP code:
The lease time received from the DHCP server is a 32bit variable stored in the DHCP_LeaseTime array (4 bytes). The IP-adress lease must to be renewed before the lease time expires. What I do is simply divide the actual lease time in half, when that time (half of the actual lease time) expires the code triggers a renew of the lease.

The code snippet you ask about performs that divide by 2 operation by shifting the value one step to the right. It uses the Buffer array as a temporary storage for moving Bit1 in LeaseTime to Bit0 in Buffer and so on. Finally it copies the shifted result back to DHCP_LeaseTime.

The reason for doing it this way is that I wanted to avoid using LONGs (32bit variables) in PBP because it adds quite a bit of overhead. Using a LONG for the lease time and doing LeaseTime=LeaseTime/2 or LeaseTime=LeaseTime >> 1 would result in the same thing.

Looking at the code now I could probably do something like this instead of using Buffer as a temporary storage

For DHCP_i = 0 To 30
DHCP_LeaseTime.0[DHCP_i] = DHCP_LeaseTime.0[DHCP_i + 1]
Next
DHCP_LeaseTime.0[31] = 0

Anyway, I hope that explains the what and why.

/Henrik.

[VaGyver]

I think that i understood it (the issue about dividing the time by 2).

What i will do is that i will simply rotate each byte by 2 to the right individually. So it will be divided by two. The DHCP_LeaseTime[3] by 2, then the DHCP_LeaseTime[2] by 2, ........ lastly the DHCP_LeaseTime[0].
I think it will have the same result.


For the dynamic variables:
I am not sure that i understood correctly what you are saying, but what i exactly do is placing the following html command. With this, the web page gets refreshed each 5 seconds.

Str Buffer="<meta http-equiv=\"refresh\" content=\"5\">",13,10,0 'refresh every 5 seconds
GoSub PutStringInTXBuffer


You could select other time too, if you wish.
So, the client (the person who sees your web page) will load again all the web page with the new variables (every 5 seconds). This is not quite effective since, sometimes, it makes a flashing on the page. But the data get refreshed correctly.
There is also an other (more effective) way to do that (in order to avoid the flashing), with xml requests, but i am yet enough familiar with it.

[HenrikOlsson]

Hi,
I'm afraid it's not quite that simple. Here's an example...

24h = 86400 seconds



86400 in binary:
00000000 00000001 01010001 10000000


If you actually rotate each byte (ie the LSB gets moved to MSB) individually the result becomes:


00000000 10000000 10101000 01000000
Which in decimal is: 8431680 seconds or 2352h


If you instead of rotate simply shift each byte individually (or divide each byte by 2 individually) the result becomes:


00000000 00000000 00101000 01000000
Which in decimal would be 10304 seconds or 2.9h


As you can see neither of the above results is the expected 43200 seconds or 12h.

You need to make sure that the LSB of each significant byte get moved into the MSB of the less significant byte so that result with the above numbers becomes:


00000000 00000000 10101000 11000000
Which in decimal is: 43200 seconds or 12h


/Henrik.

[VaGyver]

Well, i solved this problem (i created a large variable and shift it -all together- one position to the right).
I also verified it. It works.

Now i am doing some testing with the W5100 as client to be connected to servers.

I'll be in touch!

[VaGyver]

Hi,

Damn, now you resparked my interest in this... I even got a couple of WIZ820IO with the (now no longer brand new) W5200 chip that I was going to implement in the same code. Too many projects...

/Henrik.

For your info:
http://www.dacomwest.de/fileadmin/Dacom/Dokumente/Wiznet/TCP-IP-Chips/W5100/W5100_AN_Migration-to-W5200_20110923.pdf

[tayfun]

Hello,

Henrik bravo !

a wonderful project, a project I need my mail sender'm using 18F4620. I have W5100 or Amicus.


henrik can help me?

[HenrikOlsson]

Hi,
Sure, no problem, give me an hour and I'll write it up for you.
Where do you want me to send it when it's finished?

No, really. I'm trying to help as much as I can but it doesn't work like that. You give some and you get some.

With that said, getting SMTP to work is not a simple task and it will not work with SSL or any other encryption. If you haven't done anything with the W5100 previosuly, like set it up to respond to a ping, serve a simple web page etc (the stuff that already IS detailed in this thread) then forget SMTP. At least untill you HAVE done the previous stuff and know a Little bit of how it works. I'm not saying that you don't, I'm just saying that if you DO then post your effort and I or someone else might be able to help you out.

/Henrik.

[tayfun]

I agree absolutely with you the same ...

I did, the web page, UDP, TCP, with ENC28J60. but the W5100. liked very much.

I'm done with ping 5100.

I'm done with protons + 18F4620 software. I've revised your ping code for the proton.

I must take a little more now.

I bought a new module.

http://www.elektrovadi.com/w5100-ethernet-modul, PR-1389.html

then I'll continue with this module.

I wrote you about my progress from time to time. maybe I can come to the smtp portion.

You have to believe that I'm the one who worked very hard :)

ty for help.. If there is a flow diagram with smtp me if he could be good enough for me.

bye


7277

[HenrikOlsson]

Hi,
Ah, excellent work! Too bad you're using the wrong compiler. ;-)

Everything you need to know is in the RFCs covering the SMTP protocol but they can be a bit "much" from time to time. I also, strongly suggest that you do some "manual" emailing using telnet to your intended server, that way you get a feel for the protocol and what it "looks like". Anyway, here's a list from my notes

Setup a free socket with IP adress and port (usually 25 or 2525) of remote SMTP server.
Open the socket and wait for it to initialize.

Issue the sockets connect command, ARP request resolves the remote MAC adress and populate the W5100 registers.
Wait for remote server to return a response 220. If it doesn't the remote server probably isn't a SMTP server.

Send EHLO adress.domain
Wait for remote server to return a response 250

Send AUTH LOGIN
Wait for server remote server to return a response 334

Send the username of the email account, BASE64 encoded
Wait for server remote server to return a response 334

Send account password, BASE64 encoded
Wait for remote server to return a response 235 - when received the connection is authenticated.

Send the Mail From: command
Wait for remote server to return a response 250

Send the Rcpt To: command
Wait for remote server to return a response 250

Send the DATA command
Wait for remote server to return a response 354

Send a <CR><LF>, a valid email header, the content of the email, followed by <CR><LF>.<CR><LF> (note the dot)
Wait for remote server to return a response 250

Send the QUIT command
Wait for remote server to return a response 221 (bye bye).

Close the connection.


A complete email including the header might look like this, note the blank lines and the dot at the end.

Date: Tue, 25 Mar 2014 19:18:00 +0200
Message-Id: <1234>
From: Your W5100
To: Someone <[email protected]>
Subject: Test

Hello,
This is a test.
.

If you don't want to use authenticated login the use the HELO command instead of EHLO.

And again, note that the above will not work with email servers requiring SSL or other encryption.

/Henrik.

[tayfun]

hello henrik

thanks you for help and the pbp source code,

i was modifie your code for proton basic , and i was use 18F4620 , now wprking very stabile.

my web server
7319



- ping it is ok!
- web server it is ok
- send get ang command it is ok
- response and data it is ok

but i have any problem,

it is my html code,

how to use scrip in to do code?

index:


Buffer = "<html>" : GoSub PutStringInTXBuffer
Buffer = "<body>" : GoSub PutStringInTXBuffer
Buffer = "<title>my server</title>" : GoSub PutStringInTXBuffer

Buffer = "<h3>My Mini Web Server Alive</h3>" : GoSub PutStringInTXBuffer

Buffer = "<img src=http://www.enkatechnology.com/12.jpg >" : GoSub PutStringInTXBuffer


Buffer = "<p>Thenk you Henrik Olsson 2014</p>" : GoSub PutStringInTXBuffer

Buffer = "<form name=input action=/ method=get>" : GoSub PutStringInTXBuffer
Buffer = "<input type=submit align=center name=tst1 value=3 />" : GoSub PutStringInTXBuffer
Buffer = "</form>" : GoSub PutStringInTXBuffer


Buffer = "<form name=input action=/ method=get>" : GoSub PutStringInTXBuffer
Buffer = "<input type=submit align=center name=tst2 value=5 />" : GoSub PutStringInTXBuffer
Buffer = "</form>" : GoSub PutStringInTXBuffer


Buffer = "<a href=http://www.google.com>Go To Google</a>" : GoSub PutStringInTXBuffer

'Buffer = "<p> toplam request = " + Str$(Dec RequestCount) + "</p>" : GoSub PutStringInTXBuffer


Buffer ="<p>AN2 Data= </p>" : GoSub PutStringInTXBuffer
Buffer ="<script src=/s></script>" : GoSub PutStringInTXBuffer
Buffer ="<p><script>document.write(AN2)</script></p>" : GoSub PutStringInTXBuffer





Buffer = "</body></html>" : GoSub PutStringInTXBuffer



Return


i wat to use <script src=/s></script> and turn the back "<p><script>document.write(AN2)</script></p>" AN2 = 27 for exp. what you thing about?



i was receive "s" and portc.2 led is blinking but, i dont send the AN2 data to back.

my data receive code,

data_kontrol:


'----------------------------------
If Buffer[11] = "3" Then
PORTC.2 = 1'led aç
'END
EndIf
'----------------------------------
If Buffer[11] = "5" Then
PORTC.2 = 0'led kapat
'END
EndIf
'----------------------------------
If Buffer[5] = "s" Then
PORTC.2 =1
DelayMS 500
PORTC.2 =0
EndIf


Return

[HenrikOlsson]

Hi,
Nice work! I'm glad the code is getting some use!

I know very little about HTML and even less about JAVA Script (which I suspect is what Document.Write() is?

However, doing document.write(AN2) where I'm guessing AN2 is a variable in you PBP (Proton) program probably isn't working the way you expect. For it to work the way you've coded it the PutStringInTxBuffer routine would need to be able to decode and understand Java Script, find AN2 and put its content, as a string into the buffer - that's obviously NOT how it works or is intended to work.

Again, I'm very green on this HTML and JAVA stuff so I may be wrong but that's how I think it works.

In PBP, to load a "value" from a variable into an array as a string you could use ArrayWrite, I'm not sure how to do that in Proton but I'm sure there's some ToStr or whatever.

Does that make sense?

/Henrik.

[tayfun]

it is ok, is ok! work fine... :)

i will new stage , w5100 use with sd card, and screen any picture or complicate html pages. ty .. bye.




this is code,
'------------------------------------------------------------------------------------------------------------------------

send_back_result:


RequestCount = RequestCount + 1
GoSub GetFreeTXSize
TotalCount = 0

Buffer = "HTTP/1.1 200 OK "+13+10+0
GoSub PutStringInTXBuffer
Buffer = "Content-Type:text/html"+13+10+13+10+0
GoSub PutStringInTXBuffer

'---------server side------------------------------------client side----------------------------------------------------

If Buffer[5] = "s" Then '<--------------------- send me "s" block <script src=s></script>"


Buffer = "var AN2=" ' <------------send me AN2 variable <script>document.write(AN2)</script
GoSub PutStringInTXBuffer

Buffer = "1977" '-------------------> put string "1977" or any string variable
GoSub PutStringInTXBuffer

Buffer = ";"
GoSub PutStringInTXBuffer' -------------------> and put end of the string.

EndIf

'--------------------------------------- end of the cominicate

WizPtr = WizPtr + TotalCount ' load others... end.


WizAdress = W5100_S0_TX_WR0 : WizData = WizPtr.HighByte : GoSub Write_W5100
WizAdress = W5100_S0_TX_WR1 : WizData = WizPtr.LowByte : GoSub Write_W5100

' Issue a SEND command.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_SEND : GoSub Write_W5100

WaitForCompletion2:
GoSub Read_W5100
If WizData <> 0 Then GoTo WaitForCompletion2



' Then disconnect from the client.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_DISCON : GoSub Write_W5100

' And close the socket. We'll reopen the socket again in the main routine.
WizAdress = W5100_S0_CR : WizData = W5100_Sn_CLOSE : GoSub Write_W5100

GetRequest = 0




Return

[tayfun]

hi,

In this project, every socket been defined as 2K, but 2Kb web page is a very short size..

i want to change 8kb RX buffer and 8 Kb TX buffer, because total buffer size 16 Kb.

i was change ;
8Kb



WizAdress = W5100_TMSR : WizData = $03 : GoSub Write_W5100
WizAdress = W5100_RMSR : WizData = $03 : GoSub Write_W5100

'--------------------------------------------------------------------

W5100_S0_RX_START CON $6000 'Start adress of Socket 0 (not changeable)
W5100_S0_RX_END CON $7FF9
W5100_S0_RX_MASK CON $1FFF 'Mask value for 8k


'--------------------------------------------------------------------

W5100_S0_TX_START CON $4000 'Start adress of Socket 0 (not changeable)
W5100_S0_TX_END CON $5FFF 'End adress of Socket 0 (depends on the TMSR register)
W5100_S0_TX_MASK CON $1FFF 'Mask value for 8k



but not work :S have other configration? or I'm doing wrong in somewhere!

[HenrikOlsson]

Hi,
First of all I need to admit that I'm not sure if I've actually tested with anything except the default 2k per socket so it's quite possible something isn't correct there. The phrase but not work isn't much help though, do you have any other leads or hints, what does it do or what does it not do?

With that said, one thing I do see is

W5100_S0_RX_START CON $6000 'Start adress of Socket 0 (not changeable)
W5100_S0_RX_END CON $7FF9 ' <----- Shouldn't this be 7FFF?
W5100_S0_RX_MASK CON $1FFF 'Mask value for 8k
but I'm not sure if that IS the only problem or not.

/Henrik.