NetWorker - an advanced web server with a microcrontrollerFollow article
An Internet connection would be a valuable addition to many projects, but often designers are put off by the complexities involved. The ‘NetWorker’, which consists of a small printed circuit board, a free software library and a ready-to-use microcontroller-based web server, solves these problems and allows beginners to add Internet connectivity to their projects. More experienced users will benefit from features such as SPI communications, power over Ethernet (PoE) and more.
There are three key elements in connecting a device to an Ethernet network: the Ethernet connection hardware itself, the software library (called the ‘stack’) responsible for handling the various protocol layers, and finally the top-level firmware functions. For example, to allow access to a device from anywhere in the world, it is possible to use a small web server running in a microcontroller.
Compact ready-made modules are available on the market to cover the first two of these elements. They allow you to connect a microcontroller to a network without having to get involved in the gritty details of its protocols. However, programming against the libraries provided can be a daunting task for the beginner; and more experienced users often find it necessary to ‘work around’ the firmware provided with the module to enhance or modify it.
The solution provided here hides the complexity of the network protocols from the application programmer, and is simple to extend and very versatile. The circuit is based around a Microchip PIC18-family microcontroller which includes a built-in Ethernet transceiver. The author has added a few new functions to the free C software library available from the manufacturer that implements the TCP/IP stack. The final element of the project is a web server running on the microcontroller which can communicate with other hardware via the I/O pins of the device. Using this design it is possible to assemble a small system in just a few minutes, capable of being controlled at will over the Internet. Beyond that, the module can be used as a ‘network modem’ for another microcontroller. Extra advanced features such as PoE round out the project. Printed circuit boards and ready-programmed microcontrollers are available from Elektor, as well as the complete readymade module. And of course the accompanying software is also available as a free download from the Elektor website. Now let’s see what makes the module tick.
Figure 1 (below) shows the circuit of the module. At the heart of the unit is a Microchip PIC18F67J60. This device is one member of a range of microcontrollers that include a 10BASE-T Ethernet transceiver. Most of the circuitry required is provided in the device and only a few external components (including a connector!) are needed to connect to a network.
The Ethernet PICs require a 2.5 V core voltage supply, which can be derived from a higher voltage (3.3 V is recommended) using the voltage regulator included in the PIC. Pin ENVREG determines whether the voltage regulator is enabled. A 10 μF capacitor (C112) provides smoothing and capacitors C101 to C106 positioned around the microcontroller decouple the supplies. A 25 MHz crystal (Q101) provides an external clock for the microcontroller, and the Ethernet transceiver clock is also derived from this source.
The project includes a bootloader which allows new firmware to be uploaded over the Ethernet interface. Alternatively the device can be programmed via the solder pads MCLR, VCC, GND, PGD and PGC which form the standard Microchip in-circuit serial programming (ICSP) interface. This interface is needed for debugging software problems or if the module gets into a state where it no longer responds to being rebooted. A dedicated programmer (such as the ICD2) is required for debugging and programming over this interface. The reset circuit comprises R102, R103 and C107. R102 is a pull-up resistor, and R103 and C107 filter the reset signal to prevent spurious resets. C107 can interfere with the operation of the ICSP interface, and so should be removed if it is to be used. The EEPROM stores configuration data. It also includes a 48-bit ID code, called the MAC address, required for Ethernet communication. The proprietary Microchip UNI/O bus protocol allows the EEPROM to be accessed over a single wire.
The connection to the Ethernet network comprises four signals, TPOUT+, TPOUT–, TPIN+ and TPIN–. These signals are passed via the Ethernet transformer TR1, which galvanically isolates the main part of the circuit from the network. On the far side of the transformer is an RJ45 socket J1, into which a standard Ethernet cable can be plugged. A special feature of the transformer we have chosen is its compatibility with the IEEE 802.3af power over Ethernet (PoE) standard. Under this standard a 48 V supply is provided on the network and a powered device (PD) can draw current from this supply. There are two alternative arrangements possible:
1. using the spare conductor pairs (on J1 these are pins 4 and 5 and pins 7 and 8); or
2. phantom supply over the signal pairs (pins 1 and 2 and pins 3 and 6).
A PoE-standard device must support both possibilities. To take advantage of the phantom supply a special transformer is needed with centre taps to the windings on the network side.
Implementing both power options means that a total of four wires have to be brought to headers JP1 and JP2, which form the userside interface to the module. A suitable PoE regulator circuit can be connected to these headers to derive a supply for the module from the 48 V DC provided over the network (see below).
Resistors R107 to R110 provide impedance matching, and C108, C109 and FE101 help reduce the effect of interference spikes on the network connections. R104 is a bias resistor that provides the Ethernet transceiver circuit with a known reference current (hence its bizarre value). This in turn determines the signal amplitude on TPOUT+ and TPOUT–.
There are two LEDs mounted inside the Ethernet socket. LED1 lights when the link is up (in other words, when there is a connection between the module and at least one other network device), and LED2 indicates network activity. The LEDs are driven from the PIC via series current-limiting resistors R105 and R106. User-side interface
The user-side interface consists of two 10-way 0.1 inch pitch headers. As well as carrying the PoE connections and the module’s power supply, the headers are also directly connected to certain pins on the microcontroller. On the author’s printed circuit board design (see Figure 2 right) the two headers are mounted 0.1 inch apart, which allows the module to be mounted easily on ordinary prototyping board. Pins VCC and GND form the power supply to the module. The supply voltage should be between 3.1 V and 3.6 V. Pins VA1, VA2, VB1 and VB2 allow the module to be used in conjunction with a power over Ethernet regulator, sometimes called a ‘PD module’, such as the Silver Telecom Ag9000 series . Figure 3 (below) shows an example circuit using the Ag9033, which provides a regulated 3.3 V output from the PoE DC supply.
The active-low MCLR pin allows the module to be reset. It is also pulled low briefly when power is applied to the module. The other pins on the user-side interface are for use either to control external circuits or to sense their state, or for more general communications (by UART or SPI). Pins GPIO0 to GPIO2 can be configured as digital inputs or outputs; those configured as inputs can be used to generate an interrupt to the microcontroller. Used as an output, each pin can sink or source up to 25 mA. Pin GPIO3 is also configurable as a digital input or output, but can only sink or source up to 8 mA. A future software version may allow a PWM signal to be generated on this pin. GPIO4 and GPIO5 (which can be configured as inputs or outputs) can sink or source up to 2 mA. A future software version may allow these pins to be used as analogue inputs with 10 bit conversion resolution.
Pins TX and RX are connected to a UART that can support speeds of up to 115200 baud. Alternatively, these pins can be configured as ordinary digital inputs or outputs; as outputs, they can each sink or source up to 25 mA.
An SPI interface is available on the pins with labels starting ‘SPI’. The interface can operate in master mode or in slave mode, and the function of the pins depends on the mode selected.
1. In master mode SPI_INT is not used, SPI_CLK is the clock output, SPI_MOSI the data output, SPI_MISO the data input. The active-low SPI_CS output is used to enable the connected slave device.
2. In slave mode SPI_INT is used to indicate when an event has occurred in the module that needs the attention of the SPI bus master. SPI_CLK is the clock input, SPI_MOSI the data output, and SPI_MISO the data input. The active-low SPI_CS input is used to enable reception of a bus message by the module.
Again, these pins can alternatively be configured as digital inputs or outputs. As an output, SPI_CS can sink or source up to 2 mA, while the other pins can each sink or source up to 25 mA. A future software version may allow SPI_ MISO and SPI_CLK to be used as an I2C interface. In this case SPI_MISO would be the SDA (data) signal and SPI_MOSI the SCL (clock) signal.
A ‘stack’ is a collection of software implementations of protocols and drivers, usually arranged in a hierarchy of layers. At the bottom end of the stack are the hardware drivers which are responsible for getting data bits transferred onto the network wires. At the top end of the stack is a simple interface for data exchange.
In theory, the layered protocol model makes it relatively easy to make modifications to one layer (such as the hardware driver) without affecting the others. In practice however, when implemented on a microcontroller, the hardware and the stack are very closely tied together, and lots of tricks need to be used to keep memory usage low. Nevertheless, TCP/IP stack implementations are available for a wide range of microcontroller families, often for free and direct from the manufacturer.
The author’s first tentative steps towards building a TCP/IP stack for a PIC microcontroller were in assembler. Over time, however, as the code grew, maintenance became harder and harder, and an alternative solution was needed.
Microchip offers a TCP/IP stack for its PIC18, PIC24, dsPIC and PIC32 microcontroller families written in ANSI C. It is free and can be extended and modified as long as it is only used in Microchip devices. It contains a rich collection of hardware drivers, lowlevel protocol implementations (ARP, IP, TCP and so on) and a couple of important application-level protocol implementations such as DHCP and HTTP. The simple MPFS file system is also included to allow storage of web page source data. Example applications, a bootloader, tools and comprehensive documentation of all stack functions complete the package.
The author evaluated and extended this protocol stack for the project in the light of his experience designing the ‘eWicht’ handheld model railway controller . For example, the DHCP module was extended to include an AutoIP mechanism (see glossary). An mDNS server was incorporated so that the module could autonomously make its presence known on a network (see below).
A feature was added to check values entered on web forms for type and range errors, and the MPFS tool provided by the manufacturer was extended to use a Huffman code to obtain a saving in memory footprint of up to 50 %.
The module has an integrated bootloader that allows a new version of its firmware to be uploaded over the Ethernet connection for installation at any time: suitable hex files are available on the project’s web pages 
The bootloader used is the one included in the TCP/IP software package provided by Microchip for its PIC18F97J60-series devices. This includes a footprint-optimised implementation of the TFTP server and ARP protocols (see glossary), as well as a hex file parser and flash programming routines: everything needed to program new code into the chip.
There are two ways to access the bootloader. If the microcontroller has not been programmed with an application hex file, or if there has been an error programming the hex file, the bootloader is automatically invoked. This will always be the case with a ‘virgin’ microcontroller. Alternatively, if an application hex file has been successfully programmed, the bootloader will be active for just the first two seconds or so after power is applied. The bootloader’s IP address is fixed at 192.168.97.60, and its MAC address is 00-04-A3-00-00- 00. A connected PC must have an address on the same network (such as 192.168.97.61) or it will not be possible to upload a new file. The microcontroller is then programmed by simply sending a hex file using TFTP.
If you wish to upload a new hex file while the device is in operation, you simply need to start a TFTP upload to its current IP address. The bootloader will be invoked automatically and the programming operation will start. (For this to work the running application must make use of the ‘Reboot’ module, which is part of the Microchip TCP/IP stack.) TFTP clients are available for almost all operating systems. A typical invocation of the client on a Windows system looks like this:
tftp 192.168.97.60 put “file.hex”
After a short wait a message resembling the following should appear:
Transfer successful: 203557 bytes in 8 seconds, 25444 bytes/s
The device will then automatically reboot into the new version of the firmware.
The author developed the application software for the module using the MPLAB IDE. It is written entirely in ANSI C and can be compiled using the free version of the Microchip C18 compiler.
The application consists of three modules, called Main, Web and Appl. The bootloader jumps directly to Main, where the TCP/IP stack, the EEPROM driver and the Appl module are initialised. Then Appl and the TCP/IP stack receive processing time alternately in an infinite loop.
The Web module implements the web interface (web server and web pages). When a client, such as a browser running on a PC, requests a page the relevant information is prepared. If a completed web form is received from the client, its contents are checked and passed on to the relevant module. Since most data are stored in EEPROM, they are still available after a reset or loss of power.
The Appl module forms the main part of the application. Within it there are three further sub-modules: Bootup, GPIO and Serial. Each of these is initialised by Appl and then called in round-robin fashion. The Bootup module makes sure that the GPIO and Serial modules are accessible if a network connection exists: that is, when an IP address is available and unambiguously registered on the network.
The GPIO module is responsible for reading and writing the I/O pins of the device. The module communicates with the web interface to allow the levels on the pins to be read and written via a web page. However, the possibilities go well beyond simple manual operation. The unit also makes a kind of web service available (on a port configura-ble via the web interface) called ‘GPIO Service’. Another device on the network can use this service to control and read the GPIO pins using a simple protocol (see below).
As already alluded to, the unit includes serial interfaces. Data can be transferred from these ports to the network using a separate TCP connection, or in the opposite direction: more details later on. Initialisation and data handling are done in the Serial module. A further service (‘Serial Server’) is responsible for listening on a (configurable) port for requests and data.
Using the module
For first experiments with the module it should be connected to a PC using a crossover patch cable. When power is applied the application program starts up, with the link LED lighting and the network activity LED blinking irregularly. The module will first try to obtain an IP address. By default this happens over DHCP, but if no DHCP server is found, AutoIP is used (see glossary). If a network error should occur during this phase, the link LED will start to blink rapidly.
How do we know what IP address the module has acquired? A simple way to find out is to use Bonjour, a program developed by Apple which is now also available for Windows (most simply installed as a browser plug-in, for example for Firefox ).
An mDNS service runs on the module, which advertises the device’s presence to the rest of the network. Bonjour scans the network for devices running this service and displays the results in the browser (see Figure 4 above, lefthand part of the screen image). One click later and the web server responds by sending its home page. This can all be done without the user having to know the IP address of the device. The web server can serve up a variety of pages. The main page gives information on the device itself (version, IP information, network name and so on). There are also buttons to reset the module, stop it, and reset its settings (Figure 5 below).
Under the ‘General’ menu it is possible (among other things) to change the operation mode of the module (Figure 6 below).
In GPIO mode the pins listed above can be configured as inputs or outputs via a web page, with the default setting being ‘input’. Unconnected pins should be prevented from floating either using pull-up or pull-down resistors or by configuring them as outputs. When the GPIO web page is loaded the radio buttons indicate the state read from the GPIO pins (see Figure 7 below), and then when the form is submitted the values are sent to the GPIO module. If required, the initial values of pins configured as outputs can be stored in the EEPROM.
The GPIO server module is active independent of the operation mode chosen, and other devices on the network can connect to it at any time using its simple textbased protocol. A tool such as Hyperterminal can be used to communicate with the server (using a Winsock connection specifying the network name or IP address as the host, and port number as configured). The GPIO web page gives each pin an ID code: this code is used in commands to the server in the form ‘ID=value’, terminated by a character code in the range from 0x00 to 0x20. The ‘value’ can be ‘0’, ‘1’ or ‘?’, the question mark representing a command to the server to request the status of a port pin. The server replies to requests and commands with a string of the form ‘ID=0’ or ‘ID=1’. The special wild-card ID ‘x’ indicates that all GPIO pins are to be read or written by the request or command. For example, a request of the form ‘x=?’ will receive a reply of the form ‘x=1234’, where ‘1234’ is a hexadecimal value in little-endian format, that is, to be interpreted as bits in ID code order from right to left. Write commands work in an analogous way.
In UART mode the TX and RX signals form an asynchronous serial interface and are thus not under the control of the GPIO module; nevertheless, the GPIO module can still be used to read the state of these lines. The baud rate can be adjusted via the web page that is reached by clicking on the ‘Serial’ menu item (Figure 8 below). The asynchronous interface can, with suitable level conversion, be used to connect to devices with an RS232 port.
The serial server is active in this mode. Characters can be sent to the device over the configured port, and these are echoed over the serial port in 8N1 format. Likewise, characters received over the serial port can be sent to another device on the network.
SPI is an interface that can be used to connect to a wide range of storage, display and other devices. When a suitable network connection is established with the serial server data can be sent using TCP/ IP to the unit. In this mode it acts as an SPI bus master, which means that it initiates all SPI communications. When data arrive, it asserts SPI_CS (in other words, takes the signal low). Each data byte is then sent out bitwise SPI_MOSI, accompanied by a clock on SPI_CLK. Data are read back from a connected slave over SPI_MISO using the same clock for synchronisation. Bytes read in are then sent out over the TCP/IP connection. A zero byte in the data stream is used as a terminator. When the terminator byte is received over TCP/IP, the unit deactivates SPI_CS (by taking it high); the terminator byte itself is not sent out over the SPI bus. A zero byte that is to be transmitted to the SPI slave must be escaped by prefixing it with a backslash character (‘\’); the backslash character itself is represented by two backslashes. No special escaping is applied to zero bytes received from the SPI slave. The web page for the serial interface allows a choice of clock frequencies (650 Kbit/s, 2.5 Mbit/s and 10 Mbit/s) and of SPI clock polarity (Figure 9 below).
When operated in slave mode, the unit can work as a network ‘modem’ for another microcontroller. This microcontroller can, with the help of the unit, create and destroy TCP/IP connections (as client or server) as well as transmit and receive data. Further details on the additional protocols required for operation in this mode can be found in a supplementary document downloadable from the Elektor website.
Ideas for future expansion
- Extensions to existing functionality
- Changes to format and content of embedded web pages
- Special functions for use in control applications
Resistors (SMD 0603)
R101 = 1MΩ
R102 = 4.7kΩ
R103 = 1kΩ
R104 = 2.26kΩ
R105,R106 = 560Ω
R107–R110 = 49.9Ω (1%)
Capacitors (SMD 0603):
Inductors (SMD 0603):
FE101 = 1k @ 100 MHz
IC1= PIC18F67J60 (TQFP64), programmed, Elektor # 100552-41 
IC2= 11AA02E48 (SOT23)
TR1= 10/100Base-TX transformer, Halo N5 (SMD), PoE to IEEE 802.3af
J1= 8+2+2 pin RJ45 socket, w. 2 integrated LEDs
JP1,JP2 = 10-pin SIL pinheader (lead pitch 0.1 in.)
Q101 = 25 MHz SMD quartz crystal, HC49UP case
PCB, Elektor # 100552-11 
NetWorker module, ready built and tested, Elektor # 100552-91 
Article by Sven Schlendet
Taken from eTech issue 6