Web Control Panel – Building the Web Server – Raspberry Pi Pico, ESP32, Arduino

MicroPython Web Server on a Raspberry Pi Pico

Introduction

With a number of microcontrollers now including a Wi-Fi chip we can open up a whole range of possibilities for our projects. In this tutorial I’ll show you how to build a full web server in MicroPython on a Raspberry Pi Pico, but the code should work on any other MicroPython capable device which has a WiFi connection. This lets us build a web based control panel where we can display data readings and control our project with buttons and other devices from any browser on our network.

So lets hook up our Pi Pico W and get started.

Function of a Web Server

When you use your web browser to visit a website you tell it which webpage address or URL you want to view. Your web browser then sends an HTTP request to the server that handles that website. These requests can be simple GET requests for a static webpage or they can contain more complex commands as part of an API (Application Programming Interface). These requests might also be in the form of an HTTP POST that has a range of data attached to it. Our web server needs to listen for and decode this request, take the appropriate action and generate an HTTP response message that lets the client know if the request was successful and contains any code or data that the client requested.

So in essence we need to build three sections of code.

• An HTTP request parser to receive and decode the client’s request
• A request handler to take any actions required
• An HTTP response builder to send data back to the client

So let’s take a look at each of these sections and turn.

The HTTP Request

When your browser connects to a website it sends a formatted block of data as an HTTP request. We can easily see the data being sent by getting our MicroPython code to print out the request text after accepting a connection.

If we strip our Python code from the previous tutorial down to a basic HTTP request receiver we can capture the data and start to analyse its format.

To help in this process I’m using a package called PostMan. If you visit their website at postman.com you can sign up for a free account and then in your My Workspace area create collections of HTTP requests.

Here I’ve set up some GET and POST requests to my Raspberry Pi Pico. If I send one of these requests I can see the request data in my REPL console and the Pico’s response in the PostMan application.

So let’s take a look at a few of these requests to see what format the data comes across as.

This first block is the raw data from a standard GET HTTP request.

GET/page.html?action=setled&colour=blue HTTP/1.1\r\n
User-Agent: PostmanRuntime/7.30.0\r\nAccept: */*\r\n
Cache-Control: no-cache\r\n
Postman-Token: ecce0bea-5bf6-4cbc-9218-7c2cd25b9da3\r\n
Host:192.168.1.246\r\nAccept-Encoding: gzip, deflate, br\r\n
Connection: keep-alive\r\n\r\n

This is received as a continuous string of bytes. If we use MicroPython’s string.decode method we can translate this into a standard Python string. The \r\n characters are CRLF codes, or Carriage Return Line Feed codes, that are effectively new line characters. So this gives us a decoded request like this.

GET /page.html?action=setled&colour=blue HTTP/1.1
User-Agent: PostmanRuntime/7.30.0
Accept: */*
Cache-Control: no-cache
Postman-Token: ecce0bea-5bf6-4cbc-9218-7c2cd25b9da3
Host: 192.168.1.246
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

It’s important to note that there is actually a blank line at the end of this request. If you look at the raw byte stream you’ll see that it ends in a double CRLF character. You’ll see the significance of this as we start to decode the request data.

If we start to analyse this request we can really break it down into three sections.

The first line is the actual request. This itself is split into three parts separated by spaces. The first word is the HTTP method or verb. So this request is a GET request. As we’ll see in a second there are a few different methods the most common of which is the POST request. The second word in this line is the full URL of the resource being requested. Again this can also be broken down into the page URL, /page.html, and the query string, action=setled&colour=blue, which is separated from the URL by the ?. The final word is the protocol being used in this request.

The remaining lines form the header section of this request. As you can see there is one header per line which starts with the header name followed by a colon followed by the header value. These headers give us information about the request including the format if required.

As I mentioned above there is an important blank line at the end of this header section. This blank line indicates the end of the header and the start of the third section, the request body. In this GET request there was no need to send any body data.

For comparison we can look at a more complex POST request.

POST /api HTTP/1.1
User-Agent: PostmanRuntime/7.30.0
Accept: */*
Cache-Control: no-cache
Postman-Token: 42f8113f-acec-46db-abb1-a2840e5011d3
Host: 192.168.1.246
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------------------------833994107218074559113347
Content-Length: 429

----------------------------833994107218074559113347
Content-Disposition: form-data; name="action"



setLedColour
----------------------------833994107218074559113347
Content-Disposition: form-data; name="colour"

green
----------------------------833994107218074559113347
Content-Disposition: form-data; name="text"

this is line1
this is line 2
this is line 3
----------------------------833994107218074559113347--

Who you can see the request line showing that we are making a POST request to the /api endpoint followed by a header section followed by a blank line followed by a number of lines of data that make up the body of the request.

If we look at the header section you’ll see the Content-Type header. This tells us what format the body has been encoded as. In this instance it’s specifying multipart/form-data which means that we are being sent form data, in effect named variables, but in a multipart format. We then have a boundary definition which tells us how the various parts of data have been broken up.

If we now look at the body you can see that this boundary string is being used to divide the body data into parts, each of which has its own many header section which specifies the name of the variable, and a body section that specifies the value of the variable.

So our Request Parser code needs to be able to pull all of this information out of the request data and present it to us in a usable format.

As a note the format of an HTTP request is part of the official specification for HTTP. Please have a look at https://www.rfc-editor.org/rfc/rfc9110.html for full details.

So let’s have a look at my RequestParser class.

Parsing the HTTP Request

To make this code easily reusable I’ve encapsulated it in a class definition. Classes are the basis of object orientated programming and allow us to box up various data variables and the code that manages them. We can then present an easy to use object that hides all the complexities of, in this case parsing the HTTP request, and simply lets us access the information we need.

Running through the RequestParser class we first come to the constructor method. This accepts the raw request data and then prepares the class attributes to hold decoded data. We first make sure that our request data has been decoded to a standard Python string and then create our class variables to hold each part of our request. I’m creating dictionary objects to hold request headers, the decoded query parameters and any post data whether it comes across as form data or URL encoded data. I’m also making a note of the boundary marker if there is one and then finally a Python list to hold each of the lines of the request content or body.

The constructor then calls the actual parsing code which decodes the entire message. This allows our main code to simply instantiate the object, give it the raw request data and it will then respond back with a fully parsed request.

If we look through the parsing code we first split the raw data up into individual lines. The HTTP specification requires \r\n line endings but I have seen some that just use the simple \n character. So this first block of code simply tries to work out which line endings are being used so it can divide up the data correctly.

At this point we now have each line of our request as an element in our list. We can then parse the first line which we know contains the request details. As we’ve already seen this first line can first be broken using the spaces between words to give us the method, full URL and protocol. The URL can then be broken using the ? into the URL and query string. And we can then decode the query string into a set of key/value pairs which is carried out in the decode_query_string method.

Decoding the query string is a matter of breaking it apart on the ampersand (&) characters. This gives you variable=value statements. We can then split each of these at the = character to get the variable name and value. We then put these into a Python dictionary.

Going back to the main parsing method we now need to deal with the header section. In each of these blocks of code we are always checking to see if we’ve reached the end of the request. Not all requests contain all the sections and of course we need to be able to cope with malformed data.

So first we are checking each line after the first line to see if it contains a header. We basically check to see if the line contains any characters. If it does we treat it as a header line and grab the header name and value and place it in our dictionary. We continue this until we get to a blank line which denotes the end of the header section.

At this point the rest of the request is the actual content so we save a copy of the remaining lines in our class content attribute.

We now need to decide what we need to do with this content. This relies on the value of the Content-Type header, so we use a class method to extract that value if it’s there. For each type of Content-Type we call a different parsing method.

Each of these is aware of how the data is formatted so can simply work through and pull the individual data points from the request content. This data is then put into the class post_data variable.

For url-encoded POSTs the body of the request holds the url encoded data string. We use the same decode_query_string method   as the get request.

Form-data encoded POSTs are a bit more complicated but as described earlier we need to break the body into sections using the boundary markers, then pull the variable name from the Content-Disposition header and then grab the data value. Hopefully I’ve added enough comments to explain what’s happening in a bit more detail.

At this point we’ve now got a fully decoded request. The rest of the class methods are helper functions to make life easier in our main code. Again we’ll see what they do as we use them.

All of the code I’m using in this tutorial is available on a GitHub repository that I’ll link to in the description. And don’t forget that each video I make also has a project page on my website. If you like what I’m doing then do please click the like button and subscribe to my channel. The more the channel grows the more time I can put into making these projects.

Testing the RequestParser

Let’s jump onto the Pico and check that we are getting the right data from the RequestParser.

So this code is in the request_parser_test.py file. All it does is receive the request, hand it to the parser class and then print out the received data and request info. If we then throw a few requests at it from PostMan we can see the class decoding the request headers and data and allowing us to concentrate on handling the request without worrying about how and in what format the request was sent.

Now that our code is able to deal with requests we need to consider what we need to say back to the client.

The HTTP Response

As we saw in the first program we needed to send a coded message back to PostMan to stop it complaining. This is called the HTTP Response and is an important part of the whole HTTP protocol communication.

The response message allows our web server to tell the client if the request was successful and to return any data required to complete the action. This data could be the HTML code for the page browser wants to render, or it could be the data readings from the circuitry the Pico is monitoring. It could also be an error message if the page being asked for doesn’t exist. But the important point is that this response message needs to be in the correct format.

So let’s have a look at what this format looks like.

If we send a request to the time service I used in the previous tutorial we’ll get back an HTTP Response.

Request URL is https://timeapi.io/api/Time/current/zone?timeZone=Europe/London

Response message:

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Fri, 03 Feb 2023 15:42:46 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Content-Encoding: br
Vary: Accept-Encoding

{"year":2023,"month":2,"day":3,"hour":15,"minute":42,"seconds":46,"milliSeconds":73,
"dateTime":"2023-02-03T15:42:46.073709","date":"02/03/2023",
"time":"15:42","timeZone":"Europe/London","dayOfWeek":"Friday","dstActive":false}

Again it is a plain text string sent as bytes of data.

The first line sent back contains the protocol used and the response code and description. Here we are using HTTP/1.1 protocol with a 200 response code that means that the request was handled correctly (OK).

There are a whole range of response codes that you can look up in the HTTP specification.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

They are three digit numbers grouped by the first digit.

1xx codes are for information only.

2xx codes are successful responses.

3xx codes are redirection values meaning the resource has moved elsewhere.

4xx codes are client errors such as missing page etc.

5xx are errors on the server.

For our purposes we only need to work with 200 success, and 404 not found when someone asks for a page we haven’t coded.

After this response code line we then have a number of headers. These work in the same way as the response headers and again give information about the response and the return data formats. The important one for us in this response is the Content-Type header that tells us that the body of the response contains data in JSON format.

The header section ends with a blank line, followed by the response body which will contain the data being returned.

So we need to build this response for each request we receive so that we can complete the HTTP communication.

Building an HTTP Response

If we look through the ResponseBuilder.py file you’ll see that I’ve built all the logic into a class that takes the data that needs to be sent back and formats it correctly with the required headers, etc.

Going through the code…

We first set up a few constants that will be used to reply back to the client. The server just identifies this as our Raspberry Pi Pico.

We then have our class constructor which basically creates the class attributes and initialises them to sensible values.

We then have a number of setter methods. Our main code will be using the RequestParser to work out what the client is asking for. It will then need to decide how the request went and what data needs to be returned. The main code will then use these setters to store the relevant data in the class.

Some setters let you add specific values into the header of the response. But the serve_static_file and set_body_from_dict methods help with attaching the relevant return data.

To get web pages to show we usually send back mostly static HTML, JavaScript and CSS files. The serve_static_file method handles that for us. The RequestParser will return a requested url. If we decide this points to one of our web page files we can simply pass that to this method and it will attach the file data and set up the correct headers.

If we look through the code we can see that it checks to make sure we’ve got a correctly formatted url. It then checks to see if that file actually exists on the Pico. If it does exist we then work out what type of file is being requested so that we can set the correct Content-Type header. And finally we attach the file contents to the response body section.

In the set_body_from_dict method we simply send in a dictionary containing the data we want to return. This then gets turned into JSON code that fully describes the data and objects in our Python dictionary. This code is then attached as the response body and the Content-Type set to application/json.

Finally in our class we have the actual build method. This takes all the information that’s been built into the class and creates the actual data string that needs to be sent back to the client.

Testing the ResponseBuilder

Again we can use PostMan to test our response builder. Here I’m running the response_builder_test.py code. I’ve set up a couple of end points in the request handling section to send back certain responses. These simply examine the requested URL and setup the ResponseBuilder object accordingly.

If we then use PostMan to send requests to these endpoints we should get back the relevant responses.

This effectively gives either end of the web server. We just need to work out how to handle the requests.

The Request Handling Section

When the client sends an HTTP request we’ll use our RequestParser class to decode the message. This will give us a requested URL and some data. To handle the request we need to filter out any special endpoints such as an API call, etc. and then assume the remaining urls are file requests.

If we look at the full web API code that we’ll develop in the next tutorial you can see that this step is fairly straightforward.

Once we’ve parsed the request we test the url value to see if it matches our /api endpoint address. If it does we then further break down the request based on the data being sent to carry out the actions requested.

In my API the client must supply an action variable to tell the Pico what to do. Once we decode this we take the appropriate action and then set up the response status and attach the data.

If we don’t find a special case we end up in the final else clause where we try to return a file that matches the requested url. These files are simply HTML, JavaScript and CSS files that I’ve created to build the web pages and uploaded to the Pico’s internal storage in exactly the same way you would do with a Python code file. These files then become accessible from the Python code. You can even use folders and sub folders to separate our web files if required.

By encapsulating the file handling into the ResponseBuilder class our main code is kept very clean and readable.

The Next Step

So that gives us a fully working web server. All we need now are the web files to build the control panel and the API handling routines to provide control and monitoring of our project.

But our current web server does have one big flaw. It uses blocking code.

The line where we accept a connection from the client is the blocking statement. Once we execute this line of code the Pico will sit waiting for the next request. It is unable to move past this line of code. If we are running some sort of update loop for our project this means that the code grinds to a halt every time we check for a client request.

There are two ways around this. Use the second core if you have one, or use asynchronous coding. Both of these will be covered in the next tutorial where we’ll take the final step and build a fully API driven control panel that run the web server in the background so that your main code can run as normal without interruption.