Web Server for ESP32 and ESP8266 Software Update Over the Air

I developed a web server for uploading and handling binary images and serving them to ESP32 and ESP8266 boards. Also, I implemented over the air updating for my ESP32 and ESP8266 boards to enable upgrading of boards I don’t have physical access to. In this post I’ll document how I implemented the web server – and learnt Python Flask along the way. This project is also the first that I published to GitHub and Docker Hub.

Initial solution

Skip this part if you’re in a hurry, this is just me rambling. I started out searching the web for a solution to upgrade my boards over the air and stumbled upon this article. A crude solution that focuses mainly on how to do the implementation on the ESP8266. I followed the guide and fired up a NGINX Docker container on my Raspberry Pi for hosting of files. The solution is based on the binaries for the boards being named with the MAC address of the board they match to. A version file, also named with the MAC address, accompanies the binary to allow the board to determine of a newer version of software is available on the web server.

This was a good beginning but I wanted something more advanced. Based on this initial solution these were my requirements:

No sidecar file. I didn’t want a version file for each binary and I definitely didn’t want to manually update it and making sure to do it in the right order compared to when the binary is uploaded.
Same binary for multiple platforms. I would like the option to upgrade multiple platforms with the same binary but still maintain some level of MAC address filtering.
Better interface for uploading. It’s not hard, but it’s a hassle to manually scp new binaries to the Raspberry Pi and ensure correct naming and file placement. I’d like a more user friendly interface to handle this.
Auto-detect image identity. The binary should be analysed upon upload to determine platform and version number to avoid manually having to enter those.
Use of semantic versioning. I find semantic versioning more informative than a simple integer which might suggest a date.
Server-side upgrade decision. The server should decide if an upgrade is needed. No need for the added communication to let the boards make this decision.

ESP Update Server

The ESP8266 API documentation provides a lot of the code needed to implement an update server in PHP. However, I’m very fond of Python and wanted to take this opportunity to learn Flask – a Python micro framework for web development. I found a short tutorial and followed that taking what I needed and adding all the behaviour mentioned above.

The code is available on GitHub so I won’t dive into all the details about the implementation but just describe some selected features and ideas.

Storing Data

All the data the server need to keep track of about the binaries is stored in a YAML file. This way the data is easily understandable if one needs to read it directly. With the use of pyYAML library YAML is extremely simple to work with in Python.

Auto-detect Platform and Version

All my ESP projects have names and by including this name in the code I can search for it in the binary when I upload it to the web server. For this to work I have to create the platform name on the server before uploading binaries – otherwise the server won’t know what to look for. Searching can be done with Regular Expression which is also used to find the semantic version number. The version number must start with a v to avoid false positive matching other version numbers.

```
data = file.read()
```
```
for __dev in platforms.keys():
```

    if re.search(__dev.encode('UTF-8'), data, re.IGNORECASE):

      m = re.search(b'v\d+\.\d+\.\d+', data)

```
      if m:
```

          __ver = m.group()[1:].decode('utf-8')

Timely Update of YAML

During the implementation I was careful to make sure the platform file was updated in such a way that it would never point to a file or version that was not yet available. And making sure the platform file is updated to point to a new binary before the old one is deleted.

No Security

The web server doesn’t implement any security. No https nor authentication or login mechanisms. It is intended to run on an internal network with no access from the internet. Feel free to add to the code if you need it.

ESP Update Server Protocol

Devices requesting download of a binary file for upgrade must access path update and include device name and current version number in a query like below – substitute the IP address with your own. The ESP32 and ESP8266 will include the MAC Address as a HTTP header in the request and the server will extract it from there.

http://192.168.0.10:5000/update?ver=v1.0.2&amp;dev=chase

The server will respond with HTTP Error Code:
– 200 A new binary is available.
– 304 No update is needed.
– 400 or 500 An error occurred, e.g. device not whitelisted or platform name is unknown.

ESP Update Implementation

For the AirPatrol project I named the update functionality SUOTA – Software Update Over The Air – as an homage to a previous job I had.

For an introduction to the publisher I use see Logging for Sensor Networks.

ESP32 Code

```
#include <HTTPClient.h>
```
```
#include <HTTPUpdate.h>
```
```
#include <WiFi.h>
```
```
 
```
```
#include "publisher.h"
```
```
#include "suota.h"
```
```
 
```
```
#define VERSION "v1.0.1"
```
```
#define HOST "Marshall"
```
```
 
```

const char* urlBase = "http://192.168.0.10:5000/update";

```
 
```

/***************************************************/

```
void suota_checkForUpdates(void)
```
```
{
```
```
  String checkUrl = String( urlBase);
```

  checkUrl.concat( "?ver=" + String(VERSION) );

  checkUrl.concat( "&dev=" + String(HOST) );

```
 
```

  publisher_sendSyslog(SYSLOG_INFO, "SUOTA: Checking for updates at URL: " + String( checkUrl ) );

```
 
```
```
  WiFiClient client;
```

  t_httpUpdate_return ret = httpUpdate.update( client, checkUrl );

```
 
```
```
  switch (ret) {
```
```
    default:
```
```
    case HTTP_UPDATE_FAILED:
```

      publisher_sendSyslog(SYSLOG_ERROR, "SUOTA: HTTP_UPDATE_FAILD Error (" + String(httpUpdate.getLastError()) + "): " + httpUpdate.getLastErrorString().c_str());

```
      break;
```
```
    case HTTP_UPDATE_NO_UPDATES:
```

      publisher_sendSyslog(SYSLOG_INFO, "SUOTA: HTTP_UPDATE_NO_UPDATES");

```
      break;
```
```
    case HTTP_UPDATE_OK:
```

      publisher_sendSyslog(SYSLOG_INFO, "SUOTA status: HTTP_UPDATE_OK");

```
      break;
```
```
    }
```
```
}
```

ESP8266 Code

```
#include <ESP8266HTTPClient.h>
```
```
#include <ESP8266httpUpdate.h>
```
```
#include <ESP8266WiFi.h>
```
```
 
```
```
#define VERSION "v1.0.1"
```
```
#define HOST "Chase"
```
```
 
```
```
#include "publisher.h"
```
```
#include "suota.h"
```
```
 
```

const char* urlBase = "http://192.168.0.10:5000/update";

```
 
```

/***************************************************/

```
void suota_checkForUpdates(void)
```
```
{
```
```
  String checkUrl = String( urlBase);
```

  checkUrl.concat( "?ver=" + String(VERSION) );

  checkUrl.concat( "&dev=" + String(HOST) );

```
 
```

  publisher_sendSyslog(SYSLOG_INFO, "SUOTA: Checking for updates at URL: " + String( checkUrl ) );

```
 
```

  t_httpUpdate_return ret = ESPhttpUpdate.update( checkUrl );

```
 
```
```
  switch (ret) {
```
```
    default:
```
```
    case HTTP_UPDATE_FAILED:
```

      publisher_sendSyslog(SYSLOG_ERROR, "SUOTA: HTTP_UPDATE_FAILD Error (" + String(ESPhttpUpdate.getLastError()) + "): " + ESPhttpUpdate.getLastErrorString().c_str());

```
      break;
```
```
    case HTTP_UPDATE_NO_UPDATES:
```

      publisher_sendSyslog(SYSLOG_INFO, "SUOTA: HTTP_UPDATE_NO_UPDATES");

```
      break;
```
```
    case HTTP_UPDATE_OK:
```

      publisher_sendSyslog(SYSLOG_INFO, "SUOTA status: HTTP_UPDATE_OK");

```
      break;
```
```
    }
```
```
}
```

Docker Image

I want to have the server run in a Docker container so I created a dockerfile in order to make a Docker Image out of the project. An added bonus is that I could publish the server to Docker Hub for other to have easy access to it.

FROM python:alpine3.7
COPY . /esp-update-server
WORKDIR /esp-update-server
RUN pip install -r requirements.txt
EXPOSE 5000
CMD python3 ./server.py

During development I worked on the code on my laptop but wanted the final server to run on a Raspberry Pi. This presented some challenges with the generation of Docker Images which I described in Multi-architecture Images for Docker Hub.

Future

I’ve had the server running for about a week now and identified a few short comings already. It’s kind of like a painting – when is it ever really done?

Server-side logging. Logging through publisher is fine, but if that part fails then you’re a bit in the dark. A bit of server-side logging of updating events would help. The user interface is already pretty informative so no need for logging on that part.
Image size analysis. The HTTP request from the boards contain headers with information on how much space is available. This information should be taken into account before uploading a new binary to ensure it will actually fit.
Security. Some security could be added to make it feasible to run the server over the internet.