esp8266 / Esp8266 firmware management tutorial / Firmware / REST
0

esp8266 self OTA update in HTTPS (SSL/TLS) with trusted self signed certificate – 3

by Renzo Mischianti · Published 9 January 2022 · Updated 11 February 2024

Spread the love

OTA (Over the Air) update is the process of uploading firmware to an ESP module using a Wi-Fi connection rather than a serial port. Such functionality becomes extremely useful in case of limited or no physical access to the module.

esp8266 self OTA update firmware in HTTPS SSL TLS with trusted self signed certificate — esp8266 self OTA update firmware in HTTPS SSL TLS with a trusted self-signed certificate

OTA may be done using:

Arduino IDE
Web Browser
HTTP Server

First of all, read the tutorial “esp8266: flash firmware binary (.bin) compiled and signed“.

In this article, we will explain OTA update via HTTPS secure connection and with a valid self-signed certificate trusted from the client.

Update end point with version control and secure TLS connection

Now we can upgrade our example with SSL/TLS connection.

Self signed certificate generation

But if you want to use a self-signed certificate, you can use OpenSSL to generate private keys and certificates.

You can get OpenSSL by downloading the Linux version from all packet managers or downloading for windows from here. To use It with more simplicity, add to bin pah.

openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout server.key -out server.crt -subj "/CN=192.168.1.70"

and here is the response

Generating a RSA private key
.....+++++
..................................................................................+++++
writing new private key to 'server.key'
-----

Now you have a server.key

and server.crt

Valid self signed certificate

If you don’t need to create an authentic environment, go to the next paragraph, the certificate generated in the previous chapter are sufficient for our test.

First of all, we must assign a name to the server host, there are a lot of alternatives, but I think the fastest way is to use a Dynamic DNS.

Commonly Dynamic DNS is used to assign a name dynamically to the IP given to us by ISP, but you can set a static IP and use It like a typical host; here is an example with a free service NO-IP.

Configure Dynamic DNS service

Go to https://www.noip.com/ and Sign Up.

I add my email, insert a password and create a name like serverota.ddns.net, then click on Free Sign Up.

After email and subscription confirmation, go to this link https://my.noip.com/dynamic-dns, and you can see the list of dynamic DNS entries.

NOIP list of dynamic dns entry — NOIP list of dynamic DNS entry

Now click on modify and insert the IP address of your nodejs server; for me, now it is 192.168.1.70.

NOIP modify dynamic dns entry for local LAN IP address — NOIP modify dynamic DNS entry for local LAN IP address

Apply and wait some minutes. After that (on windows), go con cmd and launch. ipconfig /flushdns.

C:\Users\renzo>ipconfig /flushdns

Configurazione IP di Windows

Cache del resolver DNS svuotata.

C:\Users\renzo>ipconfig /flushdns

Windows IP Configuration

Successfully flushed the DNS Resolver Cache

Now you can try ping the name.

C:\Users\renzo>ping serverota.ddns.net

Esecuzione di Ping serverota.ddns.net [192.168.1.70] con 32 byte di dati:
Risposta da 192.168.1.70: byte=32 durata<1ms TTL=128
Risposta da 192.168.1.70: byte=32 durata<1ms TTL=128
Risposta da 192.168.1.70: byte=32 durata<1ms TTL=128

To test the specified resolve of DNS, you can use this script (Linux).

# Renzo Mischianti <www.mischianti.org>
# Specify DNS server
dnsserver="8.8.8.8"
# function to get IP address
function get_ipaddr {
  ip_address=""
    # A and AAA record for IPv4 and IPv6, respectively
    # $1 stands for first argument
  if [ -n "$1" ]; then
    hostname="${1}"
    if [ -z "query_type" ]; then
      query_type="A"
    fi
    # use host command for DNS lookup operations
    host -t ${query_type}  ${hostname} &>/dev/null ${dnsserver}
    if [ "$?" -eq "0" ]; then
      # get ip address
      ip_address="$(host -t ${query_type} ${hostname} ${dnsserver}| awk '/has.*address/{print $NF; exit}')"
    else
      exit 1
    fi
  else
    exit 2
  fi
# display ip
 echo $ip_address
}
hostname="${1}"
for query in "A-IPv4" "AAAA-IPv6"; do
  query_type="$(printf $query | cut -d- -f 1)"
  ipversion="$(printf $query | cut -d- -f 2)"
  address="$(get_ipaddr ${hostname})"
  if [ "$?" -eq "0" ]; then
    if [ -n "${address}" ]; then
    echo "The ${ipversion} adress of the Hostname ${hostname} is: $address"
    fi
  else
    echo "An error occurred"
  fi
done

And you can run It like so:

$ ./testdns.sh serverota.ddns.net
The IPv4 adress of the Hostname serverota.ddns.net is: 192.168.1.70

Generate self signed certificate

You can use the base command already used by modifying the common name.

openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout server.key -out server.crt  -subj "/CN=serverota.ddns.net"

And we have server.key

and a server.crt

Generate certificate with a configuration file

If you want to give more information to the certificate, you can use this form of the configuration file; here my local.cnf

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = IT
ST = Perugia
L = Gubbio
O = www.mischianti.org
OU = dev
CN = 192.168.1.70: Self-signed certificate
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = serverota.ddns.net

and the command becomes

openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout server.key -out server.crt  -config local.cnf

The most relevant data is the DNS.1 that identifies the alternate common name, and I use a familiar term to identify what I want to do.

Certificate configuration for local IP

This generation is not supported for ESP32 but can be helpful for all programmers.

We can generate a self-signed certificate linked to the IP address, not the hostname, and many devices support it, but not the Espressif one.

Here is the example.

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = IT
ST = Perugia
L = Gubbio
O = www.mischianti.org
OU = dev
CN = 192.168.1.70: Self-signed certificate
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
[alt_names]
IP.1 = 192.168.1.70

Add certificate to NodeJS server

First, modify the server to expose HTTPS endpoint, I select 3443 port for that purpose, and I add the two certificates to the certificate folder.

var fs = require('fs');

const express = require('express');
var http = require('http');
var https = require('https');
var privateKey  = fs.readFileSync('certificate/server.key', 'utf8');
var certificate = fs.readFileSync('certificate/server.crt', 'utf8');

var credentials = {key: privateKey, cert: certificate};

const { networkInterfaces } = require('os');
const path = require('path');

const app = express();
var httpServer = http.createServer(app);
var httpsServer = https.createServer(credentials, app);

const nets = networkInterfaces();

// Server port
const PORT = 3000;
const PORT_HTTPS = 3443;

app.get('/', (request, response) => response.send('Hello from www.mischianti.org!'));

let downloadCounter = 1;
const LAST_VERSION = 0.3;
app.get('/update', (request, response) => {
    const version = (request.header("x-esp8266-version"))?parseFloat(request.header("x-esp8266-version")):Infinity;

    if (version<LAST_VERSION){
        // If you need an update go here
        response.status(200).download(path.join(__dirname, 'firmware/httpUpdateNew.bin'), 'httpUpdateNew.bin', (err)=>{
            if (err) {
                console.error("Problem on download firmware: ", err)
            }else{
                downloadCounter++;
            }
        });
        console.log('Your file has been downloaded '+downloadCounter+' times!')
    }else{
        response.status(304).send('No update needed!')
    }
});


httpServer.listen(PORT, () => {
    const results = {}; // Or just '{}', an empty object

    for (const name of Object.keys(nets)) {
        for (const net of nets[name]) {
            // Skip over non-IPv4 and internal (i.e. 127.0.0.1) addresses
            if (net.family === 'IPv4' && !net.internal) {
                if (!results[name]) {
                    results[name] = [];
                }
                results[name].push(net.address);
            }
        }
    }

    console.log('HTTP Listening on port '+PORT+'\n', results)
});
httpsServer.listen(PORT_HTTPS, () => console.log('HTTPS Listening on port '+PORT_HTTPS+'\n'));

Now in the command prompt, you have this output.

D:\Projects\IdeaProjects\OTAWebPages\OTAServer>node index.js
HTTP Listening on port 3000
 {
  Ethernet: [ '192.168.1.70' ],
  'VirtualBox Host-Only Network': [ '192.168.56.1' ],
  'VMware Network Adapter VMnet1': [ '192.168.113.1' ],
  'VMware Network Adapter VMnet8': [ '192.168.116.1' ]
}
HTTPS Listening on port 3443

Modify sketch to do a secure HTTPs call.

There is two way to execute this example.

Prevent certificate validation

The first is to prevent any certificate validation with this command

     client.setInsecure();

or better the more specific command.

    client.allowSelfSignedCerts();

Without adding a certificate, you must use this command if you can’t generate a valid certificate.

Create and add CertStore

The second is to trust the server certificate without using a certification authority; you can do that by adding to the device a valid certificate of the server.

Practically you are going to add directly to the client the certificate needed.

static const char serverCACert[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
MIIDGzCCAgOgAwIBAgIUWd2CIejZK0uuqLPUDTUkA20SyAAwDQYJKoZIhvcNAQEL
BQAwHTEbMBkGA1UEAwwSc2VydmVyb3RhLmRkbnMubmV0MB4XDTIxMDkzMDEyMjkx
MFoXDTIzMDkzMDEyMjkxMFowHTEbMBkGA1UEAwwSc2VydmVyb3RhLmRkbnMubmV0
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxXtdGLH6IZKnpJYCyyI9
PWN6dQe4wM0J85lqF+omjxmHCY4WQfON6PwIxlhnDDGC4pj/2m2wFkIpg1sDyMmS
Ow8Q0c7Zu3nqDk5gptKWtkjQdmOlsa48jAX49fwvb6Fwyh+Hz94lLuCTQi59LG2S
UPuubXh74ct1pKnc8pwgtIuClSSbgcLCGkhAnLbjTJaet4UemgVcUcorlmqwDKbd
VQR+5DL6biVxy+cLJcwKqH93xrj9hh1I4ktWSBpM/xInlGnA1Fa0qtGrM6m+d3uq
E/GL6MpM0kIvTyxyFVk+LlProQS8zNyRU1KAw51EcT5qcj6FkVMVpSodyARoJPwn
BwIDAQABo1MwUTAdBgNVHQ4EFgQUY6S5QEqFZe7UTVQLAuet07qsW6MwHwYDVR0j
BBgwFoAUY6S5QEqFZe7UTVQLAuet07qsW6MwDwYDVR0TAQH/BAUwAwEB/zANBgkq
hkiG9w0BAQsFAAOCAQEAmxHExHoWclbgdhsIenSyb2wihbC7+QO69dAfQPwJOEQ9
9Lb4h6nmJzuHD+ohFsnNDm3cnKadG7F7to7/LdeG4g5qBNzdMVolsgMTMQ0wyFBx
iTxKPh2FGsGvzftJoYLNXAYXDKtrwK7cxn+HOQOqCw7Q2clMRljva42GQJytDsOx
7F6pVNDnDzo2H1Sni7WzwzIQGb06dUinPY4AhunYRaasbWAU4a4K35x2c5IqCrMW
5TYYCt2KMOaTNLbd0Lh9/ImeJnImAVGN8DivXbtNfhrj+Pl8McHnUztUMkcyHMNR
dVRh0YwsiXtZcu+RWatZB2eJQZJyZx04pIAwgIdhBA==
-----END CERTIFICATE-----
)EOF";

[...]

    BearSSL::WiFiClientSecure client;

    bool mfln = client.probeMaxFragmentLength(updateServer, updateServerPort, 1024);  // server must be the same as in ESPhttpUpdate.update()
    Serial.printf("MFLN supported: %s\n", mfln ? "yes" : "no");
    if (mfln) {
      client.setBufferSizes(1024, 1024);
    }

//    client.allowSelfSignedCerts();

    BearSSL::X509List x509(serverCACert);
    client.setTrustAnchors(&x509);

    setClock();

Set clock for certificate validation

I speak about NTP service in deep on this article “Network Time Protocol (NTP), Timezone and Daylight saving time (DST) with esp8266, esp32 or Arduino,” and we are going to use this function.

// Set time via NTP, as required for x.509 validation
time_t setClock() {
  configTime(2*3600, 0, "pool.ntp.org", "time.nist.gov");

  Serial.print("Waiting for NTP time sync: ");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));
  return now;
}

Complete sketch for secure update

Now we will assemble all the parts, and the result is this.

/**
 * Sketch with secure and conditionally firmware update
 * with trusted certification authority
 *
 * Renzo Mischianti <www.mischianti.org>
 *
 * https://mischianti.org
 */

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>

#include <time.h>

#ifndef APSSID
#define APSSID "<YOUR-SSID>"
#define APPSK  "<YOUR-PASSWD>"
#endif

ESP8266WiFiMulti WiFiMulti;

// Set time via NTP, as required for x.509 validation
time_t setClock() {
  configTime(2*3600, 0, "pool.ntp.org", "time.nist.gov");

  Serial.print("Waiting for NTP time sync: ");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));
  return now;
}

void update_started() {
  Serial.println("CALLBACK:  HTTP update process started");
}

void update_finished() {
  Serial.println("CALLBACK:  HTTP update process finished");
}

void update_progress(int cur, int total) {
  Serial.printf("CALLBACK:  HTTP update process at %d of %d bytes...\n", cur, total);
}

void update_error(int err) {
  Serial.printf("CALLBACK:  HTTP update fatal error code %d\n", err);
}

static const char serverCACert[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
MIIDGzCCAgOgAwIBAgIUWd2CIejZK0uuqLPUDTUkA20SyAAwDQYJKoZIhvcNAQEL
BQAwHTEbMBkGA1UEAwwSc2VydmVyb3RhLmRkbnMubmV0MB4XDTIxMDkzMDEyMjkx
MFoXDTIzMDkzMDEyMjkxMFowHTEbMBkGA1UEAwwSc2VydmVyb3RhLmRkbnMubmV0
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxXtdGLH6IZKnpJYCyyI9
PWN6dQe4wM0J85lqF+omjxmHCY4WQfON6PwIxlhnDDGC4pj/2m2wFkIpg1sDyMmS
Ow8Q0c7Zu3nqDk5gptKWtkjQdmOlsa48jAX49fwvb6Fwyh+Hz94lLuCTQi59LG2S
UPuubXh74ct1pKnc8pwgtIuClSSbgcLCGkhAnLbjTJaet4UemgVcUcorlmqwDKbd
VQR+5DL6biVxy+cLJcwKqH93xrj9hh1I4ktWSBpM/xInlGnA1Fa0qtGrM6m+d3uq
E/GL6MpM0kIvTyxyFVk+LlProQS8zNyRU1KAw51EcT5qcj6FkVMVpSodyARoJPwn
BwIDAQABo1MwUTAdBgNVHQ4EFgQUY6S5QEqFZe7UTVQLAuet07qsW6MwHwYDVR0j
BBgwFoAUY6S5QEqFZe7UTVQLAuet07qsW6MwDwYDVR0TAQH/BAUwAwEB/zANBgkq
hkiG9w0BAQsFAAOCAQEAmxHExHoWclbgdhsIenSyb2wihbC7+QO69dAfQPwJOEQ9
9Lb4h6nmJzuHD+ohFsnNDm3cnKadG7F7to7/LdeG4g5qBNzdMVolsgMTMQ0wyFBx
iTxKPh2FGsGvzftJoYLNXAYXDKtrwK7cxn+HOQOqCw7Q2clMRljva42GQJytDsOx
7F6pVNDnDzo2H1Sni7WzwzIQGb06dUinPY4AhunYRaasbWAU4a4K35x2c5IqCrMW
5TYYCt2KMOaTNLbd0Lh9/ImeJnImAVGN8DivXbtNfhrj+Pl8McHnUztUMkcyHMNR
dVRh0YwsiXtZcu+RWatZB2eJQZJyZx04pIAwgIdhBA==
-----END CERTIFICATE-----
)EOF";

#define FIRMWARE_VERSION "0.1"

void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);

  WiFiMulti.addAP(APSSID, APPSK);

  Serial.print(F("Firmware version "));
  Serial.println(FIRMWARE_VERSION);
}

const char* updateServer = "serverota.ddns.net";
const int updateServerPort = 3443;

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    BearSSL::WiFiClientSecure client;

    bool mfln = client.probeMaxFragmentLength(updateServer, updateServerPort, 1024);  // server must be the same as in ESPhttpUpdate.update()
    Serial.printf("MFLN supported: %s\n", mfln ? "yes" : "no");
    if (mfln) {
      client.setBufferSizes(1024, 1024);
    }

//    client.allowSelfSignedCerts();

    BearSSL::X509List x509(serverCACert);
    client.setTrustAnchors(&x509);

    setClock();

    // The line below is optional. It can be used to blink the LED on the board during flashing
    // The LED will be on during download of one buffer of data from the network. The LED will
    // be off during writing that buffer to flash
    // On a good connection the LED should flash regularly. On a bad connection the LED will be
    // on much longer than it will be off. Other pins than LED_BUILTIN may be used. The second
    // value is used to put the LED on. If the LED is on with HIGH, that value should be passed
    ESPhttpUpdate.setLedPin(LED_BUILTIN, LOW);

    // Add optional callback notifiers
    ESPhttpUpdate.onStart(update_started);
    ESPhttpUpdate.onEnd(update_finished);
    ESPhttpUpdate.onProgress(update_progress);
    ESPhttpUpdate.onError(update_error);

    ESPhttpUpdate.rebootOnUpdate(false); // remove automatic update

    Serial.println(F("Update start now!"));

     t_httpUpdate_return ret = ESPhttpUpdate.update(client, "https://"+String(updateServer)+":"+String(updateServerPort)+"/update", FIRMWARE_VERSION);

    switch (ret) {
      case HTTP_UPDATE_FAILED:
        Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
        Serial.println(F("Retry in 10secs!"));
        delay(10000); // Wait 10secs
        break;

      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("HTTP_UPDATE_NO_UPDATES");
        Serial.println("Your code is up to date!");
          delay(10000); // Wait 10secs
        break;

      case HTTP_UPDATE_OK:
        Serial.println("HTTP_UPDATE_OK");
        delay(1000); // Wait a second and restart
        ESP.restart();
        break;
    }
  }
}

The serial output is the same as the HTTP update.

[SETUP] WAIT 4...
[SETUP] WAIT 3...
[SETUP] WAIT 2...
[SETUP] WAIT 1...
Firmware version 0.1
MFLN supported: yes
Waiting for NTP time sync: 
Current time: Thu Sep 30 14:38:12 2021
Update start now!
CALLBACK:  HTTP update process started
CALLBACK:  HTTP update process at 0 of 420432 bytes...
CALLBACK:  HTTP update process at 0 of 420432 bytes...
CALLBACK:  HTTP update process at 4096 of 420432 bytes...
CALLBACK:  HTTP update process at 8192 of 420432 bytes...
CALLBACK:  HTTP update process at 12288 of 420432 bytes...
CALLBACK:  HTTP update process at 16384 of 420432 bytes...
CALLBACK:  HTTP update process at 20480 of 420432 bytes...
CALLBACK:  HTTP update process at 24576 of 420432 bytes...
CALLBACK:  HTTP update process at 28672 of 420432 bytes...
CALLBACK:  HTTP update process at 32768 of 420432 bytes...
CALLBACK:  HTTP update process at 36864 of 420432 bytes...
CALLBACK:  HTTP update process at 40960 of 420432 bytes...
CALLBACK:  HTTP update process at 45056 of 420432 bytes...
CALLBACK:  HTTP update process at 49152 of 420432 bytes...
CALLBACK:  HTTP update process at 53248 of 420432 bytes...
CALLBACK:  HTTP update process at 57344 of 420432 bytes...
CALLBACK:  HTTP update process at 61440 of 420432 bytes...
CALLBACK:  HTTP update process at 65536 of 420432 bytes...
CALLBACK:  HTTP update process at 69632 of 420432 bytes...
CALLBACK:  HTTP update process at 73728 of 420432 bytes...
CALLBACK:  HTTP update process at 77824 of 420432 bytes...

[...]

CALLBACK:  HTTP update process at 393216 of 420432 bytes...
CALLBACK:  HTTP update process at 397312 of 420432 bytes...
CALLBACK:  HTTP update process at 401408 of 420432 bytes...
CALLBACK:  HTTP update process at 405504 of 420432 bytes...
CALLBACK:  HTTP update process at 409600 of 420432 bytes...
CALLBACK:  HTTP update process at 413696 of 420432 bytes...
CALLBACK:  HTTP update process at 417792 of 420432 bytes...
CALLBACK:  HTTP update process at 420432 of 420432 bytes...
CALLBACK:  HTTP update process at 420432 of 420432 bytes...
CALLBACK:  HTTP update process at 420432 of 420432 bytes...
CALLBACK:  HTTP update process finished
HTTP_UPDATE_OK
 
 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4 
chksum 0xc9
csum 0xc9
v00078180 
@cp:B0
ld
 


[SETUP] WAIT 4...
[SETUP] WAIT 3...
[SETUP] WAIT 2...
[SETUP] WAIT 1...
Firmware version 0.2
MFLN supported: yes
Waiting for NTP time sync: 
Current time: Thu Sep 30 14:38:44 2021
Update start now!
HTTP_UPDATE_NO_UPDATES
Your code is up to date!