esp32 / ESP32 firmware management tutorial / Firmware / FTP / No category / SD Card
2

ESP32 firmware and filesystem update with FTP client – 2

by Renzo Mischianti · Published 24 April 2023 · Updated 11 February 2024

Spread the love

Welcome to the final installment of our comprehensive series on ESP32 firmware and Over-the-Air (OTA) update management. This series has been a deep dive into various methods and strategies for updating firmware and filesystems on the popular ESP32 microcontroller. Today, we will focus on the last article, where we will explore how to use an FTP client to update the firmware and filesystem on an ESP32 device seamlessly.

Throughout the series, we have covered a wide range of topics and update methodologies, such as flashing compiled firmware (.bin) files, OTA updates using the Arduino IDE, web browser-based updates for firmware and filesystems, self-OTA updates from HTTP servers, and even updates from SD cards. Each method has its advantages and specific use cases, but now we will concentrate on the FTP client method to complete our exploration of ESP32 update techniques.

ESP32 firmware and filesystem update with FTP client

In this article, we will guide you through the process of using an FTP client to update the firmware and filesystem on ESP32 microcontrollers, discussing its benefits and practical applications. We will provide step-by-step instructions, tips, and best practices to help you fully leverage the power and flexibility of ESP32 devices.

Here my device selection ESP32 Dev Kit v1 - TTGO T-Display 1.14 ESP32 - NodeMCU V3 V2 ESP8266 Lolin32 - NodeMCU ESP-32S - WeMos Lolin32 - WeMos Lolin32 mini - ESP32-CAM programmer - ESP32-CAM bundle - ESP32-WROOM-32 - ESP32-S

We hope that this comprehensive series has provided valuable insights and practical knowledge that you can apply to your own ESP32 projects. Let’s dive in and master the last technique in our extensive exploration of ESP32 update methods.

Update firmware from storage with Update class

To put on work this system, we must use an SD or local filesystem. And we must use the Update class.

A method, in particular, can manage updates via stream

Update.begin(firmwareSizeInBytes);
Update.writeStream(streamVar);
Update.end();

Upload firmware via FTP and automatically update

I created some applications that store logs and statistics that can be downloaded via an FTP client. To do that, I use a simple library that you can check in this article, “FTP server on esp8266 and esp32“, I create that library, and I customize It with some callback that can be used for our purpose.

We can use SPIFFS, LittleFS, FFat, or SD, check on these articles for the basics:

Update firmware with FTP server over SPIFFS

For this test, remember to modify (or check) that FtpServerKey.h is like so

#ifndef DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP32
    #define DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP32   NETWORK_ESP32
    #define DEFAULT_STORAGE_TYPE_ESP32 STORAGE_SPIFFS
#endif

To understand how to generate the compiled firmware, read this article “ESP32: flash compiled firmware (.bin)“, it is quite simple, but it is mandatory to understand all the steps of this article.

Here I write an example that uses the SimpleFTPServer library to upload the firmware.bin file on SPIFFS, and when the system store that file starts, the update of firmware.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

/*
 * Upload firmware with FtpServer (ESP32 with SPIFFS)
 * when uploaded start automatic Update and reboot
 *
 * AUTHOR:  Renzo Mischianti
 *
 * https://mischianti.org/
 *
 */
 
#include <WiFi.h>
#include <Update.h>
#include <SPIFFS.h>
 
#include <SimpleFTPServer.h>
 
const char* ssid = "<YOUR-SSID>";
const char* password = "<YOUR-PASSWD>";
 
 
FtpServer ftpSrv;   //set #define FTP_DEBUG in ESP8266FtpServer.h to see ftp verbose on serial
 
bool isFirmwareUploaded = false;
 
void progressCallBack(size_t currSize, size_t totalSize) {
    Serial.printf("CALLBACK:  Update process at %d of %d bytes...\n", currSize, totalSize);
}
 
#define FIRMWARE_VERSION 0.2
 
void _callback(FtpOperation ftpOperation, unsigned int freeSpace, unsigned int totalSpace){
  switch (ftpOperation) {
    case FTP_CONNECT:
      Serial.println(F("FTP: Connected!"));
      break;
    case FTP_DISCONNECT:
      Serial.println(F("FTP: Disconnected!"));
      break;
    case FTP_FREE_SPACE_CHANGE:
      if (isFirmwareUploaded){
          Serial.println(F("The uploaded firmware now stored in FS!"));
          Serial.print(F("\nSearch for firmware in FS.."));
          String name = "firmware.bin";
          File firmware =  SPIFFS.open(name, FTP_FILE_READ);
          if (firmware) {
            Serial.println(F("found!"));
            Serial.println(F("Try to update!"));
 
            Update.onProgress(progressCallBack);
 
            Update.begin(firmware.size(), U_SPIFFS);
            Update.writeStream(firmware);
            if (Update.end()){
            Serial.println(F("Update finished!"));
            }else{
            Serial.println(F("Update error!"));
            Serial.println(Update.getError());
            }
 
            firmware.close();
 
            String renamed = name;
            renamed.replace(".bin", ".bak");
            if (SPIFFS.rename(name, renamed.c_str())){
            Serial.println(F("Firmware rename succesfully!"));
            }else{
            Serial.println(F("Firmware rename error!"));
            }
            delay(2000);
 
            ESP.restart();
          }else{
            Serial.println(F("not found!"));
          }
          // isFirmwareUploaded = false; // not need by reset
      }
      Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
      break;
    default:
      break;
  }
};
void _transferCallback(FtpTransferOperation ftpOperation, const char* name, unsigned int transferredSize){
  switch (ftpOperation) {
    case FTP_UPLOAD_START:
      Serial.println(F("FTP: Upload start!"));
      break;
    case FTP_UPLOAD:
      Serial.printf("FTP: Upload of file %s byte %u\n", name, transferredSize);
      break;
    case FTP_TRANSFER_STOP:
      Serial.println(F("FTP: Finish transfer!"));
      break;
    case FTP_TRANSFER_ERROR:
      Serial.println(F("FTP: Transfer error!"));
      break;
    default:
      break;
  }
 
  /* FTP_UPLOAD_START = 0,
   * FTP_UPLOAD = 1,
   *
   * FTP_DOWNLOAD_START = 2,
   * FTP_DOWNLOAD = 3,
   *
   * FTP_TRANSFER_STOP = 4,
   * FTP_DOWNLOAD_STOP = 4,
   * FTP_UPLOAD_STOP = 4,
   *
   * FTP_TRANSFER_ERROR = 5,
   * FTP_DOWNLOAD_ERROR = 5,
   * FTP_UPLOAD_ERROR = 5
   */
 
  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){
    isFirmwareUploaded = true;
  }
};
 
void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");
 
  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
 
  ftpSrv.setCallback(_callback);
  ftpSrv.setTransferCallback(_transferCallback);
 
  /////FTP Setup, ensure SPIFFS is started before ftp;  /////////
  if (SPIFFS.begin()) {
      Serial.println("SPIFFS opened!");
      Serial.print(F("\nCurrent firmware version: "));
      Serial.println(FIRMWARE_VERSION);
 
      ftpSrv.begin("esp32","esp32");    //username, password for ftp.   (default 21, 50009 for PASV)
  }
}
void loop(void){
  ftpSrv.handleFTP();        //make sure in loop you call handleFTP()!!
 // server.handleClient();   //example if running a webserver you still need to call .handleClient();
 
}

The code is not so complicated, but you must understand only a few steps.

When FTP upload the file in the callback, you can distinguish three different phase, FTP_UPLOAD_START, FTP_UPLOAD, and FTP_TRANSFER_STOP.

When the transfer stop, the file is completely transferred,

if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){
  isFirmwareUploaded = true;
}

but not yet stored, you must check the event FTP_FREE_SPACE_CHANGE to be sure that the file is completely stored.

switch (ftpOperation) {
  case FTP_CONNECT:
    Serial.println(F("FTP: Connected!"));
    break;
  case FTP_DISCONNECT:
    Serial.println(F("FTP: Disconnected!"));
    break;
  case FTP_FREE_SPACE_CHANGE:
    if (isFirmwareUploaded){
        Serial.println(F("The uploaded firmware now stored in FS!"));
        Serial.print(F("\nSearch for firmware in FS.."));
        String name = "firmware.bin";
        File firmware =  LittleFS.open(name, FTP_FILE_READ);
        if (firmware) {
          Serial.println(F("found!"));
          Serial.println(F("Try to update!"));
 
          Update.onProgress(progressCallBack);
 
          Update.begin(firmware.size(), U_FLASH);
          Update.writeStream(firmware);
          if (Update.end()){
          Serial.println(F("Update finished!"));
          }else{
          Serial.println(F("Update error!"));
          Serial.println(Update.getError());
          }
 
          firmware.close();
 
          String renamed = name;
          renamed.replace(".bin", ".bak");
          if (LittleFS.rename(name, renamed.c_str())){
          Serial.println(F("Firmware rename succesfully!"));
          }else{
          Serial.println(F("Firmware rename error!"));
          }
          delay(2000);
 
          ESP.reset();
        }else{
          Serial.println(F("not found!"));
        }
        // isFirmwareUploaded = false; // not need by reset
    }
    Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
    break;
  default:
    break;
}

Now I check if the file is present and if yes update starts the stream of the file, after upload, It was renamed, and starts the reboot that does the final phase of the update.

.....
Connected to reef-casa-sopra
IP address: 192.168.1.184
SPIFFS opened!
 
Current firmware version: 0.10
FTP: Connected!
FTP: Upload start!
FTP: Upload of file /firmware.bin byte 2048
FTP: Upload of file /firmware.bin byte 4096
FTP: Upload of file /firmware.bin byte 5744
FTP: Upload of file /firmware.bin byte 7180
FTP: Upload of file /firmware.bin byte 9228
FTP: Upload of file /firmware.bin byte 11276
FTP: Upload of file /firmware.bin byte 13324
FTP: Upload of file /firmware.bin byte 15372
FTP: Upload of file /firmware.bin byte 17420
 
[...]
 
FTP: Upload of file /firmware.bin byte 805900
FTP: Upload of file /firmware.bin byte 807948
FTP: Upload of file /firmware.bin byte 809996
FTP: Upload of file /firmware.bin byte 812044
FTP: Upload of file /firmware.bin byte 814092
FTP: Upload of file /firmware.bin byte 816140
FTP: Upload of file /firmware.bin byte 817696
FTP: Finish transfer!
The uploaded firmware now stored in FS!
 
Search for firmware in FS..not found!
FTP: Free space change, free 276602 of 689120!

Update the filesystem with an FTP server over an SD

For this test, remember to modify (or check) that FtpServerKey.h is like so

#ifndef DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266
    #define DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266     NETWORK_ESP8266
    #define DEFAULT_STORAGE_TYPE_ESP8266 STORAGE_SD
#endif

This sketch isn’t more complex, I reuse a part of the previous code, and I add the management of the filesystem.bin file.

We create a version.txt file on the data folder of the sketch, and I write 0.1 inside, and I use that as a version of the FileSystem, then upload the data folder to the filesystem.

Now we are going to modify the file version.txt with version 0.2 then upload to the device, then

Change version to 0.2 in version.txt, regenerate without upload, and rename the file ArduinoOTAesp8266_fs_update.mklittlefs.bin to filesystem.bin and upload the file to the SD.

Here is the complete sketch.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

/*
 * Upload firmware or filesystem (LittleFS) with FtpServer (esp8266 with SD)
 * when uploaded start automatic Update and reboot
 *
 * AUTHOR:  Renzo Mischianti
 *
 * https://mischianti.org/
 *
 */
 
#include <ESP8266WiFi.h>
#include <SD.h>
#include <LittleFS.h>
 
#include <SimpleFTPServer.h>
 
const char* ssid = "<YOUR-SSID>";
const char* password = "<YOUR-PASSWD>";
 
 
FtpServer ftpSrv;   //set #define FTP_DEBUG in ESP8266FtpServer.h to see ftp verbose on serial
 
bool isFirmwareUploaded = false;
bool isFilesystemUploaded = false;
 
void progressCallBack(size_t currSize, size_t totalSize) {
    Serial.printf("CALLBACK:  Update process at %d of %d bytes...\n", currSize, totalSize);
}
 
#define FIRMWARE_VERSION 0.2
String FILESYSTEM_VERSION = "0.0";
 
void _callback(FtpOperation ftpOperation, unsigned int freeSpace, unsigned int totalSpace){
  switch (ftpOperation) {
    case FTP_CONNECT:
      Serial.println(F("FTP: Connected!"));
      break;
    case FTP_DISCONNECT:
      Serial.println(F("FTP: Disconnected!"));
      break;
    case FTP_FREE_SPACE_CHANGE:
      if (isFirmwareUploaded){
          Serial.println(F("The uploaded firmware now stored in FS!"));
          Serial.print(F("\nSearch for firmware in FS.."));
          String name = "firmware.bin";
          File firmware =  SD.open(name, FTP_FILE_READ);
          if (firmware) {
            Serial.println(F("found!"));
            Serial.println(F("Try to update!"));
 
            Update.onProgress(progressCallBack);
 
            Update.begin(firmware.size(), U_FLASH);
            Update.writeStream(firmware);
            if (Update.end()){
            Serial.println(F("Update finished!"));
            }else{
            Serial.println(F("Update error!"));
            Serial.println(Update.getError());
            }
 
            firmware.close();
 
            String renamed = name;
            renamed.replace(".bin", ".bak");
            if (SD.rename(name, renamed.c_str())){
            Serial.println(F("Firmware rename succesfully!"));
            }else{
            Serial.println(F("Firmware rename error!"));
            }
            delay(2000);
 
            ESP.reset();
          }else{
            Serial.println(F("not found!"));
          }
          // isFirmwareUploaded = false; // not need by reset
      }
      if (isFilesystemUploaded){
          Serial.println(F("The uploaded Filesystem now stored in FS!"));
          Serial.print(F("\nSearch for Filesystem in FS.."));
          String name = "filesystem.bin";
          File filesystem =  SD.open(name, FTP_FILE_READ);
          if (filesystem) {
            Serial.println(F("found!"));
            Serial.println(F("Try to update!"));
 
            Update.onProgress(progressCallBack);
 
            Update.begin(filesystem.size(), U_FS);
            Update.writeStream(filesystem);
            if (Update.end()){
            Serial.println(F("Update finished!"));
            }else{
            Serial.println(F("Update error!"));
            Serial.println(Update.getError());
            }
 
            filesystem.close();
 
            String renamed = name;
            renamed.replace(".bin", ".bak");
            if (SD.rename(name, renamed.c_str())){
            Serial.println(F("Filesystem rename succesfully!"));
            }else{
            Serial.println(F("Filesystem rename error!"));
            }
            delay(2000);
 
            ESP.reset();
          }else{
            Serial.println(F("not found!"));
          }
          // isFilesystemUploaded = false; // not need by reset
      }
      Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
      break;
    default:
      break;
  }
};
void _transferCallback(FtpTransferOperation ftpOperation, const char* name, unsigned int transferredSize){
  switch (ftpOperation) {
    case FTP_UPLOAD_START:
      Serial.println(F("FTP: Upload start!"));
      break;
    case FTP_UPLOAD:
      Serial.printf("FTP: Upload of file %s byte %u\n", name, transferredSize);
      break;
    case FTP_TRANSFER_STOP:
      Serial.println(F("FTP: Finish transfer!"));
      break;
    case FTP_TRANSFER_ERROR:
      Serial.println(F("FTP: Transfer error!"));
      break;
    default:
      break;
  }
 
  /* FTP_UPLOAD_START = 0,
   * FTP_UPLOAD = 1,
   *
   * FTP_DOWNLOAD_START = 2,
   * FTP_DOWNLOAD = 3,
   *
   * FTP_TRANSFER_STOP = 4,
   * FTP_DOWNLOAD_STOP = 4,
   * FTP_UPLOAD_STOP = 4,
   *
   * FTP_TRANSFER_ERROR = 5,
   * FTP_DOWNLOAD_ERROR = 5,
   * FTP_UPLOAD_ERROR = 5
   */
 
  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){
    isFirmwareUploaded = true;
  }
  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("filesystem.bin")>=0){
    isFilesystemUploaded = true;
  }
};
 
void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");
 
  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
 
  ftpSrv.setCallback(_callback);
  ftpSrv.setTransferCallback(_transferCallback);
 
  Serial.print(F("Inizializing FS..."));
  if (LittleFS.begin()){
      Serial.println(F("done."));
  }else{
      Serial.println(F("fail."));
  }
 
  Serial.print(F("FileSystem version "));
  File versionFile = LittleFS.open(F("/version.txt"), "r");
  if (versionFile) {
      FILESYSTEM_VERSION = versionFile.readString();
      versionFile.close();
  }
  Serial.println(FILESYSTEM_VERSION);
 
  /////FTP Setup, ensure SD is started before ftp;  /////////
  if (SD.begin(SS)) {
      Serial.println("SD opened!");
      Serial.print(F("\nCurrent firmware version: "));
      Serial.println(FIRMWARE_VERSION);
 
      ftpSrv.begin("esp8266","esp8266");    //username, password for ftp.   (default 21, 50009 for PASV)
  }
}
void loop(void){
  ftpSrv.handleFTP();        //make sure in loop you call handleFTP()!!
 // server.handleClient();   //example if running a webserver you still need to call .handleClient();
 
}

The serial output when I upload the file filesystem.bin

...
Connected to reef-casa-sopra 
IP address: 192.168.1.127
Inizializing FS...done.
FileSystem version 0.1
SD opened!
 
Current firmware version: 0.20
FTP: Connected!
FTP: Free space change, free 1 of 1!
FTP: Free space change, free 1 of 1!
FTP: Free space change, free 1 of 1!
FTP: Upload start!
FTP: Upload of file filesystem.bin byte 1608
FTP: Upload of file filesystem.bin byte 2680
FTP: Upload of file filesystem.bin byte 3216
FTP: Upload of file filesystem.bin byte 4288
FTP: Upload of file filesystem.bin byte 4824
FTP: Upload of file filesystem.bin byte 5360
FTP: Upload of file filesystem.bin byte 6968
FTP: Upload of file filesystem.bin byte 7504
FTP: Upload of file filesystem.bin byte 8040
FTP: Upload of file filesystem.bin byte 9112
FTP: Upload of file filesystem.bin byte 9648
FTP: Upload of file filesystem.bin byte 10184
 
[...]
 
FTP: Upload of file filesystem.bin byte 2068960
FTP: Upload of file filesystem.bin byte 2069496
FTP: Upload of file filesystem.bin byte 2070568
FTP: Upload of file filesystem.bin byte 2071104
FTP: Upload of file filesystem.bin byte 2071640
FTP: Upload of file filesystem.bin byte 2072576
FTP: Finish transfer!
The uploaded Filesystem now stored in FS!
 
Search for Filesystem in FS..found!
Try to update!
CALLBACK:  Update process at 0 of 2072576 bytes...
CALLBACK:  Update process at 4096 of 2072576 bytes...
CALLBACK:  Update process at 8192 of 2072576 bytes...
CALLBACK:  Update process at 12288 of 2072576 bytes...
CALLBACK:  Update process at 16384 of 2072576 bytes...
CALLBACK:  Update process at 20480 of 2072576 bytes...
CALLBACK:  Update process at 24576 of 2072576 bytes...
 
[...]
 
CALLBACK:  Update process at 2052096 of 2072576 bytes...
CALLBACK:  Update process at 2056192 of 2072576 bytes...
CALLBACK:  Update process at 2060288 of 2072576 bytes...
CALLBACK:  Update process at 2064384 of 2072576 bytes...
CALLBACK:  Update process at 2068480 of 2072576 bytes...
CALLBACK:  Update process at 2072576 of 2072576 bytes...
CALLBACK:  Update process at 2072576 of 2072576 bytes...
Update finished!
Filesystem rename succesfully!
 
 ets Jan  8 2013,rst cause:2, boot mode:(3,6)
 
load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00070bd0
~ld
  
.......
Connected to reef-casa-sopra 
IP address: 192.168.1.127
Inizializing FS...done.
FileSystem version 0.2
SD opened!
 
Current firmware version: 0.20

Thanks

Firmware and OTA update management
1. Firmware management
  1. ESP32: flash compiled firmware (.bin)
  2. ESP32: flash compiled firmware and filesystem (.bin) with GUI tools
2. OTA update with Arduino IDE
  1. ESP32 OTA update with Arduino IDE: filesystem, firmware, and password
3. OTA update with Web Browser
4. Self OTA uptate from HTTP server
5. Non-standard Firmware update
  1. ESP32 firmware and filesystem update from SD card
  2. ESP32 firmware and filesystem update with FTP client

Support forum

Spread the love

Tags: esp32 File System Firmware SD Card

Eduardo says:

1 August 2023 at 09:49

Hi.
I am testing your code.

After uploading the firmware.bin file to the board, it performs the update but when I reboot it gives me the following error. “E (2718) SPIFFS: mount failed, -10025”

I tried deleting the file instead of renaming it and in both cases it gives me an error when performing the operation.

Even though I update the firmware from the arduino ide, it still gives me the same error.

In order to try again, I first have to upload a different program.

I’ve been looking for information about this error but can’t find anything.

Reply
- Renzo Mischianti says:
  
  1 August 2023 at 12:07
  
  Hi Eduardo,
  check if the board configuration and FtpServerKey.h point to SPIFFS.
  Bye Renzo
  
  Reply