esp8266 firmware and filesystem update with FTP client – 2

by · Published · Updated

Spread the love
           

In this series of article about firmware and how to update It, I’d like to add a series of alternative methods of updating that are very useful for me.

esp8266 firmware and filesystem update with FTP client
esp8266 firmware and filesystem update with FTP client

In this article, we will learn how to add an FTP server to our device and use It to upload firmware and filesystem updates.

Update firmware from a storage with Update class

The basic methods to update a stream of data are contained in the Update class.

A method, in particular, can manage updates via stream

Update.begin(firmwareSizeInBytes);
Update.writeStream(streamVar);
Update.end();

Upload firmware via FTP and automatic update

Another very interesting system to update firmware is by using an FTP client.

I have created some applications for storing logs and statistics and then downloading the data via an FTP client. To do that, I use a simple library that you can check on this article “FTP server on esp8266 and esp32” I created that library, and I customized It with some callback, and you can use them for our purpose.

We can use SPIFFS LittleFS or SD. Check on these articles the basics:

Update firmware with FTP server over LittleFS

For this test, remember to modify (or check) that FtpServerKey.h is like so

#ifndef DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266
	#define DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266 	NETWORK_ESP8266
	#define DEFAULT_STORAGE_TYPE_ESP8266 STORAGE_LITTLEFS
#endif

To understand how to generate the compiled firmware, read this article “esp8266: flash firmware binary (.bin) compiled and signed“, which is quite simple but is mandatory to understand all the steps.

Here I write an example that uses the SimpleFTPServer library to upload the firmware.bin file on LittleFS, and when the system store that file, it starts the update of firmware.

/*
 * Upload firmware with FtpServer (esp8266 with LittleFS)
 * when uploaded start automatic Update and reboot
 *
 * AUTHOR:  Renzo Mischianti
 *
 * https://mischianti.org/
 *
 */

#include <ESP8266WiFi.h>
#include <LittleFS.h>

#include <SimpleFTPServer.h>

const char* ssid = "<YOUR-SSID>";
const char* password = "<YOUR-PASSWD>";


FtpServer ftpSrv;   //set #define FTP_DEBUG in ESP8266FtpServer.h to see ftp verbose on serial

bool isFirmwareUploaded = false;

void progressCallBack(size_t currSize, size_t totalSize) {
    Serial.printf("CALLBACK:  Update process at %d of %d bytes...\n", currSize, totalSize);
}

#define FIRMWARE_VERSION 0.2

void _callback(FtpOperation ftpOperation, unsigned int freeSpace, unsigned int totalSpace){
  switch (ftpOperation) {
    case FTP_CONNECT:
      Serial.println(F("FTP: Connected!"));
      break;
    case FTP_DISCONNECT:
      Serial.println(F("FTP: Disconnected!"));
      break;
    case FTP_FREE_SPACE_CHANGE:
      if (isFirmwareUploaded){
          Serial.println(F("The uploaded firmware now stored in FS!"));
		  Serial.print(F("\nSearch for firmware in FS.."));
		  String name = "firmware.bin";
		  File firmware =  LittleFS.open(name, FTP_FILE_READ);
		  if (firmware) {
			Serial.println(F("found!"));
			Serial.println(F("Try to update!"));

			Update.onProgress(progressCallBack);

			Update.begin(firmware.size(), U_FLASH);
			Update.writeStream(firmware);
			if (Update.end()){
			Serial.println(F("Update finished!"));
			}else{
			Serial.println(F("Update error!"));
			Serial.println(Update.getError());
			}

			firmware.close();

			String renamed = name;
			renamed.replace(".bin", ".bak");
			if (LittleFS.rename(name, renamed.c_str())){
			Serial.println(F("Firmware rename succesfully!"));
			}else{
			Serial.println(F("Firmware rename error!"));
			}
			delay(2000);

			ESP.reset();
		  }else{
			Serial.println(F("not found!"));
		  }
		  // isFirmwareUploaded = false; // not need by reset
      }
      Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
      break;
    default:
      break;
  }
};
void _transferCallback(FtpTransferOperation ftpOperation, const char* name, unsigned int transferredSize){
  switch (ftpOperation) {
    case FTP_UPLOAD_START:
      Serial.println(F("FTP: Upload start!"));
      break;
    case FTP_UPLOAD:
      Serial.printf("FTP: Upload of file %s byte %u\n", name, transferredSize);
      break;
    case FTP_TRANSFER_STOP:
      Serial.println(F("FTP: Finish transfer!"));
      break;
    case FTP_TRANSFER_ERROR:
      Serial.println(F("FTP: Transfer error!"));
      break;
    default:
      break;
  }

  /* FTP_UPLOAD_START = 0,
   * FTP_UPLOAD = 1,
   *
   * FTP_DOWNLOAD_START = 2,
   * FTP_DOWNLOAD = 3,
   *
   * FTP_TRANSFER_STOP = 4,
   * FTP_DOWNLOAD_STOP = 4,
   * FTP_UPLOAD_STOP = 4,
   *
   * FTP_TRANSFER_ERROR = 5,
   * FTP_DOWNLOAD_ERROR = 5,
   * FTP_UPLOAD_ERROR = 5
   */

  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){
    isFirmwareUploaded = true;
  }
};

void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  ftpSrv.setCallback(_callback);
  ftpSrv.setTransferCallback(_transferCallback);

  /////FTP Setup, ensure LittleFS is started before ftp;  /////////
  if (LittleFS.begin()) {
      Serial.println("LittleFS opened!");
      Serial.print(F("\nCurrent firmware version: "));
      Serial.println(FIRMWARE_VERSION);

      ftpSrv.begin("esp8266","esp8266");    //username, password for ftp.   (default 21, 50009 for PASV)
  }
}
void loop(void){
  ftpSrv.handleFTP();        //make sure in loop you call handleFTP()!!
 // server.handleClient();   //example if running a webserver you still need to call .handleClient();

}

The code is not so complicated, but you must understand only a few steps.

When FTP uploads the file in the callback, you can distinguish three phases: the FTP_UPLOAD_START, FTP_UPLOAD, and FTP_TRANSFER_STOP.

When transfer stops, the file is completely transferred 

  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){
    isFirmwareUploaded = true;
  }

but not yet stored, and you must check the event FTP_FREE_SPACE_CHANGE to ensure that the file is ultimately stored.

  switch (ftpOperation) {
    case FTP_CONNECT:
      Serial.println(F("FTP: Connected!"));
      break;
    case FTP_DISCONNECT:
      Serial.println(F("FTP: Disconnected!"));
      break;
    case FTP_FREE_SPACE_CHANGE:
      if (isFirmwareUploaded){
          Serial.println(F("The uploaded firmware now stored in FS!"));
		  Serial.print(F("\nSearch for firmware in FS.."));
		  String name = "firmware.bin";
		  File firmware =  LittleFS.open(name, FTP_FILE_READ);
		  if (firmware) {
			Serial.println(F("found!"));
			Serial.println(F("Try to update!"));

			Update.onProgress(progressCallBack);

			Update.begin(firmware.size(), U_FLASH);
			Update.writeStream(firmware);
			if (Update.end()){
			Serial.println(F("Update finished!"));
			}else{
			Serial.println(F("Update error!"));
			Serial.println(Update.getError());
			}

			firmware.close();

			String renamed = name;
			renamed.replace(".bin", ".bak");
			if (LittleFS.rename(name, renamed.c_str())){
			Serial.println(F("Firmware rename succesfully!"));
			}else{
			Serial.println(F("Firmware rename error!"));
			}
			delay(2000);

			ESP.reset();
		  }else{
			Serial.println(F("not found!"));
		  }
		  // isFirmwareUploaded = false; // not need by reset
      }
      Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
      break;
    default:
      break;
  }

Now I check if the file is present and if yes, the update starts the stream of the file; after upload, It is renamed and start the reboot that does the final phase of the update.

.......
Connected to reef-casa-sopra
IP address: 192.168.1.127
LittleFS opened!

Current firmware version: 0.10
FTP: Connected!
FTP: Upload start!
FTP: Upload of file firmware.bin byte 2048
FTP: Upload of file firmware.bin byte 3216
FTP: Upload of file firmware.bin byte 3752
FTP: Upload of file firmware.bin byte 5360
FTP: Upload of file firmware.bin byte 7408
FTP: Upload of file firmware.bin byte 8040
FTP: Upload of file firmware.bin byte 9112
FTP: Upload of file firmware.bin byte 11160
FTP: Upload of file firmware.bin byte 11792
FTP: Upload of file firmware.bin byte 12864
FTP: Upload of file firmware.bin byte 14912
FTP: Upload of file firmware.bin byte 15544
FTP: Upload of file firmware.bin byte 16080
FTP: Upload of file firmware.bin byte 17688

[...]


FTP: Upload of file firmware.bin byte 327400
FTP: Upload of file firmware.bin byte 329104
FTP: Upload of file firmware.bin byte 329640
FTP: Upload of file firmware.bin byte 330176
FTP: Upload of file firmware.bin byte 331784
FTP: Upload of file firmware.bin byte 333832
FTP: Upload of file firmware.bin byte 335536
FTP: Upload of file firmware.bin byte 336072
FTP: Upload of file firmware.bin byte 338120
FTP: Upload of file firmware.bin byte 338752
FTP: Upload of file firmware.bin byte 340360
FTP: Upload of file firmware.bin byte 341600
FTP: Finish transfer!
The uploaded firmware now stored in FS!

Search for firmware in FS..found!
Try to update!
CALLBACK:  Update process at 0 of 341600 bytes...
CALLBACK:  Update process at 4096 of 341600 bytes...
CALLBACK:  Update process at 8192 of 341600 bytes...
CALLBACK:  Update process at 12288 of 341600 bytes...
CALLBACK:  Update process at 16384 of 341600 bytes...
CALLBACK:  Update process at 20480 of 341600 bytes...
CALLBACK:  Update process at 24576 of 341600 bytes...
CALLBACK:  Update process at 28672 of 341600 bytes...
CALLBACK:  Update process at 32768 of 341600 bytes...
CALLBACK:  Update process at 36864 of 341600 bytes...

[...]

CALLBACK:  Update process at 319488 of 341600 bytes...
CALLBACK:  Update process at 323584 of 341600 bytes...
CALLBACK:  Update process at 327680 of 341600 bytes...
CALLBACK:  Update process at 331776 of 341600 bytes...
CALLBACK:  Update process at 335872 of 341600 bytes...
CALLBACK:  Update process at 339968 of 341600 bytes...
CALLBACK:  Update process at 341600 of 341600 bytes...
CALLBACK:  Update process at 341600 of 341600 bytes...
Update finished!
Firmware rename succesfully!

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00053100
@cp:B0
ld

.......
Connected to reef-casa-sopra
IP address: 192.168.1.127
LittleFS opened!

Current firmware version: 0.20

Update filesystem with FTP server over an SD

For this test, remember to modify (or check) that FtpServerKey.h is like so

#ifndef DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266
	#define DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266 	NETWORK_ESP8266
	#define DEFAULT_STORAGE_TYPE_ESP8266 STORAGE_SD
#endif

This sketch isn’t more complex, I reuse a part of previous code, and I add the management of the filesystem.bin file.

We create a version.txt file on the sketch’s data folder, and I write 0.1 inside, and I use that as the version of the FileSystem, then upload the data folder to the filesystem.

Now we are going to modify the file version.txt with the version 0.2 then upload to the device, then

Change version to 0.2 in version.txt, regenerate without upload, and rename the file ArduinoOTAesp8266_fs_update.mklittlefs.bin to filesystem.bin and upload the file to the SD.

Here is the complete sketch.

/*
 * Upload firmware or filesystem (LittleFS) with FtpServer (esp8266 with SD)
 * when uploaded start automatic Update and reboot
 *
 * AUTHOR:  Renzo Mischianti
 *
 * https://mischianti.org/
 *
 */

#include <ESP8266WiFi.h>
#include <SD.h>
#include <LittleFS.h>

#include <SimpleFTPServer.h>

const char* ssid = "<YOUR-SSID>";
const char* password = "<YOUR-PASSWD>";


FtpServer ftpSrv;   //set #define FTP_DEBUG in ESP8266FtpServer.h to see ftp verbose on serial

bool isFirmwareUploaded = false;
bool isFilesystemUploaded = false;

void progressCallBack(size_t currSize, size_t totalSize) {
    Serial.printf("CALLBACK:  Update process at %d of %d bytes...\n", currSize, totalSize);
}

#define FIRMWARE_VERSION 0.2
String FILESYSTEM_VERSION = "0.0";

void _callback(FtpOperation ftpOperation, unsigned int freeSpace, unsigned int totalSpace){
  switch (ftpOperation) {
    case FTP_CONNECT:
      Serial.println(F("FTP: Connected!"));
      break;
    case FTP_DISCONNECT:
      Serial.println(F("FTP: Disconnected!"));
      break;
    case FTP_FREE_SPACE_CHANGE:
      if (isFirmwareUploaded){
          Serial.println(F("The uploaded firmware now stored in FS!"));
		  Serial.print(F("\nSearch for firmware in FS.."));
		  String name = "firmware.bin";
		  File firmware =  SD.open(name, FTP_FILE_READ);
		  if (firmware) {
			Serial.println(F("found!"));
			Serial.println(F("Try to update!"));

			Update.onProgress(progressCallBack);

			Update.begin(firmware.size(), U_FLASH);
			Update.writeStream(firmware);
			if (Update.end()){
			Serial.println(F("Update finished!"));
			}else{
			Serial.println(F("Update error!"));
			Serial.println(Update.getError());
			}

			firmware.close();

			String renamed = name;
			renamed.replace(".bin", ".bak");
			if (SD.rename(name, renamed.c_str())){
			Serial.println(F("Firmware rename succesfully!"));
			}else{
			Serial.println(F("Firmware rename error!"));
			}
			delay(2000);

			ESP.reset();
		  }else{
			Serial.println(F("not found!"));
		  }
		  // isFirmwareUploaded = false; // not need by reset
      }
      if (isFilesystemUploaded){
          Serial.println(F("The uploaded Filesystem now stored in FS!"));
		  Serial.print(F("\nSearch for Filesystem in FS.."));
		  String name = "filesystem.bin";
		  File filesystem =  SD.open(name, FTP_FILE_READ);
		  if (filesystem) {
			Serial.println(F("found!"));
			Serial.println(F("Try to update!"));

			Update.onProgress(progressCallBack);

			Update.begin(filesystem.size(), U_FS);
			Update.writeStream(filesystem);
			if (Update.end()){
			Serial.println(F("Update finished!"));
			}else{
			Serial.println(F("Update error!"));
			Serial.println(Update.getError());
			}

			filesystem.close();

			String renamed = name;
			renamed.replace(".bin", ".bak");
			if (SD.rename(name, renamed.c_str())){
			Serial.println(F("Filesystem rename succesfully!"));
			}else{
			Serial.println(F("Filesystem rename error!"));
			}
			delay(2000);

			ESP.reset();
		  }else{
			Serial.println(F("not found!"));
		  }
		  // isFilesystemUploaded = false; // not need by reset
      }
      Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
      break;
    default:
      break;
  }
};
void _transferCallback(FtpTransferOperation ftpOperation, const char* name, unsigned int transferredSize){
  switch (ftpOperation) {
    case FTP_UPLOAD_START:
      Serial.println(F("FTP: Upload start!"));
      break;
    case FTP_UPLOAD:
      Serial.printf("FTP: Upload of file %s byte %u\n", name, transferredSize);
      break;
    case FTP_TRANSFER_STOP:
      Serial.println(F("FTP: Finish transfer!"));
      break;
    case FTP_TRANSFER_ERROR:
      Serial.println(F("FTP: Transfer error!"));
      break;
    default:
      break;
  }

  /* FTP_UPLOAD_START = 0,
   * FTP_UPLOAD = 1,
   *
   * FTP_DOWNLOAD_START = 2,
   * FTP_DOWNLOAD = 3,
   *
   * FTP_TRANSFER_STOP = 4,
   * FTP_DOWNLOAD_STOP = 4,
   * FTP_UPLOAD_STOP = 4,
   *
   * FTP_TRANSFER_ERROR = 5,
   * FTP_DOWNLOAD_ERROR = 5,
   * FTP_UPLOAD_ERROR = 5
   */

  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){
    isFirmwareUploaded = true;
  }
  if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("filesystem.bin")>=0){
    isFilesystemUploaded = true;
  }
};

void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  ftpSrv.setCallback(_callback);
  ftpSrv.setTransferCallback(_transferCallback);

  Serial.print(F("Inizializing FS..."));
  if (LittleFS.begin()){
      Serial.println(F("done."));
  }else{
      Serial.println(F("fail."));
  }

  Serial.print(F("FileSystem version "));
  File versionFile = LittleFS.open(F("/version.txt"), "r");
  if (versionFile) {
	  FILESYSTEM_VERSION = versionFile.readString();
	  versionFile.close();
  }
  Serial.println(FILESYSTEM_VERSION);

  /////FTP Setup, ensure SD is started before ftp;  /////////
  if (SD.begin(SS)) {
      Serial.println("SD opened!");
      Serial.print(F("\nCurrent firmware version: "));
      Serial.println(FIRMWARE_VERSION);

      ftpSrv.begin("esp8266","esp8266");    //username, password for ftp.   (default 21, 50009 for PASV)
  }
}
void loop(void){
  ftpSrv.handleFTP();        //make sure in loop you call handleFTP()!!
 // server.handleClient();   //example if running a webserver you still need to call .handleClient();

}

The serial output when I upload the file filesystem.bin

...
Connected to reef-casa-sopra 
IP address: 192.168.1.127
Inizializing FS...done.
FileSystem version 0.1
SD opened!

Current firmware version: 0.20
FTP: Connected!
FTP: Free space change, free 1 of 1!
FTP: Free space change, free 1 of 1!
FTP: Free space change, free 1 of 1!
FTP: Upload start!
FTP: Upload of file filesystem.bin byte 1608
FTP: Upload of file filesystem.bin byte 2680
FTP: Upload of file filesystem.bin byte 3216
FTP: Upload of file filesystem.bin byte 4288
FTP: Upload of file filesystem.bin byte 4824
FTP: Upload of file filesystem.bin byte 5360
FTP: Upload of file filesystem.bin byte 6968
FTP: Upload of file filesystem.bin byte 7504
FTP: Upload of file filesystem.bin byte 8040
FTP: Upload of file filesystem.bin byte 9112
FTP: Upload of file filesystem.bin byte 9648
FTP: Upload of file filesystem.bin byte 10184

[...]

FTP: Upload of file filesystem.bin byte 2068960
FTP: Upload of file filesystem.bin byte 2069496
FTP: Upload of file filesystem.bin byte 2070568
FTP: Upload of file filesystem.bin byte 2071104
FTP: Upload of file filesystem.bin byte 2071640
FTP: Upload of file filesystem.bin byte 2072576
FTP: Finish transfer!
The uploaded Filesystem now stored in FS!

Search for Filesystem in FS..found!
Try to update!
CALLBACK:  Update process at 0 of 2072576 bytes...
CALLBACK:  Update process at 4096 of 2072576 bytes...
CALLBACK:  Update process at 8192 of 2072576 bytes...
CALLBACK:  Update process at 12288 of 2072576 bytes...
CALLBACK:  Update process at 16384 of 2072576 bytes...
CALLBACK:  Update process at 20480 of 2072576 bytes...
CALLBACK:  Update process at 24576 of 2072576 bytes...

[...]

CALLBACK:  Update process at 2052096 of 2072576 bytes...
CALLBACK:  Update process at 2056192 of 2072576 bytes...
CALLBACK:  Update process at 2060288 of 2072576 bytes...
CALLBACK:  Update process at 2064384 of 2072576 bytes...
CALLBACK:  Update process at 2068480 of 2072576 bytes...
CALLBACK:  Update process at 2072576 of 2072576 bytes...
CALLBACK:  Update process at 2072576 of 2072576 bytes...
Update finished!
Filesystem rename succesfully!

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00070bd0
~ld
 
.......
Connected to reef-casa-sopra 
IP address: 192.168.1.127
Inizializing FS...done.
FileSystem version 0.2
SD opened!

Current firmware version: 0.20

Thanks

  1. Firmware management
    1. esp8266: flash firmware binary (.bin) compiled and signed
    2. esp8266: flash firmware and filesystem binary (.bin) compiled with GUI tools
  2. OTA update with Arduino IDE
    1. esp8266 OTA update with Arduino IDE: filesystem, signed and password
  3. OTA update with Web Browser
    1. esp8266 OTA update with Web Browser: firmware, filesystem and authentication
    2. esp8266 OTA update with Web Browser: sign the firmware and HTTPS (SSL/TLS)
    3. esp8266 OTA update with Web Browser: custom web interface
  4. Self OTA uptate from HTTP server
    1. esp8266 self OTA update firmware from server
    2. esp8266 self OTA update firmware from server with version check
    3. esp8266 self OTA update in HTTPS (SSL/TLS) with trusted self signed certificate
  5. Non standard Firmware update
    1. esp8266 firmware and filesystem update from SD card
    1. esp8266 firmware and filesystem update with FTP client

Support forum

Spread the love
           

Tags:

2 Responses

  1. Rob de Voer says:
    6 November 2022 at 06:03

    Hello Renzo,

    Firstly my biggest gratitude for the impressive collection of superb articles and products.

    You are making the journey of an IoT developer a lot easier.

    I wanted to let you know that the SimpleFtpServer was super easy to add to my group of ESP32 and ESP8266 hubs and now enable simple upgrade of all of them from my development desktop.

    To upload the firmware.bin files I am using WinSCP (5.21.5) and I observed that your example code does not disconnect the ftpserver when it starts to upgrade the firmware. This means the WinSCP keeps retrying and eventually after the upgraded device is back online it uploads again… and again…and… 🙂

    I have added ftpSrv.end() as shown below:
    ====start of code=====
    case FTP_FREE_SPACE_CHANGE:
    if (isFirmwareUploaded){
    ftpSrv.end();
    Serial.println(F(“The uploaded firmware now stored in FS!”));
    ===end of code====

    Nothing major but thought you might like to know.

    Keep up the great work!!

    Dont hesitate to let me know if this is not the right way to terminate the ftp session.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Terms and Conditions - Privacy Policy latest - Privacy Policy with company name - Privacy Policy