Protoson for to encode and decode payload (bytes)

[george_santiago]

Hi Renzo

Thanks for sharing your knowledge.

I use Protoson (search github for thinger-io/Protoson ) to encode and decode payload (bytes) of iot devices. It’s much faster and lighter than JSON.
Would it be feasible to use Protoson instead of strutc to perform the payload and send it through the E220?
Protoson seems to be a very interesting option to use with E32, E22 and E220.

There is the repository and a technical article with more information about Protoson:

Protoson is a library for encoding and decoding unstructured data in a binary format. Similar to JSON but fast and small. Protoson is based on some Protocol Buffers encoding techniques, but it is not compatible over the wire. However you can embed the protoson output in any Protocol Buffers bytes field. Actually you can transmit the raw bytes as you like, as it does not require Protocol Buffers itself.

Design goals
Small compiled code size. This library has been mainly designed for microcontrollers or devices with very limited resources. The whole library with encoding and decoding support takes less than 3.5KB on an Arduino. It does not require the Standard Template Library (STL).

Custom memory allocators. Protoson C++ can use different memory allocations approaches. Currently there are implemented a circular, and a dynamic memory allocator. Choose your allocation scheme according to your needs.

Small output. The output size is comparable to the well-known MessagePack. Depending on the encoded data it can be even smaller.

More than JSON. Protoson can encode binary data and other data types in the root, without explicitly requiring a parent object or array.

Integration
The single required source, pson.h file is in the src directory. All you need to do is add this file and define the memory allocator. The memory allocator must be defined only once in a single .cpp file.

#include "pson.h"
protoson::dynamic_memory_allocator alloc;
protoson::memory_allocator& protoson::pool = alloc;

Quick Example

#include "pson.h"

protoson::dynamic_memory_allocator alloc;
protoson::memory_allocator& protoson::pool = alloc;
using namespace protoson;

int main() {
    pson object;
    object["key1"] = 55;
    object["key2"] = true;
    object["key3"] = "hello";
    object["key4"] = 3.14;
}

Serialization/Deserialization
Serialization and deserialization is done over some virtual classes that provide raw methods for writing and reading bytes. In this way you can create your own wrappers to serialize/deserialize to/from multiple data sources, like memory, socket, file, etc. In the following there are two classes that allows serialization and deserialization to a memory buffer.

// Helper class for encoding to memory
// You can create your own encoders writing to file, socket, etc)
class memory_writer : public pson_encoder {
private:
    char* buffer_;

public:
    memory_writer(char *buffer) : buffer_(buffer) {
    }

protected:
    virtual void write(const void *buffer, size_t size) {
        memcpy(&buffer_[written_], buffer, size);
        pson_encoder::write(buffer, size);
    }
};

// Helper class for decoding from memory
class memory_reader : public pson_decoder {
private:
    char* buffer_;

public:
    memory_reader(char *buffer) : buffer_(buffer) {
    }

protected:
    virtual bool read(void *buffer, size_t size) {
        memcpy(buffer, &buffer_[read_], size);
        return pson_decoder::read(buffer, size);
    }
};

// Now you can use these custom classes in the following way:

// Reserve memory buffer
char memory_buffer[512];

// Create a sample object
pson object;
object["hello"] = "world";
object["value"] = 336;

// Encode
memory_writer writer(memory_buffer);
writer.encode(object);

// Decode
memory_reader reader(memory_buffer);
pson decoded_object;
reader.decode(decoded_object);

// Use the values
const char* str = decoded_object["hello"];
int value = decoded_object["value"];

Memory Allocators
In some environments with limited memory or without dynamic memory allocation can be useful to define custom memory allocators. Protoson requires memory for storing the data structure in memory, i.e., when your are building a object, or decoding it from some source. Encoding and Decoding part does not require memory itself.

Currently you can switch between two different memory allocators: circular_memory_allocator and dynamic_memory_allocator:

Use a dynamic_memory_allocator if you want to/can use dynamic memory. Internally, the dynamic memory allocator uses malloc and free.

#include "pson.h"
protoson::dynamic_memory_allocator alloc;
protoson::memory_allocator& protoson::pool = alloc;

Use a circular_memory_allocator if you want to define a static memory buffer. In this case you need to specify a buffer size that will depend on your maximum message length. Notice that the required buffer size can vary between platforms for storing the same message, depending on the pointer size (16, 32, or 64 bits), memory alignment, padding, etc.

#include "pson.h"
protoson::circular_memory_allocator<512> alloc;
protoson::memory_allocator&; protoson::pool = alloc;

[Renzo Mischianti]

Thanks for sharing George,
It’s very interesting, but I think we can get the same result by sending a binary structure like in this article.

Ebyte LoRa E220 device for Arduino, esp32 or esp8266: manage Wake On Radio and sends structured data – 5

It can be interesting to verify if It’s apply some compression system.
Bye Renzo

[Author]

Posts