Program a Connected Device Using Raft, Part 2

Before we add functionality to the basic Raft app that we started in the first part of this post, let’s have a poke around in some of the files that were generated by the Raft CLI scaffolding process and understand how the elements of a Raft app fit together. The source code is available on Github – see the Releases for the code at each step if you want to follow along.

Application Folder Structure

The folder and file structure created looked like this:

Let’s take a look at the way SysTypes work in Raft and the contents of the systypes folder.

SysTypes

SysTypes allow a developer to define different targets and configurations for the application being built. For instance, you may want to target an application at three different hardware platforms (e.g. two different ESP32 development boards – one with external RAM and one without – and an ESP32-S3) and, in this case, you would create three different systypes – each with their own name (defined by the folder name) – containing the files which define each SysType.

When you build a Raft app you can specify a particular SysType to be built (if there is more than one) using the -s option. If there is only one SysType then there is no need to specify it.

The configuration files inside the folder for the specific SysType are as follows:

• SysTypes.json – a JSON document which is used by the Raft framework to provide configuration to all system modules (SysMods)
• features.cmake – defines the target processor type and other compiler definitions that might be needed
• partitions.csv – this is a standard ESP IDF file which determines the layout of the flash memory on the device
• sdkconfig.defaults – this is a standard ESP IDF file which contains configuration options such as the use of external RAM, WiFi and BLE packages, etc

In addition:

• a folder called FSImage can contain files that the developer wants to be placed in the file-system image (this is generally in the Common folder so that it applies to all SysTypes but can differ for each SysType as required)
• a folder called WebUI which can contain a web application that is built and flashed to the file-system as part of the build process

SysTypes.json

This is the key file that is used in Raft to define configuration for SysMods. Since the file is rather long I have split the discussion of it into sections:

{
    "SysTypeName": "SysTypeMain",
    "CmdsAtStart": "",
    "SysManager": {
        "monitorPeriodMs":10000,
        "reportList":[
            "NetMan",
            "SysMan"
        ],
        "slowSysModMs": 50
    },
    "ProtExchg": {
        "RICSerial":{
            "FrameBound":"0xE7",
            "CtrlEscape":"0xD7"
        }
    },
...

• SysTypeName defines the name of this SysType – it should be identical to the name of the folder containing the SysType
• CmdsAtStart allows REST API commands to be listed (comma delimited) and these will be performed when the system starts up
• The SysManager section defines the configuration of the top-level Raft manager object.
  • The monitorPeriodMs value determines the period for gathering and reporting statistics on performance and these are presented on the debug terminal as a “heartbeat” status update.
  • The reportList enables reporting on additional SysMods (in the “heartbeat” status update message)
  • The slowSysModMs value determines the threshold for reporting that a SysMod is “slow” – this is a useful diagnostic that can pinpoint sluggish code
• The ProtExchg section configuration of the Protocol-Exchange which is a module that handles messaging between different interfaces in a Raft application. RICSerial is a protocol based on HDLC / SLIP and these settings define the boundary values for frames, etc.

    "NetMan": {
        "wifiSTAEn": 1,
        "wifiAPEn": 1,
        "ethEn": 0,
        "wifiSSID": "",
        "wifiPW": "",
        "wifiSTAScanThreshold": "OPEN",
        "wifiAPSSID": "RaftAP",
        "wifiAPPW": "raftpassword",
        "wifiAPChannel": 1,
        "wifiAPMaxConn": 4,
        "wifiAPAuthMode": "WPA2_PSK",
        "NTPServer": "pool.ntp.org",
        "timezone": "UTC",
        "logLevel": "D"
    },
    "ESPOTAUpdate": {
        "enable": 1,
        "OTADirect": 1
    },

The NetMan section is the first one which defines a conventional SysMod. NetMan (or Network Manager) handles WiFi and ethernet connectivity.

• Values for wifiSTAEn (enable WiFi station mode), wifiSSID, wifiPW and wifiSTAScanThreshold relate to station mode
• Values for wifiAPEn (enable WiFi access point mode), wifiAPSSID, wifiAPPW, wifiAPChannel, wifiAPMaxConn and wifiAPAuthMode relate to access point mode
• ethEn should be 1 (or true) to enable ethernet operation – note that settings in the sdkconfig.default file need to be added to set up the hardware correctly for ethernet to work
• NTPServer and timezone are used for time-synchronisation – set these appropriately depending on your timezone and requirements
• logLevel is setting which can be used with all SysMod configurations to set the log-level on a local basis

The ESPOTAUpdate section is used to control the availability of over-the-air updates.

• enable should be set to 1 (or true) to enable OTA update
• when OTADirect is set to 1 (or true) an OTA update can be “pushed” to the device – otherwise it needs to be started from the device and involve a server which contains the updated firmware

    "MQTTMan": {
        "enable": 0,
        "brokerHostname": "mqttbroker",
        "brokerPort": 1883,
        "clientID": "",
        "topics": [
            {
                "name": "examplein",
                "inbound": 1,
                "path": "example/in",
                "qos": 1
            },
            {
                "name": "exampleout",
                "inbound": 0,
                "path": "example/out",
                "qos": 1
            }
        ]
    },   

The MQTTMan section is used to configure MQTT support. MQTT is a publish-subscribe (pub-sub) protocol commonly used to enable IoT and other devices to send information to another computer (often a server) in the cloud or locally. Raft supports MQTT natively through the built-in publishing mechanism.

• enable (set to 1 or true to enable MQTT)
• brokerHostname and brokerPort are the hostname (or IP address) and port of the MQTT broker (see Wikipedia article for further explanation)
• clientID is an ID that should be unique on the broker – it is used to differentiate traffic from different clients – if this is left blank the system name will be used – a unique string (based on the ESP32 MAC address) will be appended to this automatically to avoid conflicts
• the topics section is used to define the topics that are published and subscribed to:
  • name refers to the source interface for information to be published or the destination interface for information to be subscribed to – interfaces refer to the communications channels inside a Raft application that are managed by the Protocol Exchange – see the Publish section below for more information on publishing interfaces
  • inbound is 1 (or true) for inbound (subscribed) topics
  • path is the topic name on the broker
  • qos is the QoS as defined by MQTT

    "LogManager": {
        "enable": 0,
        "logDests": [
            {
                "enable": false,
                "type": "Papertrail",
                "host": "xxxxx.papertrailapp.com",
                "port": 12345
            }
        ]
    },

LogManager enables logging to destinations other than the serial diagnostics port.

• enable (if set to 1 or true then logging to external destination is enabled)
• logDests is a list of destinations
  • enable
  • type – currently only SolarWinds Papertrail is supported
  • host – this is specified by Papertrail when you sign-up
  • port – as above

"SerialConsole": {
        "enable": 1,
        "uartNum": 0,
        "rxBuf": 5000,
        "txBuf": 1500,
        "crlfOnTx": 1,
        "protocol": "RICSerial",
        "logLevel": "D"
    },

Settings for the serial debug terminal:

• enable – 1 or true to enable
• uartNum – specifies which ESP32 device UART to use
• rxBuf – size of receive buffer in bytes
• txBuf – size of transmit buffer in bytes
• crlfOnTx – send a CR + LF sequence on transmit at the end of each line
• protocol – since serial can be used as a comms channel (in addition to diagnostics) this defines the framing protocol used (RICSerial is based on HDLC / SLIP)

"WebServer": {
        "enable": 1,
        "webServerPort": 80,
        "stdRespHeaders": [
            "Access-Control-Allow-Origin: *"
        ],
        "apiPrefix": "api/",
        "fileServer": 1,
        "staticFilePaths": "",
        "numConnSlots": 12,
        "clearPendingMs": 50,
        "websockets": [
            {
                "pfix": "ws",
                "pcol": "RICSerial",
                "maxConn": 4,
                "txQueueMax": 20,
                "pingMs": 30000
            }
        ],
        "logLevel": "D",
        "sendMax": 5000,
        "taskCore": 0,
        "taskStack": 5000,
        "taskPriority": 9
    },

The Raft web-server supports static pages, REST APIs, WebSockets and many other things.

• enable – set to 1 (or true) to enable
• webServerPort – the port for HTTP
• stdRespHeaders – include these headers in all responses – Access-Control-Allow-Origin avoids CORS issues
• apiPrefix – the prefix required before REST API urls
• fileServer – 1 or true to enable file serving over HTTP
• staticFilePaths – on file systems that support it this allows a sub-folder to be used for web files
• numConnSlots – maximum number of concurrent web connections
• clearPendingMs – if a socket connection is marked to be closed then wait this long before closing it – this can improve performance in some cases
• webSockets settings are a list of websocket endpoints and these can handle differing protocols
  • pfix – the name of the websocket channel
  • pcol – the protocol used on the websocket – may be JSON or RICSerial (which is a framed protocol based on HDLC / SLIP)
  • maxConn – maximum number of concurrent websocket connections of this type
  • txQueueMax – maximum number of queued frames waiting to be sent
  • pingMs – the time between sending pings
• sendMax – the maximum size of an IP payload sent by the web server
• taskCore, taskStack and taskPriority – the web server socket listener runs in its own FreeRTOS task and these settings define the task configuration

    "FileManager": {
        "LocalFsDefault": "littlefs",
        "LocalFSFormatIfCorrupt": 1,
        "CacheFileSysInfo": 0,
        "SDEnabled": 0,
        "DefaultSD": 1,
        "SDMOSI": 15,
        "SDMISO": 4,
        "SDCLK": 14,
        "SDCS": 13
    },

The FileManager SysMod handles the file system(s) which may be spiffs, littlefs and/or SD card.

• LocalFsDefault – defines the default file-system
• LocalFSFormatIfCorrupt – if 1 (or true) and the file system can’t be opened then it will be formatted
• CacheFileSysInfo – if 1 (or true) then cache file system contents – this can speed up file access but requires more RAM
• SDEnabled – 1 or true to enable SD card support
• DefaultSD – 1 or true to default to using the SD card as the primary file system
• SDMOSI, SDMISO, SDCLK, SDCS are the pin numbers for the SD card supporting hardware

    "Publish": {
        "enable": 1,
        "pubList": []
    },

Publish is the module responsible for collecting and publishing information from other SysMods.

• enable – 1 or true to enable
• pubList – is empty in the default configuration here

To explain the Publish configuration a little further, here’s a Publish section from an application that includes publishing:

    "Publish": {
        "enable": 1,
        "pubList": [
            {
                "name": "devices",
                "trigger": "Time",
                "msgID": "devices",
                "rates": [
                    {
                        "if": "BLE",
                        "protocol": "RICSerial",
                        "rateHz": 10.0
                    },                    
                    {
                        "if": "scaderOut",
                        "protocol": "RICJSON",
                        "rateHz": 0.01
                    }
                ]
            }
        ]
    },

In this example:

• name – must be unique amongst Publish records and is used when subscribing, etc
• trigger – can be either “change” or “time” – “change” means that publishing of values will happen only when a change occurs and “time” means publishing will occur on either a change of value or at a time interval defined in the “rates” record for the specific interface where data is published
• msgID – state changes are detected using callback functions and these functions are registered in code which uses the msgID as the key to indicate which pubList item it is the callback for
• rates define the protocols and rates used to communication on different interfaces
  • if – this is the name of the interface that publishing will be on – valid names are
    • BLE – the bluetooth-low-energy interface
    • the name of an MQTT topic record (see MQTT section above)
    • the name of a web-socket pfix (see Web Server section above)
  • protocol
    • RICJSON – this is a plain-text protocol containing JSON data
    • RICSerial – a framed protocol based on HDLC / SLIP
  • rateHz – the rate of publishing

    "AppSysMod": {
        "exampleGroup": {
            "exampleKey": "Welcome to Raft!"
        }
    }
}

Finally we get to the config information for the SysMod that we added when we created the app. The name AppSysMod must match the name that the SysMod is registered with (see Part 3 for more details). The contents of this record are entirely up to the developer and any information such as GPIO numbers related to devices, identification strings, etc can be stored and retrieved from the app.

The exampleGroup/exampleKey value is retrieved in the example code generated by Raft CLI and displayed on the serial terminal on start-up as a simple example of how configuration works (see the end of this post for the code which does this).

features.cmake

This file specifies:

• the target processor
• system version string
• Raft components and versions to be included in the app
• File system type and path to source files
• Web UI (optional)

# Set the target Espressif chip
set(IDF_TARGET "esp32s3")

# System version
add_compile_definitions(SYSTEM_VERSION="1.0.0")

# Raft components
set(RAFT_COMPONENTS
    RaftSysMods@main
    RaftWebServer@main
)

# File system
set(FS_TYPE "littlefs")
set(FS_IMAGE_PATH "../Common/FSImage")

# Web UI

# Uncomment the "set" line below if you want to use the web UI
# This assumes an app is built using npm run build
# it also assumes that the web app is built into a folder called "dist" in the UI_SOURCE_PATH
# set(UI_SOURCE_PATH "../Common/WebUI")

# Uncomment the following line if you do NOT want to gzip the web UI
# set(WEB_UI_GEN_FLAGS ${WEB_UI_GEN_FLAGS} --nogzip)

# Uncomment the following line to include a source map for the web UI - this will increase the size of the web UI
# set(WEB_UI_GEN_FLAGS ${WEB_UI_GEN_FLAGS} --incmap)

Valid IDF_TARGET values are esp32, esp32s3, esp32c3

The SYSTEM_VERSION must be in semver format

RAFT_COMPONENTS is list and each item specifies a library in the Raft framework. Specific git tags can be specified after the @

The FS_TYPE can be littlefs or spiffs and the path specified can either be in the Common folder or specific to the SysType

If a UI_SOURCE_PATH is specified then the folder should contain a web app that can be built using the “npm run build” command

Optionally generated Web UI files can be gzipped to save space

Optionally the source-map files generated with the web ui can be retained (normally these are large and may not fit on the file-system)

partitions.csv

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x009000,  0x015000,
otadata,  data, ota,     0x01e000,  0x002000,
app0,     app,  ota_0,   0x020000,  0x1b0000,
app1,     app,  ota_1,   0x1d0000,  0x1b0000,
fs,       data, 0x83,    0x380000,  0x080000,

The format and contents of this file are described in the ESP32 documentation. The default definition is for a 4MB device and includes an 84Kb non-volatile storage area, two app partitions of 1.7MB each (enabling OTA updates on quite a large application) and a small file-system of 0.5MB

sdkconfig.defaults

The use of the sdkconfig.defaults file is specified in the ESP IDF documentation and avoids the use of the menuconfig mechanism which is normal for other ESP IDF apps. Only settings that differ from the defaults need to be specified.

A full list of settings and their meaning is available in the ESP IDF documentation.

Here are the standard settings for Raft:

# Define configuration
# Remove/repace these comments to set level to debug/info

# Default log level
CONFIG_LOG_DEFAULT_LEVEL_DEBUG=n

# Serial Baud-Rate
CONFIG_ESP_CONSOLE_UART_BAUDRATE=115200
CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG=y
CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG=n

# Flash size
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y

# Partition Table
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="systypes/SysTypeMain/partitions.csv"

# Ethernet
CONFIG_ETH_USE_ESP32_EMAC=n
CONFIG_ETH_USE_OPENETH=n
CONFIG_ETH_USE_SPI_ETHERNET=n

# Common ESP-related
CONFIG_ESP_MAIN_TASK_STACK_SIZE=10000

# FreeRTOS
CONFIG_FREERTOS_HZ=1000

# SPIRAM
CONFIG_SPIRAM=n

# TLS
CONFIG_ESP_TLS_SERVER=y

The settings starting CONFIG_ESP_CONSOLE set up the debug serial terminal and the last two are required on the ESP32-S3 (and possibly C3?) to ensure that communication is two-way.

The flash size can be set by changing the CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y line to indicate 2MB or 8MB, etc.

Ethernet support is normally disabled but can be enabled (on hardware that supports it) as described in the ESP IDF documentation

The size of the main stack is something you might need to adjust if you want to optimize memory usage or are running out of stack space due to recursion, etc.

The FreeRTOS tick period is conventionally set to 10ms but Raft reduces this to 1ms as this makes fine-grained scheduling easier and on an ESP32 which has reasonably powerful CPU cores there isn’t much downside in this.

SPIRAM (external RAM connected over SPI bus – sometimes called PSRAM) is disabled by default – change this to CONFIG_SPIRAM=y to enable it.

TLS support is in development at the time of writing (May 2024).

Folder main

The folder named main contains the startup code for the application.

////////////////////////////////////////////////////////////////////////////////
// Main entry point
////////////////////////////////////////////////////////////////////////////////

#include "RaftCoreApp.h"
#include "RegisterSysMods.h"
#include "RegisterWebServer.h"
#include "AppSysMod.h"

// Entry point
extern "C" void app_main(void)
{
    RaftCoreApp raftCoreApp;
    
    // Register SysMods from RaftSysMods library
    RegisterSysMods::registerSysMods(raftCoreApp.getSysManager());

    // Register WebServer from RaftWebServer library
    RegisterSysMods::registerWebServer(raftCoreApp.getSysManager());

    // Register app
    raftCoreApp.registerSysMod("AppSysMod", AppSysMod::create, true);

    // Loop forever
    while (1)
    {
        // Yield for 1 tick
        vTaskDelay(1);

        // Loop the app
        raftCoreApp.loop();
    }
}

This is the code generated by Raft CLI

• app_main() is the startup function called when an ESP32 app begins
• RaftCoreApp is the class which handles the core functionality of a Raft app and raftCoreApp is the single instance of that class.
• All SysMods must be registered with the SysManager object and the next three lines register
  • The SysMods contained in the RaftSysMods library (which includes BLE, NetworkManager, StatePublisher and most other common Raft functionality)
  • The Raft Web Server library SysMod
  • The AppSysMod that was created by the Raft CLI scaffolding process
• Finally the app loops forever
  • Yielding on every loop to give other tasks an opportunity to run (if they are ready to do so)
  • Calling the raftCoreApp.loop() function which calls all of the registered (and enabled) SysMods’ loop() functions

Folder AppSysMod

This folder contains the SysMod created by Raft CLI scaffolding process and it should be extended by the developer to implement application specific functionality.

////////////////////////////////////////////////////////////////////////////////
// AppSysMod.h
////////////////////////////////////////////////////////////////////////////////

#pragma once

#include "RaftArduino.h"
#include "RaftSysMod.h"

class AppSysMod : public RaftSysMod
{
public:
    AppSysMod(const char *pModuleName, RaftJsonIF& sysConfig);
    virtual ~AppSysMod();

    // Create function (for use by SysManager factory)
    static RaftSysMod* create(const char* pModuleName, RaftJsonIF& sysConfig)
    {
        return new AppSysMod(pModuleName, sysConfig);
    }

protected:
    // Setup
    virtual void setup() override final;

    // Loop (called frequently)
    virtual void loop() override final;

private:
    // Example of how to control loop rate
    uint32_t _lastLoopMs = 0;
};

• Every SysMod must have a create() function which is used by the SysManager factory to create instances of the SysMod.
• The setup() function works exactly the same way setup() works in an Arduino app
• The loop() function also works as it does in Arduino
• In this example a variable _lastLoopMs is used as a loop timer

////////////////////////////////////////////////////////////////////////////////
// AppSysMod.h
////////////////////////////////////////////////////////////////////////////////

#include "AppSysMod.h"
#include "RaftUtils.h"

static const char *MODULE_PREFIX = "AppSysMod";

AppSysMod::AppSysMod(const char *pModuleName, RaftJsonIF& sysConfig)
    : RaftSysMod(pModuleName, sysConfig)
{
    // This code is executed when the system module is created
    // ...
}

AppSysMod::~AppSysMod()
{
    // This code is executed when the system module is destroyed
    // ...
}

void AppSysMod::setup()
{
    // The following code is an example of how to use the config object to
    // get a parameter from SysType (JSON) file for this system module
    // Replace this with your own setup code
    String configValue = config.getString("exampleGroup/exampleKey", "This Should Not Happen!");
    LOG_I(MODULE_PREFIX, "%s", configValue.c_str());
}

void AppSysMod::loop()
{
    // Check for loop rate
    if (Raft::isTimeout(millis(), _lastLoopMs, 1000))
    {
        // Update last loop time
        _lastLoopMs = millis();

        // Put some code here that will be executed once per second
        // ...
    }
}

The comments in this file should help with understanding what is going on here:

• The constructor and destructor can be used to allocate and deallocate resources (such as FreeRTOS semaphores) that might be needed by the application
• The setup() function is called once when the application starts and the example code:
  • extracts a string value from the configuration using the path “exampleGroup/exampleKey” – see the final section of the SysTypes.json file above to see how this relates to the JSON content
  • Logs the string value out to the diagnostic terminal – this will display “Welcome to Raft!” as that is what is in the JSON config
• The example also shows how to create a timer-based loop section which will execute once per second.

Build files

The remaining files are related to the build process for Raft apps. Since Raft is using the ESP IDF build system (which itself is based on CMake) the files CMakeLists.txt appear in the root folder and in each component folder.

# Raft Project
cmake_minimum_required(VERSION 3.16)
include(FetchContent)

# Fetch the RaftCore library
FetchContent_Declare(
    raftcore
    SOURCE_DIR RaftCore
    GIT_REPOSITORY https://github.com/robdobsn/RaftCore.git
    GIT_TAG        main
)
FetchContent_Populate(raftcore)
set(ADDED_PROJECT_DEPENDENCIES ${ADDED_PROJECT_DEPENDENCIES} "raftcore")
set(EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS} ${raftcore_SOURCE_DIR})

# Include the Raft CMake
include(${raftcore_SOURCE_DIR}/scripts/RaftProject.cmake)

# Define the project dependencies
project(${_build_config_name} DEPENDS ${ADDED_PROJECT_DEPENDENCIES})

# Generate File System image
include(${raftcore_SOURCE_DIR}/scripts/RaftGenFSImage.cmake)

To target a specific version of the RaftCore library, change the GIT_TAG on line 10 to a valid tag or hash in the GitHub repo for RaftCore.

Each component folder should have a CMakeLists.txt file similar to this:

idf_component_register(
    SRCS 
        "AppSysMod.cpp"
    INCLUDE_DIRS
        "."
    REQUIRES
        RaftCore
)

The Dockerfile and compose.yaml files relate to using Docker to build the Raft application. If you want to subsequently change the ESP IDF version when using Docker then change the version tag on line 1 of the Dockerfile.

FROM espressif/idf:v5.2.1
WORKDIR /project
# Install dependencies required for Node.js install
RUN apt-get update && apt-get install -y curl software-properties-common && \
    curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
    apt-get update && apt-get install -y nodejs g++
# Verify the installation of the specific Node.js version
RUN node -v && npm -v
# Configure Git to recognize /project as a safe directory
RUN git config --global --add safe.directory /project