TL;DR This blog entry describes how we added the ability to upload NoCAN sketches directly from the Arduino IDE over CAN bus, thanks to a newly added feature called "Pluggable Discovery." We take this opportunity to explore some aspects related to third-party hardware integration in the Arduino IDE. We hope this info is helpful to developers of other Arduino-compatible hardware that cannot be programmed over a typical serial port. This developer-oriented blog entry follows a previous one, which describes this new feature from a user perspective.

Introduction

Several years ago, with version 1.6.4, the Arduino IDE introduced a set of features that greatly facilitated support for third-party hardware: merely entering a specific URL in the IDE's preferences panel enables the IDE to automatically download all the information needed to configure and use boards made by companies and Arduino enthusiasts all over the world.

Almost half a year ago, when we launched NoCAN, our "wired" Arduino-compatible IoT platform, we naturally used these features to enable the integration of our CANZERO boards in the Arduino IDE.

One thing that remained imperfect in our Arduino integration was board programming. With a traditional Arduino, you select a "Board," a "Serial Port" and then you can upload your compiled sketch to your board by simply clicking on the "Upload" icon. Unfortunately, this approach does not apply to NoCAN IoT networks, which have very little in common with traditional boards connected through a serial link.

As illustrated below, a typical NoCAN network features:

  1. A set of Arduino-compatible CANZERO nodes, here numbered 1, 2 and 3, all connected together on a CAN bus.
  2. A Raspberry-Pi that acts as a gateway between the CAN bus and the "outside world," thanks to a dedicated HAT, the PiMaster.
  3. A PC running the Arduino IDE, that is used to control and program the nodes remotely through the gateway.

arduino-pi-nocan

Naturally, nodes 1, 2 and 3 cannot appear as serial ports visible to the Arduino IDE, and are only discoverable with our platform specific tool, nocanc.

To upload a sketch to a node in the network, you first need to export the compiled binary using the appropriate menu item in the Arduino IDE, and then manually issue a command similar to the following example, which uploads a sketch to node 1 in the network:

./nocanc upload /Users/Omzlo/Arduino/canzero_hello_world/canzero_hello_world.ino.omzlo_canzero.hex 1

We recently improved the situation by adding a web interface to NoCAN for people who are less comfortable with the command line: after exporting the compiled binary, the user can drag and drop the resulting file to an "upload" form in a web interface.

Still, we were hoping to find a solution that would enable the user to upload a sketch by clicking on the "upload" icon, just like on a good old Arduino UNO.

The Arduino IDE does feature a form of "network discovery" based on multicast DNS (mDNS), which is notably used by some ESP8266 boards. Unfortunately, while it works well for IP-based networks, it is not adaptable to other forms of networking, like CAN bus in our case.

Luckily, a couple of months ago, a solution suddenly appeared.

Pluggable discovery

In March 2019, Arduino released version 1.8.9 of their IDE that included a new feature called "Pluggable Discovery." This was the product of work lead by Paul Stoffregen to facilitate the integration of third-party platforms such as the Teensy, which do not use traditional "serial" ports for programming. We decided to see if it could be "hacked" for the benefit of our NoCAN platform.

Paul Stoffregen did a comprehensive explanation of Pluggable Discovery on his blog: check it out for the full details. The principle is simple: the Arduino IDE can now be configured to launch a user-defined "discovery" tool that continuously reports a list of available "ports" that are then automatically added to the "Ports" menu in the Arduino IDE. The user can then select one of those "ports" in the IDE, and the identity of the selected port can then be passed as a parameter to the "upload" tool when the user clicks on the "upload" icon. The great thing is that these "ports" don't have to be serial ports, or TCP/IP ports; they can be anything that is understood by the "uploader" tool. In our case, we wanted to be able to select the NoCAN address of a node in the NoCAN network, which is simply a number between 1 and 127.

We added a new command to the nocanc tool, aptly named arduino-discovery. Provided that you have correctly installed and configured nocanc on the same Linux or MacOS PC that runs your Arduino IDE, you can just type:

./nocanc arduino-discovery
START_SYNC

If you're on a Windows PC instead, replace ./nocanc with nocanc.exe. The tool responds with a set of JSON structures, one for each node in your NoCAN network, e.g.:

{
  "eventType": "add",
  "port": {
    "address": "3",
    "label": "node 3 [2b:4c:d2:e6:2f:6c:ae:44]",
    "boardName": "Omzlo CANZERO",
    "protocol": "nocan",
    "protocolLabel": "NoCAN Nodes",
    "prefs": {},
    "identificationPrefs": {}
  }
}
{
  "eventType": "add",
  "port": {
    "address": "4",
    "label": "node 4 [39:0c:65:d3:9d:c0:40:d3]",
    "boardName": "Omzlo CANZERO",
    "protocol": "nocan",
    "protocolLabel": "NoCAN Nodes",
    "prefs": {},
    "identificationPrefs": {}
  }
}

The above example corresponds to a NoCAN network with two CANZERO nodes, identified with addresses 3 and 4, respectively. (The additional numbers such as 39:0c:65:d3:9d:c0:40:d3 are global unique identifiers, like a MAC address). Hot-plugging additional nodes to the NoCAN network would cause the creation of additional messages with other addresses. The removal of a node with the most recent firmware (April 2019) will also cause the generation of a "remove" JSON event.

Now, if we configure the Arduino IDE (1.8.9) to run nocanc arduino-discovery, it makes our nodes "magically" appear in the "Port" selection menu of the Arduino IDE, as shown below.

The user can then select a node in the menu (here node 3) and its address can be passed as a parameter to the "upload" tool.

Putting everything in a nice JSON package

Now that we know how to modify the Port selection menu of the Arduino IDE, we can create a nice package that makes everything work automatically for Arduino users that want to program NoCAN hardware. The way to create such a package is described in detail in the Arduino IDE 1.6.x package_index.json format specification and the Arduino IDE 1.5 3rd party Hardware specification. A very simplified description of the mechanism that allows the Arduino IDE to be extended to support third party software is presented on the figure below just to give a bit of context to the following discussion:

When the user adds package.json in the preferences panel of the Arduino IDE, this results in the download of a set of files bundled together in an archive that contains all the necessary code and libraries needed to run the corresponding third-party hardware. In that bundle, two files play a particular role in the configuration of the Arduino IDE: boards.txt and platform.txt. The file boards.txt contains a description of the supported boards and board-specific parameters needed to program each one of them. The platform.txt file contains general recipes to compile, link, upload sketches for a specific platform. The package.json also optionally refers to one or more tools needed by the third party hardware, telling the Arduino IDE where to download these tools (e.g. avrdude, bossac, etc.)

For NoCAN, our JSON package references just one tool: nocanc, which is both used as a discovery tool (with the arduino-discovery command) and as an upload tool (with the upload command).

boards.txt

Currently, NoCAN only supports one board, the CANZERO, which features a SAMD21G18, a nice ARM-Cortex M0+ microcontroller featured in many new Arduino-compatible boards. Our boards.txt file contains various details about the CANZERO and a reference to our upload tool: 

omzlo_canzero.upload.tool=nocanc
omzlo_canzero.upload.protocol=nocan

platform.txt

We added two sections to platform.txt: one describing our sketch upload command, and the other configuring board discovery, following the newly introduced "pluggable discovery" mechanism we previously discussed.

Sketch upload is described as follows:

tools.nocanc.path={runtime.tools.nocanc.path}
tools.nocanc.cmd=nocanc
tools.nocanc.cmd.windows=nocanc.exe
tools.nocanc.upload.pattern="{path}/{cmd}" upload "{build.path}/{build.project_name}.hex" "{serial.port}"
tools.nocanc.upload.params.quiet=
tools.nocanc.upload.params.verbose=

The value tools.nocanc.upload.pattern describes the recipe that is used to build the sketch upload command. During runtime, the Arduino IDE will replace various variables enclosed in curly brackets with "real" value. In the context of the NoCAN platform:

  • {path}/{cmd} is replaced with the full path to the nocanc tool;
  • {build.path}/{build.project_name}.hex is replaced with the full path to the compiled NoCAN sketch;
  • most importantly, {serial.port} is replaced by the value of the port selected by the user in the Arduino IDE "Port" submenu.

Finally, the discovery mechanism is configured as follows in platform.txt: 

discovery.nocanc.pattern="{runtime.tools.nocanc.path}/nocanc" arduino-discovery
discovery.nocanc.pattern.windows="{runtime.tools.nocanc.path}/nocanc.exe" arduino-discovery

In the above, {runtime.tools.nocanc.path} gets replaced by the path to the nocanc tool that has been downloaded by the Arduino IDE.

Note: the discovery pattern above should typically be written similarly to the "upload" pattern described previously. However, due to a variable expansion bug, this workaround was needed.

Arduino got an awesome new feature

The work of Paul Stoffregen and the Arduino dev team on this "Pluggable Discovery" mechanism offers interesting perspectives for hardware integration in the Arduino IDE. While it was created to allow the integration of hardware that uses alternatives to "Serial USB" as a communications mechanism (e.g. USB HID), it also has great potential for "non-IP" networked-devices, such as our own CAN-bus based IoT platform. As such it could be used to deal with the following situations:

  • Automatic Bluetooth device discovery and programming.
  • "NAT-ed" IP-based devices, such as a set of ESP32's or Arduino Yúns sitting behind a firewall, which cannot be discovered with mDNS.
  • RS-485 systems and other bus connected devices.

Note that this is the first release of "Pluggable Discovery" in the Arduino IDE. Some bugs will be corrected in the next release and some things might change.

To stay updated on the NoCAN project, don't forget to follow us on Twitter or on our Facebook page.