Skip to main content

Building the Electronica 2018 Connected Stand Data Platform

Main27_7933684d09f87695d7eba14597dc91d963c0d970.jpg

Mosquitto, NodeRED, InfluxDB and Grafana are combined to provide an eminently flexible open source data platform.

The RS Components Connected Stand featured IoT Blockchain and LoRaWAN Environmental Sensor demonstrators, with data from these visualised via dashboards displayed on large monitors. In this post we take a look at how Node-RED was used to provide the “glue” and some simple processing, with InfluxDB used for data storage and then finally, Grafana for visualisation.

With both demonstrators, MQTT provided the data source. However, while the Environmental Sensors were connected via The Things Network and so it was simply a matter of connecting to its MQTT API to obtain a data feed, the IoT Blockchain demonstrator required a broker to publish to.

While it would have been possible to run the entire software stack locally, it was decided to host this in the cloud, so as to enable viewing from remote sites if so desired.

There are numerous different ways each of the software components can be installed. For example, from source code, via O/S packages or Docker containers. Rather than detail each installation step for every component, here we will just cover the configuration and the official documentation or a how-to guide should be consulted for software installation details.

Mosquitto

# Anonymous user access

topic read public/#

# Authenticated users

user app
topic #

user iotdemo
topic iotbc/

Mosquitto is an open source message broker for the MQTT pub/sub messaging protocol, which provides a lightweight means of sending messages between applications. It benefits from widespread support and easy to use libraries for a variety of programming languages.

While the IoT Blockchain demonstrators use Ethereum to provide a secure, immutable record of transactions, blockchain technology does not provide real-time processing, so MQTT was employed for dashboard integration. Hence the demonstrators would need a broker to publish to.

Mosquitto configuration is quite simple and by default, anyone can publish and subscribe to all the topics on the broker, which is certainly undesirable on an Internet connected host.

Above we can see the /etc/mosquitto/acl file which implements access control. In this we have said that anyone can subscribe to topics published under public/, which can prove useful for e.g. testing. The app user can publish to any topic, while iotdemo can only publish to topics under iotbc/. Here, app is the user which Node-RED will use, while iotdemo will be used by the demonstrators.

In order to use the ACL we need to specify it in the /etc/mosquitto/mosquitto.conf file, which also must configure a password file, where the broker will look up the user and confirm that their password is valid. With connections via the Internet it is also highly advisable to configure TLS. Details of how do do this and configure the password file etc. can be found in the documentation.

InfluxDB

        env:
          INFLUXDB_DB: 'iotdemo'
          INFLUXDB_READ_USER: 'dashboard'
          INFLUXDB_READ_USER_PASSWORD: 'secretReadPass'
          INFLUXDB_WRITE_USER: 'app'
          INFLUXDB_WRITE_USER_PASSWORD: 'secretWritePass'
          INFLUXDB_ADMIN_USER: 'admin'
          INFLUXDB_ADMIN_PASSWORD: 'secretAdminPass'
          INFLUXDB_HTTP_ENABLED: true
          INFLUXDB_HTTP_AUTH_ENABLED: true

So now we’ll jump a step and move on to InfluxDB, which will provide our data store. We’ll come on to Node-RED next, which will then provide the glue between MQTT data sources and storage.

InfluxDB is a time series database (TSDB), which means that it’s optimised for storing information which is ordered by time — such as readings from IoT devices. It uses HTTP as an interface and as with MQTT, there is fairly widespread support. InfluxDB comes with a command line interface to administer it, plus an optional web interface called Chronograf for viewing and administering data.

InfluxDB can be installed from O/S packages, amongst other methods. Above can be seen a fragment from an Ansible playbook used to deploy InfluxDB via Docker. With this mechanism we can use environment variables to set up an initial database, along with various user accounts and optional configuration. However, it’s just as easy to create users and databases post-installation.

Once again, if we wanted to access this resource via the Internet it would be wise to set up TLS. See the documentation for details of how to do this, and for configuration and use in general.

So at this point we will assume that both Mosquitto and InfluxDB have been installed, with security configured as required and as part of which some user accounts.

NodeRED

Node-RED is a tool for wiring the Internet of Things and, eminently extensible, it has grown over the years to support all manner of inputs and outputs — with core functionality evolving to become ever more sophisticated, while remaining easy to use and novice friendly.

Assuming we have installed Node-RED via O/S packages, Docker or some other method, we will also need to install a few additional “nodes” that extend it so that we can connect to The Things Network and our InfluxDB database. Once inside the data directory this can be done with:

$ npm install node-red-contrib-ttn node-red-contrib-influxdb

The location of the data directory will varying depending upon the O/S and how Node-RED was installed. Next we’ll move on to the flows which implement our application logic.

IoT Blockchain

CSDP_IoTBC_NR_Flow_9c31d3b0e3b30fb78e142517d2c795e1b312a982.jpg

Above we can see the flow for the IoT Blockchain demonstrators. Here we have an MQTT input node far left, that is subscribed to all topics under iotbc/ (hash is a wildcard). This node has been configured to use the Mosquitto broker, along with a username and password that we set up earlier.

The output is then connected to a node which converts JSON strings in the message payload to JavaScript object representation, making it that little bit easier to access values.

// Check sensor status and if good, format and post to InfluxDB
// Fields = sensor readings
// Tag = TTN DevID

if (msg.topic == 'iotbc/miner'){
    var influx = {
        measurement: 'blockchain',
        payload: [
            msg.payload,
        {   node: 'miner'
        }]
    }
} else if (msg.topic == 'iotbc/carcrash') {
    var influx = {
        measurement: 'blockchain',
        payload: [
            {impact: msg.payload},
        {   node: 'carcrash'
        }]
    }

} else {
    return;
}

return influx;

The partial contents of the Process Message function node are shown above. Here we first look for the full MQTT topic name and then process the message accordingly. With the miner we have a JSON formatted payload containing the latest Ethereum block number and how many transactions this contained. The other nodes (use cases), e.g. Car Crash and Machine Failure etc. just emit a single integer value. Above we have just shown the Miner and Car Crash unit code for simplicity.

The msg.measurement property gets used by the InfluxDB output node and this means that, rather than hard-coding a measurement in the output node, we can specify it in our application logic and it will be communicated along with the payload which is to be stored. The payload itself is comprised of an array containing two objects, with the first being a set of fields and the second tags.

What are measurements, fields and tags? A measurement describes the data that will be stored. Fields are key-value pairs and are not indexed. Tags are also key-value pairs, but they are indexed. Hence tags should be used for commonly queried metadata.

So now we have a measurement called blockchain, which has indexed tags of miner, carcrash and temperaturealert etc. At this point we could perform queries and retrieve values for the number of transactions contained in an Ethereum block, or the impact of the last car crash, for example.

LoRaWAN Environmental Sensors

CSDP_DSES_NR_Flow_527838358ce5af07946b57581151d29fa0ba628c.jpg

The environmental sensors flow is quite similar and also starts with an MQTT data source. However, rather than use an MQTT input node, this time we will use a TTN Uplink node, which makes it even easier to get data via the MQTT-based The Things Network API. With this we just needed to configure the App Id and Access Key, as provided by the TTN Console.

// Check sensor status and if good, format and post to InfluxDB
// Fields = sensor readings
// Tag = TTN DevID

if (msg.payload.status == 'good'){
    delete msg.payload.status;
    var influx = {
        measurement: 'environment',
        payload: [
            msg.payload,
        {   sensor: msg.dev_id
        }]
    };
}else{
    return;
}

return influx;

With this flow the Process Message function is particularly simple and we:

  1. Check the status is good (the sensor is not in its burn-in period)
  2. If the status is good we delete the status property, since there is no point logging it
  3. Set the InfluxDB measurement to environment
  4. Take the payload — itself a JSON object with temperature and humidity etc. key-value pairs — and use this to populate the InfluxDB fields
  5. Set the InfluxDB sensor tag to the LoRaWAN sensor/TTN device ID.

It should be pointed out that a Payload Format Decoder was previously configured via the TTN Console. This unpacks the sensor binary format sent via LoRaWAN network uplink and turns this into a much friendlier JSON object, with key-value pairs for all the various sensor readings.

At this stage we now have two measurements being logged to InfluxDB, with time series ordered data for both IoT Blockchain and Environmental Sensor demonstrators.

Grafana

Finally we get to visualising the data and for this we will use Grafana, a web based platform for analytics and monitoring, which features support for a wide range of data sources, including InfluxDB. Once again, for installation information please see the official documentation.

Once Grafana is installed a data source of type InfluxDB will need setting up and for this we just have to enter the server URL, a database name, and a username and password with read access.

IoT Blockchain

EDB_IoTBC_ce2da720338d1d945e8dfc7eaa01f062fabffa90.jpg

With the IoT Blockchain demonstrators we simply wanted to show single statistics, rather than graphs or similar. For example, the latest block number and how many transactions it holds, the Y-acceleration force of a car crash, or the time of the last reported machine failure. This was easily done using Singlestat type panels and with queries such as:

SELECT last("block") FROM "blockchain" WHERE ("node" = 'miner')

The Singlestat panel also provides various options, such as labels, font size and colour.

LoRaWAN Environmental Sensors

EDB_IndoorSensors_9554393a86109f39492d203bae3bb77d393f2c9d.jpg

For the Environmental Sensors, we actually had two dashboards: one for indoor sensors placed around the RS stand, and a second for outdoor sensors located at Oxfordshire, London and Halifax. Both were essentially the same and with the only difference being the sensors selected in queries.

SELECT last("temperatureAvg") FROM "environment" WHERE ("sensor" = 'sens021') AND $timeFilter GROUP BY time($__interval) fill(linear)

The above query retrieves the period average temperature logged for a sensor. These queries are then simply repeated for temperature max, plus humidity and air pressure average and max etc. for each sensor. Each panel plotting measurements of the same type from different sensors.

Summing up

So as we’ve seen we can quickly assemble an IoT data platform complete with processing, storage and visualisation, using powerful free and open source software. While our use cases were relatively simple, with a small number of devices and only very basic processing, it would equally be possible to create much more complex dashboards, supporting many more devices and with far more advanced processing of data.

Andrew Back

Open source (hardware and software!) advocate, Treasurer and Director of the Free and Open Source Silicon Foundation, organiser of Wuthering Bytes technology festival and founder of the Open Source Hardware User Group.
DesignSpark Electrical Logolinkedin