DesignSpark Metrics | REST API and The Things Network Integration

It’s now possible to post metrics to the cloud time series database using a simple REST API and directly from LoRaWAN devices connected to The Things Network (TTN).

The DesignSpark Metrics platform is based on Prometheus and pushing measurements to the time series database previously required using the remote_write mechanism, with data serialised using protocol buffers. However, two new REST API endpoints have been created which now enable data to also be posted to the Metrics platform using simple JSON formatted messages.

Please note that the original API endpoint which uses remote_write is still in place and there is no impact on applications which use this, such as the Air Quality app which runs on ESDK hardware, and custom applications which make use of the DesignSpark.Cloud Python library.

Endpoints

The first of the two new API endpoints is suited to custom applications where you have full control over the data format. The second meanwhile is for use with LoRaWAN devices which are connected to The Things Network, where there are some minor limitations with the payload format used when configuring a webhook integration.

Basic

{
        "metrics": [
                {
                        "name":"TestName",
                        "value":"42.0",
                        "optionalLabel1":"Label1",
                        "optionalLabel2":"Label2"
                },
                {
                        "name":"TestName2",
                        "value":"32.0",
                        "optionalLabel1":"Label1",
                        "optionalLabel2":"Label2"
                }
        ],
        "labels": {
                "optionalGlobalLabel1":"Label1",
                "optionalGlobalLabel2":"Label2"
        }
}

The payload format for the basic API endpoint can be seen above. With this, we can have one or more metrics, where we specify a metric name and its value. All labels are optional and if global labels are specified, these will be added to each metric, along with any metric-specific labels.

The basic API endpoint is:

https://dscapi.designspark.io/metrics/basic

In order to use this we will also need:

• Metrics/Prometheus instance ID
• Key/token with write access for the instance

Air quality project members will have already been provided with the above credentials, while wider DesignSpark community access to the platform should be opened up soon.

A cURL command line example:

$ curl -X POST "https://https://dscapi.designspark.io/metrics/basic" -u "dsm_instance:dsm_key" -H "Content-Type: application/json" -d '{"metric":{"name":"TestMetric","value":"42.0","label1":"Label1Value","label2":"Label2Value"}}'

As we can the formatting is pretty straightforward. There are JSON libraries available for most programming languages and manually constructing a suitable JSON object is not difficult either.

The Things Network

{
  "uplink_message": {
    "decoded_payload": {
      "Bat": 3.324,
      "Distance": 1247,
      "Interrupt_flag": 0,
      "Sensor_flag": 1,
      "TempC_DS18B20": "0.00",
      "labels": {
        "testLabel1": "testLabelValue1",
        "testLabel2": "testLabelValue2"
      }
    }
  }
}

Applications provisioned on The Things Network may be configured with webhook integrations so that every time data is received via LoRaWAN device uplink, this is then posted to an external API. However, while payload formatters may be configured to process data before forwarding, there is still a basic structure which cannot be changed and hence a dedicated endpoint is needed; the data (metrics and labels) will always be wrapped in decoded_payload and uplink_message objects.

Above can be seen an example JSON object structured for the TTN API endpoint. Note that because all the metrics will be associated with a single LoRaWAN device, it is assumed that any labels set will always be global and so it’s not possible to set metric-specific labels.

The TTN API endpoint is:

https://dscapi.designspark.io/metrics/ttn

Next, we’ll walk through a complete example, from TTN configuration to visualising data.

LoRaWAN Sensor Example

We have previously written about River Level Monitoring with LoRaWAN and DesignSpark Metrics, where a distance sensor was connected to The Things Network, with integration via MQTT and a custom Python script which made use of the DesignSpark.Cloud library. This worked perfectly well but does require a server — or at least an always-on computer — to run the script which subscribes to the TTN MQTT broker and publishes data to the Metrics platform.

The benefit of this new approach being that we no longer need to have a custom Python script running 24/7 and instead it’s just a matter of The Things Network configuration.

TTN Configuration

Assuming we already have devices connected to The Things Network, there are two steps involved in configuring these to post data up to DesignSpark Metrics:

• Configure a payload formatter
• Configure a webhook integration

Payload formatter

{
  var additionalLabels = {"testLabel1":"testLabelValue1", 
    "testLabel2":"testLabelValue2"};
  var data = {};
  var len=input.bytes.length;
  var value = (input.bytes[0] << 8 | input.bytes[1]) & 0x3FFF;
  switch (input.fPort) {
    case 2:
      data.Bat = value / 1000;
      value = input.bytes[2] << 8 | input.bytes[3];
      data.Distance = value;
      if (value === 0)
      {
        data.Distance = -1;
      }
      else if(value === 20)
      {
        data.Distance = -1;
      }
      
      data.Interrupt_flag = input.bytes[4]; 
  
      value = input.bytes[5] << 8 | input.bytes[6];
      if(input.bytes[5] & 0x80)
      {
        value |= 0xFFFF0000;
      }
      
      data.TempC_DS18B20 = (value / 10).toFixed(2);//DS18B20,temperature  
      data.Sensor_flag = input.bytes[7];
      data.labels = additionalLabels;
      return {
        data:data,
      };

    default:
      return {
        errors: ["unknown FPort"]
      }
	}
}

Above can be seen a custom payload formatter which will extract measurements from the binary payload of a Dragino LDDS75-8 ultrasonic distance sensor, then return a JSON object suitably formatted for the TTN API endpoint. This will contain:

• Battery voltage
• Distance measurement
• Interrupt flag
• Sensor flag
• DS18B20 temperature sensor
• Labels

Not all of which may be used. For example, the Dragino LDDS75-8 is supplied without a DS18B20 1-wire temperature sensor, but one may be connected up if required. Note also that if the distance readings are at out of bounds, e.g. 0, a value of -1 is returned. The payload formatter is based on the default one for the LDDS75-8 which is available in the TTN device repository.

LoRaWAN devices connected to The Things Network are associated with an application and payload formatters can be configured at either the device or application level. Since we wanted to use labels which are specific to a particular device, we had to configure the payload formatter at the device level, given that labels are set by the payload formatter. However, in some cases it will make more sense to configure a payload formatter at the application level.

Webhook

Now that our sensor data is appropriately formatted we need to create a webhook integration.

Here can see that we have created a webhook called designspark-metrics (this is just an arbitrary name), set the format to JSON and configured the API base URL. We need to enable basic authentication and for the username we enter our metrics instance ID, with the key/token for the password. For Filter event data we have selected up.uplink_message.decoded_payload.

For the Enabled event types we have selected only Uplink message and set the API path for this to /metrics/ttn. This will be appended to the base URL set earlier.

So just to recap: our distance sensor will send data via LoRaWAN uplink, which will be processed via our payload formatter and labels applied to metrics, before being POSTed to the TTN API endpoint, using our DesignSpark Metrics instance ID and key to authenticate.

Grafana Configuration

When DesignSpark Metrics accounts are provisioned — e.g. as part of the Air Quality project — a Prometheus data source is provisioned within Grafana. Therefore once data is being posted up to Prometheus, it’s just a question of creating a new visualisation using this data source.

Above we can see a plot of the sensor battery voltage over a 7-day period. The fluctuations in part are likely due to temperature cycling through day and night.

Here we have plotted the sensor distance measurements. This is not a “corrected” river level reading, but rather the distance measured from sensor to water surface. However, something is not right and it appears that the payload formatter is periodically returning a reading of -1.

This particular sensor is being evaluated and there are additional factors to consider, such as its location and whether this is optimal or may be too high, or perhaps too close to a wall or the riverbank. In any case, we can easily filter out measurements returned which are below zero.

Here we have updated the PromQL query to:

min(Distance{location="tenterfields"} >0)

Meaning only return values greater than zero. Which has helped somewhat, but there are clearly still some spurious measurements. Fortunately, there are more tools available to us and next, we will consider another approach.

This time we have set the query for the green plot to:

quantile_over_time(0.7, Distance{location="tenterfields"}[24h])

Here we are using the quantile_over_time function to create a 0.7 quantile from a range of values over a 24h time period. For more details, see the excellent Grafana Labs blog post, Quick tip: How Prometheus can make visualizing noisy data easier. Meanwhile the second plot, in yellow, is of values greater than 4,000. Which is by no means ideal, but we know that this threshold is at least valid for the time period in question, since the river level did not rise by ~1m or more.

The quantile plot looks much more like what we’ve seen previously with slightly more expensive sensors. Although it would be premature to blame the sensor hardware, since as noted there could be other issues at play here. In any case, more work is required to investigate things such as sensor location and Grafana configuration, but for the purposes of this article, we can see just how easy it now is to have devices connected to The Things Network, posting data to DesignSpark Metrics.

Final Words

In this article we have taken a look at the two new REST API endpoints which are available for use with DesignSpark Metrics, making it much easier to post data from devices with custom applications and from LoRaWAN devices which are connected to The Things Network.

We’ve also taken a look at how we might address potential data issues from within Grafana. We could have modified the TTN payload formatter to simply drop out of bounds measurements, but by still recording these in Prometheus, we could decide to analyse them to look for clues. Similarly, we might be able to modify device firmware to drop or smooth out what appears to be spurious data, but by taking care of it in Grafana we still have the raw data and it may later turn out that such “noise” is in fact useful and indicative of some physical issue, e.g. foliage adjacent to a sensor.

Finally, the payload formatter shown here is quite simple and this could be expanded upon, and at least customised to provide a more optimal payload. For example, there is little point in posting successive 0 values to the Metrics platform for a temperature sensor which is not present.

  — Andrew Back