Secure IoT networks with DEBIX Infinity

DEBIX Infinity (282-4027) is the latest SBC offering from Polyhex. It’s a high-quality board based on NXP’s i.MX 8M Plus Quad Core SoC with extended temperature range and dual Gigabit Ethernet RJ45 jacks, one of which supports Time Sensitive Networking (TNS). It also provides a host of other useful interfaces in a credit card format: https://www.rs-online.com/designspark/debix-infinity-power-performance-precision

Available in 2/4 GB RAM and none/32 GB eMMC options, the board is aimed at smart robots, Industry 4.0, edge computing, gateways, IoT, and security applications. 

Polyhex provides several industrial-grade, reliable and secure operating systems with long term support for Ubuntu, Android, Yocto and Windows IoT Enterprise. https://debix.io/Software/index.html

Securing IoT networks is becoming increasingly important due to the reliance being placed on these devices. More and more businesses and individuals are falling foul to hacking and ransomware attacks, with IoT devices providing a potential unauthorised entry point to the network. 

One of the issues is many IoT devices are not capable of securing their network connections and others have poor or out-of-date security features that may be open to compromise. A solution is to run IoT devices on their own network zone, entirely isolated from the main LAN. This limits the blast radius of any possible compromise. So how can these devices running MQTT protocol communicate securely with the outside world?

This article demonstrates running a secure MQTT broker on DEBIX Infinity using its dual network interfaces to allow communication between the main LAN and the IoT network. This configuration has the advantage that all MQTT clients from the external facing network can only access the Broker using TLS/SSL with Certificates but messages can pass securely through the Broker to clients on the IoT network, without additional firewall rules.

We cover flashing the eMMC on the DEBIX Infinity with Ubuntu, configuring the network interfaces, installation and secure configuration Mosquitto MQTT broker hosted in Docker and testing network security effectiveness with Wireshark.

Typical applications for this configuration are in factory automation where external access is required or in SOHO and home networks running consumer-grade IoT devices.

Difficulty: Medium

Time: 4 Hrs

Credit: None

Licence: None

 

Parts Needed:

Shopping List

Essentials for this project are the DEBIX Infinity board, with monitor, keyboard and mouse attached. Your network needs to be configured to host 2 separate subnets, one for secure connections and the other for IoT devices. For testing ideally you will have 2 Linux boxes that run MQTT clients, one on each of the networks.

We used the 4GB RAM / 32GB eMMC board variant for the project but other versions are available. We also attached the DEBIX IO Board as it has an onboard USB-to-Serial connector, handy for debugging.

The Ubuntu OS image comes in a desktop version so for the display we added the OKdo 15.6in LCD monitor which is very comfortable to use. This monitor comes with a full size HDMI connector cable which fits the DEBIX Infinity HDMI port. 

5in and 10in dedicated TFT displays are also available for embedded use. 

Power supplies for the board and monitor are separate, both are 5V / 3A via USB-C connectors.

A good quality microSD card is required to flash the OS.

Flashing eMMC

Installing the OS to eMMC storage is done by flashing a microSD card with a specially prepared system image. DEBIX OS images are also available for booting directly from microSD cards so ensure you choose the correct version. The board has a set of DIP switches which set the different Boot Modes.

We used the Ubuntu image for this installation.

• Click the link below to download the Ubuntu OS image marked Boot from eMMC

https://debix.io/Software/downloadn.html 

Flash to SD card with balenaEtcher and insert it in the Infinity:

• Connect keyboard, mouse and HDMI monitor
• Set the boot DIP switches to Micro SD boot mode: SW1 -> OFF / SW2 -> ON / SW3 -> ON
• Power on the Infinity with a USB-C 5VDC/3A power adapter.

After the system has booted to the DEBIX boot screen, the power LED will come on Purple. The Green activity LED will flash quickly to indicate the eMMC is being flashed. This takes several minutes.

• Wait for the system to display the login prompt but at this point you can’t login!
• Disconnect the power and remove the SD card.
• Now reset the DIP switches to boot from eMMC: SW1 -> OFF / SW2 -> ON / SW3 -> OFF

Note: The first boot takes several minutes to complete so be patient!

Power on and boot from eMMC. The display will be blank for several minutes.

Login with default credentials:

Username: debix

Password: debix

Networking

Here’s the example network diagram showing two network zones, 192.168.10.0/24 for secure devices and 10.0.10.0/24 for IoT devices. Devices running in each zone can only communicate with their peers and not across subnets.

We don’t cover how to set this up, as it depends on your own network, but if you want to experiment look at the OpenWrt project which has builds for many different devices like ROCK CM3, Raspberry Pi and many different router brands.

In the example, the DEBIX Infinity interfaces are configured with static addresses at 192.168.10.113 for TLS encrypted MQTT traffic on Port 8883 and 10.0.10.116 for unencrypted traffic on Port 1183. Only the secure network zone has direct internet access.

DEBIX Ubuntu comes with NetworkManager which allows GUI configuration of the network interfaces but we will show the use of Netplan as recommended by Ubuntu:

https://ubuntu.com/server/docs/configuring-networks#static-ip-address-assignment

Open a Terminal on the DEBIX Infinity and create a new netplan configuration:

Use the ip command to view the available network devices and how they are configured. On our Infinity the Ethernet devices available are ens33 & ens34:

$ ip a
..
2: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether AB:CD:EF:00:00:01 brd ff:ff:ff:ff:ff:ff
…
3: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether AB:CD:EF:00:00:02 brd ff:ff:ff:ff:ff:ff

Create a basic netplan static configuration for ens33 - this will be on the main secure subnet:

sudo netplan set --origin-hint 02-ens33-interface \
ethernets.ens33.addresses=[192.168.10.113/24]

This generates the configuration /etc/netplan/02-ens33-interface.yaml

Now edit the file as sudo and add the routes and nameservers settings for the network on the ens33 interface, so they look similar to this:

sudo vi /etc/netplan/02-ens33-interface.yaml 
network:
  version: 2
  ethernets:
    ens33:
      addresses:
      - "192.168.10.113/24"
      routes:
      - to: default
        via: "192.168.10.10"
      nameservers:
        search: [lan]
        addresses: [192.168.10.10]

Save the edits, then run netplan with the try command and it will check your yaml (which is very picky!) and If you get the syntax wrong it will tell you.

sudo netplan try

If it's all good, press ENTER to keep the configuration. If not then let the command time out, then go back and amend your changes.

Now create a second yaml for the ens34 device - this will be on the network for the IoT devices:

sudo netplan set --origin-hint 03-ens34-interface \
ethernets.ens34.addresses=[10.0.10.116/24]

This interface doesn't need a gateway and nameserver as it isn't connected to any other network.

Once these settings are in place you can check the status of your network interfaces with the following netplan command:

netplan status

To make things doubly difficult for an attacker we can restrict SSH access to the DEBIX Infinity from the IoT zone by adding rules to the hosts files, only allowing clients on the secure network to connect:

sudo vi /etc/hosts.allow
# /etc/hosts.allow: list of hosts that are allowed to access the system.
sshd : 192.168.10.

Tip: Note the unusual network notation, the last dot is required

sudo vi /etc/hosts.deny
# /etc/hosts.deny: list of hosts that are _not_ allowed to access the system.
ALL : ALL

Now you need to be either directly connected to a terminal on the DEBIX Infinity or from a host on the 192.168.10.0/24 subnet to gain SSH access.

Network Testing

You should now be able to ping the DEBIX Infinity on 192.168.10.113 from a host on the same network:

ping 192.168.10.113
PING 192.168.10.113 (192.168.10.113) 56(84) bytes of data.
64 bytes from 192.168.10.113: icmp_seq=1 ttl=64 time=0.658 ms

But not from a host on the IoT network (10.0.10.0/24) - it should be Unreachable.

ping 192.168.10.113
PING 192.168.10.113 (192.168.10.113) 56(84) bytes of data.
From 10.0.10.10 icmp_seq=1 Destination Port Unreachable

SSH access to the DEBIX Infinity from the IoT network should also be blocked.

ssh debix@imx8mp-debix
kex_exchange_identification: read: Connection reset by peer
Connection reset by 10.0.10.116 port 22

This tests our DEBIX Infinity interfaces are correctly configured and that security is nice and locked down.

DEBIX Infinity with IO Module showing Dual Network Interfaces:

Docker

We need to install Docker on the DEBIX Infinity to host the Mosquitto MQTT Broker.

It’s best practice to follow the documentation on the Docker site to Install Docker Engine on Ubuntu. Use the Manual Install method and we did the non-privileged user setup, which is good for testing only.

Tip: I like to cut and paste the commands into an SSH session from my PC.

Make sure to test that Docker works with the following command:

docker run hello-world
Hello from Docker!

This message shows that your installation appears to be working correctly.

Mosquitto

Mosquitto will run in a Docker Container. This is a very secure way to run it, as the Official Docker Image is prepared by the project owners, Cedelo at the Eclipse Project.

It also allows you to easily put up and tear down of instances for testing and evaluation purposes without impacting the underlying host OS. This keeps installations clean and secure - once you start using it you won’t want to go back!

https://hub.docker.com/_/eclipse-mosquitto

I’m going to be using the raw docker commands in this article but they can be automated for production systems with additional tools like docker-compose, Kubernetes, Podman, and Portainer.

Cedelo have produced a very detailed document on how to configure Mosquitto in a Docker Container for all the nitty-gritty, so I will only cover the essentials:

https://cedalo.com/blog/mosquitto-docker-configuration-ultimate-guide/

Anonymous Access

Before we start to lock down access to the broker, I like to start off by configuring it for Anonymous Access which is in-secure, but proves that basic functionality is working and makes testing and debugging easier.

Mosquitto configuration is done in the /etc/mosquitto/conf.d directory by adding files with a .conf extension. Don’t modify the /etc/mosquitto/mosquitto.conf file as that can be changed during upgrades.

Start by creating the file /etc/mosquitto/conf.d/broker.conf

sudo vi /etc/mosquitto/conf.d/broker.conf
listener 1883
allow_anonymous true 

Now run Docker with the following to get the image and start a container:

docker run -it -d --name mosquitto -p 10.0.10.116:1883:1883 \
-v /etc/mosquitto/conf.d/broker.conf:/mosquitto/config/mosquitto.conf \
eclipse-mosquitto:2

This will pull the latest version 2 eclipse-mosquitto image onto the host. It will then run the container with a name mosquitto mapping the host port 1883 on 10.0.10.116 (ens33) to the docker container port 1883.

It maps the host configuration broker.conf to /mosquitto/config/mosquitto.conf 

Tip: You must use this exact path and filename in order for the docker container to work

The container named mosquitto should now be running as a daemon in the DEBIX Infinity host. You can check this by running the docker ps command:

docker ps -a
CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS                   PORTS                        NAMES
1d32db15d52f   eclipse-mosquitto:2   "/docker-entrypoint.…"   2 minutes ago   Up 2 minutes             10.0.10.116:1883->1883/tcp   mosquitto
a69fcc98dad7   hello-world           "/hello"                 11 days ago     Exited (0) 11 days ago                                great_leavitt

You should now be able to send MQTT messages from any host on the 10.0.10.0/24 network without authentication. Test it with mosquitto_pub and mosquitto_sub sessions on a connected device like any of the ROCK boards or a Raspberry Pi. You can install the mosquitto clients with:

sudo apt install mosquitto-clients

From one terminal session open a subscriber client to listen for messages on the test topic using the IP address with debugging on. We can use the IP address for the DEBIX Infinity in the host parameter:

mosquitto_sub -h 10.0.10.116 -t "test" -d

From another terminal session publish a message to the test topic on the broker using the IP address, with debugging:

mosquitto_pub -h 10.0.10.116 -t "test" -m "Hello Infinity" -d

You should see the message printed in the subscriber client session:

So all is good. If there’s a problem, the debugging info should help.

MQTT clients in the IoT subnet should be able to communicate via the broker. Clients in the main LAN should have their connections refused with this configuration.

Secure Access

Now that everything is working with Mosquitto running in the Docker Container, we can increase the level of security for Mosquitto Clients running in the secure zone of our network.

The next section will configure the Broker to only accept MQTTS traffic on port 8883 for clients on the 192.168.10.0/24 subnet.

All connections are authenticated using TLS/SSL Certificates and traffic is encrypted end-to-end. This is the most secure configuration for Mosquitto and also the trickiest one to get working!

We covered all the details in this article so I will just include the highlights:

https://www.rs-online.com/designspark/mqtt-part-6-security

The following example commands assume you are creating Self Hosted Certificates.

Root Certificate

On your secure host as root (not the DEBIX Infinity) create a new CA setup and generate a Root CA certificate if you don’t already have one:

sudo -s
cd /srv/ca/certs/
openssl req -new -x509 -days 365 -extensions v3_ca -nodes \
-keyout root_ca.key -out root_ca.crt

This generates the root certificate root_ca.crt which must be copied to defined locations on each host, which we shall do later. This is publicly readable, but only writable by root.

The private key root_ca.key must be kept secure and held only on the CA host. A hacker obtaining this key can generate their own valid certificates. It can only be read and written by root.

Here is the listing on the CA host - note the permissions:

ls -l
total 8
-rw-r--r-- 1 root root 1627 Apr 20  2023 root_ca.crt
-rw------- 1 root root 1704 Apr 20  2023 root_ca.key

Broker Certificate

Still on the CA host, generate a new Broker Private Key and Certificate Signing Request (without password). The Public Key is included in the CSR. You must use the hostname for the DEBIX Infinity in the CN as this will be matched when MQTTS connections are attempted in use.

The Broker certificate is used in the MQTT authentication by the Client to ensure that the Broker is genuine. Both the Private Key and Certificate will be securely transferred to the DEBIX Infinity broker in a later step.

cd /srv/ca/broker/
openssl req -new -newkey rsa:2048 -nodes \
-keyout imx8_broker.key -out imx8_broker.csr \
-subj "/CN=imx8mp-debix"

Pass the Certificate Signing Request csr file to the Certificate Authority to generate the signed certificate imx8_broker.crt

openssl x509 -req -CA ../certs/root_ca.crt \
-CAkey ../certs/root_ca.key -CAcreateserial \
-in imx8_broker.csr -out imx8_broker.crt

Here’s the listing on the CA host:

ls -l imx8*
-rw-r--r-- 1 root root 1253 Jul  9 11:49 imx8_broker.crt
-rw-r--r-- 1 root root  895 Jul  9 11:47 imx8_broker.csr
-rw------- 1 root root 1704 Jul  9 11:47 imx8_broker.key

Copy Broker Certs

From the CA host as root, we can copy the CA Certificate, Broker Private Key and Broker Certificate to the debix user home directory on the DEBIX Infinity via secure SSH. 

cd /srv/ca
scp certs/root_ca.crt broker/imx8_broker.{crt,key} debix@imx8mp-debix:~/

On the DEBIX Infinity as the debix user, move the CA Certificate to its correct location and change ownership to root:

sudo mv root_ca.crt /etc/mosquitto/ca_certificates/
sudo chown root:root /etc/mosquitto/ca_certificates/root_ca.crt

Then, still on the DEBIX Infinity, move the Broker Private Key and Certificate to the mosquitto certificate store and change ownership to mosquitto:

sudo mv imx8_broker.{crt,key} -t /etc/mosquitto/certs/
sudo chown mosquitto:mosquitto /etc/mosquitto/certs/imx8_broker.{crt,key}

That's the Broker TLS keys and certificates all in place on the DEBIX Infinity now, phew!

Client Certificates

Each MQTT client needs its own Private Key and Certificate to authenticate with the Broker over TLS. This one’s called client01 and has its own directory under the user's home as an example. 

The next steps are similar to that of the Broker where the keys and certificates are generated on the CA host and then moved to the Client host directory. In this example, my client is on the same host as the CA. The client also needs a copy of the CA Root Certificate for the Chain of Trust.

As root generate the Private Key and Certificate Request:

cd /srv/ca/client01/
openssl req -new -newkey rsa:2048 -nodes \
-keyout client01.key -out client01.csr \
-subj "/CN=client01"

Now generate the Client Certificate from the CSR:

sudo openssl x509 -req -CA /srv/ca/certs/root_ca.crt \
-CAkey /srv/ca/certs/root_ca.key -CAcreateserial \
-in client01.csr -out client01.crt

Copy the root_ca.crt, client01.crt and client01.key to the Client and change ownership:

sudo cp /srv/ca/certs/root_ca.crt ~/client01/
sudo cp /srv/ca/client01/client01.{crt,key} ~/client01/
sudo chown $USER:$USER ~/client01/client01.{crt,key}

Here’s the listing of all the required client files with correct ownership and permissions:

ls -l ~/client01
total 12
-rw-r--r-- 1 pete pete 1249 Jul 19 17:08 client01.crt
-rw------- 1 pete pete 1704 Jul 19 17:08 client01.key
-rw-r--r-- 1 root root 1627 Jul 19 17:08 root_ca.crt

Broker Security Upgrade

The final step is to upgrade the Mosquitto configuration on the DEBIX Infinity to listen for TLS connections on port 8883 for clients on the 192.168.10.0/24 subnet and for the Broker to only authenticate and authorise clients with valid certificates.

Because the broker config file is copied from the host into the docker container, the paths need to be relative to the Docker container, not the host.

Here’s the contents of /etc/mosquitto/conf.d/broker.conf:

listener 1883
allow_anonymous true

# MQTT over TLS
listener 8883
# Paths relative to docker volumes, not docker host!
cafile /mosquitto/config/ca.crt
certfile /mosquitto/config/mosquitto.crt
keyfile /mosquitto/config/mosquitto.key
# Only allow clients with valid certs
require_certificate true
# Set client ID
use_identity_as_username true

Now stop the old container and remove it then start a new one with the addition of the new mappings:

docker stop mosquitto && docker rm mosquitto
docker run -it -d --name mosquitto -p 10.0.10.116:1883:1883 \
-p 192.168.10.113:8883:8883 \
-v /etc/mosquitto/conf.d/broker.conf:/mosquitto/config/mosquitto.conf \
-v /etc/mosquitto/ca_certificates/root_ca.crt:/mosquitto/config/ca.crt \
-v /etc/mosquitto/certs/imx8_broker.crt:/mosquitto/config/mosquitto.crt \
-v /etc/mosquitto/certs/imx8_broker.key:/mosquitto/config/mosquitto.key \
eclipse-mosquitto:2.0.18

Tip: To start an existing mosquito instance run:

docker start mosquitto

TLS Testing

If everything has been set up correctly, you should be able to publish and subscribe with clients running on the 192.168.10.0/24 network.

The clients will authenticate the Broker, to ensure that it’s valid before starting the authentication and authorization handshake using the Client Certificate. All this is encrypted end-to-end.

Note that clients must connect using the broker's hostname as this is validated against the CN in the Broker Certificate created in the previous step.

Start a subscriber on a host in the secure network zone:

cd ~/client01/
mosquitto_sub -h imx8mp-debix -p 8883 --cafile root_ca.crt \
--cert client01.crt --key client01.key -t test -d

And publish a test message from a separate terminal session in the same zone:

mosquitto_pub -h imx8mp-debix -p 8883 --cafile root_ca.crt \
--cert client01.crt --key client01.key -t test -m 'Secure hello' -d

For a final check, Anonymous Clients in the IoT zone should be able to publish messages to the Secure Zone and TLS Clients with Certificates should be able to receive those messages.

TLS Clients in the secure zone should be able to publish messages to the IoT and Anonymous Clients should be able to receive those messages.

All messages must pass between the zones through the Broker and there are no MQTT / MQTTS ports open on the IoT network.

Run a few checks to satisfy yourself that everything is working as expected.

Wireshark

Because we are concerned about security, it’s not sufficient to just rely on configuring everything correctly, we need a way to check this and troubleshoot any issues.

Wireshark runs on the Ubuntu desktop so it can be used to analyse the MQTT traffic passing through both network interfaces on the Mosquitto broker. Wireshark can be installed with:

sudo apt install wireshark

Start it up from the Activities menu. It’s the icon with the shark's fin!

• On the start screen select the ens33 interface
• CTRL + click on the ens34 interface

You will now be able to analyse traffic on both interfaces:

• Click the blue sharks fin icon in the toolbar 
• Enter tcp or mqtt in the filter box which must turn green to indicate valid syntax
• Press return to start capturing
• Now send some messages between clients

Messages on the TLS port 8883 should only show up as TCP packets, you can't even tell they are MQTTS protocol.

In this example, if you click on the Publish Message transaction you can clearly see the contents of the message published on the un-encrypted 1883 port.

Summary

This article addresses the issue of securing IoT devices using MQTT protocol running on industrial or home networks using the dual network interfaces on the DEBIX Infinity.

It demonstrates how to install and configure a Mosquitto MQTT broker running in Docker to allow only secure access from external networks using TLS/SSL with Certificates. Insecure IoT devices running on a separate subnet are able to send and receive MQTT protocol via the broker only.

We also showed how to run Wireshark on the DEBIX Infinity to visualise network packets on both network interfaces to check the Mosquitto configuration.

It would be advisable to add password authentication to the IoT devices if they can support this. Also further work could be done to limit SSH access to the Broker, SSH and specific IoT devices in a production system.

References

DEBIX Infinity User Guide: https://debix.io/Uploads/Temp/file/20240605/DEBIX%20Infinity%20User%20Manual-V1.0.pdf

Docker: https://docs.docker.com/manuals/

mosquitto-tls man page: https://mosquitto.org/man/mosquitto-tls-7.html

Wireshark docs: https://www.wireshark.org/docs/