Configuring a hierarchy of IoT Edge Devices at Home Part 3 – Configuring Business Planning and Logistics (IT)

In the last two posts Configuring a hierarchy of IoT Edge Devices at Home Part 1 – Configuring the IT Proxy and Configuring a hierarchy of IoT Edge Devices at Home Part 2 – Configuring the Enterprise Network (IT) we have set up the proxy and the top layer of the hierarchical IoT Edge network. This post will describe how to set up and configure the first nested layer in the Purdue network architecture for manufacturing – the Business Planning and Logistics (IT) layer. So, let’s get going!

A lot of the steps to configure the IoT Edge runtime on the device are repeating from the previous post. Nevertheless, though, let’s go over them again.

Configuring the Network for Layer 4 Device

The tricky part with the Layer 4 device is that it should have no Internet access. The problem with that is that you will need Internet access to install the IoT Edge runtime. In a typical scenario, the Layer 4 device will come with the IoT Edge runtime pre-installed and you will only need to plug in the device and configure it to talk to its parent. For our scenario though, we need to have the device Internet-enabled while installing the IoT Edge and lock it down after that. Here is the initial dhcpcd.conf configuration for the Layer 4 device:

interface eth0
static ip_address=10.16.6.4/16
static routers=10.16.8.4
static domain_name_servers=1.1.1.1

The only difference in the network configuration is the IP address of the device. This configuration will allow us to download the necessary software to the device before we restrict its access.

While speaking of necessary software, let’s install the netfilter-persistent and iptables-persistent and have them ready for locking the device networking later on:

sudo DEBIAN_FRONTEND=noninteractive apt install -y netfilter-persistent iptables-persistent

Create the Azure Cloud Resources

We already have created the resource group and the IoT Hub resource in Azure. The only new resource that we need to create is the IoT Edge device for Layer 4. One difference here is that, unlike the Layer 5 device, the Layer 4 device will have a parent. And this parent will be the Layer 5 device. Use the following command to create the Layer 4 device

az iot hub device-identity create --device-id L4-edge-pi --edge-enabled --hub-name tsm-ioth-eus-test
az iot hub device-identity parent set --device-id L4-edge-pi --parent-device-id L5-edge-pi --hub-name tsm-ioth-eus-test
az iot hub device-identity connection-string show --device-id L5-edge-pi --hub-name tsm-ioth-eus-test

The first command creates the IoT Edge device for Layer 4. The second command sets the Layer 5 device as the parent for the Layer 4 device. And, the third command returns the connection string for the Layer 4 device that we can use for the IoT Edge runtime configuration.

Installing Azure IoT Edge Runtime on the Layer 4 Device

Surprisingly to me, it the time between posting my last post and starting this one, the Tutorial to Create a Hierarchy of IoT Edge Devices changed. I will still use the steps from my previous post to provide consistency.

Creating Certificates

We will start again with the creation of certificates – this time for the Layer 4 device. We already have the root and the intermediate certificates for the chain. The only certificate that we need to create is the Layer 4 device certificate. We will use the following command for that:

./certGen.sh create_edge_device_ca_certificate "l4_certificate"

You should see the new certificates in the <WORKDIR>/certs folder:

drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 .
drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 ..
...
-rwxrwxrwx 1 toddysm toddysm 5891 Apr 16 10:46 iot-edge-device-ca-l4_certificate-full-chain.cert.pem
-r-xr-xr-x 1 toddysm toddysm 1931 Apr 16 10:46 iot-edge-device-ca-l4_certificate.cert.pem
-rwxrwxrwx 1 toddysm toddysm 7240 Apr 16 10:46 iot-edge-device-ca-l4_certificate.cert.pfx
...

The private keys are in the <WORKDIR>/private folder together with the root and intermediate keys:

drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 .
drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 ..
-r-xr-xr-x 1 toddysm toddysm 3326 Apr  8 14:54 azure-iot-test-only.intermediate.key.pem
-r-xr-xr-x 1 toddysm toddysm 3326 Apr  8 14:54 azure-iot-test-only.root.ca.key.pem
-r-xr-xr-x 1 toddysm toddysm 3243 Apr 16 10:46 iot-edge-device-ca-l4_certificate.key.pem
...

As before, we need to upload the relevant certificates to the Layer 4 device. From the <WORKDIR> folder on the workstation use the following commands:

scp ./certs/azure-iot-test-only.root.ca.cert.pem pi@pi-l4:.
scp ./certs/iot-edge-device-ca-l4_certificate-full-chain.cert.pem pi@pi-l4:.
scp ./private/iot-edge-device-ca-l4_certificate.key.pem pi@pi-l4:.

Connect to the Layer 4 device with ssh pi@pi-l4 and install the root CA using:

sudo cp ~/azure-iot-test-only.root.ca.cert.pem /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.pem.crt
sudo update-ca-certificates

Verify that the cert is installed with:

ls /etc/ssl/certs/ | grep azure

We will also move the device certificate chain and the private key to the /var/secrets folder:

sudo mkdir /var/secrets
sudo mkdir /var/secrets/aziot
sudo mv iot-edge-device-ca-l4_certificate-full-chain.cert.pem /var/secrets/aziot/
sudo mv iot-edge-device-ca-l4_certificate.key.pem /var/secrets/aziot/

We are all set to install and configure the Azure IoT Edge runtime.

Configuring the IoT Edge Runtime

As mentioned before the steps are described in Install or uninstall Azure IoT Edge for Linux and here is the configuration that we need to use in /etc/aziot/config.tomlfile:

hostname = "10.16.6.4"

parent_hostname = "10.16.7.4"

trust_bundle_cert = "file:///etc/ssl/certs/azure-iot-test-only.root.ca.cert.pem.pem"

[provisioning]
source = "manual"
connection_string = "HostName=tsm-ioth-eus-test.azure-devices.net;DeviceId=L4-edge-pi;SharedAccessKey=s0m#VeRYcRYpT1c$tR1nG"

[agent]
name = "edgeAgent"
type = "docker"
[agent.config]
image = "10.16.7.4:<port>/azureiotedge-agent:1.2"

[edge_ca]
cert = "file:///var/secrets/aziot/iot-edge-device-ca-l4_certificate-full-chain.cert.pem"
pk = "file:///var/secrets/aziot/iot-edge-device-ca-l4_certificate.key.pem"

Note two differences in this configuration compared to the configuration for the Layer 5 device:

  1. There is a parent_hostname configuration that has the Layer 5 device’s IP address as a value.
  2. The agent image is pulled from the parent registry 10.16.7.4:<port>/azureiotedge-agent:1.2 instead from mcr.microsoft.com (don’t forget to remove/change the <port> in the configuration depending on how you set up the parent registry).

Also, if the registry running on your Layer 5 device requires authentication, you will need to provide credentials in the [agent.config.auth] section.

Restricting the Network for the Layer 4 Device

Now, that we have the IoT Edge runtime configured, we need to restrict the traffic to and from the Layer 4 device. Here is what are we going to do:

  • We will allow SSH connectivity only from the IP addresses in the Demo/Support network, i.e. where our workstation is. However, we will use a stricter CIDR to allow only  10.16.9.x addresses. Here are the commands that will make that configuration: 
    sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -s 10.16.9.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --sport 22 -d 10.16.9.0/24 -m state --state ESTABLISHED -j ACCEPT
  • We will enable connectivity on the following ports 8080 (or whatever the registry port is), 8883, 443, and 5671 to the Layer 5 network only, i.e. 10.16.7.x addresses. Here the commands that will make that configuration: 
    sudo iptables -A OUTPUT -p tcp --dport <registry-port> -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport <registry-port> -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 8883 -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 8883 -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 5671 -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 5671 -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 443 -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT
  • We will also enable connectivity on the following ports 8080 (or whatever the registry port is), 8883, 443, and 5671 from the OT Proxy network only, i.e. 10.16.5.x addresses. Here the commands that will make that configuration: 
    sudo iptables -A OUTPUT -p tcp --dport <registry-port> -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport <registry-port> -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 8883 -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 8883 -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 5671 -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 5671 -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 443 -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
  • Last, we need to disable any other traffic to and from the Layer 4 device. Here the commands for that: 
    sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP

Don’t forget to save the configuration and restart the device with:

sudo netfilter-persistent save
sudo systemctl reboot

Deploying Modules on the Layer 4 Device

Similar to the deployment of modules on the Layer 5 Device, we need to deploy the standard $edgeAgent and $edgeHub modules as well as a registry module. For this layer, I experimented with the API proxy module. Hence, the configurations include also the $IoTEdgeApiProxy module. There is also an API Proxy configuration that you need to set on the $IoTEdgeApiProxy module’s twin – for details, take a look at Configure the API proxy module for your gateway hierarchy scenario. Here are the Gists that you can use for deploying the modules on the Layer 4 device:

By using the API proxy, you can also leverage the certificates deployed on the device to connect to the registry over HTTPS. Thus, you don’t need to configure Docker on the clients to connect to an insecured registry.

Testing the Registry Module

To test the registry module, use the following commands:

docker login -u <sync_token_name> -p <sync_token_password> 10.16.6.4:<port_if_any>
docker pull 10.16.6.4:<port_if_any>/azureiotedge-simulated-temperature-sensor:1.0

Now, we have set up the first nested layer (Layer 4) for our nested IoT Edge infrastructure. In the next post, I will describe the steps to set up the OT Proxy and the second nested layer for IoT Edge (Layer 3).

Image by ejaugsburg from Pixabay