Configuring a hierarchy of IoT Edge Devices at Home Part 3 – Configuring Business Planning and Logistics (IT)
In the last two posts Configuring a hierarchy of IoT Edge Devices at Home Part 1 – Configuring the IT Proxy and Configuring a hierarchy of IoT Edge Devices at Home Part 2 – Configuring the Enterprise Network (IT) we have set up the proxy and the top layer of the hierarchical IoT Edge network. This post will describe how to set up and configure the first nested layer in the Purdue network architecture for manufacturing – the Business Planning and Logistics (IT) layer. So, let’s get going!
A lot of the steps to configure the IoT Edge runtime on the device are repeating from the previous post. Nevertheless, though, let’s go over them again.
Configuring the Network for Layer 4 Device
The tricky part with the Layer 4 device is that it should have no Internet access. The problem with that is that you will need Internet access to install the IoT Edge runtime. In a typical scenario, the Layer 4 device will come with the IoT Edge runtime pre-installed and you will only need to plug in the device and configure it to talk to its parent. For our scenario though, we need to have the device Internet-enabled while installing the IoT Edge and lock it down after that. Here is the initial dhcpcd.conf
configuration for the Layer 4 device:
interface eth0 static ip_address=10.16.6.4/16 static routers=10.16.8.4 static domain_name_servers=1.1.1.1
The only difference in the network configuration is the IP address of the device. This configuration will allow us to download the necessary software to the device before we restrict its access.
While speaking of necessary software, let’s install the netfilter-persistent
and iptables-persistent
and have them ready for locking the device networking later on:
sudo DEBIAN_FRONTEND=noninteractive apt install -y netfilter-persistent iptables-persistent
Create the Azure Cloud Resources
We already have created the resource group and the IoT Hub resource in Azure. The only new resource that we need to create is the IoT Edge device for Layer 4. One difference here is that, unlike the Layer 5 device, the Layer 4 device will have a parent. And this parent will be the Layer 5 device. Use the following command to create the Layer 4 device
az iot hub device-identity create --device-id L4-edge-pi --edge-enabled --hub-name tsm-ioth-eus-test az iot hub device-identity parent set --device-id L4-edge-pi --parent-device-id L5-edge-pi --hub-name tsm-ioth-eus-test az iot hub device-identity connection-string show --device-id L5-edge-pi --hub-name tsm-ioth-eus-test
The first command creates the IoT Edge device for Layer 4. The second command sets the Layer 5 device as the parent for the Layer 4 device. And, the third command returns the connection string for the Layer 4 device that we can use for the IoT Edge runtime configuration.
Installing Azure IoT Edge Runtime on the Layer 4 Device
Surprisingly to me, it the time between posting my last post and starting this one, the Tutorial to Create a Hierarchy of IoT Edge Devices changed. I will still use the steps from my previous post to provide consistency.
Creating Certificates
We will start again with the creation of certificates – this time for the Layer 4 device. We already have the root and the intermediate certificates for the chain. The only certificate that we need to create is the Layer 4 device certificate. We will use the following command for that:
./certGen.sh create_edge_device_ca_certificate "l4_certificate"
You should see the new certificates in the <WORKDIR>/certs
folder:
drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 . drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 .. ... -rwxrwxrwx 1 toddysm toddysm 5891 Apr 16 10:46 iot-edge-device-ca-l4_certificate-full-chain.cert.pem -r-xr-xr-x 1 toddysm toddysm 1931 Apr 16 10:46 iot-edge-device-ca-l4_certificate.cert.pem -rwxrwxrwx 1 toddysm toddysm 7240 Apr 16 10:46 iot-edge-device-ca-l4_certificate.cert.pfx ...
The private keys are in the <WORKDIR>/private
folder together with the root and intermediate keys:
drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 . drwxrwxrwx 1 toddysm toddysm 4096 Apr 16 10:46 .. -r-xr-xr-x 1 toddysm toddysm 3326 Apr 8 14:54 azure-iot-test-only.intermediate.key.pem -r-xr-xr-x 1 toddysm toddysm 3326 Apr 8 14:54 azure-iot-test-only.root.ca.key.pem -r-xr-xr-x 1 toddysm toddysm 3243 Apr 16 10:46 iot-edge-device-ca-l4_certificate.key.pem ...
As before, we need to upload the relevant certificates to the Layer 4 device. From the <WORKDIR>
folder on the workstation use the following commands:
scp ./certs/azure-iot-test-only.root.ca.cert.pem pi@pi-l4:. scp ./certs/iot-edge-device-ca-l4_certificate-full-chain.cert.pem pi@pi-l4:. scp ./private/iot-edge-device-ca-l4_certificate.key.pem pi@pi-l4:.
Connect to the Layer 4 device with ssh pi@pi-l4
and install the root CA using:
sudo cp ~/azure-iot-test-only.root.ca.cert.pem /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.pem.crt sudo update-ca-certificates
Verify that the cert is installed with:
ls /etc/ssl/certs/ | grep azure
We will also move the device certificate chain and the private key to the /var/secrets
folder:
sudo mkdir /var/secrets sudo mkdir /var/secrets/aziot sudo mv iot-edge-device-ca-l4_certificate-full-chain.cert.pem /var/secrets/aziot/ sudo mv iot-edge-device-ca-l4_certificate.key.pem /var/secrets/aziot/
We are all set to install and configure the Azure IoT Edge runtime.
Configuring the IoT Edge Runtime
As mentioned before the steps are described in Install or uninstall Azure IoT Edge for Linux and here is the configuration that we need to use in /etc/aziot/config.toml
file:
hostname = "10.16.6.4" parent_hostname = "10.16.7.4" trust_bundle_cert = "file:///etc/ssl/certs/azure-iot-test-only.root.ca.cert.pem.pem" [provisioning] source = "manual" connection_string = "HostName=tsm-ioth-eus-test.azure-devices.net;DeviceId=L4-edge-pi;SharedAccessKey=s0m#VeRYcRYpT1c$tR1nG" [agent] name = "edgeAgent" type = "docker" [agent.config] image = "10.16.7.4:<port>/azureiotedge-agent:1.2" [edge_ca] cert = "file:///var/secrets/aziot/iot-edge-device-ca-l4_certificate-full-chain.cert.pem" pk = "file:///var/secrets/aziot/iot-edge-device-ca-l4_certificate.key.pem"
Note two differences in this configuration compared to the configuration for the Layer 5 device:
- There is a
parent_hostname
configuration that has the Layer 5 device’s IP address as a value. - The agent image is pulled from the parent registry
10.16.7.4:<port>/azureiotedge-agent:1.2
instead frommcr.microsoft.com
(don’t forget to remove/change the<port>
in the configuration depending on how you set up the parent registry).
Also, if the registry running on your Layer 5 device requires authentication, you will need to provide credentials in the [agent.config.auth]
section.
Restricting the Network for the Layer 4 Device
Now, that we have the IoT Edge runtime configured, we need to restrict the traffic to and from the Layer 4 device. Here is what are we going to do:
- We will allow SSH connectivity only from the IP addresses in the Demo/Support network, i.e. where our workstation is. However, we will use a stricter CIDR to allow only
10.16.9.x
addresses. Here are the commands that will make that configuration:sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -s 10.16.9.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --sport 22 -d 10.16.9.0/24 -m state --state ESTABLISHED -j ACCEPT
- We will enable connectivity on the following ports
8080
(or whatever the registry port is),8883
,443
, and5671
to the Layer 5 network only, i.e.10.16.7.x
addresses. Here the commands that will make that configuration:sudo iptables -A OUTPUT -p tcp --dport <registry-port> -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport <registry-port> -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --dport 8883 -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport 8883 -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --dport 5671 -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport 5671 -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --dport 443 -d 10.16.7.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport 443 -s 10.16.7.0/24 -m state --state ESTABLISHED -j ACCEPT
- We will also enable connectivity on the following ports
8080
(or whatever the registry port is),8883
,443
, and5671
from the OT Proxy network only, i.e.10.16.5.x
addresses. Here the commands that will make that configuration:sudo iptables -A OUTPUT -p tcp --dport <registry-port> -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport <registry-port> -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --dport 8883 -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport 8883 -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --dport 5671 -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport 5671 -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT sudo iptables -A OUTPUT -p tcp --dport 443 -d 10.16.5.0/24 -m state --state ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i eth0 -p tcp --sport 443 -s 10.16.5.0/24 -m state --state NEW,ESTABLISHED -j ACCEPT
- Last, we need to disable any other traffic to and from the Layer 4 device. Here the commands for that:
sudo iptables -P INPUT DROP sudo iptables -P OUTPUT DROP sudo iptables -P FORWARD DROP
Don’t forget to save the configuration and restart the device with:
sudo netfilter-persistent save sudo systemctl reboot
Deploying Modules on the Layer 4 Device
Similar to the deployment of modules on the Layer 5 Device, we need to deploy the standard $edgeAgent
and $edgeHub
modules as well as a registry module. For this layer, I experimented with the API proxy module. Hence, the configurations include also the $IoTEdgeApiProxy
module. There is also an API Proxy configuration that you need to set on the $IoTEdgeApiProxy
module’s twin – for details, take a look at Configure the API proxy module for your gateway hierarchy scenario. Here are the Gists that you can use for deploying the modules on the Layer 4 device:
- Layer 4 Modules – OSS registry with API proxy
- Layer 4 Modules – ACR connected registry with connection string and API proxy
- Azure IoT Edge API Proxy Configuration
Note: At the time of this writing, the ACR connected registry is in a limited preview. See https://aka.ms/acr/connected-registry for more details.
By using the API proxy, you can also leverage the certificates deployed on the device to connect to the registry over HTTPS. Thus, you don’t need to configure Docker on the clients to connect to an insecured registry.
Testing the Registry Module
To test the registry module, use the following commands:
docker login -u <sync_token_name> -p <sync_token_password> 10.16.6.4:<port_if_any> docker pull 10.16.6.4:<port_if_any>/azureiotedge-simulated-temperature-sensor:1.0
Now, we have set up the first nested layer (Layer 4) for our nested IoT Edge infrastructure. In the next post, I will describe the steps to set up the OT Proxy and the second nested layer for IoT Edge (Layer 3).
Image by ejaugsburg from Pixabay