Learn More About Your Home Network with Elastic SIEM – Part 1: Setting Up Elastic SIEM
Last night I had some free time to play with my network, and I ran
tcpdump out of curiosity. For a while, I’ve been interested to analyze what traffic is going through my home network, and the result of my test pushed me to get to work. I have a bunch of Raspberry Pi devices in my drawers, so, the simplest thing that I can do is get one and install Elastic SIEM on it. For those of you, who don’t know what SIEM is, it stands for Security Information and Event Management. My hope was that with it, I will be able to get a better understanding of the traffic on my home network.
Installing Elastic SIEM on Raspberry Pi
The first thing I had to do is to install the ELK stack on a Raspberry Pi. There are not too many good articles that explain how to set up Elastic SIEM on your Pi. According to Elastic, Elastic SIEM is generally available in Elastic Stack 7.6. So, installing the Elastic Stack should do the work.
A couple of notes before that:
- The first thing to keep in mind is that 8GB is the minimum requirement for the ELK stack. You can get around with 2GB Pi, but if you want to run the whole stack (Elasticsearch, Logstash, and Kibana) on a single device, make sure that you order Raspberry Pi 4 Model B Quad Core 64 Bit with 4GB[ad]. Even this one is a stretch if you collect a lot of information. A good option would be to split the stack over two devices: one for Elasticsearch and Kibana, and another one for the Logstash service
- Elastic has no builds for Raspbian. Hence, in the instructions below, I will use Debian packages and will describe the steps to install those on the Pi. This will require some custom configs and scripts, so be prepared for that. Well, this article is how to hack the installation and no warranties are provided 🙂
- You will not be able to use the ML functionality of Elasticsearch because it is not supported on the low-powered Raspbian device
- Below steps assume version 7.7.0 of the ELK stack. If you are installing a different version, make sure that you replace it accordingly in the commands below
- Last but not least (in the spirit of no warranties), Elasticsearch has a dependency on
libc6that will be ignored and will break future updates. You have to deal with this at your own risk
Here the steps to follow.
Installing Elasticsearch on Raspberry Pi
- Set up your Raspberry Pi first. Here are the steps to set up your Rasberry Pi. The Raspberry Pi Imager makes it even easier to set up the OS. Once again, I would recommend using Raspberry Pi 4 Model B Quad Core 64 Bit with 4GB[ad] and a larger SD card[ad] to save logs for longer.
- Make sure all packages are up to date, and your Raspbian OS is fully patched.
sudo apt-get update sudo apt-get upgrade
- Install the ZIP utility, we will need that later on for the Logstash configuration.
sudo apt-get install zip
- Then, install the Java JRE because Elastic Stack requires it. You can use the open-source JRE to avoid licensing troubles with Oracle’s one.
sudo apt-get install default-jre
- Once this is done, go ahead and download the Debian package for Elasticsearch. Make sure that you download the package with no JDK in it.
- Once this is done, go ahead and install Elasticsearch using the package manager.
sudo dpkg -i --force-all --ignore-depends=libc6 elasticsearch-7.7.0-no-jdk-amd64.deb
- Next, we need to configure Elasticsearch to use the installed JDK.
sudo vi /etc/default/elasticsearch
JAVA_HOMEto the location of the JDK. Normally this is
/usr/lib/jvm/default-java. You can also set the
JAVA_HOMEto the same path in the
/etc/environmentfile but this is not required.
- Last thing you need to do if to disable the ML XPack for Elasticsearch. Change the access mode to the
/etc/elasticsearchdirectory first and edit the Elasticsearch configuration file.
sudo chmod g+w /etc/elasticsearch sudo vi /etc/elasticsearch/elasticsearch.yml
The above steps install and configure the Elasticsearch service on your Raspberry Pi. You can start the service with:
sudo service elasticsearch start
Or check its status with:
sudo service elasticsearch status
Installing Logstash on Raspberry Pi
Installing Logstash on the Raspberry Pi turned out to be a bit more problematic than Elasticsearch. Again, Elastic doesn’t have a Logstash package that targets ARM architecture and you need to install it manually. StackOverflow posts and GitHub issues were particularly helpful for that – I listed the two I used in the References at the end of this article. Here the steps:
- Download the Logstash Debian package from Elastic’s repository.
- Install the downloaded package using the
sudo dpkg -i logstash-7.7.0.deb
- If you run Logstash at this point and encounter error similar to
logstash load error: ffi/ffi -- java.lang.NullPointerException: nullget Alexandre Alouit’s fix from GitHub using.
- Go to
/usr/share/logstash/logstash-core/lib/jarsand check the version of the
- Open the downloaded
fix.sh, and replace the version of the
jruby-complete-X.X.X.X.jaron line 11 with the one from your distribution. In my case, that was
- Change the permissions of the downloaded
fix.shscript, and run it.
chmod 755 fix.sh sudo ./fix.sh
- You can run Logstash with.
sudo service logstash start
You can check the Logstash logs in
/var/log/logstash/logstash-plain.logfor information on whether Logstash is successfully started.
Installing Kibana on Raspberry Pi
Installing Kibana had different challenges. The problem is that Kibana requires an older version of NodeJS, but the one that is packed with the Debian package doesn’t run on Raspbian. What you need to do is to replace the NodeJS version after you install the Debian package. Here the steps:
- Download the Kinabna Debian package from Elastic’s repository.
- Install the downloaded package using the
sudo dpkg -i --force-all kibana-7.7.0-amd64.deb
- Move the redistributed NodeJS to another folder (or delete it completely) and create a new empty directory
nodein the Kibana installation directory.
sudo mv /usr/share/kibana/node /usr/share/kibana/node.OLD sudo mkdir /usr/share/kibana/node
- Next, download version 10.19.0 of NodeJS. This is the required version of NodeJS for Kibana 7.7.0. If you are installing another version of Kibana, you may want to check what NodeJS version it requires. The best way to do that is to start the Kibana service and it will tell you.
- Unpack the TAR and move the content to the
nodedirectory under the Kibana installation directory.
sudo tar -xJvf node-v10.19.0-linux-armv7l.tar.xz sudo mv ./node-v10.19.0-linux-armv7l.tar.xz/* /usr/share/kibana/node
- You may also want to create symlinks for the NodeJS executable and its tools.
sudo ln -s /usr/share/kibana/node/bin/node /usr/bin/node sudo ln -s /usr/share/kibana/node/bin/npm /usr/bin/npm sudo ln -s /usr/share/kibana/node/bin/npx /usr/bin/npx
- Configure Kibana to accept requests on any IP address on the device.
sudo vi /etc/kibana/kibana.yml
- You can run Kibana with.
sudo service kibana start
Although not supported, you can run the complete ELK stack on a Raspberry Pi 4[ad] device. It is not the most trivial installation, but it is not so hard either. In the following posts, I will explain how you can use the Elastic SIEM to monitor the traffic on your network.
Here are some additional links that you may find useful:
- Install Logstash using APT
- StackOverflow Post: ELK Stack – Logstash doesn’t get installed properly
- GitHub Issue: Installing Logstash DEB file fails to run on Pi4+ with Buster
- Alexandre Alouit’s Fix for Logstash FFI issue on Raspbian (and my fork here)
9 thoughts on “Learn More About Your Home Network with Elastic SIEM – Part 1: Setting Up Elastic SIEM”
Whatever I try – I can’t get elasticsearch to run – every time I try to start the service it fails. Fun fact – starting the binary manually and even in the context of the elasticsearch user works.
Can you post the error you are seeing?
I receive this error:
fatal error in thread [elasticsearch[node-1][scheduler][T#1]], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
According to my research this happens with versions above 7.4 when run as systemd service: https://discuss.elastic.co/t/elasticsearch-no-longer-works-under-systemd-7-4-0-on-centos-7-7-1908/201846
But even after changing the tmp directory it still wont run.
When I switch back to version 7.3.2 it works just fine but I would have to change the other components too.
On elasticsearch-7.8.0 there were 2 JNA things to clear:
1) tmp folder permissions —
sudo mkdir -p /usr/share/elasticsearch/tmp && sudo chown elasticsearch:elasticsearch /usr/share/elasticsearch/tmp
2) loading .so file due to non-x64 environment; pickup beta raspianOS-x64 at —
(Beta OS still packs a desktop; remove if needed.)
where to find fix.sh file in pi directories?
fix.sh is a script posted on GitHub. There is a link in the post to it.
I think it was simpler if you made a fully functional image with everything installed :))
That is correct 🙂 Though, I would be reluctant to run images from random people.