If you have any kind of server connected to the Internet, you are no doubt aware that no matter how small or unimportant it might seem, it is frequently probed, tested or subject to various attempts at abuse. These attacks come from so many malicious hosts that it is impossible to keep track by hand. So I started looking for a way to implement an automated blocklist to use with iptables and firewalld which I use on my servers.
ipset
There are good solutions to detect and block hosts that are attacking your Linux system, such as denyhosts or fail2ban and I highly recommend you implement one of them depending on your needs. However, if hosts are known to be malicious in the security community, it is much more elegant to catch them at the front door, before they connect to your services. This is where a blocklist of known malicious hosts comes in.
Of course I wasn’t the first to have considered using blocklists with iptables. Many commercial firewall solutions distribute updates frequently, adding detection signatures and blocklist information. Someone must have done something similar with iptables and firewalld, I thought. And indeed, a few moments spent searching online, revealed a good deal of questions and some good answers too. One method that I particularly liked, was the use of ipset to administer large lists of IP addresses inside the kernel, eliminating the need for thousands upon thousands of iptables or firewalld rules. This sounded exactly like what I wanted achieve. Now all I needed was a good blocklist.
Creating the blocklist
A bit more searching lead me to the site beris.nl, which had both a shell script to create an extensive blocklist and a way to feed it to ipset, so iptables or firewalld could use the list. I am reproducing this script in full below, should the original ever disappear. I take no credit for writing this script which has been modified by me a little bit, just because wizcrafts list does not work anymore. I decided to use badips.com list instead.
I installed the ipset. Then created the directory in /opt called blocklist and created a file called blocklist.sh where I put the content of the script published below.
|
|
Press insert in vi editor, then copy the script below and put with the right button of the mouse into the vi editor. Then press Esc button on the keyboard, type :x and hit Enter.
|
|
Bash script Explanation
|
|
Saves file descriptors so they can be restored to whatever they were before redirection or used themselves to output to whatever they were before the following redirect.
|
|
Restore file descriptors for particular signals. Not generally necessary since they should be restored when the sub-shell exits.
|
|
Redirect stdout to file log.out then redirect stderr to stdout. Note that the order is important when you want them going to the same file. stdout must be redirected before stderr is redirected to stdout.
|
|
if it should exit upon error.
Now you can put the script in /opt/blocklist directory and run the script in the background.
|
|
If you want to see the progress just type:
|
|
To break use ctrl+c
To check is the job running type
|
|
or
|
|
to see it is running in the background.
Basically what this script does, is download lists of IP netblocks and IP addresses from various sites hosting such lists and a shodan.io local txt file (you can prepare more than one local bad IPs database), strip out everything that isn’t an IP netblock or address and then put all those lines in a single text file. This results in a file containing thousands of lines (at the moment well over 100.000 IP addresses). It would be impossible to manage this by hand. If you set this script to run once per day from your crontab, you’ll have a fairly up to date list of malicious hosts. Please refrain from running this script too often, since the websites where the various source lists are hosted, need to pay for this traffic. Updating too often will probably get you banned. At the very bottom, the ipset is flushed and the new list added line by line to the blocklist.
I have found in the internet one website, which contains a list of IPs databases and domains. Please note, that some of these websites are not updated or they have been disabled, hacked, deleted etc. However still a huge part of them is working, and this is a good source, where we can start: http://www.covert.io/threat-intelligence/ to add more resources to this script, what is quite easy, or prepare manually our own databases in txt format as I did in shodan.io case.
Shodan.io IP addresses
|
|
Please note that the instructions that follow, are written for CentOS 7 but they should translate to other distributions as well, since we are only using the command line.
Adding the script to the crontab.
Edit the crontab with this command:
|
|
Add this line:
|
|
Script runs every Sunday at midnight. I added one hour (3600 seconds) random function, what avoids the situation, when everyone will try to download IP lists from servers at the same time. I recommend to change hours in this task. Be aware, that they can ban you if you will try to download IPs too often.
Excellent cronjob examples you will find here: https://crontab.guru/
Plugging the blocklist into firewalld
Running this script from the command line will fail at the moment. While it will create the blocklist file in /etc/ip-blocklist.conf, it will not be able to load the ipset because we have not yet created an ipset called blocklist. We can create it by hand in the following way:
|
|
This command creates an ipset called “blocklist” of type “hash:net”. This type of ipset is used to store different sized IP network addresses, ranging from large netblocks right down to single hosts. Running the above script now will create the blocklist and populate the ipset with the created blocklist. Next we need to add a rule to firewalld so that it will use the blocklist. I recommend inserting this rule at or near the top of the INPUT chain so that it will be processed early in your ruleset. Let’s take a look at the INPUT chain, so we will know where to insert the new rule. The hash size and maxelem value is a power of two.
|
|
The command above will list all the rules currently in the INPUT chain of iptables, with line numbers, which makes it easy to see where to insert the new rule. A snippet of my INPUT chain can be seen above. The first rule accepts all traffic, the second rule accepts all traffic to the loopback interface and we’ll leave them at the top. The sixth rule drops all incoming packets that are in an invalid state which is good. The rule below that logs the number of incoming connections which are rejected. Since that is already quite specific, let’s insert our new rule above this one.
|
|
This command inserts our rule at position 7 in the INPUT chain and matches incoming traffic to the set called “blocklist”, dropping corresponding traffic. At this point, iptables is silently dropping all traffic coming from hosts and netblocks in the blocklist. However, if we want to see what is being blocked, we need to add a logging rule above rule 7 Since I am curious to see what gets blocked, I added the following rule.
|
|
This rule is inserted in position 7 pushing the drop rule to position 8 Now, each incoming traffic matching our blocklist is first logged with the prefix “IP Blocked:” by rule 7 and then discarded by rule 8. A look at the logging reveals this:
|
|
Here we see several attempts by hosts with different IP addresses attempting to connect to the server. These connections will never be established because iptables is on the job.
Since the rules we have written seem to be working, we need to save them so that they will be loaded again should iptables get restarted. On CentOS 7, we can use iptables for that purpose. Issuing the command below will save the rules in /etc/sysconfig/iptables.
|
|
Creating the ipset at boot
So far, we’ve managed to download and compile an extensive blocklist, learned how to load it into ipset and plug that ipset into iptables or firewalld as a blocklist. We’ve also set up a rule to log detected connection attempts from our blocklist. So far so good. There is one remaining problem, though. The moment we reboot our server, you’ll notice the firewall fails to initialize the ruleset saves by iptables/firewalld because it is referencing an ipset that doesn’t exist. We’re going to need a way to create the ipset at boot time. I’ve done quite a bit of searching on how to do that properly but there appears to be little documentation available. In the end, I decided that creating a script to create and fill the ipset when one (or more) Ethernet interfaces come up would make sense. For that purpose, I created a script to run when initializing the networking system.
Create a file called ipset.sh located in /usr/local/bin/
|
|
Put the following script into this file exactly the same way as before.
|
|
What this script does, is quite simple. It first makes sure there is no ipset called blocklist by emptying and destroying any ipset of that name. It then (re-)creates the ipset called blocklist and populates it, using the /etc/ip-blocklist.conf file we’ve created before. I then integrated that script in the new systemd service unit by adding it as a post-up script in /etc/systemd/system/ipset_blocklist.service as below.
|
|
|
|
Reload the systemd process to consider newly created sample.service OR every time when sample.service gets modified.
|
|
Enable this service to start after reboot automatically.
|
|
Start the service.
|
|
In Debian/Ubuntu you can integrate it a little bit different by adding it as a post-up script in /etc/network/interfaces as below:
|
|
As soon as the primary interface (in many cases eth0) is up, the ipset is created and populated by running the /usr/local/bin/ipset.sh script. By the time iptables is initialized, the ipset is available and filled so malicious hosts are blocked nearly immediately. After doing this, our blocklist will survive a reboot, ensuring we always have its protection.
firewalld ipset
The script contains additional part for the firewalld which is a result of looking for the solution for Red Hat family distros.
|
|
First line deletes the existing ipset. Second adds it with proper hash size and maximum elements (IP addresses) which can be added to the opset. Third adds two rules for logging and dropping bad IP addresses. The fifth rules loads bad IPS from the local file created by the script. Then it reloads the firewalld to apply new ipset settings. The penultimate line counts all IPs in the firewalld ipset and the last line counts IPs in the the ipset.
To enable logging dropped connections you need to perform this command.
|
|
It changes the value in file /etc/firewalld/firewalld.conf. You can check it with this command:
|
|
Finally I decided to remove logging bad IPs, just because it creates a really mess in message log. So, I changed it this way.
sudo vi /etc/firewalld/direct.xml
The content of this files looks like this:
|
|
I just removed the first rule
|
|
To filter this I use commands like those below:
|
|
It was better, but everything went to /var/log/messages. This is wrong. I decided to redirect these messages to separate logs. This is how I did it.
|
|
I added there these lines:
|
|
I saved the file and exited. Then I decided to create a logrotate for these logs.
|
|
Added this content:
|
|
Then I did the same for rejected.
|
|
And added:
|
|
Great this is what I wanted, but still messages was messy, so I decided to edit rsyslog.conf and change it this way according to the solution from this website: https://serverfault.com/questions/557885/remove-iptables-log-from-kern-log-syslog-messages
|
|
Look for the part with rules and modify it according to the below example.
|
|
Leave everything below as it is.
After that edit the below file:
|
|
And set the line args like below:
|
|
After that restart auditd and rsyslog with the below commands:
|
|
Then I decided to download multitail to be able to monitor them at once.
|
|
Then I have run this way:
|
|
It was nice, however still messy, so I decided to look for the solution about the kernel messages. And I have found it here: https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console
The solution was simple and I changed values this way:
|
|
Then I checked these values with this command:
|
|
Configure kernel parameters at runtime- explanation
See man sysctl - “configure kernel parameters at runtime&” for more.
Reminder of the severity levels and the four values of kernel.printk:
- CUR = current severity level; only messages more important than this level are printed
- DEF = default severity level assigned to messages with no level
- MIN = minimum allowable CUR
- BTDEF = boot-time default CUR
On my CentOS: 7 4 1 7
|
|
This is too noisy, I just want critical and up (no errors). Unlabeled messages should be regarded as warning, so DEF is good:
|
|
Set to: 3 4 1 3 and problem solved. Now when I use multitail to watch logs I see everything as it should be.
The last thing I had to do was to whitelist Google’s IP addresses, because e-mails from Gmail was rejected because the above solution blocked IP addresses. I did it this way.
|
|
Above commands let save their IP addresses into two files. You need to clean them and leave only IP addresses (one IP per line). Then I created a whitelist and added IP addresses from these two files.
After that hit ctrl+d to log out and go back to the standard user.
|
|
Then all you have to do is to reload firewalld rules.
|
|
If you have any corrections or tips pertaining to the above, I am all ears. If this post helps you in any way, I’d also like to hear about it.
References
- https://www.beris.nl/2015/04/22/using-blacklists-with-iptables/
- https://fedoramagazine.org/protect-your-system-with-fail2ban-and-firewalld-blacklists/
- https://www.thegeekdiary.com/centos-rhel-7-how-to-make-custom-script-to-run-automatically-during-boot/
- https://www.howtoforge.com/tutorial/protect-your-server-computer-with-badips-and-fail2ban/
- https://serverfault.com/questions/842749/firewalld-logging-denied-packets-enabled-not-logging
- https://www.cyberciti.biz/faq/enable-firewalld-logging-for-denied-packets-on-linux/
- https://serverfault.com/questions/859572/missed-kernel-messages
- https://serverfault.com/questions/557885/remove-iptables-log-from-kern-log-syslog-messages
- https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console
- https://wiki.gentoo.org/wiki/Rsyslog
