Optimising your Fail2Ban filters | The Art of Web
Going beyond the basics with Fail2Ban involves some experience with parsing log files and regular expressions. Below you can find a short introduction to the available tools and steps for analyzing existing filters on your server.
Examining a jail
Let's start with the ssh/sshd filter as it is the only one enabled by default after installation.
In Fail2Ban 0.8.x you can find the settings in /etc/fail2ban/jail.conf:
[ssh] enabled = true port = ssh filter = sshd logpath = /var/log/auth.log maxretry = 5
Whereas in Fail2Ban 0.9.x you need to look in both /etc/fail2ban/jail.conf:
[sshd] port = ssh logpath = %(sshd_log)s
and /etc/fail2ban/jail.d/defaults-*.conf for where it's turned on and where we can override any of the default settings:
[sshd] enabled = true maxretry = 3
In Fail2Ban 0.9.x the jail heading in square brackets also identifies the filter being used.
In both cases, the jail settings point us to the same filter /etc/fail2ban/filter.d/sshd.conf where you will find definitions for failregex and ignoreregex for this jail. And %(sshd_log)s is mapped to %(syslog_authpriv)s which in turn points to (on Debian) /var/log/auth.log.
What's important is that we've identified both the log file location as well as the filter holding the regular expressions for parsing the log file.
An alternative way to see the filter name and log path is from the command line:
# fail2ban-client status Status |- Number of jail: 6 `- Jail list: apache-auth, apache-gdayedit, apache-noscript, apache-overflows, sshd, sshd-ddos # fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed: 0 | |- Total failed: 0 | `- File list: /var/log/auth.log `- Actions |- Currently banned: 0 |- Total banned: 0 `- Banned IP list:
How to test the filter
Fail2Ban comes with some handy command line tools. The fail2ban-client interface is useful for querying and managing jails, but in this case the one we want is fail2ban-regex which can be called as follows:
# fail2ban-regex <logfile> <failregex> <ignoreregex>
But instead of typing regular expressions into the command, you can just throw at it the relevant filter configuration file. In other words, to test the sshd filter the command will be:
# fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf
Or, if you want to include an ignoreregex definition, include the filter file twice:
# fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf /etc/fail2ban/filter.d/sshd.conf
There some extra options you can include for a better oversight, such as:
- -v, --verbose
- Be verbose in output
- Print all missed lines
- Print all ignored lines
- Print all matched lines
Ignoring harmless log lines
Before tweaking our failregex settings we first set up some ignoreregex patterns to filter out what is normal server activity.
For example, on our servers programs such as Systemd, CRON and PostgreSQL can all generate a lot of auth.log activity throughout the day, and to filter it we have added the following expressions:
ignoreregex = : pam_unix\((cron|sshd|systemd-user):session\): session (open|clos)ed for user (daemon|munin|postgres|root)( by \(uid=0\))?$ : Successful su for (postgres) by root$ New session \d+ of user (postgres)\.$ Removed session \d+\.$
For security all regular expressions should be anchored to the end of line $ if at all possible.
Tweaking the filter
With all the background noise filtered out we can now focus on what is being blocked.
# fail2ban-regex -v --print-all-missed /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf /etc/fail2ban/filter.d/sshd.conf | less
This will output:
- All failregex expressions and the number of matches
- All ignoreregex expressions and the number of matches
- Date formats detected
- Missed lines (lines neither matched nor ignored)
What we're interested in is the last section "Missed lines". Anything listed here has fallen through the cracks and may need our attention. You should consider carefully these lines and modify your settings as necessary to have them included in either the 'matched' or 'ignored' list.
After modifying the filter configuration you can update an already running instance of Fail2Ban using:
# fail2ban-client reload sshd
In our case there's not much to see in auth.log as we secure our SSH by using non-standard ports, certificate based authentication, and other such devices.
Things are much more interesting when you get to http and email filters.
Analyzing an Apache filter
Following the above example, we can now examine other filters using the same approach. Here we're looking into apache-noscript which monitors the Apache error.log looking for spurious requests.
But first, to enable this filter in Fail2Ban 0.8.x you need to edit /etc/fail2ban/jail.conf:
[apache-noscript] enabled = true port = http,https filter = apache-noscript logpath = /var/log/apache2/error.log maxretry = 6 bantime = 43200
and in Fail2Ban 0.9.x you can put the settings in /etc/fail2ban/jail.d/local.conf or similar. A separate conf file in that directory of every jail is also an option. You only need to add lines that are changed from the default:
[apache-noscript] enabled = true maxretry = 6 bantime = 43200
Once the jail has been activated and running for some time, you can analyze the performance using:
# fail2ban-regex -v --print-all-missed /var/log/apache2/error.log /etc/fail2ban/filter.d/apache-noscript.conf /etc/fail2ban/filter.d/apache-noscript.conf | less
Looking at the "Missed lines" as before, we've come up with the following regular expressions:
failregex = ^%(_apache_error_client)s ((AH001(28|30): )?File does not exist|(AH01264: )?script not found or unable to stat): /\S*\.(php|aspx?|exe|pl|cgi|rar|zip|gz)(, referer: \S+)?\s*$ ^%(_apache_error_client)s script '/\S*\.(php|aspx?|exe|pl)\S*' not found or unable to stat(, referer: \S+)?\s*$ ^%(_apache_error_client)s (script not found or unable to stat|attempt to invoke directory as script): /usr/lib/cgi-bin ignoreregex = [pagespeed:(warn|error)] [client \S+] PHP (Notice|Parse error|Warning): [client \S+] script '.*\.html' not found or unable to stat(, referer: \S+)?\s*$ [client \S+] Operation timed out after \d+ milliseconds
These will obviously differ between servers and operating systems. The remaining missed lines mostly relate to IPv6 (support should be coming in a future version of Fail2Ban), or will be captured by other filters such as apache-overflows or apache-wordpress.
If you're running a mail server you definitely want to enable one or more of the sendmail jails. The following relates to the sendmail jail in Fail2Ban 0.8.x, enabled in /etc/fail2ban/jail.local:
[sendmail] enabled = true port = smtp,submission filter = sendmail logpath = /var/log/mail.log maxretry = 3
The command for testing the filter is:
# fail2ban-regex --print-all-missed /var/log/mail.log /etc/fail2ban/filter.d/sendmail.conf /etc/fail2ban/filter.d/sendmail.conf | less
And after some work our filters look something like this:
failregex = lost input channel from .*\[<HOST>\] to MTA-v\d after (data|mail|rcpt)$ \[<HOST>\] did not issue MAIL/EXPN/VRFY/ETRN during connection to (MSP|MTA)-v\d$ \[<HOST>\], reject.*\.\.\. (Relaying denied) \[<HOST>\]: Possible SMTP RCPT flood, throttling\.$ timeout waiting for input from \[<HOST>\] during server cmd read rejecting commands from( .+)? \[<HOST>\] due to pre-greeting traffic relay=([^ ]+ )?\[<HOST>\], .* Domain of sender address [\w@.-]+ does not (exist|resolve)$ ignoreregex = sm-mta\[\d+\]: \w+: (from|to)= sm-mta\[\d+\]: \S+[\d]: sm-mta\[\d+\]: STARTTLS=(client|server) sm-mta\[\d+\]: STARTTLS: (read|write) error=(generic|syscall|timeout) : timeout waiting for input from [\w.-]+ during server cmd read$ : collect: premature EOM: (unexpected close|Connection timed out with [\[\]\w.-]+|Connection reset by \S+)(, sender=\S+)?$ : collect: (I/O error|read timeout|unexpected close) on connection from [\w.-]+ did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-v\d$
As you can see we're quite strict on accepting mail, and any unusual activity will result in a Fail2Ban block - even if Sendmail itself or one of the DNSBL we subscribe to is already addressing the issue.
And any sequence of sendmail blocks will trigger a much longer block triggered by a separate, introspective, Fail2Ban jail.
You should follow the above steps for all active jails, as well as for any inactive jails that look promising for your installation. And repeat on a regular basis, or at least following any major upgrade.
Don't be afraid to create your own filters based on the ones supplied with the default installation. The command-line tools allow you to test them thoroughly before deployment, and in addition to the simple 'block' action they can also be used to generate emails or run other commands in response to log activity.
Last but not least, when you accidentially manage to block yourself, you can remove the block using:
# fail2ban-client set <jail> unbanip <ip>
Send a message to The Art of Web:
press <Esc> or click outside this box to close