fail2ban-crawlers

Automatically unban legitimate search engine bots that were banned by Fail2Ban.

This tool verifies bots using FCrDNS (Forward-confirmed reverse DNS):

1. Reverse DNS lookup: IP -> hostname (PTR)
2. Hostname suffix validation against an allowlist
3. Forward DNS lookup: hostname -> IP list
4. Confirmation that the original IP is in the forward lookup results

If all checks pass, the script unbans the IP from the target jail.

Script

• fail2ban-unban-verified-bots.sh

Supported bot domains (default)

• googlebot.com
• google.com
• search.msn.com
• applebot.apple.com
• yandex.ru
• yandex.net
• yandex.com

You can edit ALLOWED_PTR_SUFFIXES in the script to match your needs.

Requirements

• bash
• fail2ban-client
• DNS tools:
  • preferred: dig
  • fallback: host for reverse DNS and getent for forward DNS

Configuration

In the script:

• LOGFILE: log destination (default: /var/log/fail2ban-unban-verified-bots.log)
• JAILS: list of jails to scan
  • empty array = all Fail2Ban jails
  • example: JAILS=(nginx-botsearch nginx-http-auth)

Usage

Run manually (usually as root):

sudo bash fail2ban-unban-verified-bots.sh

Example cron job

Run every 15 minutes:

*/15 * * * * /path/to/fail2ban-unban-verified-bots.sh

Log output example

[2026-03-10 12:34:56] UNBAN jail=nginx-botsearch ip=66.249.66.1 ptr=crawl-66-249-66-1.googlebot.com

Notes

• The script only unbans IPs that pass all FCrDNS checks.
• This reduces false positives while keeping protection active for unknown or spoofed crawlers.