Adding CrowdSec Next to fail2ban: Crowd-Sourced Threat Intel on MicroShift

I already run fail2ban on my home server, and it does its job well: it watches the mail and SSH logs, and it bans brute-forcers within a second or two. But fail2ban has two structural limits. It’s reactive — an attacker has to hit my server, and fail long enough to trip a jail, before anything happens. And it’s local — every server learns about every attacker from scratch, on its own.

My Lead Cloud Platform Engineer finally pointed me in the right direction. CrowdSec fixes both of those. It parses logs like fail2ban does, but it also participates in a network: every participating engine reports the IPs it catches (anonymised — just the offending IP and the scenario name, never your logs), and in return you pull a community blocklist of the most aggressive IPs currently attacking everyone else. That means you can block known-bad addresses before they ever touch you.

I wanted CrowdSec alongside fail2ban, not instead of it. This post is how I wired that up on a Fedora host running MicroShift — and, more usefully, the handful of things that didn’t go to plan.

The Key Idea: Split Detection From Enforcement

fail2ban is monolithic: one process reads logs, decides, and writes firewall rules. CrowdSec deliberately separates those two jobs:

• The Security Engine (agent + a “Local API”, or LAPI) reads logs, runs detection scenarios, and stores decisions.
• Bouncers are dumb enforcers. They ask the LAPI “who should I block?” and apply it — in a firewall, a web server, a CDN, wherever.

That split maps perfectly onto my setup. Everything I run — mail, Nextcloud, DNS, and so on — lives in the MicroShift cluster, managed by GitOps with Flux. So I put the brain in the cluster (a Helm release, version-controlled like everything else) and the muscle on the host (a small systemd service that writes nftables rules).

  MicroShift cluster (managed by Flux)
  ┌────────────────────────────────────────────┐
  │  LAPI  ─ stores decisions + community list  │
  │  Agent ─ parses mail / SSH / web logs       │
  └───────────────┬────────────────────────────┘
                  │  internal API only
                  ▼
  Host: firewall bouncer (systemd)
        → its own nftables table, next to
          fail2ban, geoblocking, and firewalld

The host’s firewall already had three independent nftables layers, each in its own table at its own hook priority — country-level geo-blocking, fail2ban, and firewalld. CrowdSec simply adds a fourth. Because each tool owns a separate table, they never fight over rules:

fail2ban stays completely untouched. It remains the sole enforcer for the things it’s good at (SASL mail brute-force with a database lookup, SSH). CrowdSec runs in parallel and adds behavioural detection plus the community blocklist.

Things That Didn’t Go To Plan

The architecture above is the clean version. Getting there surfaced a series of small, instructive problems — the kind that only show up once real infrastructure is involved.

1. The container image has no journalctl.

I wanted CrowdSec to watch SSH, and on a systemd host SSH logs live in the journal. CrowdSec has a journald datasource for exactly this — except it shells out to journalctl, and the CrowdSec image is Alpine-based, which has no systemd and no journalctl. The datasource errored out at startup (harmlessly — the other sources kept working, but SSH was dark).

No need for a custom image or a host change, though: rsyslog is already running and already writes auth events to /var/log/secure. Since the agent mounts the host’s /var/log read-only anyway, I just switched SSH from a journald source to a plain file source:

- source: file
  filenames: [ /var/log/secure ]
  labels: { type: syslog }

2. You cannot feed fail2ban’s bans into the community blocklist.

This was the most interesting dead end, because it’s the thing I most wanted. My instinct was: “take the IPs fail2ban already caught and contribute them to the crowd.” There’s even a natural way to try — point CrowdSec at /var/log/fail2ban.log.

It doesn’t work, and it’s worth understanding why. The community blocklist is built by a consensus engine that only trusts signals from unmodified, official detection scenarios from the CrowdSec hub. Custom parsers, tainted (edited) scenarios, and manually imported decisions are explicitly ignored by that engine — they can’t be vetted, so they don’t count. There is no official “fail2ban” collection on the hub, so anything I built to parse fail2ban.log would be custom-by-definition, and its signals would go nowhere useful.

The resolution is nicer than the original plan, though: I don’t need to launder fail2ban’s data into CrowdSec. fail2ban and CrowdSec detect largely the same attacks (SSH brute-force, SASL abuse, HTTP probing). If I just run CrowdSec’s own official collections for those surfaces, CrowdSec independently catches the same bad actors and shares them automatically and vetted. The crowd benefits from the IPs my server sees — through CrowdSec’s own detection, not through a fragile bridge. So I dropped the fail2ban ingestion entirely. However, my fail2ban jails are a little bit more sophisticated and therefore still useful for me.

3. A brand-new distro means no packaged bouncer.

The firewall bouncer is normally a distro package. My Fedora release was new enough that the vendor’s package repository had no builds for it yet (and pointing at the previous release threw GPG signature errors). Rather than fight the packaging, I used the official static binary from the project’s GitHub releases — a single self-contained Go executable, a config file, and a systemd unit. It’s version-pinnable and trivially updatable, and it sidesteps the repo problem completely.

Actually, this is something that will hopefully change in the near future because no one wants to maintain static binaries.

4. Whitelist your own networks, or you’ll lock yourself out.

A brute-force detector will happily ban you if your own traffic ever looks noisy — and I’d been bitten by exactly this with fail2ban before, when an internal address got caught. So before enabling anything, I added a parser-level whitelist for every trusted range: loopback, the host’s own IP, the pod and service networks, and the VPN subnet. Whitelisted IPs never generate an alert, never get banned, and are never shared upstream.

Proving It Works

With the engine up and the bouncer connected, the end-to-end test is satisfying and quick. Add a decision for a test IP (from the reserved documentation range) and watch it land in the firewall:

$ cscli decisions add --ip 203.0.113.66 --duration 5m --reason test
$ nft list table ip crowdsec | grep 203.0.113.66
   elements = { 203.0.113.66 timeout 4m54s expires 4m50s }   # ~8s later

Delete the decision and it disappears from the set just as fast. Detection → decision → firewall → expiry, all confirmed.

Then the payoff. As soon as the engine registered with the central API and pulled the community list, the bouncer logged:

Created set and rule for origin CAPI ...
15000 decisions added

Roughly 15,000 known-malicious IPs, pre-emptively dropped at the host firewall, before a single one of them touches a service. That’s the part fail2ban can’t give you on its own.

What I Ended Up With

• Detection in the cluster (GitOps-managed Helm release), watching mail, SSH, and web logs.
• Enforcement on the host (static-binary bouncer, its own nftables table).
• fail2ban and geo-blocking untouched, running as independent layers beside it.
• ~15k community IPs blocked pre-emptively, refreshed continuously.
• My server’s own detections fed back to the community — vetted, automatically.
• Internal and VPN traffic whitelisted so I can’t lock myself out.

A note on management, since people ask: CrowdSec (the free version) has no self-hosted web UI. You drive it with the cscli command line, and optionally enrol the engine into the hosted cloud console for dashboards. Nothing about it is exposed to the internet — the LAPI is cluster-internal only, and the console is an outbound connection to a SaaS you log into elsewhere. For a security tool guarding the front door, keeping its own attack surface at zero is the whole point.

The nicest outcome is philosophical. fail2ban made my server defend itself. CrowdSec makes it part of a herd — it learns from everyone else’s attackers, and it teaches the herd about its own. Same logs, same nftables, one extra table, and suddenly the box is no longer fighting alone.