fabriziosalmi/patterns

Automated OWASP CRS and Bad Bot Detection for Nginx, Apache, Traefik and HaProxy

Python

312

298 commits

updated Sep 22, 2026

See the code

README

Patterns

Patterns

Production-grade WAF rules, on autopilot.

Automated OWASP Core Rule Set and bad-bot patterns, converted into native configurations for Nginx, Apache, Traefik, and HAProxy — refreshed every day.

Latest Release Update workflow Nginx tests License Documentation

Documentation · Get started · Latest release


Why Patterns

The OWASP Core Rule Set (CRS) is the de-facto open-source rule base behind ModSecurity, but plugging it into anything other than Apache is non-trivial. Patterns automates the whole pipeline:

  1. Pull the latest CRS rules straight from upstream.
  2. Convert them into the native syntax of each web server — not a generic shim.
  3. Package the output as ready-to-deploy archives, refreshed every day by GitHub Actions.

The output covers SQL injection, XSS, RCE, LFI, RFI and protocol violations. What it stops is measured, not claimed: see What this catches.

What this catches

A ModSecurity rule is not only a regular expression. It also carries a transformation chain (t:urlDecodeUni, t:htmlEntityDecode, ...) that the pattern is written to run after, and an anomaly score that lets several weak signals accumulate before anything is refused. An nginx map has neither. It matches one regex against one raw request component, and that is all it can do.

So the converted rule set is measured against ordinary traffic and against attacks, with nginx itself, by tests/test_nginx_blocking.py. Against CRS v4.29.0, 174 emitted rules, 38 ordinary requests and 21 attacks (corpus.py):

Ordinary requests refused0 of 38
Attacks refused, sent in clear14 of 21
Attacks refused, percent-encoded3 of 21

The gap between the last two rows is the transformation chain. nginx cannot url-decode inside a map, so a pattern written to run after t:urlDecodeUni sees %3Cscript%3E where CRS would have seen <script>. Everything that depends on decoding is caught in clear and missed encoded.

Three further limits, all visible in the header of the generated waf_maps.conf:

  • Rules that would refuse ordinary traffic are not emitted. Converted without its transformations, CRS 920230 (%[0-9a-fA-F]{2}, t:urlDecodeUni) means "still percent-encoded after one decode", that is, double encoding. Against a raw URI it means "contains a percent-encoded character", which is most URLs. Seven such rules are excluded, each named in the generated file.
  • Rules that record rather than refuse are not emitted. CRS 921170 is @rx ., matches any character, declares pass, and exists to count repeated parameter names.
  • Only @rx converts. @detectSQLi and @detectXSS are libinjection, @pmFromFile is a word list, @lt/@ge are anomaly-score comparisons. None of them is a regular expression, so none can become a map key. That is why a scanner User-Agent and a request for /.env pass: CRS catches both with @pmFromFile.

This is a useful first filter in front of an application, and it is not a replacement for a WAF that can apply transformations and keep score. If you need that on nginx, use ModSecurity; on Caddy, see caddy-waf.

The other three backends have not been measured this way yet.

Highlights

OWASP CRS coverageSQLi, XSS, RCE, LFI, RFI, plus generic anomaly and protocol-violation rules.
Native outputNginx map/if, Apache SecRule, Traefik middleware TOML, HAProxy ACL files.
Bad-bot blockingCurated User-Agent lists from public sources, with safe defaults that do not block major search engines.
Daily refreshA scheduled GitHub Actions workflow rebuilds every backend and publishes a fresh release.
Pre-built archivesSkip the toolchain — download nginx_waf.zip, apache_waf.zip, traefik_waf.zip, or haproxy_waf.zip.
ComposableEach backend is a small Python converter on top of one JSON intermediate. Adding a new platform is a few hundred lines.

Using Caddy? See the dedicated caddy-waf project.

Quick start

Option 1 — download a pre-built release

# Pick the archive that matches your stack
curl -LO https://github.com/fabriziosalmi/patterns/releases/latest/download/nginx_waf.zip
unzip nginx_waf.zip -d /etc/nginx/waf_patterns

Then follow the Nginx, Apache, Traefik, or HAProxy integration guide.

Option 2 — build from source

Requires Python 3.9+, pip, and git.

git clone https://github.com/fabriziosalmi/patterns.git
cd patterns
pip install -r requirements.txt

python owasp2json.py            # 1. Fetch the latest OWASP CRS into owasp_rules.json
python json2nginx.py            # 2. Convert into Nginx WAF config
python json2apache.py           #    …or Apache (ModSecurity)
python json2traefik.py          #    …or Traefik middleware
python json2haproxy.py          #    …or HAProxy ACL files
python badbots.py               # 3. Generate bad-bot blocklists

Generated files land in waf_patterns/<platform>/.

Architecture

   ┌─────────────────────┐    daily cron     ┌──────────────────────┐
   │ coreruleset/        │ ───────────────▶  │ owasp2json.py        │
   │ coreruleset (GH)    │                   │   → owasp_rules.json │
   └─────────────────────┘                   └──────────┬───────────┘
                                                        │
            ┌─────────────────┬──────────────────┬──────┴──────────┐
            ▼                 ▼                  ▼                 ▼
      json2nginx.py    json2apache.py    json2traefik.py    json2haproxy.py
            │                 │                  │                 │
            ▼                 ▼                  ▼                 ▼
       nginx_waf.zip    apache_waf.zip    traefik_waf.zip    haproxy_waf.zip
                          (published as a GitHub Release)

Each converter is independent, idempotent, and configured exclusively through environment variables (INPUT_FILE, OUTPUT_DIR). Full reference at docs/api.

Repository layout

patterns/
├── owasp2json.py            # Pull and parse OWASP CRS into a JSON intermediate
├── json2nginx.py            # JSON → Nginx (map + if directives)
├── json2apache.py           # JSON → Apache (ModSecurity SecRule)
├── json2traefik.py          # JSON → Traefik (middleware TOML)
├── json2haproxy.py          # JSON → HAProxy (ACL files)
├── badbots.py               # Public bot lists → per-platform blocklists
├── import_*_waf.py          # Optional installers for each platform
├── waf_patterns/            # Generated outputs
│   ├── nginx/
│   ├── apache/
│   ├── traefik/
│   └── haproxy/
├── docs/                    # VitePress documentation site
├── tests/                   # Validation tests for each backend
└── .github/workflows/       # Daily build + release automation

Integration in 60 seconds

Nginx

http {
    include /etc/nginx/waf_patterns/nginx/waf_maps.conf;
    include /etc/nginx/waf_patterns/nginx/bots.conf;
}
server {
    include /etc/nginx/waf_patterns/nginx/waf_rules.conf;
    if ($bad_bot) { return 403; }
}

Apache (ModSecurity)

<IfModule security2_module>
    SecRuleEngine On
    Include /etc/apache2/waf_patterns/apache/*.conf
</IfModule>

Traefik

http:
  routers:
    app:
      rule: "Host(`example.com`)"
      service: app
      middlewares: [waf-protection@file, bot-blocker@file]

HAProxy

frontend http-in
    bind *:80
    acl waf_match path,url_dec -m reg -i -f /etc/haproxy/waf.acl
    acl bad_bot   hdr(User-Agent) -m reg -i -f /etc/haproxy/bots.acl
    http-request deny deny_status 403 if waf_match || bad_bot

Full guides — with logging, whitelists, and tuning — live in the docs.

Bad-bot example output (Nginx)

map $http_user_agent $bad_bot {
    default 0;
    "~*AhrefsBot"  1;
    "~*SemrushBot" 1;
    "~*MJ12bot"    1;
    "~*GPTBot"     1;
}

if ($bad_bot) { return 403; }

The default list blocks SEO crawlers, AI training bots, and known scanners while explicitly allowing major search engines (Google, Bing, DuckDuckGo, Yandex, Baidu).

Automation

WorkflowSchedulePurpose
update_patterns.ymlDaily + manualRe-fetch CRS, regenerate every backend, publish a release
test_nginx.ymlOn PRValidate generated Nginx rules against a live container
test_apache_docker.ymlOn PRValidate generated Apache rules against ModSecurity in Docker
docs.ymlOn docs/ changeBuild and deploy the VitePress docs to GitHub Pages

All workflows run on GitHub-hosted runners (ubuntu-latest).

Documentation

The full documentation lives at fabriziosalmi.github.io/patterns — built with VitePress and deployed automatically.

Commercial support & consulting

Running these WAF rules in production? I offer paid support, custom rule development, and security consulting - WAF tuning, hardening, TLS automation, and cloud detection & alerting. Reach out: fabrizio.salmi@gmail.com.

Contributing

  1. Fork the repository.
  2. Create a feature branch: git checkout -b feature/your-change.
  3. Commit and push.
  4. Open a pull request — the test workflows will run automatically.

See CONTRIBUTING.md for details and SECURITY.md for the disclosure policy.

License

The original code of this project — the Python converters, the documentation, and the tests — is released under the MIT License (Copyright © Fabrizio Salmi).

The generated data is not covered by that MIT grant. owasp_rules.json and everything under waf_patterns/** are derived/converted from third-party sources — chiefly the OWASP Core Rule Set (Apache-2.0), plus public bad-bot and referrer-spam lists — and are redistributed under those upstream licenses, not under MIT. See THIRD_PARTY_NOTICES.md for the source-by-source breakdown and LICENSES/ for the required license texts.

Resources


Built and maintained by Fabrizio Salmi.
apache
bad-requests
bot-detection
caddy
caddyserver
crs
firewall-configuration
firewall-rules
malicious-url-detection
mod-security
nginx
owasp
waf
web-application-firewall

Contributors

fabriziosalmi/patterns

Automated OWASP CRS and Bad Bot Detection for Nginx, Apache, Traefik and HaProxy

Python

312

298 commits

updated Sep 22, 2026

See the code

README

Patterns

Patterns

Production-grade WAF rules, on autopilot.

Automated OWASP Core Rule Set and bad-bot patterns, converted into native configurations for Nginx, Apache, Traefik, and HAProxy — refreshed every day.

Latest Release Update workflow Nginx tests License Documentation

Documentation · Get started · Latest release


Why Patterns

The OWASP Core Rule Set (CRS) is the de-facto open-source rule base behind ModSecurity, but plugging it into anything other than Apache is non-trivial. Patterns automates the whole pipeline:

  1. Pull the latest CRS rules straight from upstream.
  2. Convert them into the native syntax of each web server — not a generic shim.
  3. Package the output as ready-to-deploy archives, refreshed every day by GitHub Actions.

The output covers SQL injection, XSS, RCE, LFI, RFI and protocol violations. What it stops is measured, not claimed: see What this catches.

What this catches

A ModSecurity rule is not only a regular expression. It also carries a transformation chain (t:urlDecodeUni, t:htmlEntityDecode, ...) that the pattern is written to run after, and an anomaly score that lets several weak signals accumulate before anything is refused. An nginx map has neither. It matches one regex against one raw request component, and that is all it can do.

So the converted rule set is measured against ordinary traffic and against attacks, with nginx itself, by tests/test_nginx_blocking.py. Against CRS v4.29.0, 174 emitted rules, 38 ordinary requests and 21 attacks (corpus.py):

Ordinary requests refused0 of 38
Attacks refused, sent in clear14 of 21
Attacks refused, percent-encoded3 of 21

The gap between the last two rows is the transformation chain. nginx cannot url-decode inside a map, so a pattern written to run after t:urlDecodeUni sees %3Cscript%3E where CRS would have seen <script>. Everything that depends on decoding is caught in clear and missed encoded.

Three further limits, all visible in the header of the generated waf_maps.conf:

  • Rules that would refuse ordinary traffic are not emitted. Converted without its transformations, CRS 920230 (%[0-9a-fA-F]{2}, t:urlDecodeUni) means "still percent-encoded after one decode", that is, double encoding. Against a raw URI it means "contains a percent-encoded character", which is most URLs. Seven such rules are excluded, each named in the generated file.
  • Rules that record rather than refuse are not emitted. CRS 921170 is @rx ., matches any character, declares pass, and exists to count repeated parameter names.
  • Only @rx converts. @detectSQLi and @detectXSS are libinjection, @pmFromFile is a word list, @lt/@ge are anomaly-score comparisons. None of them is a regular expression, so none can become a map key. That is why a scanner User-Agent and a request for /.env pass: CRS catches both with @pmFromFile.

This is a useful first filter in front of an application, and it is not a replacement for a WAF that can apply transformations and keep score. If you need that on nginx, use ModSecurity; on Caddy, see caddy-waf.

The other three backends have not been measured this way yet.

Highlights

OWASP CRS coverageSQLi, XSS, RCE, LFI, RFI, plus generic anomaly and protocol-violation rules.
Native outputNginx map/if, Apache SecRule, Traefik middleware TOML, HAProxy ACL files.
Bad-bot blockingCurated User-Agent lists from public sources, with safe defaults that do not block major search engines.
Daily refreshA scheduled GitHub Actions workflow rebuilds every backend and publishes a fresh release.
Pre-built archivesSkip the toolchain — download nginx_waf.zip, apache_waf.zip, traefik_waf.zip, or haproxy_waf.zip.
ComposableEach backend is a small Python converter on top of one JSON intermediate. Adding a new platform is a few hundred lines.

Using Caddy? See the dedicated caddy-waf project.

Quick start

Option 1 — download a pre-built release

# Pick the archive that matches your stack
curl -LO https://github.com/fabriziosalmi/patterns/releases/latest/download/nginx_waf.zip
unzip nginx_waf.zip -d /etc/nginx/waf_patterns

Then follow the Nginx, Apache, Traefik, or HAProxy integration guide.

Option 2 — build from source

Requires Python 3.9+, pip, and git.

git clone https://github.com/fabriziosalmi/patterns.git
cd patterns
pip install -r requirements.txt

python owasp2json.py            # 1. Fetch the latest OWASP CRS into owasp_rules.json
python json2nginx.py            # 2. Convert into Nginx WAF config
python json2apache.py           #    …or Apache (ModSecurity)
python json2traefik.py          #    …or Traefik middleware
python json2haproxy.py          #    …or HAProxy ACL files
python badbots.py               # 3. Generate bad-bot blocklists

Generated files land in waf_patterns/<platform>/.

Architecture

   ┌─────────────────────┐    daily cron     ┌──────────────────────┐
   │ coreruleset/        │ ───────────────▶  │ owasp2json.py        │
   │ coreruleset (GH)    │                   │   → owasp_rules.json │
   └─────────────────────┘                   └──────────┬───────────┘
                                                        │
            ┌─────────────────┬──────────────────┬──────┴──────────┐
            ▼                 ▼                  ▼                 ▼
      json2nginx.py    json2apache.py    json2traefik.py    json2haproxy.py
            │                 │                  │                 │
            ▼                 ▼                  ▼                 ▼
       nginx_waf.zip    apache_waf.zip    traefik_waf.zip    haproxy_waf.zip
                          (published as a GitHub Release)

Each converter is independent, idempotent, and configured exclusively through environment variables (INPUT_FILE, OUTPUT_DIR). Full reference at docs/api.

Repository layout

patterns/
├── owasp2json.py            # Pull and parse OWASP CRS into a JSON intermediate
├── json2nginx.py            # JSON → Nginx (map + if directives)
├── json2apache.py           # JSON → Apache (ModSecurity SecRule)
├── json2traefik.py          # JSON → Traefik (middleware TOML)
├── json2haproxy.py          # JSON → HAProxy (ACL files)
├── badbots.py               # Public bot lists → per-platform blocklists
├── import_*_waf.py          # Optional installers for each platform
├── waf_patterns/            # Generated outputs
│   ├── nginx/
│   ├── apache/
│   ├── traefik/
│   └── haproxy/
├── docs/                    # VitePress documentation site
├── tests/                   # Validation tests for each backend
└── .github/workflows/       # Daily build + release automation

Integration in 60 seconds

Nginx

http {
    include /etc/nginx/waf_patterns/nginx/waf_maps.conf;
    include /etc/nginx/waf_patterns/nginx/bots.conf;
}
server {
    include /etc/nginx/waf_patterns/nginx/waf_rules.conf;
    if ($bad_bot) { return 403; }
}

Apache (ModSecurity)

<IfModule security2_module>
    SecRuleEngine On
    Include /etc/apache2/waf_patterns/apache/*.conf
</IfModule>

Traefik

http:
  routers:
    app:
      rule: "Host(`example.com`)"
      service: app
      middlewares: [waf-protection@file, bot-blocker@file]

HAProxy

frontend http-in
    bind *:80
    acl waf_match path,url_dec -m reg -i -f /etc/haproxy/waf.acl
    acl bad_bot   hdr(User-Agent) -m reg -i -f /etc/haproxy/bots.acl
    http-request deny deny_status 403 if waf_match || bad_bot

Full guides — with logging, whitelists, and tuning — live in the docs.

Bad-bot example output (Nginx)

map $http_user_agent $bad_bot {
    default 0;
    "~*AhrefsBot"  1;
    "~*SemrushBot" 1;
    "~*MJ12bot"    1;
    "~*GPTBot"     1;
}

if ($bad_bot) { return 403; }

The default list blocks SEO crawlers, AI training bots, and known scanners while explicitly allowing major search engines (Google, Bing, DuckDuckGo, Yandex, Baidu).

Automation

WorkflowSchedulePurpose
update_patterns.ymlDaily + manualRe-fetch CRS, regenerate every backend, publish a release
test_nginx.ymlOn PRValidate generated Nginx rules against a live container
test_apache_docker.ymlOn PRValidate generated Apache rules against ModSecurity in Docker
docs.ymlOn docs/ changeBuild and deploy the VitePress docs to GitHub Pages

All workflows run on GitHub-hosted runners (ubuntu-latest).

Documentation

The full documentation lives at fabriziosalmi.github.io/patterns — built with VitePress and deployed automatically.

Commercial support & consulting

Running these WAF rules in production? I offer paid support, custom rule development, and security consulting - WAF tuning, hardening, TLS automation, and cloud detection & alerting. Reach out: fabrizio.salmi@gmail.com.

Contributing

  1. Fork the repository.
  2. Create a feature branch: git checkout -b feature/your-change.
  3. Commit and push.
  4. Open a pull request — the test workflows will run automatically.

See CONTRIBUTING.md for details and SECURITY.md for the disclosure policy.

License

The original code of this project — the Python converters, the documentation, and the tests — is released under the MIT License (Copyright © Fabrizio Salmi).

The generated data is not covered by that MIT grant. owasp_rules.json and everything under waf_patterns/** are derived/converted from third-party sources — chiefly the OWASP Core Rule Set (Apache-2.0), plus public bad-bot and referrer-spam lists — and are redistributed under those upstream licenses, not under MIT. See THIRD_PARTY_NOTICES.md for the source-by-source breakdown and LICENSES/ for the required license texts.

Resources


Built and maintained by Fabrizio Salmi.
apache
bad-requests
bot-detection
caddy
caddyserver
crs
firewall-configuration
firewall-rules
malicious-url-detection
mod-security
nginx
owasp
waf
web-application-firewall

Contributors

Languages

Python

100.0%