Automated OWASP CRS and Bad Bot Detection for Nginx, Apache, Traefik and HaProxy
See the codeProduction-grade WAF rules, on autopilot.
Automated OWASP Core Rule Set and bad-bot patterns, converted into native configurations for Nginx, Apache, Traefik, and HAProxy — refreshed every day.
The OWASP Core Rule Set (CRS) is the de-facto open-source rule base behind ModSecurity, but plugging it into anything other than Apache is non-trivial. Patterns automates the whole pipeline:
The output covers SQL injection, XSS, RCE, LFI, RFI and protocol violations. What it stops is measured, not claimed: see What this catches.
A ModSecurity rule is not only a regular expression. It also carries a
transformation chain (t:urlDecodeUni, t:htmlEntityDecode, ...) that the
pattern is written to run after, and an anomaly score that lets several weak
signals accumulate before anything is refused. An nginx map has neither. It
matches one regex against one raw request component, and that is all it can do.
So the converted rule set is measured against ordinary traffic and against
attacks, with nginx itself, by tests/test_nginx_blocking.py.
Against CRS v4.29.0, 174 emitted rules, 38 ordinary requests and 21 attacks
(corpus.py):
| Ordinary requests refused | 0 of 38 |
| Attacks refused, sent in clear | 14 of 21 |
| Attacks refused, percent-encoded | 3 of 21 |
The gap between the last two rows is the transformation chain. nginx cannot
url-decode inside a map, so a pattern written to run after t:urlDecodeUni
sees %3Cscript%3E where CRS would have seen <script>. Everything that
depends on decoding is caught in clear and missed encoded.
Three further limits, all visible in the header of the generated
waf_maps.conf:
%[0-9a-fA-F]{2}, t:urlDecodeUni)
means "still percent-encoded after one decode", that is, double encoding.
Against a raw URI it means "contains a percent-encoded character", which is
most URLs. Seven such rules are excluded, each named in the generated file.@rx ., matches any character, declares pass, and exists to count repeated
parameter names.@rx converts. @detectSQLi and @detectXSS are libinjection,
@pmFromFile is a word list, @lt/@ge are anomaly-score comparisons.
None of them is a regular expression, so none can become a map key. That is
why a scanner User-Agent and a request for /.env pass: CRS catches both with
@pmFromFile.This is a useful first filter in front of an application, and it is not a
replacement for a WAF that can apply transformations and keep score. If you need
that on nginx, use ModSecurity; on Caddy, see
caddy-waf.
The other three backends have not been measured this way yet.
| OWASP CRS coverage | SQLi, XSS, RCE, LFI, RFI, plus generic anomaly and protocol-violation rules. |
| Native output | Nginx map/if, Apache SecRule, Traefik middleware TOML, HAProxy ACL files. |
| Bad-bot blocking | Curated User-Agent lists from public sources, with safe defaults that do not block major search engines. |
| Daily refresh | A scheduled GitHub Actions workflow rebuilds every backend and publishes a fresh release. |
| Pre-built archives | Skip the toolchain — download nginx_waf.zip, apache_waf.zip, traefik_waf.zip, or haproxy_waf.zip. |
| Composable | Each backend is a small Python converter on top of one JSON intermediate. Adding a new platform is a few hundred lines. |
Using Caddy? See the dedicated
caddy-wafproject.
# Pick the archive that matches your stack
curl -LO https://github.com/fabriziosalmi/patterns/releases/latest/download/nginx_waf.zip
unzip nginx_waf.zip -d /etc/nginx/waf_patterns
Then follow the Nginx, Apache, Traefik, or HAProxy integration guide.
Requires Python 3.9+, pip, and git.
git clone https://github.com/fabriziosalmi/patterns.git
cd patterns
pip install -r requirements.txt
python owasp2json.py # 1. Fetch the latest OWASP CRS into owasp_rules.json
python json2nginx.py # 2. Convert into Nginx WAF config
python json2apache.py # …or Apache (ModSecurity)
python json2traefik.py # …or Traefik middleware
python json2haproxy.py # …or HAProxy ACL files
python badbots.py # 3. Generate bad-bot blocklists
Generated files land in waf_patterns/<platform>/.
┌─────────────────────┐ daily cron ┌──────────────────────┐
│ coreruleset/ │ ───────────────▶ │ owasp2json.py │
│ coreruleset (GH) │ │ → owasp_rules.json │
└─────────────────────┘ └──────────┬───────────┘
│
┌─────────────────┬──────────────────┬──────┴──────────┐
▼ ▼ ▼ ▼
json2nginx.py json2apache.py json2traefik.py json2haproxy.py
│ │ │ │
▼ ▼ ▼ ▼
nginx_waf.zip apache_waf.zip traefik_waf.zip haproxy_waf.zip
(published as a GitHub Release)
Each converter is independent, idempotent, and configured exclusively through environment variables (INPUT_FILE, OUTPUT_DIR). Full reference at docs/api.
patterns/
├── owasp2json.py # Pull and parse OWASP CRS into a JSON intermediate
├── json2nginx.py # JSON → Nginx (map + if directives)
├── json2apache.py # JSON → Apache (ModSecurity SecRule)
├── json2traefik.py # JSON → Traefik (middleware TOML)
├── json2haproxy.py # JSON → HAProxy (ACL files)
├── badbots.py # Public bot lists → per-platform blocklists
├── import_*_waf.py # Optional installers for each platform
├── waf_patterns/ # Generated outputs
│ ├── nginx/
│ ├── apache/
│ ├── traefik/
│ └── haproxy/
├── docs/ # VitePress documentation site
├── tests/ # Validation tests for each backend
└── .github/workflows/ # Daily build + release automation
http {
include /etc/nginx/waf_patterns/nginx/waf_maps.conf;
include /etc/nginx/waf_patterns/nginx/bots.conf;
}
server {
include /etc/nginx/waf_patterns/nginx/waf_rules.conf;
if ($bad_bot) { return 403; }
}
<IfModule security2_module>
SecRuleEngine On
Include /etc/apache2/waf_patterns/apache/*.conf
</IfModule>
http:
routers:
app:
rule: "Host(`example.com`)"
service: app
middlewares: [waf-protection@file, bot-blocker@file]
frontend http-in
bind *:80
acl waf_match path,url_dec -m reg -i -f /etc/haproxy/waf.acl
acl bad_bot hdr(User-Agent) -m reg -i -f /etc/haproxy/bots.acl
http-request deny deny_status 403 if waf_match || bad_bot
Full guides — with logging, whitelists, and tuning — live in the docs.
map $http_user_agent $bad_bot {
default 0;
"~*AhrefsBot" 1;
"~*SemrushBot" 1;
"~*MJ12bot" 1;
"~*GPTBot" 1;
}
if ($bad_bot) { return 403; }
The default list blocks SEO crawlers, AI training bots, and known scanners while explicitly allowing major search engines (Google, Bing, DuckDuckGo, Yandex, Baidu).
| Workflow | Schedule | Purpose |
|---|---|---|
update_patterns.yml | Daily + manual | Re-fetch CRS, regenerate every backend, publish a release |
test_nginx.yml | On PR | Validate generated Nginx rules against a live container |
test_apache_docker.yml | On PR | Validate generated Apache rules against ModSecurity in Docker |
docs.yml | On docs/ change | Build and deploy the VitePress docs to GitHub Pages |
All workflows run on GitHub-hosted runners (ubuntu-latest).
The full documentation lives at fabriziosalmi.github.io/patterns — built with VitePress and deployed automatically.
Running these WAF rules in production? I offer paid support, custom rule development, and security consulting - WAF tuning, hardening, TLS automation, and cloud detection & alerting. Reach out: fabrizio.salmi@gmail.com.
git checkout -b feature/your-change.See CONTRIBUTING.md for details and SECURITY.md for the disclosure policy.
The original code of this project — the Python converters, the documentation, and the tests — is released under the MIT License (Copyright © Fabrizio Salmi).
The generated data is not covered by that MIT grant. owasp_rules.json and
everything under waf_patterns/** are derived/converted from third-party
sources — chiefly the OWASP Core Rule Set
(Apache-2.0), plus public bad-bot and referrer-spam lists — and are
redistributed under those upstream licenses, not under MIT. See
THIRD_PARTY_NOTICES.md for the source-by-source
breakdown and LICENSES/ for the required license texts.
208 commits
70 commits
9 commits
5 commits
Python
100.0%
Automated OWASP CRS and Bad Bot Detection for Nginx, Apache, Traefik and HaProxy
See the codeProduction-grade WAF rules, on autopilot.
Automated OWASP Core Rule Set and bad-bot patterns, converted into native configurations for Nginx, Apache, Traefik, and HAProxy — refreshed every day.
The OWASP Core Rule Set (CRS) is the de-facto open-source rule base behind ModSecurity, but plugging it into anything other than Apache is non-trivial. Patterns automates the whole pipeline:
The output covers SQL injection, XSS, RCE, LFI, RFI and protocol violations. What it stops is measured, not claimed: see What this catches.
A ModSecurity rule is not only a regular expression. It also carries a
transformation chain (t:urlDecodeUni, t:htmlEntityDecode, ...) that the
pattern is written to run after, and an anomaly score that lets several weak
signals accumulate before anything is refused. An nginx map has neither. It
matches one regex against one raw request component, and that is all it can do.
So the converted rule set is measured against ordinary traffic and against
attacks, with nginx itself, by tests/test_nginx_blocking.py.
Against CRS v4.29.0, 174 emitted rules, 38 ordinary requests and 21 attacks
(corpus.py):
| Ordinary requests refused | 0 of 38 |
| Attacks refused, sent in clear | 14 of 21 |
| Attacks refused, percent-encoded | 3 of 21 |
The gap between the last two rows is the transformation chain. nginx cannot
url-decode inside a map, so a pattern written to run after t:urlDecodeUni
sees %3Cscript%3E where CRS would have seen <script>. Everything that
depends on decoding is caught in clear and missed encoded.
Three further limits, all visible in the header of the generated
waf_maps.conf:
%[0-9a-fA-F]{2}, t:urlDecodeUni)
means "still percent-encoded after one decode", that is, double encoding.
Against a raw URI it means "contains a percent-encoded character", which is
most URLs. Seven such rules are excluded, each named in the generated file.@rx ., matches any character, declares pass, and exists to count repeated
parameter names.@rx converts. @detectSQLi and @detectXSS are libinjection,
@pmFromFile is a word list, @lt/@ge are anomaly-score comparisons.
None of them is a regular expression, so none can become a map key. That is
why a scanner User-Agent and a request for /.env pass: CRS catches both with
@pmFromFile.This is a useful first filter in front of an application, and it is not a
replacement for a WAF that can apply transformations and keep score. If you need
that on nginx, use ModSecurity; on Caddy, see
caddy-waf.
The other three backends have not been measured this way yet.
| OWASP CRS coverage | SQLi, XSS, RCE, LFI, RFI, plus generic anomaly and protocol-violation rules. |
| Native output | Nginx map/if, Apache SecRule, Traefik middleware TOML, HAProxy ACL files. |
| Bad-bot blocking | Curated User-Agent lists from public sources, with safe defaults that do not block major search engines. |
| Daily refresh | A scheduled GitHub Actions workflow rebuilds every backend and publishes a fresh release. |
| Pre-built archives | Skip the toolchain — download nginx_waf.zip, apache_waf.zip, traefik_waf.zip, or haproxy_waf.zip. |
| Composable | Each backend is a small Python converter on top of one JSON intermediate. Adding a new platform is a few hundred lines. |
Using Caddy? See the dedicated
caddy-wafproject.
# Pick the archive that matches your stack
curl -LO https://github.com/fabriziosalmi/patterns/releases/latest/download/nginx_waf.zip
unzip nginx_waf.zip -d /etc/nginx/waf_patterns
Then follow the Nginx, Apache, Traefik, or HAProxy integration guide.
Requires Python 3.9+, pip, and git.
git clone https://github.com/fabriziosalmi/patterns.git
cd patterns
pip install -r requirements.txt
python owasp2json.py # 1. Fetch the latest OWASP CRS into owasp_rules.json
python json2nginx.py # 2. Convert into Nginx WAF config
python json2apache.py # …or Apache (ModSecurity)
python json2traefik.py # …or Traefik middleware
python json2haproxy.py # …or HAProxy ACL files
python badbots.py # 3. Generate bad-bot blocklists
Generated files land in waf_patterns/<platform>/.
┌─────────────────────┐ daily cron ┌──────────────────────┐
│ coreruleset/ │ ───────────────▶ │ owasp2json.py │
│ coreruleset (GH) │ │ → owasp_rules.json │
└─────────────────────┘ └──────────┬───────────┘
│
┌─────────────────┬──────────────────┬──────┴──────────┐
▼ ▼ ▼ ▼
json2nginx.py json2apache.py json2traefik.py json2haproxy.py
│ │ │ │
▼ ▼ ▼ ▼
nginx_waf.zip apache_waf.zip traefik_waf.zip haproxy_waf.zip
(published as a GitHub Release)
Each converter is independent, idempotent, and configured exclusively through environment variables (INPUT_FILE, OUTPUT_DIR). Full reference at docs/api.
patterns/
├── owasp2json.py # Pull and parse OWASP CRS into a JSON intermediate
├── json2nginx.py # JSON → Nginx (map + if directives)
├── json2apache.py # JSON → Apache (ModSecurity SecRule)
├── json2traefik.py # JSON → Traefik (middleware TOML)
├── json2haproxy.py # JSON → HAProxy (ACL files)
├── badbots.py # Public bot lists → per-platform blocklists
├── import_*_waf.py # Optional installers for each platform
├── waf_patterns/ # Generated outputs
│ ├── nginx/
│ ├── apache/
│ ├── traefik/
│ └── haproxy/
├── docs/ # VitePress documentation site
├── tests/ # Validation tests for each backend
└── .github/workflows/ # Daily build + release automation
http {
include /etc/nginx/waf_patterns/nginx/waf_maps.conf;
include /etc/nginx/waf_patterns/nginx/bots.conf;
}
server {
include /etc/nginx/waf_patterns/nginx/waf_rules.conf;
if ($bad_bot) { return 403; }
}
<IfModule security2_module>
SecRuleEngine On
Include /etc/apache2/waf_patterns/apache/*.conf
</IfModule>
http:
routers:
app:
rule: "Host(`example.com`)"
service: app
middlewares: [waf-protection@file, bot-blocker@file]
frontend http-in
bind *:80
acl waf_match path,url_dec -m reg -i -f /etc/haproxy/waf.acl
acl bad_bot hdr(User-Agent) -m reg -i -f /etc/haproxy/bots.acl
http-request deny deny_status 403 if waf_match || bad_bot
Full guides — with logging, whitelists, and tuning — live in the docs.
map $http_user_agent $bad_bot {
default 0;
"~*AhrefsBot" 1;
"~*SemrushBot" 1;
"~*MJ12bot" 1;
"~*GPTBot" 1;
}
if ($bad_bot) { return 403; }
The default list blocks SEO crawlers, AI training bots, and known scanners while explicitly allowing major search engines (Google, Bing, DuckDuckGo, Yandex, Baidu).
| Workflow | Schedule | Purpose |
|---|---|---|
update_patterns.yml | Daily + manual | Re-fetch CRS, regenerate every backend, publish a release |
test_nginx.yml | On PR | Validate generated Nginx rules against a live container |
test_apache_docker.yml | On PR | Validate generated Apache rules against ModSecurity in Docker |
docs.yml | On docs/ change | Build and deploy the VitePress docs to GitHub Pages |
All workflows run on GitHub-hosted runners (ubuntu-latest).
The full documentation lives at fabriziosalmi.github.io/patterns — built with VitePress and deployed automatically.
Running these WAF rules in production? I offer paid support, custom rule development, and security consulting - WAF tuning, hardening, TLS automation, and cloud detection & alerting. Reach out: fabrizio.salmi@gmail.com.
git checkout -b feature/your-change.See CONTRIBUTING.md for details and SECURITY.md for the disclosure policy.
The original code of this project — the Python converters, the documentation, and the tests — is released under the MIT License (Copyright © Fabrizio Salmi).
The generated data is not covered by that MIT grant. owasp_rules.json and
everything under waf_patterns/** are derived/converted from third-party
sources — chiefly the OWASP Core Rule Set
(Apache-2.0), plus public bad-bot and referrer-spam lists — and are
redistributed under those upstream licenses, not under MIT. See
THIRD_PARTY_NOTICES.md for the source-by-source
breakdown and LICENSES/ for the required license texts.
208 commits
70 commits
9 commits
5 commits
Python
100.0%