ermis-ingestion

Scraping and enrichment pipeline for the ERMIS Cyber Threat Intelligence platform. Scrapes curated security blogs and GitHub-hosted CTI reports, runs IOC/NER/ATT&CK extraction via ermis-common, performs Gemini-based relation extraction, and exports results to PostgreSQL, Neo4j, MISP, and OpenCTI.

This service is built and run as the ingestion container defined in ermis-deploy (docker compose --profile tools run --rm ingestion). See that repository for the full installation guide.

Layout

├── scraping/
│   ├── core/        # Scraper engine, HTTP utilities, ML gates
│   └── adapters/     # Per-site scrapers (Securelist, SentinelOne, etc.)
├── parsers/          # IoC extraction, MISP/OpenCTI export
├── ml/                # Custom CTI-BERT model weights & training/cross-validation scripts
├── main.py            # CLI entrypoint for scraping
└── run_ingestion.sh   # Full pipeline script (scrape → extract → export)

Manual run (CLI)

python main.py sites.txt --ignore-robots