Component Signature Generator

Overview

This tool implements the methodology described in Automatic Yara Rule Generation Using Biclustering, adapting the research for modern component/version identification workflows. It automates signature generation by mining high-entropy n-grams from reference binaries or class files, grouping them into rules, and measuring the impact against explicit negative corpora. By combining approximate counting (for scalability), counting bloom filters, and biclustering-inspired selection heuristics, the system assembles YARA-compatible rules tailored to your training dataset.

How It Works

1. N-gram discovery – NGramExtractor streams each positive sample, sketches the most frequent byte n-grams, and refines exact counts on promising candidates.
2. False-positive pruning – Counting bloom filters and explicit negative corpora are used to estimate how often each candidate fires outside the target component, penalising low-entropy grams and high FP rates.
3. Candidate selection – Signatures are prioritised by new coverage, FP rate, and entropy before assembling conjunctive groups that maximise coverage of the positive set.
4. Rule evaluation – Generated rules are scored against held-out positives and negatives to provide true/false-positive rates, helping you iterate quickly under a TDD workflow.

Dataset Preparation

• Positive corpus: Organise all files belonging to the target component/version into one or more directories. They should be representative of the bytes you expect to match in the wild.
• Negative corpus: Gather neighbouring components or versions that must not trigger the rule (e.g., other libraries or classes). Provide these via --neg-dir during rule generation; the tool reports FP rates against them.
• Bloom construction corpora (optional): Use bytes-to-bloom to pre-build counting bloom filters per component family. Supply datasets containing broader benign/mismatched samples so FP signals are meaningful. Ensure all inputs are accessible on disk; compressed archives should be decompressed ahead of time.

Installation

cd auto-yara-py
uv sync --extra dev

This creates a local environment with runtime and development dependencies. Use uv run to execute the commands below.

CLI Reference

auto-yara

Generate a YARA rule from component samples.

uv run python -m auto_yara.cli.auto_yara build \
  path/to/positive_dir \
  --neg-dir path/to/negative_dir \
  --output component_rule.yara \
  --gram-size 4 \
  --top-k 100 \
  --target-cover 2 \
  --min-group 2 \
  --max-fp 0.1

• --neg-dir may be repeated to include multiple negative directories; omit to skip FP evaluation.
• --gram-size and --top-k tune the n-gram search space; larger values increase recall but cost more time.
• Output includes coverage and false-positive summaries to validate component isolation.

bytes-to-bloom

Build a counting bloom filter from a corpus (useful for precomputed FP estimates).

uv run python -m auto_yara.cli.bytes_to_bloom \
  path/to/corpus \
  output/component.bloom \
  --gram-size 3 \
  --top-k 500 \
  --filter-size 32768 \
  --hash-functions 6

• --filter-size controls bloom slots; larger sizes reduce collisions at the cost of memory.
• --hash-functions defines the number of hash streams.
• The resulting .bloom files can be versioned and reused across experiments.

bloom-info

Inspect metadata about an existing bloom filter.

uv run python -m auto_yara.cli.bloom_info show output/component.bloom

Outputs entry counts, divisor (source file count), slot size, and estimated minimum counts to help assess bloom health.

Testing & Verification

Run the full test and quality suite to ensure reproducibility:

uv run ruff check src tests
uv run mypy src tests
uv run pytest

The project ships with unit tests covering bloom filters, n-gram discovery, signature selection, rule formatting, evaluation, and CLI integration so you can extend functionality confidently.

Next Steps

• Collect component datasets with clearly separated positives/negatives.
• Iterate on gram sizes and target coverage thresholds to balance recall and precision.
• Incorporate bloom filters built from broader corpora to strengthen FP suppression across unparallelled components.