🕵🏻‍♂️ CyberHolmes

Cyber Threat Intelligence Monitoring System
Final Year Project • Bachelor of Computer Science (Hons) (Artificial Intelligence)

---

💡 Project Overview

CyberHolmes is a full-stack, web-based Cyber Threat Intelligence (CTI) monitoring system developed as part of Final Year Project (FYP) for the completion of Bachelor of Computer Science (Honours) (Artificial Intelligence).

The project focuses on detecting and analysing posts that may pose cyber threats by applying advanced natural language processing techniques. It automates the collection of textual data from both clear and dark web forums, followed by prediction and analysis of potential threats using machine learning and deep learning models. The goal is to provide cybersecurity professionals and researchers with actionable intelligence through an interactive dashboard and evaluation, enabling more effective threat monitoring and response.

🚀 Features

1. Data Collection

• Crawls posts and discussions from both clear and dark web forums using Scrapy with Tor proxy integration to ensure anonymity.

2. Threat Prediction

• Applies machine learning model (Ensemble = LR + SGD + XGBoost) built using scikit-learn to classify text-based data and identify potential cyber threats.

3. Threat Analysis

• Utilises deep learning model, SecBERT, imported from Hugging Face to perform advanced text analysis on collected data.
• Enriches threat context using the Common Vulnerability Scoring System (CVSS) via the cvss package and MITRE ATT&CK framework data via the attackcti package.

💻 Tech Stack

📁 Folder Structure

/CyberHolmes
│
├── backend/                # Django backend application
│ |── analysis/             # Threat prediction and analysis processes (ML + DL)
│ |── config/               # Django project configuration
│ |── crawler/              # Scrapy-based crawlers for data collection
│ |── main/                 # Core app for main logics and API endpoints
│ |── .gitignore/           # Git ignore rules for backend
│ |── Dockerfile/           # Docker setup for backend service
│ |── manage.py/            # Django project management script
│ |── requirements.txt/     # Python dependencies
│ └── torrc/                # Tor configuration
|
├── frontend/               # Next.js frontend application
│ |── app/                  # App routing and page definitions
│ |── components/           # Reusable React UI components
│ |── config/               # Frontend configuration
│ |── lib/                  # Utility functions and hooks
│ |── styles/               # Global and component-specific CSS styles
│ |── .gitignore/           # Git ignore rules for frontend
│ └── Dockerfile/           # Docker setup for frontend service
|
├── nginx/                  # nginx configuration for reverse proxy
├── docker-compose.yml      # Docker container orchestration
├── LICENSE
└── README.md

⚙️ Installation & Setup

✅ Prerequisites

• Debian-based system
• Git
• Docker
• Docker Compose

🌀 1. Clone the Repository

git clone https://github.com/rydzze/CyberHolmes.git
cd CyberHolmes

🔐 2. Prepare Environment Variables

Create .env files for both backend and frontend. Use the provided templates or set your own keys.

backend/.env

DEBUG=True
SECRET_KEY=your-secret-key ¯\_(ツ)_/¯

frontend/.env

# If run locally:
NEXT_PUBLIC_API_URL=http://localhost:3001/api

# If run remotely:
NEXT_PUBLIC_API_URL=/api

🐳 3. Build and Run with Docker

From the root of the project directory:

docker-compose up --build -d

The frontend can be accessed at http://localhost:3000

The backend API can be accessed at http://localhost:3001/api

📸 Screenshots of UI

📊 Dashboard Views

Figure 1: Dashboard overview (light theme)

Figure 2: Dashboard overview (dark theme)

---

🕷️ Crawler Interface

Figure 3: Deploying crawlers to collect threat intelligence

Figure 4: Viewing collected crawler records

---

🧠 Threat Analysis - Non-Threat Examples

Figure 5: Analysis result for a non-threatening post (light theme)

Figure 6: Analysis result for a non-threatening post (dark theme)

---

🚨 Threat Analysis - Threat Examples

Figure 7: Threat detected with enriched threat context (light theme)

Figure 8: Threat detected with enriched threat context (dark theme)