PrivacyTotal: Final Year Project Deliverable

Omar Ramadan – C00286349 | BSc (Hons) Cybercrime and IT Security | Supervisor: Mark Cummins South East Technological University | April 2026

---

Deliverables Checklist

---

Table of Contents

1. Project Overview
2. Directory Structure
3. Quick Start
4. Research Report vs. Implementation: In-Depth Comparison
   • 1. Scope: Number of Applications
   • 2. Model Training Strategy
   • 3. Training Dataset
   • 4. Training Configuration & Resources
   • 5. APK & Data Acquisition Pipeline
   • 6. Privacy Health Score (PHS) Algorithm
   • 7. Web Application
   • 8. Evaluation & Validation
   • 9. Ethical Considerations
   • 10. Tools & Frameworks
5. Final Results Summary
6. Key Deviations and Rationale

---

Project Overview

PrivacyTotal is a static analysis tool for Android applications that automatically compares an app's declared APK permissions against its published privacy policy. Using a fine-tuned MobileLLaMA 2.7B language model, the system determines whether each permission is disclosed in the policy, assigns a Privacy Health Score (PHS), and presents the results through a web application.

The tool was motivated by an RTÉ PrimeTime investigation (September 2025) revealing that precise location data from tens of thousands of Irish smartphones was being sold by data brokers, highlighting that most users blindly accept privacy policies without reading them.

---

Directory Structure

PrivacyTotal/
├── README.md                         This document
├── LICENSE                           MIT licence
├── Research Report.pdf               Final research report
├── ProjectReport.pdf                 Final project report
├── requirements.txt                  Python dependencies
├── eval_f1.py                        F1 evaluation script (keyword-oracle)
│
├── src/
│   ├── end_to_end_gap_analysis.py    Main pipeline: APK download → permission
│   │                                 extraction → policy scraping → LLM gap
│   │                                 analysis → PHS → DB save
│   ├── batch_runner.py               Batch runner for sequential multi-app analysis
│   ├── database.py                   SQLite layer (gap_analysis.db schema, CRUD)
│   ├── acquire_app_artifacts.py      Standalone APK + metadata acquisition script
│   ├── train_gap_scratch.py          QLoRA training script (final working version)
│   ├── prepare_gap_training_v2.py    Builds gap_dataset_v2.jsonl from OPP-115 data
│   │
│   ├── models/
│   │   └── mobilellama_gap_scratch/  Working LoRA adapter (trained from scratch)
│   │       ├── adapter_config.json
│   │       ├── adapter_model.bin     ~20MB LoRA weights
│   │       ├── tokenizer.json
│   │       ├── tokenizer.model
│   │       └── tokenizer_config.json
│   │
│   └── acquired/                     Per-app result folders (117 apps)
│       └── com.example.app/
│           └── YYYYMMDD_HHMMSS/
│               └── gap_report.csv    Permission × coverage × evidence × PHS
│
├── data/
│   ├── gap_analysis.db               SQLite database (117 apps, 175 runs)
│   └── processed/
│       └── gap_dataset_v2.jsonl      Training dataset (5,154 examples)
│
└── webapp/
    ├── web_app.py                    Flask application (PrivacyTotal)
    └── templates/
        ├── base.html                 Shared layout, navigation, CSS
        ├── index.html                Home: Analyze App / Browse DB / Research Report
        ├── app.html                  Per-app: PHS gauge, coverage donut, permission table
        ├── run.html                  Single run view with gap report details
        ├── diff.html                 Policy text diff between two runs
        └── job.html                  Live job status / log streaming

---

Quick Start

Prerequisites

• Python 3.11
• NVIDIA GPU with 8GB+ VRAM (RTX 2060 Super or equivalent)
• Windows or Linux with Git Bash

Setup

# Create virtual environment
python -m venv venv311
source venv311/Scripts/activate  # Windows Git Bash
pip install -r requirements.txt

# Download base model (MobileLLaMA-2.7B-Chat) from HuggingFace
# The LoRA adapter in src/models/mobilellama_gap_scratch/ is already trained

Run the Web Application

cd PrivacyTotal/
source venv311/Scripts/activate
python webapp/web_app.py
# Opens at http://localhost:5000

Analyze a Single App

python src/end_to_end_gap_analysis.py --package-id com.example.app

Run Batch Analysis

python src/batch_runner.py
# Edit CANDIDATES list in batch_runner.py to specify apps

Evaluate F1

PYTHONIOENCODING=utf-8 python eval_f1.py

---

Research Report vs. Implementation: In-Depth Comparison

1. Scope: Number of Applications

Outcome: The research report outlined 1,000 applications but the implemented study analyzed 117 applications across all 10 planned categories. This reduction was driven by practical constraints: APK acquisition is rate-limited, Cloudflare-gated, and time-intensive. The 117-app dataset still represents a statistically meaningful cross-section and satisfies the original intent of cross-category comparison. The tool remains fully capable of scaling to 1,000 apps with additional runtime; all infrastructure is in place.

---

2. Model Training Strategy

The research report outlined a three-phase sequential training curriculum:

Why the planned approach was abandoned:

Phases 1 and 2 were implemented during development. The Princeton-Leuven pre-training taught the model to generate policy-style text (domain vocabulary), and OPP-115 fine-tuning reinforced verbatim policy quoting. When Phase 3 instruction tuning was applied on top (using mobilellama_stage3_final), the model suffered catastrophic forgetting; it ignored all input instructions entirely and generated hallucinated boilerplate regardless of the query. Root causes identified:

• Learning rate (2e-4) was too aggressive for Phase 3, causing the adapter to overwrite earlier learning
• Training responses in Phase 3 were verbatim policy text excerpts, not analytical gap assessments
• Stacking three sequential adapters compounded representation drift

Solution (training from scratch): A custom training dataset (gap_dataset_v2.jsonl) was built that pairs each Android permission group with policy text excerpts and produces a clean single-line analytical output: "Mentioned: [explanation]" or "Not mentioned: [explanation]". Training directly on this from the base MobileLLaMA model (without any intermediate phases) produced a stable, well-behaved adapter in 3 epochs (~2.5 hours) with training loss converging from 2.91 → 0.34.

---

3. Training Dataset

Outcome: The final dataset is smaller than planned (5,154 vs 18,000–25,000) but purpose-built for the exact task. As noted by Firecrawl (2025), 5,000–10,000 well-formed instruction pairs works well for focused fine-tuning. The higher-volume multi-task approach outlined in the report was superseded by a leaner, task-focused dataset that produced better results for this specific gap-analysis classification problem.

---

4. Training Configuration & Resources

Key deviation: The learning rate was reduced from 2e-4 to 5e-5. The report's recommendation of 2e-4 follows Raschka (2023) for general LoRA fine-tuning, but for this specific classification task on a 2.7B model, 2e-4 proved too aggressive and was the root cause of the catastrophic forgetting observed in the mobilellama_stage3_final adapter. At 5e-5 with cosine decay, training converged stably across 3 epochs.

---

5. APK & Data Acquisition Pipeline

Key upgrades over the report:

1. Playwright over Selenium: Playwright was mentioned in the report as a faster alternative but was deemed to have less documentation for legacy pages. In practice, Playwright's expect_download() context manager and response event interception proved essential for capturing APKMirror's CDN download URLs, making it the better choice.

2. 5-source fallback: APKMirror alone is insufficient because Cloudflare blocks automated search requests. The fallback chain (APKPure, APKCombo, APKMonk, Uptodown) enables acquisition of high-profile apps (WhatsApp, Instagram, TikTok, YouTube) that APKMirror's search would block.

3. Direct URL bypass: A _DEV_SLUG_MAP of ~60 publisher-to-developer-slug mappings enables constructing APKMirror app URLs directly (/apk/{dev-slug}/{app-slug}/) without triggering the Cloudflare-gated search page, unlocking apps like Homescapes, Township, Hulu, and Azure.

4. Post-download verification: The Uptodown source occasionally served wrong-app CDN responses (e.g., Roblox's slot returned Instagram's APK). Package name verification reads the binary AndroidManifest.xml from the downloaded ZIP and checks for the expected package ID as both ASCII and UTF-16-LE byte patterns, rejecting mismatches before analysis proceeds.

---

6. Privacy Health Score (PHS) Algorithm

Risk label note: The report defines "Critical Risk" as PHS < 50. The implementation uses a 0–100 scale where 100 = fully disclosed (Low Risk) and 0 = fully undisclosed (Critical Risk), consistent with the report's intent. The risk labels in the implementation are: Low Risk (≥80), Moderate Risk (60–79), High Risk (30–59), Critical Risk (<30), refined from the report's single threshold.

---

7. Web Application

Additional capabilities not in report:

• Browse Database tab: Live-search across all 117 analyzed apps with sortable PHS scores and coverage bars
• Policy diff view: Detects when an app's privacy policy text changed between runs and shows highlighted additions/removals
• Run history: Each app page shows all historical analysis runs, enabling longitudinal tracking
• Job status page: Real-time progress display as the pipeline runs, including LLM per-permission classification progress

---

8. Evaluation & Validation

Validation approach deviation: The research report planned manual review of 200 apps by a human reviewer examining each privacy policy. This would be comprehensive but labour-intensive and non-reproducible. The implementation uses a deterministic keyword-oracle approach: for each permission group, a curated set of 8–14 domain keywords is matched against the policy text to establish a reproducible ground truth label. The LLM's classification is then compared against this keyword oracle. This approach:

• Is fully reproducible (re-run eval_f1.py at any time)
• Produces consistent results not subject to human reviewer variability
• Scales to any number of apps automatically

The trade-off is that the keyword oracle may have its own false positives/negatives. However, an F1 of 0.930 against this oracle, with Precision of 0.892, provides meaningful evidence that the LLM classifications track the keyword retriever closely, though the oracle and the retriever share signal, so the figure is best read as a consistency check rather than an independent accuracy measure.

---

9. Ethical Considerations

Note on APK retention: The research report commits to deleting APKs post-processing. The current src/acquired/ directory retains downloaded APKs alongside the CSV reports (total ~12GB). For compliance with this commitment, APK files (*.apk, *.xapk) within src/acquired/ should be deleted before final submission; only the gap_report.csv and metadata files are required for reproducibility.

---

10. Tools & Frameworks

---

Final Results Summary

App Risk Distribution across 117 apps:

Notable findings:

• Gaming apps (Rovio, Outfit7, Supercell, EA) frequently request NFC, WRITE_CALENDAR, and READ_PHONE_STATE permissions without policy disclosure
• Social media apps (Instagram, Spotify) showed high PHS (100) due to comprehensive policy coverage despite many permissions
• Productivity apps (Microsoft suite, Google Workspace) scored 94–96 consistently, disclosing almost all permissions
• Finance/payment apps (Revolut, TransferWise) scored low (20–39), failing to disclose several data access permissions

---

Key Deviations and Rationale

---

PrivacyTotal was built to empower users to understand what data applications collect and whether they are honest about it. The tool bridges the gap between legal privacy policies and user comprehension, exposing the "Privacy Paradox" through automated, evidence-based gap analysis at scale.