Hasal-Eih Intelligent News Platform

Welcome to the Hasal-Eih repository!

Hasal-Eih is an advanced, AI-powered bilingual (Arabic & English) intelligent news platform designed to tackle the global challenge of information overload and hidden media bias.

1. Project Overview

Purpose

Every day, over 2 million news articles are published. Readers struggle to distinguish between factual reporting and subjective framing. Hasal-Eih aims to reduce cognitive overload by automatically aggregating, summarizing, analyzing, and detecting bias in news articles. Furthermore, it measures the impact of news sentiment on financial markets.

Target Users

• Journalists & Media Analysts: For tracking narrative framing and media sentiment.
• Investors & Financial Analysts: Seeking alpha by understanding how media bias affects indices like EGX30 and specific stocks like Egypt Gas.
• General Public: Anyone seeking transparent, unbiased, and summarized news streams.

---

2. High-Level Architecture Overview

Hasal-Eih is built as a highly scalable microservices ecosystem. It separates high-speed data ingestion from resource-intensive Machine Learning inference.

1. Ingestion Module: A multi-threaded set of scrapers fetching from NewsAPI (English) and RSS Feeds (Arabic).
2. Buffer Queue (Redis): Acts as a high-speed broker handling asynchronous tasks.
3. ML Worker Pool (Celery): Dedicated workers that run heavy Deep Learning and NLP pipelines (Named Entity Recognition, Target-level bias detection).
4. Hybrid Database (PostgreSQL + pgvector): Stores relational data and vector embeddings for instantaneous semantic search.
5. API Gateways (FastAPI): Serves data asynchronously to the frontend.
6. Frontend: A Vanilla HTML/JS dynamic dashboard to visualize timelines, bias scores, and financial predictions.

---

3. System Explanation (Core Components)

A. Data Ingestion & Routing

• What it does: Scrapes live articles and extracts noise-free text.
• Why it exists: To gather diverse news streams in real-time.
• How it works: Uses trafilatura to extract core text from raw HTML, and utilizes langdetect to instantly route articles into Arabic or English processing queues.
• Interaction: Pushes sanitized article data into Redis.

B. NLP & Bias Detection Models

The system uses differing SOTA approaches based on the language.

English Bias Pipeline (Target-Conditioned Polarity)

• What it does: Instead of predicting global political leaning (Left/Right), it predicts polarity (Anti/Neutral/Pro) towards a specific Target Entity mentioned in the sentence.
• Why it exists: Political taxonomies like Left/Right are culturally relative. Structural "Anti-entity" language remains consistent globally.
• How it works: Uses a fine-tuned RoBERTa-base model trained on the NewsMTSC dataset. It inserts custom [TARGET] tokens during tokenization to force the model to gauge sentiment exclusively towards that entity. It then aggregates these sentence scores into a weighted Article-Level bias score using Entropy.

Arabic Bias Pipeline (Article Level Subjectivity)

• What it does: Detects emotional framing, propaganda, and linguistic bias in Arabic texts.
• Why it exists: Arabic NLP lacks high-quality journalism bias datasets.
• How it works: Uses MARBERT fine-tuned on a heavily engineered custom dataset (ArmPro + AFND + BBC + XL-Sum). We implemented a "Lead Paragraph Truncator" to strictly enforce length parity (200-250 characters) so the model focuses on semantics, not text length.

C. Financial Impact Prediction (Time-Series)

• What it does: Forecasts the next 10-day market trends for standard indices.
• Why it exists: To evaluate market sensitivity to news bias over time.
• How it works: Uses Advanced Bidirectional LSTMs with Attention mechanisms. It runs in an isolated TensorFlow environment (tf_env) to avoid package dependency conflicts.

---

4. End-to-End Data Flow

1. Input: A chron job (Celery Beat) triggers the ingestion process every X minutes (default 15).
2. Processing (Queue): The article URL and raw text enter the Redis Queue. Fast data scrapers are never bottlenecked by slow ML models.
3. Processing (ML Pipeline): Celery workers pick up tasks. For each article:
   • NER (Named Entity Recognition) extracts mentioned entities.
   • Based on the detected language, sentences are fed to the corresponding Bias Model (RoBERTa / MARBERT).
   • Generates a 384-dimensional embedding using Sentence-Transformers.
   • Aggregation: Sentence-level scores are mathematically aggregated into an overall Article-Level Bias metric.
4. Storage: The finalized JSON payload (text, bias metrics, target insights, embedding) is inserted into PostgreSQL.
5. Output: The frontend queries the FastAPI instance. If a user does a semantic search, the API utilizes the HNSW index in pgvector to rapidly fetch similar articles via cosine similarity.

Edge Cases & Failure Scenarios:

• Scraper Blocking: Handled via cloudscraper and request retries.
• GPU/Memory Overflow: ML models batch requests. Redis handles buffering. If an ML worker crashes, the Celery task will be requeued.

---

5. Technical Details

Key Algorithms & Design Patterns

• Target-Conditioned Loss: In the English model, Focal Loss + Label Smoothing (0.05) is utilized to combat severe class imbalances in bias datasets.
• Directional Penalty Loss: The LSTM financial model uses a custom loss function: 0.7 * MSE + 0.3 * Directional Penalty. This forces the model to predict the trend direction accurately, rather than just the absolute value.
• Singleton Model Loading: Machine learning models instances are loaded at API boot time to prevent massive memory overhead per request.

Database Schema

• PostgreSQL Schema: Stores standard UUIDs, timestamps, raw text, and clustered entities. Includes a specialised embedding_vector column optimized with an HNSW (Hierarchical Navigable Small World) index for millisecond-level similarity searches across thousands of documents.

---

6. Setup & Usage

Prerequisites

• Docker Desktop (for PostgreSQL and Redis)
• Python

Step-by-Step Installation

1. Clone the repo and set up your .env file in the root directory:

NEWSAPI_KEY=YOUR_NEWSAPI_KEY
FETCH_INTERVAL_MINUTES=15
REDIS_HOST=localhost
REDIS_PORT=6379
DB_HOST=127.0.0.1
DB_PORT=5433
DB_NAME=news_db
DB_USER=postgres
DB_PASSWORD=postgres
CELERY_BROKER_URL=redis://localhost:6379/0

2. Boot infrastructure components:

docker-compose up -d

(This starts Redis on port 6379 and PostgreSQL with pgvector on port 5433).

3. Install Core Dependencies:

pip install -r requirements.txt

4. Install TensorFlow in a separate environment: Due to hardware and dependency conflicts, the financial API runs in an isolated tf_env.

py -3.11 -m venv tf_env
.\tf_env\Scripts\activate
.\tf_env\Scripts\python.exe -m pip install fastapi uvicorn pandas numpy tensorflow joblib scikit-learn pydantic redis celery hdbscan torch
deactivate

5. Download the Models: Create the following folder structure in ml_pipeline/models/ and place the models in their respective folders:

ml_pipeline/
├── models/
│   ├── BIAS/
│   │   ├── AR/
│   │   │   └── MARBERT/
│   │   └── EN/
│   │       ├── ANTIvsPRO/
│   │       ├── Binary/
│   │       └── FACTvsOP/
│   └── Impact/
│       ├── egGAS/
│       └── egx30/

Running the System

We have provided an automated orchestrator script. Run:

python run_services.py

This automatically boots up 6 separate command windows representing:

1. Database Initialize
2. FastAPI Main Application (http://127.0.0.1:8000)
3. Ingestion Worker
4. Celery Worker Pool
5. Celery Beat (Cron)
6. TensorFlow ML API (http://127.0.0.1:8001)

To view the UI, launch the frontend:

cd frontend
python -m http.server 3000

Navigate to http://localhost:3000/.

---

7. Troubleshooting

• Zero Articles in Dashboard: Ensure your NEWSAPI_KEY is valid. Ensure Celery Beat is actively running, and check docker ps to ensure Redis is accepting connections.
• Out of Memory (OOM) Errors on Model Loading: Decrease NEWSAPI_PAGE_SIZE to process fewer documents per batch or deploy the system on an environment with high RAM resources. Ensure you are not loading the large Transformers models globally outside of worker execution unless intended.
• TensorFlow Module Not Found: You forgot to activate the Python virtual environment (tf_env) before starting impact_api.py. The run_services.py script mitigates this locally in Windows.