[!TIP] 🤗 Official Hugging Face Hub Integration:

• 📦 Dataset: wwewtech/russian-it-community-corpus — 2,816,434 clean messages, 171.5k multi-turn SFT dialogues, and 325.7k RAG knowledge base chunks in Apache Parquet.
• 🦁 LoRA Model Zoo: wwewtech/russian-it-community-lora — 58 adapters on Hugging Face Hub (55 domain adapters + 3 flagship 7B–8B QLoRA; 56 adapters cloned locally in lora_adapters/).

from datasets import load_dataset
dataset = load_dataset("wwewtech/russian-it-community-corpus", "sft_dialogues", split="train")

---

Overview

RICC (Russian IT Community Corpus) is an open data engineering and curation stack that ingests, cleans, deduplicates, and structures 2,816,434 engineering, infrastructure, business, and software development messages from 11 community nodes spanning 2017–2026.

The platform produces datasets for instruction fine-tuning, direct preference optimization, and vector knowledge retrieval.

---

Architecture

                       ┌──────────────────────────────┐
   11 Community Nodes ──►   Multi-Source Ingestion      │
    2,816,434 raw records   └──────────────┬───────────────┘
                                      │
                       ┌──────────────▼───────────────┐
                       │   Deep Case-Aware Zero-PII   │ ──► Declension across 6 Russian cases
                       │   RegEx + Natasha Neural NER │     Tech terminology protection whitelist
                       └──────────────┬───────────────┘
                                      │
                       ┌──────────────▼───────────────┐
                       │   Deduplication & Taxonomy   │ ──► MinHash LSH with 128 permutations
                       │   8 Domain Classifiers       │     Exact hash deduplication
                       └──────────────┬───────────────┘
                                      │
                       ┌──────────────▼───────────────┐
                       │   Thread DAG Reconstruction  │ ──► Reply tree traversal
                       │   SFT, DPO, RAG Extraction   │     Temporal cluster resolution
                       └──────────────┬───────────────┘
                                      │
         ┌────────────────────────────┼────────────────────────────┐
         ▼                            ▼                            ▼
┌──────────────────┐        ┌──────────────────┐        ┌──────────────────┐
│  Apache Parquet  │        │   JSONL Datasets │        │  Local LoRA, RAG │
│  zstd compressed │        │  ShareGPT, ChatML│        │  RTX 3060 Runner │
│   full, sft, rag │        │  Alpaca, Qdrant  │        │  Streamlit Studio│
└──────────────────┘        └──────────────────┘        └──────────────────┘

---

🔗 Official Datasets & Direct Access Links

The corpus and trained adapters are available both remotely on Hugging Face Hub and locally in dataset_output/:

🤗 Hugging Face Hub Repositories:

• 📦 Dataset Hub (Full Corpus, SFT, RAG): 👉 https://huggingface.co/datasets/wwewtech/russian-it-community-corpus

  • 📄 Full Clean Corpus (Parquet) — 2.82M rows (2,816,434 records, 189 MB)
  • 💬 SFT Dialogues (Parquet) — 171,520 multi-turn dialogues (132 MB)
  • 🔍 RAG Knowledge Base (Parquet) — 325,690 knowledge chunks (159 MB)
  • ⚙️ Heuristic Benchmark Evaluation (JSON) — 50-scenario domain heuristic evaluation matrix

• 🦁 Model Hub (58 LoRA Adapters on Hub / 56 Local Adapters): 👉 https://huggingface.co/wwewtech/russian-it-community-lora

  • 🥇 Flagship Qwen 2.5 Coder 7B Adapter
  • 🥈 Flagship DeepSeek R1 Distill 7B Adapter
  • 🥉 Flagship Meta LLaMA 3.1 8B Adapter

---

📂 Local Datasets and Formats

All datasets are automatically generated and saved in dataset_output/:

---

Comparative Architectural Evaluation (Base vs RAG vs LoRA vs Hybrid)

[!WARNING] Benchmark section withdrawn from README. Earlier numbers in this section were officially retracted in reports/DATASET_AND_ANALYTICS.md, section "Empirical Evaluation — Honest Status". A code audit of the benchmark harness (src/evaluation/official_academic_benchmarks.py) found three defects (substring-based MCQ scoring, PPL on empty placeholders, silent copy of Base results into LoRA/Hybrid columns when the adapter failed to load). The numbers will be republished only after a fresh GPU re-run; meanwhile we ship the dataset and adapters without any accuracy claim on this README page. The 100-question non-executing benchmark suite remains available in reports/domain_benchmark_100.json as a corpus, not as a leaderboard.

---

Quick start

Requirements

• Python 3.11, 3.12, or 3.13
• NVIDIA GPU with 8 GB or more VRAM for training, or CPU for data processing
• Docker with Docker Compose for containerized runs

Direct Loading from Hugging Face

Load pre-built dataset splits in Python with a single line:

from datasets import load_dataset

# 1. Load SFT Multi-Turn Dialogues (171.5k dialogues)
sft_ds = load_dataset("wwewtech/russian-it-community-corpus", "sft_dialogues", split="train")

# 2. Load RAG Knowledge Base Chunks (325.7k chunks)
rag_ds = load_dataset("wwewtech/russian-it-community-corpus", "rag_knowledge_base", split="train")

# 3. Load Full Clean Messages (2.81M records)
full_ds = load_dataset("wwewtech/russian-it-community-corpus", "full_corpus", split="train")

LoRA Model Zoo (56 Local Adapters / 58 Adapters on Hugging Face Hub)

Pre-trained adapters fine-tuned in pilot domain-adaptation runs on RICC dataset are available locally in lora_adapters/ (56 adapters) and on Hugging Face: wwewtech/russian-it-community-lora (58 adapters, including codegen_350m_multi and sber_mgpt). The local lora_adapters/registry.json is the single source of truth for on-disk adapters, generated by scripts/generate_lora_registry.py. To mirror the registry to the Hub, see docs/adr/0001-hf-token-handling.md and scripts/sync_to_hub.py. Full hand-maintained catalog lives in reports/LORA_MODEL_ZOO.md.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(
    "lora_adapters/qwen2.5_1.5b_instruct"
)  # or "wwewtech/russian-it-community-lora"
model = PeftModel.from_pretrained(base, "lora_adapters/qwen2.5_1.5b_instruct")

1. Install

git clone https://github.com/wwewtech/russian-it-community-corpus.git
cd russian-it-community-corpus
pip install -r requirements.txt

2. Run Data Pipeline

python main.py

3. Launch Web Data Studio

streamlit run app.py
# or:
make ui

4. Fine-Tune LoRA

python src/lora/train_lora.py --model Qwen/Qwen2.5-0.5B-Instruct --steps 100

5. Run LoRA Inference

python src/lora/generate_demo.py --prompt "Как настроить прием платежей для SaaS из РФ?"

Command Reference

---

Project Structure

├── assets/                     # Theme-aware vector graphics
├── src/
│   ├── ingestion/              # Multi-chat JSON export parser
│   ├── pii/                    # Case-aware PII anonymizer and neural NER
│   ├── graph/                  # Thread DAG builder and dialogue extractor
│   ├── deduplication/          # MinHash LSH and exact text hashing
│   ├── taxonomy/               # Domain classifiers and keyword taggers
│   ├── exporter/               # Apache Parquet and JSONL serializers
│   ├── analytics/              # DeepChatAnalyzer statistical engine
│   ├── rag/                    # Vector search and prompt augmentation pipeline
│   ├── lora/                   # PEFT training and inference scripts
│   ├── evaluation/             # Benchmark comparator
│   └── validation/             # PII auditor and domain benchmarks
├── dataset_output/             # Parquet datasets and preview samples
├── reports/                    # Canonical dataset & model cards and benchmark suites
│   ├── DATASET_AND_ANALYTICS.md   # Dataset Card and Zero-PII Protocol
│   ├── HF_MODEL_CARD.md           # Hugging Face Model Card
│   ├── LORA_MODEL_ZOO.md          # Catalog of 58 LoRA Adapters on Hub (56 local)
│   ├── heuristic_benchmark_eval.json # 50-Scenario Heuristic Benchmark Matrix
│   └── domain_benchmark_100.json  # 100-Scenario Domain Benchmark Suite
├── tests/                      # Automated unit tests
├── app.py                      # Streamlit Web Data Studio
├── demo_walkthrough.py         # Terminal demonstration script
├── cli.py                      # CLI entrypoint
├── main.py                     # Pipeline master runner
├── Dockerfile                  # Production container definition
├── docker-compose.yml          # Container composition
├── Makefile                    # Task shortcuts
└── requirements.txt            # Python dependencies

---

Privacy and Anonymization Protocol

1. Morphological Name Redaction: Detects author display names and inflects them across 6 Russian grammatical cases to eliminate conversational mentions.
2. Deterministic Pattern Scrubbing: Removes phone numbers in international formats, email addresses, cryptocurrency wallet addresses, API keys, tokens, and database connection strings.
3. Source Community Anonymization: Replaces all source channel titles with surrogate identifiers (community_node_01..11) and re-indexes technical IDs.
4. Terminology Protection: Whitelists common technical terms, programming languages, libraries, and hosting providers to prevent false positives.
5. Automated Sanity Suite: Regression tests against synthetic adversarial vectors verify regex pattern coverage in reports/pii_validation_report.json.

---

License and Ethical Use

• Academic and Research Use: Provided for educational, non-commercial, and machine learning research purposes.
• Privacy Policy: All data has undergone heuristic multi-pass PII scrubbing and anonymization.
• Notice and Takedown: If you identify any inadvertent personal identifier or content that should be excluded, please open a takedown issue or submit a removal request. Requests are processed within 48 hours.

---