📦 Russian IT Community Corpus (RICC)

Russian IT Community Corpus (RICC) is an open, de-identified conversational dataset collected from 11 engineering community nodes spanning a 9-year timeline (2017–2026). It captures authentic discussions on backend systems, cloud infrastructure, AI/ML deployment, database internals, and software architecture.

The corpus is structured into ready-to-use splits for Instruction Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Vector Retrieval-Augmented Generation (RAG).

---

⚡ Quick Start

from datasets import load_dataset

# 1. Multi-turn SFT Dialogues (171.5k curated conversations)
sft_ds = load_dataset("wwewtech/russian-it-community-corpus", "sft_dialogues", split="train")
print(f"Loaded SFT dataset: {len(sft_ds):,} dialogues")
print("Sample dialogue:", sft_ds[0]["messages"][:2])

# 2. RAG Technical Knowledge Base (325.7k segmented documents)
rag_ds = load_dataset("wwewtech/russian-it-community-corpus", "rag_knowledge_base", split="train")
print(f"Loaded RAG knowledge base: {len(rag_ds):,} chunks")

# 3. Full Chronological Corpus (2.81M deduplicated messages)
full_ds = load_dataset("wwewtech/russian-it-community-corpus", "full_corpus", split="train")
print(f"Loaded Full corpus: {len(full_ds):,} records")

---

🏗️ Data Curation Pipeline

---

📊 Key Corpus Metrics

---

🧠 Domain & Topic Distribution

---

📁 Dataset Splits & Configurations

1. full_corpus (data/full_clean_messages.parquet)

Full chronological sequence of clean messages with domain labels, sentiment, and structural metadata.

2. sft_dialogues (data/sft_dialogues.parquet)

Reconstructed multi-turn conversation threads formatted for supervised instruction fine-tuning.

{
  "thread_id": 42056,
  "chat_name": "community_node_07",
  "topic_domain": "frontend_ui",
  "topic_tags": ["vue", "js", "di_container", "architecture"],
  "quality_score": 12.25,
  "messages": [
    {
      "role": "user",
      "author": "Developer_65546",
      "content": "Как изолировать ядро CMS при использовании Vue на фронтенде?"
    },
    {
      "role": "assistant",
      "author": "Developer_38544",
      "content": "Для изоляции выносите API в независимый сервисный слой..."
    }
  ]
}

3. rag_knowledge_base (data/rag_knowledge_base.parquet)

Chunked technical discussions formatted for dense embedding indexing (Qdrant, ChromaDB, Milvus, FAISS).

---

📊 SFT Subset — Real Composition & Known Limitations

Computed directly from data/sft_dialogues.parquet (171,520 dialogues):

Known limitations — read before training on this subset:

• This is reconstructed community chat, not curated instruction data. Many "assistant" turns are opinionated chat replies rather than expert answers.
• The quality_score heuristic rewards length and code markers; it does not measure factual correctness.
• For serious SFT runs, filter aggressively: exclude general_tech_chat, require question-first openers and multi-turn substantive answers. That leaves ~283–4,900 dialogues depending on strictness — small but much cleaner than the full set.

A representative technical exchange that does pass the filter (thread_id 30449, business_legal_fintech):

{
  "role": "user",
  "content": "От map(filter(...)) уже тошнит? Серьезно, когда вы видите вот это, то хочется плакать:
              result = list(map(lambda x: x * 2, filter(lambda x: x % 2 == 0, arr)))
              А ведь можно проще..."
},
{
  "role": "assistant",
  "content": "В этом месте JS выглядит читаемее чем Python
              users.filter(user => user.gender === gender)
                .map(user => user.age)
                .reduce((acc, age, index, arr) => acc + age / arr.length, 0);"
}

---

🔬 Empirical Evaluation — Honest Status

🚫 Academic metrics (HumanEval / RuMMLU / PPL / ROUGE) published earlier have been WITHDRAWN. A code audit of the benchmark harness (src/evaluation/official_academic_benchmarks.py) found three defects that produced invalid numbers: substring-based MCQ scoring (inflated RuMMLU to an implausible 100%), PPL computed on empty placeholder strings due to wrong column names (identical base/LoRA values), and silent copying of Base results into LoRA/Hybrid columns when the adapter failed to load. All three are fixed in code with unit tests; the numbers will be republished only after a fresh GPU re-run.

What remains interpretable today — rubric-based heuristic scores on 50 engineering scenarios (concept-overlap + AST-parseability judged programmatically; not execution-verified capability):

(AST rates recomputed as means over all 50 per-scenario ast_score values in reports/heuristic_benchmark_eval.json; an earlier version of this card quoted different numbers that did not match the machine-readable data.)

Pre-trained adapters for 58 base models are available in the LoRA Model Zoo (published as pilot domain-adaptation checkpoints on sampled domain subsets).

---

🛡️ Privacy, Anonymization & Ethical Use

1. Morphological Name Scrubbing: Author display names are extracted and declined across all 6 Russian grammatical cases (Им., Род., Дат., Вин., Твор., Предл.) to remove conversational references in text.
2. Community Node Anonymization: All 11 channel titles and supergroup IDs are strictly anonymized as surrogate nodes (community_node_01..11).
3. Deterministic Pattern Scrubbing: Removes phone numbers, personal emails, crypto wallet addresses (BTC, ETH, TRON, TON, SOL), API keys (sk-proj-..., ghp_...), JWT tokens, and database credentials.
4. Terminology Whitelist: 276 standard programming keywords, frameworks, and tools are protected against accidental redaction (see src/pii/ner_scrubber.py → TECH_WHITELIST).
5. Notice and Takedown Policy: Intended strictly for educational, academic, and non-commercial research. If you identify any inadvertent personal identifier, please open a takedown issue or submit a removal request. Requests are processed within 48 hours.

---

📖 Citation

@misc{ricc2026,
  author = {Russian IT Community Open Research Group},
  title = {Russian IT Community Corpus (RICC): A Curated Multi-Domain Conversational Dataset for LLM SFT, DPO, and RAG},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/wwewtech/russian-it-community-corpus}}
}