🎯 Ellora: Enhancing LLMs with LoRA

Ellora (Enhancing LLMs with LoRA) is a collection of standardized, high-quality LoRA recipes for enhancing Large Language Model capabilities. Instead of building new frameworks, we focus on creating reproducible training methodologies that work with existing infrastructure.

🌟 Philosophy

The LLM ecosystem has amazing infrastructure (LoRAX, PEFT, vLLM), but lacks standardized, high-quality capability adapters. Ellora bridges this gap by providing:

• 📋 Recipes, not frameworks - Reproducible training methodologies
• 🎯 Quality-first approach - Rigorous evaluation and benchmarking
• 🔄 Self-supervised data generation - No dependency on external datasets
• 🏗️ Infrastructure agnostic - Works with existing tools (PEFT, LoRAX, etc.)
• 🌍 Community-driven - Open recipes for the ecosystem

📚 Recipe Collection

🍳 Available Recipes

Recipe #1: Accuracy Recovery LoRA

Problem: Quantized models (INT4/INT8) lose accuracy compared to FP16 versions
Solution: Self-distillation LoRA adapter using Magpie-generated data

• 🎯 Goal: <5% performance degradation from FP16 baseline
• 💾 Memory: ~75% reduction in model size
• ⚡ Speed: 2-3x faster inference than FP16
• 📊 Method: Teacher (FP16) → Student (INT4+LoRA) distillation

Key Innovation: Uses Magpie self-data generation for perfect domain alignment - no external datasets needed!

Quick Start

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
)

# Load accuracy recovery adapter
model = PeftModel.from_pretrained(model, "codelion/Qwen3-0.6B-accuracy-recovery-lora")

# Use normally - now with recovered accuracy!

Results

Recipe #2: Reasoning LoRA with GRPO

Problem: LLMs often lack structured thinking patterns for complex reasoning
Solution: GRPO-trained adapter that teaches chain-of-thought with <think></think> tags

• 🧠 Goal: Enhance reasoning capabilities through preference learning
• 📝 Method: GRPO (Group Relative Policy Optimization) with self-rewarding
• 🎯 Feature: Teaches structured thinking with clear reasoning steps
• 💡 Output: Models that show their reasoning process transparently

Key Innovation: Self-generated preference data with automated quality scoring - no need for human annotations or external preference datasets!

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")

# Load reasoning adapter
model = PeftModel.from_pretrained(model, "codelion/gemma-3-1b-it-reasoning-grpo-lora")

# Use with thinking prompt
prompt = '''Think step by step and use <think></think> tags to show your reasoning process.

Problem: If a train travels 120 miles in 2 hours, then increases its speed by 30 mph for the next hour, how many total miles does it travel?

Response:'''

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Results

Recipe #3: Tool Calling LoRA

Problem: LLMs struggle with effective tool usage for code exploration
Solution: Hybrid training with Magpie scenarios + real tool execution results

• 🛠️ Goal: Teach models to use development tools effectively
• 🔄 Method: Generate scenarios with Magpie, execute on real codebases
• 🎯 Feature: OpenAI-compatible function calling format
• 💻 Tools: File operations, search, code navigation, and more

Key Innovation: Combines synthetic scenario diversity with real execution feedback - ensuring models learn authentic tool usage patterns!

Recipe #4: Progressive Context Extension LoRA

Problem: Base models limited to 32K context, need 2M tokens for large repositories
Solution: Progressive curriculum learning with vLLM + Unsloth hybrid approach

• 📈 Goal: Extend context from 32K to 2M tokens (61x increase)
• 🎓 Method: Curriculum learning across 4 stages (32K → 128K → 512K → 2M)
• ⚡ Innovation: vLLM for fast data generation, Unsloth for memory-efficient training
• 🔍 Feature: Single LoRA adapter progressively learns longer contexts

Key Innovation: Hybrid optimization combining vLLM's inference speed with Unsloth's training efficiency - achieving 61x context extension with minimal compute!

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")

# Load progressive context adapter
model = PeftModel.from_pretrained(model, "codelion/qwen2-5-coder-0-5b-instruct-progressive-2000k-lora")

# Use with 2M token context - perfect for large repositories!
long_context_prompt = "Analyze this entire repository..." # Up to 2M tokens
inputs = tokenizer(long_context_prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=1024)

Results

Recipe #5: Secure Code Generation LoRA

Problem: LLMs frequently generate code with security vulnerabilities (SQL injection, etc.)
Solution: GRPO training with automated Semgrep analysis for security scoring

• 🔒 Goal: Generate secure code by default without explicit prompting
• 🛡️ Method: Self-supervised training with automatic vulnerability detection
• 📊 Scoring: Partial credit system (40% functionality, 40% patterns, 20% vulnerabilities)
• ✅ Results: 97% reduction in vulnerabilities, 100% functional code

Key Innovation: Automated security analysis replaces manual curation - teaching secure patterns without labeled datasets!

Quick Start

from transformers import AutoModelForCausalLM
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")

# Load security adapter
model = PeftModel.from_pretrained(model, "codelion/qwen2.5-coder-security-grpo-lora")

# Generate secure code by default
prompt = "Create a function to search for products by name in a database"
# Model will automatically use parameterized queries!

Results

Recipe #6: Execution World Model Thinking LoRA

Problem: LLMs can generate and reason about code, but lack execution awareness - understanding how code behaves at runtime, predicting variable states, and comprehending dynamic program behavior Solution: GRPO-trained adapter combining Qwen3's native thinking with real execution traces, inspired by Meta's CWM (Code World Model)

• 🧠 Goal: Add execution awareness to thinking models
• 🔍 Method: Hybrid Magpie-style generation + real Python execution tracing
• 📊 Feature: Predict program states, debug with execution understanding
• 💡 Model: Built on Qwen3-4B-Thinking-2507 with 262K context

Key Innovation: Combines Qwen3's thinking capabilities with real execution traces captured via Python's trace module - creating a "neural debugger" that understands both logic AND runtime behavior!

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Thinking-2507")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Thinking-2507")

# Load execution world model adapter
model = PeftModel.from_pretrained(model, "codelion/Qwen3-4B-execution-world-model-lora")

# Analyze code with execution awareness
code = """
x = 10
y = x * 2
z = x + y
"""

prompt = f"Analyze this code and predict its execution trace step by step:\n\n```python\n{code}\n```"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
# Model will predict variable states at each line!

Results

🏆 Model Zoo

All models trained using Ellora recipes are available on HuggingFace:

Featured Models

• codelion/Qwen3-0.6B-accuracy-recovery-lora - Accuracy recovery for Qwen3-0.6B
• codelion/gemma-3-1b-it-reasoning-grpo-lora - Reasoning enhancement for Gemma-3-1B
• codelion/Llama-3.2-1B-Instruct-tool-calling-lora - Tool calling for Llama-3.2-1B
• codelion/qwen2-5-coder-0-5b-instruct-progressive-2000k-lora - 2M context extension for Qwen2.5-Coder-0.5B
• codelion/Qwen2.5-Coder-0.5B-Instruct-security-grpo-lora - Secure code generation for Qwen2.5-Coder-0.5B
• codelion/Qwen3-4B-execution-world-model-lora - Execution-aware world model for Qwen3-4B-Thinking
• More models coming as we test recipes across different model families!

🔬 Research & Citations

If you use Ellora recipes in your research, please cite:

@misc{ellora2024,
  title={Ellora: Enhancing LLMs with LoRA - Standardized Recipes for Capability Enhancement},
  author={Asankhaya Sharma},
  year={2024},
  url={https://github.com/codelion/ellora}
}

Key Papers & Inspirations

• LoRA: Low-Rank Adaptation of Large Language Models
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• QLoRA: Efficient Finetuning of Quantized LLMs
• CWM: An Open-Weights LLM for Research on Code Generation with World Models