Shadowgraph
TrendingBuzzTop stargazers
Chrome extension
Sign in

wakamex/human3090

Human Eval scores evaluated locally using a 3090

Python

9

76 commits

updated Mar 5, 2026

See the code

README

HumanEval on a local 3090 results

Unless otherwise specified:

  • maximum number of layers offloaded to GPU
  • local models run with llama.cpp server and .gguf formats
  • parameter changes carried over into following tests (temperature, penalties, etc.)
  • * denotes non-local for comparison
ModelConfigurationHuman EvalTime taken
GPT-4*Instruction-style, temperature=0.2, presence_penalty=063.4%
GPT-4*Completion-style84.1%
Mixtral8x7bmixtral-8x7b-v0.1.Q5_K_M.gguf45.7%
Mistral-medium*62.2%
Llama2*HF API, CodeLlama-34b-Instruct-hf42.1%
Mistral-large*73.2%
WizardLM2WizardLM-2-8x22B.IQ3_XS-00001-of-00005.gguf56.7%
Wizardcoderwizardcoder-33b-v1.1.Q4_K_M.gguf, temperature=0.073.8%
Wizardcoder-PythonQ4_K_M. quant, Modified prompt57.3%
CodeFuse-DeepseekCodeFuse-DeepSeek-33B-Q4_K_M.gguf68.7%
Deepseekdeepseek-coder-33b-instruct.Q4_K_M.gguf79.9%
OpenCodeInterpreterggml-opencodeinterpreter-ds-33b-q8_0.gguf, -ngl 40Failed
Deepseekggml-deepseek-coder-33b-instruct-q4_k_m.gguf78.7%
Deepseekdeepseek-coder-33b-instruct.Q5_K_M.gguf, -ngl 60, a bit slow79.3%
Llama3*together.ai API, Llama-3-70b-chat-hf75.6%
DBRX*together.ai API, dbrx-instruct48.8%
CodeQwencodeqwen-1_5-7b-chat-q8_0.gguf83.5%
Llama3-8Bbartowski/Meta-Llama-3-8B-Instruct-GGUF52.4%
Phi-3-mini4k context, 4bit quantized60.4%
Phi-3-mini4k context, fp16 quantized62.2%
Hermes-LlamaHermes-2-Pro-Llama-3-Instruct-Merged-DPO-F1653.7%
Codestral Q6_KCodestral-22B-v0.1-hf.Q6_K.gguf81.7%812.53s
Codestral Q8Codestral-22B-v0.1-hf.Q8_0.gguf79.9%2918.51s
Deepseek V2DeepSeek-Coder-V2-Lite-Instruct-Q8_0_L.gguf82.9%378.86s
Llama-3.1Meta-Llama-3.1-8B-Instruct.Q8_0_MaziyarPanahi.gguf57.9%304.09s
Qwen2.5-Coder-14BQwen2.5-Coder-14B-Instruct-Q8_0.gguf83.5%409.90s
Qwen2.5-Coder-32BQwen2.5-Coder-32B-Instruct-Q4_K_M.gguf89.6%375.33s
QwQ-32B-PreviewQwQ-32B-Preview-Q4_K_L.gguf52.4%11660.46s
DeepSeek-R1-1.5BDeepSeek-R1-Distill-Qwen-1.5B-Q6_K_L.gguf, temperature=0.657.3%4154.71s
DeepSeek-R1-8BDeepSeek-R1-Distill-Llama-8B-Q8_0.gguf, temperature=0.674.4%8836.76s
DeepSeek-R1-14BDeepSeek-R1-Distill-Qwen-14B-Q6_K_L.gguf, temperature=0.084.8%10444.61s
DeepSeek-R1-32BDeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf, temperature=0.690.2%12861.13s
Mistral-Small-24BMistral-Small-24B-Instruct-2501-Q6_K.gguf89.0%2365.32s
Simplescaling S1simplescaling_s1-32B-Q5_K_S.gguf68.3%1140.21s
Tqwendotqwendo-36b-Q4_K_L.gguf87.8%1916.67s
smollm2smollm2-1.7b-instruct-q4_k_m30.5%203.09s
tinyr1tinyr1-32b-preview-q4_k_m31.1%4992.37s
QwQQwQ-32B-Q4_K_M, max_tokens=1000078.0%19609.10s
QwQQwQ-32B-Q4_K_M, temperature=0.6, max-tokens=1000078.0%18689.78s
tinyr1tinyr1-32b-preview-q4_k_m, max-tokens=1000059.1%51286.32s
tinyr1tinyr1-32b-preview-q4_k_m, temperature=0.6, max-tokens=1000069.3%54596.08s
EXAONEEXAONE-Deep-32B-Q4_K_M, temperature=0.6, top-p=0.95, max-tokens=1000079.9%25417.01s
Mistral-Small-3.1-24Bmistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L89.0%2390.45s
DeepCoder-14B-Previewagentica-org_DeepCoder-14B-Preview-Q6_K_L, temperature=0.6, max-tokens=1000075.0%16987.95s
DevstralDevstral-Small-2505-Q4_K_M, start-problem=16390.2%42544.20s
Qwen3.5-27BQwen3.5-27B-Q4_K_M, max-tokens=1000098.2%3953.13s
Qwen3.5-35B-A3BQwen3.5-35B-A3B-Q4_K_M (MoE, 3B active), max-tokens=1000096.3%1289.29s
Qwen3.5Qwen3.5-35B-A3B-UD-Q4_K_XL, temperature=0.6, max-tokens=10000, context-size=3276897.0%1403.61s

LiveCodeBench

ModelConfigurationVersionScoreTime taken
smollm2smollm2-1.7b-instruct-q4_k_mLCBv54.2%259.23s
QwQQwQ-32B-Q4_K_MLCBv50.0%5196.35s
DevstralDevstral-Small-2505-Q4_K_M, start-problem=34LCBv58.6%57268.14s
Qwen3.5-27BQwen3.5-27B-Q4_K_M, max-tokens=10000LCBv58.8%
Qwen3.5-35B-A3BQwen3.5-35B-A3B-Q4_K_M (MoE, 3B active), max-tokens=32000LCBv565.5%5139.25s
agenticaagentica-org_DeepCoder-14B-Preview-Q6_K_L, max-tokens=10000, context-size=32768LCBv58.8%27151.90s
agenticaagentica-org_DeepCoder-14B-Preview-Q6_K_L, max-tokens=10000, context-size=32768LCBv616.4%28325.10s
DeepSeekDeepSeek-R1-Distill-Qwen-14B-Q6_K_L, temperature=0.6, max-tokens=10000, context-size=32768LCBv530.8%32852.81s
DeepSeekDeepSeek-R1-Distill-Qwen-14B-Q6/K_LLCBv634.8%33265.24s
mistralai/Mistralmistralai/Mistral-Small-3.1-24B-Instruct-2503-Q6_K_LLCBv639.3%1909.46s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-14B.i1-Q4/K_MLCBv614.9%32695.36s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-14B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv516.1%33253.31s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-14B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv615.3%32732.98s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-7B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv518.6%14692.69s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-7B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv625.1%14032.87s
Qwen2.5Qwen2.5-Coder-32B-Instruct-Q4_K_MLCBv546.0%2195.34s
Qwen2.5Qwen2.5-Coder-32B-Instruct-Q4_K_MLCBv644.7%2246.69s
Qwen3.5Qwen3.5-27B-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv535.1%45214.12s
Qwen3.5Qwen3.5-35B-A3B-UD-Q4_K_XL, temperature=0.6, max-tokens=10000, context-size=32768LCBv535.8%15681.84s
Qwen3.5Qwen3.5-35B-A3B-UD-Q4_K_XL, temperature=0.6, max-tokens=10000, context-size=32768LCBv633.3%16048.97s
JanJan-code-4b-Q4_K_M, temperature=0.7, max-tokens=10000, top-p=0.8, min-p=0.0, context-size=32768LCBv642.5%5814.66s
JanJan-code-4b-Q8_0, temperature=0.7, max-tokens=10000, top-p=0.8, min-p=0.0, context-size=32768LCBv637.6%6616.07s
IQuestIQuest-Coder-V1-14B-Thinking, temperature=1.0, max-tokens=32000, top-p=0.95LCBv51.8%6915.48s
IQuestIQuest-Coder-V1-14B-Thinking, temperature=1.0, max-tokens=32000, top-p=0.95LCBv60.0%238.64s
agenticaagentica-org_DeepCoder-14B-Preview-Q6_K_L, max-tokens=32000, context-size=32768LCBv510.2%36732.02s

DeepSeek-R1 Distilled HumanEval Results

Model SizeTemperatureFrequency PenaltyAccuracyDeathloopsAvg. CharsSpeed (s)
1.5B0.00.050.6%29464,442
1.5B0.60.057.3%11,0034,155
1.5B0.01.056.1%04175,501
1.5B0.61.050.6%07924,828
14B0.00.084.8%175910,445
32B0.00.085.4%060315,655
32B0.60.090.2%069812,861
32B0.01.075.6%064517,182
Using the following quants from @Bartowski:
  1.5B: Qwen-1.5B-Q6_K_L.gguf
  14B: Qwen-14B-Q4_K_M.gguf
  32B: Qwen-32B-Q4_K_M.gguf
Deathloop is a response over 15k chars

API Results

ModelConfigurationHuman EvalTime taken
gpt-4o-2024-11-20*Don't respond with anything but code.86.2%174.97s
gpt-4o-2024-11-20*94.5%661.51s
Claude 3.5 SonnetOpenRouter95.1%1673.41s
Deepseek-R1OpenRouter31.1%15359.62s
Deepseek-R1OpenRouter. "Don't respond with anything but code"79.3%9819.20s
Deepseek-R1api.deepseek.com97.6%7625.37s
o3-mini96.3%1140.21s

Star History Chart

Significant stargazers

Victor Muštar

170 followers · starred Mar 2025

Similar repos

onurkanbakirci/google-colab-hf-human-eval

2

WizardLMTeam/WizardCoder-33B-V1.1

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

136

WizardLMTeam/WizardCoder-15B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

762

WizardLM/WizardCoder-15B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

762

WizardLMTeam/WizardCoder-Python-34B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

773

deepseek-ai/DeepSeek-Coder-V2

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

7,011

WizardLMTeam/WizardCoder-Python-13B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

109

TheBloke/WizardCoder-15B-1.0-GGML

Model

WizardLM's WizardCoder 15B 1.0 GGML

115

wakamex/human3090

Human Eval scores evaluated locally using a 3090

Python

9

76 commits

updated Mar 5, 2026

See the code

README

HumanEval on a local 3090 results

Unless otherwise specified:

  • maximum number of layers offloaded to GPU
  • local models run with llama.cpp server and .gguf formats
  • parameter changes carried over into following tests (temperature, penalties, etc.)
  • * denotes non-local for comparison
ModelConfigurationHuman EvalTime taken
GPT-4*Instruction-style, temperature=0.2, presence_penalty=063.4%
GPT-4*Completion-style84.1%
Mixtral8x7bmixtral-8x7b-v0.1.Q5_K_M.gguf45.7%
Mistral-medium*62.2%
Llama2*HF API, CodeLlama-34b-Instruct-hf42.1%
Mistral-large*73.2%
WizardLM2WizardLM-2-8x22B.IQ3_XS-00001-of-00005.gguf56.7%
Wizardcoderwizardcoder-33b-v1.1.Q4_K_M.gguf, temperature=0.073.8%
Wizardcoder-PythonQ4_K_M. quant, Modified prompt57.3%
CodeFuse-DeepseekCodeFuse-DeepSeek-33B-Q4_K_M.gguf68.7%
Deepseekdeepseek-coder-33b-instruct.Q4_K_M.gguf79.9%
OpenCodeInterpreterggml-opencodeinterpreter-ds-33b-q8_0.gguf, -ngl 40Failed
Deepseekggml-deepseek-coder-33b-instruct-q4_k_m.gguf78.7%
Deepseekdeepseek-coder-33b-instruct.Q5_K_M.gguf, -ngl 60, a bit slow79.3%
Llama3*together.ai API, Llama-3-70b-chat-hf75.6%
DBRX*together.ai API, dbrx-instruct48.8%
CodeQwencodeqwen-1_5-7b-chat-q8_0.gguf83.5%
Llama3-8Bbartowski/Meta-Llama-3-8B-Instruct-GGUF52.4%
Phi-3-mini4k context, 4bit quantized60.4%
Phi-3-mini4k context, fp16 quantized62.2%
Hermes-LlamaHermes-2-Pro-Llama-3-Instruct-Merged-DPO-F1653.7%
Codestral Q6_KCodestral-22B-v0.1-hf.Q6_K.gguf81.7%812.53s
Codestral Q8Codestral-22B-v0.1-hf.Q8_0.gguf79.9%2918.51s
Deepseek V2DeepSeek-Coder-V2-Lite-Instruct-Q8_0_L.gguf82.9%378.86s
Llama-3.1Meta-Llama-3.1-8B-Instruct.Q8_0_MaziyarPanahi.gguf57.9%304.09s
Qwen2.5-Coder-14BQwen2.5-Coder-14B-Instruct-Q8_0.gguf83.5%409.90s
Qwen2.5-Coder-32BQwen2.5-Coder-32B-Instruct-Q4_K_M.gguf89.6%375.33s
QwQ-32B-PreviewQwQ-32B-Preview-Q4_K_L.gguf52.4%11660.46s
DeepSeek-R1-1.5BDeepSeek-R1-Distill-Qwen-1.5B-Q6_K_L.gguf, temperature=0.657.3%4154.71s
DeepSeek-R1-8BDeepSeek-R1-Distill-Llama-8B-Q8_0.gguf, temperature=0.674.4%8836.76s
DeepSeek-R1-14BDeepSeek-R1-Distill-Qwen-14B-Q6_K_L.gguf, temperature=0.084.8%10444.61s
DeepSeek-R1-32BDeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf, temperature=0.690.2%12861.13s
Mistral-Small-24BMistral-Small-24B-Instruct-2501-Q6_K.gguf89.0%2365.32s
Simplescaling S1simplescaling_s1-32B-Q5_K_S.gguf68.3%1140.21s
Tqwendotqwendo-36b-Q4_K_L.gguf87.8%1916.67s
smollm2smollm2-1.7b-instruct-q4_k_m30.5%203.09s
tinyr1tinyr1-32b-preview-q4_k_m31.1%4992.37s
QwQQwQ-32B-Q4_K_M, max_tokens=1000078.0%19609.10s
QwQQwQ-32B-Q4_K_M, temperature=0.6, max-tokens=1000078.0%18689.78s
tinyr1tinyr1-32b-preview-q4_k_m, max-tokens=1000059.1%51286.32s
tinyr1tinyr1-32b-preview-q4_k_m, temperature=0.6, max-tokens=1000069.3%54596.08s
EXAONEEXAONE-Deep-32B-Q4_K_M, temperature=0.6, top-p=0.95, max-tokens=1000079.9%25417.01s
Mistral-Small-3.1-24Bmistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L89.0%2390.45s
DeepCoder-14B-Previewagentica-org_DeepCoder-14B-Preview-Q6_K_L, temperature=0.6, max-tokens=1000075.0%16987.95s
DevstralDevstral-Small-2505-Q4_K_M, start-problem=16390.2%42544.20s
Qwen3.5-27BQwen3.5-27B-Q4_K_M, max-tokens=1000098.2%3953.13s
Qwen3.5-35B-A3BQwen3.5-35B-A3B-Q4_K_M (MoE, 3B active), max-tokens=1000096.3%1289.29s
Qwen3.5Qwen3.5-35B-A3B-UD-Q4_K_XL, temperature=0.6, max-tokens=10000, context-size=3276897.0%1403.61s

LiveCodeBench

ModelConfigurationVersionScoreTime taken
smollm2smollm2-1.7b-instruct-q4_k_mLCBv54.2%259.23s
QwQQwQ-32B-Q4_K_MLCBv50.0%5196.35s
DevstralDevstral-Small-2505-Q4_K_M, start-problem=34LCBv58.6%57268.14s
Qwen3.5-27BQwen3.5-27B-Q4_K_M, max-tokens=10000LCBv58.8%
Qwen3.5-35B-A3BQwen3.5-35B-A3B-Q4_K_M (MoE, 3B active), max-tokens=32000LCBv565.5%5139.25s
agenticaagentica-org_DeepCoder-14B-Preview-Q6_K_L, max-tokens=10000, context-size=32768LCBv58.8%27151.90s
agenticaagentica-org_DeepCoder-14B-Preview-Q6_K_L, max-tokens=10000, context-size=32768LCBv616.4%28325.10s
DeepSeekDeepSeek-R1-Distill-Qwen-14B-Q6_K_L, temperature=0.6, max-tokens=10000, context-size=32768LCBv530.8%32852.81s
DeepSeekDeepSeek-R1-Distill-Qwen-14B-Q6/K_LLCBv634.8%33265.24s
mistralai/Mistralmistralai/Mistral-Small-3.1-24B-Instruct-2503-Q6_K_LLCBv639.3%1909.46s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-14B.i1-Q4/K_MLCBv614.9%32695.36s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-14B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv516.1%33253.31s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-14B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv615.3%32732.98s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-7B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv518.6%14692.69s
OpenCodeReasoningOpenCodeReasoning-Nemotron-1.1-7B.i1-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv625.1%14032.87s
Qwen2.5Qwen2.5-Coder-32B-Instruct-Q4_K_MLCBv546.0%2195.34s
Qwen2.5Qwen2.5-Coder-32B-Instruct-Q4_K_MLCBv644.7%2246.69s
Qwen3.5Qwen3.5-27B-Q4_K_M, temperature=0.6, max-tokens=10000, context-size=32768LCBv535.1%45214.12s
Qwen3.5Qwen3.5-35B-A3B-UD-Q4_K_XL, temperature=0.6, max-tokens=10000, context-size=32768LCBv535.8%15681.84s
Qwen3.5Qwen3.5-35B-A3B-UD-Q4_K_XL, temperature=0.6, max-tokens=10000, context-size=32768LCBv633.3%16048.97s
JanJan-code-4b-Q4_K_M, temperature=0.7, max-tokens=10000, top-p=0.8, min-p=0.0, context-size=32768LCBv642.5%5814.66s
JanJan-code-4b-Q8_0, temperature=0.7, max-tokens=10000, top-p=0.8, min-p=0.0, context-size=32768LCBv637.6%6616.07s
IQuestIQuest-Coder-V1-14B-Thinking, temperature=1.0, max-tokens=32000, top-p=0.95LCBv51.8%6915.48s
IQuestIQuest-Coder-V1-14B-Thinking, temperature=1.0, max-tokens=32000, top-p=0.95LCBv60.0%238.64s
agenticaagentica-org_DeepCoder-14B-Preview-Q6_K_L, max-tokens=32000, context-size=32768LCBv510.2%36732.02s

DeepSeek-R1 Distilled HumanEval Results

Model SizeTemperatureFrequency PenaltyAccuracyDeathloopsAvg. CharsSpeed (s)
1.5B0.00.050.6%29464,442
1.5B0.60.057.3%11,0034,155
1.5B0.01.056.1%04175,501
1.5B0.61.050.6%07924,828
14B0.00.084.8%175910,445
32B0.00.085.4%060315,655
32B0.60.090.2%069812,861
32B0.01.075.6%064517,182
Using the following quants from @Bartowski:
  1.5B: Qwen-1.5B-Q6_K_L.gguf
  14B: Qwen-14B-Q4_K_M.gguf
  32B: Qwen-32B-Q4_K_M.gguf
Deathloop is a response over 15k chars

API Results

ModelConfigurationHuman EvalTime taken
gpt-4o-2024-11-20*Don't respond with anything but code.86.2%174.97s
gpt-4o-2024-11-20*94.5%661.51s
Claude 3.5 SonnetOpenRouter95.1%1673.41s
Deepseek-R1OpenRouter31.1%15359.62s
Deepseek-R1OpenRouter. "Don't respond with anything but code"79.3%9819.20s
Deepseek-R1api.deepseek.com97.6%7625.37s
o3-mini96.3%1140.21s

Star History Chart

Significant stargazers

Victor Muštar

170 followers · starred Mar 2025

Similar repos

onurkanbakirci/google-colab-hf-human-eval

2

WizardLMTeam/WizardCoder-33B-V1.1

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

136

WizardLMTeam/WizardCoder-15B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

762

WizardLM/WizardCoder-15B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

762

WizardLMTeam/WizardCoder-Python-34B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

773

deepseek-ai/DeepSeek-Coder-V2

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

7,011

WizardLMTeam/WizardCoder-Python-13B-V1.0

Model

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

109

TheBloke/WizardCoder-15B-1.0-GGML

Model

WizardLM's WizardCoder 15B 1.0 GGML

115