internlm/internlm2_5-step-prover

Model

InternLM2.5-Step-Prover

5

3 commits

1 linked in READMEs

updated Oct 22, 2024

See the code

README

InternLM2.5-Step-Prover

InternLM-Math HOT

A state-of-the-art LEAN4 step prover.

💻 Github 📊Dataset 📖 Paper

InternLM2.5-Step-Prover is a 7B language model which achieves state-of-the-art performances on MiniF2F, ProofNet, and Putnam math benchmarks, showing its formal math proving ability in multiple domains.

Dialogue Example

### Input template
f"---\nNAME: {theorem.full_name}\n\n"
f"---\nPROOF_BEFORE: {proof_before}\n\n"
f"---\nSTATE_BEFORE: {state}\n\n"
f"---\nTACTIC: "
### Input example
---
NAME: square_sub_one_divisible_eight
---
PROOF_BEFORE: rw [h, pow_two]
---
STATE_BEFORE: m n : N
h : n = 2 * m + 1
⊢ 8 | (2 * m + 1) * (2 * m + 1) - 1
---
TACTIC:
### Output example
rw [← Nat.mod_add_div (2 * m + 1) 8]

If you want to use critic model, please refer critic's model page.

Performance

MiniF2F

MethodModel sizePassminiF2F-validminiF2F-test
Whole-Proof Generation Methods
GPT-4-turbo 0409-6425.4%23.0%
DeepSeekMath-Base7B12825.4%27.5%
DeepSeek-Prover7B1-30.0%
64-46.3%
128-46.3%
8192-48.8%
65536-50.0%
cumulative60.2%52.0%
DeepSeek-Prover-1.57B32-63.5%
TheoremLlama-cumulative36.5%33.6%
Tree Search Methods
COPRA (GPT-3.5)-1-9.0%
COPRA (GPT-4)-1-26.6%
DSP(Isabelle)540B10042.6%38.9%
Proof Artifact Co-Training837M123.9%24.6%
829.3%29.2%
ReProver229M1-25.0%
Llemma7B126.2%26.2%
Llemma34B127.9%25.8%
Curriculum Learning837M133.6%29.6%
841.2%34.5%
6447.3%36.6%
Hypertree Proof Search600Mcumulative58.6%-
64-41.0%
Lean-STaR7B64-46.3%
InternLM2-Math7B129.9%30.3%
InternLM2-Math-Plus7B1-43.4%
InternLM2-Step-Prover7B159.8%48.8%
InternLM2.5-Step-Prover7B155.4%47.3%
InternLM2.5-Step-Prover+Critic7B25669.6%65.9%

Proofnet & Putnam

MethodModel sizePassresult
ProofNet benchmark
ReProver229M113.8%
InternLM2-Step-Prover7B118.1%
InternLM2.5-Step-Prover7B25627.0%
Putnam benchmark
GPT-4-101/640
COPRA (GPT-4)-101/640
DSP(Isabelle)540B104/640
ReProver229M10/640
InternLM2-Step-Prover7B15/640
InternLM2.5-Step-Prover7B16/640

Citation and Tech Report

@misc{wu2024internlm25stepproveradvancingautomatedtheorem,
      title={InternLM2.5-StepProver: Advancing Automated Theorem Proving via Expert Iteration on Large-Scale LEAN Problems}, 
      author={Zijian Wu and Suozhi Huang and Zhejian Zhou and Huaiyuan Ying and Jiayu Wang and Dahua Lin and Kai Chen},
      year={2024},
      eprint={2410.15700},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2410.15700}, 
}
conversational
custom_code
internlm2
math
pytorch
text-generation

Contributors

InternLM-Math

3 commits

internlm/internlm2_5-step-prover

Model

InternLM2.5-Step-Prover

5

3 commits

1 linked in READMEs

updated Oct 22, 2024

See the code

README

InternLM2.5-Step-Prover

InternLM-Math HOT

A state-of-the-art LEAN4 step prover.

💻 Github 📊Dataset 📖 Paper

InternLM2.5-Step-Prover is a 7B language model which achieves state-of-the-art performances on MiniF2F, ProofNet, and Putnam math benchmarks, showing its formal math proving ability in multiple domains.

Dialogue Example

### Input template
f"---\nNAME: {theorem.full_name}\n\n"
f"---\nPROOF_BEFORE: {proof_before}\n\n"
f"---\nSTATE_BEFORE: {state}\n\n"
f"---\nTACTIC: "
### Input example
---
NAME: square_sub_one_divisible_eight
---
PROOF_BEFORE: rw [h, pow_two]
---
STATE_BEFORE: m n : N
h : n = 2 * m + 1
⊢ 8 | (2 * m + 1) * (2 * m + 1) - 1
---
TACTIC:
### Output example
rw [← Nat.mod_add_div (2 * m + 1) 8]

If you want to use critic model, please refer critic's model page.

Performance

MiniF2F

MethodModel sizePassminiF2F-validminiF2F-test
Whole-Proof Generation Methods
GPT-4-turbo 0409-6425.4%23.0%
DeepSeekMath-Base7B12825.4%27.5%
DeepSeek-Prover7B1-30.0%
64-46.3%
128-46.3%
8192-48.8%
65536-50.0%
cumulative60.2%52.0%
DeepSeek-Prover-1.57B32-63.5%
TheoremLlama-cumulative36.5%33.6%
Tree Search Methods
COPRA (GPT-3.5)-1-9.0%
COPRA (GPT-4)-1-26.6%
DSP(Isabelle)540B10042.6%38.9%
Proof Artifact Co-Training837M123.9%24.6%
829.3%29.2%
ReProver229M1-25.0%
Llemma7B126.2%26.2%
Llemma34B127.9%25.8%
Curriculum Learning837M133.6%29.6%
841.2%34.5%
6447.3%36.6%
Hypertree Proof Search600Mcumulative58.6%-
64-41.0%
Lean-STaR7B64-46.3%
InternLM2-Math7B129.9%30.3%
InternLM2-Math-Plus7B1-43.4%
InternLM2-Step-Prover7B159.8%48.8%
InternLM2.5-Step-Prover7B155.4%47.3%
InternLM2.5-Step-Prover+Critic7B25669.6%65.9%

Proofnet & Putnam

MethodModel sizePassresult
ProofNet benchmark
ReProver229M113.8%
InternLM2-Step-Prover7B118.1%
InternLM2.5-Step-Prover7B25627.0%
Putnam benchmark
GPT-4-101/640
COPRA (GPT-4)-101/640
DSP(Isabelle)540B104/640
ReProver229M10/640
InternLM2-Step-Prover7B15/640
InternLM2.5-Step-Prover7B16/640

Citation and Tech Report

@misc{wu2024internlm25stepproveradvancingautomatedtheorem,
      title={InternLM2.5-StepProver: Advancing Automated Theorem Proving via Expert Iteration on Large-Scale LEAN Problems}, 
      author={Zijian Wu and Suozhi Huang and Zhejian Zhou and Huaiyuan Ying and Jiayu Wang and Dahua Lin and Kai Chen},
      year={2024},
      eprint={2410.15700},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2410.15700}, 
}
conversational
custom_code
internlm2
math
pytorch
text-generation

Contributors

InternLM-Math

3 commits