GraphCodeBERT Fine-tuned on C++ and Erlang (MLM)

A fine-tuned version of microsoft/graphcodebert-base for Masked Language Modeling (MLM) on C++ and Erlang source code. The model is additionally evaluated on Python, Java, and JavaScript to assess cross-lingual transfer.

Model Details

Model Description

This model extends GraphCodeBERT's pre-training with continued MLM training on C++ and Erlang code. GraphCodeBERT incorporates data flow graphs (DFGs) alongside token sequences, enabling the model to capture semantic relationships between variables in addition to syntactic structure. The fine-tuning objective combines standard MLM loss with an edge-prediction loss over the DFG.

• Developed by: https://github.com/MatthewsO3/GraphCode-CErl-base
• Model type: Transformer encoder (RoBERTa-based), Masked Language Model
• Languages: C++, Erlang (training); Python, Java, JavaScript (zero-shot evaluation)
• License: MIT
• Finetuned from: microsoft/graphcodebert-base

Model Sources

• Repository: https://github.com/MatthewsO3/GraphCode-CErl-base

Uses

Direct Use

This model can be used as-is for masked token prediction in C++ and Erlang codebases — for example, code completion suggestions, token-level error detection, or identifier prediction. It can be loaded with the fill-mask pipeline:

from transformers import pipeline

fill = pipeline("fill-mask", model="MatthewsO3/GraphCode-CErl-base")
fill("int <mask> = 0;")

Downstream Use

The model can serve as a backbone for downstream code intelligence tasks such as:

• Code search and clone detection
• Defect prediction
• Variable misuse detection
• Cross-language code understanding (given its zero-shot transfer capability to Python, Java, and JavaScript)

Out-of-Scope Use

• Natural language understanding tasks
• Languages not represented in the training or base model data (performance may degrade significantly)
• Code generation (autoregressive generation — this is an encoder-only model)

Bias, Risks, and Limitations

• The Erlang training data was collected via a custom scraper and is not publicly available, which limits reproducibility for that language.
• C++, Python, Java, and JavaScript training data comes from GitHub, which may reflect biases in open-source coding style, library choices, and developer demographics.
• The model may perform poorly on domain-specific dialects or coding conventions not represented in GitHub data.
• No safety filtering beyond what is present in the upstream codeparrot/github-code-clean dataset was applied.

Recommendations

Users should validate the model's predictions on their specific codebase domain before deploying in production. For Erlang specifically, results may vary depending on how closely the target code resembles OTP-style patterns present in the scraping corpus.

How to Get Started with the Model

from transformers import RobertaTokenizer, RobertaForMaskedLM
import torch

tokenizer = RobertaTokenizer.from_pretrained("microsoft/graphcodebert-base")
model = RobertaForMaskedLM.from_pretrained("MatthewsO3/GraphCode-CErl-base")

code = "int <mask> = 42;"
inputs = tokenizer(code, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

mask_idx = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_id = logits[0, mask_idx].argmax(dim=-1)
print(tokenizer.decode(predicted_id))

Note: For full GraphCodeBERT-style inference with data flow graph inputs, refer to the evaluation script in the repository.

Training Details

Training Data

Total training tokens: ~500k code snippets
Validation split: 5% held out from training data

Python, Java, and JavaScript data from codeparrot/github-code-clean was used only for evaluation (2,500 samples each).

Training Procedure

The model was fine-tuned with the standard GraphCodeBERT MLM objective, which combines:

1. MLM loss — predict randomly masked tokens in the code sequence
2. Edge prediction loss — predict edges in the data flow graph (DFG)

Training Hyperparameters

Training Loss Progression

Best checkpoint: epoch 5 (val loss: 0.3701)

Hardware

• GPU: NVIDIA Quadro RTX 4000 (16 GB VRAM)

Evaluation

Testing Data, Factors & Metrics

Testing Data

2,500 samples per language were used for evaluation:

• C++ — codeparrot/github-code-clean
• Erlang — custom scraped dataset
• Python — codeparrot/github-code-clean (zero-shot)
• Java — codeparrot/github-code-clean (zero-shot)
• JavaScript — codeparrot/github-code-clean (zero-shot)

Metrics

• Top-1 Accuracy — fraction of masked tokens where the model's top prediction matches the original token
• Top-5 Accuracy — fraction of masked tokens where the correct token appears in the top 5 predictions
• Perplexity — exponentiated mean negative log-likelihood over masked tokens (lower is better)

Evaluation uses a mask ratio of 0.15, consistent with training.

Results

Final evaluation metrics (epoch 6):

C++ and Erlang (the two training languages) achieve the strongest results. The model shows solid zero-shot transfer to Java, and reasonable transfer to Python and JavaScript despite not being trained on those languages.

Summary

The model converges steadily across all 6 epochs. C++ and Erlang show the sharpest perplexity improvements in the first two epochs (from ~5.1 → ~2.1 and ~10.5 → ~2.1 respectively), then plateau. Java, Python, and JavaScript perplexity curves are flatter throughout, consistent with zero-shot generalization rather than direct training signal.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator.

• Hardware Type: NVIDIA Quadro RTX 4000 (16 GB VRAM)
• Hours used: 24 hour

Technical Specifications

Model Architecture and Objective

GraphCodeBERT uses a 12-layer RoBERTa-style transformer encoder. The model takes as input a concatenation of (1) code token sequences and (2) data flow graph node representations, with a 2D attention mask encoding which tokens and DFG nodes can attend to each other. The training objective is a sum of MLM loss and DFG edge prediction loss.

Compute Infrastructure

Hardware

NVIDIA Quadro RTX 4000, 16 GB VRAM

Software

• transformers (HuggingFace)
• torch
• tree-sitter (for DFG extraction during evaluation)

Citation

If you use this model, please also cite the original GraphCodeBERT paper:

BibTeX:

@inproceedings{guo2021graphcodebert,
  title={GraphCodeBERT: Pre-training Code Representations with Data Flow},
  author={Guo, Daya and Ren, Shuo and Lu, Shuai and Feng, Zhangyin and Tang, Duyu and Liu, Shujie and Zhou, Long and Duan, Nan and Svyatkovskiy, Alexey and Fu, Shengyu and Tufano, Michele and Deng, Shao Kun and Clement, Colin and Drain, Dawn and Sundaresan, Neel and Yin, Jian and Jiang, Daxin and Zhou, Ming},
  booktitle={International Conference on Learning Representations},
  year={2021}
}

Model Card Contact

https://github.com/MatthewsO3