What is it?

Swallow-code-v0.1 consists of 4 staged dataset subsets and are filtered from bigcode/the-stack-v2-train-smol-ids.

What is being released?

The dataset is released in four versions:

• Swallow Code v0.1 stage 1: 36B tokens, 41M documents containing Python scripts.
• Swallow Code v0.1 stage 2: 31B tokens, 37M documents containing Python scripts that are syntax error-free.
• Swallow Code v0.1 stage 3: 20B tokens, 24M documents containing Python scripts that are filtered with pylint score.
• Swallow Code v0.1 stage 4: 16B tokens, 21M documents containing Python scripts that are filtered with code comments and literal language detection(English and Japanese).

Results and Performance

Llama-3.1-8B Performance after Continual Pretraining on 50B tokens Japanese, English, and Code(= swallow-code-v0.1) datasets.

Dataset Schema

{
  "blob_id": string,
  "path": string,
  "content_id": string,
  "language": string,
  "length_bytes": int64,
  "detected_licenses": list,
  "license_type": string,
  "src_encoding": string,
  "is_vendor": bool,
  "is_generated": bool,
  "alphanum_fraction": float64,
  "alpha_fraction": float64,
  "num_lines": int64,
  "avg_line_length": float64,
  "max_line_length": int64,
  "text": string,
  "analysis_results": list,
  "has_issues": bool,
  "language_type_issue": list,
  "language_type": string,
  "pylint_score": int64,
  "pylint_output": string
}

Licensing information

Swallow-code-v0.1 follows the license of the stack v2. The following is the license of the stack v2.

The Stack v2 is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack v2 must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point.

Citation information

@misc{fujii2024swallowcode,  
    author       = { Kazuki Fujii, Rio Yokota },  
    title        = { Swallow-Code-v0.1 }, 
    year         = 2024,  
    url          = { https://huggingface.co/datasets/tokyotech-llm/swallow-code-v0.1 },  
	publisher    = { Swallow Project }
}