STRATA

STRATA (Software daTa Repository Analysis & Testing Architecture)

---

Usage

Strata includes two main programs, mine and analyze. Analyze is dependent on the results of mine, however it does not require mine to be run before each analysis. There other programs included in strata, compare, find, and sample. compare requires results from analyze. However, find and sample are standalone programs.

Strata is built to perform lexical analysis on Python, C, C++, Objective-C, and Java repositories.

Paper Results

To replicate the results outlined in the [Changes in Vocabulary Between Human-created and LLM-assisted code](paper/Taylor Lineman - Changes in Vocabulary Between Human-created and LLM-assisted Code Repositories.pdf) paper, you can follow the instructions outlined in REPRODUCTION.md

Main Program

The main program in strata is a jump spot for all of the underlying functionality of strata.

> ./strata --help
usage: strata [-h] [--bell] {mine,analyze,predict,sample,find,database,compare} ...

Mine or analyze a software repository.

options:
  -h, --help            show this help message and exit
  --bell                Ring a bell when done!

subcommands:
  {mine,analyze,predict,sample,find,database,compare}
    mine                mine help
    analyze             analyze help
    predict             predict help
    sample              sample help
    find                find help
    database            database help
    compare             compare help

Mine Subprogram

Mining is the first step in using strata. To gather data, you must first run mine on a GitHub repository by providing the owner/repo identifier. Strata will automatically clone the repository, mine commits, and extract information from the source code.

> ./strata mine --help
usage: strata mine [-h] [--token TOKEN] [--max-commits MAX_COMMITS]
                   [--max-files MAX_FILES] [--no-record-run] [--no-cleaning] [--no-srcml]
                   [--no-prs-issues] [--no-ci] [--sampling]
                   owner_repo

positional arguments:
  owner_repo            owner/repo (e.g. octocat/Hello-World) or local repo path

options:
  -h, --help            show this help message and exit
  --token TOKEN         GitHub token (or set GITHUB_TOKEN) for private repo cloning
  --max-commits MAX_COMMITS
                        Stop after this many commits (optional)
  --max-files MAX_FILES
                        What is the maximum number of source files to mine (optional)?
  --no-record-run       Do not write a run_log entry
  --no-cleaning         Do not clean the incoming commits, pull requests and issues.
  --no-srcml            Do not run srcml on the repo.
  --no-prs-issues       Do not collect any pull requests or issues.
  --no-ci               Do not collect any continuous integration.
  --sampling            Should sampling be run. Only works with source ML enabled.

Analysis Subprogram

After running mine on any number of repositories, analyze will use their data to extract useful insights. These insights will be output in the form of graphs (in output/graphs) and tables (in the console).

The analyze subprogram creates a cache of analyzed data in output/blobs. This makes iterating on output less time consuming.

> ./strata analyze --help
usage: strata analyze [-h] [--output OUTPUT] [--clusters CLUSTERS]
                      [--max-commits MAX_COMMITS] [--max-identifiers MAX_IDENTIFIERS]
                      [--max-comments MAX_COMMENTS] [--analysis-file ANALYSIS_FILE]
                      [--no-save] [--no-graph] [--no-stem] [--markdown]

options:
  -h, --help            show this help message and exit
  --output OUTPUT       Where should the output be saved?
  --clusters CLUSTERS   How many clusters should be used for analysis? (Default 5)
  --max-commits MAX_COMMITS
                        What is the maximum number of commits to analyze?
  --max-identifiers MAX_IDENTIFIERS
                        What is the maximum number of identifiers to analyze?
  --max-comments MAX_COMMENTS
                        What is the maximum number of comments to analyze?
  --analysis-file ANALYSIS_FILE
                        Path to an analysis file to analyze. Skips long builds by using
                        the results present in the file.
  --no-save             Do not save a cache of the analysis results
  --no-graph            Do not create any output graphs
  --no-stem             Do not reduce vocabulary into stems.
  --markdown            Produce markdown output tables.

Using an existing analysis dataset

To use a cache create by a previous run of analyze you just provide the name of the file in the output/blobs directory. This file can also be located outside the output/blobs directory, but requires the full system path to be found.

Using a cache is a great way to speed up iterations on output tweaks. Since no analysis needs to be run, you can repeatadly generate graphs on the same data without the need to go through the long analysis process.

./strata analyze --analysis-file <name-of-file-in-output-blobs>

Predict Subprogram

The predict command trains a classification model on analyzed commit messages. The classification model attempts to predict a commits "type" from the language used in the message. This type can be fix, feature, refactor, test, docs.

The main argument of the predict subprogram is the analysis_file this argument is a path to an analysis cache created by the analyze command.

> ./strata predict --help
usage: strata predict [-h] [--feature-matrix-file FEATURE_MATRIX_FILE] [--clusters CLUSTERS] [--output OUTPUT]
                      [--model-type {decision_tree,random_forest}] [--max-depth MAX_DEPTH]
                      [--commit-limit COMMIT_LIMIT] [--no-save] [--no-graph] [--no-tables] [--no-stem] [--markdown]
                      analysis_file

positional arguments:
  analysis_file         The analysis file to gather comment and token vocabularies from. Generated from running the
                        analyze command.

options:
  -h, --help            show this help message and exit
  --feature-matrix-file FEATURE_MATRIX_FILE
                        Path to a commit feature matrix file. Skips long builds by using the results present in the
                        file.
  --clusters CLUSTERS   How many clusters should be used for analysis? (Default 5)
  --output OUTPUT       Where should the output be saved?
  --model-type {decision_tree,random_forest}
  --max-depth MAX_DEPTH
  --commit-limit COMMIT_LIMIT
  --no-save             Do not save a cache of the analysis results
  --no-graph            Do not create any output graphs
  --no-tables           Do not create any output tables
  --no-stem             Do not reduce vocabulary into stems.
  --markdown            Produce markdown output tables.

A sample call to predict

./strata predict <name-of-analysis-in-output-blobs>

Using an existing feature matrix

On every run where --no-save is not defined, the predict subprogram saves the generated feature matrix so graphs and models can be recreated without the long data classification step. To use a cache created by a previous run, you can provide the name of the file in the output/blobs directory. This file can also be located outside the output/blobs directory, but requires the full system path to be found.

./strata predict <name-of-analysis-in-output-blobs> --feature-matrix-file <name-of-feature-matrix-in-output-blobs>

Sample Subprogram

The sample subprogram allows you to create a sample from data within a csv file. If a csv file contains a header row, stratified sampling can be performed across any of the columns.

> ./strata sample --help
usage: strata sample [-h] [--sample-size SAMPLE_SIZE] [--population-size POPULATION_SIZE] [--proportion PROPORTION] [--margin MARGIN] [--z_score Z_SCORE] [--seed SEED]
                     [--sample-method {SampleMethod.UNIFORM,SampleMethod.STRATIFIED,SampleMethod.SYSTEMATIC}] [--sample-size-method {SampleSizeMethod.PROPORTION,SampleSizeMethod.MEAN}]
                     [--csv CSV] [--csv-has-header] [--stratified-group STRATIFIED_GROUP] [--output OUTPUT]

options:
  -h, --help            show this help message and exit
  --sample-size SAMPLE_SIZE
  --population-size POPULATION_SIZE
                        How many samples should be used for sampling?
  --proportion PROPORTION
  --margin MARGIN
  --z_score Z_SCORE
  --seed SEED           A seed for random sampling. Makes the output deterministic.
  --sample-method {SampleMethod.UNIFORM,SampleMethod.STRATIFIED,SampleMethod.SYSTEMATIC}
                        The method used for sampling from provided data.
  --sample-size-method {SampleSizeMethod.PROPORTION,SampleSizeMethod.MEAN}
                        The method used for finding the size of the population sample.
  --csv CSV             A CSV file containing the data / population to sample from.
  --csv-has-header      A flag that informs the CSV parser whether or not the first row is a header row.
  --stratified-group STRATIFIED_GROUP
                        The CSV column that should be used to group the sampled data for stratified sampling.
  --output OUTPUT       Where should the sampled data be output?

Find Subprogram

The find subprogram was created to find repositories on GitHub that were created with the assistence of an LLM. It does this by incrementally searching GitHub repository creation events over a data range (typically one year). Found repos are then scanned for common indicators of LLM usage such as AI-Agent configuration files, and authored or co-authored commits from AI-Agents.

> ./strata find --help
usage: strata find [-h] [--token TOKEN] [--languages LANGUAGES [LANGUAGES ...]] [--minimum-stars MINIMUM_STARS] [--search-chunk-length SEARCH_CHUNK_LENGTH]
                   [--seconds-between-requests SECONDS_BETWEEN_REQUESTS] [--items-per-page ITEMS_PER_PAGE] [--all-repositories-csv ALL_REPOSITORIES_CSV]
                   [--all-repositories-csv-no-header] [--skip-agent-config-files] [--skip-commit-authors] [--output OUTPUT]

options:
  -h, --help            show this help message and exit
  --token TOKEN         GitHub token (or set GITHUB_TOKEN) for using the GitHub Search API.
  --languages LANGUAGES [LANGUAGES ...]
                        The repository language(s) to limit config file searching to. Will also filter repositories found from commit searching, but can not limit the search itself.
  --minimum-stars MINIMUM_STARS
                        The minimum number of stars for a repository to be considered during search.
  --search-chunk-length SEARCH_CHUNK_LENGTH
                        The length in minutes in search chunks used during finding all repositories.
  --seconds-between-requests SECONDS_BETWEEN_REQUESTS
                        The length between requests to the GitHub API.
  --items-per-page ITEMS_PER_PAGE
                        The number of items per page of GitHub search results.
  --all-repositories-csv ALL_REPOSITORIES_CSV
                        Path to a list of repositories to analyze for being AI Assisted. Generated by the first portion of the find command.
  --all-repositories-csv-no-header
                        Informs the CSV parser that there is no header for the repositories CSV file.
  --skip-agent-config-files
                        Do not use agent config files to find repositories.
  --skip-commit-authors
                        Do not use commit authors to find repositories.
  --output OUTPUT       Where should the sampled data be output?

Database Subprogram

The database command allows a user to save, and restore full database instances. This is extremely useful for switching between datasets for running analysis. The --restore command can kind of merge two databases by restoring one ontop of the other. This is often not what you want to do, so it is recommended to use teh --reset flag whenever restoring a database. The delete-repositories command is useful for removing a repository that failed to fully mine. This allows for more data integrity in a dataset.

> ./strata database --help
usage: strata database [-h] [--dump FILENAME] [--restore FILENAME] [--reset] [--delete-repositories DELETE_REPOSITORIES [DELETE_REPOSITORIES ...]]

options:
  -h, --help            show this help message and exit
  --dump FILENAME       Dump the database using 'pg_dump'.
  --restore FILENAME    Restore a database dump created by '--dump'.
  --reset               Reset the database (DESTRUCTIVE)!
  --delete-repositories DELETE_REPOSITORIES [DELETE_REPOSITORIES ...]

Saving and restoring a database

The following example shows you how to save a mined dataset (Human Repositories), then load a previously saved dataset (LLM-assisted).

# Dump the just mined dataset
./strata database --dump human.tar.gz

# Load a previously saved dataset
./strata database --reset --restore ai.tar.gz

# Load the human dataset again
./strata dtabase --reset --restore human.tar.gz

Compare Subprogram

The compare subprogram is used to compare two analysis files (which result from the analysis subprogram). The tool allows you to specify an dataset A and dataset B, as well as their names. Any output graphs or tables will display the name assocaited with the dataset they belong to. This tool produces all of the output used within the research paper.

> ./strata compare --help
usage: strata compare [-h] [--a-name A_NAME] [--b-name B_NAME] [--output OUTPUT] [--dimension-reduction-method {pca,tsne}] [--no-save] [--no-graph] [--no-tables] [--markdown]
                      analysis_file_a analysis_file_b

positional arguments:
  analysis_file_a       Path to the first analysis file to analyze.
  analysis_file_b       Path to the second analysis file to analyze

options:
  -h, --help            show this help message and exit
  --a-name A_NAME       What is the name of dataset A?
  --b-name B_NAME       What is the name of dataset B?
  --output OUTPUT       Where should the output be saved?
  --dimension-reduction-method {pca,tsne}
                        What method should be used for dimensionality reduction (pca or tsne)?
  --no-save             Do not save a cache of the analysis results
  --no-graph            Do not create any output graphs
  --no-tables           Do not create any output tables
  --markdown            Produce markdown output tables.

---

Repository Structure

.
├── config
├── output
├── paper
│   ├── analysis
│   ├── data
│   └── figures
├── reports
│   └── reports
├── schema
│   └── public
├── src
│   ├── analysis
│   ├── cleaning
│   ├── compare
│   ├── find
│   ├── git_miner
│   ├── modeling
│   ├── sampling
│   └── srcml
├── subcommands
└── test
    └── test_data
---

Setup Instructions

Cloning

git clone https://git.gccis.rit.edu/zml9650/strata
cd strata

Database Setup

• In CI, config/gitlab-credentials.yml is copied to config/db.yml automatically.
• For local dev, create a config/db.yml and config/db-test.yml with keys matching your own Postgres instance:

database: swen344
user: swen344
password: whowatchesthewatchmen
host: localhost
port: 5432

GitHub Personal Access Token (PAT)

Your workflow needs access to GitHub (for cloning private repositories and calling the GitHub API, etc), create a fine-grained personal access token following the official GitHub guide

After creating the token, set it in your environment as GITHUB_TOKEN so tools and CI can read it. For local shells you can run:

export GITHUB_TOKEN=<your-token-here>

Install dependencies

python3 -m venv .venv
source .venv/bin/activate # On unix systems
pip install -r requirements.txt