ghorg

Pronounced [gore-guh]; similar to gorge. You can use ghorg to gorge on orgs.

Use ghorg to quickly clone all of an orgs, or users repos into a single directory. This can be useful in many situations including

1. Searching an orgs/users codebase with ack, silver searcher, grep etc..
2. Bash scripting
3. Creating backups
4. Onboarding new team members (cloning all team repos)
5. Performing Audits

With default configuration ghorg performs two actions.

1. Will clone a repo if its not inside the clone directory.
2. If repo does exists locally in the clone directory it will perform a git pull and git clean on the repo.

So when running ghorg a second time on the same org/user, all local changes in the cloned directory by default will be overwritten by what's on GitHub. If you want to work out of this directory, make sure you either rename the directory or set the --no-clean flag on all future clones to prevent losing your changes locally.

Supported Providers

• GitHub (Self Hosted & Cloud)
  • Install | Setup | Examples
• GitLab (Self Hosted & Cloud)
  • Install | Setup | Examples
• Bitbucket (Cloud & Self-hosted Server)
  • Install | Setup | Examples
• Gitea (Self Hosted Only)
  • Install | Setup | Examples
• Codeberg (Cloud & Self Hosted Forgejo)
  • Install | Setup | Examples
• Sourcehut (Limited Features)
  • Install | Setup | Examples

The terminology used in ghorg is that of GitHub, mainly orgs/repos. GitLab and BitBucket use different terminology. There is a handy chart thanks to GitLab that translates terminology here. Note, some features may be different for certain providers.

High Level Features

• Filter or select specific repositories for cloning
• Create backups of repositories
• Simplify complex clone commands using reclone shortcuts
• Update every repo in a directory with pull, without discarding local changes
• Initiate clone operations via HTTP server
• Schedule cloning tasks using cron
• Monitor and track clone metrics over time

Installation

There are a installation methods available, please choose the one that suits your fancy:

• Prebuilt Binaries
• Homebrew
• Mise
• Golang
• Scoop
• Docker

For each installation method, optionally create a ghorg configuration file. See the configuration section for more details.

mkdir -p $HOME/.config/ghorg
curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-conf.yaml > $HOME/.config/ghorg/conf.yaml
vi $HOME/.config/ghorg/conf.yaml # To update your configuration

Prebuilt Binaries

See latest release to download directly for

• Mac (Darwin)
• Windows
• Linux

If you don't know which to choose its likely going to be the x86_64 version for your operating system.

Homebrew

brew install ghorg

Mise

If you use Mise, you can install the latest version of ghorg on Linux/MacOS/Windows:

mise use -g ghorg@latest

Golang

Ensure $HOME/go/bin is in your path

go install github.com/gabrie30/ghorg@latest

Scoop

Windows users can also install ghorg using scoop

scoop bucket add main
scoop install ghorg

Configuration

Precedence for configuration is first given to the flags set on the command-line, then to what's set in your $HOME/.config/ghorg/conf.yaml. This file comes from the sample-conf.yaml and can be installed by performing the following.

mkdir -p $HOME/.config/ghorg
curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-conf.yaml > $HOME/.config/ghorg/conf.yaml
vi $HOME/.config/ghorg/conf.yaml # To update your configuration

If no configuration file is found ghorg will use its defaults and try to clone a GitHub Org, however an api token is always required.

You can have multiple configuration files which is useful if you clone from multiple SCM providers with different tokens and settings. Alternative configuration files can only be referenced as a command-line flag --config.

If you have multiple different orgs/users/configurations to clone see the ghorg reclone command as a way to manage them.

Note: ghorg will respect the XDG_CONFIG_HOME environment variable if set.

SCM Provider Setup

Note: if you are running into issues, read the troubleshooting and known issues section below

GitHub Setup

1. Create Personal Access Token with all repo scopes. Update GHORG_GITHUB_TOKEN in your ghorg/conf.yaml or as a cli flag or place it in a file and add the path to GHORG_GITHUB_TOKEN. If your org has Saml SSO in front you will need to give your token those permissions as well, see this doc.
2. For cloning GitHub Enterprise (self hosted github instances) repos you must set --base-url e.g. ghorg clone <github_org> --base-url=https://internal.github.com
3. See examples/github.md on how to run

GitHub App Authentication (Advanced)

1. Create a GitHub App in your Organization. You only need to fill out the required fields. Make sure to give Repository Permissions -> contents -> read only permissions
2. Install the GitHub App into your Organization
3. Generate a a private key from the GitHub App, set the location of the key to GHORG_GITHUB_APP_PEM_PATH
4. Locate the GitHub App ID from the GitHub App, set the value to GHORG_GITHUB_APP_ID
5. Locate the GitHub Installation ID from the URL of the GitHub app, set the value to GHORG_GITHUB_APP_INSTALLATION_ID. NOTE: you will need to use the actual GitHub url to get this ID, go to your GitHub Organization Settings Page -> Third Party Access -> GitHub Apps -> Configure -> Get ID from URL

GitLab Setup

1. Create Personal Access Token with the read_api scope (or api for self-managed GitLab older than 12.10). This token can be added to your ghorg/conf.yaml or as a cli flag.
2. Update the GitLab Specific config in your ghorg/conf.yaml or via cli flags or place it in a file and add the path to GHORG_GITLAB_TOKEN
3. Update GHORG_SCM_TYPE to gitlab in your ghorg/conf.yaml or via cli flags
4. See examples/gitlab.md on how to run

Gitea Setup

1. Create Access Token (Settings -> Applications -> Generate Token)
2. Update GHORG_GITEA_TOKEN in your ghorg/conf.yaml or use the (--token, -t) flag or place it in a file and add the path to GHORG_GITEA_TOKEN.
3. Update GHORG_SCM_TYPE to gitea in your ghorg/conf.yaml or via cli flags
4. See examples/gitea.md on how to run

Codeberg Setup

Codeberg runs Forgejo, which is API-compatible with Gitea, so ghorg reuses its Gitea backend and defaults the base URL to https://codeberg.org (no --base-url required). Self-hosted Forgejo instances are also supported by setting --base-url; instances served over http additionally require --insecure-codeberg-client.

1. Create an Access Token (Settings -> Applications -> Generate Token) with at least the read:organization and read:repository scopes
2. Update GHORG_CODEBERG_TOKEN in your ghorg/conf.yaml or use the (--token, -t) flag or place it in a file and add the path to GHORG_CODEBERG_TOKEN.
3. Update GHORG_SCM_TYPE to codeberg in your ghorg/conf.yaml or via cli flags
4. See examples/codeberg.md on how to run

Sourcehut Setup

1. Create a Personal Access Token. Click "Limit scope of access grant", check "Generate read-only access token", then ctrl-click the REPOSITORIES and OBJECTS permissions.
2. Ensure you have added an SSH key if you want to clone private repos (sourcehut does not accept PATs in https URLs)
3. Update GHORG_SOURCEHUT_TOKEN in your ghorg/conf.yaml or use the (--token, -t) flag or place it in a file and add the path to GHORG_SOURCEHUT_TOKEN.
4. Update GHORG_SCM_TYPE to sourcehut in your ghorg/conf.yaml or via cli flags

Note on usernames: You can specify sourcehut usernames with or without the ~ prefix (e.g., both ghorg clone username and ghorg clone ~username work). Local folder paths will never include the ~ prefix to avoid shell expansion issues.

For detailed examples, API limitations, and sourcehut-specific features, see examples/sourcehut.md

Bitbucket Setup

Note: ghorg supports both Bitbucket Cloud and Bitbucket Server (self-hosted instances)

API Tokens (Recommended for Bitbucket Cloud)

Bitbucket has deprecated App Passwords in favor of API Tokens. This is the recommended authentication method for Bitbucket Cloud.

1. Create an API token from your Atlassian account settings
2. Important: When creating the token, grant all read scopes (Account: Read, Workspace membership: Read, Projects: Read, Repositories: Read, etc.) to ensure ghorg can list and clone repositories
3. Set GHORG_BITBUCKET_API_TOKEN in your $HOME/.config/ghorg/conf.yaml or use the --token flag
4. Set GHORG_BITBUCKET_API_EMAIL to your Atlassian account email (or use --bitbucket-api-email)
5. Update SCM type to bitbucket in your ghorg/conf.yaml or via cli flags
6. See examples/bitbucket.md on how to run

Note: When using API tokens, ghorg automatically uses x-bitbucket-api-token-auth as the Git username for clone operations, as required by Bitbucket's API token authentication.

App Passwords (Legacy)

Note: Bitbucket has deprecated App Passwords. Consider using API Tokens instead.

PAT/OAuth token

1. Create a PAT
2. Set the token with GHORG_BITBUCKET_OAUTH_TOKEN in your $HOME/.config/ghorg/conf.yaml or using the --token flag. Make sure you do not have --bitbucket-username set.
3. Update SCM TYPE to bitbucket in your ghorg/conf.yaml or via cli flags
4. See examples/bitbucket.md on how to run

Bitbucket Server (Self-hosted)

1. To configure with Bitbucket Server you will need to provide your instance URL via GHORG_SCM_BASE_URL in your $HOME/.config/ghorg/conf.yaml or use the --base-url flag.
2. Create credentials (username/password or app password) and update your configuration or use the --bitbucket-username and --token flags.
3. For insecure connections (HTTP), set GHORG_INSECURE_BITBUCKET_CLIENT=true
4. Update SCM type to bitbucket in your ghorg/conf.yaml or via cli flags
5. See examples/bitbucket.md on how to run

How to Use

See examples directory for more SCM specific docs or use the examples command e.g. ghorg examples gitlab

$ ghorg clone kubernetes --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
# Example how to use --token with a file path
$ ghorg clone kubernetes --token=~/.config/ghorg/gitlab-token.txt
$ ghorg clone davecheney --clone-type=user --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone gitlab-examples --scm=gitlab --preserve-dir --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone gitlab-examples/wayne-enterprises --scm=gitlab --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone all-groups --scm=gitlab --base-url=https://gitlab.internal.yourcompany.com --preserve-dir
$ ghorg clone --help
# view cloned resources
$ ghorg ls
$ ghorg ls someorg
$ ghorg ls someorg | xargs -I %s mv %s bar/

Changing Clone Directories

1. By default ghorg will clone the org or user repos into a directory like $HOME/ghorg/org. If you want to clone the org to a different directory use the --path flag or set GHORG_ABSOLUTE_PATH_TO_CLONE_TO in your ghorg conf. This value must be an absolute path. For example if you wanted to clone the kubernetes org to /tmp/ghorg you would run the following command.

   $ ghorg clone kubernetes --path=/tmp/ghorg
   

   which would create...

   /tmp/ghorg
   └── kubernetes
       ├── apimachinery
       ├── gengo
       ├── git-sync
       ├── kubeadm
       ├── kubernetes-template-project
       ├── ...
   

2. If you want to change the name of the directory the repos get cloned into, set the GHORG_OUTPUT_DIR in your ghorg conf or set the --output-dir flag. For example to clone only the repos starting with sig- from the kubernetes org into a direcotry called kubernetes-sig-only. You would run the following command.

   $ ghorg clone kubernetes --match-regex=^sig- --output-dir=kubernetes-sig-only
   

   which would create...

   $HOME/ghorg
   └── kubernetes-sig-only
       ├── sig-release
       ├── sig-security
       └── sig-testing
   

Selective Repository Cloning

Ghorg provides several optional ways to narrow down which repositories get cloned. Filters are applied in this order: flag-based filters → --target-repos-path → ghorgonly → ghorgignore → --repo-filter-hook. They can work in combination for a fine-grained control.

Flag-based filters

• Match by regex: use --match-regex to include, or --exclude-match-regex to exclude, repos whose names match a regex.
• Match by prefix: use --match-prefix to include, or --exclude-match-prefix to exclude, repos whose names start with one or more prefixes.
• Skip archived repos: use --skip-archived (not supported on Bitbucket).
• Skip forked repos: use --skip-forks.
• Filter by topic: use --topics (or GHORG_TOPICS) to clone only repos tagged with a matching topic. GitHub, GitLab, and Gitea only.

--target-repos-path - explicit list of repo names

Maintain a file containing the exact repository names you want to clone (one per line) and point at it with --target-repos-path (or the GHORG_TARGET_REPOS_PATH env var). Only repos whose name appears in the file will be cloned. This is the right choice when you have a fixed, known set of repos e.g. a curated list driving a CI pipeline or a documented backup set.

• Matching is done against the repo name only (the basename of the clone URL with .git stripped) and is case-insensitive and exact — partial names will not match.
• There is no default location; you must always pass an explicit path via the flag or env var.
• Names listed in the file that don't exist on the org/user are reported in the clone summary so you can catch typos or repos that were renamed/removed.
• When --clone-wiki is enabled, each listed name will also match its corresponding .wiki repo. GitLab snippets are matched against their parent repo name.

ghorgonly - auto-detected substring allowlist file

ghorgonly is a plain-text file you create yourself. If a file exists at $HOME/.config/ghorg/ghorgonly, ghorg will pick it up automatically on every clone, no flag required, and only clone repos whose clone URL contains one of the listed substrings (one pattern per line). This is useful when you want a dynamic, pattern-based subset of a large organization e.g. "everything published under the infra- namespace" without having to enumerate every repo name by hand.

• Matching is a plain substring check against the full clone URL, so patterns like infra-, internal/platform/, or a full URL all work.
• If no file exists at the default path, this filter is skipped entirely.
• To use a different location, or to maintain several ghorgonly files for different clone scenarios (e.g. one per team or per environment), pass --ghorgonly-path (or set GHORG_ONLY_PATH) to select the right file for that run.
• ghorgonly is applied after --target-repos-path and before ghorgignore, so the three can be layered (e.g., target a known list, narrow further by substring, then drop a few specific repos).

ghorgonly vs. --target-repos-path: use --target-repos-path when you know the exact set of repo names you want and want missing-repo warnings; use ghorgonly when you want a pattern-based subset of the org's URLs and don't need an explicit list.

ghorgignore - auto-detected denylist file

ghorgignore is a plain-text file you create yourself. If a file exists at $HOME/.config/ghorg/ghorgignore, ghorg will pick it up automatically on every clone, no flag required, and skip any repo whose clone URL contains a substring listed in the file (one pattern per line). This is the right tool for permanently excluding a small set of repos (legacy mirrors, vendor forks, repos you simply don't want on disk).

• Make each entry as specific as possible to avoid unintentional matches. For example, prefer a full clone URL like https://github.com/gabrie30/ghorg.git or git@github.com:gabrie30/ghorg.git over a short name fragment.
• If no file exists at the default path, this filter is skipped entirely.
• To use a different location, or to maintain several ghorgignore files for different clone scenarios (e.g. a stricter list for backups vs. a lighter list for day-to-day clones), pass --ghorgignore-path (or set GHORG_IGNORE_PATH) to select the right file for that run.

--repo-filter-hook - custom filtering with your own executable

When the built-in filters aren't enough, point --repo-filter-hook (or GHORG_REPO_FILTER_HOOK) at any executable. After all built-in filters run, ghorg writes the remaining repos as a JSON array to the hook's stdin and clones the JSON array the hook writes to stdout. Add --include-scm-data (see below) to filter on any field your SCM providers API returns.

• The hook runs last, so it has the final say on what gets cloned.
• The hook can modify repo fields, for example setting clone_branch per repo.
• Returning [] is valid and clones nothing.
• The hook inherits ghorg's environment, including GHORG_SCM_TYPE, GHORG_CLONE_TYPE, and credentials such as GHORG_GITHUB_TOKEN. This is by design, since you own the executable.
• Write progress or diagnostics to stderr. ghorg streams it to its own output.
• ghorg aborts the run if the hook is missing, exits non-zero, or writes invalid JSON, so a broken hook never clones an unfiltered list.
• A path without a separator, such as filter.sh, is looked up on PATH. Use ./filter.sh or an absolute path for a local script.
• GHORG_DEBUG=true prints the hook's input as indented JSON just before it runs. Debug output also contains your API token, both in ghorg's other debug lines and inside each clone_url when cloning over HTTPS, so keep it private.

Each repo object has id, name, host_path, path, url, clone_url, clone_branch, is_wiki, is_gitlab_snippet, is_gitlab_root_level_snippet, is_github_gist, gitlab_snippet_info, commits, and scm_data when --include-scm-data is set.

--include-scm-data - filter on anything the SCM provider returns

Set --include-scm-data (or GHORG_INCLUDE_SCM_DATA=true) with --repo-filter-hook to add a scm_data field to each repo object. It holds the provider's API object for that repo, so your hook can filter on size, visibility, last push date, license, or any other field. It is off by default because it adds about 5 KB per GitHub repo.

• scm_data is passed through as is, so field names and units differ between providers. For example, size is in KiB on GitHub and Gitea. See the fields GitHub returns for organization repos and user repos, and GitLab returns for group projects and user projects.
• Wikis carry their parent repo's scm_data, so a hook can skip a repo and its wiki together. GitHub gists and GitLab snippets carry their own.
• GitLab's runners_token is blanked.
• Sourcehut only includes the fields ghorg queries: id, name, visibility, owner, and HEAD.
• Bitbucket Cloud is not supported.
• scm_data is dropped after the hook runs, so editing it has no effect.

See examples/hooks for complete bash and python hooks, including filter-kubernetes-scm-data.sh, which clones the kubernetes GitHub org without archived repos, read-only staging mirrors, or repos with no pushes in the last year.

Creating Backups

When taking backups the notable flags are --backup, --clone-wiki, and --include-submodules. The --backup flag will clone the repo with git clone --mirror. The --clone-wiki flag will include any wiki pages the repo has. If you want to include any submodules you will need --include-submodules. Lastly, if you want to exclude any binary files use the the flag --git-filter=blob:none to prevent them from being cloned.

ghorg clone kubernetes --backup --clone-wiki --include-submodules

This will create a kubernetes_backup directory for the org. Each folder inside will contain the .git contents for the source repo. To restore the code from the .git contents you would move all contents into a .git dir, then run git init inside the dir, then checkout branch e.g.

# inside kubernetes_backup dir, to restore kubelet source code
cd kubelet
mkdir .git
mv -f * .git # moves all contents into .git directory
git init
git checkout master

Reclone Command

The ghorg reclone command is a way to store all your ghorg clone commands in one configuration file and makes calling long or multiple ghorg clone commands easier.

Once your reclone.yaml configuration is set you can call ghorg reclone to clone each entry individually or clone all at once, see examples below.

Each reclone entry can have:

• cmd: The ghorg clone command to execute (required)
• description: A description of what the command does (optional)
• post_exec_script: Path to a script that will be called after the clone command finishes (optional). The script will always be called, regardless of success or failure, and receives two arguments: the status (success or fail) and the name of the reclone entry. This allows you to implement custom notifications, monitoring, or other automation (optional)
• token_cmd: A command whose stdout is used as the token for this entry (optional). This lets you source a token from a secrets manager (e.g. 1Password, mise, pass, a keyring CLI) instead of storing it in cleartext. It overrides any global GHORG_TOKEN_CMD set in conf.yaml for this entry only, which is useful when your reclone entries span multiple providers/accounts that each require a different token. An explicit --token in the entry's cmd still takes precedence over token_cmd.

Example reclone.yaml entry:

gitlab-examples:
  cmd: "ghorg clone gitlab-examples --scm=gitlab --token=XXXXXXX"
  post_exec_script: "/path/to/notify.sh"

# Sources the token from a secrets manager instead of cleartext
gitlab-secrets-manager:
  cmd: "ghorg clone gitlab-examples --scm=gitlab"
  token_cmd: "op item get gitlab --fields token"

Example script for post_exec_script (e.g. /path/to/notify.sh):

#!/bin/sh
STATUS="$1"
NAME="$2"

if [ "$STATUS" = "success" ]; then
  # Success webhook
  curl -fsS https://hc-ping.com/your-uuid-here
else
  # Failure webhook
  curl -fsS https://hc-ping.com/your-uuid-here/fail
fi

# To clone all the entries in your reclone.yaml omit any arguments
ghorg reclone

# To run one or more entries you can pass arguments
ghorg reclone kubernetes-sig-staging kubernetes-sig

# To view all your reclone commands
# NOTE: This command prints tokens to stdout
ghorg reclone --list

Setup

Add a reclone.yaml to your $HOME/.config/ghorg directory. You can use the following command to set it for you with examples to use as a template

curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-reclone.yaml > $HOME/.config/ghorg/reclone.yaml

Update file with the commands you wish to run.

Automating Reclone

For automated execution, ghorg ships with two companion commands. See the linked examples for usage, flags, and endpoint details:

• ghorg reclone-server — Start an HTTP server that triggers reclone commands via HTTP requests. See examples/reclone-server.md.
• ghorg reclone-cron — Run reclone on a scheduled interval. See examples/reclone-cron.md.

Pull Command

ghorg pull [dir] a helper command that updates every git repo under dir (default: the current directory) without discarding local changes. It works on any directory of repos, whether ghorg cloned them or not, and uses your local git credentials.

# Update every repo in the current directory
ghorg pull .

# Update every repo in a ghorg clone directory
ghorg pull ~/ghorg/my-org

By default only fast-forward updates are made:

• The checked out branch is fast-forwarded from its upstream.
• The local default branch (from origin/HEAD) is fast-forwarded without being checked out, so it stays current while you work on another branch.
• A repo is skipped when it has uncommitted changes or untracked files, a detached HEAD, a branch with no upstream, a branch that has diverged from origin, or an unfinished merge or rebase.

Every skipped or failed repo is listed at the end with the reason:

Skipped (2)
  billing  feature/x: uncommitted changes (4 files); main +2
  infra    main: diverged from origin/main (1 ahead, 3 behind)

Failed (1)
  legacy  fetch failed: Authentication failed for 'https://github.com/my-org/legacy.git/'

Updated 5, up to date 6, skipped 2, failed 1

Add --verbose to see what is blocking repos with local changes. Under each repo skipped for uncommitted changes, the summary lists its untracked files and the first 100 lines of git diff HEAD, with the command to see the full diff when it is cut off. It has no effect with --reset-hard, which never skips repos for local changes.

Git uses your existing credentials, such as an SSH agent or a credential helper. ghorg removes tokens from remotes after cloning, so repos cloned over HTTPS need a credential helper (for example gh auth setup-git). HTTPS credential prompts are disabled, and SSH runs in batch mode unless you configured your own SSH command (GIT_SSH_COMMAND, GIT_SSH, or core.sshCommand), so a missing credential shows up as a failure instead of hanging.

To make every repo match origin's default branch and discard local changes, add --reset-hard. This is destructive: it discards uncommitted changes, deletes untracked files, checks out the default branch, and resets it to origin. Ignored files are kept and other branches are not modified. A repo whose default branch is checked out in another worktree is skipped. --reset-hard has no short form and cannot be set in conf.yaml.

ghorg pull ~/ghorg/my-org --reset-hard

Use --concurrency to change how many repos are updated at once (default: GHORG_CONCURRENCY, 25). Worktrees of the same repo are updated one after another because they share branches. Pull does not update submodule checkouts; run git submodule update in repos that use them.

The exit code is 0 when no repo failed and 1 when any repo failed. Skipped repos do not change the exit code.

Using Docker

The provided images are built for both amd64 and arm64 architectures and are available solely on Github Container Registry ghcr.io.

# Should print help message
# You can also specify a version as the tag, such as ghcr.io/gabrie30/ghorg:v1.9.9
docker run --rm ghcr.io/gabrie30/ghorg:latest

Note: There are also tags available for the latest on trunk, such as master or master-<commit SHA 7 chars>, but these are not recommended.

The commands for ghorg are parsed as docker commands. The entrypoint is the ghorg binary, hence you only need to enter remaining arguments as follows:

docker run --rm ghcr.io/gabrie30/ghorg \
    clone kubernetes --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2

The image ships with the following environment variables set:

GHORG_CONFIG=/config/conf.yaml
GHORG_RECLONE_PATH=/config/reclone.yaml
GHORG_ABSOLUTE_PATH_TO_CLONE_TO=/data

These can be overriden, if necessary, by including the -e flag to the docker run comand, e.g. -e GHORG_GITHUB_TOKEN=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2.

Persisting Data on the Host

In order to store data on the host, it is required to bind mount a volume:

• $HOME/.config/ghorg:/config: Mounts your config directory inside the container, to access config.yaml and reclone.yaml.
• $HOME/repositories:/data: Mounts your local data directory inside the container, where repos will be downloaded by default.

docker run --rm \
        -e GHORG_GITHUB_TOKEN=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2 \
        -v $HOME/.config/ghorg:/config `# optional` \
        -v $HOME/repositories:/data \
        ghcr.io/gabrie30/ghorg:latest \
        clone kubernetes --match-regex=^sig

Note: Altering GHORG_ABSOLUTE_PATH_TO_CLONE_TO will require changing the mount location from /data to the new location inside the container.

A shell alias might make this more practical:

alias ghorg="docker run --rm -v $HOME/.config/ghorg:/config -v $HOME/repositories:/data ghcr.io/gabrie30/ghorg:latest"

# Using the alias: creates and cleans up the container
ghorg clone kubernetes --match-regex=^sig

Tracking Clone Data Over Time

To track data on your clones over time, you can use the ghorg stats feature. It is recommended to enable ghorg stats in your configuration file by setting GHORG_STATS_ENABLED=true. This ensures that each clone operation is logged automatically without needing to set the command line flag --stats-enabled every time. The ghorg stats feature is disabled by default and needs to be enabled.

When ghorg stats is enabled, the CSV file _ghorg_stats.csv is created in the directory specified by GHORG_ABSOLUTE_PATH_TO_CLONE_TO. This file contains detailed information about each clone operation, which is useful for auditing and tracking purposes such as the size of the clone and the number of new commits over time.

Below are the headers and their descriptions. Note that these headers may change over time. If there are any changes in the headers, a new file named _ghorg_stats_new_header_${sha256HashOfHeader}.csv will be created to prevent incorrect data from being added to your CSV.

• datetime: Date and time of the clone in YYYY-MM-DD hh:mm:ss format
• clonePath: Location of the clone directory
• scm: Name of the source control used
• cloneType: Either user or org clone
• cloneTarget: What is specified after the clone command ghorg clone <target>
• totalCount: Total number of resources expected to be cloned or pulled
• newClonesCount: Sum of all new repos cloned
• existingResourcesPulledCount: Sum of all repos that were pulled
• dirSizeInMB: The size in megabytes of the output dir
• newCommits: Sum of all new commits in all repos pulled
• cloneInfosCount: Number of clone Info messages
• cloneErrorsCount: Number of clone Issues/Errors
• updateRemoteCount: Number of remotes updated
• pruneCount: Number of repos pruned
• hasCollisions: If there were any name collisions, only can happen with gitlab clones
• ghorgignore: If a ghorgignore was used in the clone
• ghorgonly: If a ghorgonly was used in the clone
• totalDurationSeconds: Total time in seconds for the entire clone operation
• ghorgVersion: Version of ghorg used in the clone

Converting CSV to JSON

go install github.com/gabrie30/csvToJson@latest && \
csvToJson _ghorg_stats.csv

Profiling Clones

ghorg can profile its own clone runs with pprof (--pprof) and runtime execution tracing (--trace). See examples/profiling.md for full documentation, or read it in your terminal with ghorg examples profiling.

Troubleshooting

• If you are having trouble cloning repos. Try to clone one of the repos locally e.g. manually running git clone https://github.com/your_private_org/your_private_repo.git if this does not work, ghorg will also not work. Your git client must first be setup to clone the target repos. If you normally clone using an ssh key use the --protocol=ssh flag with ghorg. This will fetch the ssh clone urls instead of the https clone urls.
• If you are cloning a large org you may see Error: open /dev/null: too many open files which means you need to increase your ulimits, there are lots of docs online for this. Another solution is to decrease the number of concurrent clones. Use the --concurrency flag to set to lower than 25 (the default)
• If your GitHub org is behind SSO, you will need to authorize your token, see here
• If your GitHub Personal Access Token is only finding public repos, give your token all the repos permissions
• Make sure your $ git --version is >= 2.19.0
• Check for other software, such as anti-malware, that could interfere with ghorgs ability to create large number of connections, see issue 132. You can also lower the concurrency with --concurrency=n default is 25.
• To debug yourself you can call ghorg with the GHORG_DEBUG=true env e.g GHORG_DEBUG=true ghorg clone kubernetes. Note, when this env is set concurrency is set to a value of 1 and will expose the api key used to stdout.
• If you've gotten this far and still have an issue feel free to raise an issue
• If you’re cloning using https, but you have submodules which are configured to use ssh, you can force git to pull these submodules as well via https by running these commands before running ghorg:

  git config --global url."https://github.com/".insteadOf git@github.com:
  git config --global credential.https://github.com/.helper '! f() { echo username=x-access-token; echo password=$GHORG_GITHUB_TOKEN; };f'