PRI-g46: NPM Package Search

Milestone 1

Requirements

nvm

If you encounter any issues installing nvm, visit the official repo for more guidance. This is the official cURL command to download the script and install the latest version (v0.40.3) at time of writing:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash

Install node v22.20.0 using nvm:

nvm install

unzip

From Stack Overflow:

If the unzip command isn't already installed on your system (use which unzip to check), then run:

sudo apt-get install unzip

.env

GitHub Personal Token (GitHub PA) is required to run the ./milestone1/fetchGithubData.sh script. The GitHub PA is required to use the GitHub GraphQL API. Please copy the .env.example file, delete the .example from the filename (so it is .env) and set your GitHub PA token there. This GitHub PA does not need to have any kind of permissions for your own account or repos.

Example file:

GITHUB_PA_TOKEN=<YOUR_GITHUB_PA_TOKEN>

Usage

Fetch Data

Initial data is from the kaggle data set: https://www.kaggle.com/datasets/mikelanciano/top-npm-github-repositories

Run the following bash script to download and process all info:

# Fetch top npm github repo csv:
curl -L -o ./milestone1/top-npm-github-repositories.zip https://www.kaggle.com/api/v1/datasets/download/mikelanciano/top-npm-github-repositories
unzip top-npm-github-repositories -d ./milestone1/top-npm-github-repositories

# Download metadata and readme
./milestone1/run.sh

# Remove packages that returned an error from NPM
./milestone1/removeErrorPackages.sh

# Get github repo info from packages' directories and save it to githubRepos.json and githubRepoTopackage.json
./milestone1/getGithubRepos.sh

# Get github data from github graphQL api and save it to githubData.json
./milestone1/fetchGithubData.sh

# Remove packages with deleted githubData repos
./milestone1/removeDeletedRepos.sh

Load Data to DuckDB

Make sure you are using correct node version and install dependencies:

nvm use
yarn

Run the following node code to load info to duckDB:

yarn createDatabase

Save duckDB to output.json:

yarn exportToJson

Fetch info on a packge

Fetch info for a package:

yarn getPackageInfo '1-liners'

Milestone 2

Requirements

Docker

Make sure Docker is installed and running. You can check by running:

docker --version

jq (optional)

For testing search queries, install jq:

sudo apt-get install jq

Usage

Complete Setup (Recommended)

Run the complete Milestone 2 setup with command:

cd milestone2
./activateHandler.sh

This script will:

• Start Solr in a Docker container
• Configure the custom schema
• Transform data to Solr-compatible format
• Index all NPM packages
• Verify the setup

Manual Step-by-Step Setup

If you prefer to run each step individually:

Start Solr Search Engine

cd milestone2
docker run -d -p 8983:8983 --name npm -e SOLR_OPTS="-Dsolr.modules=extraction" -v ${PWD}:/data solr:9 solr-precreate npm

Configure Solr Schema

curl -X POST -H 'Content-type:application/json' --data-binary @milestone2/solr-schema.json http://localhost:8983/solr/npm/schema

Transform Data for Solr

nvm use
node ./milestone2/prepare-solr-data.js

Index Data in Solr

curl -X POST -H 'Content-Type: application/json' --data-binary @solr-data.json 'http://localhost:8983/solr/npm/update?commit=true'

Access Search Interface

Web Interface: Open http://localhost:8983/solr/#/npm/query in your browser