StackPulse

THE JAVASCRIPT ECOSYSTEM | OBSERVED DAILY

StackPulse crawls a rolling sample of public JavaScript and TypeScript repositories on GitHub and tracks which frameworks, databases, ORMs, and tools are actually used together, and how that adoption shifts over time. Every number below is generated by a daily pipeline and backed by an open dataset you can inspect yourself.

The Pulse (Updated: September 28, 2026)

🔗 View the live dashboard →

📊 Daily Snapshot

Trend Score is a log-scaled growth index: (current adoption / prior adoption) × log10(current adoption + 10), weighted so both the rate of change and the technology's overall scale matter.

🔥 Hot This Week

📉 Falling This Month

🏷️ Top Per Category

🔗 Top Pairings

Strength Score measures the co-occurrence affinity between two technologies using the Jaccard similarity coefficient:

$$ \frac{|A \cap B|}{|A| + |B| - |A \cap B|} $$

The number of repositories containing both divided by the number of repositories containing either. Scores range from 0 to 1.

Full dataset

These are committed to the repo automatically by the GitHub Actions pipeline (see Daily Pipeline), so you can browse or consume the data directly from this repository without running the app or standing up a database.

• technologies.json
• repositories.json
• pairings.json
• trending.json
• weekly-trending.json

License note: these datasets are licensed under CC-BY 4.0, separate from the MIT-licensed codebase. Attribution is required if you reuse them.

Note: cloning this repo gets you the code and the exported datasets. It does not give you a running instance of the website. The live app requires a deployed Next.js instance (Vercel) connected to a hosted PostgreSQL database. See Deployment.

Scope

Deliberately out of scope: full repository browsing/search. Technology pages include a read-only "top repositories" panel instead.

May be revisited later: knowledge graph visualization, migration detection, AI recommendations, multi-language ecosystem analysis, user accounts.

Repository Eligibility

Repositories are discovered by scanning recently updated public JS/TS repositories that meet eligibility requirements, and the crawler maintains a rolling sample rather than attempting full ecosystem coverage. We process repository technology links over a 30-day rolling window to ensure that data remains relevant and representative of current usage.

repoCount indicates the number of active, eligible repositories where a technology is currently detected.

Excluded: forks, archived repositories, disabled repositories, empty repositories.

Pages

Pages are React Server Components that fetch from the API routes above rather than querying the database directly, so there's a single data-access path for both the UI and any external consumer of the API.

Stack

Local Development

Target: a fully running local environment in under 15 minutes.

Prerequisites

• Git
• Node.js 20+
• Docker (for local PostgreSQL)
• A GitHub account and a GitHub Personal Access Token for the crawler

Setup

1. Clone the repository

git clone https://github.com/FarazKelhini/stack-pulse
cd stack-pulse

2. Install dependencies

npm install

3. Configure environment variables

Copy the example environment file:

Linux/macOS:

cp .env.example .env.local

Windows PowerShell:

Copy-Item .env.example .env.local

Then open .env.local and configure the required variables.

For the local PostgreSQL database created in the next step, use:

DATABASE_URL="postgresql://stackpulse:stackpulse@localhost:5432/stackpulse"

You'll also need to provide a GitHub token:

GITHUB_TOKEN="your_github_token"

The GitHub token is used by the crawler to access the GitHub API. Keep it private and never commit .env.local to Git.

4. Start local PostgreSQL

The following Linux/macOS command creates a PostgreSQL 16 container named stackpulse-db:

docker run -d \
  --name stackpulse-db \
  -e POSTGRES_DB=stackpulse \
  -e POSTGRES_USER=stackpulse \
  -e POSTGRES_PASSWORD=stackpulse \
  -p 5432:5432 \
  postgres:16

Or use this one line command that works identically in both Linux Bash and Windows PowerShell:

docker run -d --name stackpulse-db -e POSTGRES_DB=stackpulse -e POSTGRES_USER=stackpulse -e POSTGRES_PASSWORD=stackpulse -p 5432:5432 postgres:16

The database will be available at localhost:5432 with:

Already have a stackpulse-db container? You can start it again with:

docker start stackpulse-db

5. Run database migrations

npx prisma migrate dev

This creates/updates the local database schema.

6. Seed the technology dictionary

npx prisma db seed

This populates the database with the initial technology dictionary.

7. (Optional) Crawl some GitHub data

To populate the application with a small amount of data, run a crawl with a batch size of 1.

Linux/macOS:

CRAWL_BATCH_SIZE=1 npm run crawl

Windows PowerShell:

$env:CRAWL_BATCH_SIZE="1"; npm run crawl

The crawler uses the GitHub API, so a valid GITHUB_TOKEN is required.

8. Start the development server

npm run dev

Then open the URL shown in the terminal, typically:

http://localhost:3000

Troubleshooting

PostgreSQL connection error

Make sure the container is running:

docker ps

If it isn't running:

docker start stackpulse-db

Port 5432 is already in use

Another PostgreSQL instance may already be using port 5432. Either stop it or change the host-side port in the Docker command and update DATABASE_URL accordingly.

Port 3000 is already in use

The development server may choose another available port. Use the URL printed by npm run dev.

Environment variables

See .env.example for the full list. Required:

Optional (sensible defaults shown):

Scripts

npm run dev               # Start the Next.js dev server
npm run crawl             # Discover and ingest repositories
npm run refresh           # Maintenance: refresh existing repo links
npm run aggregate         # Compute technology pairings
npm run snapshot          # Generate daily adoption snapshots
npm run trends            # Compute trend scores
npm run export            # Export datasets to public/datasets/
npm run validate-exports  # Validate exported dataset schemas
npm run monitor           # Integrity check: find stale tech links
npm run generate-network  # Generate pairing network file
npm run test              # Run the Vitest suite

Daily pipeline

A scheduled GitHub Actions workflow (.github/workflows/daily-pipeline.yml) runs once per day (02:00 UTC) and can also be triggered manually:

1. prisma migrate deploy
2. crawl — Discovery-oriented repository traversal (crawl.ts)
3. refresh — Maintenance-oriented repository refresh (refresh.ts). Updates technology links for existing repos to keep the 30-day rolling window accurate.
4. aggregate — Compute technology pairings (aggregate.ts)
5. snapshot — Generate daily snapshots (snapshot.ts)
6. trends — Compute trend scores (trends.ts)
7. export — Generate and commit datasets (export.ts)
8. validate-exports — Validate exported dataset schemas
9. monitor — Non-blocking integrity check (monitor_stale.ts). Final step; failures do not halt the pipeline.
10. network - Generate pairing network file
11. commit — Commit updated datasets back to the repo

Every step is idempotent and resumable. Crawl failures and DB connection failures are treated as critical and halt the pipeline; failures in other steps degrade gracefully so later steps still run with whatever data is available.

Testing

• Framework: Vitest
• Unit tests: dictionary matching and deduplication, trend calculation, pairing (Jaccard) calculation
• Integration tests: repository ingestion upsert/diff logic, /api/search, /api/trending, /api/technology/[slug], export generation
• Integration tests run against TEST_DATABASE_URL, with migrations applied before the suite runs

Deployment

The app is designed for Vercel with a hosted PostgreSQL database. Deploying requires:

• A PostgreSQL 16+ instance reachable via DATABASE_URL
• The environment variables above configured as Vercel/Actions secrets
• The daily pipeline's GitHub Actions secrets (DATABASE_URL, GH_PAT) configured in the repo settings

Without a deployment, the datasets in public/datasets/ are still browsable directly from GitHub even though the interactive site (search, trending, technology pages) isn't live.

Success criteria

• 150+ technologies tracked
• 5,000+ repositories analyzed
• Search returns results in < 100ms p95
• Technology pages load with pairings and top repositories
• Pairings and trends precomputed
• Daily exports committed with valid schema
• Daily pipeline runs green unassisted
• Crawl resumes correctly after interruption
• Local setup completes in under 15 minutes

License

Code is licensed under MIT; datasets in public/datasets/ are licensed under CC-BY 4.0.