---

What is Nexus?

Nexus is a standalone notification service that businesses call over an API to deliver notifications to their users across Email, In-App, and Webhook channels, without ever touching a mail provider's SDK or a retry loop themselves. It exists because notification delivery is not a feature you bolt onto a service, it's infrastructure, with the same correctness requirements as a payments system: a request must never be silently dropped, and it must never be processed twice. A company like Swiggy, Razorpay, or Groww doesn't want its order service to go down because Resend had a bad afternoon; Nexus is the isolation layer that makes that impossible.

---

The Problem

Every company that sends a user a "your order is confirmed" email, an in-app alert, or a server-to-server webhook eventually hits the same three failure modes if notification logic lives inside the business service itself:

1. Tight coupling kills reliability. Bury resend.emails.send() inside your order-placement code, and the day your email provider has an outage, your order service has an outage too, for a concern that has nothing to do with placing orders.

2. Synchronous delivery destroys latency. External providers are slow, occasionally unreachable, and entirely outside your control. Making a user's HTTP request wait on that round trip is an unforced error.

3. Network retries create duplicates and this is the one that actually breaks production. A client times out, retries the same logical request, and a naive system processes it twice: the user gets two "Order Confirmed" emails, delivery metrics are corrupted, and the database eats double the write load. Solving this correctly, at the database level, is the hardest part of this entire system and the part most side projects skip.

Nexus solves all three by being a dedicated, decoupled service: tenants register, get an API key, and send notifications through Nexus without ever thinking about delivery, duplication, or provider outages again.

---

Architecture

flowchart TD
    A["Client<br/>POST /api/v1/notification<br/>x-api-key + Idempotency-Key"] --> B[Correlation ID]
    B --> C[Auth: hash key → tenant]
    C --> D[Zod validation]
    D --> E["Idempotency<br/>Redis SET NX"]
    E --> F["Rate limit<br/>sliding window, Redis sorted set"]
    F --> G["Controller<br/>save Notification: PROCESSING"]
    G -->|201, non-blocking| A
    G --> H{channel}
    H -->|EMAIL| Q1[Email Queue]
    H -->|INAPP| Q2[In-App Queue]
    H -->|WEBHOOK| Q3[Webhook Queue]
 
    Q1 & Q2 & Q3 --> W["worker-1 / worker-2 / worker-3<br/>BullMQ Redis job locking"]
    W -->|success| DB[("PostgreSQL<br/>status: COMPLETED")]
    W -->|failure| R["Custom backoff<br/>1s→2s→4s→8s→16s (cap 30s)"]
    R -->|3 attempts exhausted| DLQ[Dead Letter Queue]
 
    W --> M["/metrics ×3<br/>ports 9101/9102/9103"]
    G --> M2["/metrics<br/>API process — queue_depth"]
    M & M2 --> P[Prometheus scrape + sum]
    P --> GR[Grafana dashboards]

Client → POST /api/v1/notification (with x-api-key + Idempotency-Key)
           │
           ▼
    ┌─────────────────────────────────────────────────────────┐
    │                   MIDDLEWARE CHAIN                      │
    │                                                         │ 
    │  1. Correlation ID  →  UUID stamped on every log        │
    │  2. Auth            →  Hash key → lookup tenant         │
    │  3. Zod Validate    →  Reject malformed before Redis    │
    │  4. Idempotency     →  Redis atomic SET NX              │
    │  5. Rate Limit      →  Sliding window (Redis sorted set)│
    └─────────────────────────────────────────────────────────┘
           │
           ▼
    Controller → Save to PostgreSQL (status: PROCESSING)
           │
           ▼
    Enqueue job to BullMQ (email / inapp / webhook queue)
           │
           ▼
    API responds 201 immediately ← non-blocking, fast
           │
           ▼
    ┌────────────────────────────────────────────────────────┐
    │              WORKER LAYER (3 containers)               │
    │                                                        │
    │  worker-1 ──┐                                          │
    │  worker-2 ──┼──► Pick job → Send → Record attempt      │
    │  worker-3 ──┘         │                                │
    │                       ▼                                │
    │              Retry with custom exponential backoff     │
    │              1s → 2s → 4s → 8s → 16s (capped at 30s)   │
    │                       │                                │
    │              After 3 attempts → Dead Letter Queue      │
    └────────────────────────────────────────────────────────┘
           │
           ▼
    Prometheus scrapes /metrics from API + all 3 workers
           │
           ▼
    Grafana dashboards → queue depth, delivery latency, error rates

The Three Channels

---

Tech Stack

---

Key Engineering Decisions

1. PostgreSQL over MongoDB

The schema is relational, not document-shaped: notifications belong to tenants, delivery attempts belong to notifications, and idempotency keys must be unique per tenant. Foreign keys enforce that integrity at the database level regardless of application bugs, and ACID guarantees mean a worker that crashes mid-write never leaves half-committed state. MongoDB was explicitly considered and rejected.

2. A queue, not synchronous delivery

Calling Resend's API inside the HTTP request cycle means the client waits on a third-party network call, a provider outage becomes an API outage, and there's no retry path, one failure is permanent. BullMQ decouples "accept the request" from "do the work": the API responds in milliseconds, and delivery with retries happens invisibly to the client.

3. Three separate queues, not one shared queue

If email, in-app, and webhook jobs shared a single queue and Resend went down, failing email jobs would pile up with backoff delays and block webhook and in-app jobs behind them classic head-of-line blocking. Three independent queues mean a broken email provider has zero effect on the other two channels, and each channel's health is observable on its own.

4. Three worker containers, not one

BullMQ's Redis-based job locking guarantees no two workers can grab the same job, exactly-once processing regardless of how many workers are running. Three containers give 3x throughput and fault isolation: if one crashes, two keep processing. Workers scale independently of the API, with zero code changes.

5. Idempotency runs before rate limiting in the middleware chain

A client retrying after a network timeout is resending the same logical request that retry should never cost the tenant a slot in their rate limit. Run rate limiting first, and a tenant with flaky network conditions burns their quota purely on retries of requests that already succeeded. Idempotency catches duplicates first; only genuinely new requests reach the limiter.

6. Sliding window rate limiting, not fixed window

Fixed windows have a boundary exploit: 100 requests at second 59 and 100 more at second 61 is 200 requests inside a 2-second span, invisible to a limiter that just resets a counter every 60 seconds. Nexus implements a sliding window with a Redis sorted set timestamps as scores. ZREMRANGEBYSCORE evicts anything older than the window, ZCARD counts what's left, ZADD records the new request. At any instant, looking back 60 seconds, the count is exact atomic, and shared correctly across every API instance. Optimization: To prevent Redis network latency from stacking up, the rate limiter uses a single, optimistic pipeline. It adds the new request and counts the window in one round-trip, only issuing a rollback command if the limit is exceeded. This halved the Redis latency on the happy path.

7. A custom BullMQ backoff function, not the built-in one

BullMQ's default exponential backoff has no ceiling a persistently failing job can end up retrying hours later. Nexus overrides it with Math.min(1000 * 2^attemptsMade, 30000), producing delays of 1s → 2s → 4s → 8s → 16s, hard-capped at 30 seconds. After 3 attempts the job moves to BullMQ's failed set, functioning as the dead letter queue.

8. API keys are hashed with SHA-256; the raw key is never stored

The raw key is shown exactly once, at tenant creation, then discarded server-side only its hash lives in the database, the same pattern used for passwords. If the database is ever breached, no usable key is exposed.

9. channel lives on Notification, not Template

The original schema put channel on Template. The flaw: a notification needs its channel to route to the correct BullMQ queue, and if that lived on the template, the service would need an extra DB round trip mid-request just to read it back. Channel is a property of the send, not the content the same "Welcome" template can legitimately go out as an email in one call and an in-app notification in another. It was moved to Notification.

10. One /metrics endpoint per worker, not a shared one

Each of the three worker containers is a separate Node.js process with separate memory metrics genuinely cannot be shared between them. Each exposes its own /metrics (mapped to host ports 9101/9102/9103), and Prometheus scrapes all three and aggregates with sum() at query time. queue_depth is deliberately reported by the API process only if all three workers also reported it, Prometheus would sum it to 3x the real value.

11. TypeScript across the entire codebase

A notification service moves data through five different systems on the way to delivery HTTP body, Postgres row, Redis cache entry, BullMQ job payload, Prisma-typed query result. A silent shape mismatch at any one of those boundaries either drops a notification or duplicates one. TypeScript makes every one of those boundaries a compile-time check instead of a runtime surprise.

12. Zod validation before anything touches Redis or Postgres

Validation sits immediately after auth and before idempotency, so a malformed payload never reaches the idempotency check, never writes a Redis key, and never opens a database round trip. Rejecting garbage input is cheap; rejecting it early is what keeps Redis and Postgres doing useful work under load.

13. Winston over console.log

console.log gives you unstructured text that's unsearchable at scale and impossible to thread a request through once it crosses into worker processes. Winston emits structured JSON with the correlation ID attached to every line, so a single failed delivery can be traced end-to-end API middleware, controller, queue enqueue, worker pickup, send attempt by searching one ID.

14. Developing inside Docker, not just deploying with it

macOS and Windows have case-insensitive filesystems by default; Linux what Render actually runs does not. Docker Compose locally runs the same Linux filesystem as production, which means a casing bug surfaces on a laptop instead of in a deploy log. This is not a hypothetical see below.

---

Production Bugs I Actually Fixed

The Prometheus scraping problem that came from replicas: 3

Symptom: three worker containers were running and processing jobs correctly, but Prometheus only ever showed metrics from one of them never all three, and never consistently the same one.

Root cause: Docker Compose's deploy: replicas: 3 shortcut creates three anonymous containers behind one service name. They're identical and interchangeable from Compose's point of view which is exactly the problem, because Prometheus needs a distinct address per target to scrape. It was hitting whichever container the network happened to route to at that moment and calling it done.

Fix: replaced the replica shortcut with three explicitly named services worker-1, worker-2, worker-3 each bound to its own host port (9101, 9102, 9103), and updated the Prometheus scrape config to list all three targets by name. Each worker's metrics are now individually addressable, and sum() at query time gives the real aggregate instead of a lucky guess.

The idempotency race condition that application logic alone can't close

The scenario: a client's request times out and it retries but the retry lands before the first request has finished writing its idempotency record. Both requests hit the Redis check simultaneously, both find nothing (correct, nothing's been written yet), and both proceed to process the notification. Two "Order Confirmed" emails go out. The Redis-first check didn't lie it was accurate at the exact instant it ran; the failure is in the gap between check and write.

Why the obvious fix doesn't work: adding an application-level "check, then write" guard doesn't close the window it's still two separate operations, and two concurrent requests can both pass the check before either completes the write. This has to be enforced somewhere that can't race with itself.

Fix: A database-level @@unique([idempotency_key, tenant_id]) constraint works, but executing fire-and-forget INSERTs on the hot path completely exhausts the Postgres connection pool under load, driving p95 latencies past 6 seconds. Instead, the race condition is closed entirely in memory using a single, atomic Redis SET NX (Not Exists) command. When two identical requests race, Redis guarantees only the first one successfully sets the key. The second request is atomically rejected and handled as a duplicate without ever opening a database connection. Redis acts as a distributed lock, providing correctness while maintaining sub-40ms median latencies.

---

Idempotency, in Detail

Every write request carries an Idempotency-Key header. On arrival:

1. Redis check (fast path). Key found and the body hash matches → the cached response is returned immediately, no database touch.
2. Body hash mismatch, same key. → 409 Conflict. The client is reusing a key for a genuinely different request; it needs a new one.
3. New request. Not found anywhere → the Redis SET NX succeeds, locking the key atomically for 24 hours. The request proceeds to rate limiting and the notification is queued. The atomic nature of SET NX guarantees that even if two simultaneous first-time requests race exactly at the same millisecond, only one will ever acquire the lock and proceed.

---

Observability

Metrics Collected

Dashboard Queries

# P95 delivery latency
histogram_quantile(0.95, sum(rate(notification_delivery_duration_seconds_bucket[5m])) by (le, channel))

# Error rate per channel
sum(rate(notifications_failed_total[5m])) by (channel)

# Queue depth   the primary scaling signal
queue_depth

A rising queue_depth means "spin up more workers." A notifications_failed_total spike isolated to one channel means "that channel's provider is down" without touching the other two. These are answered by a live dashboard, not a guess.

---

Database Schema

model Tenant {
  id                  String              @id @default(uuid())
  name                String
  from_email          String?
  api_keys            ApiKey[]
  notifications       Notification[]
  templates           Template[]
}

model ApiKey {
  id        String  @id @default(uuid())
  key_hash  String  @unique   // raw key is never persisted
  tenant_id String
  tenant    Tenant  @relation(fields: [tenant_id], references: [id])
}

enum Channel { WEBHOOK  INAPP  EMAIL }

model Notification {
  id                String             @id @default(uuid())
  channel           Channel            // lives here, not on Template   see decision #9
  tenant_id         String
  status            NotificationStatus // PROCESSING | COMPLETED | FAILED | SCHEDULED
  attempts          Int                @default(0)
  message           String?
  subject           String?
  template_id       String?
  recipient         String
  variables         Json?
  scheduledFor      DateTime?
  delivery_attempts DeliveryAttempt[]

  @@index([tenant_id])
  @@index([status])
  @@index([created_at])
}



model DeliveryAttempt {
  id              String                @id @default(uuid())
  notification_id String
  status          DeliveryAttemptStatus // PROCESSING | COMPLETED | FAILED
  error_message   String?

  @@index([notification_id])
}

DeliveryAttempt exists specifically so a notification's full retry history every attempt, its status, and its error is queryable after the fact, instead of only ever knowing the final outcome.

---

Performance

The load test (k6/load-test.js) ramps from 0 to 200 concurrent virtual users over two minutes and holds there, posting notifications against the live /api/v1/notification endpoint with a unique idempotency key per request, against thresholds of p95 latency under 1s and error rate under 1%. Run it yourself:

k6 run k6/load-test.js

At a sustained load of 100 requests per second (3,000 requests over 30s), Nexus achieves:

• Median Latency: ~35ms
• p95 Latency: ~957ms
• Error Rate: 0.01%
• Throughput: 100% success rate with zero dropped requests.

---

Known Limitations

Being upfront about the gaps is part of the engineering, not a weakness in it:

• No Outbox pattern yet. A worker sends the email, then updates the notification's status to COMPLETED in a separate DB write. If the worker crashes in that exact gap, the notification was delivered but its status stays PROCESSING forever. The correct fix is the Outbox pattern making the send confirmation and the DB write part of one atomic transaction. Documented, not yet implemented.

---

Local Setup

git clone https://github.com/priyansh00soni/Nexus
cd Nexus
cp .env.example .env
# fill in DATABASE_URL, REDIS_URL / REDIS_PASSWORD, RESEND_API_KEY
docker-compose up -d
npx prisma migrate deploy
npm run dev

Run the test suite and typecheck exactly as CI does:

npx tsc --noEmit
npm run test

---

API Reference

Full request/response schemas are live at SwaggerUi : /api/v1/api-docs. Summary:

Example: Send a Notification

curl -X POST https://nexus-w21t.onrender.com/api/v1/notification \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: order-confirmed-user-123" \
  -H "Content-Type: application/json" \
  -d '{
    "channel": "EMAIL",
    "recipient": "user@example.com",
    "subject": "Order Confirmed",
    "message": "Your order #456 has been placed successfully."
  }'

{
  "success": true,
  "statusCode": 201,
  "message": "Notification queued successfully.",
  "data": { "notification_id": "uuid", "status": "PROCESSING" }
}

Author

Priyansh Soni

Drop a 🌟 if you found this useful.