🧠💻 Reimagine the Internet as Self-Organizing Mind Map

Critical times call for critical thinkers to create a crowdsourced argument reasoning dataset, for AI models to recommend research quotes, to evolve crowdsourced chain-of-thought reasoning, to unlock faster ways to read long articles, to monitor developments by topic modeling a knowledge base graph, and to provide a public service of answers to research. Language Models can distill the essence of collective thought into a vector space where every point has a weighted value representing its contribution to the overall decision-making process. AI will show its reasoning based on what sentences and cites it used from the collective research, so that people can see it is aligned with our interests. Research Agents recommend articles for human researchers working alongside AI to develop a summarized topic outline as a public service. The agents monitor for any related articles via web searches for keywords associated with that Topic Model. Imagine uploading a research paper, then the app extracts full text of reference cites and creates topic model and keyword summaries, then monitors that literature base and stores highlights. People will make personal knowledge bases of what influences them to create AI assistants cloning their mind-uploaded perspective and interests in a self-organizing mind map.

# Download Source
bun x git0 vtempest/qwksearch-research-agent

# Import API Client
bun i qwksearch-api-client

🤖🔎 STREAM: Search with Top Result Extraction & Answer Model

Reimagine The Web as Self-Organizing Mind Map

• 🔍 Web Search - 100+ popular sites search across 10 categories: Web Search, Academic, Videos, Images, Files, News, etc
• 📝 Article Preview - Extract, format with APA cite, and summarize articles, PDFs, Youtube, and URLs before reading them
• 🤖 Choice of Language Model - Claude, GPT, Gemini, Grok, Llama, Chinese LLMs, etc
• 📄 File Upload Support - Ask questions about PDFs, URLs, DOCX, Google Docs, and Youtube
• 📚 Search History - All searches saved with memories, except for privacy mode
• ❓ Follow-up Questions - Generate follow-up questions to ask language models

REASON Writing Agent

Research Editor for Annotated Summaries in Outline Notation

• 📝 Complex Rich Text Editor: full featured alternative to Google Docs based on Meta's Lexical with easy minimalist UI
• 📂 Nested Document Tree: organize research notes with a nested document organizer with drag-and-drop, tabs, and custom storage sources
• 🖱️ Context Menu: right-click to access quick actions for seamless document management
• 🔍 Full-Text Search: instantly find documents by title or content with full-text search
• ✨ AI Rewriting: leverage AI to rewrite and improve your text directly within the editor
• 👥 Team Management: collaborate with team members and manage access rights
• 🔄 View Modes: switch between Formatted, parsing HTML, and Markdown views for versatile editing
• 🛠️ Find & Replace: powerful search and replace functionality with match highlighting
• 📥 Google Docs Integration: seamless export, import, and sharing capabilities
• 💾 Persistent Storage: reliable SQLite storage ensures your data is safe and accessible
• ⌨️ Keyboard Navigation: efficient keyboard shortcuts for power users
• 💬 Research Quotes: capture and organize key quotes and insights from your research
• 🏗️ Built with: Claude Code, Lexical Editor, shadcn/ui, Cloudflare, Next.js, Google Docs API

🚜📜 Tractor the Text Extractor

1. Main Content Detection: Extract the main content from a URL by combining Mozilla Readability and Postlight Mercury algorithms, utilizing over 100 custom adapters for major sites for article, author, date HTML classes.
2. YouTube Transcript Processing: When a YouTube video URL is detected, retrieve the complete video transcript including both manual captions and auto-generated subtitles, maintaining proper timestamp synchronization.
3. PDF to HTML: Extracts formatted text from PDF with parsing of linebreaks , page headers, footnotes, and section headings. Supports fonts, links, bold, italics, lists, headings, headers, footnotes, and Table of Contents, Quotes, and Code Blocks. Removes repeated headers, links footnote anchors to the footnote, and preserves number of the PDF page with invisible I element. This function uses pdfjs-serverless to work in more environments than PDF.js-based tools: Cloudflare workers, serverless, node.js, and front-end only.
4. Basic HTML Standardization: Transform complex HTML into a simplified reading-mode format of basic HTML, making it ideal for research note archival and focused reading, with headings, images and links.
5. Cite: Identify and extract citation metadata including author names, publication dates, sources, and titles using HTML meta tags and common class name patterns. The system validates author names against a comprehensive database of 90,000 first and last names, distinguishing between personal and organizational authors to properly format citations.

🖥️ Desktop App

QwkSearch Desktop — Built with Tauri. Select any text on screen, press ` to instantly search the web without leaving your workflow.

🧩 Browser Extension

QwkSearch Tab Manager AI — AI-powered tab manager that organizes, searches, and manages browser tabs intelligently.

🧠 VS Code Extension

QwkSearch for VS Code — Ask cited research questions from a sidebar in your editor, backed by the same QwkSearch API. Sign in with your account's API key, or use it signed-out as a guest.

📦 Packages

• chat-agent-toolkit — A multi-provider AI agent toolkit that generates language responses, searches the web, extracts content, and manages memory across 10+ LLM providers. It integrates the Vercel AI SDK, Mastra framework, and MCP protocol to orchestrate research agent workflows.

• domain-rank — Looks up top-ranked domains from the Tranco List and CommonCrawl backlink data to retrieve their human-readable source label, influence rank, and favicon. Useful for search/URL autocomplete, bookmark launchers, and domain reputation scoring.

• extract-pdf — Converts a PDF from a URL or ArrayBuffer into clean HTML with structural tagging including headings, lists, footnotes, and code blocks. Slim by default — PDF.js loads at runtime from the pdfjs-serverless CDN build — with optional OCR via IBM's granite-docling-258M model: run all pages through the frontend JS parser, all through Docling, hybrid (a regex scan flags pages with infographics/tables and OCRs only those), or point at the URL of another docling-compatible processor. Ships the Hono HTTP OCR service in its server/ folder. Works in Node.js, Cloudflare Workers, and browser environments.

• extract-webpage — Searches, extracts, cites, and outlines web content for a topic using an AI Research Agent. Combines PDF extraction, YouTube transcript extraction, DOM parsing, and LLM-based summarization to produce structured content from arbitrary web pages.

• extract-youtube — A fast, no-browser, serverless-optimized YouTube transcript extractor that fetches subtitles and captions without requiring a headless browser. Supports multiple output formats (SRT, WebVTT) and runs on edge/serverless platforms.

• html-renderer-api — A Cloudflare Worker that renders pages with Puppeteer via Browser Rendering. A Durable Object keeps the browser warm and persists cookies per session id, so pages behind a login or a Cloudflare challenge can be scraped, with Swagger UI and an OpenAPI spec on the worker itself.

• notebooklm-api-client — A Cloudflare Worker and on-demand Python container that drive Google's NotebookLM, which has no public API. One authenticated POST endpoint covers list/create/ask/summarize/delete, and a Puppeteer login flow captures the Google session for the container. The container sleeps after 5 minutes idle to conserve cost.

• qwksearch-api-client — An auto-generated TypeScript API client for the QwkSearch platform, built from an OpenAPI specification. Provides typed fetch-based bindings for interacting with the QwkSearch backend API.

• qwksearch-mcp-server — An MCP server that exposes web search, page extraction and JavaScript rendering as tools over stdio, so any MCP client — Claude Desktop, Claude Code, or your own — can search and read the web through QwkSearch.

• reason-editor — A formatted text editor built on Tiptap and Plate (React), published as react-reason-editor, with a toolbar, documents manager, and note outlines. Ships 59 editor extensions on individual subpath exports, 21 locales, remappable shortcuts, drag-and-drop, and collaborative editing via Yjs.

• reason-editor-sidebar — The REASON editor's file/folder tree, open-tabs panel, outline and split-view menu as a standalone package, published as react-reason-editor-sidebar. Includes the file-source layer for local, SSH, S3, R2, B2, Google Docs and Turso backends.

• render-url-to-html — A collection of URL-to-HTML rendering strategies using Cloudflare Browser Rendering, Puppeteer with stealth plugins, and JSDOM. Fetches URLs and returns fully-rendered DOM as HTML, capable of bypassing bot-detection on JavaScript-rendered pages.

• research-agent-ui — The chat research agent UI: conversation window, article reader, search config, file uploads, and chat history, along with the shadcn primitives and icons it depends on. Drops into a Next.js app behind a small config/injection surface for auth, branding, and media-search preferences.

• search-web-api — Provides access to 70+ search engines across 10 categories (web, academic, news, images, etc.) plus a scrape/extract API served via a Hono HTTP server. Includes Hugging Face Transformers integration for AI-powered processing.

• searxng-search-cloudflare — A deployment configuration for running a private SearXNG metasearch engine proxy in Docker. Aggregates results from multiple search engines without tracking the user, providing a privacy-respecting search backend.

• shadcn-app-dock — A prop-driven, macOS-style category dock React component with icon magnification on hover and a built-in shadcn theme switcher. Uses Framer Motion for animations and integrates with next-themes for light/dark mode toggling.

• shadcn-settings — A schema-driven settings form renderer built on shadcn/ui. Feed it plain-data field declarations and value/commit callbacks and it renders string, password, textarea, select and switch controls in card, inline or ghost layouts, with custom field types supplied as renderers.

• language-model-training — A from-scratch GPT-style transformer implementation built on Tinygrad that trains a next-word-prediction language model with a full Wikipedia pipeline. Ships with a FastAPI control API, Docker Compose orchestration, and a Next.js dashboard for monitoring training jobs.

• trending-news-api — A React trending-news widget backed by Wikipedia's daily pageviews joined to The News API through a bundled Cloudflare Worker proxy, with a 10-minute client cache.

• react-weather-forecast — A React weather forecast widget using Open-Meteo, published as use-weather-forecast. Resolves the viewer's location from explicit coordinates, a bundled Cloudflare geo Worker, or IP geolocation.

• use-voice-control — React voice control with speech transcription, vocalization and interruption (STT/TTS/VAD), plus a CLI that reads Markdown and text files aloud to audio files.

• user-help-docs — The user-facing help documentation served at /docs in the web app via Fumadocs. Content is inlined at build time so the docs run on Cloudflare Workers, where there is no filesystem to scan.

• write-language — A multi-provider language generation toolkit using the Vercel AI SDK that generates text responses via 10+ LLM providers including OpenAI, Anthropic, Google, Groq, and more. Provides a unified interface for streaming and non-streaming text generation.

🧠 Agent Skills

Every package has a matching Agent Skill — setup, the calls worth knowing, recipes, and a troubleshooting table, written from the source rather than the README. Install all of them, or just the one you need:

npx skills@latest add https://github.com/OpenSourceAGI/qwksearch-research-agent
npx skills@latest add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-search-web-api

See skills/README.md for the full index.

Further Research

• ThoughtSource Reasoning Datasets
• Awesome-LLMs-Datasets
• GPT Researcher
• NLP Papers Latest Updates
• Anthropic Persuasion Overview
• NLP Research Progress
• NLP Datasets
• Google Search Algorithm
• Can LLMs Generate Novel Research Ideas?
• Graph Algorithms Playground
• CommonCrawl C4 Download
• Knowledge Graphs Prompts Papers
• Paper - Iterative Research Idea Generation
• Open Deep Search
• LangChain Hub

🌟 Star this repo so it will grow new features!

LobeHub foundation (apps/qwk-in-lobe/)

apps/qwk-in-lobe/ holds a copy of the LobeHub monorepo adapted to run qwksearch.com on the same Cloudflare stack as this repo's Worker (Workers + D1 + KV + R2 + Email Routing + Better Auth). It ships all LobeHub features plus the QwkSearch article-extract side panel and D1-backed Docs. It is a separate pnpm workspace; see apps/qwk-in-lobe/README.md for build, deploy and the list of changes.