Search Engine Suite

Quick Jumps:

• Description
• Overview
• Client / Server Configurations
• Files
  • TL;DR Version
  • Full Description Version
• Configuring The System
• Building The System
• Using The System
• URL Parameters
• Notes on Operation
  • Preview Lines and Full File Viewing
  • Sliding Windows
  • Literal Substring Mode
  • Phrase Search
  • Proximity Indexes
  • Thresholds (TF-IDF vs. BM25)
• Personal Notes and Why This Exists
  • Not Everything was a Success - I blame sleepiness.
• Since We Can, Some Fun?
• Author

Description

TL;DR: A small C-based search engine using TF-IDF and BM25, with optional proximity search. Includes indexers, query daemons, terminal clients, and a web interface.

Search Engine Suite is a compact C-based search engine built around TF-IDF and BM25 scoring, with optional proximity search. It includes fast indexers, query daemons (secure and non-secure), terminal clients, and a lightweight Node.js web interface. The system is designed to be simple, transparent, and easy to understand - no external databases or heavyweight dependencies, while still supporting multiple search modes, pagination, sorting, and preview features.

Disclaimer: This is not a search engine that indexes the whole internet - it was never meant to be. But having said that, TF-IDF and especially BM25 are the classical scoring methods used by early search engines, and BM25 still serves as the first-stage retrieval method in modern systems.

(Above screenshots were taken using an older browser because I prefer the 3D look over the flat UI style of modern browsers. Here's a more modern browser)

Overview

A quick look at how the system fits together - indexers, daemons, clients, and the web interface,

This project provides a complete search engine stack implemented in C. It includes:

• Indexers for TF-IDF, BM25, and proximity scoring

• Query daemons (regular and OpenSSL-secured) that keep indexes in memory

• Terminal clients for querying daemons or running direct queries

• Node.js web API layers using exec, spawn, or daemon sockets

• Browser UI with multiple search modes, pagination, sorting, and preview options

• Scripts for starting/stopping daemons and generating local certificates

The design emphasizes speed and simplicity. Index files are plain, memory-mapped structures; daemons keep them loaded for fast queries; and the web interface remains lightweight and easy to inspect.

This system is a genuine "choose your own adventure": indexers, daemons, clients, and web layers can be mixed and matched to suit whatever workflow the user prefers.

Client / Server Configurations

This diagram shows how the daemons and web interface interact. The daemons keep index files in memory for fast queries, while the web interface can either call the query programs directly or communicate with the daemons over sockets (secure or non-secure).

This diagram shows how the daemons and terminal clients interact. The daemons keep index files in memory for fast queries, while the terminal clients communicate with the daemons over sockets (secure or non-secure).

Or, of course, for local only queries, the tfquery and bm25query programs can be used instead of daemons and clients.

Files

This repository contains several components - programs, scripts, build files, and configuration files. A compact summary section is next followed by a more detailed section.

TL;DR Version

This is the quick reference version of the file list. The full version appears in the next section.

Full Description Version

The detailed version of the file list, with expanded notes.

Indexers (TF-IDF and BM25)

Query Programs (TUI) (Direct query tools)

Query Daemons (Non-secure daemon processes)

Secure Query Daemons (OpenSSL) (TLS-enabled daemon processes)

Query Clients (JSON / Terminal) (Programs that talk to daemons)

Core Headers / Config (Shared definitions and configuration)

Web API (Node.js) (Interfaces between browser and query system)

Web UI (Browser interfaces)

Scripts (Certificates) (Local CA + key generation)

Scripts (Daemon Control) (Start/stop daemons in various modes)

Build Files (Makefiles and package metadata)

Proximity Versions (v2/) (Proximity indexers and related files)

Other (Stuff I didn't create)

Configuring The System

If you want to use the secure versions of the query daemons, you would need to edit config.h to specify the locations of the certificate files you will generate (or have generated). If you are only using this system on either Linux or Windows, you can ignore the OS you're not using. If using Windows, remember to escape the backslashes (i.e. double backslashes). Change this file before compiling the system (or rerun make/nmake after changing this file).

You would also need to set the correct path for OpenSSL in makecerts.bat if using Windows. Or if using Linux, make sure openssl is in your search path when using makecerts.

If you want to use the non-daemon version of the web interface, you would have to edit search.js and/or searchn.js to point to the location of the query programs. If using Windows, remember to escape the backslashes (i.e. double backslashes).

Building The System

If you want to use the web interface, you would need to install node.js if you don't already have it. If using Linux, it's probably available in your distro. Or it can be downloaded by going to: https://nodejs.org or github.com/nodejs/node. Once node is installed, you can run:

npm install

to install the Express.js dependency. Alternatively, you could manually install Express.js by running npm install express or npm install express@4.21.2 (for example) if you wanted to install express version 4.21.2.

If you want to use the Jansson JSON parser (and if you want to use queryct or querycts) you would need to install Jansson (if you don't already have it), by going to: https://github.com/akheron/jansson. Or alternatively, you could use YYJSON: https://github.com/ibireme/yyjson.

If you want to use the secure versions, you would need OpenSSL installed. If using Linux, it will be available in your distro. Or it can be downloaded by going to: www.openssl.org or github.com/openssl/openssl.

To build the executables, start up a terminal or Windows cmd shell and then change the directory to the appropriate source directory (cd src or cd src/v2 (for Windows: cd src\v2)).

You should then be able to build the system by either:

make -f Makefile.linux

if on Linux, or

nmake

if using Visual Studio on Windows (with the DOS environment set correctly).

This will make everything. The executables will be put in this suite's main directory (in the case of the v2 versions, they would be put in the v2 directory under the main directory). However, if you want to only build a specific thing, you could specify the desired program, for example: make -f Makefile.linux ../tfquery or nmake ..\tfquery.exe. If you don't have any of the required C code dependencies, you would either have to use this option or remove the uncompilable programs from the ALL variable in the makefile. For those who are more adventurous, you might want to create a configure script that would generate a Makefile specific to your system.

Also if you want to use the secure versions, you would need to create the certificate files needed (or use ones you might have made before). To do this, you can run:

makecerts localhost

or if you want to specify other hostnames that should be recognized with the certificates, you could run, for example:

makecerts localhost mycomputer myserver.mycompany.com

Note: If you are using Linux and your download didn't preserve executable permissions (common when transferring between filesystems), you would need to run chmod 755 makecerts and chmod 755 startd stopd.

If you want to use secure web connections (https:), you would need to add the generated localCA.pem file to your browser in order for your browser to recognize the local Certificate Authority that this script created. That local Certificate Authority would then verify your computer's certificate (that was generated with this script). To add localCA.pem to some browsers you would go to Tools -> Settings -> Privacy & Security -> View Certificates -> Import (make sure you are in the Authorities tab before importing). Your browser may be different. You can always remove this Certificate Authority ("Local Development CA") from your browser if you change your mind.

Using The System

Help on all the programs can be accessed by specifying -h on the commandline (example: tfquery -h). This also applies to the node.js programs (example: node search3.js -h).

The first thing that needs to be done is create the index files. You should create a file containing the filenames of every file you wish to index (full paths) one file per line. This file could be called whatever you like, but calling it filelist would be easiest. Since I wanted this search engine to index the source code I have written over the years, I modified my own recursive directory lister to create this filelist file for me by adding filters for the various types of source code files I wanted to index. Here's a snippet of my code:

char *s;
j=strlen(dirslot->d_name);
for (i=0;i<j;i++) { str2[i]=tolower(dirslot->d_name[i]); } str2[j]='\0';
s=str2+j;
if ((strcmp(s-2,".c")!=0)&&(strcmp(s-2,".h")!=0)&&(strcmp(s-4,".cpp")!=0)&&
	(strcmp(s-4,".pas")!=0)&&(strcmp(s-2,".a")!=0)&&(strcmp(s-3,".js")!=0)&&
	(strcmp(s-5,".html")!=0)&&(strcmp(s-4,".htm")!=0)&&(strcmp(s-4,".tex")!=0)
	&&(strcmp(s-5,".java")!=0)&&(strcmp(s-3,".py")!=0)&&(strcmp(s-4,".asm")!=0)
	&&(strcmp(s-4,".vhd")!=0)&&(strcmp(s-5,".vhdl")!=0)&&(strcmp(s-2,".v")!=0)
	&&(strcmp(s-4,".css")!=0)&&(strcmp(s-2,".f")!=0)&&(strcmp(s-4,".for")!=0))
	{
	if (!allsource) { return; }
	if ((strcmp(s-5,".glsl")!=0)&&(strcmp(s-5,".frag")!=0)&&
		(strcmp(s-4,".mak")!=0)&&(strcmp(s-4,".bat")!=0)&&
		(strcmp(s-4,".prg")!=0)&&(strcmp(s-4,".cxx")!=0)&&
		(strcmp(s-4,".xml")!=0)&&(strcmp(s-4,".inc")!=0)&&
		(strcmp(s-4,".pov")!=0)&&(strcmp(s-4,".acd")!=0)&&
		(strcmp(s-4,".hpp")!=0)&&(strcmp(s-4,".bas")!=0)&&
		(strcmp(s-3,".rc")!=0)&&(strcmp(s-3,".sh")!=0)&&
		(strcmp(s-3,".cc")!=0)&&(strcmp(s-3,".cs")!=0)&&
		(strcmp(s-3,".vb")!=0)&&(strcmp(s-2,".p")!=0)&&
		(strcmp(s-3,".ts")!=0)&&(strcmp(s-3,".go")!=0)&&
		(strcmp(s-3,".rs")!=0)&&(strcmp(s-3,".hs")!=0)&&
		(strcmp(s-4,".ini")!=0)&&(strcmp(s-4,".cfg")!=0)&&
		(strcmp(s-5,".json")!=0)&&(strcmp(s-5,".yaml")!=0))
		{
		if (allsource<2) { return; }
		if ((strcmp(s-3,".md")!=0)&&(strcmp(s-4,".txt")!=0)) { return; }
		}
	}
// survived the filter, write out the filename

There probably are more efficient ways to do it, but I just needed something I could use right away - no need for it to be super efficient. It also allowed for optional more inclusive filtering (via allsource flag setting the inclusion level).

Once you have the filelist, run the indexers:

tfindex -v
bm25index -v

if you want to see the progress (verbose output), or:

tfindex
bm25index

if you don't. Optionally run the v2 versions.

Once the indexers have indexed your files, you can test things out by running:

tfquery {whatever you want to search for}

For example: tfquery createwindow or tfquery "simple class" (Unless using the case-sensitive version of literal substring search, the query is case-insensitive. Also, multiple word searches need to be in quotes.)

Tip: To ensure the search term appears in the preview lines, use the -S option.

To see things the best way possible, you will need a terminal that understands ANSI escape codes (to set the color). If you don't have an ANSI code capable terminal, use the -C option to eliminate coloring. If you use a light-mode terminal you probably would want to use the -l option to switch to the light mode specific colors. I don't use a light-mode terminal, so I can't be sure those are the best colors, so you may need to modify the source code to suit your likings.

If you want to use the daemon versions, you likely need to decide if you want to use secure or non-secure sockets. If this system is only going to be used by you and only on your computer (not remotely), there probably wouldn't be any reason to use secure sockets.

The easiest option is to use the "new socket per request" non-secure versions of the daemons, in which case you would run:

startd

Note: If you use a firewall, you will likely need to approve these programs in your firewall. Depending on your OS, you may be prompted to approve of their use when first run.

Once the daemons are running, you can try them out by running:

queryct {whatever you want to search for}

If you get a json error, there's a pretty good chance the file has Unicode characters, in which case you should try using the -u option.

If you want to use the web interface, run the node server:

node search3.js

If you are using Linux, you might want to use node search3.js & instead.

Then start up your browser either normally or by running (for example):

firefox localhost:3000/search.html

(optionally with & if on Linux). If normally, then type localhost:3000\search.html in the search box of the browser.

The page's search box will automatically have the focus. You can press Enter after typing your search string rather than pressing the search button. The search button will "spin" the clock hands to show it's performing the search. If all goes well, the results will be shown and the search button will return to the magnifying glass.

Click in any empty area of the page (not the control bar input boxes) and press h to get a help screen for key controls.

The control bar will show new input boxes and/or buttons automatically as they are needed and remove them when not needed anymore.

If you would prefer not to have to type :3000 in the URL, you could use the standard http: and https: ports. If using Linux, in order to use the standard ports you would need to use sudo. For example, sudo node search3.js -p80 for http: and sudo node search3.js -p443 -s for https:. Windows has no problem doing it (though I don't know if you would have to run as Administrator since my Windows account is Administrator).

When finished, you can terminate the node server (if you ran it), and run:

stopd

to terminate the daemons.

If you want to incrementally update an index, you can optionally create any combination of filelist.delete, filelist.change, and filelist.add which contain filenames (full paths, one file per line) of files that were deleted, changed, and added, respectively. Then run tfincr and bm25incr to update the indexes accordingly. The filenames of the new index files will have .new added to them. Since the full indexers run fast enough at this point, there's really no reason to use the incremental indexers.

If you ever need to know what files were indexed in any index file, you can run tfquery -F (or tfquery -F -i{filename} if you named the index file something other than filelist). Same with bm25query.

If you want to use the no jQuery version of the search engine web page, you would need to rename search.html to something else and rename search-no-jquery.html to search.html since the search engine node servers will be serving search.html.

URL Parameters

These optional parameters allow you to customize search behavior, pagination, sorting, preview size, font mode, proximity distance, and more.

Use these like this, for example (If specifying this on a terminal commandline, you probably will need to escape the special characters, depending on your OS (for example: search.html?pagin=2\&q=createwindow\&page=4 if using Linux, "search.html?pagin=2&q=createwindow&page=4" if using Windows)):

search.html?tabs=2&q=mark%20craig&search=true

• q={string} - Search for the specified ({string}) keyword(s). Spaces must be replaced with %20 and some other characters must be replaced with the %hex value.
• keepontop=false - Don't always keep the control bar at top of screen
• togglethresh=true - Show score threshold input boxes on control bar
• autoadjustsearchbox=false - Don't automatically reduce search input box when other items appear
• searchboxsize={number} - Specify maximum size of the search input box (default = 39)
• mode={number} - Specify the search mode (1 = OR, 2 = AND, 3 = Phrase, 4 = Literal substring, 5 = Literal substring (case sensitive), 6 = Proximity (if available))
• pagin={number} - Specify pagination (1 = All Results (no pagination), 2 = 10 Results per page, 3 = 20 Results per page, 4 = 50 Results per page)
• page={number} - Get the specified result page (pagination must be enabled, q string must already be specified, and page= must be last)
• sort={number} - Specify result sorting (1 = Sort by score, 2 = Sort by filename, then by score, 3 = Sort by filename, then by line #)
• algor={number} - Specify search algorithm (1 = TF-IDF, 2 = BM25)
• preview={number} - Specify number of preview lines (0 = no file preview)
• proxdist={number} - Specify proximity distance
• tthresh={number} - Specify TF-IDF score threshold (floating point number)
• bthresh={number} - Specify BM25 score threshold (floating point number)
• tabs={number} - Specify tab stop size
• forcemono={number} - If 1 then force monospace font for all preview lines and file listing
• forceprop={number} - If 1 then force proportional space font for all preview lines and file listing
• uniconvert={number} - If 1 then convert unicode preview lines to utf8
• host={string} - Specify remote host URL to use (i.e. not http://localhost:3000). Should be unnecessary since it will get the remote host from the URL.
• search=true - Run the search (if specified, it must appear last in the parameter list) (Don't use this if page number was specified)

Notes on Operation

Preview Lines and Full File Viewing

The query system doesn't need the files that were indexed to find what you're looking for (since each file was broken up into snippets which then were tokenized, storing the IDs of the tokens that each snippet had), BUT the query system does need the indexed files for the preview lines that are optionally shown. Because of this, if any indexed file no longer exists or was changed, the preview lines for that file either won't be available or might not be correct (respectively).

When using the web interface, you can click on the filename to open the file in a new tab (hover over filename for further instructions). This feature also requires access to the files that were indexed. So similar caveat - the file might not be available any longer or might not match the preview lines.

Sliding Windows

The system indexes the files using overlapping sliding windows. By default, the snippet window size is 30 lines and the stride is 10 lines (therefore overlap is 20 lines). Be aware that a line number reported in the search results is the line number of the start of the window, not necessarily the line number of the occurrence of the searched item - but the searched item will be within the 30 line window (assuming you indexed using a 30 line window (it can be specified on the commandline)). Indexing with a different window size or stride will affect the size of the index file and the quality of the results, so it's a delicate balance (30/10 is common in code search and early information retrieval research).

Literal Substring Mode

The Literal substring search mode is a last resort to be used if (for example) you don't remember the exact word for what you're searching for. Since it likely is a substring, it likely won't be indexed (since indexing uses complete "words" (defined as any continuous string of letters, numbers, or "_")). With Literal substring search, the index is only used for a list of files - and each file is read to see if the substring appears in the file. So, of course, this also requires the indexed files to be available.

Phrase Search

Phrase match (non-v2 versions) uses the index only to identify candidate snippets, the actual phrase check is performed by scanning the file thus the indexed file must be available - if it's not available, it would be best to switch the search mode to AND.

Proximity Indexes

The v2 (proximity) versions have the token locations preserved in the index, so the indexed files aren't needed for the search since it can be determined from the index whether the words appear together in the specified order (but you still need the indexed files for the preview lines). Because of the preserved token locations, a v2 version index file will be larger and take longer to load. Be aware that the compiler flag FASTER_READ can be set - if it is set, the index file will be even larger, but will load a little faster. Decide for yourself what your situation can withstand.

Thresholds (TF-IDF vs. BM25)

TF-IDF is more stable for short, structured documents like source code, while BM25 is optimized for longer natural-language text and tends to produce more matches unless thresholds are raised. In my system, the default BM25 score threshold is perhaps too low (it can be set both in the query programs and in the web page). However, for very short documents with few words, BM25 can produce scores lower than the default threshold of 0.05 and even negative scores for snippets that actually do contain the search term. Adjust the thresholds to your liking if desired.

Personal Notes and Why This Exists

Its raison d'être (my two years of French class not wasted 😎 ):

I like to reuse code I've already written. But with so many projects over the many years, it's sometimes hard to remember which project I did a given thing in or even what the parameters of a rarely used Win32 API function are. I needed something that could easily find what I needed without the slowness of a recursive grep (especially since my projects can span many different locations in the filesystems and even many different computers/OSes) or the increasing difficulty in getting what you want from search engines on the internet.

Concerning the choice of language:

I don't know about you, but I'm sick of seeing open-source software that is written in Python, Rust, Go, or even C++26 (or whatever number they come up with after that). My only concession is node.js. But node.js is optional - it's only needed if you want to use the browser version - and its only dependency is express.js. I've written this system to compile on pretty much the oldest C compilers you might have (such as Visual Studio 6) and even the node.js part is written to run on old versions (tested on as low as 8.17.0) - which is why the package.json file contains a much lower express.js version than the current one. As an aside, I've reluctantly written code in Python, so I do know how somewhat, I just don't like to.

Why most everything is in one directory:

This project has many optional components (choose your own adventure), and users may come from very different experience levels. Keeping scripts, executables, HTML files, JavaScript (node.js) files, and generated indexes and certificates in a single directory avoids path issues and makes the system easier to use without requiring environment variables or custom search paths. Advanced users can reorganize things if they want, but the default layout favors simplicity and reliability.

Although the system was designed for indexing source code, it works just as well for any structured text, for example recipes. Someone might want to search for "oregano" to find that forgotten meal they liked that had oregano in it, or "leftover chicken" to figure out what to do with their leftover chicken - plain ASCII recipe files can be indexed the same way as code. For those adventurous enough, PDFs can also be indexed if the PROCESS_PDF functionality is enabled (described in the next section).

Not Everything was a Success - I blame sleepiness.

I came up with an idea to modify the indexer to run ps2ascii (pdftotext and pstotext also work) on any PDF file that is in the filelist of files to index and then process the resulting temporary text file. The indexing took considerably more time to index. Then I ran tfquery on it at which point I saw the disappointing result - I forgot the indexer doesn't store actual text from the file (in this case the temporary ascii text file) and the preview lines are extracted from the actual file (therefore mostly binary garbage). I blame this failure (of remembering how it works) on the fact that I came up with the idea while waking up from a nap and then was so blinded by how great this feature would be to have. By the way, when I looked at the vocab list I was surprised to see the word dementia. I thought "I don't have any PDF about dementia, what would have that in it?". I was even more surprised when I found out the pdf that had dementia in it was MY OWN MASTER'S THESIS. Running ps2ascii manually showed that it was a bibliography entry in my thesis:

[10] A. K. Dewdney. A program for rotating hypercubes induces four-dimensional dementia. Scientific American, 254(4):14 - 23, April 1986.

I think this shows how useful this feature would have been - I wrote that document and I didn't remember dementia was in it. I then modified tfquery to run ps2ascii on any PDF in the results hoping it would run better than I expected it to - it didn't (the preview line output was fine, but it took too long to do it). By the way, many of my PDF files are owner's manuals for various electronic equipment I have. Often manufacturers don't give the pdf files meaningful filenames, so it sometimes can be difficult to find a desired manual in a directory full of PDFs. That's another reason this feature would have been useful. Maybe it's better this way, because as I was coding it, I didn't know if I would even be able to get Windows versions of ps2ascii, etc.

If you feel masochistic, ahem, adventurous enough you can compile tfindex, bm25index, tfquery, and bm25query with the PROCESS_PDF flag and try it for yourself. It could be made more efficient by caching the text produced by the PDF converters rather than deleting it right away (during query) or even permanently keeping them during indexing.

Since We Can, Some Fun?

It occurred to me that since the index contains every word from every document and stores how often those words appear in snippets, a word cloud could be created for either a specific document or the whole vocabulary. At first I figured I would create a simple HTML/Javascript 2d-canvas and draw the word cloud myself. After looking into word clouds, I decided it would be better just to use one from the internet. I then modified the daemons and client programs to optionally output the vocabulary and weights.

If you want to use this feature, you can download wordcloud2.js from: https://github.com/timdream/wordcloud2.js. Then uncomment the <script src="wordcloud2.js"></script> line in search.html and/or search-no-jquery.html. Then you can click on the result number in the search results (hover over for instructions) to get a word cloud of that document. Alternatively, press Ctrl while clicking on the result number to get a word cloud of the whole index (but be careful, a word cloud of the entire index can be resource intensive). Adjust the options variable to suit your needs (refer to the API documentation available at the github page for wordcloud2.js).

Note: Other word cloud generators can be used, but you'll have to code the interface between them and my web page yourself.

Here are some examples of what this feature produced:

But there is a catch: Because of the way this system indexes files (and how TF-IDF and BM25 work), this is not necessarily a true representation of the exact number of occurrences of each word. The weights used for the word clouds come from the snippet-level term frequencies that TF-IDF/BM25 rely on, not raw term frequency across the whole file. Raw term frequency is computed during indexing, but only inside the sliding windows, and it is not stored in the index file because the search algorithms don't need it. Storing raw TF in the index file would make the index file larger and slower to read - not worth it for a feature this trivial.

Also, the weights used for the word clouds are based on occurrences in the snippets, and keep in mind the snippets use overlapping sliding windows. Having said that, if you were to do the discouraged step of indexing with a stride equal to the window size and use BM25, the weights would be equal to the raw term frequencies for the given file (since there would no longer be overlapping windows).

Note: Clicking on the result number while in TF-IDF mode will result in a different word cloud than clicking on the result number of the same filename while in BM25 mode (different beyond the randomness provided by wordcloud2.js that is).

It should also be noted that full index word clouds may not result in useful results because of the likelihood that there will be many terms at the low frequency end and many terms at the high frequency end (if you're adventurous enough, you may want to edit the query daemons to filter out unwanted noise (such as perhaps void, int, char, float, include, if, while, for, etc.)).

So basically this feature is an interesting curiosity and your results may vary depending on what type of files you're indexing (ones where TF-IDF provide better results or ones where BM25 provide better results, etc.).

Author

Mark Craig https://www.youtube.com/MrMcSoftware