A self‑hosted search engine for documents
See the code
A self-hosted search engine to find stories in any files.
| Status | |
|---|---|
| Download |  |
| CI checks |  |
| Translations |  |
| Latest version |  |
| Release date |  |
| Open issues |  |
| Documentation |   |
Datashare is an open‑source, self‑hosted document search and analysis platform built by the International Consortium of Investigative Journalists (ICIJ). It ingests heterogeneous data (PDFs, emails, spreadsheets, images, archives, etc.), extracts text (including via OCR), enriches it with metadata and named entities, and exposes everything through a powerful search UI and REST API. Because Datashare runs on your own machines, you keep full control over sensitive material—no external cloud services required.
📣 Help us improve Datashare! What do you think about the new design of Datashare? Tell us your feedback through our survey, it will directly influences our roadmap, and lets you opt‑in for early previews/beta testing.
This section explains how to set up a development environment, build the project, run tests, and manage database migrations. It assumes you are comfortable with Java/Maven projects and basic service orchestration.
make help to see available targets)datashare (dev) and test (tests)A docker-compose.yml is provided to start all required services:
docker compose up -d
Datashare uses a properties file to configure service URIs for local development and testing. This allows you to run tests against services running on different hosts (e.g., in Docker containers or on localhost). To bootstrap the devenv configuration file, simply run:
make devenv
This creates a gitignored datashare-devenv.properties from the template file. The default configuration expects services to be available at these URIs:
| Service | Property | Default URI |
|---|---|---|
| AMQP | amqpUri | amqp://guest:guest@localhost:5673 |
| Elasticsearch | elasticsearchUri | http://localhost:9200 |
| PostgreSQL | postgresUri | jdbc:postgresql://localhost/dstest?user=dstest&password=test |
| Redis | redisUri | redis://localhost:6379 |
| S3 Mock | s3mockUri | http://localhost:9090 |
The properties file is loaded automatically when running tests via the -Ddevenv.file system property.
After cloning the repo, consider configuring git in the following way, to ignore commits that are actively noted by developers as to be ignored for git blame
git config blame.ignoreRevsFile .git-blame-ignore-revs
The project is modular. Using Make:
# Build and install all modules (runs migrations first, then jOOQ codegen)
make install
# Or build distribution JARs only
make build
The install and build targets automatically run database migrations before building, ensuring jOOQ sources are generated from the current schema.
Elasticsearch setup:
Datashare can automatically download and install Elasticsearch locally:
# Download and install Elasticsearch (done automatically during 'make dist')
make elasticsearch
The distribution package (make dist) automatically downloads Elasticsearch 8.x to ~/.local/share/datashare/elasticsearch/ if not already present. The installation script detects your platform (Linux/macOS, x86_64/aarch64) and downloads the appropriate version.
Datashare has both unit and integration tests. Integration tests expect Postgres, Elasticsearch, and Redis to be reachable.
# Run the whole test suite
make test
# Or run a single module
mvn -pl datashare-api test
# Or a single test class
mvn -pl datashare-api -Dtest=org.icij.datashare.PropertiesProviderTest test
Datashare uses Liquibase for schema migrations and jOOQ for type-safe SQL.
# Apply pending migrations
make migrate
# Regenerate jOOQ sources from DB schema
make generate
# Reset DB and reapply all migrations (DESTRUCTIVE)
make reset-db
Adding a new changeset:
datashare-db/src/main/resources/liquibase/changelog/changes/datashare-db/src/main/resources/liquibase/changelog/db.changelog.ymlmake migrate locally to verifyThe web UI is built with Vue 3 and maintained in a separate repository. When building the backend, you must also build the client and copy its compiled files into the ./app directory. The backend bundles these static assets using FluentHTTP, which serves resources from ./app (relative to the repo root). If this folder is missing or empty, only the API will be available, no UI.
The easiest way to get the frontend is to download a pre-built release:
make app
# Or a specific version of the front
make app VERSION=20.8.1
This downloads the frontend release matching the backend VERSION (from pom.xml) and extracts it to the app/ directory. If the matching version doesn't exist, it falls back to the latest release.
Datashare can also be developed using a VS Code Devcontainer, which provides a reproducible development environment with all required dependencies (JDK, Maven, PostgreSQL, Elasticsearch, Redis, etc.) running in Docker.
This approach helps avoid local environment inconsistencies and ensures a setup closer to CI and production-like conditions.
ms-vscode-remote.remote-containers)VS Code will build the container image and start the development environment.
This step may take several minutes on first run.
Once VS Code is connected to the devcontainer:
Run all commands from the VS Code integrated terminal, inside the container.
Initialize the project:
make install # Build all modules (runs migrations + jOOQ codegen)
make test # Run tests
Reload Java projects so that VS Code correctly picks up generated sources (jOOQ) and dependencies:
To publish Datashare's Temporal test container run
make build-temporal-testcontainer NEW_VERSION=<your-version>
For practical command-line examples covering CLI mode, server mode, OAuth setup, and more, see the Cookbook.
Datashare is distributed under the GNU Affero General Public License v3.0.
The International Consortium of Investigative Journalists (ICIJ) is a global network of reporters and media organizations collaborating on cross‑border investigations (e.g., Panama Papers, Luanda Leaks, Uber Files, Pandora Papers). The tech team at ICIJ builds tools like Datashare to empower investigative journalism at scale, handling millions of documents securely and efficiently. We open‑sourced Datashare to empower solo reporters and small newsrooms with advanced investigative tools, enable larger organizations to audit, extend, and self‑host the platform, and foster collaboration within the investigative community to continually improve the software.
Contact & Community
[email protected]Overall
Maintained
Code review
Latest release
Releases this year
863 followers · starred Mar 2018
1,025 followers · starred Mar 2023
209 followers · starred May 2024
181 followers · starred Oct 2021
A self‑hosted search engine for documents
See the code
A self-hosted search engine to find stories in any files.
| Status | |
|---|---|
| Download | |
| CI checks | |
| Translations | |
| Latest version | |
| Release date | |
| Open issues | |
| Documentation | |
Datashare is an open‑source, self‑hosted document search and analysis platform built by the International Consortium of Investigative Journalists (ICIJ). It ingests heterogeneous data (PDFs, emails, spreadsheets, images, archives, etc.), extracts text (including via OCR), enriches it with metadata and named entities, and exposes everything through a powerful search UI and REST API. Because Datashare runs on your own machines, you keep full control over sensitive material—no external cloud services required.
📣 Help us improve Datashare! What do you think about the new design of Datashare? Tell us your feedback through our survey, it will directly influences our roadmap, and lets you opt‑in for early previews/beta testing.
This section explains how to set up a development environment, build the project, run tests, and manage database migrations. It assumes you are comfortable with Java/Maven projects and basic service orchestration.
make help to see available targets)datashare (dev) and test (tests)A docker-compose.yml is provided to start all required services:
docker compose up -d
Datashare uses a properties file to configure service URIs for local development and testing. This allows you to run tests against services running on different hosts (e.g., in Docker containers or on localhost). To bootstrap the devenv configuration file, simply run:
make devenv
This creates a gitignored datashare-devenv.properties from the template file. The default configuration expects services to be available at these URIs:
| Service | Property | Default URI |
|---|---|---|
| AMQP | amqpUri | amqp://guest:guest@localhost:5673 |
| Elasticsearch | elasticsearchUri | http://localhost:9200 |
| PostgreSQL | postgresUri | jdbc:postgresql://localhost/dstest?user=dstest&password=test |
| Redis | redisUri | redis://localhost:6379 |
| S3 Mock | s3mockUri | http://localhost:9090 |
The properties file is loaded automatically when running tests via the -Ddevenv.file system property.
After cloning the repo, consider configuring git in the following way, to ignore commits that are actively noted by developers as to be ignored for git blame
git config blame.ignoreRevsFile .git-blame-ignore-revs
The project is modular. Using Make:
# Build and install all modules (runs migrations first, then jOOQ codegen)
make install
# Or build distribution JARs only
make build
The install and build targets automatically run database migrations before building, ensuring jOOQ sources are generated from the current schema.
Elasticsearch setup:
Datashare can automatically download and install Elasticsearch locally:
# Download and install Elasticsearch (done automatically during 'make dist')
make elasticsearch
The distribution package (make dist) automatically downloads Elasticsearch 8.x to ~/.local/share/datashare/elasticsearch/ if not already present. The installation script detects your platform (Linux/macOS, x86_64/aarch64) and downloads the appropriate version.
Datashare has both unit and integration tests. Integration tests expect Postgres, Elasticsearch, and Redis to be reachable.
# Run the whole test suite
make test
# Or run a single module
mvn -pl datashare-api test
# Or a single test class
mvn -pl datashare-api -Dtest=org.icij.datashare.PropertiesProviderTest test
Datashare uses Liquibase for schema migrations and jOOQ for type-safe SQL.
# Apply pending migrations
make migrate
# Regenerate jOOQ sources from DB schema
make generate
# Reset DB and reapply all migrations (DESTRUCTIVE)
make reset-db
Adding a new changeset:
datashare-db/src/main/resources/liquibase/changelog/changes/datashare-db/src/main/resources/liquibase/changelog/db.changelog.ymlmake migrate locally to verifyThe web UI is built with Vue 3 and maintained in a separate repository. When building the backend, you must also build the client and copy its compiled files into the ./app directory. The backend bundles these static assets using FluentHTTP, which serves resources from ./app (relative to the repo root). If this folder is missing or empty, only the API will be available, no UI.
The easiest way to get the frontend is to download a pre-built release:
make app
# Or a specific version of the front
make app VERSION=20.8.1
This downloads the frontend release matching the backend VERSION (from pom.xml) and extracts it to the app/ directory. If the matching version doesn't exist, it falls back to the latest release.
Datashare can also be developed using a VS Code Devcontainer, which provides a reproducible development environment with all required dependencies (JDK, Maven, PostgreSQL, Elasticsearch, Redis, etc.) running in Docker.
This approach helps avoid local environment inconsistencies and ensures a setup closer to CI and production-like conditions.
ms-vscode-remote.remote-containers)VS Code will build the container image and start the development environment.
This step may take several minutes on first run.
Once VS Code is connected to the devcontainer:
Run all commands from the VS Code integrated terminal, inside the container.
Initialize the project:
make install # Build all modules (runs migrations + jOOQ codegen)
make test # Run tests
Reload Java projects so that VS Code correctly picks up generated sources (jOOQ) and dependencies:
To publish Datashare's Temporal test container run
make build-temporal-testcontainer NEW_VERSION=<your-version>
For practical command-line examples covering CLI mode, server mode, OAuth setup, and more, see the Cookbook.
Datashare is distributed under the GNU Affero General Public License v3.0.
The International Consortium of Investigative Journalists (ICIJ) is a global network of reporters and media organizations collaborating on cross‑border investigations (e.g., Panama Papers, Luanda Leaks, Uber Files, Pandora Papers). The tech team at ICIJ builds tools like Datashare to empower investigative journalism at scale, handling millions of documents securely and efficiently. We open‑sourced Datashare to empower solo reporters and small newsrooms with advanced investigative tools, enable larger organizations to audit, extend, and self‑host the platform, and foster collaboration within the investigative community to continually improve the software.
Contact & Community
[email protected]Overall
Maintained
Code review
Latest release
Releases this year
863 followers · starred Mar 2018
1,025 followers · starred Mar 2023
209 followers · starred May 2024
181 followers · starred Oct 2021