Karapace - Your Apache Kafka® essentials in one tool
See the codeKarapace
========
``karapace``. Your Apache Kafka® essentials in one tool.
An `open-source <https://github.com/Aiven-Open/karapace/blob/master/LICENSE>`_ implementation
of `Kafka REST <https://docs.confluent.io/platform/current/kafka-rest/index.html#features>`_ and
`Schema Registry <https://docs.confluent.io/platform/current/schema-registry/index.html>`_.
|Tests| |Contributor Covenant|
.. |Tests| image:: https://github.com/Aiven-Open/karapace/actions/workflows/tests.yml/badge.svg?branch=main
:target: https://github.com/Aiven-Open/karapace/actions/workflows/tests.yml?query=branch%3Amain
.. |Contributor Covenant| image:: https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg
:target: CODE_OF_CONDUCT.md
Overview
========
Karapace supports the storing of schemas in a central repository, which clients can access to
serialize and deserialize messages. The schemas also maintain their own version histories and can be
checked for compatibility between their different respective versions.
Karapace rest provides a RESTful interface to your Apache Kafka cluster, allowing you to perform tasks such
as producing and consuming messages and perform administrative cluster work, all the while using the
language of the WEB.
Features
========
* Drop-in replacement both on pre-existing Schema Registry / Kafka Rest Proxy client and
server-sides
* Moderate memory consumption
* Asynchronous architecture based on aiohttp
* Supports Avro, JSON Schema, and Protobuf
* Leader/Replica architecture for HA and load balancing
* Observability with Metrics and OpenTelemetry
* Based on aiokafka (rdkafka) kafka library mostly
* Schema registry based on FastApi
Compatibility details
---------------------
Karapace is compatible with Schema Registry 6.1.1 on API level. When a new version of SR is released, the goal is
to support it in a reasonable time. Karapace supports all operations in the API.
There are some caveats regarding the schema normalization, and the error messages being the same as in Schema Registry, which
cannot be always fully guaranteed.
Setup
=====
Using Docker
------------
To get you up and running with the latest build of Karapace, a docker image is available::
# Fetch the latest build from main branch
docker pull ghcr.io/aiven-open/karapace:develop
# Fetch the latest release
docker pull ghcr.io/aiven-open/karapace:latest
Versions `3.7.1` and earlier are available from the `ghcr.io/aiven` registry::
docker pull ghcr.io/aiven/karapace:3.7.1
An example setup including configuration and Kafka connection is available as compose example::
docker compose -f ./container/compose.yml up -d
Then you should be able to reach two sets of endpoints:
* Karapace schema registry on http://localhost:8081
* Karapace REST on http://localhost:8082
For local development with Docker, use the convenient make commands (see `Development`_ section below).
Configuration
^^^^^^^^^^^^^
Each configuration key can be overridden with an environment variable prefixed with ``KARAPACE_``,
exception being configuration keys that actually start with the ``karapace`` string. For example, to
override the ``bootstrap_uri`` config value, one would use the environment variable
``KARAPACE_BOOTSTRAP_URI``. Here_ you can find an example configuration file to give you an idea
what you need to change.
.. _`Here`: https://github.com/Aiven-Open/karapace/blob/master/karapace.config.json
Using Sources
-------------
Install
^^^^^^^
You can do a source install using::
pip install .
Troubleshooting notes :
- An updated version of wheel (https://pypi.org/project/wheel/) is required.
- An updated version of 'go' and 'rust' is required
- Create and activate virtual environment (venv) to manage dependencies
Run
^^^
- Make sure kafka is running.
Start Karapace. This should start karapace on http://localhost:8081 ::
python -m karapace
Verify in browser http://localhost:8081/subjects should return an array of subjects if exist or an empty array.
or with curl ::
$ curl -X GET http://localhost:8081/subjects
To enable oidc authentication on the karapace, configure oidc jwks url config details ::
sasl_oauthbearer_authentication_enabled: bool = True
sasl_oauthbearer_jwks_endpoint_url = "https://idp.example.com/realms/karapace/protocol/openid-connect/certs",
sasl_oauthbearer_expected_issuer = "https://idp.example.com/realms/karapace",
sasl_oauthbearer_expected_audience = "karapace-audience",
sasl_oauthbearer_sub_claim_name = "sub",
When ``sasl_oauthbearer_authentication_enabled`` is true, every request must carry a valid
Bearer token. The JWT signature, expiry and the configured subject claim are always verified
via JWKS. ``sasl_oauthbearer_expected_issuer`` and ``sasl_oauthbearer_expected_audience`` are
optional, as in Kafka's ``sasl.oauthbearer.expected.*``: leaving one unset skips that claim
check. Leaving the audience unset means any token the provider issues for any client is
accepted, so set it whenever the provider can mint a usable ``aud``. Authorization (role-based
access) is opt-in on top of this with ``sasl_oauthbearer_authorization_enabled``.
Optional hardening flags::
sasl_oauthbearer_leeway_seconds: int = 0 # clock-skew tolerance for exp/nbf/iat (must be >= 0)
sasl_oauthbearer_require_access_token_typ: bool = False # require typ "at+jwt" header
There is a detailed section about OAuth2 authentication for karapace below.
To enable oidc authorization on karapace, configure the below params together with the above ::
sasl_oauthbearer_authorization_enabled: bool = False
sasl_oauthbearer_roles_claim_path: str | None = None
# Defaults to the karapace.* roles shown in the example below; override per deployment.
sasl_oauthbearer_method_roles: dict[str, list[str]]
Note: ``sasl_oauthbearer_authorization_enabled=true`` requires
``sasl_oauthbearer_authentication_enabled=true``. For backwards compatibility, enabling
authorization without explicitly enabling authentication will auto-enable authentication and
log a deprecation warning — set both flags explicitly.
There is a detailed section about OAuth2 authorization for karapace below.
Start Karapace rest proxy. This shout start karapace on http://localhost:8082 ::
python -m karapace.kafka_rest_apis
To enable authorization & authentication on the rest proxy, configure 'sasl_mechanism' in the config with values like PLAIN/OAUTHBEARER ::
sasl_mechanism = "OAUTHBEARER",
sasl_oauth_token_provider = token_provider,
security_protocol="SASL_SSL",
ssl_cafile="ca.pem",
If 'sasl_mechanism' is configured to PLAIN::
sasl_mechanism = "PLAIN",
security_protocol = "SASL_PLAINTEXT",
sasl_plain_username = "your_username",
sasl_plain_password = "your_password"
There is a detailed section about OAuth2 authentication for rest proxy below.
Verify with list topics::
$ curl "http://localhost:8082/topics"
Schema Registry Api reference
=============================
To register the first version of a schema under the subject "test" using Avro schema::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"}' \
http://localhost:8081/subjects/test-key/versions
{"id":1}
To register a version of a schema using JSON Schema, one needs to use `schemaType` property::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schemaType": "JSON", "schema": "{\"type\": \"object\",\"properties\":{\"age\":{\"type\": \"number\"}},\"additionalProperties\":true}"}' \
http://localhost:8081/subjects/test-key-json-schema/versions
{"id":2}
To list all subjects (including the one created just above)::
$ curl -X GET http://localhost:8081/subjects
["test-key"]
To list all the versions of a given schema (including the one just created above)::
$ curl -X GET http://localhost:8081/subjects/test-key/versions
[1]
To fetch back the schema whose global id is 1 (i.e. the one registered above)::
$ curl -X GET http://localhost:8081/schemas/ids/1
{"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}
To get the specific version 1 of the schema just registered run::
$ curl -X GET http://localhost:8081/subjects/test-key/versions/1
{"subject":"test-key","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}
To get the latest version of the schema under subject test-key run::
$ curl -X GET http://localhost:8081/subjects/test-key/versions/latest
{"subject":"test-key","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}
In order to delete version 10 of the schema registered under subject "test-key" (if it exists)::
$ curl -X DELETE http://localhost:8081/subjects/test-key/versions/10
10
To Delete all versions of the schema registered under subject "test-key"::
$ curl -X DELETE http://localhost:8081/subjects/test-key
[1]
Test the compatibility of a schema with the latest schema under subject "test-key"::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"int\"}"}' \
http://localhost:8081/compatibility/subjects/test-key/versions/latest
{"is_compatible":true}
NOTE: if the subject's compatibility mode is transitive (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE or FULL_TRANSITIVE) then the
compatibility is checked not only against the latest schema, but also against all previous schemas, as it would be done
when trying to register the new schema through the `subjects/<subject-key>/versions` endpoint.
Get current global backwards compatibility setting value::
$ curl -X GET http://localhost:8081/config
{"compatibilityLevel":"BACKWARD"}
Change compatibility requirements for all subjects where it's not
specifically defined otherwise::
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"compatibility": "NONE"}' http://localhost:8081/config
{"compatibility":"NONE"}
Change compatibility requirement to FULL for the test-key subject::
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"compatibility": "FULL"}' http://localhost:8081/config/test-key
{"compatibility":"FULL"}
Get the current global mode. Karapace supports ``READWRITE`` (the default) and ``IMPORT``::
$ curl -X GET http://localhost:8081/mode
{"mode":"READWRITE"}
Switch the test-key subject to IMPORT mode, which allows schemas to be registered with
their original schema ids and version numbers. The subject must be empty, unless
``?force=true`` is given::
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"mode": "IMPORT"}' http://localhost:8081/mode/test-key
{"mode":"IMPORT"}
Register a schema with an explicit id and version, which is only accepted in IMPORT mode::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"string\"}", "id": 1001, "version": 5}' \
http://localhost:8081/subjects/test-key/versions
{"id":1001}
Read the effective mode of a subject. Add ``?defaultToGlobal=true`` to get the global mode
instead of a 404 for an unknown subject::
$ curl -X GET http://localhost:8081/mode/test-key
{"mode":"IMPORT"}
Remove the subject level mode so the subject follows the global mode again::
$ curl -X DELETE http://localhost:8081/mode/test-key
{"mode":"READWRITE"}
IMPORT mode skips compatibility checks and content deduplication, schema ids are global and
immutable once bound, and a failed import is resumed by replaying it rather than rolled
back. See the `Import mode <https://www.karapace.io/docs/import-mode>`_ documentation for
the full constraints and recovery steps.
Schema Registry Rest proxy Api reference
========================================
List topics::
$ curl "http://localhost:8082/topics"
Get info for one particular topic::
$ curl "http://localhost:8082/topics/my_topic"
Produce a message backed up by schema registry::
$ curl -H "Content-Type: application/vnd.kafka.avro.v2+json" -X POST -d \
'{"value_schema": "{\"namespace\": \"example.avro\", \"type\": \"record\", \"name\": \"simple\", \"fields\": \
[{\"name\": \"name\", \"type\": \"string\"}]}", "records": [{"value": {"name": "name0"}}]}' http://localhost:8082/topics/my_topic
A record may carry optional Kafka message ``headers``, a list of ``{"name": <string>, "value": <base64 string | null>}`` objects (header values are raw bytes, so they are base64-encoded)::
$ curl -H "Content-Type: application/vnd.kafka.json.v2+json" -X POST -d \
'{"records": [{"value": {"name": "name0"}, "headers": [{"name": "traceId", "value": "YWJjMTIz"}]}]}' \
http://localhost:8082/topics/my_topic
Create a consumer with consumer group 'avro_consumers' and consumer instance 'my_consumer' ::
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" -H "Accept: application/vnd.kafka.v2+json" \
--data '{"name": "my_consumer", "format": "avro", "auto.offset.reset": "earliest"}' \
http://localhost:8082/consumers/avro_consumers
Subscribe to the topic we previously created ::
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["my_topic"]}' \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer/subscription
Consume previously produced message::
$ curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer/records?timeout=1000
When a consumed record has headers, they are returned in a ``headers`` field using the same ``{"name": ..., "value": <base64 string | null>}`` form (the field is omitted for records without headers).
Commit offsets for a particular topic partition::
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{}' \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer/offsets
Delete consumer::
$ curl -X DELETE -H "Accept: application/vnd.kafka.v2+json" \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer
Backing up your Karapace
========================
.. note::
The built-in backup functionality (``karapace_schema_backup``) has been removed
as of version 6.0.0. Users who need this functionality can continue to use
Karapace versions prior to 6.0.0, however it will no longer be maintained
within the project.
Performance comparison to Confluent stack
==========================================
Latency
-------
* 50 concurrent connections, 50.000 requests
====== ========== ===========
Format Karapace Confluent
====== ========== ===========
Avro 80.95 7.22
Binary 66.32 46.99
Json 60.36 53.7
====== ========== ===========
* 15 concurrent connections, 50.000 requests
====== =========== ===========
Format Karapace Confluent
====== =========== ===========
Avro 25.05 18.14
Binary 21.35 15.85
Json 21.38 14.83
====== =========== ===========
* 4 concurrent connections, 50.000 requests
====== =========== ===========
Format Karapace Confluent
====== =========== ===========
Avro 6.54 5.67
Binary 6.51 4.56
Json 6.86 5.32
====== =========== ===========
Also, it appears there is quite a bit of variation on subsequent runs, especially for the lower numbers, so once
more exact measurements are required, it's advised we increase the total req count to something like 500K
We'll focus on Avro serialization only after this round, as it's the more expensive one, plus it tests the entire stack
Consuming RAM
-------------
A basic push pull test , with 12 connections on the publisher process and 3 connections on the subscriber process, with a
10 minute duration. The publisher has the 100 ms timeout and 100 max_bytes parameters set on each request so both processes have work to do
Heap size limit is set to 256M on Rest proxy
Ram consumption, different consumer count, over 300s
=========== =================== ================
Consumers Karapace combined Confluent rest
=========== =================== ================
1 47 200
10 55 400
20 83 530
=========== =================== ================
Commands
========
Once installed, the ``karapace`` program should be in your path. It is the
main daemon process that should be run under a service manager such as
``systemd`` to serve clients.
Configuration keys
==================
Keys to take special care are the ones needed to configure Kafka and advertised_hostname.
.. list-table::
:header-rows: 1
* - Parameter
- Default Value
- Description
* - ``advertised_hostname``
- ``socket.gethostname()``
- The hostname being advertised to other instances of Karapace that are attached to the same Kafka group. All nodes within the cluster need to have their ``advertised_hostname``'s set so that they can all reach each other.
* - ``advertised_port``
- ``None``
- The port being advertised to other instances of Karapace that are attached to the same Kafka group. Fallbacks to ``port`` if not set.
* - ``advertised_protocol``
- ``http``
- The protocol being advertised to other instances of Karapace that are attached to the same Kafka group.
* - ``bootstrap_uri``
- ``localhost:9092``
- The URI to the Kafka service where to store the schemas and to run
coordination among the Karapace instances.
* - ``client_id``
- ``sr-1``
- The ``client_id`` Karapace will use when coordinating with
other Karapace instances. The instance with the ID that sorts
first alphabetically is chosen as master from the services with
master_eligibility set to true.
* - ``consumer_enable_autocommit``
- ``True``
- Enable auto commit on rest proxy consumers
* - ``consumer_idle_disconnect_timeout``
- ``0``
- Disconnect idle consumers after timeout seconds if not used. Inactivity leads to consumer leaving consumer group and consumer state. 0 (default) means no auto-disconnect.
* - ``consumer_request_max_bytes``
- ``67108864``
- Rest proxy consumers maximum bytes to be fetched per request
* - ``consumer_request_timeout_ms``
- ``11000``
- Rest proxy consumers timeout for reads that do not limit the max bytes or provide their own timeout
* - ``fetch_min_bytes``
- ``1``
- Rest proxy consumers minimum bytes to be fetched per request.
* - ``group_id``
- ``schema-registry``
- The Kafka group name used for selecting a master service to coordinate the storing of Schemas.
* - ``host``
- ``127.0.0.1``
- Listening host for the Karapace server. Use an empty string to
listen to all available networks.
* - ``http_request_max_size``
- ``1048576``
- The maximum client HTTP request size. This value controls how large (POST) payloads are allowed. When configuration of ``karapace_rest`` is set to `true` and ``http_request_max_size`` is not set, Karapace configuration adapts the allowed client max size from the ``producer_max_request_size``. In cases where automatically selected size is not enough the configuration can be overridden by setting a value in configuration. For schema registry operation set the client max size according to expected size of schema payloads if default size is not enough.
* - ``kafka_retriable_errors_silenced``
- ``true``
- If enabled, kafka errors which can be retried or custom errors specified for the service will not be raised,
instead, a warning log is emitted. This will denoise issue tracking systems, i.e. sentry
* - ``kafka_schema_reader_strict_mode``
- ``false``
- If enabled, causes the Karapace schema-registry service to shutdown when there are invalid schema records in the `_schemas` topic
* - ``karapace_registry``
- ``true``
- If the registry part of the app should be included in the starting process
At least one of this and ``karapace_rest`` options need to be enabled in order
for the service to start
* - ``karapace_rest``
- ``true``
- If the rest part of the app should be included in the starting process
At least one of this and ``karapace_registry`` options need to be enabled in order
for the service to start
* - ``librdkafka_debug``
- ``null``
- librdkafka ``debug`` contexts applied to every confluent-kafka client of both the schema-registry and the REST proxy
(consumers, producers, admin clients), e.g. ``broker,metadata,topic``. See the
`librdkafka configuration reference <https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md>`_
for the available contexts, an unknown context is rejected at startup. The debug messages are written to stderr. Meant for troubleshooting only, as it is verbose.
* - ``log_format``
- ``%(name)-20s\t%(threadName)s\t%(levelname)-8s\t%(message)s``
- Log format
* - ``log_handler``
- ``stdout``
- Select the log handler. Default is standard output. Alternative log handler is ``systemd``.
* - ``log_level``
- ``DEBUG``
- Logging level. Default level is debug.
* - ``master_election_strategy``
- ``lowest``
- Decides on what basis the Karapace cluster master is chosen (only relevant in a multi node setup)
* - ``master_eligibility``
- ``true``
- Should the service instance be considered for promotion to the master
service. One reason to turn this off would be to have an instance of Karapace
running somewhere else for HA purposes but which you wouldn't want to
automatically promote to master if the primary instances become
unavailable.
* - ``metadata_max_age_ms``
- ``60000``
- Period of time in milliseconds after Kafka metadata is force refreshed.
* - ``name_strategy``
- ``topic_name``
- Name strategy to use when storing schemas from the kafka rest proxy service. You can opt between ``topic_name`` , ``record_name`` and ``topic_record_name``
* - ``name_strategy_validation``
- ``true``
- If enabled, validate that given schema is registered under used name strategy when producing messages from Kafka Rest
* - ``port``
- ``8081``
- Listening port for the Karapace server.
* - ``producer_acks``
- ``1``
- Level of consistency desired by each producer message sent on the rest proxy.
More on `Kafka Producer <https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html>`__
* - ``producer_compression_type``
- ``None``
- Type of compression to be used by rest proxy producers
* - ``producer_linger_ms``
- ``0``
- Time to wait for grouping together requests.
More on `Kafka Producer <https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html>`__
* - ``producer_max_request_size``
- ``1048576``
- The maximum size of a request in bytes.
More on `Kafka Producer configs <https://kafka.apache.org/documentation/#producerconfigs_max.request.size>`_
* - ``protobuf_runtime_directory``
- ``runtime``
- Runtime directory for the ``protoc`` protobuf schema parser and code generator
* - ``registry_authfile``
- ``/path/to/authfile.json``
- Filename to specify users and access control rules for Karapace Schema Registry.
If this is set, Schema Registry requires authentication for most of the endpoints and applies per endpoint authorization rules.
* - ``registry_ca``
- ``/path/to/cafile``
- Kafka Registry CA certificate, used by Kafka Rest for Avro related requests.
If this is set, Kafka Rest will use HTTPS to connect to the registry.
If running both in the same process, it should be left to its default value
* - ``registry_host``
- ``127.0.0.1``
- Schema Registry host, used by Kafka Rest for schema related requests.
If running both in the same process, it should be left to its default value
* - ``registry_password``
- ``None``
- Schema Registry password for authentication, used by Kafka Rest for schema related requests.
A client ``Authorization`` header forwarded to the registry takes precedence; this is the fallback when none is forwarded.
* - ``registry_port``
- ``8081``
- Schema Registry port, used by Kafka Rest for schema related requests.
If running both in the same process, it should be left to its default value
* - ``registry_scheme``
- ``http``
- Schema Registry scheme to use for rest-proxy, http | https (if certificates are provided).
* - ``registry_user``
- ``None``
- Schema Registry user for authentication, used by Kafka Rest for schema related requests.
A client ``Authorization`` header forwarded to the registry takes precedence; this is the fallback when none is forwarded.
* - ``replication_factor``
- ``1``
- The replication factor to be used with the schema topic.
* - ``rest_authorization``
- ``false``
- Use REST API's calling authorization credentials to invoke Kafka operations over SASL authentication of ``sasl_bootstrap_uri`` to delegate REST proxy authorization to Kafka. If false, then use configured common credentials for all Kafka connections of REST proxy operations.
* - ``rest_base_uri``
- ``None``
- Publicly available URI of this instance advertised to the clients using stateful operations such as creating consumers. If not set, then construct URI using ``advertised_protocol``, ``advertised_hostname``, and ``advertised_port``.
* - ``sasl_bootstrap_uri``
- ``None``
- The URI to the Kafka service to use with the Kafka REST API when SASL authorization with REST is used.
* - ``sasl_mechanism``
- ``None``
- SASL mechanism for Kafka authentication (e.g. ``PLAIN``, ``SCRAM-SHA-512``, ``OAUTHBEARER``). Used together with ``security_protocol``.
* - ``sasl_plain_password``
- ``None``
- Password for Kafka SASL ``PLAIN`` authentication.
* - ``sasl_plain_username``
- ``None``
- Username for Kafka SASL ``PLAIN`` authentication.
* - ``security_protocol``
- ``PLAINTEXT``
- Default Kafka security protocol needed to communicate with the Kafka
cluster. Other options is to use SSL for SSL client certificate
authentication, or SASL_PLAINTEXT / SASL_SSL for simple authentication.
* - ``sentry``
- ``None``
- Used to configure parameters for sentry integration (dsn, tags, ...). Setting the
environment variable ``SENTRY_DSN`` will also enable sentry integration.
* - ``server_tls_cafile``
- ``/path/to/cafile``
- Filename to the SSL CA certificate.
* - ``server_tls_certfile``
- ``/path/to/certfile``
- Filename to a certificate chain for the Karapace server in HTTPS mode.
* - ``server_tls_client_auth``
- ``none``
- Client certificate requirement: ``none``, ``optional``, or ``required``.
* - ``server_tls_keyfile``
- ``/path/to/keyfile``
- Filename to a private key for the Karapace server in HTTPS mode.
* - ``ssl_cafile``
- ``/path/to/cafile``
- Used when ``security_protocol`` is set to SSL, the path to the SSL CA certificate.
* - ``ssl_certfile``
- ``/path/to/certfile``
- Used when ``security_protocol`` is set to SSL, the path to the SSL certfile.
* - ``ssl_keyfile``
- ``/path/to/keyfile``
- Used when ``security_protocol`` is set to SSL, the path to the SSL keyfile.
* - ``topic_name``
- ``_schemas``
- The name of the Kafka topic where to store the schemas.
* - ``allow_duplicate_schema_ids``
- ``true``
- Whether one schema may be imported under more than one schema id. Set to ``false`` to refuse an import whose content is already
registered under a different id, keeping ids one to one with content.
* - ``mode_mutability``
- ``true``
- Whether mode changes are allowed. Set to ``false`` to reject every ``PUT``/``DELETE`` on ``/mode``, so ``IMPORT`` mode cannot be
entered at runtime.
* - ``use_protobuf_formatter``
- ``false``
- If protobuf formatter should be used on protobuf schemas in order to normalize schemas. The formatter is used on top and independent of regular normalization and schemas will be persisted in a formatted state.
* - ``waiting_time_before_acting_as_master_ms``
- ``5000``
- The time that a master wait before becoming an active master if at the previous round of election wasn't the master (in that case the waiting time its skipped).
Should be an upper bound of the time required for a master to write a message in the kafka topic + the time required from a node in the cluster to consume the
Log of messages. If the value its too low there is the risk under high load of producing different schemas with the ID.
Authentication and authorization of Karapace Schema Registry REST API
=====================================================================
To enable HTTP Basic Authentication and user authorization the authorization configuration file is set in the main configuration key ``registry_authfile`` of the Karapace.
Karapace Schema Registry authorization file is an optional JSON configuration, which contains a list of authorized users in ``users`` and a list of access control rules in ``permissions``.
Each user entry contains following attributes:
.. list-table::
:header-rows: 1
* - Parameter
- Description
* - ``username``
- A string
* - ``algorithm``
- One of supported hashing algorithms, ``scrypt``, ``sha1``, ``sha256``, or ``sha512``
* - ``salt``
- Salt used for hashing the password
* - ``password_hash``
- Hash string of the password calculated using given algorithm and salt.
Password hashing can be done using ``karapace_mkpasswd`` tool, if installed, or by invoking directly with ``python -m karapace.core.auth``. The tool generates JSON entry with these fields. ::
$ karapace_mkpasswd -u user -a sha512 secret
{
"username": "user",
"algorithm": "sha512",
"salt": "iuLouaExTeg9ypqTxqP-dw",
"password_hash": "R6ghYSXdLGsq6hkQcg8wT4xkD4QToxBhlp7NerTnyB077M+mD2qiN7ZxXCDb4aE+5lExu5P11UpMPYAcVYxSQA=="
}
Each access control rule contains following attributes:
.. list-table::
:header-rows: 1
* - Parameter
- Description
* - ``username``
- A string to match against authenticated user
* - ``operation``
- Exact value of ``Read`` or ``Write``. Write implies also read permissions. Write includes all mutable operations, e.g. deleting schema versions
* - ``resource``
- A regular expression used to match against accessed resource.
Supported resource authorization:
.. list-table::
:header-rows: 1
* - Resource
- Description
* - ``Config:``
- Controls authorization to global schema registry configuration.
* - ``Subject:<subject_name>``
- Controls authorization to subject. The ``<subject_name>`` is a regular expression to match against the accessed subject.
Example of complete authorization file
--------------------------------------
::
{
"users": [
{
"username": "admin",
"algorithm": "scrypt",
"salt": "<put salt for randomized hashing here>",
"password_hash": "<put hashed password here>"
},
{
"username": "plainuser",
"algorithm": "sha256",
"salt": "<put salt for randomized hashing here>",
"password_hash": "<put hashed password here>"
}
],
"permissions": [
{
"username": "admin",
"operation": "Write",
"resource": ".*"
},
{
"username": "plainuser",
"operation": "Read",
"resource": "Subject:general.*"
},
{
"username": "plainuser",
"operation": "Read",
"resource": "Config:"
}
]
}
Karapace Schema Registry access to the _schemas topic
=====================================================
The principal used by the Karapace Schema Registry has to have adequate access to the _schemas topic (see the ``topic_name`` configuration option above).
In addition to what is required to access the topic, as described in the Confluent Schema Registry documentation_, the unique, single-member consumer group
used by consumers in the schema registry needs ``Describe`` and ``Read`` permissions_ on the group.
These unique (per instance of the schema registry) consumer group names are prefixed by ``karapace-autogenerated-``, followed by a random string.
.. _`documentation`: https://docs.confluent.io/platform/current/schema-registry/security/index.html#authorizing-access-to-the-schemas-topic
.. _`permissions`: https://docs.confluent.io/platform/current/kafka/authorization.html#group-resource-type-operations
OAuth2 authentication and authorization of Karapace
===================================================
Karapace supports OAuth2 authentication and authorization. The JSON Web Token (JWT) is extracted from the ``Authorization`` HTTP header if the authorization scheme is ``Bearer``,
eg. ``Authorization: Bearer $JWT``. If a ``Bearer`` token is present in schema registry requests, karapace will validate the token against OpenId connect provider and continue.
Below here is an example of karapace OpenId connect config ::
sasl_oauthbearer_authentication_enabled: bool = True
sasl_oauthbearer_jwks_endpoint_url = "http://localhost:8383/realms/karapace/protocol/openid-connect/certs",
sasl_oauthbearer_expected_issuer = "http://localhost:8383/realms/karapace",
sasl_oauthbearer_expected_audience = "account",
sasl_oauthbearer_sub_claim_name = "sub",
``sasl_oauthbearer_expected_issuer`` and ``sasl_oauthbearer_expected_audience`` are optional; an
unset value skips verification of that claim. ``sasl_oauthbearer_expected_audience`` accepts a
comma-separated list and a token matching any entry is accepted. With Keycloak, the default
access token carries ``aud: account``, so add an audience mapper to the client if you want to
pin the audience to the client itself.
For authorization (requires authentication to be enabled)::
sasl_oauthbearer_authentication_enabled: bool = True
sasl_oauthbearer_authorization_enabled: bool = False
sasl_oauthbearer_roles_claim_path: str | None = None
# Defaults to the karapace.* roles shown in the example below; override per deployment.
sasl_oauthbearer_method_roles: dict[str, list[str]]
Below here is an example of karapace OpenId connect docker config ::
KARAPACE_SASL_OAUTHBEARER_AUTHENTICATION_ENABLED: True
KARAPACE_SASL_OAUTHBEARER_JWKS_ENDPOINT_URL: http://keycloak:8080/realms/karapace/protocol/openid-connect/certs
KARAPACE_SASL_OAUTHBEARER_EXPECTED_ISSUER: http://keycloak:8080/realms/karapace
KARAPACE_SASL_OAUTHBEARER_EXPECTED_AUDIENCE: "account"
KARAPACE_SASL_OAUTHBEARER_SUB_CLAIM_NAME: sub
For authorization, example config::
KARAPACE_SASL_OAUTHBEARER_AUTHENTICATION_ENABLED: True
KARAPACE_SASL_OAUTHBEARER_AUTHORIZATION_ENABLED: True
KARAPACE_SASL_OAUTHBEARER_ROLES_CLAIM_PATH: "resource_access.karapace-client.roles"
KARAPACE_SASL_OAUTHBEARER_METHOD_ROLES: dict[str, list[str]] = {"GET": ["karapace.schema:read", "karapace.subject:read"],
"POST": ["karapace.schema:write", "karapace.subject:write"], "PUT": [], "DELETE": []}
.. note::
``sasl_oauthbearer_client_id`` and ``sasl_oauthbearer_enforce_azp`` have been removed. The
``azp`` claim is no longer validated. The ``[client_id]`` placeholder in
``sasl_oauthbearer_roles_claim_path`` is no longer substituted — write the literal client id
into the path (e.g. ``resource_access.karapace-client.roles``).
Production hardening notes
--------------------------
* ``sasl_oauthbearer_jwks_endpoint_url`` must use ``https://``; startup fails otherwise.
The ``sasl_oauthbearer_allow_insecure_jwks`` flag overrides this for dev/test only.
* ``/docs``, ``/redoc``, ``/openapi.json`` bypass the auth gate (by design, for Swagger UI).
Block them at the reverse proxy in production to avoid leaking the API surface.
Testing the authentication and authorization with docker
--------------------------------------------------------
Get token
---------
If you are running with docker, login to karapace container with ::
docker exec -it containerid /bin/sh
and get a token like below. ::
curl -s -X POST "http://keycloak:8080/realms/karapace/protocol/openid-connect/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials" \
-d "client_id=karapace-client" \
-d "client_secret=client-secret" \
Note : client id and client secret can be retrieved from Oidc provider
Response of the above curl should be a access token, and other scope and expiry details.
Export the token into ACCESS_TOKEN variable::
export ACCESS_TOKEN='token..'
Access schema registry
----------------------
Send the token to get subjects url::
curl --insecure -H "Authorization: Bearer $ACCESS_TOKEN" https://localhost:8081/subjects/testtopic-value/versions
Response should be list of available subjects, as it is a valid token, and GET (schema read, subject read) is allowed as per the defined method roles.
Send an invalid token to get subjects url::
curl --insecure -H "Authorization: Bearer aninvalidtoken" https://localhost:8081/subjects
Response should be {"error":"Unauthorized","reason":"Invalid token/payload"}
OAuth2 authentication and authorization of Karapace REST proxy
===================================================================
The Karapace REST proxy supports passing OAuth2 credentials to the underlying Kafka service (defined in the ``sasl_bootstrap_uri`` configuration parameter). The JSON Web Token (JWT) is extracted from the ``Authorization`` HTTP header if the authorization scheme is ``Bearer``,
eg. ``Authorization: Bearer $JWT``. If a ``Bearer`` token is present, the Kafka clients managed by Karapace will be created to use the SASL ``OAUTHBEARER`` mechanism and the JWT will be passed along. The Karapace REST proxy does not verify the token, that is done by
the underlying Kafka service itself, if it's configured accordingly.
Authorization is also done by Kafka itself, typically using the ``sub`` claim (although it's configurable) from the JWT as the username, checked against the configured ACLs.
OAuth2 and ``Bearer`` token usage is dependent on the ``rest_authorization`` configuration parameter being ``true``.
Forwarding credentials to the Schema Registry
----------------------------------------------
For schema operations (registration and lookup during produce/consume), the REST proxy forwards the inbound
``Authorization`` header to the Schema Registry when ``sasl_oauthbearer_authentication_enabled`` is ``true``.
Both ``Bearer`` (validated by the registry via OIDC) and ``Basic`` (validated against ``registry_authfile``) schemes
are forwarded as-is, so a client can authenticate to the registry with its own token or credentials.
A forwarded ``Authorization`` header takes precedence over the proxy's own ``registry_user`` / ``registry_password``;
those service credentials are used only for requests that arrive without a forwarded header. The two may be configured
together, which lets clients migrate to forwarded tokens one at a time. This forwarding is independent of
``rest_authorization`` (which governs Kafka-side authentication).
When the Schema Registry rejects the credentials or token during a schema operation, the REST proxy reports the failure
with a matching status: ``401`` (error code ``401``) for authentication and ``403`` (error code ``403``) for
authorization. Other schema registration failures keep their existing errors and are not reported as auth failures — an
incompatible or otherwise rejected schema returns ``40801``, and a schema that fails to parse returns ``42205``.
Token expiry
------------
The REST proxy process manages a set of producer and consumer clients, which are identified by the OAuth2 JWT token. These are periodically cleaned up if they are idle, as well as *before* the JWT token expires (the clean up currently runs every 5 minutes).
Before a client refreshes its OAuth2 JWT token, it is expected to remove currently running consumers (eg. after committing their offsets) and producers using the current token.
Schema Normalization
--------------------
If specified as a rest parameter for the POST ``/subjects/{subject}/versions?normalize=true`` endpoint and the POST ``subjects/{subject}?normalize=true`` endpoint,
Karapace uses a schema normalization algorithm to ensure that the schema is stored in a canonical form.
This normalization process is done so that schemas semantically equivalent are stored in the same way and should be considered equal.
Normalization is currently only supported for Protobuf schemas. Karapace does not support all normalization features implemented by Confluent Schema Registry.
Currently the normalization process covers:
- Ordering of optional fields in the schema.
- Restoration of map field shorthand: binary-registered schemas that contain the expanded entry-message
form (a ``repeated`` field pointing to a synthetic nested message with ``option map_entry = true``)
are converted back to ``map<K, V>`` syntax, matching Confluent Schema Registry behaviour.
Use the feature with the assumption that it will be extended in the future and so two schemas that are semantically equivalent could be considered
different by the normalization process in different future versions of Karapace.
The safe choice, when using a normalization process, is always to consider as different two schemas that are semantically equivalent while the problem is when two semantically different schemas are considered equivalent.
In that view the future extension of the normalization process isn't considered a breaking change but rather an extension of the normalization process.
JSON Schema support
===================
Karapace supports registering schemas of type ``JSON`` (set ``"schemaType": "JSON"`` when registering). This section
describes which JSON Schema drafts are supported, how compatibility is evaluated, current limitations, and guidance for
migrating between drafts.
Supported drafts and ``$schema`` URIs
-------------------------------------
Karapace selects the JSON Schema draft from the ``$schema`` keyword and validates the schema against that draft's
metaschema. When ``$schema`` is absent or its value is not recognized, the schema is validated as **Draft-7**, which
preserves the previous behaviour and matches the default used by Confluent Schema Registry.
.. list-table::
:header-rows: 1
* - Draft
- ``$schema`` URI
- Status
* - draft-04
- ``http://json-schema.org/draft-04/schema#``
- Validated with its own draft validator
* - draft-06
- ``http://json-schema.org/draft-06/schema#``
- Validated with its own draft validator
* - draft-07
- ``http://json-schema.org/draft-07/schema#``
- Validated with its own draft validator (also the default when ``$schema`` is absent or unrecognized)
* - 2019-09
- ``https://json-schema.org/draft/2019-09/schema``
- Validated with its own draft validator
* - 2020-12
- ``https://json-schema.org/draft/2020-12/schema``
- Validated with its own draft validator
Compatibility checking canonicalizes the renamed newer-draft keywords and rejects the ones it cannot yet evaluate — see
`Compatibility behavior`_ and `Limitations`_ below.
Compatibility behavior
----------------------
Compatibility between schema versions is evaluated using the compatibility mode configured for the subject (or the
global default). The supported modes are ``NONE``, ``BACKWARD``, ``BACKWARD_TRANSITIVE``, ``FORWARD``,
``FORWARD_TRANSITIVE``, ``FULL`` and ``FULL_TRANSITIVE``. Transitive modes check the new schema against *all* previous
versions; non-transitive modes only check against the latest one.
The following keywords are taken into account when comparing versions: ``type``, ``properties``, ``required``,
``additionalProperties``, ``enum``, ``$ref``, the numeric bounds (``minimum``, ``maximum``, ``exclusiveMinimum``,
``exclusiveMaximum``, ``multipleOf``), the string constraints (``minLength``, ``maxLength``, ``pattern``), the array
constraints (``items``, ``additionalItems``, ``minItems``, ``maxItems``, ``uniqueItems``), the object-size constraints
(``minProperties``, ``maxProperties``, ``propertyNames``, ``patternProperties``) and the dependency keywords
(``dependencies``, ``dependentSchemas``).
Draft 2019-09 / 2020-12 keywords that are renames or relocations of Draft-7 keywords are **canonicalized** before
comparison, so they are evaluated correctly and are interchangeable across drafts within a subject:
* ``$defs`` is treated as ``definitions``
* ``prefixItems`` is treated as the array (tuple) form of ``items``; a sibling ``items`` schema is treated as
``additionalItems``
* ``dependentRequired`` is treated as the string-array form of ``dependencies``
Examples of backward-compatible evolutions (a new schema can read data written with the old one):
* Widening a numeric range (lowering ``minimum`` or raising ``maximum``)
* Relaxing string constraints (lowering ``minLength``, raising ``maxLength``, removing ``pattern``)
* Adding options to an ``enum``
Examples of backward-**incompatible** evolutions:
* Narrowing ``type`` (e.g. from ``number`` to ``integer``)
* Adding or decreasing ``maxLength`` / ``maximum``
* Removing options from an ``enum``
* Removing a property from a closed content model (``additionalProperties: false``)
* Tightening ``additionalProperties`` (from ``true`` to ``false``)
* Adding a ``required`` property to a content model that is not open
* Adding a positional ``prefixItems`` entry the previous version did not require
Forward compatibility applies the mirror of these rules, and full compatibility requires both directions. Because the
renamed keywords share a single canonical form, a subject can evolve across drafts (e.g. a Draft-7 version followed by a
2020-12 version) and compatibility is evaluated on that canonical form.
Limitations
-----------
Schema **validation** is draft-specific: a schema is validated against the draft declared in ``$schema`` (defaulting to
Draft-7), so newer-draft keywords are validated with their intended semantics.
Some newer-draft keywords cannot yet be evaluated correctly during **compatibility checking**. To avoid returning a
result that looks safe but is not, Karapace **rejects the compatibility check** (fails closed) when either the previous
or the new version uses any of: ``unevaluatedProperties``, ``unevaluatedItems``, ``$dynamicRef``, ``$dynamicAnchor``,
``$recursiveRef``, ``$recursiveAnchor`` or ``$vocabulary``. The registration is rejected with an error naming the
keyword, rather than accepted and later surprising a producer or consumer at runtime.
Such schemas can still be registered under a subject whose compatibility is set to ``NONE`` (no compatibility check is
performed). Evaluating these keywords correctly requires resolving dynamic references and reasoning across subschemas,
which the compatibility checker does not do yet; until it does, they remain fail-closed rather than being compared
under an approximation that could report a breaking change as safe.
There is no strict/lenient mode for JSON Schema validation or compatibility. (The ``kafka_schema_reader_strict_mode``
configuration is unrelated — it controls error handling when reading the internal ``_schemas`` topic, not JSON Schema
validation.)
Migration guidance
------------------
* **Validation:** schemas declaring Draft 2019-09 or 2020-12 (or draft-04 / draft-06) via ``$schema`` are validated with
that draft's rules; omitting ``$schema`` defaults to Draft-7.
* **Compatibility:** the renamed/relocated keywords (``$defs``, ``prefixItems``, ``dependentRequired``) are evaluated
correctly and are interchangeable with their Draft-7 spelling, so you can migrate a subject from Draft-7 to
2019-09 / 2020-12 without a compatibility break as long as the schema shape is otherwise unchanged.
* **Unsupported keywords fail closed:** if a version uses ``unevaluatedProperties`` / ``unevaluatedItems``,
``$dynamicRef`` / ``$dynamicAnchor``, ``$recursiveRef`` / ``$recursiveAnchor`` or ``$vocabulary``, the compatibility
check is rejected. Use compatibility ``NONE`` for that subject if you need to register such schemas.
Uninstall
=========
To uninstall Karapace from the system, you can follow the instructions described below. We would love to hear your reasons for uninstalling though. Please file an issue if you experience any problems or email us_ with feedback
.. _`us`: mailto:[email protected]
Installed via Docker
--------------------
If you installed Karapace via Docker, you would need to first stop and remove the images like described:
First obtain the container IDs related to Karapace, you should have one for the registry itself and another one for the rest interface::
docker ps | grep karapace
After this, you can stop each of the containers with::
docker stop <CONTAINER_ID>
If you don't need or want to have the Karapace images around you can now proceed to delete them using::
docker rm <CONTAINER_ID>
Installed from Sources
----------------------
Karapace is installed ``pip install .``, it can be uninstalled with the following ``pip`` command::
pip uninstall karapace
Development
===========
For local development and testing with Docker, use the convenience make commands.
These handle all the setup automatically and ensure consistency across development environments.
Quick Start
-----------
**Set `.env`:** ``cp .env.example .env``
Edit ``.env`` to customize:
- ``PYTHON_VERSION`` - Python version for Docker builds (default: 3.12)
- ``KARAPACE_VERSION`` - Karapace version (default: 5.0.3)
- ``DOCKER_COMPOSE`` - Docker compose command (default: docker-compose)
- ``RUNNER_UID`` / ``RUNNER_GID`` - Container user IDs (check .env.example on how to get)
- ``PYTEST_ARGS`` - Pytest arguments for test runs
Available Commands
------------------
**Container Management:**
- ``make cli`` - Open interactive shell inside Karapace container
- ``make start-karapace-docker-resources`` - Start all Docker containers (Kafka, Schema Registry, REST Proxy)
- ``make stop-karapace-docker-resources`` - Stop all Docker containers
**Testing:**
- ``make unit-tests-in-docker`` - Run unit tests in Docker
- ``make integration-tests-in-docker`` - Run integration tests in Docker
- ``make e2e-tests-in-docker`` - Run end-to-end tests in Docker
- ``make smoke-test-schema-registry`` - Run schema registry smoke tests
- ``make smoke-test-rest-proxy`` - Run REST proxy smoke tests
- ``make type-check-mypy-in-docker`` - Run type checking in Docker
For local development and testing without Docker, see the instructions below.
Execute ``make`` (GNU, usually ``gmake`` on BSD and Mac) to set up a ``venv``
and install the required software for development. Use ``make unit-tests`` and
``make integration-tests`` to execute the respective test suite, or simply
``make test`` to execute both. You can set ``PYTEST_ARGS`` to customize the
execution (e.g. ``PYTEST_ARGS=--maxfail=1 make test``).
Karapace currently depends on various system software to be installed. The
installation of these is automated for some operation systems, but not all. At
the time of writing Java, the Protobuf Compiler, and the Snappy shared library
are required to work with Karapace. You need to install them manually if your
operating system is not supported by the automatic installation scripts. Note
that the scripts are going to ask before installing any of these on your system.
Note that Karapace requires a Protobuf Compiler older than 3.20.0, because
3.20.0 introduces various breaking changes. The tests are going to fail if the
Protobuf Compiler is newer than that. However, you can work around this locally
by running ``pip install --upgrade protobuf`` in your venv. We are going to fix
this soon.
Note that the integration tests are currently not working on Mac. You can use
Docker, just be sure to set ``VENV_DIR`` to a directory outside the working
directory so that the container is not overwriting files from the host (e.g.
``docker run --env VENV_DIR=/tmp/venv ...``).
Note that the ``runtime`` directory **MUST** exist and that Karapace is going to
fail if it does not. The ``runtime`` directory is also not cleaned between test
runs, and left over data might result in failing tests. Use the ``make`` test
targets that correctly clean the ``runtime`` directory without deleting it, but
keep this in mind whenever you are not using ``make`` (e.g. running tests from
your IDE).
Use ``pipx`` or ``brew`` to install ``pre-commit`` and use the global installation,
there is also no dependency on it.
License
=======
Karapace is licensed under the Apache license, version 2.0. Full license text is
available in the ``LICENSE`` file.
Please note that the project explicitly does not require a CLA (Contributor
License Agreement) from its contributors.
Contact
=======
Bug reports and patches are very welcome, please post them as GitHub issues
and pull requests at https://github.com/Aiven-Open/karapace . Any possible
vulnerabilities or other serious issues should be reported directly to the
maintainers <[email protected]>.
Trademark
=========
Apache Kafka is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries. Kafka Rest and Schema Registry are trademarks and property of their respective owners. All product and service names used in this page are for identification purposes only and do not imply endorsement.
Credits
=======
Karapace was created by, and is maintained by, Aiven_ cloud data hub
developers.
The schema storing part of Karapace loans heavily from the ideas of the
earlier Schema Registry implementation by Confluent and thanks are in order
to them for pioneering the concept.
.. _`Aiven`: https://aiven.io/
Recent contributors are listed on the GitHub project page,
https://github.com/Aiven-Open/karapace/graphs/contributors
Copyright ⓒ 2021 Aiven Ltd.
Not written in Markdown, so it's shown here as plain text — view it formatted on GitHub.
2,477 followers · starred Apr 2024
326 followers · starred Apr 2023
42 followers · starred Mar 2023
823 followers · starred Jan 2022
Karapace - Your Apache Kafka® essentials in one tool
See the codeKarapace
========
``karapace``. Your Apache Kafka® essentials in one tool.
An `open-source <https://github.com/Aiven-Open/karapace/blob/master/LICENSE>`_ implementation
of `Kafka REST <https://docs.confluent.io/platform/current/kafka-rest/index.html#features>`_ and
`Schema Registry <https://docs.confluent.io/platform/current/schema-registry/index.html>`_.
|Tests| |Contributor Covenant|
.. |Tests| image:: https://github.com/Aiven-Open/karapace/actions/workflows/tests.yml/badge.svg?branch=main
:target: https://github.com/Aiven-Open/karapace/actions/workflows/tests.yml?query=branch%3Amain
.. |Contributor Covenant| image:: https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg
:target: CODE_OF_CONDUCT.md
Overview
========
Karapace supports the storing of schemas in a central repository, which clients can access to
serialize and deserialize messages. The schemas also maintain their own version histories and can be
checked for compatibility between their different respective versions.
Karapace rest provides a RESTful interface to your Apache Kafka cluster, allowing you to perform tasks such
as producing and consuming messages and perform administrative cluster work, all the while using the
language of the WEB.
Features
========
* Drop-in replacement both on pre-existing Schema Registry / Kafka Rest Proxy client and
server-sides
* Moderate memory consumption
* Asynchronous architecture based on aiohttp
* Supports Avro, JSON Schema, and Protobuf
* Leader/Replica architecture for HA and load balancing
* Observability with Metrics and OpenTelemetry
* Based on aiokafka (rdkafka) kafka library mostly
* Schema registry based on FastApi
Compatibility details
---------------------
Karapace is compatible with Schema Registry 6.1.1 on API level. When a new version of SR is released, the goal is
to support it in a reasonable time. Karapace supports all operations in the API.
There are some caveats regarding the schema normalization, and the error messages being the same as in Schema Registry, which
cannot be always fully guaranteed.
Setup
=====
Using Docker
------------
To get you up and running with the latest build of Karapace, a docker image is available::
# Fetch the latest build from main branch
docker pull ghcr.io/aiven-open/karapace:develop
# Fetch the latest release
docker pull ghcr.io/aiven-open/karapace:latest
Versions `3.7.1` and earlier are available from the `ghcr.io/aiven` registry::
docker pull ghcr.io/aiven/karapace:3.7.1
An example setup including configuration and Kafka connection is available as compose example::
docker compose -f ./container/compose.yml up -d
Then you should be able to reach two sets of endpoints:
* Karapace schema registry on http://localhost:8081
* Karapace REST on http://localhost:8082
For local development with Docker, use the convenient make commands (see `Development`_ section below).
Configuration
^^^^^^^^^^^^^
Each configuration key can be overridden with an environment variable prefixed with ``KARAPACE_``,
exception being configuration keys that actually start with the ``karapace`` string. For example, to
override the ``bootstrap_uri`` config value, one would use the environment variable
``KARAPACE_BOOTSTRAP_URI``. Here_ you can find an example configuration file to give you an idea
what you need to change.
.. _`Here`: https://github.com/Aiven-Open/karapace/blob/master/karapace.config.json
Using Sources
-------------
Install
^^^^^^^
You can do a source install using::
pip install .
Troubleshooting notes :
- An updated version of wheel (https://pypi.org/project/wheel/) is required.
- An updated version of 'go' and 'rust' is required
- Create and activate virtual environment (venv) to manage dependencies
Run
^^^
- Make sure kafka is running.
Start Karapace. This should start karapace on http://localhost:8081 ::
python -m karapace
Verify in browser http://localhost:8081/subjects should return an array of subjects if exist or an empty array.
or with curl ::
$ curl -X GET http://localhost:8081/subjects
To enable oidc authentication on the karapace, configure oidc jwks url config details ::
sasl_oauthbearer_authentication_enabled: bool = True
sasl_oauthbearer_jwks_endpoint_url = "https://idp.example.com/realms/karapace/protocol/openid-connect/certs",
sasl_oauthbearer_expected_issuer = "https://idp.example.com/realms/karapace",
sasl_oauthbearer_expected_audience = "karapace-audience",
sasl_oauthbearer_sub_claim_name = "sub",
When ``sasl_oauthbearer_authentication_enabled`` is true, every request must carry a valid
Bearer token. The JWT signature, expiry and the configured subject claim are always verified
via JWKS. ``sasl_oauthbearer_expected_issuer`` and ``sasl_oauthbearer_expected_audience`` are
optional, as in Kafka's ``sasl.oauthbearer.expected.*``: leaving one unset skips that claim
check. Leaving the audience unset means any token the provider issues for any client is
accepted, so set it whenever the provider can mint a usable ``aud``. Authorization (role-based
access) is opt-in on top of this with ``sasl_oauthbearer_authorization_enabled``.
Optional hardening flags::
sasl_oauthbearer_leeway_seconds: int = 0 # clock-skew tolerance for exp/nbf/iat (must be >= 0)
sasl_oauthbearer_require_access_token_typ: bool = False # require typ "at+jwt" header
There is a detailed section about OAuth2 authentication for karapace below.
To enable oidc authorization on karapace, configure the below params together with the above ::
sasl_oauthbearer_authorization_enabled: bool = False
sasl_oauthbearer_roles_claim_path: str | None = None
# Defaults to the karapace.* roles shown in the example below; override per deployment.
sasl_oauthbearer_method_roles: dict[str, list[str]]
Note: ``sasl_oauthbearer_authorization_enabled=true`` requires
``sasl_oauthbearer_authentication_enabled=true``. For backwards compatibility, enabling
authorization without explicitly enabling authentication will auto-enable authentication and
log a deprecation warning — set both flags explicitly.
There is a detailed section about OAuth2 authorization for karapace below.
Start Karapace rest proxy. This shout start karapace on http://localhost:8082 ::
python -m karapace.kafka_rest_apis
To enable authorization & authentication on the rest proxy, configure 'sasl_mechanism' in the config with values like PLAIN/OAUTHBEARER ::
sasl_mechanism = "OAUTHBEARER",
sasl_oauth_token_provider = token_provider,
security_protocol="SASL_SSL",
ssl_cafile="ca.pem",
If 'sasl_mechanism' is configured to PLAIN::
sasl_mechanism = "PLAIN",
security_protocol = "SASL_PLAINTEXT",
sasl_plain_username = "your_username",
sasl_plain_password = "your_password"
There is a detailed section about OAuth2 authentication for rest proxy below.
Verify with list topics::
$ curl "http://localhost:8082/topics"
Schema Registry Api reference
=============================
To register the first version of a schema under the subject "test" using Avro schema::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"}' \
http://localhost:8081/subjects/test-key/versions
{"id":1}
To register a version of a schema using JSON Schema, one needs to use `schemaType` property::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schemaType": "JSON", "schema": "{\"type\": \"object\",\"properties\":{\"age\":{\"type\": \"number\"}},\"additionalProperties\":true}"}' \
http://localhost:8081/subjects/test-key-json-schema/versions
{"id":2}
To list all subjects (including the one created just above)::
$ curl -X GET http://localhost:8081/subjects
["test-key"]
To list all the versions of a given schema (including the one just created above)::
$ curl -X GET http://localhost:8081/subjects/test-key/versions
[1]
To fetch back the schema whose global id is 1 (i.e. the one registered above)::
$ curl -X GET http://localhost:8081/schemas/ids/1
{"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}
To get the specific version 1 of the schema just registered run::
$ curl -X GET http://localhost:8081/subjects/test-key/versions/1
{"subject":"test-key","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}
To get the latest version of the schema under subject test-key run::
$ curl -X GET http://localhost:8081/subjects/test-key/versions/latest
{"subject":"test-key","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}
In order to delete version 10 of the schema registered under subject "test-key" (if it exists)::
$ curl -X DELETE http://localhost:8081/subjects/test-key/versions/10
10
To Delete all versions of the schema registered under subject "test-key"::
$ curl -X DELETE http://localhost:8081/subjects/test-key
[1]
Test the compatibility of a schema with the latest schema under subject "test-key"::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"int\"}"}' \
http://localhost:8081/compatibility/subjects/test-key/versions/latest
{"is_compatible":true}
NOTE: if the subject's compatibility mode is transitive (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE or FULL_TRANSITIVE) then the
compatibility is checked not only against the latest schema, but also against all previous schemas, as it would be done
when trying to register the new schema through the `subjects/<subject-key>/versions` endpoint.
Get current global backwards compatibility setting value::
$ curl -X GET http://localhost:8081/config
{"compatibilityLevel":"BACKWARD"}
Change compatibility requirements for all subjects where it's not
specifically defined otherwise::
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"compatibility": "NONE"}' http://localhost:8081/config
{"compatibility":"NONE"}
Change compatibility requirement to FULL for the test-key subject::
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"compatibility": "FULL"}' http://localhost:8081/config/test-key
{"compatibility":"FULL"}
Get the current global mode. Karapace supports ``READWRITE`` (the default) and ``IMPORT``::
$ curl -X GET http://localhost:8081/mode
{"mode":"READWRITE"}
Switch the test-key subject to IMPORT mode, which allows schemas to be registered with
their original schema ids and version numbers. The subject must be empty, unless
``?force=true`` is given::
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"mode": "IMPORT"}' http://localhost:8081/mode/test-key
{"mode":"IMPORT"}
Register a schema with an explicit id and version, which is only accepted in IMPORT mode::
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\": \"string\"}", "id": 1001, "version": 5}' \
http://localhost:8081/subjects/test-key/versions
{"id":1001}
Read the effective mode of a subject. Add ``?defaultToGlobal=true`` to get the global mode
instead of a 404 for an unknown subject::
$ curl -X GET http://localhost:8081/mode/test-key
{"mode":"IMPORT"}
Remove the subject level mode so the subject follows the global mode again::
$ curl -X DELETE http://localhost:8081/mode/test-key
{"mode":"READWRITE"}
IMPORT mode skips compatibility checks and content deduplication, schema ids are global and
immutable once bound, and a failed import is resumed by replaying it rather than rolled
back. See the `Import mode <https://www.karapace.io/docs/import-mode>`_ documentation for
the full constraints and recovery steps.
Schema Registry Rest proxy Api reference
========================================
List topics::
$ curl "http://localhost:8082/topics"
Get info for one particular topic::
$ curl "http://localhost:8082/topics/my_topic"
Produce a message backed up by schema registry::
$ curl -H "Content-Type: application/vnd.kafka.avro.v2+json" -X POST -d \
'{"value_schema": "{\"namespace\": \"example.avro\", \"type\": \"record\", \"name\": \"simple\", \"fields\": \
[{\"name\": \"name\", \"type\": \"string\"}]}", "records": [{"value": {"name": "name0"}}]}' http://localhost:8082/topics/my_topic
A record may carry optional Kafka message ``headers``, a list of ``{"name": <string>, "value": <base64 string | null>}`` objects (header values are raw bytes, so they are base64-encoded)::
$ curl -H "Content-Type: application/vnd.kafka.json.v2+json" -X POST -d \
'{"records": [{"value": {"name": "name0"}, "headers": [{"name": "traceId", "value": "YWJjMTIz"}]}]}' \
http://localhost:8082/topics/my_topic
Create a consumer with consumer group 'avro_consumers' and consumer instance 'my_consumer' ::
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" -H "Accept: application/vnd.kafka.v2+json" \
--data '{"name": "my_consumer", "format": "avro", "auto.offset.reset": "earliest"}' \
http://localhost:8082/consumers/avro_consumers
Subscribe to the topic we previously created ::
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["my_topic"]}' \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer/subscription
Consume previously produced message::
$ curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer/records?timeout=1000
When a consumed record has headers, they are returned in a ``headers`` field using the same ``{"name": ..., "value": <base64 string | null>}`` form (the field is omitted for records without headers).
Commit offsets for a particular topic partition::
$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{}' \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer/offsets
Delete consumer::
$ curl -X DELETE -H "Accept: application/vnd.kafka.v2+json" \
http://localhost:8082/consumers/avro_consumers/instances/my_consumer
Backing up your Karapace
========================
.. note::
The built-in backup functionality (``karapace_schema_backup``) has been removed
as of version 6.0.0. Users who need this functionality can continue to use
Karapace versions prior to 6.0.0, however it will no longer be maintained
within the project.
Performance comparison to Confluent stack
==========================================
Latency
-------
* 50 concurrent connections, 50.000 requests
====== ========== ===========
Format Karapace Confluent
====== ========== ===========
Avro 80.95 7.22
Binary 66.32 46.99
Json 60.36 53.7
====== ========== ===========
* 15 concurrent connections, 50.000 requests
====== =========== ===========
Format Karapace Confluent
====== =========== ===========
Avro 25.05 18.14
Binary 21.35 15.85
Json 21.38 14.83
====== =========== ===========
* 4 concurrent connections, 50.000 requests
====== =========== ===========
Format Karapace Confluent
====== =========== ===========
Avro 6.54 5.67
Binary 6.51 4.56
Json 6.86 5.32
====== =========== ===========
Also, it appears there is quite a bit of variation on subsequent runs, especially for the lower numbers, so once
more exact measurements are required, it's advised we increase the total req count to something like 500K
We'll focus on Avro serialization only after this round, as it's the more expensive one, plus it tests the entire stack
Consuming RAM
-------------
A basic push pull test , with 12 connections on the publisher process and 3 connections on the subscriber process, with a
10 minute duration. The publisher has the 100 ms timeout and 100 max_bytes parameters set on each request so both processes have work to do
Heap size limit is set to 256M on Rest proxy
Ram consumption, different consumer count, over 300s
=========== =================== ================
Consumers Karapace combined Confluent rest
=========== =================== ================
1 47 200
10 55 400
20 83 530
=========== =================== ================
Commands
========
Once installed, the ``karapace`` program should be in your path. It is the
main daemon process that should be run under a service manager such as
``systemd`` to serve clients.
Configuration keys
==================
Keys to take special care are the ones needed to configure Kafka and advertised_hostname.
.. list-table::
:header-rows: 1
* - Parameter
- Default Value
- Description
* - ``advertised_hostname``
- ``socket.gethostname()``
- The hostname being advertised to other instances of Karapace that are attached to the same Kafka group. All nodes within the cluster need to have their ``advertised_hostname``'s set so that they can all reach each other.
* - ``advertised_port``
- ``None``
- The port being advertised to other instances of Karapace that are attached to the same Kafka group. Fallbacks to ``port`` if not set.
* - ``advertised_protocol``
- ``http``
- The protocol being advertised to other instances of Karapace that are attached to the same Kafka group.
* - ``bootstrap_uri``
- ``localhost:9092``
- The URI to the Kafka service where to store the schemas and to run
coordination among the Karapace instances.
* - ``client_id``
- ``sr-1``
- The ``client_id`` Karapace will use when coordinating with
other Karapace instances. The instance with the ID that sorts
first alphabetically is chosen as master from the services with
master_eligibility set to true.
* - ``consumer_enable_autocommit``
- ``True``
- Enable auto commit on rest proxy consumers
* - ``consumer_idle_disconnect_timeout``
- ``0``
- Disconnect idle consumers after timeout seconds if not used. Inactivity leads to consumer leaving consumer group and consumer state. 0 (default) means no auto-disconnect.
* - ``consumer_request_max_bytes``
- ``67108864``
- Rest proxy consumers maximum bytes to be fetched per request
* - ``consumer_request_timeout_ms``
- ``11000``
- Rest proxy consumers timeout for reads that do not limit the max bytes or provide their own timeout
* - ``fetch_min_bytes``
- ``1``
- Rest proxy consumers minimum bytes to be fetched per request.
* - ``group_id``
- ``schema-registry``
- The Kafka group name used for selecting a master service to coordinate the storing of Schemas.
* - ``host``
- ``127.0.0.1``
- Listening host for the Karapace server. Use an empty string to
listen to all available networks.
* - ``http_request_max_size``
- ``1048576``
- The maximum client HTTP request size. This value controls how large (POST) payloads are allowed. When configuration of ``karapace_rest`` is set to `true` and ``http_request_max_size`` is not set, Karapace configuration adapts the allowed client max size from the ``producer_max_request_size``. In cases where automatically selected size is not enough the configuration can be overridden by setting a value in configuration. For schema registry operation set the client max size according to expected size of schema payloads if default size is not enough.
* - ``kafka_retriable_errors_silenced``
- ``true``
- If enabled, kafka errors which can be retried or custom errors specified for the service will not be raised,
instead, a warning log is emitted. This will denoise issue tracking systems, i.e. sentry
* - ``kafka_schema_reader_strict_mode``
- ``false``
- If enabled, causes the Karapace schema-registry service to shutdown when there are invalid schema records in the `_schemas` topic
* - ``karapace_registry``
- ``true``
- If the registry part of the app should be included in the starting process
At least one of this and ``karapace_rest`` options need to be enabled in order
for the service to start
* - ``karapace_rest``
- ``true``
- If the rest part of the app should be included in the starting process
At least one of this and ``karapace_registry`` options need to be enabled in order
for the service to start
* - ``librdkafka_debug``
- ``null``
- librdkafka ``debug`` contexts applied to every confluent-kafka client of both the schema-registry and the REST proxy
(consumers, producers, admin clients), e.g. ``broker,metadata,topic``. See the
`librdkafka configuration reference <https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md>`_
for the available contexts, an unknown context is rejected at startup. The debug messages are written to stderr. Meant for troubleshooting only, as it is verbose.
* - ``log_format``
- ``%(name)-20s\t%(threadName)s\t%(levelname)-8s\t%(message)s``
- Log format
* - ``log_handler``
- ``stdout``
- Select the log handler. Default is standard output. Alternative log handler is ``systemd``.
* - ``log_level``
- ``DEBUG``
- Logging level. Default level is debug.
* - ``master_election_strategy``
- ``lowest``
- Decides on what basis the Karapace cluster master is chosen (only relevant in a multi node setup)
* - ``master_eligibility``
- ``true``
- Should the service instance be considered for promotion to the master
service. One reason to turn this off would be to have an instance of Karapace
running somewhere else for HA purposes but which you wouldn't want to
automatically promote to master if the primary instances become
unavailable.
* - ``metadata_max_age_ms``
- ``60000``
- Period of time in milliseconds after Kafka metadata is force refreshed.
* - ``name_strategy``
- ``topic_name``
- Name strategy to use when storing schemas from the kafka rest proxy service. You can opt between ``topic_name`` , ``record_name`` and ``topic_record_name``
* - ``name_strategy_validation``
- ``true``
- If enabled, validate that given schema is registered under used name strategy when producing messages from Kafka Rest
* - ``port``
- ``8081``
- Listening port for the Karapace server.
* - ``producer_acks``
- ``1``
- Level of consistency desired by each producer message sent on the rest proxy.
More on `Kafka Producer <https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html>`__
* - ``producer_compression_type``
- ``None``
- Type of compression to be used by rest proxy producers
* - ``producer_linger_ms``
- ``0``
- Time to wait for grouping together requests.
More on `Kafka Producer <https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html>`__
* - ``producer_max_request_size``
- ``1048576``
- The maximum size of a request in bytes.
More on `Kafka Producer configs <https://kafka.apache.org/documentation/#producerconfigs_max.request.size>`_
* - ``protobuf_runtime_directory``
- ``runtime``
- Runtime directory for the ``protoc`` protobuf schema parser and code generator
* - ``registry_authfile``
- ``/path/to/authfile.json``
- Filename to specify users and access control rules for Karapace Schema Registry.
If this is set, Schema Registry requires authentication for most of the endpoints and applies per endpoint authorization rules.
* - ``registry_ca``
- ``/path/to/cafile``
- Kafka Registry CA certificate, used by Kafka Rest for Avro related requests.
If this is set, Kafka Rest will use HTTPS to connect to the registry.
If running both in the same process, it should be left to its default value
* - ``registry_host``
- ``127.0.0.1``
- Schema Registry host, used by Kafka Rest for schema related requests.
If running both in the same process, it should be left to its default value
* - ``registry_password``
- ``None``
- Schema Registry password for authentication, used by Kafka Rest for schema related requests.
A client ``Authorization`` header forwarded to the registry takes precedence; this is the fallback when none is forwarded.
* - ``registry_port``
- ``8081``
- Schema Registry port, used by Kafka Rest for schema related requests.
If running both in the same process, it should be left to its default value
* - ``registry_scheme``
- ``http``
- Schema Registry scheme to use for rest-proxy, http | https (if certificates are provided).
* - ``registry_user``
- ``None``
- Schema Registry user for authentication, used by Kafka Rest for schema related requests.
A client ``Authorization`` header forwarded to the registry takes precedence; this is the fallback when none is forwarded.
* - ``replication_factor``
- ``1``
- The replication factor to be used with the schema topic.
* - ``rest_authorization``
- ``false``
- Use REST API's calling authorization credentials to invoke Kafka operations over SASL authentication of ``sasl_bootstrap_uri`` to delegate REST proxy authorization to Kafka. If false, then use configured common credentials for all Kafka connections of REST proxy operations.
* - ``rest_base_uri``
- ``None``
- Publicly available URI of this instance advertised to the clients using stateful operations such as creating consumers. If not set, then construct URI using ``advertised_protocol``, ``advertised_hostname``, and ``advertised_port``.
* - ``sasl_bootstrap_uri``
- ``None``
- The URI to the Kafka service to use with the Kafka REST API when SASL authorization with REST is used.
* - ``sasl_mechanism``
- ``None``
- SASL mechanism for Kafka authentication (e.g. ``PLAIN``, ``SCRAM-SHA-512``, ``OAUTHBEARER``). Used together with ``security_protocol``.
* - ``sasl_plain_password``
- ``None``
- Password for Kafka SASL ``PLAIN`` authentication.
* - ``sasl_plain_username``
- ``None``
- Username for Kafka SASL ``PLAIN`` authentication.
* - ``security_protocol``
- ``PLAINTEXT``
- Default Kafka security protocol needed to communicate with the Kafka
cluster. Other options is to use SSL for SSL client certificate
authentication, or SASL_PLAINTEXT / SASL_SSL for simple authentication.
* - ``sentry``
- ``None``
- Used to configure parameters for sentry integration (dsn, tags, ...). Setting the
environment variable ``SENTRY_DSN`` will also enable sentry integration.
* - ``server_tls_cafile``
- ``/path/to/cafile``
- Filename to the SSL CA certificate.
* - ``server_tls_certfile``
- ``/path/to/certfile``
- Filename to a certificate chain for the Karapace server in HTTPS mode.
* - ``server_tls_client_auth``
- ``none``
- Client certificate requirement: ``none``, ``optional``, or ``required``.
* - ``server_tls_keyfile``
- ``/path/to/keyfile``
- Filename to a private key for the Karapace server in HTTPS mode.
* - ``ssl_cafile``
- ``/path/to/cafile``
- Used when ``security_protocol`` is set to SSL, the path to the SSL CA certificate.
* - ``ssl_certfile``
- ``/path/to/certfile``
- Used when ``security_protocol`` is set to SSL, the path to the SSL certfile.
* - ``ssl_keyfile``
- ``/path/to/keyfile``
- Used when ``security_protocol`` is set to SSL, the path to the SSL keyfile.
* - ``topic_name``
- ``_schemas``
- The name of the Kafka topic where to store the schemas.
* - ``allow_duplicate_schema_ids``
- ``true``
- Whether one schema may be imported under more than one schema id. Set to ``false`` to refuse an import whose content is already
registered under a different id, keeping ids one to one with content.
* - ``mode_mutability``
- ``true``
- Whether mode changes are allowed. Set to ``false`` to reject every ``PUT``/``DELETE`` on ``/mode``, so ``IMPORT`` mode cannot be
entered at runtime.
* - ``use_protobuf_formatter``
- ``false``
- If protobuf formatter should be used on protobuf schemas in order to normalize schemas. The formatter is used on top and independent of regular normalization and schemas will be persisted in a formatted state.
* - ``waiting_time_before_acting_as_master_ms``
- ``5000``
- The time that a master wait before becoming an active master if at the previous round of election wasn't the master (in that case the waiting time its skipped).
Should be an upper bound of the time required for a master to write a message in the kafka topic + the time required from a node in the cluster to consume the
Log of messages. If the value its too low there is the risk under high load of producing different schemas with the ID.
Authentication and authorization of Karapace Schema Registry REST API
=====================================================================
To enable HTTP Basic Authentication and user authorization the authorization configuration file is set in the main configuration key ``registry_authfile`` of the Karapace.
Karapace Schema Registry authorization file is an optional JSON configuration, which contains a list of authorized users in ``users`` and a list of access control rules in ``permissions``.
Each user entry contains following attributes:
.. list-table::
:header-rows: 1
* - Parameter
- Description
* - ``username``
- A string
* - ``algorithm``
- One of supported hashing algorithms, ``scrypt``, ``sha1``, ``sha256``, or ``sha512``
* - ``salt``
- Salt used for hashing the password
* - ``password_hash``
- Hash string of the password calculated using given algorithm and salt.
Password hashing can be done using ``karapace_mkpasswd`` tool, if installed, or by invoking directly with ``python -m karapace.core.auth``. The tool generates JSON entry with these fields. ::
$ karapace_mkpasswd -u user -a sha512 secret
{
"username": "user",
"algorithm": "sha512",
"salt": "iuLouaExTeg9ypqTxqP-dw",
"password_hash": "R6ghYSXdLGsq6hkQcg8wT4xkD4QToxBhlp7NerTnyB077M+mD2qiN7ZxXCDb4aE+5lExu5P11UpMPYAcVYxSQA=="
}
Each access control rule contains following attributes:
.. list-table::
:header-rows: 1
* - Parameter
- Description
* - ``username``
- A string to match against authenticated user
* - ``operation``
- Exact value of ``Read`` or ``Write``. Write implies also read permissions. Write includes all mutable operations, e.g. deleting schema versions
* - ``resource``
- A regular expression used to match against accessed resource.
Supported resource authorization:
.. list-table::
:header-rows: 1
* - Resource
- Description
* - ``Config:``
- Controls authorization to global schema registry configuration.
* - ``Subject:<subject_name>``
- Controls authorization to subject. The ``<subject_name>`` is a regular expression to match against the accessed subject.
Example of complete authorization file
--------------------------------------
::
{
"users": [
{
"username": "admin",
"algorithm": "scrypt",
"salt": "<put salt for randomized hashing here>",
"password_hash": "<put hashed password here>"
},
{
"username": "plainuser",
"algorithm": "sha256",
"salt": "<put salt for randomized hashing here>",
"password_hash": "<put hashed password here>"
}
],
"permissions": [
{
"username": "admin",
"operation": "Write",
"resource": ".*"
},
{
"username": "plainuser",
"operation": "Read",
"resource": "Subject:general.*"
},
{
"username": "plainuser",
"operation": "Read",
"resource": "Config:"
}
]
}
Karapace Schema Registry access to the _schemas topic
=====================================================
The principal used by the Karapace Schema Registry has to have adequate access to the _schemas topic (see the ``topic_name`` configuration option above).
In addition to what is required to access the topic, as described in the Confluent Schema Registry documentation_, the unique, single-member consumer group
used by consumers in the schema registry needs ``Describe`` and ``Read`` permissions_ on the group.
These unique (per instance of the schema registry) consumer group names are prefixed by ``karapace-autogenerated-``, followed by a random string.
.. _`documentation`: https://docs.confluent.io/platform/current/schema-registry/security/index.html#authorizing-access-to-the-schemas-topic
.. _`permissions`: https://docs.confluent.io/platform/current/kafka/authorization.html#group-resource-type-operations
OAuth2 authentication and authorization of Karapace
===================================================
Karapace supports OAuth2 authentication and authorization. The JSON Web Token (JWT) is extracted from the ``Authorization`` HTTP header if the authorization scheme is ``Bearer``,
eg. ``Authorization: Bearer $JWT``. If a ``Bearer`` token is present in schema registry requests, karapace will validate the token against OpenId connect provider and continue.
Below here is an example of karapace OpenId connect config ::
sasl_oauthbearer_authentication_enabled: bool = True
sasl_oauthbearer_jwks_endpoint_url = "http://localhost:8383/realms/karapace/protocol/openid-connect/certs",
sasl_oauthbearer_expected_issuer = "http://localhost:8383/realms/karapace",
sasl_oauthbearer_expected_audience = "account",
sasl_oauthbearer_sub_claim_name = "sub",
``sasl_oauthbearer_expected_issuer`` and ``sasl_oauthbearer_expected_audience`` are optional; an
unset value skips verification of that claim. ``sasl_oauthbearer_expected_audience`` accepts a
comma-separated list and a token matching any entry is accepted. With Keycloak, the default
access token carries ``aud: account``, so add an audience mapper to the client if you want to
pin the audience to the client itself.
For authorization (requires authentication to be enabled)::
sasl_oauthbearer_authentication_enabled: bool = True
sasl_oauthbearer_authorization_enabled: bool = False
sasl_oauthbearer_roles_claim_path: str | None = None
# Defaults to the karapace.* roles shown in the example below; override per deployment.
sasl_oauthbearer_method_roles: dict[str, list[str]]
Below here is an example of karapace OpenId connect docker config ::
KARAPACE_SASL_OAUTHBEARER_AUTHENTICATION_ENABLED: True
KARAPACE_SASL_OAUTHBEARER_JWKS_ENDPOINT_URL: http://keycloak:8080/realms/karapace/protocol/openid-connect/certs
KARAPACE_SASL_OAUTHBEARER_EXPECTED_ISSUER: http://keycloak:8080/realms/karapace
KARAPACE_SASL_OAUTHBEARER_EXPECTED_AUDIENCE: "account"
KARAPACE_SASL_OAUTHBEARER_SUB_CLAIM_NAME: sub
For authorization, example config::
KARAPACE_SASL_OAUTHBEARER_AUTHENTICATION_ENABLED: True
KARAPACE_SASL_OAUTHBEARER_AUTHORIZATION_ENABLED: True
KARAPACE_SASL_OAUTHBEARER_ROLES_CLAIM_PATH: "resource_access.karapace-client.roles"
KARAPACE_SASL_OAUTHBEARER_METHOD_ROLES: dict[str, list[str]] = {"GET": ["karapace.schema:read", "karapace.subject:read"],
"POST": ["karapace.schema:write", "karapace.subject:write"], "PUT": [], "DELETE": []}
.. note::
``sasl_oauthbearer_client_id`` and ``sasl_oauthbearer_enforce_azp`` have been removed. The
``azp`` claim is no longer validated. The ``[client_id]`` placeholder in
``sasl_oauthbearer_roles_claim_path`` is no longer substituted — write the literal client id
into the path (e.g. ``resource_access.karapace-client.roles``).
Production hardening notes
--------------------------
* ``sasl_oauthbearer_jwks_endpoint_url`` must use ``https://``; startup fails otherwise.
The ``sasl_oauthbearer_allow_insecure_jwks`` flag overrides this for dev/test only.
* ``/docs``, ``/redoc``, ``/openapi.json`` bypass the auth gate (by design, for Swagger UI).
Block them at the reverse proxy in production to avoid leaking the API surface.
Testing the authentication and authorization with docker
--------------------------------------------------------
Get token
---------
If you are running with docker, login to karapace container with ::
docker exec -it containerid /bin/sh
and get a token like below. ::
curl -s -X POST "http://keycloak:8080/realms/karapace/protocol/openid-connect/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials" \
-d "client_id=karapace-client" \
-d "client_secret=client-secret" \
Note : client id and client secret can be retrieved from Oidc provider
Response of the above curl should be a access token, and other scope and expiry details.
Export the token into ACCESS_TOKEN variable::
export ACCESS_TOKEN='token..'
Access schema registry
----------------------
Send the token to get subjects url::
curl --insecure -H "Authorization: Bearer $ACCESS_TOKEN" https://localhost:8081/subjects/testtopic-value/versions
Response should be list of available subjects, as it is a valid token, and GET (schema read, subject read) is allowed as per the defined method roles.
Send an invalid token to get subjects url::
curl --insecure -H "Authorization: Bearer aninvalidtoken" https://localhost:8081/subjects
Response should be {"error":"Unauthorized","reason":"Invalid token/payload"}
OAuth2 authentication and authorization of Karapace REST proxy
===================================================================
The Karapace REST proxy supports passing OAuth2 credentials to the underlying Kafka service (defined in the ``sasl_bootstrap_uri`` configuration parameter). The JSON Web Token (JWT) is extracted from the ``Authorization`` HTTP header if the authorization scheme is ``Bearer``,
eg. ``Authorization: Bearer $JWT``. If a ``Bearer`` token is present, the Kafka clients managed by Karapace will be created to use the SASL ``OAUTHBEARER`` mechanism and the JWT will be passed along. The Karapace REST proxy does not verify the token, that is done by
the underlying Kafka service itself, if it's configured accordingly.
Authorization is also done by Kafka itself, typically using the ``sub`` claim (although it's configurable) from the JWT as the username, checked against the configured ACLs.
OAuth2 and ``Bearer`` token usage is dependent on the ``rest_authorization`` configuration parameter being ``true``.
Forwarding credentials to the Schema Registry
----------------------------------------------
For schema operations (registration and lookup during produce/consume), the REST proxy forwards the inbound
``Authorization`` header to the Schema Registry when ``sasl_oauthbearer_authentication_enabled`` is ``true``.
Both ``Bearer`` (validated by the registry via OIDC) and ``Basic`` (validated against ``registry_authfile``) schemes
are forwarded as-is, so a client can authenticate to the registry with its own token or credentials.
A forwarded ``Authorization`` header takes precedence over the proxy's own ``registry_user`` / ``registry_password``;
those service credentials are used only for requests that arrive without a forwarded header. The two may be configured
together, which lets clients migrate to forwarded tokens one at a time. This forwarding is independent of
``rest_authorization`` (which governs Kafka-side authentication).
When the Schema Registry rejects the credentials or token during a schema operation, the REST proxy reports the failure
with a matching status: ``401`` (error code ``401``) for authentication and ``403`` (error code ``403``) for
authorization. Other schema registration failures keep their existing errors and are not reported as auth failures — an
incompatible or otherwise rejected schema returns ``40801``, and a schema that fails to parse returns ``42205``.
Token expiry
------------
The REST proxy process manages a set of producer and consumer clients, which are identified by the OAuth2 JWT token. These are periodically cleaned up if they are idle, as well as *before* the JWT token expires (the clean up currently runs every 5 minutes).
Before a client refreshes its OAuth2 JWT token, it is expected to remove currently running consumers (eg. after committing their offsets) and producers using the current token.
Schema Normalization
--------------------
If specified as a rest parameter for the POST ``/subjects/{subject}/versions?normalize=true`` endpoint and the POST ``subjects/{subject}?normalize=true`` endpoint,
Karapace uses a schema normalization algorithm to ensure that the schema is stored in a canonical form.
This normalization process is done so that schemas semantically equivalent are stored in the same way and should be considered equal.
Normalization is currently only supported for Protobuf schemas. Karapace does not support all normalization features implemented by Confluent Schema Registry.
Currently the normalization process covers:
- Ordering of optional fields in the schema.
- Restoration of map field shorthand: binary-registered schemas that contain the expanded entry-message
form (a ``repeated`` field pointing to a synthetic nested message with ``option map_entry = true``)
are converted back to ``map<K, V>`` syntax, matching Confluent Schema Registry behaviour.
Use the feature with the assumption that it will be extended in the future and so two schemas that are semantically equivalent could be considered
different by the normalization process in different future versions of Karapace.
The safe choice, when using a normalization process, is always to consider as different two schemas that are semantically equivalent while the problem is when two semantically different schemas are considered equivalent.
In that view the future extension of the normalization process isn't considered a breaking change but rather an extension of the normalization process.
JSON Schema support
===================
Karapace supports registering schemas of type ``JSON`` (set ``"schemaType": "JSON"`` when registering). This section
describes which JSON Schema drafts are supported, how compatibility is evaluated, current limitations, and guidance for
migrating between drafts.
Supported drafts and ``$schema`` URIs
-------------------------------------
Karapace selects the JSON Schema draft from the ``$schema`` keyword and validates the schema against that draft's
metaschema. When ``$schema`` is absent or its value is not recognized, the schema is validated as **Draft-7**, which
preserves the previous behaviour and matches the default used by Confluent Schema Registry.
.. list-table::
:header-rows: 1
* - Draft
- ``$schema`` URI
- Status
* - draft-04
- ``http://json-schema.org/draft-04/schema#``
- Validated with its own draft validator
* - draft-06
- ``http://json-schema.org/draft-06/schema#``
- Validated with its own draft validator
* - draft-07
- ``http://json-schema.org/draft-07/schema#``
- Validated with its own draft validator (also the default when ``$schema`` is absent or unrecognized)
* - 2019-09
- ``https://json-schema.org/draft/2019-09/schema``
- Validated with its own draft validator
* - 2020-12
- ``https://json-schema.org/draft/2020-12/schema``
- Validated with its own draft validator
Compatibility checking canonicalizes the renamed newer-draft keywords and rejects the ones it cannot yet evaluate — see
`Compatibility behavior`_ and `Limitations`_ below.
Compatibility behavior
----------------------
Compatibility between schema versions is evaluated using the compatibility mode configured for the subject (or the
global default). The supported modes are ``NONE``, ``BACKWARD``, ``BACKWARD_TRANSITIVE``, ``FORWARD``,
``FORWARD_TRANSITIVE``, ``FULL`` and ``FULL_TRANSITIVE``. Transitive modes check the new schema against *all* previous
versions; non-transitive modes only check against the latest one.
The following keywords are taken into account when comparing versions: ``type``, ``properties``, ``required``,
``additionalProperties``, ``enum``, ``$ref``, the numeric bounds (``minimum``, ``maximum``, ``exclusiveMinimum``,
``exclusiveMaximum``, ``multipleOf``), the string constraints (``minLength``, ``maxLength``, ``pattern``), the array
constraints (``items``, ``additionalItems``, ``minItems``, ``maxItems``, ``uniqueItems``), the object-size constraints
(``minProperties``, ``maxProperties``, ``propertyNames``, ``patternProperties``) and the dependency keywords
(``dependencies``, ``dependentSchemas``).
Draft 2019-09 / 2020-12 keywords that are renames or relocations of Draft-7 keywords are **canonicalized** before
comparison, so they are evaluated correctly and are interchangeable across drafts within a subject:
* ``$defs`` is treated as ``definitions``
* ``prefixItems`` is treated as the array (tuple) form of ``items``; a sibling ``items`` schema is treated as
``additionalItems``
* ``dependentRequired`` is treated as the string-array form of ``dependencies``
Examples of backward-compatible evolutions (a new schema can read data written with the old one):
* Widening a numeric range (lowering ``minimum`` or raising ``maximum``)
* Relaxing string constraints (lowering ``minLength``, raising ``maxLength``, removing ``pattern``)
* Adding options to an ``enum``
Examples of backward-**incompatible** evolutions:
* Narrowing ``type`` (e.g. from ``number`` to ``integer``)
* Adding or decreasing ``maxLength`` / ``maximum``
* Removing options from an ``enum``
* Removing a property from a closed content model (``additionalProperties: false``)
* Tightening ``additionalProperties`` (from ``true`` to ``false``)
* Adding a ``required`` property to a content model that is not open
* Adding a positional ``prefixItems`` entry the previous version did not require
Forward compatibility applies the mirror of these rules, and full compatibility requires both directions. Because the
renamed keywords share a single canonical form, a subject can evolve across drafts (e.g. a Draft-7 version followed by a
2020-12 version) and compatibility is evaluated on that canonical form.
Limitations
-----------
Schema **validation** is draft-specific: a schema is validated against the draft declared in ``$schema`` (defaulting to
Draft-7), so newer-draft keywords are validated with their intended semantics.
Some newer-draft keywords cannot yet be evaluated correctly during **compatibility checking**. To avoid returning a
result that looks safe but is not, Karapace **rejects the compatibility check** (fails closed) when either the previous
or the new version uses any of: ``unevaluatedProperties``, ``unevaluatedItems``, ``$dynamicRef``, ``$dynamicAnchor``,
``$recursiveRef``, ``$recursiveAnchor`` or ``$vocabulary``. The registration is rejected with an error naming the
keyword, rather than accepted and later surprising a producer or consumer at runtime.
Such schemas can still be registered under a subject whose compatibility is set to ``NONE`` (no compatibility check is
performed). Evaluating these keywords correctly requires resolving dynamic references and reasoning across subschemas,
which the compatibility checker does not do yet; until it does, they remain fail-closed rather than being compared
under an approximation that could report a breaking change as safe.
There is no strict/lenient mode for JSON Schema validation or compatibility. (The ``kafka_schema_reader_strict_mode``
configuration is unrelated — it controls error handling when reading the internal ``_schemas`` topic, not JSON Schema
validation.)
Migration guidance
------------------
* **Validation:** schemas declaring Draft 2019-09 or 2020-12 (or draft-04 / draft-06) via ``$schema`` are validated with
that draft's rules; omitting ``$schema`` defaults to Draft-7.
* **Compatibility:** the renamed/relocated keywords (``$defs``, ``prefixItems``, ``dependentRequired``) are evaluated
correctly and are interchangeable with their Draft-7 spelling, so you can migrate a subject from Draft-7 to
2019-09 / 2020-12 without a compatibility break as long as the schema shape is otherwise unchanged.
* **Unsupported keywords fail closed:** if a version uses ``unevaluatedProperties`` / ``unevaluatedItems``,
``$dynamicRef`` / ``$dynamicAnchor``, ``$recursiveRef`` / ``$recursiveAnchor`` or ``$vocabulary``, the compatibility
check is rejected. Use compatibility ``NONE`` for that subject if you need to register such schemas.
Uninstall
=========
To uninstall Karapace from the system, you can follow the instructions described below. We would love to hear your reasons for uninstalling though. Please file an issue if you experience any problems or email us_ with feedback
.. _`us`: mailto:[email protected]
Installed via Docker
--------------------
If you installed Karapace via Docker, you would need to first stop and remove the images like described:
First obtain the container IDs related to Karapace, you should have one for the registry itself and another one for the rest interface::
docker ps | grep karapace
After this, you can stop each of the containers with::
docker stop <CONTAINER_ID>
If you don't need or want to have the Karapace images around you can now proceed to delete them using::
docker rm <CONTAINER_ID>
Installed from Sources
----------------------
Karapace is installed ``pip install .``, it can be uninstalled with the following ``pip`` command::
pip uninstall karapace
Development
===========
For local development and testing with Docker, use the convenience make commands.
These handle all the setup automatically and ensure consistency across development environments.
Quick Start
-----------
**Set `.env`:** ``cp .env.example .env``
Edit ``.env`` to customize:
- ``PYTHON_VERSION`` - Python version for Docker builds (default: 3.12)
- ``KARAPACE_VERSION`` - Karapace version (default: 5.0.3)
- ``DOCKER_COMPOSE`` - Docker compose command (default: docker-compose)
- ``RUNNER_UID`` / ``RUNNER_GID`` - Container user IDs (check .env.example on how to get)
- ``PYTEST_ARGS`` - Pytest arguments for test runs
Available Commands
------------------
**Container Management:**
- ``make cli`` - Open interactive shell inside Karapace container
- ``make start-karapace-docker-resources`` - Start all Docker containers (Kafka, Schema Registry, REST Proxy)
- ``make stop-karapace-docker-resources`` - Stop all Docker containers
**Testing:**
- ``make unit-tests-in-docker`` - Run unit tests in Docker
- ``make integration-tests-in-docker`` - Run integration tests in Docker
- ``make e2e-tests-in-docker`` - Run end-to-end tests in Docker
- ``make smoke-test-schema-registry`` - Run schema registry smoke tests
- ``make smoke-test-rest-proxy`` - Run REST proxy smoke tests
- ``make type-check-mypy-in-docker`` - Run type checking in Docker
For local development and testing without Docker, see the instructions below.
Execute ``make`` (GNU, usually ``gmake`` on BSD and Mac) to set up a ``venv``
and install the required software for development. Use ``make unit-tests`` and
``make integration-tests`` to execute the respective test suite, or simply
``make test`` to execute both. You can set ``PYTEST_ARGS`` to customize the
execution (e.g. ``PYTEST_ARGS=--maxfail=1 make test``).
Karapace currently depends on various system software to be installed. The
installation of these is automated for some operation systems, but not all. At
the time of writing Java, the Protobuf Compiler, and the Snappy shared library
are required to work with Karapace. You need to install them manually if your
operating system is not supported by the automatic installation scripts. Note
that the scripts are going to ask before installing any of these on your system.
Note that Karapace requires a Protobuf Compiler older than 3.20.0, because
3.20.0 introduces various breaking changes. The tests are going to fail if the
Protobuf Compiler is newer than that. However, you can work around this locally
by running ``pip install --upgrade protobuf`` in your venv. We are going to fix
this soon.
Note that the integration tests are currently not working on Mac. You can use
Docker, just be sure to set ``VENV_DIR`` to a directory outside the working
directory so that the container is not overwriting files from the host (e.g.
``docker run --env VENV_DIR=/tmp/venv ...``).
Note that the ``runtime`` directory **MUST** exist and that Karapace is going to
fail if it does not. The ``runtime`` directory is also not cleaned between test
runs, and left over data might result in failing tests. Use the ``make`` test
targets that correctly clean the ``runtime`` directory without deleting it, but
keep this in mind whenever you are not using ``make`` (e.g. running tests from
your IDE).
Use ``pipx`` or ``brew`` to install ``pre-commit`` and use the global installation,
there is also no dependency on it.
License
=======
Karapace is licensed under the Apache license, version 2.0. Full license text is
available in the ``LICENSE`` file.
Please note that the project explicitly does not require a CLA (Contributor
License Agreement) from its contributors.
Contact
=======
Bug reports and patches are very welcome, please post them as GitHub issues
and pull requests at https://github.com/Aiven-Open/karapace . Any possible
vulnerabilities or other serious issues should be reported directly to the
maintainers <[email protected]>.
Trademark
=========
Apache Kafka is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries. Kafka Rest and Schema Registry are trademarks and property of their respective owners. All product and service names used in this page are for identification purposes only and do not imply endorsement.
Credits
=======
Karapace was created by, and is maintained by, Aiven_ cloud data hub
developers.
The schema storing part of Karapace loans heavily from the ideas of the
earlier Schema Registry implementation by Confluent and thanks are in order
to them for pioneering the concept.
.. _`Aiven`: https://aiven.io/
Recent contributors are listed on the GitHub project page,
https://github.com/Aiven-Open/karapace/graphs/contributors
Copyright ⓒ 2021 Aiven Ltd.
Not written in Markdown, so it's shown here as plain text — view it formatted on GitHub.
2,477 followers · starred Apr 2024
326 followers · starred Apr 2023
42 followers · starred Mar 2023
823 followers · starred Jan 2022