A library to migrate Elasticsearch and OpenSearch mappings. Inspired by Flyway.
Java
93
974 commits
updated Sep 28, 2026
A library to migrate Elasticsearch and OpenSearch mappings. Inspired by Flyway.
Elasticsearch-Evolution executes versioned migration scripts reliably and persists the execution state in an internal Elasticsearch/OpenSearch index. Successfully executed migration scripts will not be executed again!
| Compatibility | Spring Boot | Elasticsearch | OpenSearch |
|---|---|---|---|
| elasticsearch-evolution >= 0.9.0 | 3.3 - 4.0 | 8.x - 9.x | 2.x - 3.x |
| elasticsearch-evolution >= 0.8.0 | 3.2 - 4.0 | 7.5.x - 9.x | 2.x - 3.x |
| elasticsearch-evolution >= 0.7.0 | 3.2 - 3.5 | 7.5.x - 9.x | 2.x - 3.x |
| elasticsearch-evolution >= 0.6.1 | 3.0 - 3.2 | 7.5.x - 8.19.x | 1.x - 2.x |
| elasticsearch-evolution >= 0.4.2 | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0, 3.1, 3.2 | 7.5.x - 8.13.x | 1.x - 2.x |
| elasticsearch-evolution >= 0.4.0 | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 | 7.5.x - 8.6.x | 1.x - 2.x |
| elasticsearch-evolution 0.3.x | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 | 7.5.x - 7.17.x | |
| elasticsearch-evolution 0.2.x | 1.5, 2.0, 2.1, 2.2, 2.3, 2.4 | 7.0.x - 7.4.x, 6.8.x |
First, add the latest version of the Elasticsearch-Evolution Spring Boot starter as a dependency to your Maven pom.xml
and choose an elasticsearch-evolution-rest-abstraction implementation (here: Elasticsearch RestClient implementation):
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>spring-boot-starter-elasticsearch-evolution</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-client</artifactId>
<version>1.0.0</version>
</dependency>
Place your migration scripts in your application classpath under es/migration.
That's it. Elasticsearch-Evolution runs at application startup and expects your Elasticsearch/OpenSearch instance at http://localhost:9200.
First, add the latest version of Elasticsearch-Evolution core as a dependency
and choose an elasticsearch-evolution-rest-abstraction implementation (here: Elasticsearch RestClient implementation):
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-client</artifactId>
<version>1.0.0</version>
</dependency>
Place your migration scripts in your application classpath under es/migration.
Create an ElasticsearchEvolution instance and execute the migration:
// First, create an Elasticsearch RestClient
RestClient restClient = RestClient.builder(HttpHost.create("http://localhost:9200")).build();
// Then, create an EvolutionRestClient abstraction for Elasticsearch
EvolutionRestClient evolutionRestClient = new EvolutionESRestClient(restClient);
// Then, create an ElasticsearchEvolution configuration and instance
ElasticsearchEvolution elasticsearchEvolution = ElasticsearchEvolution.configure()
.load(evolutionRestClient);
// Execute the migration
elasticsearchEvolution.migrate();
If you just want to validate your migration scripts without executing them, you can use the validate() method:
ElasticsearchEvolution elasticsearchEvolution = ...;
// Just validate the migrations
elasticsearchEvolution.validate();
This will validate applied migrations against resolved ones (on the filesystem or classpath) to detect accidental changes that may prevent the schema(s) from being recreated exactly.
Validation fails if:
validateOnMigrate)When validation fails, a ValidateException is thrown.
Elasticsearch-Evolution uses a REST client abstraction (EvolutionRestClient). Currently, these implementations exist:
EvolutionESRestClient
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-client</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionESRest5Client which is designed for Elasticsearch 9.x and Spring Boot 4.x.
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-rest5client</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionOpenSearchRestClient which is designed for OpenSearch 2.x because the RestClientTransport is deprecated for removal since 3.x.
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-os-restclient</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionOpenSearchGenericClient which is designed for OpenSearch 3.x and later.
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-os-genericclient</artifactId>
<version>1.0.0</version>
</dependency>
You can provide your own implementation of EvolutionRestClient if you want to use another HTTP client like Apache HTTPClient or OkHttpClient. This interface is located in:
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionRestClient implementation like this:
@Bean
public EvolutionRestClient customEvolutionRestClient() {
return new MyOkHttpClientEvolutionRestClient(...);
}
An Elasticsearch-Evolution migration script represents a REST call. Here is an example:
PUT /_template/my_template
Content-Type: application/json
{
"index_patterns": [
"my_index_*"
],
"order": 1,
"version": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 20,
"similarity": "boolean"
},
"locked": {
"type": "boolean"
}
}
}
}
The first line defines the HTTP method PUT and the relative path to the Elasticsearch/OpenSearch endpoint /_template/my_template to create a new mapping template.
This is followed by an HTTP header Content-Type: application/json.
After a blank line, the HTTP body is defined.
The pattern is strongly oriented towards ordinary HTTP requests and consists of 4 parts:
GET, HEAD, POST, PUT, DELETE, OPTIONS and PATCH.
The first non-comment line must always start with an HTTP method./my_index_1/_doc/1?refresh=true&op_type=create.:.Elasticsearch-Evolution supports line comments in its migration scripts. Every line starting with # or // will be interpreted as a comment line.
Comment lines are not sent to Elasticsearch/OpenSearch; they will be filtered by Elasticsearch-Evolution.
Elasticsearch-Evolution supports named placeholder substitution. Placeholders are marked in your migration script like this: ${my-placeholder}
placeholderPrefix, which is by default ${ and is configurable.placeholder name, which can be any string but must not contain placeholderPrefix or placeholderSuffix.placeholderSuffix, which is by default } and is configurable.Here is an example filename: V1.0__my-description.http
The filename must follow a pattern:
esMigrationPrefix, which is by default V and is configurable...versionDescriptionSeparator: __.esMigrationSuffixes, which is by default .http and is configurable and case-insensitive.Elasticsearch-Evolution uses the version for ordering your scripts and enforces strictly ordered execution of your scripts by default. Out-of-order execution is supported but disabled by default. Elasticsearch-Evolution interprets the version parts as integers, so each version part must be between 1 (inclusive) and 2,147,483,647 (inclusive).
Here is an example that indicates the ordering: 1.0.1 < 1.1 < 1.2.1 < (2.0.0 == 2).
In this example, version 1.0.1 is the smallest version and is executed first, followed by versions 1.1, 1.2.1, and finally 2.
2 is the same as 2.0 or 2.0.0 - trailing zeros will be trimmed.
NOTE: Versions with major version 0 are reserved for internal usage, so the smallest version you can define is 1.
Elasticsearch-Evolution also supports Java Migrations. A Java Migration is a Java class that implements the JavaMigration interface.
. can be replaced with underscores _ in the class name, so V1_2__my_description.java is a valid filename for a Java Migration with version 1.2.JavaMigration, just overwrite the default implementation of the getMetadata() method. In this case, the filename is not relevant for the version and description.classpath:es/migration).
classpath locations are supported for Java Migrations, file locations are not supported.JavaMigration must have a public no-args constructor, so that Elasticsearch-Evolution can create an instance of it via reflection.Here is an example of a Java Migration:
public class V1_2__AddDocument implements JavaMigration {
@Override
public void migrate(Context context) throws Exception {
String body = """
{
"doc": {
"version": "2",
"success": true,
"a": "a a a"
}
}""";
final EvolutionRestResponse res = context.getEvolutionRestClient().execute(HttpMethod.PUT,
"/test_1/_doc/2?refresh",
Map.of("Content-Type", "application/json"),
null,
body);
if (res.statusCode() != 201) {
throw new IllegalArgumentException("Failed to add document. " + res.asString());
}
}
}
Elasticsearch-Evolution can be configured to your needs:
true): Whether to enable or disable Elasticsearch-Evolution.[classpath:es/migration]): List of locations of migration scripts. Supported are classpath:some/path and file:/some/path. The location is scanned recursively, but only to a depth of 10. NOTE: All scripts in all locations/subdirectories will be flattened, and only the version number will be used to order them.UTF-8): Encoding of migration files.application/json; charset=UTF-8): This content type will be used as the default if no Content-Type header is specified in the header section of a migration script. If no charset is defined, the encoding charset is used.V): File name prefix for migration files.[.http]): List of file name suffixes for migration files. The suffix is checked case-insensitively.true): Whether to enable or disable placeholder replacement in migration scripts.[]): Map of placeholders and their replacements to apply to migration scripts.${): Prefix of placeholders in migration scripts.}): Suffix of placeholders in migration scripts.es_evolution): Name of the history index that will be used by Elasticsearch-Evolution. In this index, Elasticsearch-Evolution will persist its internal state and track which migration scripts have already been executed.1000): The maximum query size while validating already executed scripts. This query size must be higher than the total count of your migration scripts.true): Whether to fail when a previously applied migration script has been modified after it was applied.1.0): Version to use as a baseline. Versions lower than this will not be applied.\n): Line separator, used only temporarily between reading raw migration file line-by-line and parsing it later. Only needed for backward compatibility/checksum stability! Should be one of \n, \r or \r\n.false): Allows migrations to be run "out of order". If you already have versions 1.0 and 3.0 applied, and now version 2.0 is found, it will be applied too instead of being rejected.false): Whether to remove a trailing newline in migration scripts. Only needed for backward compatibility/checksum stability![]): These are not Java-based migrations discovered through classpath scanning and instantiated by Elasticsearch-Evolution. Instead, these are manually added instances of JavaMigration This is particularly useful when working with a dependency injection container, where you may want the DI container to instantiate the class and wire up its dependencies for you.null): A custom ClassProvider to be used to look up JavaMigration classes. If not set, the default strategy will be used which is described in the Java Migrations section.You can set the above configurations via Spring Boot's default configuration mechanism. Just use the prefix spring.elasticsearch.evolution. Here is an example application.properties:
spring.elasticsearch.evolution.locations[0]=classpath:es/migration
spring.elasticsearch.evolution.locations[1]=classpath:es/more_migration_scripts
spring.elasticsearch.evolution.placeholderReplacement=true
spring.elasticsearch.evolution.placeholders.indexname=myIndexReplacement
spring.elasticsearch.evolution.placeholders.docType=_doc
spring.elasticsearch.evolution.placeholders.foo=bar
spring.elasticsearch.evolution.historyIndex=es_evolution
Since Spring Boot 2.1, AutoConfiguration for Elasticsearch's REST client is provided (see org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchRestClientAutoConfiguration or org.springframework.boot.elasticsearch.autoconfigure.ElasticsearchRestClientAutoConfiguration for Spring Boot 4).
You can configure the client, required for Elasticsearch-Evolution, just like this in your application.properties:
spring.elasticsearch.uris[0]=https://example.com:9200
spring.elasticsearch.username=my-user-name
spring.elasticsearch.password=my-secret-pw
Elasticsearch-Evolution tries to be compatible with spring-data-opensearch-starter and its AutoConfiguration.
AutoConfiguration for OpenSearch's REST client is provided (see org.opensearch.spring.boot.autoconfigure.OpenSearchRestClientAutoConfiguration).
AutoConfiguration for OpenSearch's java client is provided (see org.opensearch.spring.boot.autoconfigure.OpenSearchClientAutoConfiguration).
You can configure the client, required for Elasticsearch-Evolution, just like this in your application.properties:
opensearch.uris[0]=https://example.com:9200
opensearch.username=my-user-name
opensearch.password=my-secret-pw
Elasticsearch-Evolution just needs an EvolutionRestClient as a Spring bean. The Elasticsearch EvolutionESRestClient implementation needs a RestClient Spring bean.
If you don't have Spring Boot 2.1 or later, or you need a special RestClient configuration (e.g., to accept self-signed certificates or disable hostname validation), you can provide a custom RestClient like this:
@Bean
public RestClient myRestClient() {
RestClientBuilder builder = RestClient.builder(HttpHost.create("https://localhost:9200"))
.setHttpClientConfigCallback(httpClientBuilder -> {
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("my-user-name", "my-secret-pw"));
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
try {
httpClientBuilder
.setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build())
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
} catch (GeneralSecurityException e) {
throw new IllegalStateException("could not configure http client to accept all certificates", e);
}
return httpClientBuilder;
}
);
return builder.build();
}
Elasticsearch-Evolution just needs an EvolutionRestClient as a Spring bean. The OpenSearch EvolutionOpenSearchRestClient implementation needs a RestClient Spring bean.
If you don't use already the spring-data-opensearch-starter or you need a special RestClient configuration (e.g., to accept self-signed certificates or disable hostname validation), you can provide a custom RestClient similar to the Elasticsearch example above.
Elasticsearch-Evolution just needs an EvolutionRestClient as a Spring bean. The OpenSearch EvolutionOpenSearchGenericClient implementation needs a OpenSearchGenericClient or OpenSearchClient Spring bean.
If you don't use already the spring-data-opensearch-starter or you need a special OpenSearch*Client configuration (e.g., to accept self-signed certificates or disable hostname validation), you can provide a custom OpenSearchGenericClient or OpenSearchClient as Spring bean like this:
@Bean
public OpenSearchClient myOpenSearchClient() throws URISyntaxException {
final HttpHost httpHost = HttpHost.create("https://localhost:9200");
OpenSearchTransport openSearchTransport = ApacheHttpClient5TransportBuilder.builder(httpHost)
// I want to enable compression for better performance
.setCompressionEnabled(true)
.build();
return new OpenSearchClient(openSearchTransport);
}
Providing a OpenSearchTransport Spring bean is also possible, the ElasticsearchEvolutionAutoConfiguration will create the EvolutionOpenSearchGenericClient from it:
@Bean
public OpenSearchTransport myOpenSearchTransport() throws URISyntaxException {
final HttpHost httpHost = HttpHost.create("https://localhost:9200");
return ApacheHttpClient5TransportBuilder.builder(httpHost)
// I want to enable compression for better performance
.setCompressionEnabled(true)
.build();
}
If you want to provide a customized initializer for Elasticsearch-Evolution (e.g., with a different order):
@Bean
public ElasticsearchEvolutionInitializer customElasticsearchEvolutionInitializer(ElasticsearchEvolution elasticsearchEvolution) {
return new ElasticsearchEvolutionInitializer(elasticsearchEvolution) {
@Override
public int getOrder() {
return Ordered.LOWEST_PRECEDENCE;
}
};
}
If you want to customize the Elasticsearch-Evolution configuration after the configuration properties have been applied, you can provide a Spring Bean implementing ElasticsearchEvolutionConfigCustomizer like this:
@Bean
ElasticsearchEvolutionConfigCustomizer javaMigrationBeansCustomizer(ObjectProvider<JavaMigration> javaMigrations) {
// provide additional Java Migrations to Elasticsearch-Evolution managed as Spring Beans
return config -> config.setJavaMigrations(javaMigrations.stream().toList());
}
You can set the above configurations via the ElasticsearchEvolutionConfig fluent builder like this:
ElasticsearchEvolution.configure()
.setLocations(Collections.singletonList("classpath:es/migration"))
.setPlaceholderReplacement(true)
.setPlaceholders(Collections.singletonMap("indexname", "myIndexReplacement"))
.setHistoryIndex("es_evolution");
EvolutionOpenSearchRestClient (elasticsearch-evolution-rest-abstraction-os-restclient) compatibility with OpenSearch 2.x client libs (#565),EvolutionRestClient implementation: EvolutionESRest5Client. It uses the Apache HttpClient 5 based Rest5Client from co.elastic.clients:elasticsearch-rest5-client.EvolutionRestClient implementations (#198, #220, #287, #348).
EvolutionOpenSearchRestClient for OpenSearch RestClient.EvolutionOpenSearchGenericClient for OpenSearchGenericClient and OpenSearchClient.actions/create-release with softprops/action-gh-release (#554).MigrationException.EvolutionRestClient in new artifact com.senacor.elasticsearch.evolution:elasticsearch-evolution-rest-abstraction) and an implementation for the Elasticsearch RestClient (EvolutionESRestClient in artifact com.senacor.elasticsearch.evolution:elasticsearch-evolution-rest-abstraction-es-client) (#553).
com.senacor.elasticsearch.evolution:elasticsearch-evolution-rest-abstraction-es-client.ignore_throttled from ES requests.org.reflections:reflections to 0.10.1.elasticsearch-evolution to 0.4.2.\n. For backward compatibility, you can set a different line separator via the lineSeparator config property.validateOnMigrate to false (default: true) (#155).baselineVersion to skip migrations with versions lower than the defined baselineVersion (#164).org.elasticsearch.client.RestHighLevelClient and replace with org.elasticsearch.client.RestClient (LowLevelClient). This will drop the big transitive dependency org.elasticsearch:elasticsearch and opens compatibility to Elasticsearch 8 and OpenSearch.historyMaxQuerySize.We welcome contributions to Elasticsearch-Evolution! Here's how you can help: CONTRIBUTING.md
Java
98.8%
Shell
1.2%
A library to migrate Elasticsearch and OpenSearch mappings. Inspired by Flyway.
Java
93
974 commits
updated Sep 28, 2026
A library to migrate Elasticsearch and OpenSearch mappings. Inspired by Flyway.
Elasticsearch-Evolution executes versioned migration scripts reliably and persists the execution state in an internal Elasticsearch/OpenSearch index. Successfully executed migration scripts will not be executed again!
| Compatibility | Spring Boot | Elasticsearch | OpenSearch |
|---|---|---|---|
| elasticsearch-evolution >= 0.9.0 | 3.3 - 4.0 | 8.x - 9.x | 2.x - 3.x |
| elasticsearch-evolution >= 0.8.0 | 3.2 - 4.0 | 7.5.x - 9.x | 2.x - 3.x |
| elasticsearch-evolution >= 0.7.0 | 3.2 - 3.5 | 7.5.x - 9.x | 2.x - 3.x |
| elasticsearch-evolution >= 0.6.1 | 3.0 - 3.2 | 7.5.x - 8.19.x | 1.x - 2.x |
| elasticsearch-evolution >= 0.4.2 | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0, 3.1, 3.2 | 7.5.x - 8.13.x | 1.x - 2.x |
| elasticsearch-evolution >= 0.4.0 | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 | 7.5.x - 8.6.x | 1.x - 2.x |
| elasticsearch-evolution 0.3.x | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 | 7.5.x - 7.17.x | |
| elasticsearch-evolution 0.2.x | 1.5, 2.0, 2.1, 2.2, 2.3, 2.4 | 7.0.x - 7.4.x, 6.8.x |
First, add the latest version of the Elasticsearch-Evolution Spring Boot starter as a dependency to your Maven pom.xml
and choose an elasticsearch-evolution-rest-abstraction implementation (here: Elasticsearch RestClient implementation):
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>spring-boot-starter-elasticsearch-evolution</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-client</artifactId>
<version>1.0.0</version>
</dependency>
Place your migration scripts in your application classpath under es/migration.
That's it. Elasticsearch-Evolution runs at application startup and expects your Elasticsearch/OpenSearch instance at http://localhost:9200.
First, add the latest version of Elasticsearch-Evolution core as a dependency
and choose an elasticsearch-evolution-rest-abstraction implementation (here: Elasticsearch RestClient implementation):
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-client</artifactId>
<version>1.0.0</version>
</dependency>
Place your migration scripts in your application classpath under es/migration.
Create an ElasticsearchEvolution instance and execute the migration:
// First, create an Elasticsearch RestClient
RestClient restClient = RestClient.builder(HttpHost.create("http://localhost:9200")).build();
// Then, create an EvolutionRestClient abstraction for Elasticsearch
EvolutionRestClient evolutionRestClient = new EvolutionESRestClient(restClient);
// Then, create an ElasticsearchEvolution configuration and instance
ElasticsearchEvolution elasticsearchEvolution = ElasticsearchEvolution.configure()
.load(evolutionRestClient);
// Execute the migration
elasticsearchEvolution.migrate();
If you just want to validate your migration scripts without executing them, you can use the validate() method:
ElasticsearchEvolution elasticsearchEvolution = ...;
// Just validate the migrations
elasticsearchEvolution.validate();
This will validate applied migrations against resolved ones (on the filesystem or classpath) to detect accidental changes that may prevent the schema(s) from being recreated exactly.
Validation fails if:
validateOnMigrate)When validation fails, a ValidateException is thrown.
Elasticsearch-Evolution uses a REST client abstraction (EvolutionRestClient). Currently, these implementations exist:
EvolutionESRestClient
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-client</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionESRest5Client which is designed for Elasticsearch 9.x and Spring Boot 4.x.
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-es-rest5client</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionOpenSearchRestClient which is designed for OpenSearch 2.x because the RestClientTransport is deprecated for removal since 3.x.
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-os-restclient</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionOpenSearchGenericClient which is designed for OpenSearch 3.x and later.
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction-os-genericclient</artifactId>
<version>1.0.0</version>
</dependency>
You can provide your own implementation of EvolutionRestClient if you want to use another HTTP client like Apache HTTPClient or OkHttpClient. This interface is located in:
<dependency>
<groupId>com.senacor.elasticsearch.evolution</groupId>
<artifactId>elasticsearch-evolution-rest-abstraction</artifactId>
<version>1.0.0</version>
</dependency>
EvolutionRestClient implementation like this:
@Bean
public EvolutionRestClient customEvolutionRestClient() {
return new MyOkHttpClientEvolutionRestClient(...);
}
An Elasticsearch-Evolution migration script represents a REST call. Here is an example:
PUT /_template/my_template
Content-Type: application/json
{
"index_patterns": [
"my_index_*"
],
"order": 1,
"version": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 20,
"similarity": "boolean"
},
"locked": {
"type": "boolean"
}
}
}
}
The first line defines the HTTP method PUT and the relative path to the Elasticsearch/OpenSearch endpoint /_template/my_template to create a new mapping template.
This is followed by an HTTP header Content-Type: application/json.
After a blank line, the HTTP body is defined.
The pattern is strongly oriented towards ordinary HTTP requests and consists of 4 parts:
GET, HEAD, POST, PUT, DELETE, OPTIONS and PATCH.
The first non-comment line must always start with an HTTP method./my_index_1/_doc/1?refresh=true&op_type=create.:.Elasticsearch-Evolution supports line comments in its migration scripts. Every line starting with # or // will be interpreted as a comment line.
Comment lines are not sent to Elasticsearch/OpenSearch; they will be filtered by Elasticsearch-Evolution.
Elasticsearch-Evolution supports named placeholder substitution. Placeholders are marked in your migration script like this: ${my-placeholder}
placeholderPrefix, which is by default ${ and is configurable.placeholder name, which can be any string but must not contain placeholderPrefix or placeholderSuffix.placeholderSuffix, which is by default } and is configurable.Here is an example filename: V1.0__my-description.http
The filename must follow a pattern:
esMigrationPrefix, which is by default V and is configurable...versionDescriptionSeparator: __.esMigrationSuffixes, which is by default .http and is configurable and case-insensitive.Elasticsearch-Evolution uses the version for ordering your scripts and enforces strictly ordered execution of your scripts by default. Out-of-order execution is supported but disabled by default. Elasticsearch-Evolution interprets the version parts as integers, so each version part must be between 1 (inclusive) and 2,147,483,647 (inclusive).
Here is an example that indicates the ordering: 1.0.1 < 1.1 < 1.2.1 < (2.0.0 == 2).
In this example, version 1.0.1 is the smallest version and is executed first, followed by versions 1.1, 1.2.1, and finally 2.
2 is the same as 2.0 or 2.0.0 - trailing zeros will be trimmed.
NOTE: Versions with major version 0 are reserved for internal usage, so the smallest version you can define is 1.
Elasticsearch-Evolution also supports Java Migrations. A Java Migration is a Java class that implements the JavaMigration interface.
. can be replaced with underscores _ in the class name, so V1_2__my_description.java is a valid filename for a Java Migration with version 1.2.JavaMigration, just overwrite the default implementation of the getMetadata() method. In this case, the filename is not relevant for the version and description.classpath:es/migration).
classpath locations are supported for Java Migrations, file locations are not supported.JavaMigration must have a public no-args constructor, so that Elasticsearch-Evolution can create an instance of it via reflection.Here is an example of a Java Migration:
public class V1_2__AddDocument implements JavaMigration {
@Override
public void migrate(Context context) throws Exception {
String body = """
{
"doc": {
"version": "2",
"success": true,
"a": "a a a"
}
}""";
final EvolutionRestResponse res = context.getEvolutionRestClient().execute(HttpMethod.PUT,
"/test_1/_doc/2?refresh",
Map.of("Content-Type", "application/json"),
null,
body);
if (res.statusCode() != 201) {
throw new IllegalArgumentException("Failed to add document. " + res.asString());
}
}
}
Elasticsearch-Evolution can be configured to your needs:
true): Whether to enable or disable Elasticsearch-Evolution.[classpath:es/migration]): List of locations of migration scripts. Supported are classpath:some/path and file:/some/path. The location is scanned recursively, but only to a depth of 10. NOTE: All scripts in all locations/subdirectories will be flattened, and only the version number will be used to order them.UTF-8): Encoding of migration files.application/json; charset=UTF-8): This content type will be used as the default if no Content-Type header is specified in the header section of a migration script. If no charset is defined, the encoding charset is used.V): File name prefix for migration files.[.http]): List of file name suffixes for migration files. The suffix is checked case-insensitively.true): Whether to enable or disable placeholder replacement in migration scripts.[]): Map of placeholders and their replacements to apply to migration scripts.${): Prefix of placeholders in migration scripts.}): Suffix of placeholders in migration scripts.es_evolution): Name of the history index that will be used by Elasticsearch-Evolution. In this index, Elasticsearch-Evolution will persist its internal state and track which migration scripts have already been executed.1000): The maximum query size while validating already executed scripts. This query size must be higher than the total count of your migration scripts.true): Whether to fail when a previously applied migration script has been modified after it was applied.1.0): Version to use as a baseline. Versions lower than this will not be applied.\n): Line separator, used only temporarily between reading raw migration file line-by-line and parsing it later. Only needed for backward compatibility/checksum stability! Should be one of \n, \r or \r\n.false): Allows migrations to be run "out of order". If you already have versions 1.0 and 3.0 applied, and now version 2.0 is found, it will be applied too instead of being rejected.false): Whether to remove a trailing newline in migration scripts. Only needed for backward compatibility/checksum stability![]): These are not Java-based migrations discovered through classpath scanning and instantiated by Elasticsearch-Evolution. Instead, these are manually added instances of JavaMigration This is particularly useful when working with a dependency injection container, where you may want the DI container to instantiate the class and wire up its dependencies for you.null): A custom ClassProvider to be used to look up JavaMigration classes. If not set, the default strategy will be used which is described in the Java Migrations section.You can set the above configurations via Spring Boot's default configuration mechanism. Just use the prefix spring.elasticsearch.evolution. Here is an example application.properties:
spring.elasticsearch.evolution.locations[0]=classpath:es/migration
spring.elasticsearch.evolution.locations[1]=classpath:es/more_migration_scripts
spring.elasticsearch.evolution.placeholderReplacement=true
spring.elasticsearch.evolution.placeholders.indexname=myIndexReplacement
spring.elasticsearch.evolution.placeholders.docType=_doc
spring.elasticsearch.evolution.placeholders.foo=bar
spring.elasticsearch.evolution.historyIndex=es_evolution
Since Spring Boot 2.1, AutoConfiguration for Elasticsearch's REST client is provided (see org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchRestClientAutoConfiguration or org.springframework.boot.elasticsearch.autoconfigure.ElasticsearchRestClientAutoConfiguration for Spring Boot 4).
You can configure the client, required for Elasticsearch-Evolution, just like this in your application.properties:
spring.elasticsearch.uris[0]=https://example.com:9200
spring.elasticsearch.username=my-user-name
spring.elasticsearch.password=my-secret-pw
Elasticsearch-Evolution tries to be compatible with spring-data-opensearch-starter and its AutoConfiguration.
AutoConfiguration for OpenSearch's REST client is provided (see org.opensearch.spring.boot.autoconfigure.OpenSearchRestClientAutoConfiguration).
AutoConfiguration for OpenSearch's java client is provided (see org.opensearch.spring.boot.autoconfigure.OpenSearchClientAutoConfiguration).
You can configure the client, required for Elasticsearch-Evolution, just like this in your application.properties:
opensearch.uris[0]=https://example.com:9200
opensearch.username=my-user-name
opensearch.password=my-secret-pw
Elasticsearch-Evolution just needs an EvolutionRestClient as a Spring bean. The Elasticsearch EvolutionESRestClient implementation needs a RestClient Spring bean.
If you don't have Spring Boot 2.1 or later, or you need a special RestClient configuration (e.g., to accept self-signed certificates or disable hostname validation), you can provide a custom RestClient like this:
@Bean
public RestClient myRestClient() {
RestClientBuilder builder = RestClient.builder(HttpHost.create("https://localhost:9200"))
.setHttpClientConfigCallback(httpClientBuilder -> {
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("my-user-name", "my-secret-pw"));
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
try {
httpClientBuilder
.setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build())
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
} catch (GeneralSecurityException e) {
throw new IllegalStateException("could not configure http client to accept all certificates", e);
}
return httpClientBuilder;
}
);
return builder.build();
}
Elasticsearch-Evolution just needs an EvolutionRestClient as a Spring bean. The OpenSearch EvolutionOpenSearchRestClient implementation needs a RestClient Spring bean.
If you don't use already the spring-data-opensearch-starter or you need a special RestClient configuration (e.g., to accept self-signed certificates or disable hostname validation), you can provide a custom RestClient similar to the Elasticsearch example above.
Elasticsearch-Evolution just needs an EvolutionRestClient as a Spring bean. The OpenSearch EvolutionOpenSearchGenericClient implementation needs a OpenSearchGenericClient or OpenSearchClient Spring bean.
If you don't use already the spring-data-opensearch-starter or you need a special OpenSearch*Client configuration (e.g., to accept self-signed certificates or disable hostname validation), you can provide a custom OpenSearchGenericClient or OpenSearchClient as Spring bean like this:
@Bean
public OpenSearchClient myOpenSearchClient() throws URISyntaxException {
final HttpHost httpHost = HttpHost.create("https://localhost:9200");
OpenSearchTransport openSearchTransport = ApacheHttpClient5TransportBuilder.builder(httpHost)
// I want to enable compression for better performance
.setCompressionEnabled(true)
.build();
return new OpenSearchClient(openSearchTransport);
}
Providing a OpenSearchTransport Spring bean is also possible, the ElasticsearchEvolutionAutoConfiguration will create the EvolutionOpenSearchGenericClient from it:
@Bean
public OpenSearchTransport myOpenSearchTransport() throws URISyntaxException {
final HttpHost httpHost = HttpHost.create("https://localhost:9200");
return ApacheHttpClient5TransportBuilder.builder(httpHost)
// I want to enable compression for better performance
.setCompressionEnabled(true)
.build();
}
If you want to provide a customized initializer for Elasticsearch-Evolution (e.g., with a different order):
@Bean
public ElasticsearchEvolutionInitializer customElasticsearchEvolutionInitializer(ElasticsearchEvolution elasticsearchEvolution) {
return new ElasticsearchEvolutionInitializer(elasticsearchEvolution) {
@Override
public int getOrder() {
return Ordered.LOWEST_PRECEDENCE;
}
};
}
If you want to customize the Elasticsearch-Evolution configuration after the configuration properties have been applied, you can provide a Spring Bean implementing ElasticsearchEvolutionConfigCustomizer like this:
@Bean
ElasticsearchEvolutionConfigCustomizer javaMigrationBeansCustomizer(ObjectProvider<JavaMigration> javaMigrations) {
// provide additional Java Migrations to Elasticsearch-Evolution managed as Spring Beans
return config -> config.setJavaMigrations(javaMigrations.stream().toList());
}
You can set the above configurations via the ElasticsearchEvolutionConfig fluent builder like this:
ElasticsearchEvolution.configure()
.setLocations(Collections.singletonList("classpath:es/migration"))
.setPlaceholderReplacement(true)
.setPlaceholders(Collections.singletonMap("indexname", "myIndexReplacement"))
.setHistoryIndex("es_evolution");
EvolutionOpenSearchRestClient (elasticsearch-evolution-rest-abstraction-os-restclient) compatibility with OpenSearch 2.x client libs (#565),EvolutionRestClient implementation: EvolutionESRest5Client. It uses the Apache HttpClient 5 based Rest5Client from co.elastic.clients:elasticsearch-rest5-client.EvolutionRestClient implementations (#198, #220, #287, #348).
EvolutionOpenSearchRestClient for OpenSearch RestClient.EvolutionOpenSearchGenericClient for OpenSearchGenericClient and OpenSearchClient.actions/create-release with softprops/action-gh-release (#554).MigrationException.EvolutionRestClient in new artifact com.senacor.elasticsearch.evolution:elasticsearch-evolution-rest-abstraction) and an implementation for the Elasticsearch RestClient (EvolutionESRestClient in artifact com.senacor.elasticsearch.evolution:elasticsearch-evolution-rest-abstraction-es-client) (#553).
com.senacor.elasticsearch.evolution:elasticsearch-evolution-rest-abstraction-es-client.ignore_throttled from ES requests.org.reflections:reflections to 0.10.1.elasticsearch-evolution to 0.4.2.\n. For backward compatibility, you can set a different line separator via the lineSeparator config property.validateOnMigrate to false (default: true) (#155).baselineVersion to skip migrations with versions lower than the defined baselineVersion (#164).org.elasticsearch.client.RestHighLevelClient and replace with org.elasticsearch.client.RestClient (LowLevelClient). This will drop the big transitive dependency org.elasticsearch:elasticsearch and opens compatibility to Elasticsearch 8 and OpenSearch.historyMaxQuerySize.We welcome contributions to Elasticsearch-Evolution! Here's how you can help: CONTRIBUTING.md
Java
98.8%
Shell
1.2%