Remove HLRC from ES codebase #83423

dakrone · 2022-02-02T18:59:27Z

Description

With the launch of the new Java client the Elasticsearch High Level Rest Client (HLRC) is deprecated, no longer shipped as an artifact, and can be removed from the ES codebase. This meta issue is to track the the removal of the HLRC for 8.x.

Steps (subject to change):

Remove all HLRC tests Remove all HLRC tests #83425
Remove all usages of HLRC sub-clients by tests, replacing them with the low-level RestClient. These classes can be found inside of RestHighLevelClient.java
- Remove usages of IndicesClient (indices()) (@martijnvg, Remove HLRC IndicesClient and related classes. #85492)
- Remove usages of ClusterClient (cluster()) (@gmarouli, Remove Cluster & XPack clients from HLRC (#83423) #83593)
- Remove usages of IngestClient (ingest()) (@martijnvg, Remove HLRC IngestClient class. #85483)
- Remove usages of SnapshotClient (snapshot()) (@dakrone, Remove SnapshotClient from HLRC #85845)
- Remove usages of TasksClient (tasks()) (@danhermann, Remove tasks client from HLRC #83649)
- Remove usages of XPackClient (xpack()) (@gmarouli, Remove Cluster & XPack clients from HLRC (#83423) #83593)
- Remove usages of GraphClient (graph()) (@danhermann, Remove graph client from deprecated HLRC #83471)
- Remove usages of IndexLifecycleClient (indexLifecycle()) (@dakrone, Remove IndexLifecycleClient from High Level Rest Client #83476)
- Remove usages of SearchableSnapshotsClient (searchableSnapshots()) (@dakrone, Remove HLRC's SearchableSnapshotsClient #85472)
- Remove usages of FeaturesClient (features()) (@danhermann, Remove features client from HLRC #83651)
- Remove usages of MigrationClient (migration()) (@joegallo, Remove MigrationClient from High Level Rest Client #83609)
- Remove usages of MachineLearningClient (machineLearning()) ([ML] Remove uses of ML HLRC classes #83885, [ML] Delete ML HLRC client and request/response classes #83912)
- Remove usages of SecurityClient (security())
  - Remove unused methods (@tvernum, Remove unused security methods/classes from HLRC #83571)
  - Remove "user" methods (@tvernum Remove enable/disableUser methods from HLRC #83626, Remove all "user" related methods from HLRC #84011)
  - Remove role & role mapping methods (@tvernum Remove role related methods from HLRC #84206)
  - Remove token & api key methods
  - Remove remaining methods (@tvernum [HLRC] Remove SecurityClient #85983)
- Remove usages of TransformClient (transform()) ([Transform] Remove HLRC Transform client #83921)
- Remove usages of EqlClient (eql()) (@joegallo, Remove EqlClient from HLRC #85730)
- Remove unused clients (@danhermann, Remove unused clients from HLRC #83472)
  - Remove usages of CcrClient (ccr())
  - Remove usages of WatcherClient (watcher())
  - Remove usages of EnrichClient (enrich())
  - Remove usages of LicenseClient (license())
  - Remove usages of TextStructureClient (textStructure())
  - Remove usages of AsyncSearchClient (asyncSearch())
  - Remove usages of RollupClient (rollup())
Remove remaining usages of RestHighLevelClient
- RestHighLevelClient(RestClientBuilder restClientBuilder) (constructor)
  - getLowLevelClient()
  - close()
  - bulk(BulkRequest bulkRequest, RequestOptions options)
  - bulkAsync(BulkRequest bulkRequest, RequestOptions options, ActionListener<BulkResponse> listener)
  - reindex(ReindexRequest reindexRequest, RequestOptions options)
  - updateByQuery(UpdateByQueryRequest updateByQueryRequest, RequestOptions options)
  - deleteByQuery(DeleteByQueryRequest deleteByQueryRequest, RequestOptions options)
  - info(RequestOptions options)
  - get(GetRequest getRequest, RequestOptions options)
  - multiGet(MultiGetRequest multiGetRequest, RequestOptions options)
  - mget(MultiGetRequest multiGetRequest, RequestOptions options)
  - multiGetAsync(
  - mgetAsync(MultiGetRequest multiGetRequest, RequestOptions options, ActionListener<MultiGetResponse> listener)
  - exists(GetRequest getRequest, RequestOptions options)
  - existsSource(GetRequest getRequest, RequestOptions options)
  - existsSourceAsync(GetRequest getRequest, RequestOptions options, ActionListener<Boolean> listener)
  - getSource(GetSourceRequest getSourceRequest, RequestOptions options)
  - index(IndexRequest indexRequest, RequestOptions options)
  - count(CountRequest countRequest, RequestOptions options)
  - update(UpdateRequest updateRequest, RequestOptions options)
  - delete(DeleteRequest deleteRequest, RequestOptions options)
  - search(SearchRequest searchRequest, RequestOptions options)
  - searchAsync(SearchRequest searchRequest, RequestOptions options, ActionListener<SearchResponse> listener)
  - multiSearch(MultiSearchRequest multiSearchRequest, RequestOptions options)
  - msearch(MultiSearchRequest multiSearchRequest, RequestOptions options)
  - multiSearchAsync(
  - msearchAsync(
  - searchScroll(SearchScrollRequest searchScrollRequest, RequestOptions options)
  - scroll(SearchScrollRequest searchScrollRequest, RequestOptions options)
  - searchScrollAsync(
  - scrollAsync(
  - explain(ExplainRequest explainRequest, RequestOptions options)
  - termvectors(TermVectorsRequest request, RequestOptions options)
  - fieldCaps(FieldCapabilitiesRequest fieldCapabilitiesRequest, RequestOptions options)

As each sub client is removed its creator inside of RestHighLevelClient to indicate that portion of the removal is "done".

The text was updated successfully, but these errors were encountered:

elasticmachine · 2022-02-02T18:59:30Z

Pinging @elastic/es-data-management (Team:Data Management)

We are in the process of removing the High Level Rest Client from the ES codebase. This starts by removing all the HLRC tests themselves so that the other usages can be removed incrementally. Relates to elastic#83423

We are in the process of removing the High Level Rest Client from the ES codebase. This starts by removing all the HLRC tests themselves so that the other usages can be removed incrementally. Relates to #83423

It was used in the SLM permissions tests to check permissions for policy and retention CRUD. This commit replaces it with the LLRC equivalent. Relates to elastic#83423

It was used in the SLM permissions tests to check permissions for policy and retention CRUD. This commit replaces it with the LLRC equivalent. Relates to #83423

The High Level Rest Client (HLRC) is deprecated and is no longer shipped as a standalone artifact. It can now be safely removed from the ES codebase. This commit removes all security methods from the HLRC "SecurityClient", along with their associated Request/Response classes. All remaining methods and classes are used by existing tests and will need to be reviewed. Relates: elastic#83423

The High Level Rest Client (HLRC) is deprecated and is no longer shipped as a standalone artifact. It can now be safely removed from the ES codebase. This commit removes unused security methods from the HLRC "SecurityClient", along with their associated Request/Response classes. All remaining methods and classes are used by existing tests and will need to be reviewed. Relates: #83423

Removes the `enableUser` and `disableUser` methods from the High Level Rest Client's `SecurityClient` and replaces existing usage in tests with a helper method. Relates: elastic#83423

Part of #83423

Removes the `enableUser` and `disableUser` methods from the High Level Rest Client's `SecurityClient` and replaces existing usage in tests with a helper method. Relates: #83423

Removes the following methods from the SecurityClient component of the High Level Rest Client - putUser - deleteUser - changePassword - authenticate As part of this change, I renamed the SecurityClientTestHelper class to TestSecurityClient and made it a real object rather than a set of utility methods. This was needed because different tests need different RequestOptions objects, but passing it into every method made it cumbersome. The code is clearer if we use a field in the test client itself. Relates: #83423

This commit removes role and role-mapping related methods from the High Level Rest Client, and adds light weight replacements to the TestSecurityClient Relates: elastic#83423

elasticsearchmachine · 2022-11-30T19:01:25Z

Pinging @elastic/clients-team (Team:Clients)

joegallo · 2022-11-30T19:09:54Z

Only 8 public (well, public final) methods remaining on RestHighLevelClient, all of which have some actual caller somewhere. And only 12 java files left in the rest-high-level directory. 🎉

edit: And only ten classes that import the HLRC:

joegallo@galactic:~/Code/elastic/elasticsearch $ git grep 'import org.elasticsearch.client.RestHighLevelClient' | wc -l
      10

joegallo · 2022-12-02T17:16:56Z

joegallo@galactic:~/Code/elastic/elasticsearch $ git ls-files | grep rest-high-level | grep analytics
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/ParsedStringStats.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/ParsedTopMetrics.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/StringStatsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/TopMetricsAggregationBuilder.java
joegallo@galactic:~/Code/elastic/elasticsearch $ git grep 'import org.elasticsearch.client.analytics' | grep -v rest-high-level
x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/stringstats/InternalStringStatsTests.java:import org.elasticsearch.client.analytics.ParsedStringStats;
x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/topmetrics/InternalTopMetricsTests.java:import org.elasticsearch.client.analytics.ParsedTopMetrics;

I chatted with @nik9000 about these four classes today, and he brought up #26220. Those classes do still (or, well, may still) have value w.r.t. to testing bwc of json responses. Adding better testing of that via our normal rest tests is under the banner of #26220, so at some point I expect that we'll just be able to drop those entirely, as long as we're sure the for-testing-bwc-purposes value of them has already been captured elsewhere.

nik9000 · 2022-12-02T17:20:59Z

I think it's worth confirming that we get good coverage of the json response in the yaml tests and then removing our usage of the classes in ES. It's a little bit of surgery, but we are capable of it.

joegallo · 2023-11-02T15:42:00Z

joegallo@simulacron:~/Code/elastic/elasticsearch $ git grep 'import org.elasticsearch.client.RestHighLevelClient'
qa/ccs-unavailable-clusters/src/javaRestTest/java/org/elasticsearch/search/CrossClusterSearchUnavailableClusterIT.java:import org.elasticsearch.client.RestHighLevelClient;
qa/multi-cluster-search/src/test/java/org/elasticsearch/search/CCSDuelIT.java:import org.elasticsearch.client.RestHighLevelClient;
joegallo@simulacron:~/Code/elastic/elasticsearch $ git ls-files | grep rest-high-level | grep java
client/rest-high-level/src/main/java/org/elasticsearch/client/RequestConverters.java
client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
client/rest-high-level/src/main/java/org/elasticsearch/client/Validatable.java
client/rest-high-level/src/main/java/org/elasticsearch/client/ValidationException.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/ParsedStringStats.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/ParsedTopMetrics.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/StringStatsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/TopMetricsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/asyncsearch/AsyncSearchResponse.java
client/rest-high-level/src/main/java/org/elasticsearch/client/core/MainResponse.java

Only two more test classes left using RestHighLevelClient. 🎉

iverase · 2023-11-13T11:02:09Z

FYI: #102049

iverase · 2023-11-15T11:20:18Z

FYI: #102222

joegallo · 2023-11-16T15:18:36Z

#102305 drops RestHighLevelClient itself (as well as some supporting classes and files), but the client/rest-high-level directory isn't empty, so we're not finished here yet:

joegallo@simulacron:~/Code/elastic/elasticsearch $ git ls-files | grep client/rest-high-level
client/rest-high-level/build.gradle
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/ParsedStringStats.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/ParsedTopMetrics.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/StringStatsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/TopMetricsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/asyncsearch/AsyncSearchResponse.java
client/rest-high-level/src/main/resources/forbidden/rest-high-level-signatures.txt
joegallo@simulacron:~/Code/elastic/elasticsearch $ git grep 'import org.elasticsearch.client.analytics'
x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/stringstats/InternalStringStatsTests.java:import org.elasticsearch.client.analytics.ParsedStringStats;
x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/topmetrics/InternalTopMetricsTests.java:import org.elasticsearch.client.analytics.ParsedTopMetrics;
joegallo@simulacron:~/Code/elastic/elasticsearch $ git grep 'import org.elasticsearch.client.asyncsearch'
x-pack/plugin/async-search/qa/security/src/javaRestTest/java/org/elasticsearch/xpack/search/AsyncSearchSecurityIT.java:import org.elasticsearch.client.asyncsearch.AsyncSearchResponse;

There are still five classes remaining in client/rest-high-level and they are used by three distinct tests. The AsyncSearchResponse change looks like relatively low-hanging fruit, and then there's still the conversation from #83423 (comment).

joegallo · 2024-02-07T19:13:08Z

joegallo@simulacron:~/Code/elastic/elasticsearch $ git ls-files | grep rest-high-level
client/rest-high-level/build.gradle
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/StringStatsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/analytics/TopMetricsAggregationBuilder.java
client/rest-high-level/src/main/java/org/elasticsearch/client/asyncsearch/AsyncSearchResponse.java
client/rest-high-level/src/main/resources/forbidden/rest-high-level-signatures.txt
joegallo@simulacron:~/Code/elastic/elasticsearch $ git grep -E '(org.elasticsearch.client.analytics|org.elasticsearch.client.asyncsearch)' | grep -v package | sed -e "s/:.*//g" | sort | uniq
x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/stringstats/StringStatsAggregationBuilderTests.java
x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/topmetrics/TopMetricsAggregationBuilderTests.java
x-pack/plugin/async-search/qa/security/src/javaRestTest/java/org/elasticsearch/xpack/search/AsyncSearchSecurityIT.java

Not much change since November, there are three classes remaining in client/rest-high-level and three distinct tests that rely on them.

iverase · 2024-02-08T16:37:17Z

FYI: #105294

dakrone added :Data Management/Java High Level REST Client >tech debt labels Feb 2, 2022

elasticmachine added the Team:Data Management Meta label for data/management team label Feb 2, 2022

dakrone mentioned this issue Feb 2, 2022

Remove all HLRC tests #83425

Merged

This was referenced Feb 3, 2022

Remove graph client from deprecated HLRC #83471

Merged

Remove unused clients from HLRC #83472

Merged

dakrone mentioned this issue Feb 3, 2022

Remove IndexLifecycleClient from High Level Rest Client #83476

Merged

tvernum mentioned this issue Feb 7, 2022

Remove unused security methods/classes from HLRC #83571

Merged

gmarouli added a commit to gmarouli/elasticsearch that referenced this issue Feb 7, 2022

Remove Cluster & XPack clients from HLRC (elastic#83423)

4c62643

joegallo mentioned this issue Feb 7, 2022

Remove MigrationClient from High Level Rest Client #83609

Merged

tvernum mentioned this issue Feb 8, 2022

Remove enable/disableUser methods from HLRC #83626

Merged

This was referenced Feb 8, 2022

Remove tasks client from HLRC #83649

Merged

Remove features client from HLRC #83651

Merged

gmarouli added a commit that referenced this issue Feb 10, 2022

Remove Cluster & XPack clients from HLRC (#83593)

d4caeea

Part of #83423

davidkyle mentioned this issue Feb 14, 2022

[ML] Remove uses of ML HLRC classes #83885

Merged

tvernum mentioned this issue Feb 16, 2022

Remove all "user" related methods from HLRC #84011

Merged

This was referenced Nov 30, 2022

Remove IndexerState from HLRC #92023

Merged

Remove unused methods and classes from HLRC (again) #92030

Merged

elasticsearchmachine added Team:Clients Meta label for clients team and removed Team:Data Management Meta label for data/management team labels Nov 30, 2022

joegallo added the Team:Data Management Meta label for data/management team label Nov 30, 2022

elasticsearchmachine removed the Team:Data Management Meta label for data/management team label Nov 30, 2022

iverase mentioned this issue Nov 13, 2023

Remove HighLevelRestClient from CrossClusterSearchUnavailableClusterIT #102049

Merged

joegallo mentioned this issue Nov 14, 2023

More miscellaneous tidying towards removing the HLRC #102175

Merged

iverase mentioned this issue Nov 15, 2023

Remove HighLevelRestClient from CCSDuelIT #102222

Merged

joegallo mentioned this issue Nov 16, 2023

Remove the RestHighLevelClient and some supporting classes #102305

Merged

joegallo mentioned this issue Jan 29, 2024

Remove gradle references to the rest-high-level client #104871

Merged

This was referenced Feb 12, 2024

Move test-only search response x-content-parsing code to test codebase #105308

Merged

Remove HLRC from ES codebase #105406

Merged

[DOCS] Drop these unused references #105409

Merged

joegallo closed this as completed in #105406 Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove HLRC from ES codebase #83423

Remove HLRC from ES codebase #83423

dakrone commented Feb 2, 2022 •

edited by joegallo

Loading

elasticmachine commented Feb 2, 2022

elasticsearchmachine commented Nov 30, 2022

joegallo commented Nov 30, 2022 •

edited

Loading

joegallo commented Dec 2, 2022 •

edited

Loading

nik9000 commented Dec 2, 2022

joegallo commented Nov 2, 2023

iverase commented Nov 13, 2023

iverase commented Nov 15, 2023

joegallo commented Nov 16, 2023

joegallo commented Feb 7, 2024

iverase commented Feb 8, 2024

Remove HLRC from ES codebase #83423

Remove HLRC from ES codebase #83423

Comments

dakrone commented Feb 2, 2022 • edited by joegallo Loading

Description

elasticmachine commented Feb 2, 2022

elasticsearchmachine commented Nov 30, 2022

joegallo commented Nov 30, 2022 • edited Loading

joegallo commented Dec 2, 2022 • edited Loading

nik9000 commented Dec 2, 2022

joegallo commented Nov 2, 2023

iverase commented Nov 13, 2023

iverase commented Nov 15, 2023

joegallo commented Nov 16, 2023

joegallo commented Feb 7, 2024

iverase commented Feb 8, 2024

dakrone commented Feb 2, 2022 •

edited by joegallo

Loading

joegallo commented Nov 30, 2022 •

edited

Loading

joegallo commented Dec 2, 2022 •

edited

Loading