DRAFT [META] REST Compatible API V7 completeness

[pgomulka]

REST Compatible API v7 completeness

As described in #51816 all breaking changes could be subject to applying some compatibility at the REST layer to assist with upgrades. This issue exists to ensure that there is a complete and exhaustive list of all breaking changes and make the determination REST API compatibility should be applied.

A complete list of all breaking changes categorized by

• Requires REST API compatibility with v7
• Does not require REST API compatibility with v7
• Unsure - needs more investigation (temporary status while we work through the list)
• Potentially leverage REST API compatibility, but out of immediate scope

The number of check boxes here should equal number of closed issues from this query:
Eldest to newest breaking changes

Each line item should have a independent validation, and line items that require changes should be linked to the associated PR and issues. For specific compatibilities that require discussions, please open a new issue (as opposed to the comments of this issue).

Requires REST API compatibility with v7

Misc

//TODO: break up into better categories

• Remove deprecated _time and _term sort orders Remove deprecated _time and _term sort orders #39450 - //Przemek: this parses directly using XContentParser (not using ObjectParser and field declaration). Should be a fairly easy to do with surgical conditional (if parser.getRestVersion=V7)
• Parse empty first line in msearch request body as action metadata Parse empty first line in msearch request body as action metadata #41011 //Przemek: a request shape change. A fairly minor, but would be breaking users. The compatible change is basicallly bringing back the 7.x code (deprecation and parsing) and wrapping with if(isCompatible). The readMultiLineFormat would have to get that flag from restrequest in RestMultiSearchAction
• Get snapshots support for multiple repositories Get snapshots support for multiple repositories #42090 //Przemek: it is a fairly big change and might require help from es-distributed team. There is an issue for this work already Support v7 response format for get-snapshots API #69108
• Remove CommonTermsQuery and cutoff_frequency param Remove CommonTermsQuery and cutoff_frequency param  #42654 //Przemek: it looks like a fairly simple change. For cutoff_frequency we could simply allow to parse these but do nothing. But for CommonTermsQuery it looks more complex and I am not sure how to implement this without bringing too much code back.
• Remove deprecated sort options: nested_path and nested_filter Remove deprecated sort options: nested_path and nested_filter #42809
• Reindex remove outer level size Reindex remove outer level size #43373 compatible reindex [REST Compatible API] Reindex size field #69037 compatible update/delete by query [Rest Compatible Api] update and delete by query using size field #69606
• [ML][Data Frame] removing format support in date_histogram group_by [ML][Data Frame] removing format support in date_histogram group_by #43659 (feature was beta when this change was made)
• Force Merge should reject requests with only_expunge_deletes and max_num_segments set Force Merge should reject requests with only_expunge_deletes and max_num_segments set #44761
• Remove deprecated endpoints containing _xpack. Remove deprecated endpoints containing _xpack. #48170
• Remove the 'template' field in index templates Remove the 'template' field in index templates. #49460
• Remove the 'local' parameter of /_cat/nodes Remove the 'local' parameter of /_cat/nodes #50594
• Goodbye and thank you synced flush! Goodbye and thank you synced flush! #50882
• Remove support for settings in restore requests Remove support for settings in restore requests #53284
• Remove the object format for indices_boost Remove the object format for indices_boost. #55078
• Remove local parameter for get field mapping request Remove local parameter for get field mapping request #55100
• Remove use_field_mapping format option for docvalue fields Remove use_field_mapping format option for docvalue fields. #55622
• Remove node filters for voting config exclusions Remove node filters for voting config exclusions #55673
• Scripting: Move script_cache into _nodes/stats Scripting: Move script_cache into _nodes/stats #59265
• Consolidate script parsing from object Consolidate script parsing from object #59507
• [ML] Remove deprecated _xpack endpoints [ML] Remove deprecated _xpack endpoints #59870
• Rename binary_soft_classification evaluation to outlier_detection Rename binary_soft_classification evaluation to outlier_detection #59951 (feature was experimental when this change was made)
• Remove deprecated _upgrade API Remove deprecated _upgrade API #64732
• Remove the deprecated local parameter for _cat/shards Remove the deprecated local parameter for _cat/shards #64867
• Remove the deprecated local parameter for _cat/indices Remove the deprecated local parameter for _cat/indices #64868
• [Transform] use ISO dates in output instead of epoch millis [Transform] use ISO dates in output instead of epoch millis #65584
• Returning nested data in fields API Returning nested data in fields API #67432
• Remove MovingAverage pipeline aggregation Remove MovingAverage pipeline aggregation #39328

Types

These are mostly managed by #54160. The issues here match 1:1 to the issues with >breaking label to help ensure the counts of breaking changes and number of checkboxes are equal.

• Removes typed URLs Removes typed URLs #41636 not merged. splitted into multiple prs
• Removes typed endpoint from search and related APIs Removes typed endpoint from search and related APIs #41640
• Removes tpyed endpoints for msearch and mtermvector APIs Removes tpyed endpoints for msearch and mtermvector APIs #41674
• Removes typed URLs from mapping APIs Removes typed URLs from mapping APIs #41676
• Removes types from bulk API Removes types from bulk API #42194
• Removes type from TermVectors APIs Removes type from TermVectors APIs #42198
• Remove types from Get/MultiGet Remove types from Get/MultiGet #46587
• Remove per-type indexing stats Remove per-type indexing stats #47203
• Remove type query Remove type query #47207
• Remove include_type_name parameter from REST layer Remove include_type_name parameter from REST layer #48632
• Remove support for _type in searches Remove support for _type in searches #68564

Mappings

• Remove support for chained multi-fields. Remove support for chained multi-fields. #42333
• Remove support for string in unmapped_type Remove support for string in unmapped_type. #45675
• Remove support for sparse vectors Remove support for sparse vectors. #48781
• Remove xpack actions for the flattened field Remove xpack actions for the flattened field. #53076
• Remove TypeFieldMapper Remove TypeFieldMapper #53611
• Fix updating include_in_parent/include_in_root of nested field Fix updating include_in_parent/include_in_root of nested field. #54386
• Disallow changing 'enabled' on the root mapper Disallow changing 'enabled' on the root mapper. #54463
• Add declarative parameters to FieldMappers Add declarative parameters to FieldMappers #58663 (needs a better look)

Unsure - needs more investigation

• [ML] add new flag exclude_generated that removes generated fields in GET config APIs [ML] add new flag exclude_generated that removes generated fields in GET config APIs #63899
• Don't expose TextFieldMapper subfields Don't expose TextFieldMapper subfields #64597
• QL: "fields" api implementation in QL QL: "fields" api implementation in QL #68802

Potentially leverage REST API compatibility, but out of immediate scope

scripting

• Update the signature of vector script functions Update the signature of vector script functions. #48260
• Remove deprecated vector functions Remove deprecated vector functions. #48725

EQL

• EQL: Forbid usage of ['] for string literals EQL: Forbid usage of ['] for string literals #62458
• EQL: Replace ?"..." with """...""" for unescaped strings EQL: Replace ?"..." with """...""" for unescaped strings #62539
• EQL: Replace [`] with ['] for identifiers EQL: Replace [`] with ['] for identifiers #62638
• EQL: Remove support for = for comparisons EQL: Remove support for = for comparisons #62756

Does not require REST API compatibility with v7

Setttings

Settings are not subject to REST API compatibility. Often the changes behind the settings represent some behavior that is not carried forward.

• Remove obsolete security settings Remove obsolete security settings #40496
• Remove deprecated search.remote settings Remove deprecated search.remote settings #42381
• Remove node.max_local_storage_nodes Remove node.max_local_storage_nodes #42428
• Do not set a NameID format in Policy by default Do not set a NameID format in Policy by default #44090
• Remove processors setting Remove processors setting #45905
• Remove the pidfile setting Remove the pidfile setting #45940
• Forbid settings without a namespace Forbid settings without a namespace #45947
• Disallow _field_names enabled setting Disallow _field_names enabled setting #46681
• Remove include_relocations setting Remove include_relocations setting #47717
• [Transform] introduce new roles and deprecate old ones [Transform] introduce new roles and deprecate old ones #47780
• Make order setting mandatory for Realm config Make order setting mandatory for Realm config #51195
• Make soft-deletes mandatory in 8.0 Make soft-deletes mandatory in 8.0 #51122
• Remove translog retention settings Remove translog retention settings #51697
• Remove fixed_auto_queue_size threadpool type Remove fixed_auto_queue_size threadpool type #52280
• Remove the cluster.remote.connect setting Remove the cluster.remote.connect setting #54175
• Remove the node local storage setting Remove the node local storage setting #54381
• Remove support for delaying state recovery pending master Remove support for delaying state recovery pending master #53845
• Remove deprecated AUTH_PASSWORD setting Remove deprecated AUTH_PASSWORD setting #54892
• Remove deprecated basic license feature enablement settings from 8.0 Remove deprecated basic license feature enablement settings from 8.0 #56211
• Remove slowlog level Remove slowlog level #57591
• Default gateway.auto_import_dangling_indices to false Default gateway.auto_import_dangling_indices to false #58898
• Scripting: Remove general cache settings Scripting: Remove general cache settings #59262
• Set specific keepalive options by default on supported platforms Set specific keepalive options by default on supported platforms #59278
• Remove the listener thread pool Remove the listener thread pool #53314
• Remove join timeout Remove join timeout #60873
• Remove content type required setting Remove content type required setting #61043
• Add a cluster setting to disallow loading fielddata on _id field Add a cluster setting to disallow loading fielddata on _id field #49166
• Remove insecure settings Remove insecure settings #46147
• Limit processors by available processors Limit processors by available processors #44894

Validation

Additional or stricter validation is a type of breaking change that can change the response of a request from a success to a specific failure. However, success -> error responses are not covered by REST API compatibility and is considered behavior.

• Make Fuzziness reject illegal values earlier Make Fuzziness reject illegal values earlier #33511
• Allow parsing Content-Type and Accept headers with version Allow parsing Content-Type and Accept headers with version #61427
• API key name should always be required for creation API key name should always be required for creation #59836
• Add validation for dynamic templates Add validation for dynamic templates #51233
• Reject misconfigured/ambiguous SSL server config Reject misconfigured/ambiguous SSL server config #45892
• SQL: Test and fix the NULL handling of the String functions SQL: Test and fix the NULL handling of the String functions #68379
• Highlighters skip ignored keyword values Highlighters skip ignored keyword values #53408
• Move pipeline agg validation to coordinating node Move pipeline agg validation to coordinating node #53669
• Distinguish between simple matches with and without the terms index Distinguish between simple matches with and without the terms index #63945
• Fail node containing ancient closed index Fail node containing ancient closed index #44264
• Enforce Completion Context Limit Enforce Completion Context Limit #38675
• Add lower bound on poll_interval Add lower bound on poll_interval #39593
• Percentiles aggregation: disallow specifying same percentile values twice Percentiles aggregation: disallow specifying same percentile values twice #52257
• Reject port ranges in `discovery.seed_hosts Reject port ranges in discovery.seed_hosts #41404

Is actually breaking for 8.0 ?

Perhaps mislabeled or was beta/experimental and/or change was made in a minor

• Stop returning cluster state size by default Stop returning cluster state size by default #40016
• Remove Migration Upgrade and Assistance APIs Remove Migration Upgrade and Assistance APIs #40075
• Remove cluster state size Remove cluster state size #40061
• [ML-DataFrame] Combine task_state and indexer_state in _stats [ML-DataFrame] Combine task_state and indexer_state in _stats #45276 (feature was beta when this change was made)
• [ML] Improve response format of data frame stats endpoint [ML] Improve response format of data frame stats endpoint #44350 (feature was beta when this change was made)
• [ML] Improve progress reportings for DF analytics [ML] Improve progress reportings for DF analytics #45856 (feature was experimental when this change was made)
• [ML][Transforms] remove force flag from _start [ML][Transforms] remove force flag from _start #46414 (feature was beta when this change was made)
• Remove Repository Stats API Remove Repository Stats API #62309

Misc

• Drop support for the low-level REST client on JDK 7 Drop support for the low-level REST client on JDK 7 #38540 (drop support)
• Blob Store compress default to true Blob Store compress default to true #40033 (behavior)
• Make remote cluster resolution stricter Make remote cluster resolution stricter #40419 (behavior)
• Drop support for Ubuntu 14.04 Drop support for Ubuntu 14.04 #40709 (not rest api)
• Bump the minimum Java version to Java 11 Bump the minimum Java version to Java 11 #40754 (not rest api)
• Fixed synchronizing inflight breaker with internal variable Fixed synchronizing inflight breaker with internal variable #40878 (not rest api)
• Reindex from Remote encoding Reindex from Remote encoding #41007 (behavior)
• RollupStart endpoint should return OK if job already started RollupStart endpoint should return OK if job already started #41502 (behavior and response status code)
• Cleanup versioned deprecations in analysis Cleanup versioned deprecations in analysis #41560 (clean up no longer relvant)
• Provide an Option to Use Path-Style-Access with S3 Repo Provide an Option to Use Path-Style-Access with S3 Repo #41966 (behavior)
• Remove the migrate tool Remove the migrate tool #42174 (clean up)
• Update the schema for the REST API specification Update the schema for the REST API specification #42346 (not relevant)
• Remove "nodes/0" folder prefix from data path Remove "nodes/0" folder prefix from data path #42489 (not relevant)
• Remove client jar support from build Remove client jar support from build #42640 (cleanup)
• Do not allow modify aliases on followers Do not allow modify aliases on followers #43017 (behavior)
• Remove the client transport profile filter Remove the client transport profile filter #43236 (cleanup)
• Remove preconfigured `delimited_payload_filter Remove preconfigured delimited_payload_filter #43686 (behavior)
• Forbid empty doc values on vector functions Forbid empty doc values on vector functions #43944 (behavior)
• [ML] Remove the ability to move datafeeds between jobs [ML] Remove the ability to move datafeeds between jobs #44616 (behavior ...but maybe this should be considered ?)
• Remove client feature tracking Remove client feature tracking #44929 (cleanup)
• Decouple shard allocation awareness from search and get requests Decouple shard allocation awareness from search and get requests #45735 (behavior)
• Use float instead of double for query vectors Use float instead of double for query vectors. #46004 (behavior)
• Add support for aliases in queries on _index Add support for aliases in queries on _index. #46640 (behavior, but needs another look)
• [Transform]rename classes in transform plugin [Transform]rename classes in transform plugin #46784 (not rest api)
• [Transform] Rename internal indexes for transform plugin [Transform] Rename internal indexes for transform plugin #47788 (behavior)
• Link child requests to the root task that initiated the action Link child requests to the root task that initiated the action #48422 (closed...probably should remove breaking label)
• Remove type metadata from ingest documents Remove type metadata from ingest documents #50131 (behvior)
• Make range query rounding consistent Make range query rounding consistent #50237
• Remove DEBUG-level default logging from actions Remove DEBUG-level default logging from actions #51459 (behavior)
• Drop support for older OS in the next major release Drop support for older OS in the next major release #51480 (dropping support)
• Remove SysV init support Remove SysV init support #51716 (dropping support)
• Use a tiny base image for Docker builds Use a tiny base image for Docker builds #52519 (build change)
• Remove support for fractional byte size values Remove support for fractional byte size values #53927 (behavior)
• value_count Aggregation optimization value_count Aggregation optimization #54854 (behavior)
• Change prefer_v2_templates parameter to default to true Change prefer_v2_templates parameter to default to true #55489 (behavior)
• Add auto create action Add auto create action #55858 (behavior)
• Remove dangling index auto import functionality Remove dangling index auto import functionality #59698 (behavior)
• EQL: Disallow chained comparisons EQL: Disallow chained comparisons #62567 (behavior)
• Fix range query on date fields for number inputs Fix range query on date fields for number inputs #63692 (behavoir)
• Move watcher history to data stream Move watcher history to data stream  #64252 (behvaior)
• Compress audit logs Compress audit logs #64472 (behavior)
• Remove escape hatch permitting incompatible builds Remove escape hatch permitting incompatible builds #65753 (build)
• Remove grace period from license expiration check Remove grace period from license expiration check #67316 (behavior)
• Remove support for JAVA_HOME Remove support for JAVA_HOME #69149 (build)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[joegallo]

re: Remove Repository Stats API #62309: see #62309 as well as #62297 and #62308 -- this was /_snapshot/{repository}/_stats and as @tlrx observed "we're talking about an experimental API that was never released(behind a feature flag)." I think it's fair to assume we will not need to cover this via REST compatibility. @jakelandis do you concur?