Skip to content

Commit

Permalink
[DOCS] Custom routing for data streams
Browse files Browse the repository at this point in the history
  • Loading branch information
danhermann committed Nov 10, 2021
1 parent 0aa9767 commit 0d21b03
Show file tree
Hide file tree
Showing 4 changed files with 43 additions and 36 deletions.
53 changes: 27 additions & 26 deletions docs/reference/docs/bulk.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<titleabbrev>Bulk</titleabbrev>
++++

Performs multiple indexing or delete operations in a single API call.
Performs multiple indexing or delete operations in a single API call.
This reduces overhead and can greatly increase indexing speed.

[source,console]
Expand Down Expand Up @@ -71,33 +71,33 @@ optional_source\n
--------------------------------------------------
// NOTCONSOLE

The `index` and `create` actions expect a source on the next line,
and have the same semantics as the `op_type` parameter in the standard index API:
The `index` and `create` actions expect a source on the next line,
and have the same semantics as the `op_type` parameter in the standard index API:
`create` fails if a document with the same ID already exists in the target,
`index` adds or replaces a document as necessary.
`index` adds or replaces a document as necessary.

NOTE: <<data-streams,Data streams>> support only the `create` action. To update
or delete a document in a data stream, you must target the backing index
containing the document. See <<update-delete-docs-in-a-backing-index>>.

`update` expects that the partial doc, upsert,
`update` expects that the partial doc, upsert,
and script and its options are specified on the next line.

`delete` does not expect a source on the next line and
`delete` does not expect a source on the next line and
has the same semantics as the standard delete API.

[NOTE]
====
The final line of data must end with a newline character `\n`.
The final line of data must end with a newline character `\n`.
Each newline character may be preceded by a carriage return `\r`.
When sending NDJSON data to the `_bulk` endpoint, use a `Content-Type` header of
`application/json` or `application/x-ndjson`.
====

Because this format uses literal `\n`'s as delimiters,
Because this format uses literal `\n`'s as delimiters,
make sure that the JSON actions and sources are not pretty printed.

If you provide a `<target>` in the request path,
If you provide a `<target>` in the request path,
it is used for any actions that don't explicitly specify an `_index` argument.

A note on the format: The idea here is to make processing of this as
Expand All @@ -109,10 +109,10 @@ Client libraries using this protocol should try and strive to do
something similar on the client side, and reduce buffering as much as
possible.

There is no "correct" number of actions to perform in a single bulk request.
There is no "correct" number of actions to perform in a single bulk request.
Experiment with different settings to find the optimal size for your particular workload.

When using the HTTP API, make sure that the client does not send HTTP chunks,
When using the HTTP API, make sure that the client does not send HTTP chunks,
as this will slow things down.

[discrete]
Expand Down Expand Up @@ -192,8 +192,9 @@ Each bulk item can include the routing value using the
`routing` field. It automatically follows the behavior of the
index / delete operation based on the `_routing` mapping.

NOTE: Data streams do not support custom routing. Instead, target the
appropriate backing index for the stream.
NOTE: Data streams do not support custom routing unless they were created with
the <<put-index-template-api-request-body,`allow_custom_routing`>> setting
enabled in the template.

[discrete]
[[bulk-wait-for-active-shards]]
Expand Down Expand Up @@ -271,13 +272,13 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards

[[bulk-api-request-body]]
==== {api-request-body-title}
The request body contains a newline-delimited list of `create`, `delete`, `index`,
The request body contains a newline-delimited list of `create`, `delete`, `index`,
and `update` actions and their associated source data.

`create`::
(Optional, string)
(Optional, string)
Indexes the specified document if it does not already exist.
The following line must contain the source data to be indexed.
The following line must contain the source data to be indexed.
+
--
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
Expand All @@ -290,7 +291,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-dynamic-templates
--

`delete`::
(Optional, string)
(Optional, string)
Removes the specified document from the index.
+
--
Expand All @@ -303,10 +304,10 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
--

`index`::
(Optional, string)
Indexes the specified document.
(Optional, string)
Indexes the specified document.
If the document exists, replaces the document and increments the version.
The following line must contain the source data to be indexed.
The following line must contain the source data to be indexed.
+
--
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
Expand All @@ -319,8 +320,8 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-dynamic-templates
--

`update`::
(Optional, string)
Performs a partial document update.
(Optional, string)
Performs a partial document update.
The following line must contain the partial document and update options.
+
--
Expand All @@ -333,14 +334,14 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
--

`doc`::
(Optional, object)
(Optional, object)
The partial document to index.
Required for `update` operations.
Required for `update` operations.

`<fields>`::
(Optional, object)
(Optional, object)
The document source to index.
Required for `create` and `index` operations.
Required for `create` and `index` operations.

[role="child_attributes"]
[[bulk-api-response-body]]
Expand Down
5 changes: 3 additions & 2 deletions docs/reference/docs/index_.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -346,8 +346,9 @@ additional document parsing pass. If the `_routing` mapping is defined
and set to be `required`, the index operation will fail if no routing
value is provided or extracted.

NOTE: Data streams do not support custom routing. Instead, target the
appropriate backing index for the stream.
NOTE: Data streams do not support custom routing unless they were created with
the <<put-index-template-api-request-body,`allow_custom_routing`>> setting
enabled in the template.

[discrete]
[[index-distributed]]
Expand Down
14 changes: 9 additions & 5 deletions docs/reference/indices/put-index-template.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Creates or updates an index template. Index templates define
<<index-modules-settings,settings>>, <<mapping,mappings>>, and <<aliases,aliases>>
that can be applied automatically to new indices.
that can be applied automatically to new indices.

[source,console]
--------------------------------------------------
Expand Down Expand Up @@ -51,7 +51,7 @@ privilege>> to use this API.
{es} applies templates to new indices based on an
wildcard pattern that matches the index name.

Index templates are applied during data stream or index creation.
Index templates are applied during data stream or index creation.
For data streams, these settings and mappings are applied when the stream's backing indices are created.

Settings and mappings specified in a <<indices-create-index, create index>>
Expand Down Expand Up @@ -106,6 +106,10 @@ See <<create-index-template,create an index template>>.
`hidden`::
(Optional, Boolean) If `true`, the data stream is <<hidden,hidden>>. Defaults to
`false`.
`allow_custom_routing`::
(Optional, Boolean) If `true`, the data stream supports
<<mapping-routing-field,custom routing>>. Defaults to `false`.
====

`index_patterns`::
Expand Down Expand Up @@ -194,7 +198,7 @@ actual index name that the template gets applied to, during index creation.
[[multiple-templates]]
===== Multiple matching templates

If multiple index templates match the name of a new index or data stream,
If multiple index templates match the name of a new index or data stream,
the template with the highest priority is used. For example:

[source,console]
Expand Down Expand Up @@ -240,7 +244,7 @@ identical priorities.
[[versioning-templates]]
===== Template versioning

You can use the `version` parameter to add a version number to an index template.
You can use the `version` parameter to add a version number to an index template.
External systems can use these version numbers to simplify template management.

The `version` parameter is optional and not automatically generated or used by {es}.
Expand All @@ -267,7 +271,7 @@ To check the `version`, you can use the <<indices-get-template, get index templa
[[template-metadata]]
===== Template metadata

You can use the `_meta` parameter to add arbitrary metadata to an index template.
You can use the `_meta` parameter to add arbitrary metadata to an index template.
This user-defined object is stored in the cluster state,
so keeping it short is preferable.

Expand Down
7 changes: 4 additions & 3 deletions docs/reference/mapping/fields/routing-field.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

A document is routed to a particular shard in an index using the following
formulas:

routing_factor = num_routing_shards / num_primary_shards
shard_num = (hash(_routing) % num_routing_shards) / routing_factor

Expand Down Expand Up @@ -48,8 +48,9 @@ GET my-index-000001/_search

<1> Querying on the `_routing` field (also see the <<query-dsl-ids-query,`ids` query>>)

NOTE: Data streams do not support custom routing. Instead, target the
appropriate backing index for the stream.
NOTE: Data streams do not support custom routing unless they were created with
the <<put-index-template-api-request-body,`allow_custom_routing`>> setting
enabled in the template.

==== Searching with custom routing

Expand Down

0 comments on commit 0d21b03

Please sign in to comment.