Skip to content

Commit

Permalink
merge master into feat/discover-elastic-charts
Browse files Browse the repository at this point in the history
  • Loading branch information
Emma Cunningham committed May 21, 2019
2 parents 3f142d2 + ef38c4d commit 8523aed
Show file tree
Hide file tree
Showing 467 changed files with 6,439 additions and 3,447 deletions.
79 changes: 28 additions & 51 deletions docs/dev-tools/searchprofiler/getting-started.asciidoc
Original file line number Diff line number Diff line change
@@ -1,52 +1,22 @@
[role="xpack"]
[[profiler-getting-started]]

ifndef::gs-mini[]
=== Getting Started
endif::gs-mini[]

ifdef::gs-mini[]
== Getting Started
endif::gs-mini[]

The {searchprofiler} is automatically enabled in {kib}. It is located under the
*Dev Tools* tab in {kib}.
The {searchprofiler} is automatically enabled in {kib}. Go to *Dev Tools > Search Profiler*
to get started.

[[first-profile]]
To start profiling queries:
{searchprofiler} displays the names of the indices searched, the shards in each index,
and how long it took for the query to complete. To try it out, replace the default `match_all` query
with the query you want to profile and click *Profile*.

. Open Kibana in your web browser and log in. If you are running Kibana
locally, go to `http://localhost:5601/`.
The following example shows the results of profiling the `match_all` query.
If we take a closer look at the information for the `.kibana_1` sample index, the
Cumulative Time field shows us that the query took 0.067ms to execute.

. Click **DevTools** in the side navigation to open the {searchprofiler}.
Console is the default tool to open when first accessing DevTools.
+
image::dev-tools/searchprofiler/images/gs1.png["Opening DevTools"]
+
On the top navigation bar, click the second item: *Search Profiler*
+
image::dev-tools/searchprofiler/images/gs2.png["Opening the {searchprofiler}"]
[role="screenshot"]
image::dev-tools/searchprofiler/images/query.png["{searchprofiler} example"]

. This opens the {searchprofiler} interface.
+
image::dev-tools/searchprofiler/images/gs3.png["{searchprofiler} Interface"]

. Replace the default `match_all` query with the query you want to profile and click *Profile*.
+
image::dev-tools/searchprofiler/images/gs4.png["Profiling the match_all query"]
+
{searchprofiler} displays the names of the indices searched, the shards in each index,
and how long the query took. The following example shows the results of profiling
the match_all query. Three indices were searched: `.monitoring-kibana-2-2016.11.30`,
`.monitoring-data-2` and `test`.
+
If we take a closer look at the information for the test index, we can see from the
Cumulative Time that the query took 0.132ms to execute. Of the five shards in the
index (`DWZD0iosQNeJMTvb4q1JDw` 0 through 5), shard 3 was the slowest (0.053ms), followed by shard 2 (0.038ms). Shards are
sorted by their time in descending order.
+
image::dev-tools/searchprofiler/images/gs5.png["Profile details for the test index"]
+
[NOTE]
====
The Cumulative Time metric is the sum of individual shard times.
Expand All @@ -59,14 +29,21 @@ While the Cumulative Time metric is useful for comparing the performance of your
indices and shards, it doesn't necessarily represent the actual physical query times.
====

. To view more detailed profiling information for a shard, click the Expand button.
This displays details about the query component(s) that ran on the shard.
+
In this example, there is a single `"MatchAllDocsQuery"` that ran on the shard.
Since it was the only query run, it took 100% of the time. When you mouse over
a row, the {searchprofiler} displays additional information about the query component."
+
image::dev-tools/searchprofiler/images/gs6.png["Profile details for the first shard"]
+
This panel shows the timing breakdown of low-level Lucene methods. For more information,
see the reference docs for timing breakdowns in {ref}/search-profile-queries.html[Profiling queries].
You can select the name of the shard and then click *View details* to see more profiling information,
including details about the query component(s) that ran on the shard, as well as the timing
breakdown of low-level Lucene methods. For more information, see {ref}/search-profile-queries.html[Profiling queries].

[float]
=== Index and type filtering

By default, all queries executed by the {searchprofiler} are sent
to `GET /_search`. It searches across your entire cluster (all indices, all types).

If you need to query a specific index or type (or several), you can use the Index
and Type filters at the top left.

In the following example, the query is executed against the indices `test` and `kibana_1`
and the type `my_type`. This is equivalent making a request to `GET /test,kibana_1/my_type/_search`.

[role="screenshot"]
image::dev-tools/searchprofiler/images/filter.png["Filtering by index and type"]
12 changes: 6 additions & 6 deletions docs/dev-tools/searchprofiler/gs-index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
[role="xpack"]
[[xpack-profiler]]
= Profiling your Queries and Aggregations
= Profiling queries and aggregations

[partintro]
--
Elasticsearch has a powerful profiler API which can be used to inspect and analyze
your search queries. The response, however, is a very large JSON blob which is
difficult to analyze by hand.
{es} has a powerful {ref}/search-profile.html[Profile API] which can be used to inspect and analyze
your search queries. The response returns a large JSON blob, which can be
difficult to analyze manually.

The {searchprofiler} tool can transform this JSON output
into a visualization that is easy to navigate, allowing you to diagnose and debug
poorly performing queries much faster.


image::dev-tools/searchprofile/images/overview.png["{searchprofiler} Visualization"]
[role="screenshot"]
image::dev-tools/searchprofiler/images/overview.png["{searchprofiler} Visualization"]

--

Expand Down
Binary file modified docs/dev-tools/searchprofiler/images/filter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/dev-tools/searchprofiler/images/gs1.png
Binary file not shown.
Binary file modified docs/dev-tools/searchprofiler/images/gs10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/dev-tools/searchprofiler/images/gs2.png
Binary file not shown.
Binary file removed docs/dev-tools/searchprofiler/images/gs3.png
Binary file not shown.
Binary file removed docs/dev-tools/searchprofiler/images/gs4.png
Binary file not shown.
Binary file removed docs/dev-tools/searchprofiler/images/gs5.png
Binary file not shown.
Binary file removed docs/dev-tools/searchprofiler/images/gs6.png
Binary file not shown.
Binary file removed docs/dev-tools/searchprofiler/images/gs7.png
Binary file not shown.
Binary file modified docs/dev-tools/searchprofiler/images/gs8.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/dev-tools/searchprofiler/images/gs9.png
Binary file not shown.
Binary file modified docs/dev-tools/searchprofiler/images/overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/dev-tools/searchprofiler/images/pasting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/dev-tools/searchprofiler/images/query.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 4 additions & 6 deletions docs/dev-tools/searchprofiler/index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
[role="xpack"]
[[xpack-profiler]]
== Profiling your Queries and Aggregations
== Profiling queries and aggregations

Elasticsearch has a powerful profiler API which can be used to inspect and analyze
your search queries. The response, however, is a very large JSON blob which is
difficult to analyze by hand.
{es} has a powerful {ref}/search-profile.html[Profile API] which can be used to inspect and analyze
your search queries. The response returns a large JSON blob, which can be
difficult to analyze manually.

The {searchprofiler} tool can transform this JSON output
into a visualization that is easy to navigate, allowing you to diagnose and debug
Expand All @@ -18,5 +18,3 @@ include::getting-started.asciidoc[]
include::more-complicated.asciidoc[]

include::pasting.asciidoc[]

include::misc.asciidoc[]
16 changes: 0 additions & 16 deletions docs/dev-tools/searchprofiler/misc.asciidoc

This file was deleted.

59 changes: 20 additions & 39 deletions docs/dev-tools/searchprofiler/more-complicated.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@
To understand how the query trees are displayed inside the {searchprofiler},
let's look at a more complicated query.

. Index the following data:
. Index the following data via *Console*:
+
--
[source,js]
--------------------------------------------------
POST test/test/_bulk
POST test/_bulk
{"index":{}}
{"name":"aaron","age":23,"hair":"brown"}
{"index":{}}
Expand All @@ -25,16 +25,11 @@ POST test/test/_bulk
// CONSOLE
--

. Enter "test" in the Index filter above the query editor (the input box with a
grayed-out `_all`). This restricts profiled queries to the `test` index.
+
--
image::dev-tools/searchprofiler/images/gs7.png["Using the index filter"]
--
. From the {searchprofiler}, enter "test" in the Index field above the query editor to restrict profiled
queries to the `test` index.

. Replace the default `match_all` query with a query that has two sub-query
components and includes a simple aggregation. For example, copy and paste
the following query into the query editor.
. Replace the default `match_all` query in the query editor with a query that has two sub-query
components and includes a simple aggregation, like the example below.
+
--
[source,js]
Expand Down Expand Up @@ -71,52 +66,38 @@ the following query into the query editor.
// NOTCONSOLE
--

. Click *Profile* to profile the query and visualize the results.
. Click *Profile* to profile the query and visualize the results.
. Select the shard to view the query details.
+
--
[role="screenshot"]
image::dev-tools/searchprofiler/images/gs8.png["Profiling the more complicated query"]

As before, you'll see a list of shards appear in the center panel. You'll notice
that the query was slightly slower (15ms vs 0.13ms) because it actually had to do
a bit of work this time, unlike the `match_all` query.
--

. Click the first shard's Expand button to view the query details.
+
--
image::dev-tools/searchprofiler/images/gs9.png["Drilling into the first shard's details"]

--

You'll notice several interesting things in the results. The shard details
contain a row for each query component:
The detail view contains a row for each query component:

- The top-level `BooleanQuery` component corresponds to the bool in the query.
- The second `BooleanQuery` corresponds to the terms query, which is internally
converted to a `Boolean` of should clauses. It has two child queries that correspond
to "sue" and "sally" from the terms query.
- The `TermQuery` that's labeled with _"name:fred"_ corresponds to match: fred in the query.
- The `TermQuery` that's labeled with "name:fred" corresponds to match: fred in the query.

If you look at the timings, you can see that "Self Time" and "Total Time" are no longer
If you look at the time columns, you can see that "Self time" and "Total time" are no longer
identical on all the rows. Self time represents how long the query component took to execute.
Total time is the time a query component _and all its children_ took to execute.
Therefore, queries like the Boolean queries often have larger Total than Self.

In particular, you can see that the `BooleanQuery` for "name:sue name:sally" took 3.8ms total,
but 2.8ms of that was "self" time. That means 2.8ms was spent by the `BooleanQuery` itself, processing
the should clauses
Total time is the time a query component and all its children took to execute.
Therefore, queries like the Boolean queries often have a larger total time than self time.


==== Aggregations

This particular query also includes a aggregation (a `stats` agg on the `"age"` field).
To view Aggregation profiling statistics, click the *Aggregation Profile* tab. This tab
is only enabled if the query being profiled contains an aggregation.
Click *Aggregation Profile* to view aggregation profiling statistics (this tab
is only enabled if the query being profiled contains an aggregation).

image::dev-tools/searchprofiler/images/gs10.png["Drilling into the first shard's details"]

Click a shard's Expand button to view the aggregation details. Hover over an
aggregation row to view the timing breakdown.
Select the name of the shard to view the aggregation details and timing breakdown.

[role="screenshot"]
image::dev-tools/searchprofiler/images/gs10.png["Drilling into the first shard's details"]

For more information about how the {searchprofiler} works, how timings are calculated, and
how to interpret various results, see
Expand Down
13 changes: 7 additions & 6 deletions docs/dev-tools/searchprofiler/pasting.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,18 @@
The {searchprofiler} queries the cluster that the Kibana node is attached to.
It does this by executing the query against the cluster and collecting the results.

This is convenient, but sometimes performance problems are temporal in nature. For example,
a query might only be slow at certain time of day when many customers are using your system.
But sometimes you may want to investigate performance problems that are temporal in nature.
For example, a query might only be slow at certain time of day when many customers are using your system.
You can setup a process to automatically profile slow queries when they occur and then
save those profile responses for later analysis.

The {searchprofiler} supports this workflow by enabling you to paste the
pre-captured JSON. The tool will detect that this is a profiler response JSON
rather than a query, and render the visualization rather than querying the cluster.
The {searchprofiler} supports this workflow by allowing you to paste the
pre-captured JSON in the query editor. The {searchprofiler} will detect that you
have entered a JSON response (rather than a query) and will just render the visualization,
rather than querying the cluster.

To see how this works, copy and paste the following profile response into the
query editor and click *Profile*:
query editor and click *Profile*.

[source,js]
--------------------------------------------------
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Loading

0 comments on commit 8523aed

Please sign in to comment.