add top_hit metric

[scampi]

close #6877

Display the first/last value of a field, for a set of documents sorted on some field (e.g., a date).

TODO

• aggregation of multivalued field
• object in array use case
• use AggConfigResult.toString for displaying arrays in the metric vis

Test

Index

Create the weather index with the following mapping:

$ curl -XPUT 'http://localhost:9200/weather' -d '
{
   "mappings": {
      "Weather": {
         "properties": {
            "timestamp": {
               "type": "date",
               "format": "date_optional_time"
            },
            "location": {
               "index": "not_analyzed",
               "type": "string"
            }
         }
      }
   }
}
'

Data

Add a document at a time.

$ curl -XPOST 'http://localhost:9200/weather/Weather/' -d '
{
  "location": "France",
  "temperature": 28,
  "timestamp": "2016-05-26T21:22:18+01:00"
}
'
$ curl -XPOST 'http://localhost:9200/weather/Weather/' -d '
{
  "location": "France",
  "temperature": 23.5,
  "timestamp": "2016-05-26T22:59:04+01:00"
}
'

Visualization

We can see what is the latest recorded temperature in France by doing the following:

Create either a vertical bar or a data table:

• add a Latest metric:
  • "Field": the field which value to display, here temperature
  • "Sort On": the field to sort the data on, here timestamp
• add a terms bucket on the field location.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[panda01]

Jenkins, test it.

[scampi]

In the jenkins logs, it fails with the exception java.lang.IllegalArgumentException: unknown setting [es.path.conf] did you mean [path.conf]?. Do you know what that is ?

[scampi]

With this fix elastic/libesvm@e3fd2ca I am able to run tests on my machine. Thanks @spalger for the fix ;o)

[spalger]

jenkins, test it

[scampi]

jenkins, test it

[panda01]

Only, we (elastic members) can trigger the tests.

[panda01]

Jenkins, test it.

[panda01]

So it seems that this doesn't work with nested fields, through (I believe) no fault of your own, I'm not really sure where to go to fix this either.

[panda01]

Some other images that helped me reach this conclusion.

[panda01]

Also, I'm not a fan this implementation, it is IMO is a bit limiting. I feel like the top_hits agg has other uses and by you only limiting it to order: desc it becomes a latest.

I say we take this away and leave it as kind of a top_hits agg. As an extra we can add if you want to do latest here is how you do it.

[panda01]

Hmm, also I don't think these tests are something you caused. By the time you come back around they should be good to go though.

[scampi]

Thanks for spotting the problem with nested fields.

As an extra we can add if you want to do latest here is how you do it.

I think there is something missing to your comment, since I don't understand that sentence as it is now.

[panda01]

@scampi Nope, I just worded that poorly. What I mean to say is after you add this metric, and remove the limitation on order, you can add steps that people can follow in order to get the latest value.

[scampi]

@panda01 Sorry I am still confused...

Do you want me to provide a combobox with, e.g., two options (desc and asc), so that the order option is not hardcoded ?

Otherwise, can you clarify what those steps are ? For the issue #6877 I need to either sort by descending (or ascending for some use case?) order. From what I read in the sort options, the default sort does not do what is needed for this issue.

[panda01]

Do you want me to provide a combobox with, e.g., two options (desc and asc), so that the order option is not hardcoded ?

Exactly this! Maybe radio boxes though since i hate select boxes.

[scampi]

@panda01 I added a select box for choosing the order instead of radio boxes. I thought it was best to be consistent with the rest of kibana, e.g., the terms bucket aggregation uses a select box for the order. I can revert though.

In addition, the sort field is now by default the time field associated to the index pattern, as I had forgotten it mentioned in the issue.

[scampi]

@panda01 there is anyway another issue about the select box #2130 (comment) where the cool ui-select is mentioned ;o)

[panda01]

@scampi in regards to using chosen, I would say the normal select box is appropriate for now. I will check out your changes.

[panda01]

Hmm, this is a bit nitpicky of me, but I think the sort, and sort order should use the same property names, since they are dependent on one another, and I would hate for someone later to not realize that.

For instance you could call the sort field agg.params.sort.field and the order agg.params.sort.dir or something to that effect. but with that it's clear they're both dealing with the sort, no matter where it's put.

[scampi]

@ppisljar Regarding #7302 (comment) it seems that some documents don't have the memory field from what I saw locally. In your case, it would mean that the last before last has that field, but not the last one.

The last commits add the following:

• added top hits metric to the heatmap.
• fixed min/max of an array of values: it was returning infinity if all values were undefined/null.

Regarding that 2nd point, I would like to know if there is some default "zero" value. For example, if a field is missing:

• if a string, then we'd return "N/A" or something set in the advanced settings
• if a number, then we'd return "0" or "1" or something set in the advanced settings

The PR is now ready for review, but pending the answer to the previous question, some changes might be needed.

[ppisljar]

regarding second point it should probably behave the same as for non-array missing values ? @Bargs whats your opinion ?

[Bargs]

Discover displays a - for any missing fields regardless of type, so I think we should do the same here to remain consistent.

[scampi]

ok then PR is ready ;o) I thought there was such a setting so I wanted to make sure

[ppisljar]

congrats scampi, this is a great PR and i am sure many are excited about finally having top hits in kibana!

[ppisljar]

the backport #9735 has some conflicts you'll need to resolve.

[Bargs]

@scampi thanks for the great work on another giant PR! This is an awesome new metric, users are gonna love it.

[scampi]

Thanks @Bargs and @ppisljar for your help on this ;o)