[Maps] URL length issues with filter by geometry

[alexfrancoeur]

I'm using 7.4 BC3. I have a feeling this issue has to do with the geo-bounding box in general. Because some of these polygons can be pretty complex, I'd imagine that we'll start to see this error more and more as the feature gets utilized. The error shown below doesn't provide any good options really either. Is there anything we can do here? I'm worried this will dramatically affect the experience of this feature. Every time you zoom in the error continues to pop up as well.

Separately, it feels to me like filtering the geometry should have also filtered out the shape from the layer I was interacting with. In the GIF below, I would have expected all districts to also be filtered out and not just the underlying data. Is there a way we can filter out the points and the joined field as well? This feels less like a bug but more of a UX issue.

Here's what I'm using:
The data set is Get It Done Requests year-to-date csv from here.
You'll need to tweak the mappings and ingest pipeline when uploading via file upload

Add this to the mapping:

"location"{
   "type":"geo_point"
}

and this processor to the pipeline

{
  "processors": [
    {
      "set": {
        "field": "location",
        "value": "{{lat}},{{lng}}"
      }
    }
  ]
}

Due to #45520, I can't seem to make the actual export but here are the original objects I imported from 7.3:
get-it-done_saved_objects.zip

This was the geojson file I uploaded for san diego counties with the sd index pattern:
sd.geojson.zip

cc: @bcamper

[elasticmachine]

Pinging @elastic/kibana-gis

[nreese]

Most Kibana applications, like Dashboard and Maps, put filters in the URL. Spatial filters can get really large for shapes with lots of vertices.

One solution we can implement for shapes indexed in elasticsearch is to use the document id instead of polygon in the filter. This will avoid this problem because regardless of number of vertices, the filter length will be the same.

For spatial filters from shapes not indexed in elasticsearch, the application could round lat/lon to 5 decimal places to limit the serialized string length a good amount. But the problem will always exist for for shapes with lots of vertices.

[bcamper]

If the shape was uploaded via the UI, it should always be indexed and could be referred to by id, right?

What are the other sources of filter shapes -- drawing on the map is one, are there others right now (could imagine a link in from another part of the app that could contain a shape)?

[nreese]

What are the other sources of filter shapes

Layers from EMS Boundaries or Configured GeoJSON sources display shapes that are not indexed in Elasticsearch. In these cases, creating a filter from these shapes has to include the shape in the filter.

[nreese]

There is a problem with using pre-indexed shapes in Elasticsearch. Currently, only geo-shape query supports pre-indexed shapes. Geo-polygon query does not support pre-indexed shapes (geo-polygon is the spatial query used to query geo_point fields). We can not provide a solution until geo-polygon queries support pre-indexed shapes. I created elastic/elasticsearch#46730 to track that issue in Elasticsearch.

[bcamper]

I came across a case where the shape intersect filter appeared to be behaving differently depending on the zoom level I was at when I added the filter. @nreese and I investigated further and found the cause is that the app is grabbing the shape data for the filter from the Mapbox map (not from the underlying ES _source). But Mapbox simplifies data by zoom level, so the filter shape being used is not the actual geometry, but a view-optimized version.

Comparing two filter shapes for the same document field, but selected at two different zoom levels in the UI, you can see that they don't line up, and this discrepancy causes inconsistent intersection results.

We concluded that the best fix for this is the one already identified in this issue above, which is to use the document id for the filter shape.

[nreese]

I came across a case where the shape intersect filter appeared to be behaving differently depending on the zoom level I was at when I added the filter

This can also effect non-elasticsearch sources. For example, when using EMS boundary. If you create a filter while zoomed in really far, the EMS boundary will be tiled and not contain the entire shape, but just the part of the shape in the tile.

There is an open issue in mapbox for getting the original geometry. mapbox/mapbox-gl-js#5639.

Until that gets resolved, what are people's thoughts about storing a copy of the geometry in the feature property so the original geometry can be retrieved?

[bcamper]

Until that gets resolved, what are people's thoughts about storing a copy of the geometry in the feature property so the original geometry can be retrieved?

That's probably workable. Could you not use a feature id (either existing or assigned at run-time) to reference back to the original geometry (we fetch it ourselves anyway before feeding it to MBGL, right?) instead, to avoid extra copies? That tiling happens on a worker thread, so the extra feature data is going to get copied over there too, but this data might be lightweight enough to not have a noticeable impact.

I still think we are going to want to index these eventually, to decouple querying detail from display detail. Right now some of the source data shapes like zip codes don't really hold up past ~z5, which doesn't serve either case well. But you ought to be able to query/filter against the true full-fidelity shape, while only maintaining a zoom-appropriate amount of data in the client (e.g. either querying against a complex shape like a neighborhood border while zoomed way in, or a small shape like a census block while zoomed way out). The shape fidelity hasn't been an issue for querying until now because it was only doing a terms join, not an actual spatial filter.

[nreese]

closed by #62966. Now geo_points can use geo_shape pre-indexed shapes