Scripted Update NPE

[dansimpson]

Elasticsearch version (bin/elasticsearch --version): 7.3.2

Plugins installed:
analysis-icu
analysis-kuromoji
analysis-nori
analysis-phonetic
analysis-smartcn
analysis-stconvert
analysis-stempel
ingest-attachment
repository-s3

JVM version (java -version):
1.8.0_181

OS version (uname -a if on a Unix-like system):
Linux ip-172-31-2-219 4.4.0-1074-aws #84-Ubuntu SMP Thu Dec 6 08:57:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux and Others

Description of the problem including expected versus actual behavior:

I've encountered what looks like a bug with scripted updates. Running the following request (on a non-data node) results in the error below:

curl -XPOST "localhost:9200/test/_update/1" -H 'Content-Type: application/json' -d '{
  "upsert": {
    "meta": {
      "version": "0"
    }
  },
  "scripted_upsert": true,
  "script": {
     "source": "if (params.version == ctx._source?.meta?.version) { ctx._source.deleted = true; ctx._source.meta = new HashMap(); } else { ctx.op = '"'"'none'"'"'}",
    "params": {
      "version": "1"
    }
  }
}'

Error:

[2019-10-17T22:17:56,230][WARN ][r.suppressed             ] [ip-172-31-2-219] path: /test/_update/1, params: {index=test, id=1}
java.lang.NullPointerException: null
  at org.elasticsearch.index.get.GetResult.toXContentEmbedded(GetResult.java:262) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.action.update.UpdateResponse.innerToXContent(UpdateResponse.java:96) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.action.DocWriteResponse.toXContent(DocWriteResponse.java:296) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:57) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:33) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:42) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:34) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37) ~[elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleResponse(TransportInstanceSingleOperationAction.java:198) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleResponse(TransportInstanceSingleOperationAction.java:182) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660) [elasticsearch-7.3.2.jar:7.3.2]
  at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) [transport-netty4-client-7.3.2.jar:7.3.2]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

If the command is run on the data node, everything works as expected.

Steps to reproduce:

1. Create a cluster of 1 master and 1 data node
2. Issue the above curl command to the master node
3. Optionally issue the above curl command to the data node to see how it should work

Provide logs (if relevant):
See Above

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

[andreassepp]

The same also happens with version 7.4.0.
@dansimpson do you happen to know the latest version where it still works?

The script can be reduced to just
"source": "ctx.op = "none";"
to reproduce

Edit:
Last working version seems to be 7.2.1

[stu-elastic]

Potentially related to #30356

[stu-elastic]

We need to investigate. It appears to be due to a serialization issue based on the stack trace.

[stu-elastic]

@rjernst debugged this.

We can repro when the document does not exist. When we read in the internal serialization of the result, the boolean value exists is false.

if exists is false, we do not initialize metaFields, so we get an null pointer exception when we try to iterate over metaFields on serialization to the client.

[slovdahl]

What's the actual impact of the bug? Are e.g. all indexing/update actions in a bulk request properly handled even if a single one of them fails like this? I'm just trying to understand how critical it is to update to a fixed version (we're seeing these errors as well).

[rjernst]

@slovdahl Yes, bulk requests should continue processing the rest of the documents if some fail.