[Bug]: [benchmark][standalone] Milvus standalone restart: SIGSEGV: segmentation violation in concurrent dql & upsert scene

[wangting0128]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20240816-2736a8b8-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-concurrent-qvs7s
test case name: test_bitmap_locust_dql_dml_upsert_standalone

server:

[2024-08-16 11:16:42,169 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|bitmap-dql-dml-upsert-milvus|bitmap-dql-dml-upsert-minio|bitmap-dql-dml-upsert-etcd|bitmap-dql-dml-upsert-pulsar|bitmap-dql-dml-upsert-zookeeper|bitmap-dql-dml-upsert-kafka|bitmap-dql-dml-upsert-log|bitmap-dql-dml-upsert-tikv'  (util_cmd.py:14)
[2024-08-16 11:17:02,571 -  INFO - fouram]: [CliClient] pod details of release(bitmap-dql-dml-upsert): 
 I0816 11:16:43.410546     381 request.go:665] Waited for 1.170312669s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/source.toolkit.fluxcd.io/v1?timeout=32s
I0816 11:16:53.610118     381 request.go:665] Waited for 4.197159647s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/scheduling.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
bitmap-dql-dml-upsert-etcd-0                                      1/1     Running     0               3m30s   10.104.30.143   4am-node38   <none>           <none>
bitmap-dql-dml-upsert-milvus-standalone-775f66cc44-nrxzk          1/1     Running     1 (2m9s ago)    3m30s   10.104.18.221   4am-node25   <none>           <none>
bitmap-dql-dml-upsert-minio-86f79657f7-6k7dx                      1/1     Running     0               3m30s   10.104.18.222   4am-node25   <none>           <none>

NAME                                                              READY   STATUS             RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
bitmap-dql-dml-upsert-etcd-0                                      1/1     Running            0                 2d16h   10.104.30.143   4am-node38   <none>           <none>
bitmap-dql-dml-upsert-milvus-standalone-775f66cc44-nrxzk          1/1     Running            2 (2d16h ago)     2d16h   10.104.18.221   4am-node25   <none>           <none>
bitmap-dql-dml-upsert-minio-86f79657f7-6k7dx                      1/1     Running            0                 2d16h   10.104.18.222   4am-node25   <none>           <none>

bitmap-dql-dml-upsert-milvus-standalone-775f66cc44-nrxzk_restart.log

client pod name: fouramf-concurrent-qvs7s-3328890162
client log:

[2024-08-16 11:17:03,200 -  INFO - fouram]: [Base] Collection schema: 
{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}},
            {'name': 'sparse_float_vector', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}, {'name': 'int8_1', 'description': '', 'type': <DataType.INT8: 2>},
            {'name': 'int16_1', 'description': '', 'type': <DataType.INT16: 3>}, {'name': 'int32_1', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'int64_1', 'description': '', 'type': <DataType.INT64: 5>},
            {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}}, {'name': 'bool_1', 'description': '', 'type': <DataType.BOOL: 1>},
            {'name': 'array_int8_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT8: 2>},
            {'name': 'array_int16_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT16: 3>},
            {'name': 'array_int32_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT32: 4>},
            {'name': 'array_int64_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT64: 5>},
            {'name': 'array_varchar_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 100, 'max_capacity': 13}, 'element_type': <DataType.VARCHAR: 21>},
            {'name': 'array_bool_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.BOOL: 1>}],
 'enable_dynamic_field': False} (base.py:328)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64`, shards_num=16
            1. building `BITMAP` index on all supported 12 scalar fields
            2. 2 fields of different vector types
            3. verify DQL & DML(upsert) requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=13
            2. build indexes:
                IVF_SQ8: 'float_vector'
                SPARSE_WAND: 'sparse_float_vector'
                BITMAP: all scalar fields
            3. insert 10w data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - search
                - query
                - hybrid_search
                - load
                - upsert: batch=10
                - flush: ignore RateLimiter

Milvus Log

No response

Anything else?

test config:

{"dataset_params":{"metric_type":"L2","dim":128,"max_length":100,"scalars_index":{"int8_1":{"index_type":"BITMAP"},"int16_1":{"index_type":"BITMAP"},"int32_1":{"index_type":"BITMAP"},"int64_1":{"index_type":"BITMAP"},"varchar_1":{"index_type":"BITMAP"},"bool_1":{"index_type":"BITMAP"},"array_int8_1":{"index_type":"BITMAP"},"array_int16_1":{"index_type":"BITMAP"},"array_int32_1":{"index_type":"BITMAP"},"array_int64_1":{"index_type":"BITMAP"},"array_varchar_1":{"index_type":"BITMAP"},"array_bool_1":{"index_type":"BITMAP"}},"vectors_index":{"sparse_float_vector":{"index_type":"SPARSE_INVERTED_INDEX","index_param":{"drop_ratio_build":0.2},"metric_type":"IP"}},"scalars_params":{"array_int8_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-128,128],"max_capacity":13}}},"array_int16_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-200,200],"max_capacity":13}}},"array_int32_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"specify_scope","specify_range":[-300,300],"max_capacity":13}}},"array_int64_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"fixed_value_range","specify_range":[-400,432],"batch":50,"max_capacity":13}}},"array_varchar_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-1500,1500],"max_capacity":13}}},"array_bool_1":{"params":{"max_capacity":13}},"int8_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-128,128],"max_capacity":13}}},"int16_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-200,200],"max_capacity":13}}},"int32_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"specify_scope","specify_range":[-300,300],"max_capacity":13}}},"int64_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"fixed_value_range","specify_range":[-400,432],"batch":50,"max_capacity":13}}},"varchar_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-1500,1500],"max_capacity":13}}}},"dataset_name":"sift","dataset_size":100000,"ni_per":5000},"collection_params":{"other_fields":["sparse_float_vector","int8_1","int16_1","int32_1","int64_1","varchar_1","bool_1","array_int8_1","array_int16_1","array_int32_1","array_int64_1","array_varchar_1","array_bool_1"],"shards_num":16},"resource_groups_params":{"reset":false},"database_user_params":{"reset_rbac":false,"reset_db":false},"index_params":{"index_type":"IVF_SQ8","index_param":{"nlist":1024}},"concurrent_params":{"concurrent_number":[30],"during_time":"3h","interval":20},"concurrent_tasks":[{"type":"search","weight":1,"params":{"nq":1000,"top_k":10,"search_param":{"nprobe":16},"expr":"int8_1 == 100","guarantee_timestamp":null,"partition_names":null,"output_fields":["id","float_vector","int64_1"],"ignore_growing":false,"group_by_field":null,"timeout":null,"random_data":true,"check_task":"check_search_output","check_items":null}},{"type":"query","weight":1,"params":{"ids":null,"expr":"int64_1 > -1","output_fields":["*"],"offset":null,"limit":10,"ignore_growing":false,"partition_names":null,"timeout":null,"random_data":false,"random_count":0,"random_range":[0,1],"field_name":"id","field_type":"int64","check_task":"check_query_output","check_items":{"expect_length":10}}},{"type":"hybrid_search","weight":1,"params":{"nq":10,"top_k":10,"reqs":[{"search_param":{"nprobe":128},"anns_field":"float_vector","expr":"(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, 1)) || ((varchar_1 like \"1%\") and (bool_1 == True))","top_k":100},{"search_param":{"drop_ratio_search":0.1},"anns_field":"sparse_float_vector","expr":"not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, 1])"}],"rerank":{"RRFRanker":[]},"output_fields":["*"],"ignore_growing":false,"guarantee_timestamp":null,"partition_names":null,"timeout":null,"random_data":true,"check_task":"check_search_output","check_items":{"output_fields":["sparse_float_vector","int8_1","int16_1","int32_1","int64_1","varchar_1","bool_1","array_int8_1","array_int16_1","array_int32_1","array_int64_1","array_varchar_1","array_bool_1","id","float_vector"],"nq":10}}},{"type":"load","weight":1,"params":{"replica_number":1,"timeout":180,"check_task":"check_response","check_items":null}},{"type":"upsert","weight":1,"params":{"nb":10,"timeout":30,"random_id":true,"random_vector":true,"varchar_filled":false,"start_id":100000,"check_task":"check_response","check_items":null}},{"type":"flush","weight":1,"params":{"timeout":180,"check_task":"check_ignore_rate_limit","check_items":null}}]}

test result:

[2024-08-16 14:21:16,057 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-08-16 14:21:16,058 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     flush                                                                           4430    17(0.38%) |  66633       0  190606  22000 |    0.41        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     hybrid_search                                                                   4478     0(0.00%) |   1643      32   80459    560 |    0.41        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     load                                                                            4518     0(0.00%) |    549       4   99765    180 |    0.42        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     query                                                                           4511     0(0.00%) |    790       8  108806     99 |    0.42        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     search                                                                          4435     0(0.00%) |   2011     222  128948    740 |    0.41        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     upsert                                                                          4361     4(0.09%) |    191       7   30697     25 |    0.40        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]:          Aggregated                                                                     26733    21(0.08%) |  11908       0  190606    440 |    2.48        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]:  (stats.py:790)
[2024-08-16 14:21:16,062 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_16c64m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '16.0', 'memory': '64Gi'}, 'requests': {'cpu': '9.0', 'memory': '33Gi'}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1, 'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone', 'metrics': {'podMonitor': {'enabled': True}}},
                       'pulsar': {'enabled': False},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20240816-2736a8b8-amd64'}}},
            'host': 'bitmap-dql-dml-upsert-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_bitmap_locust_dql_dml_upsert_standalone',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'max_length': 100,
                                                    'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
                                                                      'int16_1': {'index_type': 'BITMAP'},
                                                                      'int32_1': {'index_type': 'BITMAP'},
                                                                      'int64_1': {'index_type': 'BITMAP'},
                                                                      'varchar_1': {'index_type': 'BITMAP'},
                                                                      'bool_1': {'index_type': 'BITMAP'},
                                                                      'array_int8_1': {'index_type': 'BITMAP'},
                                                                      'array_int16_1': {'index_type': 'BITMAP'},
                                                                      'array_int32_1': {'index_type': 'BITMAP'},
                                                                      'array_int64_1': {'index_type': 'BITMAP'},
                                                                      'array_varchar_1': {'index_type': 'BITMAP'},
                                                                      'array_bool_1': {'index_type': 'BITMAP'}},
                                                    'vectors_index': {'sparse_float_vector': {'index_type': 'SPARSE_INVERTED_INDEX',
                                                                                              'index_param': {'drop_ratio_build': 0.2},
                                                                                              'metric_type': 'IP'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
                                                                                        'other_params': {'dataset': 'random_algorithm',
                                                                                                         'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                              'specify_range': [-128, 128],
                                                                                                                              'max_capacity': 13}}},
                                                                       'array_int16_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                               'specify_range': [-200, 200],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int32_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                               'specify_range': [-300, 300],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int64_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                               'specify_range': [-400, 432],
                                                                                                                               'batch': 50,
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 13},
                                                                                           'other_params': {'dataset': 'random_algorithm',
                                                                                                            'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                                 'specify_range': [-1500, 1500],
                                                                                                                                 'max_capacity': 13}}},
                                                                       'array_bool_1': {'params': {'max_capacity': 13}},
                                                                       'int8_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                   'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                        'specify_range': [-128, 128],
                                                                                                                        'max_capacity': 13}}},
                                                                       'int16_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                         'specify_range': [-200, 200],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int32_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                         'specify_range': [-300, 300],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int64_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                         'specify_range': [-400, 432],
                                                                                                                         'batch': 50,
                                                                                                                         'max_capacity': 13}}},
                                                                       'varchar_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                      'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                           'specify_range': [-1500, 1500],
                                                                                                                           'max_capacity': 13}}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 100000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['sparse_float_vector', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
                                                                        'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
                                                                        'array_bool_1'],
                                                       'shards_num': 16},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 30, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 10,
                                                                  'search_param': {'nprobe': 16},
                                                                  'expr': 'int8_1 == 100',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['id', 'float_vector', 'int64_1'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': None,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': 10,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': None,
                                                                  'random_data': False,
                                                                  'random_count': 0,
                                                                  'random_range': [0, 1],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'check_task': 'check_query_output',
                                                                  'check_items': {'expect_length': 10}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': '(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, '
                                                                                    '1)) || ((varchar_1 like "1%") and (bool_1 == True))',
                                                                            'top_k': 100},
                                                                           {'search_param': {'drop_ratio_search': 0.1},
                                                                            'anns_field': 'sparse_float_vector',
                                                                            'expr': 'not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, '
                                                                                    '1])'}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': None,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'output_fields': ['sparse_float_vector', 'int8_1', 'int16_1', 'int32_1',
                                                                                                    'int64_1', 'varchar_1', 'bool_1', 'array_int8_1',
                                                                                                    'array_int16_1', 'array_int32_1', 'array_int64_1',
                                                                                                    'array_varchar_1', 'array_bool_1', 'id', 'float_vector'],
                                                                                  'nq': 10}}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1, 'timeout': 180, 'check_task': 'check_response', 'check_items': None}},
                                                      {'type': 'upsert',
                                                       'weight': 1,
                                                       'params': {'nb': 10,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 100000,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 180, 'check_task': 'check_ignore_rate_limit', 'check_items': None}}]},
            'run_id': 2024081668108011,
            'datetime': '2024-08-16 11:13:30.200652',
            'client_version': '2.5'},
 'result': {'test_result': {'index': {'RT': 220.5692,
                                      'sparse_float_vector': {'RT': 0.5159},
                                      'int8_1': {'RT': 0.5732},
                                      'int16_1': {'RT': 0.5125},
                                      'int32_1': {'RT': 0.513},
                                      'int64_1': {'RT': 0.5137},
                                      'varchar_1': {'RT': 0.5117},
                                      'bool_1': {'RT': 0.5127},
                                      'array_int8_1': {'RT': 0.5181},
                                      'array_int16_1': {'RT': 0.5122},
                                      'array_int32_1': {'RT': 0.5109},
                                      'array_int64_1': {'RT': 0.5102},
                                      'array_varchar_1': {'RT': 0.5122},
                                      'array_bool_1': {'RT': 0.51}},
                            'insert': {'total_time': 10.0554, 'VPS': 9944.9052, 'batch_time': 0.5028, 'batch': 5000},
                            'flush': {'RT': 3.0719},
                            'load': {'RT': 1.6284},
                            'Locust': {'Aggregated': {'Requests': 26733,
                                                      'Fails': 21,
                                                      'RPS': 2.48,
                                                      'fail_s': 0.0,
                                                      'RT_max': 190606.14,
                                                      'RT_avg': 11908.43,
                                                      'TP50': 440.0,
                                                      'TP99': 182000.0},
                                       'flush': {'Requests': 4430,
                                                 'Fails': 17,
                                                 'RPS': 0.41,
                                                 'fail_s': 0.0,
                                                 'RT_max': 190606.14,
                                                 'RT_avg': 66633.72,
                                                 'TP50': 22000.0,
                                                 'TP99': 183000.0},
                                       'hybrid_search': {'Requests': 4478,
                                                         'Fails': 0,
                                                         'RPS': 0.41,
                                                         'fail_s': 0.0,
                                                         'RT_max': 80459.57,
                                                         'RT_avg': 1643.12,
                                                         'TP50': 560.0,
                                                         'TP99': 12000.0},
                                       'load': {'Requests': 4518,
                                                'Fails': 0,
                                                'RPS': 0.42,
                                                'fail_s': 0.0,
                                                'RT_max': 99765.71,
                                                'RT_avg': 549.06,
                                                'TP50': 180.0,
                                                'TP99': 5300.0},
                                       'query': {'Requests': 4511,
                                                 'Fails': 0,
                                                 'RPS': 0.42,
                                                 'fail_s': 0.0,
                                                 'RT_max': 108806.46,
                                                 'RT_avg': 790.28,
                                                 'TP50': 99,
                                                 'TP99': 8800.0},
                                       'search': {'Requests': 4435,
                                                  'Fails': 0,
                                                  'RPS': 0.41,
                                                  'fail_s': 0.0,
                                                  'RT_max': 128948.58,
                                                  'RT_avg': 2011.3,
                                                  'TP50': 740.0,
                                                  'TP99': 13000.0},
                                       'upsert': {'Requests': 4361,
                                                  'Fails': 4,
                                                  'RPS': 0.4,
                                                  'fail_s': 0.0,
                                                  'RT_max': 30697.33,
                                                  'RT_avg': 191.92,
                                                  'TP50': 25,
                                                  'TP99': 2100.0}}}}}

[yanliang567]

/assign @zhagnlu
/unassign

[cydrain]

/assign

[cydrain]

runtime.gcResetMarkState
	/usr/local/go/src/runtime/mgc.go:1620 pc=0x1eb0f8f


SIGSEGV: segmentation violation
PC=0x1eb0f8f m=6931 sigcode=1

goroutine 0 [idle]:
runtime.gcResetMarkState()
	/usr/local/go/src/runtime/mgc.go:1620 +0x6f fp=0x7f10dd8bb850 sp=0x7f10dd8bb820 pc=0x1eb0f8f
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f10dd8bb860 sp=0x7f10dd8bb850 pc=0x1f0548a

goroutine 2153222 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0012177e0 sp=0xc0012177d0 pc=0x1f05428
runtime.gcStart({0x1fc0?, 0x0?, 0x0?})
	/usr/local/go/src/runtime/mgc.go:669 +0x235 fp=0xc001217878 sp=0xc0012177e0 pc=0x1eaee15
runtime.GC()
	/usr/local/go/src/runtime/mgc.go:466 +0x3e fp=0xc0012178b0 sp=0xc001217878 pc=0x1eae91e
runtime/debug.freeOSMemory()
	/usr/local/go/src/runtime/mheap.go:1672 +0x13 fp=0xc0012178c8 sp=0xc0012178b0 pc=0x1f00093
runtime/debug.FreeOSMemory()
	/usr/local/go/src/runtime/debug/garbage.go:104 +0xf fp=0xc0012178d8 sp=0xc0012178c8 pc=0x23b2b6f
github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).LoadSegment(0xc002dcc500, {0x616d4a8, 0xc00ab5cf60}, {0x61f47e0?, 0xc0114ca000?}, 0xc01005b7a0)
	/workspace/source/internal/querynodev2/segments/segment_loader.go:733 +0xa67 fp=0xc001217be0 sp=0xc0012178d8 pc=0x4df2a87
github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4(0xe5cf1d2afc8417?)
	/workspace/source/internal/querynodev2/segments/segment_loader.go:302 +0x58a fp=0xc001217e00 sp=0xc001217be0 pc=0x4dec56a
github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3(0x0)
	/workspace/source/pkg/util/funcutil/parallel.go:86 +0x151 fp=0xc001217fc8 sp=0xc001217e00 pc=0x2bbf5f1
github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func4()
	/workspace/source/pkg/util/funcutil/parallel.go:104 +0x27 fp=0xc001217fe0 sp=0xc001217fc8 pc=0x2bbf467
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001217fe8 sp=0xc001217fe0 pc=0x1f07401
created by github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel in goroutine 2153359
	/workspace/source/pkg/util/funcutil/parallel.go:73 +0x1b4

[ThreadDao]

I guess my problem is the same as this one.

• image: master-20240819-14ef88f4-amd64
• SIGV error log:
  stand_b4p85.log

• pods:

zong-go-latest-etcd-0                                             1/1     Running     0                 2d19h   10.104.24.58    4am-node29   <none>           <none>
zong-go-latest-milvus-standalone-767bd55f6b-b4p85                 1/1     Running     2 (22h ago)       2d19h   10.104.24.62    4am-node29   <none>           <none>
zong-go-latest-minio-865844ffdb-w6rr8                             1/1     Running     0                 2d19h   10.104.24.59    4am-node29   <none>           <none>

[xiaofan-luan]

this seems to be related with sparse embedding index

@zhengbuqian could you please take a look at it

[zhengbuqian]

@ThreadDao this issue is about a different sigsegv, they have different stacktrace.

[wangting0128]

The sparse vector written in this test case has no null value

[zhengbuqian]

/assign @pingliu

assigning to pingliu for now, I believe he was looking at a similar issue

[sre-ci-robot]

@zhengbuqian: GitHub didn't allow me to assign the following users: pingliu.

Note that only milvus-io members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @pingliu

assigning to pingliu for now, I believe he was looking at a similar issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pingliu]

@zhengbuqian This instance does not enable mmap, maybe it is not the same issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[wangting0128]

keep it open