Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Encountered a LoadSegment: stoi error during the query node restart. #36255

Closed
1 task done
jaime0815 opened this issue Sep 13, 2024 · 0 comments
Closed
1 task done
Assignees
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@jaime0815
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.4.5
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

An error at LoadSegment: stoi occurs during the restart of querynode. Due to the lack of a detailed error message, we cannot identify which numeric string failed to convert.

[2024/09/13 01:42:22.135 +00:00] [WARN] [segments/load_index_info.go:233] ["CStatus returns err"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [error=stoi] [extra="AppendIndex failed"]
[2024/09/13 01:42:22.144 +00:00] [WARN] [segments/segment_loader.go:695] ["load segment failed when load data into memory"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [partitionID=451632464416318926] [segmentID=451632464418111575] [segmentType=L1] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/13 01:42:22.144 +00:00] [INFO] [segments/segment_loader.go:697] ["load segment done"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [partitionID=451632464416318926] [segmentID=451632464418111575] [segmentType=L1]
[2024/09/13 01:42:22.144 +00:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:88"]
[2024/09/13 01:42:22.144 +00:00] [WARN] [segments/segment_loader.go:734] ["failed to load some segments"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/13 01:42:22.144 +00:00] [WARN] [segments/segment_loader.go:643] ["release new segment created due to load failure"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [segmentID=451632464418111575] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

@jaime0815 jaime0815 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 13, 2024
@jaime0815 jaime0815 self-assigned this Sep 13, 2024
sre-ci-robot pushed a commit that referenced this issue Sep 14, 2024
issue: #36255

Signed-off-by: jaime <yun.zhang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Sep 18, 2024
issue: #36255
pr: #36267

Signed-off-by: jaime <yun.zhang@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Development

No branches or pull requests

1 participant