Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is there a way to show which cpu id is processing the current thread in a tcp server? #598

Open
kolinfluence opened this issue May 21, 2023 · 6 comments

Comments

@kolinfluence
Copy link

is there a way to show which cpu id is processing the current thread in a tcp server?

@glommer
Copy link
Collaborator

glommer commented May 22, 2023

what exactly do you want to know? the logical cpu id ?

You shouldn't need anything special from the framework to do that. Syscalls like getcpu() will get the info you need, and I am sure there is some Rust package exposing that. Keep in mind though that unless the thread is pinned, it can move to different CPUs.

@kolinfluence
Copy link
Author

kolinfluence commented May 23, 2023

@glommer glommer
how do u pin the thread?

i'm trying to process ip address by cpu ID. e.g. hashed(123.123.123.123)%cpunum == cpuid to process this ip only

@glommer
Copy link
Collaborator

glommer commented May 23, 2023

You can pin the thread by creating an executor with a fixed placement:
https://docs.rs/glommio/latest/glommio/#pinned-threads

However, in your case, why not use the thread_id or executor_id instead?

You can get the local executor like this:https://docs.rs/glommio/latest/glommio/fn.executor.html

and then get the id from the ExecutorProxy.

This is better because it will always be 0,1,2,3, etc, whereas the CPU IDs can have gaps.

@kolinfluence
Copy link
Author

@glommer totally new to glommio.
how do you pin specific ip addresses for processing by specific cpu id only? i've checked the references u mentioned but cant seem to wrap my head around how to implement it.

hash(ip) % [total number of cpus] = cpuid (which is used to process that ip only)

can provide some sample code?

@glommer
Copy link
Collaborator

glommer commented May 24, 2023

I don't think that's doable in the current incarnation of glommio - and for the record, it's a problem that we battled for years on seastar.

If you can listen on different ports, and then map a port to a CPU, then that's easy: just have different threads listening on the different ports.

But what you usually want is to listen on a single port, and then split the traffic across multiple CPUs. The problem here is that when you call accept() on multiple servers, you have no idea where the kernel will pass the connection. It can pass it anywhere.

What you can do is accept the connection, then figure out where you want to handle it, and then move the data to a new executor (which you can do with shared channels), but this is complex, and this transfer itself can be quite expensive (meaning it will not work well at all for short lived connections)

@bryandmc
Copy link
Collaborator

bryandmc commented May 24, 2023

You can use ebpf to program where the kernel sends the connections on accept and SO_REUSEPORT @glommer -- that seems like the situation you are describing.. Examples are hard to find but it's already supported on many kernels.

EDIT: found one example that's pretty good: https://github.com/network-analytics/ebpf-loadbalancer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants