Connection breaks in between during a sustained transfer #1475
Comments
|
Speaking for the sheer unpredictability of the error here, I just happened to set the log level from
In the above picture, the top left part is the sending client, the top right part is the receiving client, the bottom left part is the broker server and the bottom right part is a watch command to check just how far is the transmission progressing, according to codebase changes made. |
|
If you wrote the loop that sends the file in 640 bytes chunks in the obvious way, there's a good chance that you're running into the problem described here: https://websockets.readthedocs.io/en/stable/faq/asyncio.html#why-does-my-program-never-receive-any-messages |
|
@aaugustin I do have a couple of functions (or coroutines? My The receiving end gets put on a sustained receiving mode until all the chunks are not received. async def collect_contents(sock: WebSocketClientProtocol, pack: bytes = b"") -> bool:
fuse_file(pack)
with logging_redirect_tqdm():
with tqdm(total=standard.client_filesize, unit="B", unit_scale=True, unit_divisor=1024, leave=False, initial=len(pack)) as prog:
for indx in range(standard.client_chks - 1):
mesgcont = await sock.recv()
if isinstance(mesgcont, bytes):
fuse_file(mesgcont)
prog.update(len(mesgcont))
# general(f"Collecting file contents from {standard.client_endo} - (SHA256 {sha256(pack).hexdigest()})")
return TrueThe sending end gets put on a sustained sending mode until all the chunks are not sent. async def deliver_contents(sock: WebSocketClientProtocol) -> bool:
with logging_redirect_tqdm():
with tqdm(total=standard.client_filesize, unit="B", unit_scale=True, unit_divisor=1024, leave=False, initial=0) as prog:
for indx in range(0, len(standard.client_bind) - 1):
bite = read_file(standard.client_bind[indx], standard.client_bind[indx + 1])
# general(f"Delivering file contents to {standard.client_endo} (SHA256 {sha256(bite).hexdigest()}).")
prog.update(standard.client_bind[indx + 1] - standard.client_bind[indx])
await sock.send(bite)
return TrueSo, a couple of questions.
|
itIf you're concerned about performance, I would recommend a larger chunk size than 640 bytes — that's extremely small by modern standards. |
|
For now, I would keep the |
On a decent modern network probably it doesn't matter. I'd go for 16kB to 64kB — probably anything between 4kB and 256kB would be indistinguishable — and let the TCP/IP layer do its job. It's better at optimizing network throughput than us :-) |
|
If you expect usage on poor networks, mobile, etc. AND if you have a good resume strategy, then you could go for a smaller size to increase the chances that a chunk go through before the connection breaks; and then you can resume. |
|
While I plan to include resumability to address weaker connections, I am sticking to the vanilla "If it fails, just try again from the start" kind of transfer. For a file that is approximately 993 MiB in size and a server that is run on the same host as where the data is sent from and received, these are the numbers that I get to see.
Until I can devise a resuming strategy (with a checkpoint write to a temporary directory every after a certain count of chunks are received), I think I would like to keep the range of chunk sizes from 64KiB to 768KiB. I am considering keeping 256KiB as the default chunk size for those users who do not want to tweak things. |
|
Sounds sensible. |
I am creating a file transfer service, the code of which can be found here. While transferring file contents from a sender client to a receiver client, the connection breaks if the transfer is sustained for a long period of time. Please note that I am fitting a maximum of 640 bytes in a message using chunking of file contents. Instead of the making one big transmission, I have doing multiple smaller ones.
Any help is appreciated.
Here is a screenshot of the logs that are produced per send and receive activity performed at all ends.
After getting rid of the logs, this is what it looks like. The receiving end seems to get stuck but maybe, it is the absence of message from the sending end due to either a problem at the sending end or at the broker end that leads to it happening.
The text was updated successfully, but these errors were encountered: