Figure out root cause of spikes in control lag time #46
Comments
|
Thanks @alanwells. I remember that the trace shows this hangs on the connection creation. Maybe reusing the connection would prevent these large lags. The requests module provides this functionality with sessions. http://docs.python-requests.org/en/master/user/advanced/ |
|
Pull request #47 adds a potential fix and lag logging. |
|
@alanwells could you try #47 and let me know if it solves the issues you're having? Once confirmed I'll merge to master. |
|
@wroscoe I just tested the latest code on the dev branch. There's a new issue now - throttle and steering controls are reseting back to 0 even with consistent input from the controlling page on the iOS device. The presence of this issue makes it difficult to gauge whether the previous lag was eliminated, as I can't get consistent steering/throttle control long enough to see if the car response is lagging relative to my control input. This issue happens when I try to control the device from a remote (iPad or iPhone), but does not occur when I control the device from a browser on the laptop that is running the donkey server. Here's a video (ignore lack of power to wheels here, I had the motor unplugged. the same resetting to zero happens with throttle when motor is plugged in): https://www.dropbox.com/s/unb8pr1pq6v6old/donkey-issue-46.m4v?dl=0 Console log from the pi: server log: |
|
Unfortunately I don't have time to debug further today but I can look into it over the weekend. |
|
Also, it appears to be unrelated to wifi signal strength - the video above was taken when laptop running the server, iOS control device, and the pi were all within 5 feet of my router. |
|
Thanks for trying this @alanwells I'm unable to replicate this but it looks like the bug is on the server. The server should be sending continuous values shown as the "Drive:" line in the output. In your output the values coming from the web input are switching between 1/-1 and zero. I'll post here if I think of anything that could be causing this. |
|
@wroscoe I just tried again and I'm not able to reproduce the resetting to zero issue from this morning. Thinking back on it, I had my PS4 gamepad sitting on my desk and connected to my laptop via bluetooth at the same time I was testing this morning. Is it possible the resetting I saw was a result of competing control inputs (on my laptop, the gamepad was not being held and could have been sending 0 angle, 0 throttle to the open vehicle browser window on the laptop, while I was sending non-zero throttle and steering inputs via the browser page on the iOS device)? |
|
@alanwells that is possible. I don't have a game pad controller so I can't test it. I did just do a fairly long test drive and can say that the connection fix seems pretty robust. Here are my lag times running with a Mifi and the timout set to .2 seconds. I'm not sure how the lag ever gets above .2 but --''/-- (or whatever that symbol is) |
|
@wroscoe interesting. I'm planning to do some more in depth testing tomorrow morning. I'll let you know what I find. |
|
@wroscoe Here's what I found in testing this morning (running dev branch):
For comparison to your stats, I also logged my lag times on my home router and my hotspot. I had my timeout set to 0.2 as well. These were measured with the routers positioned within a few feet of the pi and my laptop:
There's a minor increase in lag time on the hotspot, but probably not enough to be concerned about. When I moved the hotspot and pi about 30 ft and two walls away from my laptop, mean lag increased to 0.037s. Again, probably not enough to be concerned about at this point. I also measured the failure rates of connections (connection errors and read timeouts). Interestingly, the hotspot had a significantly lower failure rate than my home router:
This surprised me, but it might be because of other traffic on my home network (between computers and mobile devices we have about 10 active devices on our home network, vs just the pi and laptop on the hotspot). Anyway, I think you can close this issue. |
|
@alanwells That makes sense because I added a javascript function to repeatedly send stop signals to the car when the joystick was not moving to keep the car from driving off if it missed the one stop command. This may not be needed now that the throttle is reduced if the connection is lost. This also highlights the fact that the web app is not safe or multiple users to use at the same time. It's nice to see the low lag times when using a low network. My ~.1 lag times were likely due to going through the internet to AWS. Closing this issue. |
This was the main issue I experienced at the Feb 18th track day that prevented me from reliably driving the car around the track for training. It always manifests as an intermittent problem but I am also able to observe it at home, although less frequently than I saw it at the track day.
I'm running a local donkey server over wifi, so 4G latency is not a factor here. On wifi, I'm frequently seeing lag times spike above 1s, sometimes as long as 30 or more seconds. The clues that I've seen so far are:
Here's a sample console log from the pi that shows the spikes. Lag time of ~0.06 is about normal on my home network.
The text was updated successfully, but these errors were encountered: