'tokio-runtime-worker' panicked

[saraiva]

Describe the bug
feroxbuster exits since version 1.11.0 with tokio runtime panick error message

To Reproduce
Steps to reproduce the behavior:
feroxbuster --url URL -o feroxbuster.txt (in this case URL is something in 192.168..)

Traceback / Error Output
thread 'tokio-runtime-worker' panicked at 'supplied instant is later than self', library/std/src/time.rs:275:48
stack backtrace:
0: 0x6f771a -
1: 0x4a61fc -
2: 0x6f6d71 -
3: 0x6f67da -
4: 0x7119fa -
5: 0x7119c4 -
6: 0x71197d -
7: 0x4a3900 -
8: 0x4aa742 -
9: 0x646c63 -
10: 0x62e8b5 -
11: 0x62d94e -
12: 0x6e0259 -
13: 0x43d72d -
14: 0x438cbf -
15: 0x421d4b -
16: 0x72ba20 -
17: 0x7295ad -
18: 0x7288ff -
19: 0x72861b -
20: 0x722915 -
zsh: abort RUST_BACKTRACE=full ../feroxbuster --url URL -o

Environment (please complete the following information):

• feroxbuster version: 1.11.0
• Linux kali 5.9.0-kali5-amd64 #1 SMP Debian 5.9.15-1kali1 (2020-12-18) x86_64 GNU/Linux

Additional context
The previous version did not panicked on me...

[epi052]

Good morning, and thanks for the report!

Judging by supplied instant is later than self, my guess is that your system clock is off (are you in a VM?).

It looks like you removed traceback information. Could you please find any lines that deal directly with feroxbuster code and let me know which portion of the code is triggering this in your environment?

Could you please respond back with the information from the following commands?

date
date -u
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource
grep flags /proc/cpuinfo
cat /proc/cmdline
sudo dmesg | grep -i tsc

[saraiva]

Hi epi052,
correct, I'm using Kali in a VM.
Strange enough, I didn't remove anything, but the copy/paste didn't work correctly, apparently. But the missing information is not helpful:
0: 0x6f771a -
1: 0x4a61fc -
2: 0x6f6d71 -
3: 0x6f67da -
4: 0x7119fa -
5: 0x7119c4 -
6: 0x71197d -
7: 0x4a3900 -
8: 0x4aa742 -
9: 0x646c63 -
10: 0x62e8b5 -
11: 0x62d94e -
12: 0x6e0259 -
13: 0x43d72d -
14: 0x438cbf -
15: 0x421d4b -
16: 0x72ba20 -
17: 0x7295ad -
18: 0x7288ff -
19: 0x72861b -
20: 0x722915 -

date
Tue 05 Jan 2021 04:32:33 PM CET

date -u
Tue 05 Jan 2021 03:32:50 PM UTC

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock acpi_pm

grep flags /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni ssse3 pcid sse4_1 sse4_2 hypervisor lahf_lm invpcid_single pti fsgsbase invpcid md_clear flush_l1d arch_capabilities
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni ssse3 pcid sse4_1 sse4_2 hypervisor lahf_lm invpcid_single pti fsgsbase invpcid md_clear flush_l1d arch_capabilities

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.9.0-kali5-amd64 root=UUID=a6825b63-bb13-4904-a565-b3e60153ab45 ro quiet splash

sudo dmesg | grep -i tsc
[sudo] password for kali:
[ 0.000017] tsc: Detected 2592.000 MHz processor
[ 6.388823] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x255cb6cc5db, max_idle_ns: 440795203504 ns
[ 6.555688] TSC synchronization [CPU#0 -> CPU#1]:
[ 6.555691] Measured 386374 cycles TSC warp between CPUs, turning off TSC clock.
[ 6.555694] tsc: Marking TSC unstable due to check_tsc_sync_source failed

Best regards, Pedro

[saraiva]

damnit, again the copy/paste issue :)

0: 0x6f771a - 'unknown'
all memory addresses have a trailing "unknown" string

[saraiva]

regarding "Could you please find any lines that deal directly with feroxbuster code and let me know which portion of the code is triggering this in your environment?"
How can I identify those lines? (keep in mind, I'm not a really good developer, quite the opposite :))

[epi052]

regarding "Could you please find any lines that deal directly with feroxbuster code and let me know which portion of the code is triggering this in your environment?"
How can I identify those lines? (keep in mind, I'm not a really good developer, quite the opposite :))

I was hoping that the traceback included something like

0x800c22ed - std::panicking::default_hook::{{closure}}::h59672b733cc6a455
            at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:351

where the at /buildslave...... was a source file in feroxbuster (i.e. src/reporter.rs:351 or similar)

[epi052]

After quite a bit of back and forth on discord, @saraiva had a problem where the Instant::now/elapsed calls in logger::initialize caused a panic on his machine. Dropping his VM down to a single core fixed the problem. The assumption is that his VM cores were out of sync with regard to timing.

In order to avoid the code in that function, I exposed a RUST_LOG=off environment configuration that will skip the timing code tied up in the custom formatter used in logger.rs. This helped, but tokio appears to use timing for its workers as well, rendering my workaround useless.

Marking this as wont-fix, as the issue lies with @saraiva's hardware or specific environment.

[luyangcy]

have same issue on virtualbox, ubuntu, reduce 4 cores to 1 fix it.

[MaximeFrancoeur]

Tokio thread related -> tokio-rs/tokio#3619

[king0ni]

Same issue today with:

• Linux kali 5.15.0-kali3-amd64 #1 SMP Debian 5.15.15-1kali2 (2022-01-25) x86_64 GNU/Linux
• running on Oracle VM Virtualbox 6.1.30 r148432

Ferox was working until recently, then just started giving error as OP describes. Verified VM datetime was correct and shutting down and restarting machine did not correct issue.

Decreasing down to one processor core corrected issue (at the cost of a slower VM). Tried to add processor back later and it didn't work and then on another attempt later it did.

[davidhacktive]

EDIT:
nevermind. This problem is a bigger than I thought.
Clocks are only guaranteed monotonic on the same core. Suggested fixes involve shared memory ^_^.

https://stackoverflow.com/questions/46893072/clock-gettimeclock-monotonic-monotonicity-across-cores-threads

[epi052]

For posterity, the information that @davidhacktive dug up is below. While not a fix, it may be interesting/useful for others.

⚠️ ⚠️ ⚠️ does not correct the issue ⚠️ ⚠️ ⚠️

Just as an FYI (and for others that have been googling this problem):
I've traced back the issue to the Guest Additions and the interactions with the VB host in regards to timing syncing.

So I suspect that any distro will have this issue when VB Guest Additions are installed. With Kali they just have added functionality to detect if it is running in a VM, and then automatically install the guest additions. This will automatically turn on the time syncing with the host.

This makes it a more apparent problem in Kali+VB due to the above and it's nature being a host for infosec tools.

The Fix
Make sure VB is updated, and has the latest Guest Additions installed in Kali.

kali@kali:~$ sudo apt update
...
kali@kali:~$ sudo apt install -y --reinstall virtualbox-guest-x11
...
kali@kali:~$ sudo reboot -f

Reference:
https://www.kali.org/docs/virtualization/install-virtualbox-guest-additions/

It is more of a a workaround to Guest Additions, but the fix is to disable syncing the time with the host.

This can be done by running the following in a shell:

VboxManage setextradata <your_vm_name> "VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled" 1

(in Windows hosts go to the path of virtual box, usually C:\Program Files\Oracle\VirtualBox\ if you don't have VB added to your $PATH environment variable).

Reference: (it has the above command)
https://docs.oracle.com/en/virtualization/virtualbox/6.0/admin/fine-tune-timers.html

[epi052]

Update on this bug.

As part of the rustc 1.60.0 release, this commit was included: rust-lang/rust@9d8ef11

The big takeaway is listed here.

Incorrectly ordered Instant comparisons no longer panic.

Since feroxbuster 2.6.3 was built with rustc 1.60.0, my hope is that we shouldn't see this bug crop up anymore from 2.6.3 and up. If you find yourself reading this thread, and still experiencing this bug, please let me know by replying here.

[0xm4v3rick]

Hey @epi052
Still seeing this issue on ver: 2.7.0. I am using Kali on VirtualBox

thread 'tokio-runtime-worker' panicked at 'supplied instant is later than self', library/std/src/time.rs:313:48 note: run withRUST_BACKTRACE=1environment variable to display a backtrace zsh: done cat ips.txt | zsh: IOT instruction feroxbuster --stdin -w -f -k -t 2 --rate-limit 2

I tried Guest Additions update as well as
VboxManage setextradata <your_vm_name> "VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled" 1

[kruug]

Hey @epi052 Still seeing this issue on ver: 2.7.0. I am using Kali on VirtualBox

Quit running Kali as a server. That is not what Kali is for.

[wifiuk]

i am also getting the same issue, clock time is as expected, have the latest guest additions installed, latest kali version

[epi052]

Thank you all for updating this thread.

I audited feroxbuster's code to make sure I wasn't doing anything specific to trigger this behavior. Fortunately (or unfortunately, depending on your point of view), feroxbuster isn't doing any operations on Instants that would trigger this.

I checked the compiler used by the github pipeline, and it's of a version high enough that the reported rust-lang fix should be present.

For now, you can try running with RUST_LOG=off to see if the logging stuff is at fault.

Additionally, if you can add RUST_BACKTRACE=1 or RUST_BACKTRACE=full to your runs so that we can hopefully capture a bit more information that i can pass on to other places.