rtl8169: Kernel panic on FreeBSD 14 when loading if_re kernel module

[LekKit]

Steps to reproduce

• Use this U-Boot EFI firmware: https://github.com/LekKit/RVVM/releases/download/v0.6/fw_payload.bin
• Use this guest image: https://download.freebsd.org/snapshots/VM-IMAGES/14.0-STABLE/riscv64/20240314/FreeBSD-14.0-STABLE-riscv-riscv64-20240314-91c1c36102a6-266971.raw.xz
• Login with root:root
• Run # kldload if_re
• Kernel panic

Panic backtrace

root@freebsd:~ # kldload if_re
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> mem 0x40000000-0x400000ff irq 7 at device 1.0 on pci0
re0: Chip rev. 0x00800000
re0: MAC rev. 0x00000000
   t[0]: 0x0000000000000201
   t[1]: 0xffffffc06440b35c
   t[2]: 0xffffffffa3ce9d4f
   t[3]: 0xffffffc000400e76
   t[4]: 0x000000000c938ce0
   t[5]: 0xffffffc045368200
   t[6]: 0xffffffc045368200
   s[0]: 0xffffffc045368270
   s[1]: 0x0000000000008802
   s[2]: 0xffffffd001c64e00
   s[3]: 0xffffffc0453682a8
   s[4]: 0xffffffd0021c6810
   s[5]: 0xffffffc045368348
   s[6]: 0xffffffffffffffff
   s[7]: 0xffffffc0009b18d8
   s[8]: 0xffffffc0007f5cc2
   s[9]: 0xffffffc05afea130
  s[10]: 0xffffffc00071c230
  s[11]: 0x0000000000000801
   a[0]: 0xffffffd001c6f800
   a[1]: 0xffffffc0453682a8
   a[2]: 0x0000000000000000
   a[3]: 0x00000000c0306938
   a[4]: 0xffffffd0021c6820
   a[5]: 0x0000000000000020
   a[6]: 0x0000000000000020
   a[7]: 0x0000000000000800
     ra: 0xffffffc000400f7a
     sp: 0xffffffc045368240
     gp: 0xffffffc0007f4fa0
     tp: 0xffffffc0009c4600
   sepc: 0x0000000000000000
sstatus: 0x0000000a00000120
stval  : 0x0000000000000000
panic: Fatal page fault at 0: 0
cpuid = 0
time = 1710864605
KDB: stack backtrace:
#0 0xffffffc00034110a at kdb_backtrace+0x4a
#1 0xffffffc0002fc03a at vpanic+0x116
#2 0xffffffc0002fbf20 at panic+0x26
#3 0xffffffc0005ac980 at page_fault_handler+0x21a
#4 0xffffffc0005ac0a4 at do_trap_supervisor+0x56
#5 0xffffffc00059c934 at cpu_exception_handler_supervisor+0x74
#6 0xffffffc0003f8256 at if_ioctl+0xa
#7 0xffffffc0004b4dc6 at dump_iface+0x130
#8 0xffffffc0004b4742 at rtnl_handle_ifevent+0x7c
#9 0xffffffc0004b4a40 at rtnl_handle_ifattach+0x48
#10 0xffffffc0003f1f2a at if_attach_internal+0x36c
#11 0xffffffc0003f1bb2 at if_attach+0xe
#12 0xffffffc0003fc58c at ether_ifattach+0x34
#13 0xffffffc06440694a at .Lpcrel_hi96+0x48
#14 0xffffffc000331d26 at device_attach+0x368
#15 0xffffffc0003319a2 at device_probe_and_attach+0x3e
#16 0xffffffc00012e82a at pci_driver_added+0x102
#17 0xffffffc00032f736 at devclass_driver_added+0x34
Uptime: 47s

Workarounds

• Put if_re_load="YES" into /boot/loader.conf and reboot instead of a kldload
• Downgrading to FreeBSD 13.3

[LekKit]

UPD: Still happening on FreeBSD 14.1

https://download.freebsd.org/snapshots/VM-IMAGES/14.1-STABLE/riscv64/Latest/FreeBSD-14.1-STABLE-riscv-riscv64.raw.xz

root@freebsd:~ # kldload if_re
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> mem 0x40000000-0x400000ff irq 7 at device 1.0 on pci0
re0: Chip rev. 0x00800000
re0: MAC rev. 0x00000000
   t[0]: 0x0000000000000800
   t[1]: 0xffffffc06fe0b28c
   t[2]: 0x0a9cefdd868226ea
   t[3]: 0xffffffc0003f01fc
   t[4]: 0x0a45b27c50c94637
   t[5]: 0x639e09a0a6608882
   t[6]: 0x5baaff10f28a8661
   s[0]: 0xffffffc050c94280
   s[1]: 0xffffffd0025ba100
   s[2]: 0xffffffc050c942b8
   s[3]: 0x0000000000000000
   s[4]: 0xffffffd0024b4810
   s[5]: 0xffffffc050c94358
   s[6]: 0xffffffc00099f9d8
   s[7]: 0xffffffc0007e3d08
   s[8]: 0xffffffc0686dd138
   s[9]: 0xffffffc0686e0130
  s[10]: 0xffffffc00070a2d0
  s[11]: 0x0000000000000801
   a[0]: 0xffffffd0025c1800
   a[1]: 0xffffffc050c942b8
   a[2]: 0xffffffd0025ba100
   a[3]: 0x0000000000000000
   a[4]: 0x00000000c0306938
   a[5]: 0x0000000000000010
   a[6]: 0x0000000000000020
   a[7]: 0x0000000000000016
     ra: 0xffffffc0003f02ec
     sp: 0xffffffc050c94250
     gp: 0xffffffc0007e3218
     tp: 0xffffffc0009b2700
   sepc: 0x0000000000000000
sstatus: 0x0000000a00000120
stval  : 0x0000000000000000
panic: Fatal page fault at 0: 0
cpuid = 0
time = 1718785949
KDB: stack backtrace:
#0 0xffffffc000331076 at kdb_backtrace+0x4a
#1 0xffffffc0002ec40c at vpanic+0x10c
#2 0xffffffc0002ec2fc at panic+0x26
#3 0xffffffc00059a1fc at page_fault_handler+0x210
#4 0xffffffc000599930 at do_trap_supervisor+0x52
#5 0xffffffc00058a194 at cpu_exception_handler_supervisor+0x74
#6 0xffffffc0003e7584 at if_ioctl+0xc
#7 0xffffffc0004a3e0c at dump_iface+0x130
#8 0xffffffc0004a3780 at rtnl_handle_ifevent+0x7c
#9 0xffffffc0004a3a86 at rtnl_handle_ifattach+0x48
#10 0xffffffc0003e1256 at if_attach_internal+0x36c
#11 0xffffffc0003e0ede at if_attach+0xe
#12 0xffffffc0003eb92e at ether_ifattach+0x34
#13 0xffffffc06fe06910 at .Lpcrel_hi96+0x46
#14 0xffffffc000321ce2 at device_attach+0x36a
#15 0xffffffc00032195c at device_probe_and_attach+0x3e
#16 0xffffffc000121278 at pci_driver_added+0x102
#17 0xffffffc00031f714 at devclass_driver_added+0x34

[LekKit]

Seems related https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275920

[clausecker]

I just stumbled upon the same issue and reported it up stream: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=284283

[clausecker]

As a workaround, load the module through /boot/loader.conf instead of through kldload.

[LekKit]

See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277849

[LekKit]

@clausecker FreeBSD guest networking stutter is likely fixed by 6093bc1, but /boot/loader.conf workaround is still needed.

Judging the relevant FreeBSD bug tickets, kldload crash is kernel module initialization order issue, and the fact that ifmedia ioctl()'s are not protected from NULL dereference.

[clausecker]

Thank you for looking into this! I did not expect an interrupt handling issue.

The kernel panic is not that critical of a problem as there is an easy workaround. What is more confusing is why the driver doesn't attach automatically on boot—maybe a problem with the device tasting?

[LekKit]

To be fair, rtl8169 is not the only device which has a FreeBSD driver but not probed on boot automatically.

RVVM has an IDE ATA controller, which is attached via -ata disk.img, and it is not found on FreeBSD boot automatically. When doing kldload atapci kernel crashes too... Although here i'm more confident it's ATA emulation bug (And i'm not sure whether it's worth investing time to debug this legacy device)

[clausecker]

Any reason why you went for IDE over AHCI?

[LekKit]

It was very long ago when RVVM didn't even have PCI, so @cerg2010cerg2010 implemented fdt-based ATA PIO over MMIO and it was the only disk controller RVVM had back then.
Then after PCI introduction ATA was promoted to a PCI UDMA IDE controller, and soon NVMe was imlemented, which superseded ATA. U-Boot has no ATA support on RISC-V (driver has hardcoded x86 stuff), so the only way to use ATA is either as a secondary storage device or with direct kernel boot (-k linux_payload)

[clausecker]

Makes sense. I guess ATA on a non-x86 device is just rare and poorly supported. AHCI may work a lot better.