Opened 16 months ago

Last modified 2 months ago

#856 new defect

XHCI driver does not start reliably on amd64

Reported by: Colin Parker Owned by:
Priority: major Milestone: 0.14.2
Component: helenos/drv/other Version: mainline
Keywords: xhci Cc:
Blocker for: Depends on:
See also:


When starting the XHCI driver, there appears to be a race condition that prevents it from starting in some cases. Steps to reproduce:

Configure HelenOS for amd64 with UEFI and SMP.

Launch QEMU with UEFI firmware (EDK2):
qemu-system-x86_64 -enable-kvm —bios OVMF.fd -drive file=hdisk.img,index=0,media=disk,format=raw -device e1000 -device qemu-xhci,id=xhci -device usb-mouse -device usb-kbd -device intel-hda -device hda-duplex -serial mon:stdio -boot d -cdrom image.iso

OVMF.fd is the firmware description from EDK2 build.

The symptom will be that although you can use the mouse (it falls back to PS2 mouse), the usb-kbd will be inoperable and you can't type. You will see xhci does not register the devices, but does not report an error either.

The issue is sensitive to timing or sequencing, and does not happen in all cases. To try to trigger it, it helps to add "fibril_usleep(10000);" to line 357 (at the bottom of xhci_event_ring_init, before "fibril_mutex_initialize(&ring→guard);") of trb_ring.c in the xhci driver code. I use values around 10000 or so and it usually causes the issue but you might need to try 50000, 100000, etc, although if you set it too long it might start working again. Note that changing the log warning level to get debug information will also alter timing, so it is challenging to investigate.

This problem can also be triggered on Mac Mini hardware, although in that case there is no PS2 fallback.

Attachments (2)

Makefile.config (4.8 KB ) - added by Colin Parker 16 months ago.
trb_ring.c (13.6 KB ) - added by Colin Parker 16 months ago.
uspace/drv/bus/usb/trb_ring.c showing timing modifications to trigger the issue

Download all attachments as: .zip

Change History (5)

by Colin Parker, 16 months ago

Attachment: Makefile.config added

by Colin Parker, 16 months ago

Attachment: trb_ring.c added

uspace/drv/bus/usb/trb_ring.c showing timing modifications to trigger the issue

comment:1 by Colin Parker, 16 months ago

Investigating this more, I have found two possible causes, both related to interrupt handling.

The first is that the interrupts are enabled in XHCI HC driver (and all USB HC drivers) prior to the device being reset. Hence, if the boot process left the HC in a state where interrupts can be triggered, that might happen between installing the interrupts and resetting the device. I have made an effort to fix that here:

The second concerns reliability of the interrupt delivery when two devices share the same IRQ. I'm not an expert on this, but my understanding is that with edge-triggered interrupts there's no guarantee that the irq handler will be entered once for every time a device raises the interrupt line. I see 3 ways to make the system resilient if the boot process assigns shared IRQs:

1) Make each driver resilient against missed interrupts. For example, a periodic timer could be set to poll the device at a reasonably slow interval (like 100 ms), so that the outcome of missed interrupts is a small delay rather than permanent lockup.

2) Improve the interrupt reliability, either by better using the existing framework (e.g. reassigning interrupts to avoid conflicts) or switching to something like MSI-X.

3) Have the kernel call every registered irq handler that claims an irq, rather than descending through a list and calling the first one. I have made an effort in that direction here:

Combined, these two commits do get the system full functional on both QEMU (using UEFI/EDK2) and on my Mac Mini hardware.

comment:2 by Jiri Svoboda, 2 months ago


Milestone renamed

comment:3 by Jiri Svoboda, 2 months ago

Note: See TracTickets for help on using tickets.