[Bug]: Kernel NULL pointer dereference randomly

[shankerwangmiao]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

This problem is hard to reliably reproduce. I noticed that this problem happens especially when doing large disk IO operations (e.g. building Debian packages). This problem can happen at any time. Ext4 related functions are involved in the backtrace, but the backtrace keeps changing.

I have searched the backtrace and found no one on the internet complaining similar issue with Linux kernel. So I guess it might be related with this project.

Current behavior

The container got stuck. When using container logs --boot <name>, there is Unable to handle kernel NULL pointer dereference in that log.

Expected behavior

The container should run without kernel error.

Environment

- OS: macOS 15.6.1 (24G90)
- Xcode: 26.0 17A324
- Container: 0.4.1 (build: release, commit: 4ac18b5)
- Linux Kernel: 6.12.42 from [kata-static-3.20.0-arm64.tar.xz](https://github.com/kata-containers/kata-containers/releases/download/3.20.0/kata-static-3.20.0-arm64.tar.xz)

Relevant log output

[    0.066267] random: crng init done
[    0.068079] brd: module loaded
[    0.069116] loop: module loaded
[    0.069503] virtio_blk virtio2: 1/0/0 default/read/poll queues
[    0.070138] virtio_blk virtio2: [vda] 1048576 512-byte logical blocks (537 MB/512 MiB)
[    0.070943] virtio_blk virtio3: 1/0/0 default/read/poll queues
[    0.071458] virtio_blk virtio3: [vdb] 1073741824 512-byte logical blocks (550 GB/512 GiB)
[    0.072072] tun: Universal TUN/TAP device driver, 1.6
[    0.074030] VFIO - User Level meta-driver version: 0.3
[    0.074258] rtc-pl031 20050000.pl031: registered as rtc0
[    0.074338] rtc-pl031 20050000.pl031: setting system clock to 2025-09-16T11:26:41 UTC (1758022001)
[    0.074909] xt_time: kernel timezone is -0000
[    0.074985] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[    0.075035] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
[    0.075163] IPVS: ipvs loaded.
[    0.075193] IPVS: [rr] scheduler registered.
[    0.075225] IPVS: [wrr] scheduler registered.
[    0.075266] IPVS: [lc] scheduler registered.
[    0.075308] IPVS: [wlc] scheduler registered.
[    0.075369] IPVS: [fo] scheduler registered.
[    0.075417] IPVS: [ovf] scheduler registered.
[    0.075454] IPVS: [lblc] scheduler registered.
[    0.075489] IPVS: [lblcr] scheduler registered.
[    0.075550] IPVS: [dh] scheduler registered.
[    0.075581] IPVS: [sh] scheduler registered.
[    0.075612] IPVS: [sed] scheduler registered.
[    0.075658] IPVS: [nq] scheduler registered.
[    0.075697] IPVS: [sip] pe registered.
[    0.075780] Initializing XFRM netlink socket
[    0.075835] NET: Registered PF_INET6 protocol family
[    0.076261] Segment Routing with IPv6
[    0.076317] In-situ OAM (IOAM) with IPv6
[    0.076368] NET: Registered PF_PACKET protocol family
[    0.076436] 9pnet: Installing 9P2000 support
[    0.076602] NET: Registered PF_VSOCK protocol family
[    0.078159] registered taskstats version 1
[    0.082949] alg: No test for fips(ansi_cprng) (fips_ansi_cprng)
[    0.083500] clk: Disabling unused clocks
[    0.084531] EXT4-fs (vda): mounted filesystem 09e7721c-3bff-4adf-8118-5ebd1342f54d ro without journal. Quota mode: disabled.
[    0.084596] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
[    0.084767] devtmpfs: mounted
[    0.085186] Freeing unused kernel memory: 1856K
[    0.085322] Run /sbin/vminitd as init process
2025-09-16T11:26:41+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=nil
2025-09-16T11:26:41+0000 info vminitd : [vminitd] running vminitd under pid1
2025-09-16T11:26:41+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=Optional("1")
2025-09-16T11:26:41+0000 info vminitd : [vminitd] vminitd booting...
2025-09-16T11:26:41+0000 info vminitd : [vminitd] serve vminitd api
2025-09-16T11:26:41+0000 debug vminitd : [vminitd] starting process supervisor
2025-09-16T11:26:41+0000 debug vminitd : port=1024 [vminitd] booting grpc server on vsock
2025-09-16T11:26:41+0000 info vminitd : port=1024 [vminitd] grpc api serving on vsock
2025-09-16T11:26:41+0000 debug vminitd : interface=lo up=true [vminitd] ip-link-set
2025-09-16T11:26:41+0000 debug vminitd : key=PATH value=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [vminitd] setenv
2025-09-16T11:26:41+0000 debug vminitd : destination=/sys source=sysfs type=sysfs [vminitd] mount
2025-09-16T11:26:41+0000 debug vminitd : destination=/tmp source=tmpfs type=tmpfs [vminitd] mount
2025-09-16T11:26:41+0000 debug vminitd : destination=/dev/pts source=devpts type=devpts [vminitd] mount
2025-09-16T11:26:41+0000 debug vminitd : destination=/sys/fs/cgroup source=none type=cgroup2 [vminitd] mount
2025-09-16T11:26:41+0000 debug vminitd : dataSize=39 mode=0 path=/sys/fs/cgroup/cgroup.subtree_control [vminitd] writeFile
2025-09-16T11:26:41+0000 debug vminitd : destination=/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs source=/dev/vdb type=ext4 [vminitd] mount
[    0.341020] EXT4-fs (vdb): mounted filesystem 1a779faa-678a-4a44-b958-309d2821b59e r/w without journal. Quota mode: disabled.
2025-09-16T11:26:41+0000 debug vminitd : addr=192.168.64.2/24 interface=eth0 [vminitd] ip-addr-add
2025-09-16T11:26:41+0000 debug vminitd : interface=eth0 up=true [vminitd] ip-link-set
2025-09-16T11:26:41+0000 debug vminitd : gateway=192.168.64.1 interface=eth0 [vminitd] ip-route-add-default
2025-09-16T11:26:41+0000 debug vminitd : domain= location=/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs nameservers=["192.168.64.1"] [vminitd] configure-dns
2025-09-16T11:26:41+0000 debug vminitd : [vminitd] writing to path /run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs/etc/resolv.conf nameserver 192.168.64.1

2025-09-16T11:26:41+0000 debug vminitd : path=/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs/etc/resolv.conf [vminitd] wrote resolver configuration
2025-09-16T11:26:41+0000 debug vminitd : location=/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs [vminitd] configureHosts
2025-09-16T11:26:41+0000 debug vminitd : path=/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs/etc/hosts [vminitd] wrote /etc/hosts configuration
2025-09-16T11:26:41+0000 debug vminitd : containerID=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 stderr=Port: 0 stdin=Port: 268435456 stdout=Port: 268435457 [vminitd] createProcess
2025-09-16T11:26:41+0000 debug vminitd : [vminitd] created bundle with spec Spec(version: "", hooks: nil, process: Optional(ContainerizationOCI.Process(cwd: "/Volumes/tmp/ipxe/ipxe", env: ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "HOME=/root", "TERM=xterm"], consoleSize: nil, selinuxLabel: "", noNewPrivileges: false, commandLine: "", oomScoreAdj: nil, capabilities: nil, apparmorProfile: "", user: ContainerizationOCI.User(uid: 0, gid: 0, umask: nil, additionalGids: [0], username: ""), rlimits: [], args: ["bash"], terminal: true)), hostname: "3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033", domainname: "", mounts: [ContainerizationOCI.Mount(type: "proc", source: "proc", destination: "/proc", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "sysfs", source: "sysfs", destination: "/sys", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "devtmpfs", source: "none", destination: "/dev", options: ["nosuid", "mode=755"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "mqueue", source: "mqueue", destination: "/dev/mqueue", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "tmpfs", source: "tmpfs", destination: "/dev/shm", options: ["nosuid", "noexec", "nodev", "mode=1777", "size=65536k"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "cgroup2", source: "none", destination: "/sys/fs/cgroup", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "devpts", source: "devpts", destination: "/dev/pts", options: ["nosuid", "noexec", "gid=5", "mode=620", "ptmxmode=666"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "virtiofs", source: "32a394adb5a4df173f6199b332e9d5cfb778", destination: "/Volumes/tmp/ipxe/ipxe/..", options: [], uidMappings: [], gidMappings: [])], annotations: nil, root: Optional(ContainerizationOCI.Root(path: "/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs", readonly: false)), linux: Optional(ContainerizationOCI.Linux(uidMappings: [], gidMappings: [], sysctl: Optional([:]), resources: Optional(ContainerizationOCI.LinuxResources(devices: [], memory: nil, cpu: nil, pids: nil, blockIO: nil, hugepageLimits: [], network: nil, rdma: nil, unified: Optional([:]))), cgroupsPath: "/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033", namespaces: [], devices: [], seccomp: nil, rootfsPropagation: "", maskedPaths: [], readonlyPaths: [], mountLabel: "", personality: nil)))
2025-09-16T11:26:41+0000 info vminitd : mountpoint=/sys/fs/cgroup path=/sys/fs/cgroup/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] creating cgroup manager
2025-09-16T11:26:41+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] setting up terminal IO
2025-09-16T11:26:41+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] starting io
2025-09-16T11:26:41+0000 info vminitd : [vminitd] created managed init process
2025-09-16T11:26:41+0000 debug vminitd : containerID=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] startProcess
2025-09-16T11:26:41+0000 debug vminitd : [vminitd] in supervisor lock to start process
2025-09-16T11:26:41+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] starting managed process
2025-09-16T11:26:42+0000 info vminitd : id=88 [vminitd] got back pid data
2025-09-16T11:26:42+0000 debug vminitd : mountpoint=/sys/fs/cgroup path=/sys/fs/cgroup/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] adding new proc to cgroup
2025-09-16T11:26:42+0000 debug vminitd : [vminitd] received SIGCHLD, reaping processes
2025-09-16T11:26:42+0000 debug vminitd : [vminitd] starting to wait4 processes
2025-09-16T11:26:42+0000 debug vminitd : [vminitd] finished wait4 of 1 processes
2025-09-16T11:26:42+0000 debug vminitd : exits=[87: 0] processes=1 [vminitd] checking for exit of managed process
2025-09-16T11:26:42+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 pid=88 [vminitd] sending pid acknowledgement
2025-09-16T11:26:42+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] wait for pty fd
2025-09-16T11:26:42+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] received pty fd from container, attaching
2025-09-16T11:26:42+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] setting up relay for TerminalIO stdin
2025-09-16T11:26:42+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] setting up relay for TerminalIO stdout
2025-09-16T11:26:42+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 pid=88 [vminitd] started managed process
2025-09-16T11:26:42+0000 debug vminitd : [vminitd] out of supervisor lock to start process
2025-09-16T11:26:42+0000 debug vminitd : containerID=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] waitProcess
2025-09-16T11:26:42+0000 debug vminitd : containerID=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] resizeProcess
2025-09-16T11:27:13+0000 debug vminitd : sec=1758022033 usec=204989 [vminitd] setTime
2025-09-16T11:27:45+0000 debug vminitd : sec=1758022065 usec=182211 [vminitd] setTime
2025-09-16T11:28:17+0000 debug vminitd : sec=1758022097 usec=11175 [vminitd] setTime
2025-09-16T11:28:49+0000 debug vminitd : sec=1758022129 usec=15210 [vminitd] setTime
2025-09-16T11:29:21+0000 debug vminitd : sec=1758022161 usec=9943 [vminitd] setTime
2025-09-16T11:29:53+0000 debug vminitd : sec=1758022193 usec=5134 [vminitd] setTime
2025-09-16T11:30:25+0000 debug vminitd : sec=1758022225 usec=2117 [vminitd] setTime
2025-09-16T11:30:56+0000 debug vminitd : sec=1758022257 usec=2980 [vminitd] setTime
2025-09-16T11:31:29+0000 debug vminitd : sec=1758022289 usec=7387 [vminitd] setTime
2025-09-16T11:32:01+0000 debug vminitd : sec=1758022321 usec=23520 [vminitd] setTime
2025-09-16T11:32:33+0000 debug vminitd : sec=1758022353 usec=8138 [vminitd] setTime
2025-09-16T11:33:05+0000 debug vminitd : sec=1758022385 usec=13814 [vminitd] setTime
2025-09-16T11:33:37+0000 debug vminitd : sec=1758022417 usec=9368 [vminitd] setTime
2025-09-16T11:34:09+0000 debug vminitd : sec=1758022449 usec=2907 [vminitd] setTime
2025-09-16T11:34:41+0000 debug vminitd : sec=1758022481 usec=12093 [vminitd] setTime
2025-09-16T11:35:12+0000 debug vminitd : sec=1758022513 usec=7435 [vminitd] setTime
2025-09-16T11:35:45+0000 debug vminitd : sec=1758022545 usec=16919 [vminitd] setTime
2025-09-16T11:36:17+0000 debug vminitd : sec=1758022577 usec=763 [vminitd] setTime
2025-09-16T11:36:49+0000 debug vminitd : sec=1758022609 usec=15715 [vminitd] setTime
2025-09-16T11:37:21+0000 debug vminitd : sec=1758022641 usec=1174 [vminitd] setTime
2025-09-16T11:37:52+0000 debug vminitd : sec=1758022673 usec=1074 [vminitd] setTime
2025-09-16T11:38:25+0000 debug vminitd : sec=1758022705 usec=4601 [vminitd] setTime
2025-09-16T11:38:57+0000 debug vminitd : sec=1758022737 usec=1409 [vminitd] setTime
2025-09-16T11:39:28+0000 debug vminitd : sec=1758022769 usec=4438 [vminitd] setTime
2025-09-16T11:40:01+0000 debug vminitd : sec=1758022801 usec=13583 [vminitd] setTime
2025-09-16T11:40:33+0000 debug vminitd : sec=1758022833 usec=1132 [vminitd] setTime
2025-09-16T11:41:04+0000 debug vminitd : sec=1758022865 usec=182 [vminitd] setTime
2025-09-16T11:41:37+0000 debug vminitd : sec=1758022897 usec=1081 [vminitd] setTime
2025-09-16T11:42:09+0000 debug vminitd : sec=1758022929 usec=1198 [vminitd] setTime
2025-09-16T11:42:41+0000 debug vminitd : sec=1758022961 usec=1079 [vminitd] setTime
2025-09-16T11:43:12+0000 debug vminitd : sec=1758022993 usec=316 [vminitd] setTime
2025-09-16T11:43:45+0000 debug vminitd : sec=1758023025 usec=1062 [vminitd] setTime
2025-09-16T11:44:16+0000 debug vminitd : sec=1758023057 usec=1100 [vminitd] setTime
2025-09-16T11:44:49+0000 debug vminitd : sec=1758023089 usec=1112 [vminitd] setTime
2025-09-16T11:45:20+0000 debug vminitd : sec=1758023121 usec=1136 [vminitd] setTime
2025-09-16T11:45:52+0000 debug vminitd : sec=1758023153 usec=616 [vminitd] setTime
2025-09-16T11:46:25+0000 debug vminitd : sec=1758023185 usec=1117 [vminitd] setTime
2025-09-16T11:46:57+0000 debug vminitd : sec=1758023217 usec=1116 [vminitd] setTime
2025-09-16T11:47:28+0000 debug vminitd : sec=1758023249 usec=1137 [vminitd] setTime
2025-09-16T11:48:01+0000 debug vminitd : sec=1758023281 usec=1328 [vminitd] setTime
2025-09-16T11:48:32+0000 debug vminitd : sec=1758023313 usec=1080 [vminitd] setTime
2025-09-16T11:49:05+0000 debug vminitd : sec=1758023345 usec=9144 [vminitd] setTime
2025-09-16T11:49:37+0000 debug vminitd : sec=1758023377 usec=8201 [vminitd] setTime
2025-09-16T11:50:04+0000 debug vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] received EPOLLHUP with no EPOLLIN
2025-09-16T11:50:04+0000 debug vminitd : [vminitd] received SIGCHLD, reaping processes
2025-09-16T11:50:04+0000 debug vminitd : [vminitd] starting to wait4 processes
2025-09-16T11:50:04+0000 debug vminitd : [vminitd] finished wait4 of 1 processes
2025-09-16T11:50:04+0000 debug vminitd : exits=[88: 0] processes=1 [vminitd] checking for exit of managed process
2025-09-16T11:50:04+0000 debug vminitd : count=0 pid=88 status=0 [vminitd] managed process exited
2025-09-16T11:50:04+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 status=0 [vminitd] managed process exit
2025-09-16T11:50:04+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] closing relay for TerminalIO stdin
2025-09-16T11:50:04+0000 info vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] closing relay for TerminalIO stdout
2025-09-16T11:50:04+0000 debug vminitd : id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] 1 managed process waiters signaled
2025-09-16T11:50:04+0000 debug vminitd : pid=-1 signal=9 [vminitd] kill
2025-09-16T11:50:04+0000 debug vminitd : containerID=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 id=3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033 [vminitd] waitProcess
2025-09-16T11:50:04+0000 debug vminitd : flags=0 path=/run/container/3ebb8e87-d2c5-4dbe-9f3c-f6990a7ce033/rootfs [vminitd] umount
[ 1404.167140] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 1404.167402] Mem abort info:
[ 1404.167441]   ESR = 0x0000000096000046
[ 1404.167484]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1404.167606]   SET = 0, FnV = 0
[ 1404.167694]   EA = 0, S1PTW = 0
[ 1404.167757]   FSC = 0x06: level 2 translation fault
[ 1404.167821] Data abort info:
[ 1404.167858]   ISV = 0, ISS = 0x00000046, ISS2 = 0x00000000
[ 1404.167979]   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[ 1404.168100]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 1404.168222] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001002a2000
[ 1404.168357] [0000000000000000] pgd=080000010089e003, p4d=080000010089e003, pud=0800000101130003, pmd=0000000000000000
[ 1404.168534] Internal error: Oops: 0000000096000046 [#1] SMP
[ 1404.168623] CPU: 0 UID: 0 PID: 16682 Comm: vminitd Not tainted 6.12.42 #1
[ 1404.168701] pstate: 21400005 (nzCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1404.168776] pc : selinux_inode_free_security+0x7c/0xbc
[ 1404.168933] lr : security_inode_free+0x58/0x74
[ 1404.169038] sp : ffff800083f33b60
[ 1404.169140] x29: ffff800083f33b60 x28: ffff6ae440330548 x27: ffff6ae442f06000
[ 1404.169221] x26: ffff6ae440798608 x25: 00000000000000c1 x24: ffffa983b3d7bf28
[ 1404.169321] x23: ffff800083f33c98 x22: 0000000000000000 x21: ffff6ae441460018
[ 1404.169427] x20: ffff6ae4402bc0c0 x19: ffff6ae441460010 x18: 0000000000000001
[ 1404.169549] x17: 0000000000000002 x16: ffff6ae4410e1da8 x15: 000008fe172e32ba
[ 1404.169719] x14: ffff6ae442f06080 x13: 000000000014a0c0 x12: 0000000000000005
[ 1404.169834] x11: 0000000000000001 x10: 0000000000000001 x9 : 0000000000000000
[ 1404.170003] x8 : 00000000ffffffe7 x7 : 0000000000000000 x6 : 0000000000660006
[ 1404.170077] x5 : 0000000000660005 x4 : 0000000000000000 x3 : 0000000000000001
[ 1404.170149] x2 : 0000000000000000 x1 : ffff6ae4402bc0f0 x0 : 0000000000000000
[ 1404.170221] Call trace:
[ 1404.170246]  selinux_inode_free_security+0x7c/0xbc
[ 1404.170323]  security_inode_free+0x58/0x74
[ 1404.170434]  __destroy_inode+0x74/0x1ec
[ 1404.170547]  destroy_inode+0x2c/0x70
[ 1404.170657]  evict+0x184/0x2dc
[ 1404.170694]  evict_inodes+0x16c/0x220
[ 1404.170750]  generic_shutdown_super+0x44/0x15c
[ 1404.170795]  kill_block_super+0x18/0x38
[ 1404.170846]  ext4_kill_sb+0x1c/0x48
[ 1404.170902]  deactivate_locked_super+0x50/0x160
[ 1404.171015]  deactivate_super+0x68/0x7c
[ 1404.171054]  cleanup_mnt+0x94/0x11c
[ 1404.171140]  __cleanup_mnt+0x10/0x18
[ 1404.171220]  task_work_run+0xb0/0x118
[ 1404.171285]  do_notify_resume+0x138/0x13c
[ 1404.171331]  el0_svc+0xf4/0x1e8
[ 1404.171401]  el0t_64_sync_handler+0x134/0x150
[ 1404.171476]  el0t_64_sync+0x160/0x164
[ 1404.171513] Code: 35ffff84 35000221 a9408261 f9000420 (f9000001) 
[ 1404.171559] ---[ end trace 0000000000000000 ]---
2025-09-16T11:50:08+0000 debug vminitd : sec=1758023408 usec=32105 [vminitd] setTime
2025-09-16T11:50:40+0000 debug vminitd : sec=1758023440 usec=13553 [vminitd] setTime
2025-09-16T11:51:11+0000 debug vminitd : sec=1758023472 usec=5770 [vminitd] setTime
2025-09-16T11:51:44+0000 debug vminitd : sec=1758023504 usec=10060 [vminitd] setTime
2025-09-16T11:52:16+0000 debug vminitd : sec=1758023536 usec=1288 [vminitd] setTime
2025-09-16T11:52:48+0000 debug vminitd : sec=1758023568 usec=19042 [vminitd] setTime
2025-09-16T11:53:20+0000 debug vminitd : sec=1758023600 usec=8298 [vminitd] setTime
2025-09-16T11:53:52+0000 debug vminitd : sec=1758023632 usec=6851 [vminitd] setTime
2025-09-16T11:54:24+0000 debug vminitd : sec=1758023664 usec=11984 [vminitd] setTime
2025-09-16T11:54:55+0000 debug vminitd : sec=1758023696 usec=18029 [vminitd] setTime
2025-09-16T11:55:28+0000 debug vminitd : sec=1758023728 usec=2386 [vminitd] setTime
2025-09-16T11:55:59+0000 debug vminitd : sec=1758023760 usec=503 [vminitd] setTime
2025-09-16T11:56:32+0000 debug vminitd : sec=1758023792 usec=22129 [vminitd] setTime
2025-09-16T11:57:04+0000 debug vminitd : sec=1758023824 usec=375 [vminitd] setTime
2025-09-16T11:57:36+0000 debug vminitd : sec=1758023856 usec=18982 [vminitd] setTime
2025-09-16T11:58:08+0000 debug vminitd : sec=1758023888 usec=1369 [vminitd] setTime
2025-09-16T11:58:39+0000 debug vminitd : sec=1758023920 usec=1470 [vminitd] setTime
2025-09-16T11:59:11+0000 debug vminitd : sec=1758023952 usec=2069 [vminitd] setTime
2025-09-16T11:59:44+0000 debug vminitd : sec=1758023984 usec=2522 [vminitd] setTime
2025-09-16T12:00:15+0000 debug vminitd : sec=1758024016 usec=1697 [vminitd] setTime
2025-09-16T12:00:48+0000 debug vminitd : sec=1758024048 usec=3146 [vminitd] setTime
2025-09-16T12:01:19+0000 debug vminitd : sec=1758024080 usec=967 [vminitd] setTime

Code of Conduct

• I agree to follow this project's Code of Conduct

[mpolden]

I think what I observed in #576 (comment) is essentially the same issue as this. This morning I ran into this problem again and both times there were a lot of ext4 errors being logged leading up to the null pointer.

I suspect a bug with how the Containerization package handles volumes. When there is a lot of I/O on the volume (e.g. when compiling a big project) some invariant is eventually broken putting the container/kernel/VM in a half-broken state. vdc here is a volume created with container create. But there are also errors on vdb (root filesystem):

[Wed Sep 24 06:21:01 2025] EXT4-fs (vdc): error count since last fsck: 9
[Wed Sep 24 06:21:01 2025] EXT4-fs (vdc): initial error at time 1758653011: ext4_lookup:1815: inode 163842
[Wed Sep 24 06:21:01 2025] EXT4-fs (vdc): last error at time 1758655647: ext4_lookup:1815: inode 483329
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483425
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483426
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483425
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483426
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483426
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483425
[Wed Sep 24 06:32:16 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483425
[Wed Sep 24 06:32:24 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483427
[Wed Sep 24 06:32:24 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483427
[Wed Sep 24 06:32:26 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483427
[Wed Sep 24 06:32:26 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483427
[Wed Sep 24 06:32:31 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483439
[Wed Sep 24 06:32:31 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483439
[Wed Sep 24 06:32:32 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483439
[Wed Sep 24 06:32:43 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483428
[Wed Sep 24 06:32:44 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483428
[Wed Sep 24 06:32:44 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483428
[Wed Sep 24 06:32:54 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483429
[Wed Sep 24 06:32:55 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483429
[Wed Sep 24 06:32:55 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483429
[Wed Sep 24 06:33:02 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483437
[Wed Sep 24 06:33:02 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483436
[Wed Sep 24 06:33:02 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483438
[Wed Sep 24 06:33:03 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483436
[Wed Sep 24 06:33:03 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483438
[Wed Sep 24 06:33:03 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483437
[Wed Sep 24 06:33:03 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483437
[Wed Sep 24 06:33:05 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483436
[Wed Sep 24 06:33:05 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483436
[Wed Sep 24 06:33:05 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483438
[Wed Sep 24 06:33:11 2025] EXT4-fs error: 1 callbacks suppressed
[Wed Sep 24 06:33:11 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483430
[Wed Sep 24 06:33:12 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483430
[Wed Sep 24 06:33:12 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483430
[Wed Sep 24 06:33:17 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483432
[Wed Sep 24 06:33:18 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483432
[Wed Sep 24 06:33:21 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483432
[Wed Sep 24 06:33:21 2025] EXT4-fs error (device vdc): ext4_lookup:1815: inode #483329: comm ccache: deleted inode referenced: 483432
[Wed Sep 24 06:40:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3.12: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:40:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:40:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3.12: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:40:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:50:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:50:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3.12: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:50:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:50:36 2025] EXT4-fs error (device vdb): ext4_readdir:262: inode #124038: block 1283830: comm python3.12: path /usr/local/lib/python3.12/site-packages: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[Wed Sep 24 06:52:59 2025] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[Wed Sep 24 06:52:59 2025] Mem abort info:
[Wed Sep 24 06:52:59 2025]   ESR = 0x0000000096000004
[Wed Sep 24 06:52:59 2025]   EC = 0x25: DABT (current EL), IL = 32 bits
[Wed Sep 24 06:52:59 2025]   SET = 0, FnV = 0
[Wed Sep 24 06:52:59 2025]   EA = 0, S1PTW = 0
[Wed Sep 24 06:52:59 2025]   FSC = 0x04: level 0 translation fault
[Wed Sep 24 06:52:59 2025] Data abort info:
[Wed Sep 24 06:52:59 2025]   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
[Wed Sep 24 06:52:59 2025]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[Wed Sep 24 06:52:59 2025]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[Wed Sep 24 06:52:59 2025] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010882f000
[Wed Sep 24 06:52:59 2025] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[Wed Sep 24 06:52:59 2025] Internal error: Oops: 0000000096000004 [#1] SMP
[Wed Sep 24 06:52:59 2025] CPU: 8 UID: 0 PID: 143 Comm: systemd Not tainted 6.12.42 #1
[Wed Sep 24 06:52:59 2025] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[Wed Sep 24 06:52:59 2025] pc : __bpf_prog_put+0xc/0xe0
[Wed Sep 24 06:52:59 2025] lr : bpf_prog_release+0x10/0x1c
[Wed Sep 24 06:52:59 2025] sp : ffff800081133db0
[Wed Sep 24 06:52:59 2025] x29: ffff800081133db0 x28: ffff365d472e0000 x27: 0000000000000000
[Wed Sep 24 06:52:59 2025] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[Wed Sep 24 06:52:59 2025] x23: ffff365d400cd0a0 x22: ffff365d40486680 x21: ffff365d4581c000
[Wed Sep 24 06:52:59 2025] x20: 00000000000e0003 x19: ffff365d4d41b0c0 x18: 0000000000000000
[Wed Sep 24 06:52:59 2025] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[Wed Sep 24 06:52:59 2025] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[Wed Sep 24 06:52:59 2025] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000007
[Wed Sep 24 06:52:59 2025] x8 : ffff365d408f5ec0 x7 : ffff365d408f5e80 x6 : 0000000000000000
[Wed Sep 24 06:52:59 2025] x5 : ffff365d40486680 x4 : 0000000000000000 x3 : 0000000000000000
[Wed Sep 24 06:52:59 2025] x2 : ffffb43ba4513d14 x1 : 0000000000000001 x0 : 0000000000000000
[Wed Sep 24 06:52:59 2025] Call trace:
[Wed Sep 24 06:52:59 2025]  __bpf_prog_put+0xc/0xe0
[Wed Sep 24 06:52:59 2025]  __fput+0xc4/0x274
[Wed Sep 24 06:52:59 2025]  __fput_sync+0x30/0x38
[Wed Sep 24 06:52:59 2025]  __arm64_sys_close+0x34/0x74
[Wed Sep 24 06:52:59 2025]  invoke_syscall.constprop.0+0x58/0xf4
[Wed Sep 24 06:52:59 2025]  do_el0_svc+0x44/0xc8
[Wed Sep 24 06:52:59 2025]  el0_svc+0x50/0x1e8
[Wed Sep 24 06:52:59 2025]  el0t_64_sync_handler+0x134/0x150
[Wed Sep 24 06:52:59 2025]  el0t_64_sync+0x160/0x164
[Wed Sep 24 06:52:59 2025] Code: d65f03c0 d2800021 f9401c00 f9800011 (c85f7c02)
[Wed Sep 24 06:52:59 2025] ---[ end trace 0000000000000000 ]---

[tiancheng91]

I encountered a similar issue, and after panic, container stop devbox not works

2025-10-08T12:45:48+0000 debug vminitd : sec=1759927548 usec=955632 [vminitd] setTime
2025-10-08T12:46:20+0000 debug vminitd : sec=1759927580 usec=558172 [vminitd] setTime
2025-10-08T12:46:52+0000 debug vminitd : sec=1759927612 usec=21853 [vminitd] setTime
2025-10-08T12:47:23+0000 debug vminitd : sec=1759927643 usec=572137 [vminitd] setTime
2025-10-08T12:47:55+0000 debug vminitd : sec=1759927675 usec=26830 [vminitd] setTime
[939835.611567] BUG: Bad rss-counter state mm:00000000da60edc6 type:MM_FILEPAGES val:14
[939835.611594] BUG: Bad rss-counter state mm:000000001efe311e type:MM_FILEPAGES val:14
[939835.611684] BUG: Bad rss-counter state mm:00000000da60edc6 type:MM_ANONPAGES val:2
[939835.611710] BUG: Bad rss-counter state mm:000000001efe311e type:MM_ANONPAGES val:12
[939835.611712] BUG: non-zero pgtables_bytes on freeing mm: 36864
[939835.611740] BUG: Bad rss-counter state mm:00000000482c3895 type:MM_FILEPAGES val:13
[939835.611791] BUG: non-zero pgtables_bytes on freeing mm: 12288
[939835.611829] BUG: Bad rss-counter state mm:00000000482c3895 type:MM_ANONPAGES val:2
[939835.611830] BUG: non-zero pgtables_bytes on freeing mm: 32768
[939835.611938] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[939835.611939] Mem abort info:
[939835.611940]   ESR = 0x0000000086000004
[939835.611941]   EC = 0x21: IABT (current EL), IL = 32 bits
[939835.611942]   SET = 0, FnV = 0
[939835.612251]   EA = 0, S1PTW = 0
[939835.612279]   FSC = 0x04: level 0 translation fault
[939835.612321] user pgtable: 4k pages, 48-bit VAs, pgdp=000000011e0bd000
[939835.612366] BUG: Bad rss-counter state mm:00000000b10f0513 type:MM_FILEPAGES val:11
[939835.612375] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[939835.612378] Internal error: Oops: 0000000086000004 [#1] SMP
[939835.612381] CPU: 1 UID: 0 PID: 755 Comm: bash Not tainted 6.12.28 #1
[939835.612384] pstate: 614000c5 (nZCv daIF +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[939835.612470] BUG: Bad rss-counter state mm:00000000b10f0513 type:MM_ANONPAGES val:2
[939835.612484] pc : 0x0
[939835.612487] lr : 0x0
[939835.612488] sp : ffff8000807ebbf0
[939835.612488] x29: 0000000000000000 x28: ffff2e04c1017000 x27: 0000000000000000
[939835.612490] x26: 0000000000000000 x25: ffff2e04c1017000 x24: ffff2e04c1017580
[939835.612491] x23: ffffb131b5b1ab94 x22: ffffb131b5f2a910 x21: ffff2e052ffd9480
[939835.612493] x20: 0000000000000000
[939835.612561] BUG: non-zero pgtables_bytes on freeing mm: 12288
[939835.612584]  x19: 0000000000000000 x18: ffff8000806cbae0
[939835.612585] x17: 0000000000000000 x16: 1fffe5c09bc39f61 x15: 0000000000000002
[939835.612586] x14: ffff2e04c0fed090 x13: 0000000000000000 x12: 0000000001bee400
[939835.612587] x11: 000000000000019a x10: 00000000000009c0 x9 : ffff8000807ebbd0
[939835.612589] x8 : ffff2e04c1017a20 x7 : ffffb131b601cf38 x6 : 0000000000000000
[939835.612590] x5 : 0000000000000000 x4 : 000000000000080d x3 : ffffffffffffefff
[939835.612591] x2 : 000000000000080d x1 : ffff2e04c1017000 x0 : ffff2e04c100c000
[939835.612593] Call trace:
[939835.612884] BUG: Bad rss-counter state mm:00000000369b925c type:MM_FILEPAGES val:15
[939835.612895]  0x0
[939835.612899] Code: ???????? ???????? ???????? ???????? (????????)
[939835.612899] ---[ end trace 0000000000000000 ]---
[939835.612900] note: bash[755] exited with irqs disabled
[939835.613690] BUG: Bad rss-counter state mm:00000000369b925c type:MM_ANONPAGES val:6
[939835.613741] BUG: non-zero pgtables_bytes on freeing mm: 28672

[katiewasnothere]

@mpolden and @tiancheng91 these two kernel panics look different than @shankerwangmiao. Could you two open new issues for these panics?

[katiewasnothere]

I haven't gotten too far into investigating this, however this is what I've found so far:

The call trace indicates that the issue is around freeing inode security when we unmount the ext4 device for the container's rootfs. This code path should only be hit when CONFIG_SECURITY is set in the kernel, which it is set in the default kata kernel we're using right now 6.12.28-153.

[ 1404.170246]  selinux_inode_free_security+0x7c/0xbc
[ 1404.170323]  security_inode_free+0x58/0x74

Looking through the linux source code, I see that we're eventually calling this function.

I also know we don't have the security field defined on the inode type we define in our ext4 code here. Maybe the inode is getting corrupt somehow? Another question is what conditions could cause the device to be corrupted given this is a rare issue?

I don't have the cycles to continue looking at this right now, but wanted to post what I had so far in case someone else wants to take up this investigation.

[shankerwangmiao]

Hi, the attached log of output happened when the main process of the container exits. However, similar issues, the log of which also contained ext4 related symbols, had happened during the container running. I did not preserve the logs. I'll post new logs if I meet it again.

[mdempsky]

I ran container run --rm -it --platform=linux/amd64 golang:alpine and then go test crypto/internal/hpke and then (after confirming that Rosetta still fails to emulate it correctly in macOS 26.1 beta 4 -- FB20553663) pressed Ctrl+D to exit the container. The shell exited, but then I was left with a hung terminal, as the container command never exited.

Here's vminitd.log from the container before I ran `pkill container-runtime-linux`:

[    0.117015] random: crng init done
[    0.117702] brd: module loaded
[    0.118103] loop: module loaded
[    0.118170] virtio_blk virtio2: 1/0/0 default/read/poll queues
[    0.118331] virtio_blk virtio2: [vda] 1048576 512-byte logical blocks (537 MB/512 MiB)
[    0.120066] virtio_blk virtio3: 1/0/0 default/read/poll queues
[    0.120224] virtio_blk virtio3: [vdb] 1073741824 512-byte logical blocks (550 GB/512 GiB)
[    0.120717] VFIO - User Level meta-driver version: 0.3
[    0.120822] rtc-pl031 20050000.pl031: registered as rtc0
[    0.120848] rtc-pl031 20050000.pl031: setting system clock to 2025-10-21T03:42:05 UTC (1761018125)
[    0.121006] xt_time: kernel timezone is -0000
[    0.121032] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[    0.121054] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
[    0.121097] IPVS: ipvs loaded.
[    0.121118] IPVS: [rr] scheduler registered.
[    0.121134] IPVS: [wrr] scheduler registered.
[    0.121157] IPVS: [lc] scheduler registered.
[    0.121178] IPVS: [wlc] scheduler registered.
[    0.121199] IPVS: [fo] scheduler registered.
[    0.121212] IPVS: [ovf] scheduler registered.
[    0.121236] IPVS: [lblc] scheduler registered.
[    0.121260] IPVS: [lblcr] scheduler registered.
[    0.121282] IPVS: [dh] scheduler registered.
[    0.121309] IPVS: [sh] scheduler registered.
[    0.121331] IPVS: [sed] scheduler registered.
[    0.121345] IPVS: [nq] scheduler registered.
[    0.121369] IPVS: [sip] pe registered.
[    0.121408] Initializing XFRM netlink socket
[    0.121443] NET: Registered PF_INET6 protocol family
[    0.121686] Segment Routing with IPv6
[    0.121721] In-situ OAM (IOAM) with IPv6
[    0.121757] NET: Registered PF_PACKET protocol family
[    0.121785] 9pnet: Installing 9P2000 support
[    0.121843] NET: Registered PF_VSOCK protocol family
[    0.122684] registered taskstats version 1
[    0.123779] alg: No test for fips(ansi_cprng) (fips_ansi_cprng)
[    0.124073] clk: Disabling unused clocks
[    0.125205] EXT4-fs (vda): mounted filesystem 5cf97579-5f59-4158-b71c-0f4075a57cc4 ro without journal. Quota mode: disabled.
[    0.125260] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
[    0.125357] devtmpfs: mounted
[    0.125501] Freeing unused kernel memory: 1856K
[    0.125566] Run /sbin/vminitd as init process
2025-10-21T03:42:05+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=nil
2025-10-21T03:42:05+0000 info vminitd : [vminitd] running vminitd under pid1
2025-10-21T03:42:05+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=Optional("1")
2025-10-21T03:42:05+0000 info vminitd : [vminitd] vminitd booting...
2025-10-21T03:42:05+0000 info vminitd : [vminitd] serve vminitd api
2025-10-21T03:42:05+0000 debug vminitd : [vminitd] starting process supervisor
2025-10-21T03:42:05+0000 debug vminitd : port=1024 [vminitd] booting grpc server on vsock
2025-10-21T03:42:05+0000 info vminitd : port=1024 [vminitd] grpc api serving on vsock
2025-10-21T03:42:06+0000 debug vminitd : destination=/run/rosetta source=rosetta type=virtiofs [vminitd] mount
2025-10-21T03:42:06+0000 debug vminitd : request=Containerization.Com_Apple_Containerization_Sandbox_V3_SetupEmulatorRequest:
binary_path: "/run/rosetta/rosetta"
name: "x86_64"
type: "M"
magic: "\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x3e\\x00"
mask: "\\xff\\xff\\xff\\xff\\xff\\xfe\\xfe\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff"
flags: "CF"
 [vminitd] setupEmulator
2025-10-21T03:42:06+0000 debug vminitd : interface=lo up=true [vminitd] ip-link-set
2025-10-21T03:42:06+0000 debug vminitd : key=PATH value=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [vminitd] setenv
2025-10-21T03:42:06+0000 debug vminitd : destination=/sys source=sysfs type=sysfs [vminitd] mount
2025-10-21T03:42:06+0000 debug vminitd : destination=/tmp source=tmpfs type=tmpfs [vminitd] mount
2025-10-21T03:42:06+0000 debug vminitd : destination=/dev/pts source=devpts type=devpts [vminitd] mount
2025-10-21T03:42:06+0000 debug vminitd : destination=/sys/fs/cgroup source=none type=cgroup2 [vminitd] mount
2025-10-21T03:42:06+0000 debug vminitd : dataSize=39 mode=0 path=/sys/fs/cgroup/cgroup.subtree_control [vminitd] writeFile
2025-10-21T03:42:06+0000 debug vminitd : destination=/run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs source=/dev/vdb type=ext4 [vminitd] mount
[    0.959737] EXT4-fs (vdb): mounted filesystem 4fdc317b-ca0a-4d3b-ab74-b5a98c32d242 r/w without journal. Quota mode: disabled.
2025-10-21T03:42:06+0000 debug vminitd : addr=192.168.64.2/24 interface=eth0 [vminitd] ip-addr-add
2025-10-21T03:42:06+0000 debug vminitd : interface=eth0 up=true [vminitd] ip-link-set
2025-10-21T03:42:06+0000 debug vminitd : gateway=192.168.64.1 interface=eth0 [vminitd] ip-route-add-default
2025-10-21T03:42:06+0000 debug vminitd : domain=test location=/run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs nameservers=["192.168.64.1"] [vminitd] configure-dns
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] writing to path /run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs/etc/resolv.conf nameserver 192.168.64.1
domain test

2025-10-21T03:42:06+0000 debug vminitd : path=/run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs/etc/resolv.conf [vminitd] wrote resolver configuration
2025-10-21T03:42:06+0000 debug vminitd : location=/run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs [vminitd] configureHosts
2025-10-21T03:42:06+0000 debug vminitd : path=/run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs/etc/hosts [vminitd] wrote /etc/hosts configuration
2025-10-21T03:42:06+0000 debug vminitd : containerID=3cba1913-8231-47b1-a6ea-49522fc99773 id=3cba1913-8231-47b1-a6ea-49522fc99773 stderr=Port: 0 stdin=Port: 268435456 stdout=Port: 268435457 [vminitd] createProcess
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] created bundle with spec Spec(version: "", hooks: nil, process: Optional(ContainerizationOCI.Process(cwd: "/go", env: ["PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOLANG_VERSION=1.25.1", "GOTOOLCHAIN=local", "GOPATH=/go", "HOME=/root", "TERM=xterm"], consoleSize: nil, selinuxLabel: "", noNewPrivileges: false, commandLine: "", oomScoreAdj: nil, capabilities: nil, apparmorProfile: "", user: ContainerizationOCI.User(uid: 0, gid: 0, umask: nil, additionalGids: [0], username: ""), rlimits: [], args: ["/bin/sh"], terminal: true)), hostname: "3cba1913-8231-47b1-a6ea-49522fc99773", domainname: "", mounts: [ContainerizationOCI.Mount(type: "proc", source: "proc", destination: "/proc", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "sysfs", source: "sysfs", destination: "/sys", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "devtmpfs", source: "none", destination: "/dev", options: ["nosuid", "mode=755"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "mqueue", source: "mqueue", destination: "/dev/mqueue", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "tmpfs", source: "tmpfs", destination: "/dev/shm", options: ["nosuid", "noexec", "nodev", "mode=1777", "size=65536k"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "cgroup2", source: "none", destination: "/sys/fs/cgroup", options: ["nosuid", "noexec", "nodev"], uidMappings: [], gidMappings: []), ContainerizationOCI.Mount(type: "devpts", source: "devpts", destination: "/dev/pts", options: ["nosuid", "noexec", "gid=5", "mode=620", "ptmxmode=666"], uidMappings: [], gidMappings: [])], annotations: nil, root: Optional(ContainerizationOCI.Root(path: "/run/container/3cba1913-8231-47b1-a6ea-49522fc99773/rootfs", readonly: false)), linux: Optional(ContainerizationOCI.Linux(uidMappings: [], gidMappings: [], sysctl: Optional([:]), resources: Optional(ContainerizationOCI.LinuxResources(devices: [], memory: nil, cpu: nil, pids: nil, blockIO: nil, hugepageLimits: [], network: nil, rdma: nil, unified: Optional([:]))), cgroupsPath: "/container/3cba1913-8231-47b1-a6ea-49522fc99773", namespaces: [], devices: [], seccomp: nil, rootfsPropagation: "", maskedPaths: [], readonlyPaths: [], mountLabel: "", personality: nil)))
2025-10-21T03:42:06+0000 info vminitd : mountpoint=/sys/fs/cgroup path=/sys/fs/cgroup/container/3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] creating cgroup manager
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] setting up terminal IO
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] starting io
2025-10-21T03:42:06+0000 info vminitd : [vminitd] created managed init process
2025-10-21T03:42:06+0000 debug vminitd : containerID=3cba1913-8231-47b1-a6ea-49522fc99773 id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] startProcess
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] in supervisor lock to start process
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] starting managed process
2025-10-21T03:42:06+0000 info vminitd : id=89 [vminitd] got back pid data
2025-10-21T03:42:06+0000 debug vminitd : mountpoint=/sys/fs/cgroup path=/sys/fs/cgroup/container/3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] adding new proc to cgroup
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] received SIGCHLD, reaping processes
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] starting to wait4 processes
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] finished wait4 of 1 processes
2025-10-21T03:42:06+0000 debug vminitd : exits=[88: 0] processes=1 [vminitd] checking for exit of managed process
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 pid=89 [vminitd] sending pid acknowledgement
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] wait for pty fd
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] received pty fd from container, attaching
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] setting up relay for TerminalIO stdin
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] setting up relay for TerminalIO stdout
2025-10-21T03:42:06+0000 info vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 pid=89 [vminitd] started managed process
2025-10-21T03:42:06+0000 debug vminitd : [vminitd] out of supervisor lock to start process
2025-10-21T03:42:06+0000 debug vminitd : containerID=3cba1913-8231-47b1-a6ea-49522fc99773 id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] waitProcess
2025-10-21T03:42:06+0000 debug vminitd : containerID=3cba1913-8231-47b1-a6ea-49522fc99773 id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] resizeProcess
2025-10-21T03:42:37+0000 debug vminitd : sec=1761018157 usec=412762 [vminitd] setTime
2025-10-21T03:43:09+0000 debug vminitd : sec=1761018189 usec=178143 [vminitd] setTime
[   75.186564] U2025-10-21T03:43:20+0000 debug vminitd : id=3cba1913-8231-47b1-a6ea-49522fc99773 [vminitd] received EPOLLHUP with no EPOLLINnable to handle kernel NULL poin
ter dereference at virtual address 0000000000000000
[   75.191846] Mem abort info:
[   75.191932]   ESR = 0x0000000096000046
[   75.191971]   EC = 0x25: DABT (current EL), IL = 32 bits
[   75.192034]   SET = 0, FnV = 0
[   75.192074]   EA = 0, S1PTW = 0
[   75.192110]   FSC = 0x06: level 2 translation fault
[   75.192151] Data abort info:
[   75.192189]   ISV = 0, ISS = 0x00000046, ISS2 = 0x00000000
[   75.192227]   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[   75.192264]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[   75.192301] user pgtable: 4k pages, 48-bit VAs, pgdp=000000007294f000
[   75.192346] [0000000000000000] pgd=0800000072947003, p4d=0800000072947003, pud=08000000717ff003, pmd=0000000000000000
[   75.192425] Internal error: Oops: 0000000096000046 [#1] SMP
[   75.192532] CPU: 1 UID: 0 PID: 81 Comm: vminitd Not tainted 6.12.28 #1
[   75.192585] pstate: 41400005 (nZcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   75.192655] pc : __memset+0x16c/0x188
[   75.192743] lr : copy_process+0x374/0x1af8
[   75.192780] sp : ffff800080433b90
[   75.192807] x29: ffff800080433ba0 x28: ffff0a0cbab752c0 x27: 00000000ffffffff
[   75.192865] x26: ffff0a0cb1d6a000 x25: ffff0a0cbab752c0 x24: 00000000007d0f00
[   75.192920] x23: ffff0a0cb2371000 x22: ffffd61178c1cf70 x21: 0000000000000000
[   75.192957] x20: 0000000000000000 x19: ffff800080433d90 x18: 0000000000000000
[   75.193019] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[   75.193061] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[   75.193090] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[   75.193130] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 000000000000003f
[   75.193185] x5 : 0000000000000040 x4 : 0000000000000000 x3 : 0000000000000004
[   75.193227] x2 : 0000000000003fc0 x1 : 0000000000000000 x0 : 0000000000000000
[   75.193280] Call trace:
[   75.193290]  __memset+0x16c/0x188
[   75.193317]  kernel_clone+0x88/0x33c
[   75.193332]  __do_sys_clone+0x6c/0xa0
[   75.193369]  __arm64_sys_clone+0x1c/0x24
[   75.193397]  invoke_syscall.constprop.0+0x58/0xf4
[   75.193418]  do_el0_svc+0x44/0xc8
[   75.193444]  el0_svc+0x50/0x1b8
[   75.193461]  el0t_64_sync_handler+0x134/0x150
[   75.193490]  el0t_64_sync+0x160/0x164
[   75.193525] Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428) 
[   75.193555] ---[ end trace 0000000000000000 ]---
[   96.225327] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[   96.225913] Mem abort info:
[   96.225996]   ESR = 0x0000000096000046
[   96.226144]   EC = 0x25: DABT (current EL), IL = 32 bits
[   96.226188]   SET = 0, FnV = 0
[   96.226261]   EA = 0, S1PTW = 0
[   96.226295]   FSC = 0x06: level 2 translation fault
[   96.226373] Data abort info:
[   96.226410]   ISV = 0, ISS = 0x00000046, ISS2 = 0x00000000
[   96.226555]   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[   96.226630]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[   96.226867] user pgtable: 4k pages, 48-bit VAs, pgdp=000000007294f000
[   96.226994] [0000000000000000] pgd=0800000072947003, p4d=0800000072947003, pud=08000000717ff003, pmd=0000000000000000
[   96.227079] Internal error: Oops: 0000000096000046 [#2] SMP
[   96.227119] CPU: 0 UID: 0 PID: 78 Comm: NIO-ELT-0-#2 Tainted: G      D            6.12.28 #1
[   96.227182] Tainted: [D]=DIE
[   96.227220] pstate: 41400005 (nZcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   96.227262] pc : __memset+0x16c/0x188
[   96.227301] lr : copy_process+0x374/0x1af8
[   96.227331] sp : ffff800080653b90
[   96.227394] x29: ffff800080653ba0 x28: ffff0a0cbab759c0 x27: 00000000ffffffff
[   96.227455] x26: ffff0a0cb26e6000 x25: ffff0a0cbab759c0 x24: 00000000007d0f00
[   96.227574] x23: ffff0a0cb5a4c000 x22: ffffd61178c1cf70 x21: 0000000000000000
[   96.227651] x20: 0000000000000000 x19: ffff800080653d90 x18: 0000000000000000
[   96.227770] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[   96.227832] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[   96.227883] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[   96.227964] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 000000000000003f
[   96.228003] x5 : 0000000000000040 x4 : 0000000000000000 x3 : 0000000000000004
[   96.228038] x2 : 0000000000003fc0 x1 : 0000000000000000 x0 : 0000000000000000
[   96.228076] Call trace:
[   96.228089]  __memset+0x16c/0x188
[   96.228109]  kernel_clone+0x88/0x33c
[   96.228131]  __do_sys_clone+0x6c/0xa0
[   96.228165]  __arm64_sys_clone+0x1c/0x24
[   96.228244]  invoke_syscall.constprop.0+0x58/0xf4
[   96.228277]  do_el0_svc+0x44/0xc8
[   96.228304]  el0_svc+0x50/0x1b8
[   96.228336]  el0t_64_sync_handler+0x134/0x150
[   96.228426]  el0t_64_sync+0x160/0x164
[   96.228490] Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428) 
[   96.228532] ---[ end trace 0000000000000000 ]---

[vsarunas]

Also running into this; containers can hang with this and consume all CPU cores that are given to the container on the host.

For example this has affected a buildkit several times for me while doing simple modifications and rebuilding the container:

container logs buildkit --boot
[...]
2025-11-17T14:32:34+0000 debug vminitd : sec=1763389955 usec=1123 [vminitd] setTime
2025-11-17T14:33:06+0000 debug vminitd : sec=1763389987 usec=1110 [vminitd] setTime
2025-11-17T14:33:39+0000 debug vminitd : sec=1763390019 usec=1819 [vminitd] setTime
2025-11-17T14:34:11+0000 debug vminitd : sec=1763390051 usec=1892 [vminitd] setTime
[307071.781701] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
[307071.781963] Mem abort info:
[307071.782027]   ESR = 0x0000000096000006
[307071.782175]   EC = 0x25: DABT (current EL), IL = 32 bits
[307071.782327]   SET = 0, FnV = 0
[307071.782442]   EA = 0, S1PTW = 0
[307071.782558]   FSC = 0x06: level 2 translation fault
[307071.782692] Data abort info:
[307071.782807]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[307071.782954]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[307071.783146]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[307071.783211] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000072639000
[307071.783293] [0000000000000008] pgd=0800000071e2e003, p4d=0800000071e2e003, pud=0800000071e31003, pmd=0000000000000000
[307071.783393] Internal error: Oops: 0000000096000006 [#1] SMP
[307071.783481] CPU: 0 UID: 0 PID: 9677 Comm: kworker/0:1 Not tainted 6.12.28 #1
[307071.783560] Workqueue: events delayed_vfree_work
[307071.783762] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[307071.783840] pc : rb_insert_color+0x10/0x140
[307071.783942] lr : insert_vmap_area+0xac/0xfc
[307071.784010] sp : ffff800080963cc0
[307071.784046] x29: ffff800080963cc0 x28: 0000000000000000 x27: ffffcb3431742bd8
[307071.784127] x26: 00000000000000e0 x25: ffff0844effc5e40 x24: ffffcb34317c0000
[307071.784188] x23: ffff0844714070c0 x22: 0000000000003434 x21: ffff084471405880
[307071.784249] x20: ffff0844741a84a8 x19: ffff084471467bd0 x18: 0000000000000001
[307071.784310] x17: 00000000f5257d14 x16: 00000000f5257d14 x15: 0000000000000002
[307071.784370] x14: ffff084472003080 x13: 0000000000000005 x12: 0000000000000000
[307071.784425] x11: 00011747b876eaa4 x10: 00000000000009c0 x9 : ffff800080963d40
[307071.784485] x8 : ffff084484118ec0 x7 : fefefefefefefeff x6 : ffff0844741a8498
[307071.784543] x5 : ffff0844741a8490 x4 : 0000000000000000 x3 : 0000000000000000
[307071.784597] x2 : ffff0844741a8490 x1 : ffff0844714070a8 x0 : ffff084471467be0
[307071.784657] Call trace:
[307071.784680]  rb_insert_color+0x10/0x140
[307071.784709]  free_vmap_area_noflush+0x104/0x180
[307071.784748]  remove_vm_area+0x38/0x78
[307071.784778]  vfree+0x38/0x27c
[307071.784808]  delayed_vfree_work+0x38/0x48
[307071.784838]  process_one_work+0x13c/0x2a8
[307071.784882]  worker_thread+0x298/0x468
[307071.784918]  kthread+0xe0/0xe4
[307071.784948]  ret_from_fork+0x10/0x20
[307071.784987] Code: f9400002 b40006c2 f9400043 37000323 (f9400464)
[307071.785171] ---[ end trace 0000000000000000 ]---

All 4 cpu cores being used up:

Probably a loop in the kernel?

[vsarunas]

Have migrated some CI builds for Linux arm64 to be within a container and unfortunately just swift build of a large project ends up killing the container:

[78964.040923] rcu: INFO: rcu_sched self-detected stall on CPU
[78964.041096] rcu: 	4-....: (1855968 ticks this GP) idle=6724/1/0x4000000000000000 softirq=16797/16797 fqs=796616
[78964.041238] rcu: 	(t=1864276 jiffies g=57929 q=10818 ncpus=6)
[78964.041307] CPU: 4 UID: 1001 PID: 8816 Comm: node Tainted: G      D            6.12.28 #1
[78964.041309] Tainted: [D]=DIE
[78964.041310] pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[78964.041311] pc : queued_spin_lock_slowpath+0x58/0x2c8
[78964.041316] lr : selinux_inode_free_security+0xb8/0xbc
[78964.041319] sp : ffff800088e43c10
[78964.041319] x29: ffff800088e43c10 x28: ffff1377c035f000 x27: 0000000000000000
[78964.041321] x26: ffff1377e0b08a80 x25: ffff1377c10a8020 x24: ffffa78c8415ac68
[78964.041322] x23: ffffa78c8414e300 x22: ffff1377e16bc728 x21: ffff1377c17befa0
[78964.041322] x20: ffff1377c16103c0 x19: ffff1377c17bef98 x18: 0000000000000000
[78964.041323] x17: 0000c00000000000 x16: 0000000000000014 x15: 0000000000000000
[78964.041324] x14: 0000000000000000 x13: ffffa78c84159a48 x12: 0000000000000000
[78964.041325] x11: ffff1377c0ab2400 x10: 0000000000000040 x9 : 0000000000000000
[78964.041326] x8 : 0000000000000019 x7 : 0000000000000019 x6 : ffff6bebabbcc000
[78964.041326] x5 : 0000000000000019 x4 : 0000000000000001 x3 : 0000000000000000
[78964.041327] x2 : 0000000000000000 x1 : 0000000000000001 x0 : ffff1377c1610400
[78964.041328] Call trace:
[78964.041329]  queued_spin_lock_slowpath+0x58/0x2c8
[78964.041330]  security_inode_free+0x58/0x74
[78964.041332]  __destroy_inode+0x74/0x1ec
[78964.041334]  destroy_inode+0x2c/0x70
[78964.041335]  evict+0x180/0x2d4
[78964.041336]  iput+0xd4/0x228
[78964.041337]  vfs_rmdir+0x1f4/0x224
[78964.041339]  do_rmdir+0x14c/0x1a4
[78964.041340]  __arm64_sys_unlinkat+0x5c/0x74
[78964.041342]  invoke_syscall.constprop.0+0x58/0xf4
[78964.041343]  do_el0_svc+0x44/0xc8
[78964.041345]  el0_svc+0x50/0x1b8
[78964.041347]  el0t_64_sync_handler+0x134/0x150
[78964.041348]  el0t_64_sync+0x160/0x164

[vsarunas]

@jglogan / @dcantah , do you need a reproducible ?

It just reproduces with many different stacks; for example here is running https://github.com/ncabatoff/process-exporter; within 30 minutes of system being up it become hung with:

2025-11-19T14:21:26+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=nil
[...]
2025-11-19T14:51:18+0000 debug vminitd : sec=1763563878 usec=36382 [vminitd] setTime
2025-11-19T14:51:50+0000 debug vminitd : sec=1763563910 usec=6811 [vminitd] setTime
[ 1826.419781] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000070
[ 1826.420068] Mem abort info:
[ 1826.420135]   ESR = 0x0000000096000007
[ 1826.420193]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1826.422026]   SET = 0, FnV = 0
[ 1826.422107]   EA = 0, S1PTW = 0
[ 1826.422162]   FSC = 0x07: level 3 translation fault
[ 1826.422224] Data abort info:
[ 1826.422250]   ISV = 0, ISS = 0x00000007, ISS2 = 0x00000000
[ 1826.422316]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 1826.422375]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 1826.422435] user pgtable: 4k pages, 48-bit VAs, pgdp=000000012ebbb000
[ 1826.422548] [0000000000000070] pgd=0800000145f3f003, p4d=0800000145f3f003, pud=0800000143a4e003, pmd=0800000141dd6003, pte=0000000000000000
[ 1826.422640] Internal error: Oops: 0000000096000007 [#1] SMP
[ 1826.422703] CPU: 3 UID: 0 PID: 603 Comm: process-exporte Not tainted 6.12.28 #1
[ 1826.425477] pstate: 81400005 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1826.425699] pc : bpf_prog_free+0x10/0x64
[ 1826.425820] lr : bpf_prog_destroy+0x3c/0x54
[ 1826.425885] sp : ffff80008145bb40
[ 1826.425940] x29: ffff80008145bb40 x28: ffff2ba068579000 x27: 0000000000000008
[ 1826.426012] x26: ffff2ba040386a40 x25: 0000000000000000 x24: 00000000400004d8
[ 1826.426082] x23: 0000000000040100 x22: 0000000000000200 x21: 0000000000000000
[ 1826.426159] x20: 0000000000000000 x19: 0000000000000000 x18: 0000000000000000
[ 1826.426231] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 1826.426301] x14: ffff2ba068579080 x13: 0000000000000000 x12: 0000000000000001
[ 1826.426371] x11: 000001a93f192143 x10: 00000000000009c0 x9 : ffff80008145bb30
[ 1826.426448] x8 : ffff2ba068579a20 x7 : 0000000000000000 x6 : 00007953f7a192e0
[ 1826.426518] x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff2ba06eb5d450
[ 1826.426588] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80008018d000
[ 1826.426663] Call trace:
[ 1826.426804]  bpf_prog_free+0x10/0x64
[ 1826.426901]  bpf_prog_destroy+0x3c/0x54
[ 1826.426959]  __put_seccomp_filter+0x5c/0x9c
[ 1826.427023]  seccomp_filter_release+0x70/0xb0
[ 1826.428242]  do_exit+0x150/0x898
[ 1826.428354]  do_group_exit+0x30/0x90
[ 1826.428408]  copy_siginfo_to_user+0x0/0x7c
[ 1826.428462]  do_signal+0x78/0x9a8
[ 1826.428522]  do_notify_resume+0xdc/0x13c
[ 1826.428576]  el0_svc+0x100/0x1b8
[ 1826.428642]  el0t_64_sync_handler+0x134/0x150
[ 1826.428702]  el0t_64_sync+0x160/0x164
[ 1826.428847] Code: a9be7bfd 910003fd f9000bf3 f9401c13 (f9403a60)
[ 1826.428958] ---[ end trace 0000000000000000 ]---
[ 1826.429063] Fixing recursive fault but reboot is needed!

[vsarunas]

Upgrading to Kata container kernel in 3.23.0 instead of the stock 3.17.0 still had a crash:

[    0.093742] Run /sbin/vminitd as init process
2025-11-21T09:34:46+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=nil
[...]
2025-11-21T10:03:33+0000 debug vminitd : sec=1763719414 usec=269 [vminitd] setTime
2025-11-21T10:04:06+0000 debug vminitd : sec=1763719446 usec=178 [vminitd] setTime
2025-11-21T10:04:37+0000 debug vminitd : sec=1763719478 usec=10058 [vminitd] setTime
[ 1818.407813] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
[ 1818.410035] Mem abort info:
[ 1818.410407]   ESR = 0x0000000096000006
[ 1818.411686]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1818.412448]   SET = 0, FnV = 0
[ 1818.412555]   EA = 0, S1PTW = 0
[ 1818.412721]   FSC = 0x06: level 2 translation fault
[ 1818.412819] Data abort info:
[ 1818.412920]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[ 1818.413009]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 1818.413130]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 1818.413228] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001337fd000
[ 1818.413396] [0000000000000008] pgd=0800000135407003, p4d=0800000135407003, pud=08000001355c3003, pmd=0000000000000000
[ 1818.413560] Internal error: Oops: 0000000096000006 [#1] SMP
[ 1818.413951] CPU: 3 UID: 0 PID: 728 Comm: nomad Not tainted 6.12.47 #1
[ 1818.414081] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1818.414188] pc : down_write+0xc/0x4c
[ 1818.414311] lr : vma_prepare+0x6c/0x140
[ 1818.414398] sp : ffff800080c13a60
[ 1818.414496] x29: ffff800080c13a60 x28: 0000000000000000 x27: ffff3594355ea428
[ 1818.414643] x26: 0000000000000000 x25: 0000000000000000 x24: 0000ffff20000000
[ 1818.414789] x23: 0000000000000000 x22: 0000000000000000 x21: ffffffffff800000
[ 1818.414913] x20: ffff800080c13a98 x19: ffff800080c13ba0 x18: ffff800080c13cf0
[ 1818.415045] x17: 0000004000c00000 x16: 1fffe6b280006ba1 x15: 0000000000000001
[ 1818.415176] x14: ffffffffffffffff x13: 0000000000000000 x12: 0000000000000003
[ 1818.416073] x11: 0000ffff28020fff x10: ffffce55492957c8 x9 : ffff359400035d08
[ 1818.416261] x8 : ffff800080c13cf0 x7 : 0000ffff20020fff x6 : 0000000000000003
[ 1818.416436] x5 : 0000ffff1f7f0000 x4 : 0000ffff1f800000 x3 : 0000000000000001
[ 1818.416584] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000008
[ 1818.416747] Call trace:
[ 1818.416835]  down_write+0xc/0x4c
[ 1818.416905]  commit_merge+0xb4/0x2bc
[ 1818.416972]  vma_modify+0x2f0/0x684
[ 1818.417047]  vma_modify_flags+0x8c/0xc0
[ 1818.417139]  mprotect_fixup+0x88/0x294
[ 1818.417208]  do_mprotect_pkey.constprop.0+0x224/0x340
[ 1818.417290]  __arm64_sys_mprotect+0x18/0x70
[ 1818.417386]  invoke_syscall.constprop.0+0x58/0xf4
[ 1818.417475]  do_el0_svc+0x44/0xc8
[ 1818.417541]  el0_svc+0x50/0x1e8
[ 1818.418388]  el0t_64_sync_handler+0x134/0x150
[ 1818.418495]  el0t_64_sync+0x160/0x164
[ 1818.418571] Code: 97ffe634 d2800002 d2800023 f9800011 (c85ffc04)
[ 1818.418747] ---[ end trace 0000000000000000 ]---
2025-11-21T10:05:10+0000 debug vminitd : sec=1763719510 usec=10053 [vminitd] setTime
2025-11-21T10:05:42+0000 debug vminitd : sec=1763719542 usec=599 [vminitd] setTime
2025-11-21T10:06:13+0000 debug vminitd : sec=1763719574 usec=690 [vminitd] setTime
2025-11-21T10:06:46+0000 debug vminitd : sec=1763719606 usec=963 [vminitd] setTime
[ 1936.621573] Unable to handle kernel paging request at virtual address fffffffffffffff8
[ 1936.621652] Mem abort info:
[ 1936.621675]   ESR = 0x0000000096000006
[ 1936.621696]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1936.621723]   SET = 0, FnV = 0
[ 1936.621742]   EA = 0, S1PTW = 0
[ 1936.621760]   FSC = 0x06: level 2 translation fault
[ 1936.621782] Data abort info:
[ 1936.621796]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[ 1936.621813]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 1936.621834]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 1936.621851] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000070bfb000
[ 1936.621894] [fffffffffffffff8] pgd=0000000070eda003, p4d=0000000070eda003, pud=0000000070edb003, pmd=0000000000000000
[ 1936.621944] Internal error: Oops: 0000000096000006 [#2] SMP
[ 1936.621973] CPU: 2 UID: 0 PID: 8360 Comm: containerd-shim Tainted: G      D            6.12.47 #1
[ 1936.622112] Tainted: [D]=DIE
[ 1936.622133] pstate: 21400005 (nzCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1936.622161] pc : plist_add+0x54/0x104
[ 1936.622184] lr : __futex_queue+0x4c/0x5c
[ 1936.622213] sp : ffff80008176bbd0
[ 1936.622228] x29: ffff80008176bbd0 x28: ffff35941adc8000 x27: 0000000000000000
[ 1936.622267] x26: 0000000000c81090 x25: 0000000000000000 x24: ffff35941adc8000
[ 1936.622310] x23: 0000000000000000 x22: 0000000000000002 x21: ffff3594003a0680
[ 1936.622341] x20: ffff35941adc8000 x19: ffff80008176bc28 x18: 0000000000000000
[ 1936.622379] x17: 0000000000000000 x16: 0000000000000000 x15: 0000004000075e78
[ 1936.622448] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 1936.622481] x11: 0000000000000000 x10: ffff80008176bc30 x9 : fffffffffffffff8
[ 1936.622521] x8 : ffff80008176bc40 x7 : ffff3594003a0688 x6 : 0000000000000000
[ 1936.622558] x5 : ffff800080b2bc28 x4 : 0000000000000064 x3 : ffff800080b2bc28
[ 1936.622588] x2 : 0000000000000000 x1 : fffffffffffffff8 x0 : ffff80008176bc28
[ 1936.622625] Call trace:
[ 1936.622640]  plist_add+0x54/0x104
[ 1936.622652]  futex_wait_queue+0x38/0xa8
[ 1936.622670]  __futex_wait+0xb8/0x164
[ 1936.622680]  futex_wait+0x74/0xf8
[ 1936.622699]  do_futex+0x12c/0x1d0
[ 1936.622710]  __arm64_sys_futex+0x74/0x13c
[ 1936.622729]  invoke_syscall.constprop.0+0x58/0xf4
[ 1936.622750]  do_el0_svc+0x44/0xc8
[ 1936.622769]  el0_svc+0x50/0x1e8
[ 1936.622792]  el0t_64_sync_handler+0x134/0x150
[ 1936.622808]  el0t_64_sync+0x160/0x164
[ 1936.622875] Code: aa0503e3 d1002129 aa0903e1 1400000c (b9400022)
[ 1936.622938] ---[ end trace 0000000000000000 ]---
2025-11-21T10:07:18+0000 debug vminitd : sec=1763719638 usec=1073 [vminitd] setTime
[ 1957.622574] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1957.622711] rcu: 	1-....: (5249 ticks this GP) idle=04bc/1/0x4000000000000000 softirq=37802/37805 fqs=2405
[ 1957.622830] rcu: 	(t=5251 jiffies g=82857 q=1252 ncpus=4)
[ 1957.622890] CPU: 1 UID: 0 PID: 8359 Comm: containerd-shim Tainted: G      D            6.12.47 #1
[ 1957.622891] Tainted: [D]=DIE
[ 1957.622892] pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1957.622893] pc : queued_spin_lock_slowpath+0x58/0x2c8
[ 1957.622898] lr : futex_wake+0x1b8/0x1cc
[ 1957.622900] sp : ffff8000817ebd20
[ 1957.622901] x29: ffff8000817ebd50 x28: ffff35941adc9000 x27: 0000000000000000
[ 1957.622902] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000001
[ 1957.622903] x23: ffff3594003a0680 x22: 0000000000000001 x21: 00000000ffffffff
[ 1957.622904] x20: 0000000000000000 x19: 0000000000000002 x18: 0000000000000000
[ 1957.622904] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 1957.622905] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 1957.622906] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 1957.622906] x8 : 0000000000000000 x7 : 00000000ffff3594 x6 : 00000000c2fb59be
[ 1957.622907] x5 : ffffce55495ea840 x4 : ffff3594003a0000 x3 : 0000000000000000
[ 1957.622908] x2 : 0000000000000000 x1 : 0000000000000001 x0 : ffff3594003a0684
[ 1957.622908] Call trace:
[ 1957.622909]  queued_spin_lock_slowpath+0x58/0x2c8
[ 1957.622911]  do_futex+0xbc/0x1d0
[ 1957.622911]  __arm64_sys_futex+0x74/0x13c
[ 1957.622912]  invoke_syscall.constprop.0+0x58/0xf4
[ 1957.622914]  do_el0_svc+0x44/0xc8
[ 1957.622915]  el0_svc+0x50/0x1e8
[ 1957.622915]  el0t_64_sync_handler+0x134/0x150
[ 1957.622916]  el0t_64_sync+0x160/0x164
[ 1962.982651] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-.... } 5681 jiffies s: 5325 root: 0x2/.
[ 1962.982812] rcu: blocking rcu_node structures (internal RCU debug):
[ 1962.982842] Sending NMI from CPU 3 to CPUs 1:
[ 1962.982861] NMI backtrace for cpu 1
[ 1962.982865] CPU: 1 UID: 0 PID: 8359 Comm: containerd-shim Tainted: G      D            6.12.47 #1
[ 1962.982867] Tainted: [D]=DIE
[ 1962.982868] pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1962.982869] pc : queued_spin_lock_slowpath+0x58/0x2c8
[ 1962.982875] lr : futex_wake+0x1b8/0x1cc
[ 1962.982876] sp : ffff8000817ebd20
[ 1962.982877] x29: ffff8000817ebd50 x28: ffff35941adc9000 x27: 0000000000000000
[ 1962.982879] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000001
[ 1962.982879] x23: ffff3594003a0680 x22: 0000000000000001 x21: 00000000ffffffff
[ 1962.982880] x20: 0000000000000000 x19: 0000000000000002 x18: 0000000000000000
[ 1962.982881] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 1962.982882] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 1962.982882] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 1962.982883] x8 : 0000000000000000 x7 : 00000000ffff3594 x6 : 00000000c2fb59be
[ 1962.982884] x5 : ffffce55495ea840 x4 : ffff3594003a0000 x3 : 0000000000000000
[ 1962.982885] x2 : 0000000000000000 x1 : 0000000000000001 x0 : ffff3594003a0684
[ 1962.982886] Call trace:
[ 1962.982887]  queued_spin_lock_slowpath+0x58/0x2c8
[ 1962.982889]  do_futex+0xbc/0x1d0
[ 1962.982889]  __arm64_sys_futex+0x74/0x13c
[ 1962.982890]  invoke_syscall.constprop.0+0x58/0xf4
[ 1962.982892]  do_el0_svc+0x44/0xc8
[ 1962.982893]  el0_svc+0x50/0x1e8
[ 1962.982894]  el0t_64_sync_handler+0x134/0x150
[ 1962.982895]  el0t_64_sync+0x160/0x164
2025-11-21T10:07:50+0000 debug vminitd : sec=1763719670 usec=820 [vminitd] setTime
2025-11-21T10:08:21+0000 debug vminitd : sec=1763719702 usec=694 [vminitd] setTime

Pointing to FS issues; this data is on a named volume:

[  316.386999] EXT4-fs (vdc): error count since last fsck: 49
[  316.389360] EXT4-fs (vdc): initial error at time 1763047411: mb_free_blocks:1945: inode 7389190: block 32900176
[  316.389498] EXT4-fs (vdc): last error at time 1763717687: ext4_mb_generate_buddy:1212
[ 1098.910243] EXT4-fs error (device vdc): ext4_readdir:262: inode #8298505: block 33194498: comm minio: path (unknown): bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 1098.961471] EXT4-fs error (device vdc): ext4_readdir:262: inode #8323077: block 33292802: comm minio: path (unknown): bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 2058.912364] EXT4-fs error (device vdc): ext4_readdir:262: inode #8298505: block 33194498: comm minio: path (unknown): bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 2058.944860] EXT4-fs error (device vdc): ext4_readdir:262: inode #8323077: block 33292802: comm minio: path (unknown): bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 2058.962065] EXT4-fs error (device vdc): ext4_readdir:262: inode #8331272: block 33325570: comm minio: path /data/ordo-minio/local/resources/library: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 3018.920173] EXT4-fs error (device vdc): ext4_readdir:262: inode #8298505: block 33194498: comm minio: path (unknown): bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 3018.950306] EXT4-fs error (device vdc): ext4_readdir:262: inode #8323077: block 33292802: comm minio: path (unknown): bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
[ 3018.966068] EXT4-fs error (device vdc): ext4_readdir:262: inode #8331272: block 33325570: comm minio: path /data/ordo-minio/local/resources/library: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0

After volume wipe; it is behaving slightly better and worked for ~1 hour fine so far.

I had this break as well:

container run --detach --name redpanda redpandadata/redpanda:v25.3.1 "start --overprovisioned --check=false"
container start redpanda

For me it broke on stop after running for an hour:

2025-11-21T09:28:14+0000 info vminitd : [vminitd] checking for shim var FOREGROUND=nil
[...]
2025-11-21T10:31:57+0000 debug vminitd : containerID=redpanda id=redpanda [vminitd] waitProcess
2025-11-21T10:31:57+0000 debug vminitd : flags=0 path=/run/container/redpanda/rootfs [vminitd] umount
[ 1309.199291] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 1309.199404] Mem abort info:
[ 1309.199424]   ESR = 0x0000000086000006
[ 1309.199450]   EC = 0x21: IABT (current EL), IL = 32 bits
[ 1309.199469]   SET = 0, FnV = 0
[ 1309.199482]   EA = 0, S1PTW = 0
[ 1309.199492]   FSC = 0x06: level 2 translation fault
[ 1309.199505] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000072600000
[ 1309.199535] [0000000000000000] pgd=08000000717d5003, p4d=08000000717d5003, pud=08000000717d6003, pmd=0000000000000000
[ 1309.199570] Internal error: Oops: 0000000086000006 [#1] SMP
[ 1309.199585] CPU: 0 UID: 0 PID: 105 Comm: kworker/R-dio/v Not tainted 6.12.28 #1
[ 1309.199627] pstate: 614000c5 (nZCv daIF +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 1309.199648] pc : 0x0
[ 1309.199661] lr : 0x0
[ 1309.199672] sp : ffff8000806d3d50
[ 1309.199685] x29: 0000000000000000 x28: ffff520ab5a55300 x27: 0000000000000000
[ 1309.199722] x26: 0000000000000000 x25: ffff520ab2680000 x24: ffff520ab2680580
[ 1309.199791] x23: ffffc492b871ab94 x22: ffffc492b8b2a910 x21: ffff520aeffa0480
[ 1309.199834] x20: 0000000000000000 x19: 0000000000000000 x18: 0000000000000001
[ 1309.199858] x17: 0000000000000001 x16: ffff520ab2213ea8 x15: 0000000000000000
[ 1309.199882] x14: ffff520ab2680080 x13: 0000000000001000 x12: ffff520ab5499300
[ 1309.199904] x11: 00000130d25b71a5 x10: 00000000000009c0 x9 : ffff8000806d3d30
[ 1309.200013] x8 : ffff520ab2680a20 x7 : 0000000000000000 x6 : 00000130d25b7000
[ 1309.200092] x5 : 00000000000001a5 x4 : ffff520ab26aca50 x3 : 00000000000000ff
[ 1309.200168] x2 : 0000000000000048 x1 : ffff520ab2680000 x0 : ffffc492b8b3b9c0
[ 1309.200252] Call trace:
[ 1309.200278]  0x0
[ 1309.200298] Code: ???????? ???????? ???????? ???????? (????????)
[ 1309.200363] ---[ end trace 0000000000000000 ]---
[ 1309.200416] note: kworker/R-dio/v[105] exited with irqs disabled

[vsarunas]

@jglogan / @dcantah reproduces with stress testing utility for Linux kernel, stress-ng:

container run --name stress --memory 8G --detach ghcr.io/colinianking/stress-ng --class filesystem --all 2

[  203.678832] EXT4-fs warning (device vdb): ext4_block_to_path:105: block 1589763404 > max in inode 31252771
[  203.678916] EXT4-fs warning (device vdb): ext4_block_to_path:105: block 1755249628 > max in inode 31252771
[  203.679018] EXT4-fs warning (device vdb): ext4_block_to_path:105: block 1593514375 > max in inode 31252771
[  203.679112] EXT4-fs warning (device vdb): ext4_block_to_path:105: block 2135646739 > max in inode 31252771
[...]
[ 4771.979389] stress-ng-iomix (467): drop_caches: 1
[ 4777.296423] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000004
[ 4777.296959] Mem abort info:
[ 4777.297086]   ESR = 0x0000000096000006
[ 4777.297161]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 4777.297303]   SET = 0, FnV = 0
[ 4777.297423]   EA = 0, S1PTW = 0
[ 4777.297451]   FSC = 0x06: level 2 translation fault
[ 4777.297486] Data abort info:
[ 4777.297541]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[ 4777.297591]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 4777.297626]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 4777.297682] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000107535000
[ 4777.297715] [0000000000000004] pgd=0800000107537003, p4d=0800000107537003, pud=0800000107538003, pmd=0000000000000000
[ 4777.297784] Internal error: Oops: 0000000096000006 [#1] SMP
[ 4777.297838] CPU: 3 UID: 0 PID: 122 Comm: stress-ng-fcntl Not tainted 6.12.28 #1
[ 4777.297927] pstate: 21400005 (nzCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 4777.297996] pc : selinux_task_kill+0x50/0x9c
[ 4777.298046] lr : security_task_kill+0x64/0x88
[ 4777.298093] sp : ffff800080a1bce0
[ 4777.298127] x29: ffff800080a1bce0 x28: ffff27d9477f6000 x27: 0000000000000000
[ 4777.298264] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 4777.298357] x23: 0000000000000000 x22: 0000000000000000 x21: ffff800080a1bdf8
[ 4777.298451] x20: ffff27d960696000 x19: ffffaf97f0072340 x18: 0000000000000000
[ 4777.298526] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 4777.298603] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 4777.298663] x11: 0000000000000040 x10: ffff27d941bdf008 x9 : ffff27d941bdf000
[ 4777.298754] x8 : ffff27d95f8bb4a0 x7 : 0000000000000000 x6 : 0000000000000000
[ 4777.298820] x5 : 0000000000000000 x4 : ffff27d960696000 x3 : 0000000000000020
[ 4777.298894] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000007
[ 4777.299224] Call trace:
[ 4777.299237]  selinux_task_kill+0x50/0x9c
[ 4777.299345]  security_task_kill+0x64/0x88
[ 4777.299481]  check_kill_permission+0xc0/0x118
[ 4777.299658]  kill_pid_info_type+0x50/0x9c
[ 4777.299926]  __arm64_sys_kill+0x198/0x1f0
[ 4777.300041]  invoke_syscall.constprop.0+0x58/0xf4
[ 4777.300372]  do_el0_svc+0x44/0xc8
[ 4777.300977]  el0_svc+0x50/0x1b8
[ 4777.301185]  el0t_64_sync_handler+0x134/0x150
[ 4777.301711]  el0t_64_sync+0x160/0x164
[ 4777.301935] Code: f9432081 b983f842 f9404021 8b020021 (b9400421)
[ 4777.302132] ---[ end trace 0000000000000000 ]---
[ 4777.702465] ------------[ cut here ]------------

The container had to be killed as container stop was hanging; there has been numerous errors on the filesystem:

container run -ti -v  ~/Library/Application\ Support/com.apple.container/containers/stress:/fs ubuntu /bin/bash

root@6cac2d6a-24e8-44a5-aebb-8467029f9e74:/fs# e2fsck -y -f /fs/rootfs.ext4
e2fsck 1.47.0 (5-Feb-2023)
/fs/rootfs.ext4 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 6692 has zero dtime.  Fix<y>? yes
Deleted inode 6693 has zero dtime.  Fix<y>? yes
Deleted inode 6729 has zero dtime.  Fix<y>? yes
Deleted inode 6748 has zero dtime.  Fix<y>? yes
Inode 6801 extent tree (at level 2) could be narrower.  Optimize<y>? yes
Deleted inode 7086 has zero dtime.  Fix<y>? yes
Deleted inode 7103 has zero dtime.  Fix<y>? yes
Deleted inode 7139 has zero dtime.  Fix<y>? yes
Deleted inode 7728 has zero dtime.  Fix<y>? yes
Deleted inode 7897 has zero dtime.  Fix<y>? yes
Deleted inode 7898 has zero dtime.  Fix<y>? yes
Deleted inode 7976 has zero dtime.  Fix<y>? yes
Deleted inode 7977 has zero dtime.  Fix<y>? yes
Deleted inode 7985 has zero dtime.  Fix<y>? yes
Deleted inode 7992 has zero dtime.  Fix<y>? yes
Inode 8153 extent tree (at level 2) could be narrower.  Optimize<y>? yes
Deleted inode 8197 has zero dtime.  Fix<y>? yes
[....]
Deleted inode 32926964 has zero dtime.  Fix? yes

Deleted inode 32926965 has zero dtime.  Fix? yes

Deleted inode 32926966 has zero dtime.  Fix? yes

Deleted inode 32926967 has zero dtime.  Fix? yes

Deleted inode 32926969 has zero dtime.  Fix? yes

Deleted inode 32926970 has zero dtime.  Fix? yes

Deleted inode 32926971 has zero dtime.  Fix? yes
[...]

Entry '0e' in /tmp-stress-ng-filerace-43-1 (7910) has deleted/unused inode 31253486.  Clear? yes

Entry '11' in /tmp-stress-ng-filerace-43-1 (7910) has deleted/unused inode 31253542.  Clear? yes

Entry '0c' in /tmp-stress-ng-filerace-43-1 (7910) has deleted/unused inode 31253506.  Clear? yes

Entry '08' in /tmp-stress-ng-filerace-43-1 (7910) has an incorrect filetype (was 7, should be 1).
Fix? yes

Entry '12' in /tmp-stress-ng-filerace-43-1 (7910) has deleted/unused inode 31253569.  Clear? yes

Entry '08' in /tmp-stress-ng-filerace-42-0 (7899) has deleted/unused inode 31164655.  Clear? yes

Entry 'icta-0000' in /tmp-stress-ng-unlink-102-0 (10799) has deleted/unused inode 163236.  Clear? yes

Entry 'cxes-0000' in /tmp-stress-ng-unlink-103-1 (10797) has deleted/unused inode 115232.  Clear? yes

Entry 'stress-ng-metamix-82-0-885718026' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104478.  Clear? yes

Entry 'stress-ng-metamix-82-0-3397133553' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104509.  Clear? yes

Entry 'stress-ng-metamix-82-0-898341385' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104668.  Clear? yes

Entry 'stress-ng-metamix-82-0-1876526950' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104671.  Clear? yes

Entry 'stress-ng-metamix-82-0-2207258001' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104582.  Clear? yes

Entry 'stress-ng-metamix-82-0-3271850575' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104670.  Clear? yes

Entry 'stress-ng-metamix-82-0-2939373415' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104673.  Clear? yes

Entry 'stress-ng-metamix-82-0-3583782668' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104669.  Clear? yes

Entry 'stress-ng-metamix-82-0-1360320681' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104597.  Clear? yes

Entry 'stress-ng-metamix-82-0-3628824137' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104629.  Clear? yes

Entry 'stress-ng-metamix-82-0-3345030039' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104672.  Clear? yes

Entry 'stress-ng-metamix-82-0-1020483367' in /tmp-stress-ng-metamix-82-0 (12909) has deleted/unused inode 5104614.  Clear? yes

Pass 3: Checking directory connectivity
Unconnected directory inode 46926 (was in /tmp-stress-ng-filerace-43-1)
Connect to /lost+found? yes

Unconnected directory inode 206803 (was in /tmp-stress-ng-filerace-43-1)
Connect to /lost+found? yes

Pass 4: Checking reference counts
Inode 6699 ref count is 126, should be 2.  Fix? yes

Inode 46926 ref count is 3, should be 2.  Fix? yes

Inode 206803 ref count is 3, should be 2.  Fix? yes

Unattached zero-length inode 298561.  Clear? yes

Unattached zero-length inode 300414.  Clear? yes

Unattached zero-length inode 300569.  Clear? yes

Unattached zero-length inode 300596.  Clear? yes

Unattached zero-length inode 301347.  Clear? yes

Unattached inode 5103734
Connect to /lost+found? yes

Inode 5103734 ref count is 2, should be 1.  Fix? yes

Unattached inode 5103841
Connect to /lost+found? yes

Inode 5103841 ref count is 2, should be 1.  Fix? yes

Unattached inode 5103859
Connect to /lost+found? yes

Inode 5103859 ref count is 2, should be 1.  Fix? yes

Unattached inode 5103873
Connect to /lost+found? yes

Inode 5103873 ref count is 2, should be 1.  Fix? yes

Unattached inode 5103890
Connect to /lost+found? yes

Inode 5103890 ref count is 2, should be 1.  Fix? yes

Unattached inode 5103905
Connect to /lost+found? yes

Inode 5103905 ref count is 2, should be 1.  Fix? yes

Unattached inode 5103921
Connect to /lost+found? yes

Inode 5103921 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104264
Connect to /lost+found? yes

Inode 5104264 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104308
Connect to /lost+found? yes

Inode 5104308 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104311
Connect to /lost+found? yes

Inode 5104311 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104314
Connect to /lost+found? yes

Inode 5104314 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104477
Connect to /lost+found? yes

Inode 5104477 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104496
Connect to /lost+found? yes

Inode 5104496 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104523
Connect to /lost+found? yes

Inode 5104523 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104538
Connect to /lost+found? yes

Inode 5104538 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104554
Connect to /lost+found? yes

Inode 5104554 ref count is 2, should be 1.  Fix? yes

Unattached inode 5104568
Connect to /lost+found? yes

Inode 5104568 ref count is 2, should be 1.  Fix? yes

[...]
Free blocks count wrong for group #39 (3574, counted=32246).
Fix? yes

Free blocks count wrong for group #40 (3574, counted=32247).
Fix? yes

Free blocks count wrong for group #41 (25966, counted=32240).
Fix? yes

Free blocks count wrong for group #42 (32202, counted=32212).
Fix? yes

Free blocks count wrong for group #43 (29876, counted=32241).
Fix? yes

Free blocks count wrong for group #44 (26084, counted=32231).
Fix? yes

Free blocks count wrong for group #45 (13765, counted=32228).
Fix? yes

Free blocks count wrong for group #46 (13508, counted=32239).

[...]

Free inodes count wrong for group #3940 (6920, counted=8191).
Fix? yes

Free inodes count wrong for group #4019 (7247, counted=8191).
Fix? yes

Free inodes count wrong (33547762, counted=33380056).
Fix? yes

/fs/rootfs.ext4: ***** FILE SYSTEM WAS MODIFIED *****
/fs/rootfs.ext4: 174376/33554432 files (0.0% non-contiguous), 2171210/134217728 blocks

On 2 systems it reproduced the first time I ran this within an hour of running; but the 2nd time is running for longer.