Debian Becomes Unresponsive after 6.1.0-55 Kernel Update

(bugs.debian.org)

174 points | by artooro a day ago ago

81 comments

  • flowerthoughts a day ago ago

    (Happy I upgraded to trixie two weeks ago.)

    > From: Salvatore Bonaccorso <carnil@debian.org>

    > I think this is this one: https://lore.kernel.org/linux-ext4/20261102-ext4-ntfs3-null-...

    https://salsa.debian.org/kernel-team/linux/-/merge_requests/...

    Seems people are happy with the proposed fix.

  • bajard 21 hours ago ago

    I'm not using Debian, but I'm getting gigabytes of Linux updates every week. This is getting out of control.

    • chneu 21 hours ago ago

      The other day I updated like 15 things on my Linux machine. When i restarted and updated my repos again I had almost 40 more updates.

      Its definitely getting ridiculous.

    • bikvfeee 3 hours ago ago

      I'm using Debian and I get a lot too. Why is that, does anybody know?

      • az09mugen 3 hours ago ago

        Mostly because of security issues/CVEs that were found with AI. For example I have an Apache running and it got updated to 2.4.71 because there was an critical CVE with Remote Code Execution https://www.cve.org/CVERecord?id=CVE-2026-119618 I think it will last for a while and then we will hit a "plateau", a "stable state" like before AI but with more secure software overall.

    • fhn 16 hours ago ago

      then don't uypdate

      • bajard 15 hours ago ago

        I got the policy: I don't update until I accumulate 1,000 packages. Last month, the total was over 8 GB, and that doesn't even account for packages that received multiple updates in the meantime. Some versioning jumped 4 or 5 times.

  • RetroTechie 21 hours ago ago

    In that dmesg log

      [    2.920022] i915 0000:00:02.0: firmware: failed to load i915/kbl_dmc_ver1_04.bin (-2)
    
      ...
    
      [    3.818174] r8169 0000:02:00.0: firmware: failed to load rtl_nic/rtl8168h-2.fw (-2)
    
    And a few more, just search for "fail". If NIC doesn't come up that explains why machine unreachable over ssh etc, right?

    Might be a filesystem hiccup, and some firmware blobs went missing, or something? Just speculating here. EDIT: links in the article mention some ext4 issues (and maybe ntfs too?).

    Might also be handy to save dmesg output from the kernel version that works fine, and lay output side-by-side.

    • whisperenjoyer 20 hours ago ago

      I was able to reproduce the ext4 issue with qemu/kvm guest using debian cloud bookworm qcow2 image. You just need to run stress-ng --class filesystem for some time. Definitely not network problem and not xen-specific one. I think most of ext4 users (at least with common mount options) are affected.

      • artooro 19 hours ago ago

        Yep, I was initially experiencing this problem with VMs on GCP. The busier the VM was handling user traffic, the sooner it triggered the bug.

  • hmm37 17 hours ago ago

    Interesting.. I was suddenly having issues with 6.1 and Manjaro on an old laptop a week ago. It kept randomly freezing, sometimes after an hour, other times 12 hours of uptime, etc. Once I changed the kernel to a different version like 6.18 or 7.3, it worked with no issues. I wonder if it's related.

  • walrus01 a day ago ago

    When people say "this is a VM on xen" I really wish they would provide on essential piece of troubleshooting data. As a domU, is it PV, PVH, full HVM, or HVM with PV drivers? There's a huge difference for how a modern xen dom0 handles the bootloader and kernel boot for those.

    At this point trixie is also extremely mature and has been out for a long time, and while I'm not saying this isn't a major issue, if lots of people are seeing it, it's also a solid argument in favor of upgrading those bookworm systems to the latest bookworm and then to trixie to begin following the 13.x stable branch. The 6.12.x kernel in trixie fixes a lot of issues related to both being a hypervisor and running inside a hypervisor.

    • shellwizard a day ago ago

      You can get mostly the same kernel 6.12 from debían Trixie in debian bookworm using backports, it was recently updated too

    • mitchweaver a day ago ago

      Nobody runs oldstable by choice.

      Often there are unresolved compatibility issues across Debian versions. Some are not necessarily tied to the kernel but other components.

      "Just upgrade bro" can be a recipe for punishment.

      • prmoustache a day ago ago

        Technologies such as the venerable chroot, schroot, lxc, lxd, docker, podman or systemd-nspawn have made sure that the applications component can always be independent of an unresolved compatibility across Debian versions.

        And it has been the case for literally decades.

      • pmontra 21 hours ago ago

        A customer of mine has been running a Debian 10 until a couple of months ago.

        "It's inside our VPN, no customer accesses that server, what could happen?"

        We upgraded only to be able to run a newer Java version.

      • fl0ki 17 hours ago ago

        Definitely not by choice, but sometimes I don't have physical access to a machine for years at a time, and don't want to risk losing my network access as well due to an upgrade hiccup.

      • walrus01 a day ago ago

        I would agree with you if trixie had just been released, but a lot of the rough edges have been worn off by now. Running the present branch of debian stable (as opposed to debian testing) is already an extremely conservative choice and gets you a lot of revisions of packages that are quite old compared to what ships in other distros...

        Trixie is 14 months old now

        • currysausage a day ago ago

          To give an example, there is a rough edge called Dovecot 2.4 (new config syntax, Director removed, master/master replication removed) that will not wear off by itself. Upgrading to 2.4 could, depending on the complexity of your setup, require days to weeks of planning, testing and implementation.

          • jeroenhd a day ago ago

            Upgrading complex setups is never easy, but people have had over a year to plan, test, and implement their upgrades.

            These people will need to upgrade at some point, though that point will probably be at the very last moment midway through 2028 if they're still on oldstable today.

            • nikanj 21 hours ago ago

              Spending mythical man months running an upgrade just to deliver the exact same service is a hard sell for the budget season.

          • erikvanoosten a day ago ago

            As a small self hoster, I found it easier tot upgrade the stack to Stalwart than to fix the Dovecot config.

          • harshreality a day ago ago

            Not only did a parallel subcomment point out the year they had to find some weeks or even months to plan an upgrade... but there are also dovecot 2.3 containers for the desperate: https://hub.docker.com/layers/dovecot/dovecot/2.3.21.1/image...

            • currysausage 19 hours ago ago

              > dovecot 2.3 containers for the desperate

              "Last pushed: over 2 years", while Dovecot 2.3 in Debian Bookworm still receives security patches backported from 2.4 through Debian LTS [1]. Relying on backported fixes (and LTS in general) isn't footgun-free, but I would prefer that over an abandoned Docker image.

              [1] https://tracker.debian.org/news/1817828/accepted-dovecot-123...

              • harshreality 15 hours ago ago

                It's worse than that. It's not just containers, but the core software. Dovecot 2.3.21.1: August 2024. Dovecot 2.4.0: January 2025.

                What does it say about the quality of the software if they stop maintaining an old version prior to the release of the new one?

                Maybe everyone would be better off switching to Stalwart then. I think this subthread has convinced me to do just that, even though I had no problems upgrading to dovecot 2.4.

                The container route is still feasible and easy: build a bullseye (or rhel-based rocky or anything like that with dovecot 2.3) container from base by installing the distro's dovecot, and trust that it's patched all the significant vulnerabilities. Anything to get away from old stable versions for a core OS that provides kernel and networking; not every container needs to be new stable.

        • nbernard a day ago ago

          Hardware doesn't grow younger magically. Upgrading is not always an option, even as a release becomes mature.

          • walrus01 a day ago ago

            The hardware requirements of a base CLI only server trixie are not greater than the same base CLI environment of bookworm. we're talking like, 300MB RAM when system booted with full systemd and openssh and a few other standard things running. The only major difference is that 32-bit/i386 cpu support was dropped. If you have an original intel core duo (not core 2 duo) from 2006 that isn't 64-bit, or older, yeah I could see that being a problem... but otherwise,

        • j16sdiz a day ago ago

          In many case, many software runs on the same server. Docker don't help much because the image can't build from dockerfile, because some dependency are not downloading.

        • mschuster91 a day ago ago

          > I would agree with you if trixie had just been released, but a lot of the rough edges have been worn off by now.

          That doesn't help if you need to run a piece of software only certified to run under a certain Java version or, worse, a certain OS version. It's either "run on something really old" or "lose the ability for commercial support / you're on your own if shit happens".

          • ada-h00k a day ago ago

            Can't you just put that in a container with an old version of Java?

            • pjmlp a day ago ago

              Older Java versions aren't container aware, thus it might not run as expected, depending on which version it is.

              • roryirvine 21 hours ago ago

                If you really need a JRE from before they became container aware by default then you're in luck!

                Both bookworm and trixie contain the obsolete OpenJDK v8 JRE for both amd64 and ppc64el as it's needed for nvidia-visual-profiler.

                • pjmlp 20 hours ago ago

                  Thankfully I have long moved away from Java 8.

            • natebc a day ago ago

              sometimes, sure but

              > "lose the ability for commercial support / you're on your own if shit happens"

              is not something you can clever your way into. It's the vendors choice what support characteristics look like.

            • burnt-resistor a day ago ago

              Containers !~= VMs (except Kata Containers and similar). VMs ensure specific, isolated, supported compatible kernel versions in controlled environments because what's supported / should work isn't the same as what's technically possible / might work. At scale, tiny differences matter.

              Containers seem lightweight and magical until the tyranny of the quirks of the kernel matters with some gotchas of supported configurations as opposed to unsupported gremlins running custom stuff that other vendors refuse to touch.

          • monegator a day ago ago

            oh yes, i've almost given up trying to run FPGA software on linux, official tools for some vendors require JDK 11 (sigh) or just ubuntu (double sigh)

            • mschuster91 a day ago ago

              eh I'd like a "just ubuntu" requirement much more than the usual "only RHEL is supported" slop...

              Yeah I get it, vendors want to keep their support matrix small. But RHEL (and its bunch of derivatives) just plain suck ass. At least offer support for something based on Debian...

        • mitchweaver a day ago ago

          How do I best explain this?

          I upgraded to Trixie. Stuff broke.

          I rolled back to Bookworm, which was extremely painful, because unlike other OSes rollback is not possible and a reinstall is necessary.

          Stuff worked again.

          Maybe I'll give it another go in 2028 when it goes EOL. Or maybe not. I'm not under any obligation to keep trying.

          • jeroenhd a day ago ago

            OS rollback is possible if you've configured it. A combination of btrfs + timeshift has served me very well after a failed Ubuntu upgrade. You can just boot into the previous snapshot from Grub.

            Although you can set this up after the fact (converting ext4 to btrfs is supported and you can update your mounts from any recovery distro), that's probably just something to keep in mind for your next fresh install. It's also extremely for any other kind of failed upgrade, though, not just full OS upgrades.

          • nbernard a day ago ago

            FYI, for the next time: with Debian, while rolling back is not supported, it often works.

            I had issues too with Trixie on a laptop (after upgrade everything became unbearably slow, probably due to an Intel graphic chip which was dropped by Mesa or some other library) and did a rollback to bookworm. IIRC, I only had to put back the old apt source entries then apt update and apt dist-upgrade...

            • mitchweaver 13 hours ago ago

              In this case it did not; I hit a circular dependency. The system ended up in an inconsistent state and I had to start over.

              So I am 0/1 on rollbacks.

            • tremon a day ago ago

              > IIRC, I only had to put back the old apt source entries then apt update and apt dist-upgrade...

              The problem with rollbacks is not the packages themselves, that usually works without issues. The problem is with configuration and data items: if a package changes configuration format on upgrade (e.g. move from pkg.conf to pkg.conf.d), a downgrade will not restore the old version (e.g. rebuild pkg.conf from the .d snippets).

              Also, apt will not downgrade packages without prodding. At the very least, you would need --allow-downgrades.

          • e12e a day ago ago

            You didn't have a backup? Or at least make a backup before upgrade?

          • NaaYYs a day ago ago

            Was a snapshot before upgrading not possible?

          • pvaldes a day ago ago

            Sometimes problems arise when trying to upgrade from "version 1" to "version 3" without passing by "version 2". Moving from oldstable to testing can broke things. You need to upgrade first your sources to stable and then to testing.

            Don't know if this is your problem without data. just my 2cents. But you may need to go back two versions and install it properly.

            Also if some command "X" don't work, launch it as "strace X" to have a deeper look at the problem.

          • LtWorf a day ago ago

            How do i explain this?

            If you don't report the bugs of what breaks (possibly while it's in testing) nobody can help you.

            • nbernard a day ago ago

              Not GP, but I've been using Debian for 25+ years and, as a user, I still don't understand how to report a bug the right way when it is unclear what package is the cause for a regression...

              • LtWorf a day ago ago

                The proper way is to figure out which package is causing a regression.

                As an alternative you can report a general bug that will end up on the debian-devel mailing list where someone might reassign it to the proper place. But low effort reports sent to generals are basically a waste of everyone's time.

                • LoganDark a day ago ago

                  This is only economical if you can afford the time wastage yourself. If you need a working system, you don't have the time to fix the bug yourself (hoping you won't find two more by the time you're done)

                  • LtWorf a day ago ago

                    You can pay freexian to fix it for you.

                    • LtWorf 21 hours ago ago

                      Lol, everyone has a successful startup here but the second you suggest buying for a service (that unlike AI actually provides value) here come the downvotes.

                      • LoganDark 20 hours ago ago

                        I don't think it's because it's paid, but because it doesn't entirely solve a time and energy issue.

          • walrus01 a day ago ago

            I'm not going to take seriously a single person's anecdote of "I upgraded to trixie, stuff broke" when I weigh it against the large number of people I know who run debian stable with very few operating-system-base caused problems for very serious production things.

            Your comment is devoid of any useful technical detail on what broke or why, and quite frankly reads like a very linux noob complaint. Leading me to think you did something wrong in the upgrade process, or failed to manually handle any basic configuration files that you were prompted with diffs/merge/install/keep existing version in the upgrade process.

            • csdreamer7 a day ago ago

              > Your comment is devoid of any useful technical detail on what broke or why, and quite frankly reads like a very linux noob complaint. Leading me to think you did something wrong in the upgrade process, or failed to manually handle any basic configuration files that you were prompted with diffs/merge/install/keep existing version in the upgrade process.

              Why is your first reaction is to assume the person broke something in the upgrade process or did something in the config files? The parent comment is devoid of useful technical detail, but assuming the user is at fault is terrible. Debian is very conservative between releases with not upgrading packages from upstream and keeps a lot out of tree patches. It is very possible something broke between the major versions.

            • LtWorf a day ago ago

              I mean, linux kernel broke userspace in a couple different instances between oldstable to stable. If his software happened to use those APIs it could crash. Just an example. But a normal person would fix their software (if they have the sources to do that!)

            • zettabomb a day ago ago

              Classic Linux bro response. Doing a very normal thing (system upgrade) leads to an abnormal result (broken software), blame the user.

              • prmoustache a day ago ago

                Sruff broke and broken software literally doesn't mean anything.

                FWIW people use the very same words when they forgot to turn on the monitor or plug their desktop computer to an outlet.

              • LtWorf 21 hours ago ago

                > blame the user

                The user who is just complaining without providing any sort of useful information that could be used to help them?

                • wrappajack 21 hours ago ago

                  It doesn’t appear the user is seeking help with the issue, they solved the problem themselves. They seem to be simply advising on their experience with the process.

                  Not sure how it’s any less valid than others or your own anecdotal experience.

                  • LtWorf 21 hours ago ago

                    > they solved the problem themselves

                    Leaving hundreds of unpatched security vulnerabilities open hardly seems "solving"

        • atoav a day ago ago

          Yes, but there is software you may rely on that has not yet adjusted to Trixie or if it has you have not had the time/resources to test that upgrade path.

          As someone who works in IT around administration of 30+ Debian boxes for a decade now: Updates are easy in theory only or if the software doesn't rely on any OS package. Sure, most updates of well written software are easy, but we also had to manually migrate database schemas and similar things.

  • anthk a day ago ago

    I always set two different kernels at boot because of this, an LTS one and the mainline version.

    • InsidiousWombat a day ago ago

      Debian based distribution generally keep older kernel by default for that reason

      • bolangi a day ago ago

        Well, with every upgrade, apt suggests that old linux-image packages be removed.

        • dizhn a day ago ago

          That works out because it does that while you're already booted into the other kernel.

        • dieulot 17 hours ago ago

          Not all of them — the previous kernel happens to be kept.

        • seba_dos1 a day ago ago

          So what? Count the remaining ones.

  • artooro 16 hours ago ago

    This is now fixed in package version 6.1.192-2

  • mitchweaver a day ago ago

    Oof. This is bad.

    I couldn't understand which Debian ran this strange kernel version, until I realized all my oldstable boxes run a backport kernel.

    Phew.

    Because oldstable was... pretty freaking old even when it was new.

    Either that or Santa Altman came down the chimney during the night and swapped all their RAM sticks out for bad ones.

    • cable2600 a day ago ago

      >Either that or Santa Altman came down the chimney during the night and swapped all their RAM sticks out for bad ones.

      Back in the early 1990s, when they had PC Clone shops, some PC Clone shops would buy from others when RAM chips were in short supply. Sometimes they were bad RAM chips and were sold to sabotage the other PC Clone shop.

  • okeuro49 a day ago ago

    Weren't people criticising Ubuntu recently for putting the TCP bug kernel update through thorough testing?

  • jalla a day ago ago

    And this affects stable versions how?

  • Aldipower a day ago ago

    I am glad I stayed with Ubuntu after the recent MPTCP kernel incident. SCNR

  • mococa 14 hours ago ago

    I blame vibe coding.

  • taxmantakes 21 hours ago ago

    Yeah I moved off Debian to red flag, they’re really good linuxers, runs really fast and in half the ram and cpu