Hacker News new | past | comments | ask | show | jobs | submit login
Ubuntu 24.04 (and Debian) removed libsystemd from SSH server dependencies (fosstodon.org)
117 points by 3v1n0 7 months ago | hide | past | favorite | 60 comments



As per the well known XZ-utils backdoor, we decided to take a step further and drop libsystemd dependency altogether by implementing the missing bits with few small patches (one upstream, the other to be forwarded).


Bravo. On BSD somehow OpenSSH doesn't require systemd, but on Linux it did. Magic.


As a Debian user since the 1990s, this situation has once again made me thankful that I've moved important systems I'm responsible for away from Debian and over to OpenBSD and FreeBSD whenever possible.

Even if Debian wasn't perfect before systemd was introduced, at least I knew there was a very high probability that I could trust it to function well.

That stopped being the case after systemd was introduced. I've had far too many problems caused by systemd, to the point where all trust I had in newer versions of Debian has been lost.

Initially, I thought that maybe the problem was with me. But as I investigated the issues I was having with systemd, I'd see so many other bug reports, forum postings, mailing list postings, IRC logs, blog articles, and other online communications from people who were also having many other problems with systemd.

Debian offered a much better user experience before it switched to systemd, and a much worse user experience since.


What problems did you have? I really haven't come across problems that were much different than what I had with init.


It’s interesting you say that? It runs completely counter to my experience coming from the “BSD-style” rc.d init Arch used to use, and migrating to systemd.

Any specific issues? I didn’t see any. No offense. One factor may be that Arch prioritizes not patching upstream - helped save them from targeting here, and it doesn’t go overboard with default configs, which I’ve long appreciated.

Not to distro-war, I’m very grateful for Debian. My background is finding Linux in the mid-00s and breaking many SuSE, Ubuntu, and one or two Debian systems before finding something I could understand, repair, and maintain in 2008 Arch.

systemd accentuated its ability to stay relevant with enterprise Linux, made it even easier to package for, and has been a useful tool in diagnosing service issues and managing bad software for me.

I’m not sure how often it’s posted here but Benno Rice formerly of FreeBSD Core Team has an excellent and amusing discussion of systemd’s technical merits.

https://youtu.be/o_AIw9bGogo


> I’m not sure how often it’s posted here but Benno Rice formerly of FreeBSD Core Team has an excellent and amusing discussion of systemd’s technical merits.

IMO he makes a couple good points (and a couple poor ones), but it’s about everything except technical merits. It’s more about social and philosophical aspects.


He talks about a lot of design decisions like the units architecture and users being able to deploy their own services, but does keep it fairly high-level.


On BSD, OpenSSH doesn't need to detect whether systemd is running or not (that's all that libsystemd does). The answer is always no.

Magic, indeeded.


Why would it require even systemd?


Systemd manages daemons. Sshd is a daemon. It knows when sshd has finished starting up, in case something else is waiting on that, and if it dies, and whether it needs to be shut down. If systemd didn't do it, something else would.

That is not to say systemd isn't a hypertrophied pig, but it does do important work.


A back door in the same library is not likely. It drew attention and many looked into it.

What could be done to prevent supply chain attacks more broadly?


> A back door in the same library is not likely.

But libsystemd is not linked to xz only. By removing it, sshd is free of many other potential risks.


Correct, I misread.

Still the question remains: what technology could be implemented to mitigate this type of attack (beyond sshd)?

For example, Linux sandboxing is poor, and SeLinux is not usually enforced.


Disable calling functions in transitive dependencies, force them to be direct dependencies.

Why should sshd be allowed to call an xz function directly without xz being an immediate dependency.

I'm not sure what all that would entail with the ifunc stuff, but I remember encountering a glibc linking change moving from Red Hat 6 to RH7 that did something similar and broke the build process for some legacy code.


> what technology could be implemented to mitigate this type of attack (beyond sshd)?

UNIX ? (do one thing and do it well).

When your program links with 20 libraries, i have i very hard time to believe that security is one of your goals.


See: Qubes OS.


I pointed out this weakness the other day on the internet. I got attacked by open source software armies.


Maybe it was because you weren't pointing out anything new?

There was a pull request to stop linking liblzma into libsystemd a month before the backdoor was found

https://github.com/systemd/systemd/pull/31550

This was likely one of many things that pushed the attackers to work faster, and forced them into making mistakes.


No. They couldn't get along with the idea that some open source software packages should also sometimes be cut away, just like all other packages.

They felt threatened by the idea that open source maintenance can ever go wrong and started attacking me. They argued closed source was worse.

That was not my point at all. I was not raising a weakness of open source. I was just pointing out that linking to libsystemd had that kind of problem.


FOSS zealots are not always security experts.


Please don't throw despicable systemd zealots and honorable open source zealots into the same bucket ;)


Please don't bring group-identity politics here.


Plot twist: systemd zealots were all sockpuppets


Thanks for sharing.


One way to reduce supply chain risks would be to reduce the supply chain. Less dependencies results in less risk.

For example, dropping the libsystemd dependency.


> What could be done to prevent supply chain attacks more broadly?

The UNIX philosophy was right all along - each tool does one simple thing.


I can't tell if this is sarcasm or not.

Most things we want to do are necessarily complex, so dogmatically adhering to "one simple thing" necessarily drives you towards a towering heap of composed dependencies.

If you want to get rid of the supply chain, you want everything specifically to be non-composable, so that everything has to be reinvented from scratch for that specific solution.


Try that supply chain attack on dropbear running as ssh server on devuan. Simple utility on a simple system made of simple utilities


Indeed it's not.

But this implies removing dependencies on various libraries, and keep a such important process small is already relevant, despite the fact that it will load PAM libraries, making it quite still prone to issues.


If you requested this 5 years ago every would think you were crazy...

Same as the suckless people. They were right after all.


Seems like the code they added in place of the call to the huge libsystemd should be in a little library that all the other programs that need to perform systemd_notify may use. Seems like systemd should ship that library, or split out the part of their massive (789981 bytes in the text section) library that does just that bit. And probably split out other parts.


> If you requested this 5 years ago every would think you were crazy...

That's a straw man. Nobody was ever pushing for people to import all of libsystemd just to use the communication protocol with systemd, that protocol was designed to be very easy and simple to implement on your own precisely so you didn't have to depend on anything else, libsystemd just happened to provide an implementation too, and somebody was lazy and imported that instead, but I seriously doubt that's what anyone thought was best practice.

Contrary to popular belief, systemd isn't about linking gigantic binaries and libraries together into a giant blob like everyone things (e.g. the standard nonsense line from detractors that e.g. `systemd-resolved` is "part of systemd" as in "part of the same binary", which it isn't), but about just letting programs talk to each other, so that you can get reliable, featureful integration on your desktop instead of everything being a half-working mess of shims and ad hoc communication, and providing a centralized service that can consistently and from first principles solve certain tasks, so that you don't have to have every single daemon reimplementing their own, or a central implementation that's a big pile of preprocessed shell scripts, spinlocks, edge cases, and bullshit.

> Same as the suckless people. They were right after all.

Right about what? Right in myopically judging software quality by "lines of code" and setting nonsensical arbitrary line limits, and as a consequence confusing a (poor) map for the territory, because while, yes, "few lines of code" can make software good in some ways (less buggy, etc), it can also make it quite bad in others (less featureful, doesn't handle edge cases, brittle, annoying to use), or just be completely unrelated? Or finding every possible way to vrware software that most definitely "sucks more" for the vast majority of users that don't randomly happen to perfectly align with it, software that sucks so bad people have to maintain patch lists to make it useable, with all the inherent problems with maintainability and stability over time patches incur?

Suckless is a cargo cult of tradition following a fundamentalist interpretation of its holy texts, refusing to innovate and actually make computing better, stubbornly sticking to a model left unchanged since the 70s, which wasn't even that great back then, and proud of it.


libsystem is such a bizarre abstraction covering far too much surface area. The name alone is a code smell. Why is the same library used for both internal service management and for services implementing on-demand launches and notifications?


I like parts of systemd very much: units and their sandboxing/isolation/chroot of processes.

Many other parts are terrible beyond measure: journald for example.

I think the concept of on-demand processes managed by the end manager is a good idea, but systemd is strong arming the services into accepting its philosophy.


What's wrong with journald? I do not know much about it, aside from having to use journalctl to view logs occasionally.


Logs being stored in binary is the main problem, I think.


You can of course copy them into whatever format you want, systemd has support for syslog style export.


Versus the logical dmesg|grep {what you are looking for} or tail -n 30 /var/log/messages.


Tell me you never tried

  journalctl | grep {term}
without telling anything.

Also you are probably never worked with a localised distros. Go try your luck at least on a non-Latin one.


you can just run journalctl -g {term} too lol


That is the systemd way, and what many of us have pointed out for years as a major risk of the approach.


This reads as though your objection is to the scope of systemd rather than its implementation detail, which isn’t where my objection lies.

I have nothing against the service management stack also addressing common principles like logging and on-demand starts a la inetd, but the notion that applications should link against a component of the service manager which is also used by the service manager boggles my tiny mind.


libsystemd is not being used by systemd itself.


I have never seen anybody point to it as a security risk before this happened. Would be happy to see a reference of somebody saying that prior to the xz event


Report is that shortly before the hole was reported, a PR had been posted requesting to remove the dependency on xz.


> Why is the same library used for both internal service management and for services implementing on-demand launches and notifications?

Poettering was a Microsoft fan. I guess he designed SystemD like svchost.


and the fact that it worked and still working means that he was right, also systemd was already fixing the huge dependecies for a while, this only accelerated the process


I thought Ubuntu is a downstream of Debian, did I take it wrong?


the relationship shifts, many parts are unclear.. especially after MSFT took direct interest in Ubuntu, with adding custom partition types (edit msft-reserved, msft-data) to the OS installer, and new code to require signed encryption keys to the UEFI boot; keys registered or directly issued by MSFT.


> MSFT_PRIVATE

The only evidence I can find of this... Is another comment from you. (Googled: '"MSFT_PRIVATE" partition'. One result.)


OK - try an LUbuntu Install disk for 22.04; run the installer, after Welcome,Location,Keyboard in the Partition task, create a new Partition Table; make a new disk partition, see the list of available partition types, choose ext4; see FLAGS [apple-tv-recovery,bios-grub,boot,diag,hidden,hpservice,lba,legacy-boot,lvm,msft-data,msft-reserved,palo,prep,raid,root,swap]


Those are standard partition types that have existed for decades.


That's from 'parted', the disk utility - not Canonical, Ubuntu, or even Microsoft

Best I can tell support for the 'res' type has existed at least since 2006; search for 'msft': https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdi...

Then in 2012 it was... an individual and Red Hat who introduced the 'msftdata' flag: https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdi...

I don't get the insinuation, it conflates things that span at least a decade. It sucks that Microsoft is where they are in secure boot... but one can manage their own policies/keys.

Support is a good thing, nobody is forced to use secure boot or things signed by Microsoft.


> Support is a good thing, nobody is forced to use secure boot or things signed by Microsoft.

ok it is true that these partition types are older than I previously assumed.. but the statement about "forced to use secure boot" .. that is not true at all, yes lots of devices are forced to use UEFI secure boot. In fact, a quote from a Debian derivative recently stated "we use a kernel from Canonical to support booting a wider range of devices" .. because they just want to do that? no one is making them do that? really, the market truth of laptops and cloud VMs is requiring UEFI in a calculated way, with the keys being issued and managed centrally from MSFT. info welcome


Is it commonplace for laptops to not allow disabling Secure Boot? I work with desktops and big iron; it's entirely optional here.

Even in clouds - it's optional based on your compliance goals/requirements. 90% of the time it's KVM/QEMU with an API. Secure boot isn't slowly taking that scene over either, it's still optional support.

I really don't get the impression that we're losing control over our boot processes. We use the things signed by MS because it's convenient and the average user can't be bothered to do their own enrollment.

I see how this can be "boiling the frog", in a sense, but it's a bit close to conspiracy for me.


yes It Is commonplace for laptops to not allow disabling Secure Boot .. one recent low-end laptop did not allow booting from any device except the soldered-in boot disk, which used only UEFI and signing keys.


It's not common, no offense but you clearly have an axe to grind and started out with an incomplete understanding, so [citation needed].

Also, please do name and shame, especially since MICROSOFT MANDATES THAT SECURE BOOT IS DISABLE-ABLE (on x86 devices, anyway, we'll see what happens with Snapdragon X Elite devices).


let's agree that real surveys of real equipment, with names and model numbers, are a "thing" .. yes


LOL mods.

1. The comment doesn't respond to any claims I made.

2. The comment doesn't do the bare minimum to back up their claims, after spouting already wrong information in this thread. (Sorry, not sorry, but if you're going to complain a device is non-compliant with MS's own SecureBoot requirements, it's really shouldn't be unexpected that someone might ask you to specify which device.)

3. Their comment is pithy, non-responsive, and doesn't advance the conversation in any meaningful way.

And mine is the one that gets modded?

EDIT: Also, a 22-hour old thread, with a reply getting flagged in <10 minutes. Hmmmmmmmmmm, wonder what that means.


It is, but some of these patches were originally from ubuntu side




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: