I've seen lots of XZ hot takes lots of places, I'm sure more excrement will hit the impeller on Monday.
My thoughts:
- this was well planned, as many have said - it appears that the sole previous xz maintainer was having a rough time, and that was capitalized on, to introduce a new maintainer
- it looks like the reason the timeline was rushed or hurried was because systemd was going to be dropping the liblzma dependency, but the merge window for Fedora 40 and even more critically Ubuntu 24.04 were closing - can you imagine a backdoored xz making it into Ubuntu 24.04? The long tail would be a nightmare
- this wasn't caught because of OSS, but it did make RE on the bug possible or at least much more likely, and it happened out in the open
- the reason this was caught was because one engineer was curious about half a second of latency in SSH.
- everyone is going to be losing their minds about any version of xz made after Jia Tan became a contributor, anywhere, even on something like OpenBSD, for the next long while
my takeaways:
- OSS maintainers need more support, both financially and emotionally/professionally. Burnout is real.
- the xkcd meme probably has holes worn in it by now
- this was planned and orchestrated, though it is not clear if the maintainer's account was hacked or this was the long play
- slow software hides backdoors - can you imagine if someone backdoored Electron? no one would notice a difference.
Happy Easter! This is a good visualization of stuff that I spent too many hours reading through during the two days after the Big Reveal.
@larsmb Solarwinds, Midnight Blizzard, the Cisco / Juniper backdoor of the month, etc etc. Plenty of examples. Usually take months (if not years) to be found, often way longer to be fixed.
People buy proprietary products to pass along responsibility, then invest heavily to maintain their illusion of "having done everything that can be done" to stay secure.
I can't wrap my head around how almost all of the #xz reporting focuses on the failures of #opensource.
Yeah, sure, but ...
Good luck finding such an attack in proprietary code.
Via the cliché paid off/blackmailed employee, hacked dev servers/repos, or via capitalism's favorite cost-cutting measure: a remote "offshored" contracted temporary developer (or nowadays, embedded into some LLM output).
If anything, Open Source Security has *worked*.
travelynx broke so I can't see which trains are departing from the station I'm on
call that trains day of invisibility
Alright, here it is, our entry in the Wild compo for #revision2024 : "A Statement on the Platform Wars" by Moonbase Allstars, a fun little binary that's rather portable
https://demozoo.org/productions/342318/ / https://www.pouet.net/prod.php?which=96567
(video is on YouTube now: https://www.youtube.com/watch?v=7j818mS77qQ&t=1684 )
"Semicolon is very much needed in C++ because what if you need to put two statements on the same line huh?"
*Rarely ever puts two statements on the same line because it's "bad form" or something.*
I remember some two decades ago... a professional C++ programmer contacted me in frustration, having pulled his hair out for literally hours. His code just wouldn't run. It was a missing semicolon which I spotted in one minute.
As bad as this #xz compromise is, it shows some things:
* #OpenSource "works". If this would have happened with a closed source app/lib, noone would have been able to spot this.
* Curiosity is nice. "Why is this taking longer" (by something of zero dot small seconds) and ending up with this....
* #Reproducible builds seem to allow to check that this didn't affect more - rebuild things with a known clean install and compare output with what the #distros had.
* Critical projects need a way to *reliably* get more maintainer power. And yes, that is easily said, not easily solved, I know. How likely do you think the xz maintainer is now with trust for others? How much would you trust others?
(Also, what is critical?)
* Proper funding might help. Companies that use and rely on open source (you know who you are. MANY you are!) ought to give something back. (And not just a token amount). Also, yes, another not-easy-to-solve problem comes up with this. But not-easy is no reason to not try.
As the xz thinkpieces start showing up about What Should Be Done, a couple of questions I'd encourage you to keep in mind while reading them:
- Is this advocating security nihilism and giving up because stopping 100% of badness is impossible?
- Is this pushing a random hobby horse like "sign your commits" that wouldn't have helped this incident in any way?
- Is this equating employment/nationality/notoriety with trustworthiness?
- Is this pushing a technical solution to a social problem?
The sad part about today's xz/liblzma discovery is that again critical infrastructure was maintained by overworked volunteers without sufficient assistance or support. We, as professional software engineers, or even we, as society, relying on their volunteer work, failed them.
https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html
So, kids, what's the moral of the XZ story?
If you're going to backdoor something, make sure that your changes don't impact its performance. Nobody cares about security - but if your backdoor makes the thing half a second slower, some nerd is going to dig it up.
I'm watching some folks reverse engineer the xz backdoor, sharing some *preliminary* analysis with permission.
The hooked RSA_public_decrypt verifies a signature on the server's host key by a fixed Ed448 key, and then passes a payload to system().
It's RCE, not auth bypass, and gated/unreplayable.
More details in this thread: https://bsky.app/profile/did:plc:x2nsupeeo52oznrmplwapppl/post/3kowjkx2njy2b
by age 30 you should have spent years contributing to an open source project in your free time, building trust and becoming a maintainer, and then have skilfully inserted a highly obfuscated backdoor that successfully made its way into multiple linux distributions
The FTC has confirmed what we suspected: During the pandemic supply chain disruptions, big grocery chains pressured food suppliers to favor them over smaller competitors and hiked food prices to rake in record profits.
Those prices have stayed high even as disruptions eased.