had a dream last night that I was trying to ungzip a log archive and it wouldn't work, and it turned out that the software had independently gzipped every single line, terminated with a \n

man, fuck my brain for coming up with that one

bet Mark Adler never saw this one coming back in 1992

@gsuberland that nightmare is very similar to what the "bokeh" package does, at least in one project I worked on -- numpy arrays' data segments got gzipped and then base64-d into a... I forget... json literal?

@dstndstn aaaaaa, gzip into base64 is such a sin outside of narrow contexts like "I need to put this in a URL".

@gsuberland @dstndstn i know that bzip2|base64 will start with Qlp... (usually Qlpo... actually)

wanna know how? XMPP. XMPP file transfer, specifically, which did not work back when i used XMPP

@gsuberland related fun fact: all our lovely line-oriented unix tools (or maybe just the terminal, idk), and emacs, don't actually work super well on million-character-long lines

@gsuberland @dstndstn but also, the Amaranth playground uses a custom Zstandard dictionary so that it can compress your code snippets and put them in the URL and share them (this way i don't need to maintain a backend service)

@whitequark @gsuberland @dstndstn augh do they really force binary data into xml too what the hell

@whitequark @dstndstn oh yeah I forgot about that feature of zstd; I'm gonna have to remember that one for future use.

@gsuberland @dstndstn the space savings we got were in the tens-of-% range due to the dictionary (on our representative inputs)

@charlotte @gsuberland @dstndstn oh, no, it's far worse. i was pasting base64 data manually as the chat messages, because file transfer was so broken

iirc, XMPP file transfer used 0 and 1 characters (8 bit per bit) in a huge CDATA block or something