Everyone a happy random "ഊ" in your filenames.

That's UTF-16 0x0D0A which when read as UTF-8 gets interpreted as CRLF aka. a windows line break.

And when you used the UTF-8 function for trimming a string it'll break everything.

Oh and while we're on it also a happy "슜" which when read the same way will be "Â" followed by a string terminator control character.

Why I mention this? Well it just broke things. Also #KDE doesn't render them in the GUI.

#UTF8 #UTF16 #programming #development

@agowa338 @gsuberland the linux kernel treating paths and file names as binary without any specific encoding will never cease to cause fun issues

my favorite will always be adding 0x07 to file names so things that print the name as is to the terminal will cause it to ding

That not being rendered in the GUI actually made figuring this shit out even harder.

#bash also doesn't render them.

Thankfully #7zip showed them as "[]" for whatever reason and allowed me to copy the foldername raw into a hex editor to see what was going on.

I hate #encodings and especially #UTF16...

In #KDE these characters were just stripped from the GUI. However clicking on the folder worked and it in general behaved as if these characters weren't part of the folder name at all.

Where I stumbled over the bug was another application running in wine that was supposed to generate a folder tree and hash all of the files. Well it trimmed these characters not just from the GUI but also internally apparently and then failed to browse into that folder as - well - it mangled the name...

Oh and you can also have a UTF-16 whitespace.

#C and #Cpp programmers probably like them even more, as "EN QUAD" is 0x2000, aka. it ends on 0x00 which is also a string terminator...

Or just a regular space 0x0020

Oh and funnily depending on how I try to read that folder name I get all three ways returned...

So I don't really know what this folder is actually called on disk.

#Linux really needs some debug tool to just hand you the hex of files and folders in a specific path.

And I feel pitty for everyone that has to deal with #UTF8 vs #UTF16 on the daily.

And if you wonder what wrote these cursed file-/foldernames to disk, it was #rclone.

But they probably were already cursed on the HTTP remote it pulled them from...

Oh and finally the filesystem all of this played out on is #btrfs, so if it does have a way to dump the raw name that would also help in this case.

So is this folder now called:
* 0x20C29C0D0A
* 0x20C2
* 0x2000
* 0x0020

who knows.

@ChlorideCull @agowa338 @gsuberland they don’t even go all the way because you can’t have 0x00 or 0x2F in it

@charlotte @gsuberland @ChlorideCull

See the threat, apparently you can have 0x00 in it (or at least some ways of querying it show it that way).

Even the posix spec is inconsistent. In some places it says you can't have 0x00 (null byte) but in other places it says "NUL character". which with e.g. UTF-16 encoding is two null bytes.

I have no idea where this screwup is coming from, I'm still trying to make sense of it.

@agowa338 @charlotte @gsuberland @ChlorideCull ... does a posix system actually use utf-16?

if you forbid null bytes (which linux does), and you treat filenames as utf16, you couldn't have basic ASCII in your filenames. which is Obviously Wrong.

@5225225 @charlotte @gsuberland @ChlorideCull

I don't know, we're now at 5 different ways that different APIs return this filename.

I'm utterly confused and have no idea what it is called.

I suspect it is "somethingASCII<space><ChineseChars>", that would make sense given the origin of the file...

@agowa338 @5225225 @gsuberland @ChlorideCull yeah i don’t think any posix or linux api lets you have NULs in the file name since a NUL is the end of a c string

@charlotte @gsuberland @5225225 @ChlorideCull

In the posix docs is also something called a "wide string". But as I said before I've no idea about all of the quirks. I just know the basics that there shouldn't be a NUL and so on...

@charlotte @gsuberland @5225225 @ChlorideCull

But see my other posts here in the thread from a few minutes ago. It may just be a bunch of tooling being stupid with a name like "foo " aka. a trailing whitespace.

@agowa338 Who in their right mind uses anything other than alphanumeric characters in filenames anyway?

@hund @agowa338 Normal people.

@agowa338 @gsuberland @5225225 @ChlorideCull for wide strings the same applies. i also don’t think it’s supported past “it’s decoded to utf-8 and then passed to the 8 bit calls”

@charlotte @gsuberland @5225225 @ChlorideCull

Then it's literally tooling being stupid. I thought wide-string may be UTF-16 as there a NUL-character would have been two null bytes...

@agowa338 @gsuberland @5225225 @ChlorideCull i think it could be UTF-32 in some implementations. unclear. i don’t think anything uses wchar in practice on posix

@agowa338 @5225225 @ChlorideCull @gsuberland either way the same rule applies with NUL in wide strings. they mark the end of the wide string, meaning that a string cannot contain NUL. the underlying system APIs on linux do not support wide characters, but use regular c strings

@charlotte @gsuberland @5225225 @ChlorideCull

It was created by rclone cloning something from a http remote that probably was "<englishName><space><chineseName>". That's the reason I assumed it may be UTF-16 to begin with.

But it more and more looks like it's just "<englishName><space>" and tooling supposed to help debug such issues is dumb and does encoding and conversions internally before outputting something it pretends to be "RAW"...

@charlotte @gsuberland @5225225 @ChlorideCull

My thought process was that something creates an UTF-16 string where NUL would be 0x0000 and then it is incorrectly treated as a C-string later and the name is getting mangled.

But someone suggested to use strace on find -print0 and that one showed syscalls that literally returned just a trailing space without all of the other odities. So I tend to believe that one over the rest.