Counting the characters in ONE CHARACTER remains an open problem in computer science.

@fluepke it's very fun to, in rust, have someone ask "how do i do character indexing in a string, string[i] doesn't work" and then get to do the "okay so "character" is not particularly a meaningful term, what are you trying to do?" discussion

(anyways, IMO byte indexing/byte strlen is the most sensible way of doing it, especially if you're using UTF-8)

* PHP's strrev reverses a string byte-wise, thereby corrupting UTF-8. 🤦
* For Python's encode() UTF-16 is neither UTF-16-LE nor UTF-16-BE. 🤨
* NUL-terminated C-Strings. 💀
* strlen() 👻

Do you know further string processing horrors?

@5225225 @fluepke indexing by character changes depending on what font the user uses

@5225225 @fluepke also depending on what one means by character, the single codepoinr ﷽ consists of several characters

@fluepke

Another hole in this hole about the string size is the question about how many columns this string needs on your terminal!

@Kowalski @fluepke which is unanswerable in general(but generally correct) unless you actually print it and measure it

see https://www.5snb.club/w/terminal-width-libraries/

@fluepke @ryanc To be fair computer science has this solved, where we write \math{1=1} and move along, possibly with a \snarkyfootnote about edge cases. It’s software that’s screwed it all up. 😇

@adamshostack @ryanc

Computer scientists be like:
Let Σ = {a, b, c} be an alphabet. A string is an element of Σⁿ. DONE. STRINGS SOLVED. QED. 👏

Engineers meanwhile:
Humanity has spent thousands of years developing hundreds of writing systems and countless ways to express meaning through text. Let's account for that!

Meanwhile, some guy writing C since 1978:
Wait, there's writing beyond ASCII? Sounds like a skill issue. If you need umlauts, you deserve the extra screening at US immigration!

@fluepke This is your alarm clock: It's time to exit this rabbit hole and enter the next rabbit hole.

@fluepke can I interest you in unsigned char based strings? :3

@fluepke isn’t there utf-16-le/be in python to explicitly define the byte order?

@souliboi yes. If you do not specify endianess, python's encode("utf-16") adds a byte order mark, that is not present when using encode("utf-16-le") or utf-16-be.

I find it kinda funny, that str.encode("utf-16") != str.encode("utf-16-le") != str.encode("utf-16-be")

@fluepke @souliboi

Use of UTF (other than perhaps UTF8) without a BOM is an anti-pattern of assumed context.
If it conforms to LOCALE, that assumption is valid, but only until the octet stream is shared to other LOCALEs.

Collecting up test cases for multiple encodings is worthwhile (i18n good) but a PITA, made worse (2×) by BOM optionality. 🤦‍♂️🤷‍♂️

@n1vux @souliboi where does the BOM go during string concatenation?

Shall it be removed upon octet_string.decode() or shall it be kept in-place?

What's your take regarding this cluster fuck?

>>> (s.encode("utf-16")).decode("utf-16-le") == s
False

>>> (s.encode("utf-16")).decode("utf-16-be") == s
False

@fluepke In MySQL, when you ask for "utf8" encoding, you don't get UTF-8 encoding. You get "UTF-8, but not beyond U+FFFF".

If you want "UTF-8, like, really" you have to ask for "utf8mb4" encoding.

@fluepke@chaos.social

wait which language gives string length in grapheme clusters as the default normal string length????

@ramona Swift (if .count() … counts (pun intended!)):

🏳️‍⚧️".count

and Elixir:

String.length("🏳️‍⚧️")

@ramona @fluepke swift, kinda

from the docs:
let flag = "🇵🇷"
print(flag.count)
// Prints "1"
print(flag.unicodeScalars.count)
// Prints "2"
print(flag.utf16.count)
// Prints "4"
print(flag.utf8.count)
// Prints "8"

@fluepke@chaos.social @buffet@chaos.social

woah, wild
though at this point i am of the opinion that having a .count/.length/whatever method on a string is a language design mistake and all the methods should explicitly say which length they give (and in case of grapheme cluster allow specifying which kind and in which locale because text is a mess)

@ramona @buffet 💯

@ramona @fluepke @buffet I mean, that's just good common sense. Like you should never have a `delay` parameter/variable name, but it should be `delay_s` or `delay_ms` or whatever the case may be.

@henryk @ramona @buffet the programming language should be strongly typed with compile-time checks and only accept a duration as input param.

Not everything is an int of unclear size (or even worse: void* 👻)

@fluepke @adamshostack @ryanc

The problem with Computer Science is much of it has pro Capitalism/Colonialism Bias.

Others will disagree, but I think that's a problem. You cannot disagree because consider who built these systems, what do you think the purpose was?

@cyberquixote @adamshostack @ryanc I do fully agree. ASCII is modern-days colonialism and racism, expressed as code.

Everyone, including Mastodon, should be severely ashamed for only supporting ASCII. There is absolutely no excuse for artificially restricting text input to something that is incapable of Unicode (using whatever encoding).

@fluepke Well… once you know how to do it, it all makes perfect sense… 🤡

@alvar I always thought, Perl is famously known for being very good at text manipulation …

For me, a programming languages' string representation and APIs have always been a good litmus test for the quality of the language's design.

Some languages get the edge cases remarkably right. Rust, for example, understands that a filesystem path isn't necessarily a Unicode string and gives it its own type.

And then, there's Python 2.7 or Java. Or PHP even.

@5225225 @fluepke how is byte indexing useful? You can’t do anything meaningful at the byte level, it’s as meaningful as accessing the bits in a floating point value.

@thias @fluepke because you know that "the string you're interested in" is &string[500..600] (as in, you did a regex match or something and got back the start,end, and now you can use that to index into the string).

and it doesn't pretend to be useful in isolation the way unicode scalar value indexing does (you can write code that is "correct" up until you write 🏳️‍⚧️ or any other multi-codepoint emoji)

@goes2hard @fluepke (omg kris hi)

yeah. indexing by that is really expensive (since it's basically a linear scan) but you can definitely iterate over bytes/codepoints/USVs

swift does something like this (and it means their "char" has to basically be a straight up string, see https://developer.apple.com/documentation/swift/character )

@goes2hard @fluepke uhhhh that works, yeah, though at the cost of memory(you need to store an index for everything you might want to index by) and it makes mutation hard

ultimately a lot of use cases for string processing don't actually need indexing, they need iteration, and iteration can be done cheaply without a precomputed index

@fluepke Japanese characters can be written in wide or narrow form, which have different Unicode encodings. the width depends on the device you're typing on, with narrow being common as a default on mobile soft-keyboards and wide being the typical default when writing on a desktop keyboard. as such, passwords written in CJK scripts may encode to different byte sequences depending on whether the user logs in via mobile or desktop, and therefore produce different hashes/keys.

@fluepke you have to run NFC normalisation over the string first to get a consistent byte sequence.

however! this is not actually sufficient on its own. another gotcha is that the normalisation result depends on the Unicode version. for long term stability (e.g. if you're using the password as an input to a KDF for bulk storage encryption) you need to pin the Unicode version alongside the hash. failing to do so many result in a user no longer being able to recover data.

@fluepke and since a new user password might contain grapheme clusters that only exist in newer Unicode versions, you either need to have a version upgrade path (no global version pin) or reject any password that contains an unrecognised cluster.

this is even more difficult if you're doing any kind of client-side / server-side agreement, especially across distinct platform and language stacks, because both sides need to be on the same Unicode version and be doing the same normalisation.

@fluepke also: if your threat model includes timing attacks, you need to do the Unicode cluster validity checks and NFC normalisation in constant time.

@fluepke also also: if you're doing any kind of nonrepudiation of string equality where malicious modification is in scope you also need to authenticate the Unicode version that was used, e.g. in the AD portion of an AEAD.

@fluepke the noun phrase “Unicode code point” in this meme should be replaced with “Unicode scalar value”.

the surrogate pairs are two code points which encode a single scalar value. and although it is most correct to say the UTF-16 string contains 6 code units, it is also valid to say it contains 6 code points; whereas the same string in UTF-8 definitely only contains 5 code points. both, however, unambiguously contain exactly 5 scalar values (not counting the BOM or the NUL, which are depicted in the encoding and are both scalars, but they’re definitely not part of the 🏳️‍⚧️) (and yes, a scalar value is a code point, so the UTF-16 🏳️‍⚧️ contains both 5 and 6 code points at the exact same time, which is exactly why i think the meme shouldn’t use this term)

@charlotte @fluepke @5225225 i think that by ANY sensible definition of character that any programmer can ever make sense of, ﷽ cannot possibly be more than one character, because it is NFC, NFD, NFKC, and NFKD, of itself.

however, “ä” could reasonably be considered multiple characters programmatically, even though it is a single codepoint, its NFD and NFKD aren’t