Conversation

joepie91 needs more railing đźš…đź’¦ (was at eth0)

so anyway, does anyone happen to know whether the same series of UTF-16/UCS-2 codepoints *always* encodes to the same binary representation, or whether this can vary by some environmental factor like the unicode version?

2
0
0

@joepie91 utf-16 doesn't have codepoints. unicode has codepoints, utf-16 is an encoding.

and for an obvious 'no', there are big and little endian orderings.

1
0
0

joepie91 needs more railing đźš…đź’¦ (was at eth0)

@dysfun (I'm having to poke through a couple of layers of abstraction here to figure out whether I can trust Buffer.from in Node to be deterministic, hence the slightly fuzzy question)

1
0
0

@joepie91 trust what operation on a buffer to be deterministic? encode codepoints to utf16? that'll be potentially version dependent

1
0
0

joepie91 needs more railing đźš…đź’¦ (was at eth0)

@dysfun basically, trust `Buffer.from("some string")` to produce the same byte array across Node.js versions and across different systems

1
0
0

@joepie91 what kind of buffer is that? because as i recall js uses utf16 internally. so if you're reading and writing utf16, it's utf16?

1
0
0

joepie91 needs more railing đźš…đź’¦ (was at eth0)

@dysfun this is part of the complications; by default its 'encoding' argument defaults to 'utf8' which, as I understand it, means that even though the string internally is encoded as UTF-16, it will produce a byte array representing the string in UTF-8 encoding

0
0
0

@joepie91 it varies by endianness (so the bytes could be swapped between your average computer and big endian powerpc machines) only

0
0
1