Redefine char and unicodeChar for correct Unicode usage #71
Conversation
mbtaylor
commented
Aug 7, 2025
mbtaylor
commented
- char primitives are now UTF-8-encoded bytes,
- unicodeChar primitives are now UTF-16-encoded byte pairs for BMP characters (non-BMP is excluded)
- unicodeChar is deprecated
- a little bit of surrounding text is rephrased to match Unicode concepts
- there is some non-normative text explaining the implications of UTF-8 being a variable-width encoding
- references to UCS-2 are removed
- char primitives are now UTF-8-encoded bytes, - unicodeChar primitives are now UTF-16-encoded byte pairs for BMP characters (non-BMP is excluded) - unicodeChar is deprecated - a little bit of surrounding text is rephrased to match Unicode concepts - there is some non-normative text explaining the implications of UTF-8 being a variable-width encoding - references to UCS-2 are removed
|
This is the PR that I threatened on the Apps mailing list on 17 July, and follows on from the discussion on that thread and from @msdemlei's presentation in College Park. It tackles issues #55 and #69; covering both getting rid of references to UCS-2 and redefining the |
rra
left a comment
rra
left a comment
There was a problem hiding this comment.
In the world of Unicode RFCs, the standards are fairly careful to always use the term "octet" for the individual 8-bit storage unit of an encoding. This text uses "byte" throughout. I think that's a reasonable choice these days, given that all the computing architectures that used non-8-bit bytes are very obsolete, but it might be worth aligning the terminology on octet just to avoid any confusion for readers who are moving back and forth between the RFC world and the IVOA standard world and might wonder if there's some difference between byte and octet.
I'm not sure where to put this in the document, but it feels worthwhile to add a fairly explicit warning for char that if a value is truncated to fit a length restriction on the column, it may be shorter than the number of octets given in arraysize, and therefore implementations cannot use length == arraysize as a flag to detect possibly truncated values.
| For this type the primitive size of two bytes corresponds to a 2-byte | ||
| UTF-16 {\em code unit}. | ||
| Only characters in the Unicode Basic Multilingual Plane, | ||
| which all have 2-byte representations, are permitted for this datatype, | ||
| so that the primitive count matches the character count. |
There was a problem hiding this comment.
I know that you were trying to drop all the UCS-2 references, but I think you've essentially defined UCS-2 here without using that term explicitly. Maybe it would be a bit easier for implementers to understand if the text says that this is UCS-2? As far as I understand it, UCS-2 is exactly UTF-16 with all planes other than the Unicode Basic Multilingual Plane banned so that every character is exactly two octets.
Should there be a recommendation here about what implementations should do if given unicodeChar that is actually in UTF-16 and therefore contains surrogate pairs? I know that we would like to leave unicodeChar behind us, but we discovered that current PyVO generates unicodeChar fields when given a CSV table to upload, so we may be living with it for a while and I bet implementations will encounter people uploading higher plane characters in the wild.
There was a problem hiding this comment.
I'm happy to revert the language to UCS-2 (which is indeed identical to BMP-only UTF-16) if people think that's more comprehensible.
It may be that there are unicodeChar fields containing higher-plane characters out there somewhere, but that would have been illegal for earlier versions of VOTable and would be illegal with this version too. Given that, I feel like software is within its rights to do whatever it likes... but in practice it's likely to be UTF-16 so using a UTF-16 decoder would probably do the right thing even in the presence of illegal data, so maybe it's worth recommending that.
There was a problem hiding this comment.
I'm happy to revert the language to UCS-2 (which is indeed identical to BMP-only UTF-16) if people think that's more comprehensible.
I didn't express that well -- I like the change and I think it's great to connect this to UTF-16 because UTF-16 encoders are readily available but UCS-2 encoders may be a bit rarer. I was just thinking that it might be good to also note that UTF-16 with this restriction is just UCS-2.
It may be that there are
unicodeCharfields containing higher-plane characters out there somewhere, but that would have been illegal for earlier versions of VOTable and would be illegal with this version too. Given that, I feel like software is within its rights to do whatever it likes... but in practice it's likely to be UTF-16 so using a UTF-16 decoder would probably do the right thing even in the presence of illegal data, so maybe it's worth recommending that.
I like the idea of saying explicitly that you can use a UTF-16 decoder and accept technically invalid VOTables that contain surrogate pairs if you want to be generous in what you accept and can handle UTF-16, but when creating a VOTable, you must not include surrogate pairs.
VOTable.tex
Outdated
| the 2-byte big-endian UTF-16 encoding | ||
| of a Unicode character from the Basic Multilingual Plane. |
There was a problem hiding this comment.
Here too, I think this is just another way of saying UCS-2.
|
I've been trying to read a variety of sources on UCS-2, all non-authoritative, and have not been able to get a completely clear answer to the question of whether there are any valid UCS-2 code points that would be interpreted differently in UTF-16. This is roughly, but not precisely, equivalent to asking whether |
| Note that the primitive size of one byte refers to a single | ||
| UTF-8-encoded byte, not to a single character. | ||
| Since UTF-8 is a variable-width encoding, | ||
| a character may require multiple bytes, and for arrays the | ||
| string length (length in characters) and primitive count (length in bytes) | ||
| will in general differ. | ||
| 7-bit ASCII characters are however all encoded as a single byte in UTF-8, | ||
| so in the case of ASCII characters, which were required for this | ||
| datatype in earlier VOTable versions, the primitive and character count | ||
| are equal. |
There was a problem hiding this comment.
Perhaps I'm overlooking it, and it's already there, but I think it might be worth an explicit statement in the text that clarifies that a bare char without a length (a one-octet string) is limited to being able to store an ASCII character.
It's hard to find a formal definition of UCS-2 now because the UCS stuff is basically deprecated, but my understanding is that it is a direct mapping of the Unicode code points to a two-octet number per code point (and comes in either a big-endian or a little-endian variant). UTF-16 is defined to be exactly that mapping (see https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G31699) except for surrogate pairs, and the code space for surrogate codes used in surrogate pairs is reserved for that purpose exclusively in https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/#G24089. |
Following comments from Russ Allberry and Gregory D-F, make a couple of adjustments to the description of the (now deprecated) unicodeChar type: - note that the required encoding is just a rebadged UCS-2 - explicitly allow readers to treat it as standard UTF-16
I have to confess to working from wikipedia rather than the RFCs for my Unicode information; the wikipedia pages on e.g. UTF-8 and UTF-16 use the term "byte" throughout with no mention of octets. I feel like if it's good enough for wikipedia it's probably good enough here, and given that the rest of the VOTable document uses the term byte throughout as well, I think confusion will be minimised by leaving the terminology as is. But if majority opinion is against me I'll change it. |
I've tried to address this in c9859ed. |
|
With apologies for being late to the party, I have a few thoughts on this. I do think that this specification is clear and unambiguous. Usually that would be enough for me, but... Clearly for deserializing BINARY and BINARY2 we need to know the number of bytes per value. This is also probably true for 2D My main concern is for providers, for whom one of the most common VOTable errors we see is A safe For consumers, the practical issues are smaller and more about possible confusion.
|
|
On Mon, Oct 27, 2025 at 05:36:42PM -0700, Tom wrote:
Clearly for deserializing BINARY and BINARY2 we need to know the
number of bytes per value. This is also probably true for 2D
`char` arrays in the TABLEDATA serialization, but there it is less
clear to me since xml parsing and string storage techniques vary
Uh. Right. nD char arrays in Tabledata could be a killer.
The only way we can define that is to say "You need to UTF-8 encode
the payload before splitting it". Yikes.
|
That's right. I've included the following text in Section 2.2:
Do you think this needs spelling out more explicitly? I've presented it here from a read rather than write point of view, it should maybe be written "To encode or decode a multi-dimensional char array...". Is more rewording required? |
|
@tomdonaldson, thanks for your thoughts.
If it was just TABLEDATA then you could (and probably should) count number of characters rather than number of bytes when looking at 2-d char arrays. But since for BINARY/2 it really has to be by (UTF-8) byte, I'm proposing it works the same way for TABLEDATA. Otherwise, the meaning of
Fortunately, this isn't true: if you write code that's correct for VOTable 1.6 (the system proposed by this PR) it will work correctly for all legal instances of earlier VOTable versions. The reason is that in VOTable versions to date it is only allowed to have 7-bit ASCII characters in
That is true; if you're in a language where there is no facility for converting between a character string and a UTF-8 byte array then implementation is going to be tricky. Is this likely to be the case for any language we're interested in? The other points you make generally relate to difficulties in working out how many characters are required for a given arraysize declaration or vice versa. It's true that's hard and/or inefficient in the proposed scheme. My feeling is that at least for scalar strings (1-d char arrays) the best thing in most cases for providers will be now to use variable-length arrays rather than a fixed arraysize which is, as you say, of questionable utility. When parsing these things I'd consider just reading them in as variable-length strings rather than fixed-length ones even in the case of fixed arraysize, though there may be efficiency considerations for that depending on the platform. Admittedly you can't dodge a fixed arraysize dimension when working with 2-d char arrays, but that is a fairly unusual data type (and might possibly be addressed one day by xtype="strings"). One exception to the always-use-variable-arraysize rule of thumb could be (when authoring a VOTable) to use a fixed arraysize for strings known to be ASCII-like, for instance ISO-8601 dates. I should note that @fxpineau made a suggestion relevant to this, to do with using the |
VOTable.tex
Outdated
| and a character string may therefore be terminated by an ASCII | ||
| NULL [0x00] | ||
| indicates a Unicode string composed of UTF-8 encoded text. | ||
| A string may be terminated by a NULL code point |
There was a problem hiding this comment.
With the previous wording, it was explicit that the NULL termination applied to BINARY/BINARY2 (not TABLEDATA). Is it worth clarifying that here?
VOTable.tex
Outdated
| by this standard, but readers MAY treat such characters normally | ||
| if encountered, for instance by using a UTF-16 decoder on BINARY data, | ||
| though note in this case the arraysize may no longer match the character count. | ||
| Note this datatype is {\bf deprecated} from VOTable 1.6. |
There was a problem hiding this comment.
For emphasis, would it be useful to move this deprecation note to the beginning of the section.
tomdonaldson
left a comment
tomdonaldson
left a comment
There was a problem hiding this comment.
Thanks @mbtaylor for the responses and for this PR in general. I am now comfortable that this approach is a very reasonable way to handle the subtle complexities of UTF-8 values. In any case, I can't think of another approach that would be cleaner to readers or writers. I'm particularly compelled by the point that for 1D char arrays, writers are not required or even encouraged to use a fixed arraysize, and that readers other than validators are similarly not required care use the arraysize.
I'm comfortable with what's written as is, but left a couple minor suggestions for your consideration.
Another question occurred to me but also seems minor. That is, PARAM values use the same datatypes as FIELDs, but we don't explicitly mention that their value is essentially a quoted string matching a TABLEDATA serialization. In the unlikely event that one used multidimensional strings in a PARAM, is that worth clarifying in the value desccription?
There are a couple of adjustments to the text clarifying the intention of the Unicode-related changes.
|
Thanks @tomdonaldson I've made those adjustments as suggested. Incidentally a multi-dimensional char array in a |
|
In discussion of this proposal during the Apps session in Görlitz (see my presentation) @stvoutsin suggested the possibility of restricting the content of On reflection, I don't support this idea for a few reasons:
If I've misunderstood/misrepresented the proposal, or if Stelios or others want to argue for it, please do so in the comments here. If nobody does that in a week or so, I will shelve this suggestion, and move to merge the PR as is. |
I'm fine with the current approach and I do see how trying to limit fixed-size to ASCII would get messy. We're also now basically saying that you need to scan the data once to get the character count for schema allocation, numpy arrays, etc. With astropy in mind in particular, this will probably have a performance impact since they'd have to either scan the data once or use object arrays, which would lose some of the advantages of fixed-length arrays (memory overhead, loss of vectorization) That seems like we're sacrificing practicality for consistency in the spec. The alternative of letting arraysize mean characters in TABLEDATA and bytes in BINARY/BINARY2 would avoid the truncation complexity as well as the need for tools to either scan all rows or guess conservatively about max length. While this would potentially add ambiguity, I don't think having serialization specific rules is the complete mess that others seem to think, or at least I see it as the lesser evil. That said I get the consistency argument. And you're right that pushing people toward variable-length strings probably helps anyway. Those are just some thoughts I wanted to raise, but happy with this if there's consensus. |
|
@stvoutsin wrote:
I am thinking about implementation (TAP mainly) and there arraysize is in the Also, changing meaning of arraysize has a direct impact on the TAP specification since TAP uses the VOTable type system. I envision doing the following:
Does that sound sane and sufficient for correctness?Aside: in OpenCADC TAP services the VOSI-tables output use VOTableType exclusively for output, eg: Our code also supports the type system from VODataService and I don't know the current state of that... but technically TAP may still depend on the VODataService type system which, iirc, also includes an equivalent to arraysize. |
|
Something related that came to mind... One thing we'll need to make clear to users is in the table creation in WD-TAP-1.2 where a user can use a VOTable to create a database table with (in principle) fields using datatype="char":
So in principle users creating tables are impacted by the change in meaning of arraysize in a subtle way because they have to think "max UTF8 byte length" rather than "max string length". I don't think I can hide this. |
Wouldn't it have to be 4N to handle the worst-case scenario (emojis, etc.)?
I think this has to be A, because if the string is ASCII, it may contain up to A full characters. |
|
I was thinking UTF8 is 1-2 bytes per character (because I knew UTF16 exists), but you are right: 1-4 bytes/char. yeah, arraysize="A*" can have A bytes of ascii which is A characters in the database column 🤦 if you think about input. But a varchar(A) database column would let you put up to 4A bytes in there (silently) and you might truncate output to A bytes on output. Ugh. So I think now I'd have to enforce the arraysize (byte) limit in code to avoid storing input that could not be faithfully output later... it's a complication (in principle) because we allow input in a variety of formats (not just VOTable, where we'd interpret/enforce arraysize and detect arraysize mismatch vs table), but maybe those other forms are all pure ascii so not an actual problem. Anyway, it seems like creating a database table from VOTable will be more subtle and complicated to implement correctly. |
|
So far, I have not received any answer, so I am trying again: If nobody has an answer, we may state in the VOTable document that, if present, A reader will then understand that:
means:
|
|
On Tue, Dec 02, 2025 at 09:08:31AM -0800, François-Xavier Pineau wrote:
fxpineau left a comment (ivoa-std/VOTable#71)
If nobody has an answer, we may state in the VOTable document that,
if present, `width` is to be interpreted as the maximum number of
characters contained in the fixed length character column, and that
it is _recommended_ to include this attribute when possible.
We could do that; but is this actually something people want? Or
should want? Put another way: Is there a strong use case for
fixed-display-length strings in the 2020ies? [fixed-storage-length
is a different matter; perhaps not for VOTables, but being able to
compute row offsets in memory-mapped FITS binary tables admittedly is
cool].
A reader will then understand that:
> datatype="char" arraysize="N" width="M"
means:
* ASCII string of M characters (and bytes) if M = N → corresponds to CHAR(M) in PostgreSQL and SQL Server
* UTF-8 string of up to M characters if M < N → corresponds to VARCHAR(M) in PostgreSQL and CHAR(N) in SQL Server
At least Postgres is rather clear in the docs and has been that way
for a long time:
There is no performance difference among these three types [char,
varchar, text], apart from increased storage space when using the
blank-padded type, and a few extra CPU cycles to check the length
when storing into a length-constrained column. While character(n)
has performance advantages in some other database systems, there is
no such advantage in PostgreSQL; in fact character(n) is usually
the slowest of the three because of its additional storage costs.
In most situations text or character varying should be used
instead.
That said, I'd not veto employing width for this purpose. I'm not
sure I'd put it into DaCHS, though.
|
My point here is not about display; it is about having a way to tag ASCII strings (i.e., pre-v1.6 VOTable Strings).
Yes: "character(n) has performance advantages in some other database systems".
Yes, this is why I prefer |
|
On Thu, Dec 04, 2025 at 02:28:16AM -0800, François-Xavier Pineau wrote:
fxpineau left a comment (ivoa-std/VOTable#71)
> Is there a strong use case for fixed-display-length strings in the 2020ies?
My point here is not about display; it is about having a way to tag
ASCII strings (i.e., pre-v1.6 VOTable Strings).
Ah, ok. Yeah, that's a convincing use case.
* How: Two possible solutions are: using `xtype="ascii"`; using `arraysize == width`.
I think I finally prefer the `xtype` solution, which also makes
it possible to tag variable-length strings and arrays of strings.
Hm. I'd hope there are more options, because I don't particularly
like either of them. xtype="ascii" I think is clumsy because all
string-valued xtypes we have defined so far als qualify: The all are
expressed in 7-bit ASCII. And no, having multiple xtypes is even
worse.
Against that, making arraysize==width a somewhat cryptic marker is
clearly preferable to me, and I'd certainly implement that in some
way if we really don't find anything less cryptic.
So... do we? A thought that comes to mind is VALUES. This has
always been painful with char[], as I bet if you look, you will find
that OPTIONS for char[] VALUES (where they exist at all) enumerate
full strings (i.e. arrays), where we otherwise have agreed in 1.5
that VALUES is element-wise for arrays unless there's an xtype.
*If* we accept our element-wise VALUES policy for char[] as well,
then "have an ASCII character in VALUES/MAX" would work nicely as a
relatively clear signal for "it's ASCII".
But then, admittedly, this shows another slight uglyness, because if
there is non-ASCII in a char array, the MAX will be some UTF-8 escape
unless we do some crazy definition (and I'd argue we shouldn't). For
instance, if a column contained the röckdötty German word "Äußerung",
VALUES/MAX would arguably be 'Ã' (that's LATIN CAPITAL LETTER A WITH
TILDE in case you're squinting), the character corresponding to the
code point u+00c3, a thing that cannot even be represented in VOTable
char (only in char[2]).
So, I could even imagine saying: VALUES is undefined for char[] that
have non-ASCII; I actually think we should do that regardless of how
we do the ASCII signal. And then "do VALUES/MAX" to signal pure
ASCII isn't too far-fetched...
|
|
@fxpineau's suggestion about co-opting the But on second thoughts (thanks for your persistence FX) I'm coming to like it more. I think it's at least harmless and if documented sufficiently prominently it would be able to address some of the concerns raised here. It not only can serve as a flag for ASCII content, but can also be used to signal length in characters (code points) to consumers that want that information for resource allocation as raised by @stvoutsin:
The |
8 participants
