Various use-cases require the possibility to reference a part of the message body or a specific position in it. This was realized by providing offsets from the beginning of the message (when referencing a region, those offsets would define begin and end of a region). XEPs doing so include &xep0301;, &xep0372; (and thereof derived &xep0385;) and &xep0394;.

For these use-cases, it is highly relevant to decide how to count "characters" in a message body. While it at first sounds trivial, there are various ways of doing so in modern font systems. The purpose of this XEP is to define how characters shall be counted for the purpose of the aforementioned XEPs and any future XEP relying on a similar feature.

When counting characters in a body, they shall be counted by their number of Unicode code points. Message bodies must be used as strings of the XML characters (as defined in §2.2 of &w3xml;). This means that, i.e. no Unicode normalization may be performed before determining offsets when receiving or after determining offsets when sending. Any kind of further body processing shall be performed after counting (e.g. /me·The middle dot is used to represent a space character and is not meant to be taken verbatim. as described in &xep0245; is always counted as 4 characters without considering the sending user's name). All references (as defined in §4.1 of &w3xml;) must be counted by their referenced character(s) and not the reference characters (e.g. the encoded & is counted as one decoded character &).

The most obvious way of counting characters is to count them how humans would. This sounds easy when only having western scripts in mind but becomes more complicated in other scripts and most importantly is not well-defined across Unicode versions. New unicode versions regularly added new possibilities to build grapheme clusters, including from existing code points. To be forward compatible, counting grapheme clusters, graphemes, glyphs or similar is thus not an option. This leaves basically the two options of using the number of code units of the encoded string or the number of code points.

The main advantage of using the code units would be that those are native to many programming languages, easing the task for developers. However programming languages do not share a common encoding for their string type (C/C++ use UTF-8, C#/Java use UTF-16, Python 3 hides the internal encoding from the developer and only presents it in code points), so there is no best pick here. If one was to choose an encoding, the best choice would be UTF-8, the native encoding of XMPP. However this makes counting bytes a more complex task for programming languages that use a different encoding like UTF-16, as strings would need to be transcoded first.

Counting code points has the advantage that offset counts cannot point inside a code point. This could happen when using code units of any encoding that may use more than one unit to represent a code point (such as UTF-8 and UTF-16). If an offset count points inside a code point, that would be an invalid offset, raising more uncertainty of the correct behavior in such cases. Most notably the opportunity of splitting (as it exists for grapheme cluster) is not an option in that case, because splitting a code point would not create any usable output. Counting code points is widely supported in programming languages and can easily be implemented for encoded strings when not. The &w3xml; standard also defines a character as a unicode code point, thus counting code points is equivalent to counting XML characters.

Unicode terminology used across this document, can be looked up in the Unicode glossary at https://www.unicode.org/glossary/.

This document requires no interaction with &IANA;.

This document requires no interaction with ®ISTRAR;.

The author would like to thank Guus der Kinderen, Ralph Meijer, Jonas Schäfer, Lance Stout and others that provided feedback.