accepted for publication

xep-0322.xml

@@ -1,4 +1,5 @@
 <?xml version='1.0' encoding='UTF-8'?>
+<!-- TODO: Add sequence diagrams. -->
 <!DOCTYPE xep SYSTEM 'xep.dtd' [
   <!ENTITY % ents SYSTEM 'xep.ent'>
 %ents;
@@ -6,7 +7,7 @@
 <?xml-stylesheet type='text/xsl' href='xep.xsl'?>
 <xep>
     <header>
-        <title>Using Efficient XML Interchange (EXI) Format in XMPP</title>
+        <title>Efficient XML Interchange (EXI) Format</title>
         <abstract>This specification describes how EXI compression can be used in XMPP networks.</abstract>
         <legal>
             <copyright>This XMPP Extension Protocol is copyright (c) 1999 - 2013 by the XMPP Standards Foundation (XSF).</copyright>
@@ -37,6 +38,32 @@
             <jid>peter.waher@jabber.org</jid>
             <uri>http://se.linkedin.com/pub/peter-waher/1a/71b/a29/</uri>
         </author>
+		<revision>
+			<version>0.0.4</version>
+			<date>2013-03-19</date>
+			<initials>pwa</initials>
+			<remark>
+				<p>Added support for uploading EXI-compressed schema files.</p>
+			</remark>
+		</revision>
+		<revision>
+			<version>0.0.3</version>
+			<date>2013-03-15</date>
+			<initials>pwa</initials>
+			<remark>
+				<p>Added definition: EXI body.</p>
+				<p>Added note regarding preserverance of namespace prefixes.</p>
+				<p>Corrected the language.</p>
+			</remark>
+		</revision>
+		<revision>
+			<version>0.0.2</version>
+			<date>2013-03-13</date>
+			<initials>pwa</initials>
+			<remark>
+				<p>Added support for session-wide buffers and string tables.</p>
+			</remark>
+		</revision>
         <revision>
             <version>0.0.1</version>
             <date>2013-03-12</date>
@@ -54,7 +81,7 @@
             communicate efficiently. The strong support in EXI for generating efficient stubcodes is also vital to build efficient code in constrained devices.
         </p>
         <p>
-            Activating EXI compression requires a handshake to take place prior, where the server and client agrees on a set of parameters. Some of these parameters may increase compression ratio,
+            Activating EXI compression requires a handshake to take place prior, where the server and client agree on a set of parameters. Some of these parameters may increase the compression ratio,
             at the cost of processing power and readability. These parameters include:
         </p>
         <ul>
@@ -71,10 +98,10 @@
             <li>Value partition capacity.</li>
         </ul>
         <p>
-            These parameters will be discussed deeper in the following sections. There are also default values users can use to commence evaluating EXI compression.
+            These parameters will be discussed in greater depth in the following sections. There are also default values that can be used to commence evaluating EXI compression.
         </p>
         <p>
-            The single most important property to agree on however, is the set of schemas to use during EXI compression. EXI compresses XML much more efficiently if there exist schemas 
+            The single most important property to agree on however, is the set of schemas to use during EXI compression. EXI compresses XML much more efficiently if schemas exist
             describing the format of the expected XML. Since the server is not supposed to know all possible XML schemas, a mechanism is provided in this document whereby schemas can be
             interchanged, so that the server can adapt its compression to the needs of the client.
         </p>
@@ -96,12 +123,12 @@
     </stream:features>]]>
             </example>
             <p>
-                Support for EXI compression is detected by the existance of the <strong>exi</strong> compression method in the <strong>features</strong> stanza.
+                Support for EXI compression is detected by the existence of the <strong>exi</strong> compression method in the <strong>features</strong> stanza.
             </p>
         </section2>
         <section2 topic='Invalid setup'>
             <p>
-                If the client attempts to activate EXI stream at this point, before negotiation of EXI properties have been performed, the server must respond with a
+                If the client attempts to activate an EXI stream at this point, before the negotiation of EXI properties has been performed, the server must respond with a
                 <strong>setup-failed</strong> response.
             </p>
             <example caption='Invalid setup'>
@@ -143,27 +170,27 @@
                 <strong>MD5 hash</strong>. The <strong>MD5 hash</strong> provides a way to detect small changes in the file, even if the byte size and namespace are the same.
             </p>
             <p>
-                Schema files that the server does not have, based on namespace, byte size and MD5 hash, are marked with the <strong>missingSchema</strong> element instead of the 
+                Schema files that the server does not have (based on namespace, byte size and MD5 hash) are marked with the <strong>missingSchema</strong> element instead of the 
                 normal <strong>schema</strong> element.
             </p>
             <p>
-                The client can, at this point, choose to abort the EXI enablement sequence, if it cannot accommodate itself with the proposed parameter settings provided by the server.
+				At this point the client can choose to abort the EXI enablement sequence, if it cannot accommodate itself with the proposed parameter settings provided by the server.
-                The XMPP session will continue to work in its current state.
+				The XMPP session will continue to work in its current state. Aborting does not require taking further action from the client.
 			</p>
         </section2>
         <section2 topic='Uploading new schema files'>
             <p>
-                If the server lacks information about a schema file, this is specified in the response through the <strong>missingSchema</strong> elements. The client can at this point
+                If the server lacks information about a schema file, it is specified in the response through the <strong>missingSchema</strong> elements. At this point, the client can
 				either choose to accept that these schema files are not available, making compression less efficient, or choose to upload the missing schema files to the server. Of course,
-                uploading schema files would require the device to have sufficient buffers and memory to store and upload the schema files in the first place. (If not possible to upload the
+				uploading schema files would require the device to have sufficient buffers and memory to store and upload the schema files in the first place. (If it is not possible to upload the
-                schema files, consideration should be taken to install the schema files manually at the server.)
+				schema files, consideration should be given to installing the schema files manually at the server.)
 			</p>
             <p>
                 To upload a schema file, the client simply sends the schema file using an <strong>uploadSchema</strong> element, as follows:
             </p>
             <example caption='Uploading schema file'>
                 <![CDATA[
-    <uploadSchema xmlns='http://jabber.org/protocol/compress/exi'>
+    <uploadSchema xmlns='http://jabber.org/protocol/compress/exi' contentType='Text'>
         PD94bWwgdmVyc2lvbj0nMS4wJyBlbmNvZGluZz0nVVRGLTgnPz4NCjx4czpzY2hlbWENCiAgICB4
         bWxuczp4cz0naHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEnDQogICAgdGFyZ2V0TmFt
         ZXNwYWNlPSd1cm46eG1wcDpzbjpwcm92aXNpb25pbmcnDQogICAgeG1sbnM9J3Vybjp4bXBwOnNu
@@ -175,11 +202,11 @@
     </uploadSchema>]]>
             </example>
             <p>
-                The schema itself is sent using base64 encoding to the server. This, to make sure a binary exact copy is transferred, maintaining encoding, processing instructions, etc. The
+                The schema itself is sent using base64 encoding to the server. This is to make sure a binary exact copy is transferred, maintaining encoding, processing instructions, etc. The
-                server then computes the <strong>target namespace</strong>, <strong>byte size</strong> and <strong>MD5 Hash</strong> from this schema file sent.
+                server then computes the <strong>target namespace</strong>, <strong>byte size</strong> and <strong>MD5 Hash</strong> from the sent schema file.
             </p>
             <p>
-                If the client desires, it can test the EXI setup again. This is however optional, but can be used to test that uploading the schema files, and any new property values
+                If the client desires, it can test the EXI setup again. This is optional, but can be used to test that uploading the schema files, and any new property values
                 are accepted by the server.
             </p>
             <example caption='Testing newly uploaded schema files'>
@@ -190,7 +217,7 @@
     </setup>]]>
             </example>
             <p>
-                And the server should now respond:
+                And the server should then respond:
             </p>
             <example caption='Agreement between client and server'>
                 <![CDATA[
@@ -205,6 +232,94 @@
                 The client in turn can check this attribute as a quick way to check if agreement exists.
             </p>
         </section2>
+		<section2 topic='Uploading compressed schema files'>
+			<p>
+				The <strong>uploadSchema</strong> command has an optional attribute called <strong>contentType</strong> that can be used to send different types of documents 
+				to the server. This is not a MIME content type, but an enumeration with the following options:
+			</p>
+			<table caption='contentType values'>
+				<tr>
+					<th>Value</th>
+					<th>Description</th>
+				</tr>
+				<tr>
+					<td>Text</td>
+					<td>The schema is sent as plain text. If no encoding is provided in the XML header of the schema file, UTF-8 encoding is assumed. This is the default value.</td>
+				</tr>
+				<tr>
+					<td>ExiBody</td>
+					<td>The schema file is sent as an EXI compressed file, but only the body is sent. *</td>
+				</tr>
+				<tr>
+					<td>ExiDocument</td>
+					<td>The schema file is sent as an EXI compressed file. The entire file, including Exi header is provided. *</td>
+				</tr>
+			</table>
+			<p>
+				(*) These options assume the following set of default EXI options are used. It is assumed the XMPP server has more capabilities than the client, so the following
+				set of options must be supported by the XMPP server. The schema files can be precompressed and stored as binary files on the client for easier transmission.
+			</p>
+			<table caption='Default EXI options'>
+				<tr>
+					<th>Option</th>
+					<th>Default value</th>
+				</tr>
+				<tr>
+					<td>Version</td>
+					<td>1</td>
+				</tr>
+				<tr>
+					<td>alignment</td>
+					<td>bit-packed</td>
+				</tr>
+				<tr>
+					<td>compression</td>
+					<td>false</td>
+				</tr>
+				<tr>
+					<td>strict</td>
+					<td>false</td>
+				</tr>
+				<tr>
+					<td>fragment</td>
+					<td>false</td>
+				</tr>
+				<tr>
+					<td>preserve</td>
+					<td>all false</td>
+				</tr>
+				<tr>
+					<td>selfContained</td>
+					<td>false</td>
+				</tr>
+				<tr>
+					<td>schemaId</td>
+					<td>No schema</td>
+				</tr>
+				<tr>
+					<td>datatypeRepresentationMap</td>
+					<td>No map</td>
+				</tr>
+				<tr>
+					<td>blockSize</td>
+					<td>1000000 (one million)</td>
+				</tr>
+				<tr>
+					<td>valueMaxLength</td>
+					<td>unbounded</td>
+				</tr>
+				<tr>
+					<td>valuePartitionCapacity</td>
+					<td>unbounded</td>
+				</tr>
+			</table>
+			<p>
+				Since EXI compression does not perserve the extact binary representation of the schema file (for instance it doesn't preserve white space), the server
+				cannot correctly compute byte size and an MD5 hash for the file. Therefore, the client needs to provide this information in the <strong>uploadSchema</strong>
+				command using the <strong>bytes</strong> and <strong>md5Hash</strong> attributes. They are mandatory in case EXI compressed schema files are uploaded to the
+				server. Also note that the byte length and MD5 Hash should be computed on the original XML Schema file, not the compressed or decompressed version.
+			</p>
+		</section2>
         <section2 topic='Downloading new schema files on server'>
             <p>
                 As an alternative to uploading a schema file to the server, the client can ask the server to download a schema file by itself. This is done using the <strong>downloadSchema</strong>
@@ -219,7 +334,7 @@
                 from the downloaded schema.
             </p>
             <p>
-                When the schema has been downloaded, a successful download response is returned, as follows:
+                When the schema has been downloaded, the following successful download response is returned:
             </p>
             <example caption='Schema successfully downloaded'>
                 <![CDATA[
@@ -235,7 +350,7 @@
     </downloadSchemaResponse>]]>
             </example>
             <p>
-                If the URL could not be resolved, a response as follows is returned:
+                If the URL could not be resolved, the following response is returned:
             </p>
             <example caption='Invalid URL'>
                 <![CDATA[
@@ -244,7 +359,7 @@
     </downloadSchemaResponse>]]>
             </example>
             <p>
-                If a timeout occurred during the download attempt, a response as follows is returned:
+                If a timeout occurred during the download attempt, the following response is returned:
             </p>
             <example caption='Timeout'>
                 <![CDATA[
@@ -253,7 +368,7 @@
     </downloadSchemaResponse>]]>
             </example>
             <p>
-                If the url points to something that is not a schema, a response as follows is returned:
+                If the url points to something that is not a schema, the following response is returned:
             </p>
             <example caption='Invalid Content Type'>
                 <![CDATA[
@@ -262,7 +377,7 @@
     </downloadSchemaResponse>]]>
             </example>
             <p>
-                If some other error occurs, unforseen by this specification, the server can simply respond with a generic error message, as follows:
+                If an error occurs that is unforeseen by this specification, the server can simply respond with a generic error message, as follows:
             </p>
             <example caption='Other types of errors'>
                 <![CDATA[
@@ -272,10 +387,10 @@
             </example>
             <p>
                 <strong>Note:</strong> Downloading a schema, might download a version which does not correspond to the desired version
-                of the schema. So, it's more important in this case, the client checks that the server actually has the version of the schema required by the client.
+                of the schema. It's more important in this case so the client checks that the server actually has the version of the schema required by the client.
             </p>
         </section2>
-        <section2 topic='Starting compression'>
+        <section2 topic='Start compression'>
             <p>
                 When EXI option negotiation has been completed, the client can tell the server that it is ready to start compression. It does this using the normal <strong>compress</strong>
                 stanza, as follows:
@@ -287,15 +402,16 @@
     </compress>]]>
             </example>
             <p>
-                The server now having necessary knowledge on how the EXI engine should be configured for the current session, responds:
+                The server now has the necessary knowledge on how the EXI engine should be configured for the current session and it responds as follows:
             </p>
-            <example>
+            <example caption='Compression accepted'>
                 <![CDATA[
     <compressed xmlns='http://jabber.org/protocol/compress'/>]]>
             </example>
             <p>
                 When the client receives acknowledgement that the compression method has been accepted, it restarts the stream, as explained in 
-                <link url='http://xmpp.org/extensions/xep-0138.html#usecase'>XEP 0138</link>.
+                <link url='http://xmpp.org/extensions/xep-0138.html#usecase'>XEP 0138</link>, except that it <strong>must not</strong> resend the <strong>&lt;stream&gt;</strong>
+				start element sequence. Similarly, the client must not send a <strong>&lt;/stream&gt;</strong> element when closing the session. Closing the connection is sufficient.
             </p>
         </section2>
     </section1>
@@ -340,6 +456,63 @@
             <p>
                  The <strong>valuePartitionCapacity option</strong> specifies the maximum number of value content items in the string table at any given time. The default value "unbounded" is assumed when the "valuePartitionCapacity" element is absent in the EXI Options document. Section <link url='http://www.w3.org/TR/exi/#encodingOptimizedForMisses'>7.3.3 Partitions Optimized for Frequent use of String Literals</link> specifies the behavior of the string table when this capacity is reached.
             </p>
+            <p>
+                The <strong>sessionWideBuffers</strong> controls buffer and string table life time. If set to true, all buffers, string tables, etc. will be maintained during the entire session. 
+                This may improve performance during time since strings can be omitted in the compressed binary stream, but it might also in some cases degrade performance since more options are 
+                available in the tables, requiring more bits to encode strings. The default value is false, meaning that buffers, string tables, etc., are cleared between each stanza. (This option
+                is EXI/XMPP specific.)
+            </p>
+        </section2>
+        <section2 topic='Transmission of EXI bodies and Session-wide Buffers'>
+			<p>
+				The transmission of EXI-compressed stanzas takes the form of a sequence of EXI bodies. In order for the recipient to be able to correctly interpret these incoming
+				EXI bodies, the sender is required to flush any pending bits at the end of the last End Document (ED) event for each stanza and then send any pending bytes available
+				in the output buffer. Since this makes sure each EXI body starts at an even byte boundary, it permits the recipient to decompress the body into an XML stanza.
+			</p>
+			<p>
+				Therefore, each stanza sent on the stream, must be compressed separately, reusing the same options as used by the stream.
+				(Options are not sent on the stream, only the generated EXI bodies).
+			</p>
+			<p>
+				Compression of the stanza must be done in document mode, not fragment mode, including the Start Document (SD) and End Document (ED) events.
+				If there are unwritten bits pending after the last End Document (ED) event (after the end of the stanza), Zero-bits are written until a byte boundary is created. 
+				The receptor must ignore bits in the last byte after the last End Document event has been received.
+			</p>
+            <p>
+                During setup of the EXI compression engine, the client can choose if buffers are to be reused between stanzas, or cleared between each stanza. This is done
+                using the EXI over XMPP specific option <strong>sessionWideBuffers</strong>, which is false by default, meaning buffers and string tables are cleared between
+                each stanza.
+            </p>
+            <p>
+                There may be cases where maintaining buffers and string tables throughout the session is preferrable. Since strings are already available in the buffers, 
+				they don't need to be output in the stream the first time they appear in a stanza. However, the number of strings in tables increase, and so does the number of bits required to
+                encode them. Depending on what type of communication is performed, this option might give better results one way or another. If the same type of message is always
+				sent, maintaining string buffers may be more efficient. But if the client sends many many different types of messages, clearing buffers may be more efficient.
+            </p>
+			<p>
+				Note that the stream of EXI bodies is indefinite. It only stops when the session is closed, i.e. when the socket connection is dropped. Therefore, the buffers can grow
+				indefinitely unless control is maintained on what types of messages are sent, their contents (specifically string values), and to whom they are sent (JIDs being strings).
+				All string tables and buffers must be cleared when a connection is lost.
+			</p>
+            <p>
+                Note also that if you want the option to enter a session in the middle of the flow to listen to the communication, you need to clear tables and buffers between each
+                stanza, or you will not be able to decode the binary stream appropriately.
+            </p>
+        </section2>
+		<section2 topic='Preserving prefixes'>
+			<p>
+				Normally, prefixes are not preserved during EXI compression and decompression. If the communicating parties (sending client, XMPP server(s) and receiving clients)
+				interpret incoming stanzas and content according to namespace, this should be sufficient. However, some implementations do not check namespaces, but prefix names used.
+				In such cases, all communicating parties are required to enable the preserve prefixes option during negotionating.
+			</p>
+			<p>
+				<strong>Note:</strong> It is not sufficient that one party enable this option. Both sender and received are required to enable this option, or prefix names will be
+				lost in the transmission.
+			</p>
+			<p>
+				Note also, that preserving prefix names result in less efficient compression. Therefore, all clients implementing EXI compression should strive to parse incoming
+				XML based on namespace, not prefix name.
+			</p>
 		</section2>
         <section2 topic='Networks containing clients having limited memory'>
             <p>
@@ -369,13 +542,13 @@
                 download one.
             </p>
             <p>
-                Uloading a schema has the advantage, that the client knows exactly the version that the server requires. It has the disadvantage, that the client needs to store the schema
+                Uploading a schema has the advantage, that the client knows exactly the version that the server requires. It has the disadvantage, that the client needs to store the schema
                 and send a possible large schema to the server. If EXI is used because the device has limited memory, uploading a schema might not be an option.
             </p>
             <p>
-                Downloading a schema has the advantage, that size of schema does not matter. The disadvantage is that asynchronous errors might occur, so the client needs to pay respect
+                Downloading a schema has the advantage, that size of schema does not matter. The disadvantage is that asynchronous errors might occur, so the client needs to pay attention
                 to the responses returned by the server when downloading schemas. Also, downloading a schema, might download a version which does not correspond to the desired version
-                of the schema. So, it's more important in this case, the client checks that the server actually has the version of the schema required by the client.
+                of the schema. So, it's more important in this case that the client checks that the server actually has the version of the schema required by the client.
             </p>
         </section2>
         <section2 topic='Server decompression and recompression vs. binary forwarding'>
@@ -545,6 +718,15 @@
                 capacity is reached.</xs:documentation>
             </xs:annotation>
         </xs:attribute>
+        <xs:attribute name='sessionWideBuffers' type='xs:boolean' use='optional' default='false'>
+            <xs:annotation>
+                <xs:documentation>If set to true, all buffers, string tables, etc. will be maintained during the 
+                entire session. This may improve performance during time since strings can be omitted in the 
+                compressed binary stream, but it might also in some cases degrade performance since more options 
+                are available in the tables, requiring more bits to encode strings. The default value is false, 
+                meaning that buffers, string tables, etc., are cleared between each stanza.</xs:documentation>
+            </xs:annotation>
+        </xs:attribute>
     </xs:attributeGroup>
  
     <xs:simpleType name='MD5Hash'>
@@ -583,7 +765,25 @@
         </xs:restriction>
     </xs:simpleType>
  
-    <xs:element name='uploadSchema' type='xs:base64Binary'/>
+	<xs:element name='uploadSchema'>
+		<xs:complexType>
+			<xs:simpleContent>
+				<xs:extension base='xs:base64Binary'>
+					<xs:attribute name='contentType' type='ContentType' use='optional' default='Text'/>
+					<xs:attribute name='bytes' type='xs:positiveInteger' use='optional'/>
+					<xs:attribute name='md5Hash' type='MD5Hash' use='optional'/>
+				</xs:extension>
+			</xs:simpleContent>
+		</xs:complexType>
+	</xs:element>
+ 
+	<xs:simpleType name='ContentType'>
+		<xs:restriction base='xs:string'>
+			<xs:enumeration value='Text'/>
+			<xs:enumeration value='ExiBody'/>
+			<xs:enumeration value='ExiDocument'/>
+		</xs:restriction>
+	</xs:simpleType>
  
     <xs:element name='downloadSchema' type='DownloadSchema'/>
     <xs:element name='downloadSchemaResponse' type='DownloadSchemaResponse'/>
@@ -633,6 +833,6 @@
         </code>
     </section1>
     <section1 topic='Acknowledgements' anchor='ack'>
-		<p>Thanks to Joachim Lindborg for all valuable feedback.</p>
+        <p>Thanks to Joachim Lindborg, Yusuke Doi, Takuki Kamiya, Tina Beckman, Karin Forsell, Jeff Freund and Rumen Kyusakov for all valuable feedback.</p>
     </section1>
 </xep>