xeps/xep-0385.xml

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE xep SYSTEM 'xep.dtd' [
  <!ENTITY % ents SYSTEM 'xep.ent'>
%ents;
]>
<?xml-stylesheet type='text/xsl' href='xep.xsl'?>
<xep>
<!--
TODO:
  - Example showing component cache usage
  - encrypted files
  - 3rdParty hosted reference (GDrive, Dropbox, etc.)
-->
<header>
  <title>Stateless Inline Media Sharing (SIMS)</title>
  <abstract>This specification describes a protocol for stateless asynchronous media sharing with integrity and transport flexibility. It allows clients to provide a good interoperable user experience in combination with Carbons and MAM.</abstract>
  &LEGALNOTICE;
  <number>0385</number>
  <status>Experimental</status>
  <type>Standards Track</type>
  <sig>Standards</sig>
  <approver>Council</approver>
  <dependencies>
    <spec>XMPP Core</spec>
    <spec>XEP-0001</spec>
    <spec>XEP-0234</spec>
    <spec>XEP-0300</spec>
    <spec>XEP-0363</spec>
  </dependencies>
  <supersedes/>
  <supersededby/>
  <shortname>sims</shortname>
  &tobias;
  <revision>
    <version>0.2.1</version>
    <date>2018-01-25</date>
    <initials>vv</initials>
    <remark><p>Fix reference to XEP-0234.</p></remark>
  </revision>
  <revision>
    <version>0.2.0</version>
    <date>2017-09-21</date>
    <initials>fs</initials>
    <remark><p>Use 'urn:xmpp:hashes:2' and 'urn:xmpp:jingle:apps:file-transfer:5'.</p></remark>
  </revision>
  <revision>
    <version>0.1.0</version>
    <date>2017-01-04</date>
    <initials>XEP Editor: ssw</initials>
    <remark><p>Initial version approved by the council.</p></remark>
  </revision>
  <revision>
    <version>0.0.1</version>
    <date>2016-12-21</date>
    <initials>tm (XEP Editor: ssw)</initials>
    <remark><p>First draft processed by editor.</p></remark>
  </revision>
</header>
<section1 topic='Introduction' anchor='intro'>
  <p>File sharing in XMPP has mainly been addressed by synchronous solutions like &xep0096; and &xep0234;.
    However, these extensions only address the transfer of files and there is more to file sharing than the simple transfer of the data.</p>
  <p>Extentions that go beyond the simple transfer of data are &xep0329; and &xep0363;. XEP-0329 allows sharing folder structures to other users, allowing them to browse the remote folder and fetch interesting files using existing file-transfer protocols. XEP-0363 describes a protocol to ask a server component for a HTTP storage URL where a client can use HTTP PUT to save a file to and afterwards share the public URL with other users to share the file. While this provides some form of asynchronus file sharing it does not provide integrity protection and requires a server component.</p>
  <p>This proposal aims to provide a protocol that will enable XMPP clients to implement a great user experience (UX) around the process of sharing media in conversations. Shared media can take any form of static media like photos, videos, documents, compresses archives, etc.
  This is directly refelected in the requirents of this extension lined out in the following sections.</p>
</section1>
<section1 topic='Related XEPs' anchor='related'>
  <p>The state of sharing media with chat partners in the XMPP community is a protocol zoo in 2016. There are three major protocols for sharing media in XMPP.</p>
  <section2 topic='Bits of Binary'>
    <p>&xep0231; is designed for small media, i.e. less than 8 KB in size, that is hosted server-side and transferred Base64 encoded in-band of an existing XMPP stream. Example use-cases are custom emoticons that are referenced in &xep0071; img-tags, or thumbnails for &xep0234;.</p>
  </section2>
  <section2 topic='Jingle File Transfer'>
    <p>&xep0234; describes a peer-to-peer protocol for synchronous file-transfer between two XMPP entities. It attempts a direct transmission, followed by a proxied transmission, via &xep0260;. If neither works it will fallback to &xep0261; which will transfer the data inband of the exsiting XMPP stream.</p>
  </section2>
  <section2 topic='HTTP File Update'>
    <p>&xep0363; was designed as a simpler to implement alternative to &xep0231;. This is achieved by reusing the HTTP APIs in todays mobile and language SDKs. It requires a server component where clients can request HTTP URLs to upload data to and share the corresponding download URL as part of plain text in a conversation.</p>
  </section2>
  <section2 topic='Comparison of File Transfer Protocols'>
    <table caption='Comparison of File Transfer Protocols in 2016'>
      <tr>
        <th>Protocol</th>
        <th>File Size Limit</th>
        <th>Integrity Verification</th>
        <th>Transport</th>
        <th>Multi Receiver Support</th>
        <th>Server Support</th>
        <th>Resumption</th>
      </tr>
      <tr>
        <td>&xep0231;</td>
        <td>8 KB</td>
        <td>Yes</td>
        <td>Inband</td>
        <td>Yes</td>
        <td>Required</td>
        <td>No</td>
      </tr>
      <tr>
        <td>&xep0234;</td>
        <td>No</td>
        <td>Yes</td>
        <td>Inband/Direct/Proxy</td>
        <td>No</td>
        <td>Optional</td>
        <td>Yes</td>
      </tr>
      <tr>
        <td>&xep0363;</td>
        <td>Service Dependent</td>
        <td>No</td>
        <td>Outband (HTTP)</td>
        <td>Yes</td>
        <td>Required</td>
        <td>Download only</td>
      </tr>
    </table>
  </section2>
</section1>
<section1 topic='Requirements' anchor='reqs'>
  <ul>
    <li>Not require server components to work to ease deployment</li>
    <li>Can be improved by server components for taking load of clients</li>
    <li>Media sharing should work and enable a good UX in multi-user chats like &xep0045; and &xep0369;</li>
    <li>Media sharing should work great together with conversation synchronization protocols like &xep0280; and &xep0313;</li>
    <li>Reuse exiting protocols for the actual transport of the data, i.e. &xep0096;, &xep0234; or &xep0363;</li>
    <li>Guarantee file integrity</li>
    <li>Enable aggresive caching</li>
    <li>Provide users with metadata, e.g. file size, file type or thumbnail, to help them decide whether or not they want to load the media file</li>
    <li>Support referring to third party hosting services</li>
  </ul>
</section1>
<section1 topic='Use Cases' anchor='usecases'>
  <section2 topic='Sharing a photo' anchor='usecases-sending-photo'>
  <p>To share a photo, or any kind of media, a user sends a message stanza to the contact. If the message has an empty body, it is recommended
  to add a message processing hint, see &xep0334;, to indicate the message to be stored in message stores like &xep0313;.</p>
  <p>Clients supporting &xep0363; can upload the media file to a HTTP service and add the corresponding HTTP URL to the sources.</p>
  <example caption='Sharing an image with a contact'><![CDATA[<message to='julient@shakespeare.lit' from='romeo@montague.lit'>
  <body>Look at the nice view from the summit.</body>
  <reference xmlns='urn:xmpp:reference:0' begin='17' end='20' type='data'>
    <media-sharing xmlns='urn:xmpp:sims:1'>
      <file xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
        <media-type>image/jpeg</media>
        <name>summit.jpg</name>
        <size>3032449</size>
        <hash xmlns='urn:xmpp:hashes:2' algo='sha3-256'>2XarmwTlNxDAMkvymloX3S5+VbylNrJt/l5QyPa+YoU=</hash>
        <hash xmlns='urn:xmpp:hashes:2' algo='id-blake2b256'>2AfMGH8O7UNPTvUVAM9aK13mpCY=</hash>
        <desc>Photo from the summit.</desc>
        <thumbnail xmlns='urn:xmpp:thumbs:1'uri='cid:sha1+ffd7c8d28e9c5e82afea41f97108c6b4@bob.xmpp.org' media-type='image/png' width='128' height='96'/>
      </file>
      <sources>
        <reference xmlns='urn:xmpp:reference:0' type='data' uri='https://download.montague.lit/4a771ac1-f0b2-4a4a-9700-f2a26fa2bb67/summit.jpg' />
        <reference xmlns='urn:xmpp:reference:0' type='data' uri='xmpp:romeo@montague.lit/resource?jingle;id=9559976B-3FBF-4E7E-B457-2DAA225972BB' />
      </sources>
    </media-sharing>
  </reference>
</message>]]></example>
  <p>The file element is the same as from &xep0234;. It MUST specify media-type, size, description, and one or multiple hash elements as described in &xep0300;.
  The hash elements are essential as they provide end-to-end file integrity and allow efficient caching and flexible retrieval methods.</p>
  </section2>
  <section2 topic='Receiving a shared photo'>
    <p>On receive of a reference to a &lt;media-sharing&gt; element inside a message, a client SHOULD lookup in a local storage, whether the media with any of the proivded hashes has already
      been retrieved and is available. In that case no transfer needs to be initated and the image can be displayed in-line of the chat.</p>
    <p>If the media file is not available locally, the media file can be obtained by one of the references in the &lt;sources&gt; element. If a client support HTTP downloads, it can simply download HTTP references.</p>
    <p>If not, it can fetch the media file via a &xep0358; URI reference in the sources and initiate a Jingle File-Transfer. If the client does not support &xep0358;, it can attempt fetching the media file via &xep0234; by using the hash elements in the file element as described in <link url='http://xmpp.org/extensions/xep-0234.html#requesting'>Jingle File-Transfer</link>.</p>
    <p>A client MAY retrieve the file from other sources than these mentioned in the sources element. This may be via &xep0234; from the senders' other resources or from a
      media caching service located at the local service. The standardization of such cache is out of scope for this document.</p>
    <p>Regardless of the transport method used to obtain the file, the received content MUST be verified against one of the hashes.
      If the verification fails, the retrieved contant MUST be discarded and retrieval using a different source can be attempted.</p>
  </section2>
</section1>
<section1 topic='Business Rules' anchor='rules'>
  <section2 topic='Transport Method Preference'>
    <p>This XEP delegates actual transport of the media data to one of the existing file-transfer XEPs. Thus a client supporting this XEP MUST implement &xep0234; and &xep0363;.</p>
    <p>If a users server supports &xep0363;, it SHOULD upload the file to the service and add the retrieval URL to the &lt;sources&gt; tag, unless the user specifically asked to not store media in the cloud.</p>
    <p>Using &xep0363; for media file transfer highly increases the UX, since the HTTP server has a higher availability than XMPP end-user clients and can easily handle the load of lots of requests that result from sharing media in &xep0045; and &xep0369; rooms.</p>
  </section2>
  <section2 topic='Media Support'>
    <p>Sharing the raw data of media does not provide a complete user experience. Clients ideally need to be able to display the media inline of the chat. For this we set baseline requirements for audio, video and picture formats, that a client supports to display. These requrirements are shown in the following table.</p>
    <p>A client usually will always send in one format per media type, if it creates that media itself.</p>
    <table caption='Mandated encoding and in-line display support.'>
      <tr>
        <th>Media Type</th>
        <th>Mime Type</th>
        <th>Format/Container</th>
        <th>Codec</th>
        <th>Requirement</th>
        <th>Comment</th>
      </tr>
      <tr>
        <td>Audio</td>
        <td>audio/m4a</td>
        <td>MPEG4</td>
        <td>AAC</td>
        <td>SHOULD</td>
        <td>Can be encoded/decoded by stock <link url='https://developer.android.com/guide/appendix/media-formats.html'>Android</link> and <link url='https://developer.apple.com/library/content/documentation/Miscellaneous/Conceptual/iPhoneOSTechOverview/MediaLayer/MediaLayer.html'>iOS</link> systems.</td>
      </tr>
      <tr>
        <td>Image</td>
        <td>image/jpeg</td>
        <td>-</td>
        <td>JPEG</td>
        <td>SHOULD</td>
        <td>Supported on common desktop and mobile systems. Use for photos.</td>
      </tr>
      <tr>
        <td>Image</td>
        <td>image/png</td>
        <td>-</td>
        <td>PNG</td>
        <td>SHOULD</td>
        <td>Supported on common desktop and mobile systems. Use for non-photos.</td>
      </tr>
      <tr>
        <td>Video</td>
        <td>image/gif</td>
        <td>-</td>
        <td>GIF</td>
        <td>SHOULD</td>
        <td>Widespread history animation format supported everywhere.</td>
      </tr>
      <tr>
        <td>Video</td>
        <td>video/mp4</td>
        <td>MPEG4</td>
        <td>H.264 AVC</td>
        <td>SHOULD</td>
        <td>Can be encoded/decoded by stock <link url='https://developer.android.com/guide/appendix/media-formats.html'>Android</link> and <link url='https://developer.apple.com/library/content/documentation/Miscellaneous/Conceptual/iPhoneOSTechOverview/MediaLayer/MediaLayer.html'>iOS</link> systems.</td>
      </tr>
    </table>
  </section2>
  <section2 topic='Atomatic retrieval of shared media'>
    <p>Depending on the size of the shared media, a client MAY want to automatically download and display the media instead of fetching and displaying the thumbnail. The size threshold depends on the network environment the client currently runs in.</p>
    <p>If a client supports automatic retrieval it MUST disclose this feature to the end user and provide a way to disable it, as it may result in high network traffic.</p>
  </section2>
  <section2 topic='MUC and MIX related rules'>
    <p>In cases where media is shared in a &xep0045; or &xep0369; room the sender has to expect that a large number of clients may retrieve the shared media automatically. Ideally multiple sources, including HTTP or other high availability sources, are provided in the &lt;sources&gt; tag of the &lt;media-sharing&gt; tag in case the media is shared in a MUC/MIX room.</p>
    <p class='box'>TODO: Describe protocol for MIX members to advertise media availablilty to peers in a dedicated MIX channel PubSub node. Maybe as a dedicated XEP.</p>
  </section2>
  <section2 topic='MAM and Carbons related rules'>
    <p>For the media sharing described in this XEP to work, it is REQUIRED for MAM to store the whole stanza instead of only the body content. If the MAM component of the user's server strips away the &lt;media-sharing&gt; tag, any shared media will be missing in archived messages.</p>
    <p>If sensitve media is shared a client MAY add relevant hints for the server via &xep0334;.</p>
  </section2>
  <section2 topic='XHTML-IM related rules'>
    <p>To refer to shared media in a XHTML-IM message, this XEP takes advantage of the requirement for hash elements in the file metadata and &rfc6920; and its ni URI format. Using the URI format, XHTML-IM can easily refer to media that is attached to a message via a &lt;media-sharing&gt; element, as shown in the following example.</p>
    <example caption='Sharing an image with a contact'><![CDATA[<message to='julient@shakespeare.lit' from='romeo@montague.lit'>
    <body>Look at the nice view from the summit.</body>
    <html xmlns='http://jabber.org/protocol/xhtml-im'>
      <body xmlns='http://www.w3.org/1999/xhtml'>
        <p>Look at the nice <p style='font-weight:bold; display:inline">view</p> from the summit.</p>
        <img src="ni:///sha3-256;wqfDv8OGw7jCvx7Dl2ZRw4FHVsKgYcOWYsO14oKsw79Nw6Q7ScO64oCaw5gKS-KCrMO4Q0o" />
      </body>
    </html>
    <reference xmlns='urn:xmpp:reference:0' begin='17' end='20' type='data'>
      <media-sharing xmlns='urn:xmpp:sims:1'>
        <file xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
          <media-type>image/jpeg</media>
          <name>summit.jpg</name>
          <size>3032449</size>
          <hash xmlns='urn:xmpp:hashes:2' algo='sha3-256'>2XarmwTlNxDAMkvymloX3S5+VbylNrJt/l5QyPa+YoU=</hash>
          <hash xmlns='urn:xmpp:hashes:2' algo='id-blake2b256'>2AfMGH8O7UNPTvUVAM9aK13mpCY=</hash>
          <desc>Photo from the summit.</desc>
          <thumbnail xmlns='urn:xmpp:thumbs:1'uri='cid:sha1+ffd7c8d28e9c5e82afea41f97108c6b4@bob.xmpp.org' media-type='image/png' width='128' height='96'/>
        </file>
        <sources>
          <reference xmlns='urn:xmpp:reference:0' type='data' uri='https://download.montague.lit/4a771ac1-f0b2-4a4a-9700-f2a26fa2bb67/summit.jpg' />
          <reference xmlns='urn:xmpp:reference:0' type='data' uri='xmpp:romeo@montague.lit/resource?jingle-ft' />
        </sources>
      </media-sharing>
    </reference>
  </message>]]></example>
    <p>This way the client can aquire the content addressable resource mentioned in the img-tag in the XHTML-IM message, and when finished show in in the rendered XHTML-IM message.</p>
  </section2>
</section1>
<section1 topic='Implementation Notes' anchor='impl'>
  <p></p>
</section1>
<section1 topic='Accessibility Considerations' anchor='access'>
  <p>The size element in the file element provides clients to automatically load small files and if not provide the users with a hint on how long a transfer might take.</p>
  <p>The OPTIONAL thumbnail element in the file element improves the user experience as it provides further hints for users on whether the file could be of interest to them.</p>
  <p>The desc element in the file element is criticial for clients to enable them to provide accessibility to users who use screen readers.</p>
</section1>
<section1 topic='Internationalization Considerations' anchor='i18n'>
  <p>OPTIONAL.</p>
</section1>
<section1 topic='Security Considerations' anchor='security'>
  <section2 topic='Clearing of privacy sensitive metadata'>
    <p>Mobile devices are able to attach the geographic location of where a photo was taken to the photo. It is RECOMMENDED that a client implementing this XEP attempts to detect privacy exposing metadata in media shared and if found provides the user with an option to clear the media of such metadata.</p>
  </section2>
  <section2 topic='The value and cost of end-to-end media integrity'>
    <p>Requiring end-to-end media integrity prevents trival server side optimizations or other processing on shared media as it will change the cryptographic hash of the media file. On the other hand, requring a matching cryptographic hash guarantees that everybody sees the exact same media a user has shared in a group conversation.</p>
  </section2>
</section1>
<section1 topic='Acknowledgements' anchor='ack'>
  <p>Thanks to Kim Alvefur, Emmanuel Gil Peyrot, Kevin Smith, Nicolas Vérité, and Florian Schmaus for their helpful comments.</p>
</section1>
<section1 topic='IANA Considerations' anchor='iana'>
  <p>REQUIRED.</p>
</section1>
<section1 topic='XMPP Registrar Considerations' anchor='registrar'>
  <p>The &REGISTRAR; includes the following information in its registries.</p>
  <section2 topic='Protocol Namespaces' anchor='registrar-protocol'>
    <p>The XMPP Registrar will include the following namespace in its registry of protocol namespaces at &NAMESPACES;:</p>
    <ul>
      <li>urn:xmpp:sims:1</li>
    </ul>
  </section2>
</section1>
<section1 topic='XML Schema' anchor='schema'>
  <p>REQUIRED for protocol specifications.</p>
</section1>
</xep>