spdylay

Spdylay - SPDY C Library
========================

This is an experimental implementation of Google's SPDY protocol in C.

This library provides SPDY version 2 and 3 framing layer
implementation.  It does not perform any I/O operations.  When the
library needs them, it calls the callback functions provided by the
application. It also does not include any event polling mechanism, so
the application can freely choose the way of handling events. This
library code does not depend on any particular SSL library (except for
example programs which depend on OpenSSL 1.0.1 or later).

This project also develops SPDY client, server and proxy on top of
Spdylay library. See `SPDY Client and Server Programs`_ section.

Development Status
------------------

Most of the SPDY/2 and SPDY/3 functionality has been implemented.  In
both versions, the direct support of server-push has not been
available yet.  The application can achieve server-push using
primitive APIs though.

As described below, we can create SPDY client and server with the
current Spdylay API.

Requirements
------------

The following packages are needed to build the library:

* pkg-config >= 0.20
* zlib >= 1.2.3

To build and run the unit test programs, the following packages are
needed:

* cunit >= 2.1

To build and run the example programs, the following packages are
needed:

* OpenSSL >= 1.0.1

To enable ``-a`` option (getting linked assets from the downloaded
resouce) in ``spdycat`` (one of the example program), the following
packages are needed:

* libxml2 >= 2.7.7

To build SPDY/HTTPS to HTTP reverse proxy ``shrpx`` (one of the
example program), the following packages are needed:

* libevent-openssl >= 2.0.8

If you are using Ubuntu 12.04, you need the following packages
installed:

* autoconf
* automake
* autotools-dev
* libtool
* pkg-config
* zlib1g-dev
* libcunit1-dev
* libssl-dev
* libxml2-dev
* libevent-dev

Build from git
--------------

Building from git is easy, but please be sure that at least autoconf 2.68 is
used::

    $ autoreconf -i
    $ automake
    $ autoconf
    $ ./configure
    $ make

Building documentation
----------------------

To build documentation, run::

    $ make html

The documents will be generated under ``doc/manual/html/``.

The generated documents will not be installed with ``make install``.

Building Android binary
------------------------

In this section, we briefly describe how to build Android binary using
`Android NDK <http://developer.android.com/tools/sdk/ndk/index.html>`_
cross-compiler on Debian Linux.

We offer ``android-config`` and ``android-make`` scripts to make the
build easier. To make these script work, NDK toolchain must be
installed in the following way. First, let introduce ``ANDROID_HOME``
environment variable. We need to install toolchain under
``$ANDROID_HOME/toolchain``. An user can freely choose the path for
``ANDROID_HOME``.  For example, to install toolchain under
``$ANDROID_HOME/toolchain``, do this in the the directory where NDK is
unpacked::

    $ build/tools/make-standalone-toolchain.sh --platform=android-9 --install-dir=$ANDROID_HOME/toolchain

The platform level is not important here because we don't use Android
specific C/C++ API.

The dependent libraries, such as OpenSSL and libevent should be built
with the toolchain and installed under ``$ANDROID_HOME/usr/local``.
We recommend to build these libraries as static library to make the
deployment easier. libxml2 support is currently disabled.

We use zlib which comes with Android NDK, so we don't have to build it
by ourselves.

Before running ``android-config`` and ``android-make``,
``ANDOIRD_HOME`` environment variable must be set to point to the
correct path.

After ``android-config``, run ``android-make`` to compile sources.
``android-make`` is just include path to cross compiler in ``PATH``
and run make. So if you include path to corss compiler by yourself,
you can just run make to build spdylay and tools as usual.

API
---

The public API reference is available on online. Visit
http://spdylay.sourceforge.net/.  All public APIs are in
*spdylay/spdylay.h*. All public API functions as well as the callback
function typedefs are documented.

SPDY Client and Server Programs
-------------------------------

The *src* directory contains SPDY client and server implementations
using Spdylay library. These programs are intended to make sure that
Spdylay API is acutally usable for real implementation and also for
debugging purposes. Please note that OpenSSL with `NPN
<http://technotes.googlecode.com/git/nextprotoneg.html>`_ support is
required in order to build and run these programs.  At the time of
this writing, the OpenSSL 1.0.1 supports NPN.

Spdycat - SPDY client
+++++++++++++++++++++

The SPDY client is called ``spdycat``. It is a dead simple downloader
like wget/curl. It connects to SPDY server and gets resources given in
the command-line::

    $ src/spdycat -h
    Usage: spdycat [-Oansv23] [-t <SECONDS>] [-w <WINDOW_BITS>] [--cert=<CERT>]
                   [--key=<KEY>] [--no-tls] [-d <FILE>] [-m <N>] <URI>...

    OPTIONS:
        -v, --verbose      Print debug information such as reception/
                           transmission of frames and name/value pairs.
        -n, --null-out     Discard downloaded data.
        -O, --remote-name  Save download data in the current directory.
                           The filename is dereived from URI. If URI
                           ends with '/', 'index.html' is used as a
                           filename. Not implemented yet.
        -2, --spdy2        Only use SPDY/2.
        -3, --spdy3        Only use SPDY/3.
        -t, --timeout=<N>  Timeout each request after <N> seconds.
        -w, --window-bits=<N>
                           Sets the initial window size to 2**<N>.
        -a, --get-assets   Download assets such as stylesheets, images
                           and script files linked from the downloaded
                           resource. Only links whose origins are the
                           same with the linking resource will be
                           downloaded.
        -s, --stat         Print statistics.
        -H, --header       Add a header to the requests.
        --cert=<CERT>      Use the specified client certificate file.
                           The file must be in PEM format.
        --key=<KEY>        Use the client private key file. The file
                           must be in PEM format.
        --no-tls           Disable SSL/TLS. Use -2 or -3 to specify
                           SPDY protocol version to use.
        -d, --data=<FILE>  Post FILE to server. If - is given, data
                           will be read from stdin.
        -m, --multiply=<N> Request each URI <N> times. By default, same
                           URI is not requested twice. This option
                           disables it too.

    $ src/spdycat -nv https://www.google.com/
    [  0.025] NPN select next protocol: the remote server offers:
              * spdy/3
              * spdy/2
              * http/1.1
              NPN selected the protocol: spdy/3
    [  0.035] recv SETTINGS frame <version=3, flags=0, length=20>
              (niv=2)
              [4(1):100]
              [7(0):12288]
    [  0.035] send SYN_STREAM frame <version=3, flags=1, length=106>
              (stream_id=1, assoc_stream_id=0, pri=3)
              :host: www.google.com
              :method: GET
              :path: /
              :scheme: https
              :version: HTTP/1.1
              accept: */*
              user-agent: spdylay/0.2.0
    [  0.077] recv SYN_REPLY frame <version=3, flags=0, length=558>
              (stream_id=1)
              :status: 302 Found
              :version: HTTP/1.1
              cache-control: private
              content-length: 222
              content-type: text/html; charset=UTF-8
              date: Sun, 13 May 2012 08:02:54 GMT
              location: https://www.google.co.jp/
              server: gws
              x-frame-options: SAMEORIGIN
              x-xss-protection: 1; mode=block
    [  0.077] recv DATA frame (stream_id=1, flags=1, length=222)
    [  0.077] send GOAWAY frame <version=3, flags=0, length=8>
              (last_good_stream_id=0)

Spdyd - SPDY server
+++++++++++++++++++

SPDY server is called ``spdyd`` and serves static files. It is single
threaded and multiplexes connections using non-blocking socket. The
static files are read using blocking I/O system call, ``read(2)``. It
speaks SPDY/2 and SPDY/3::

    $ src/spdyd --htdocs=/your/htdocs/ -v 3000 server.key server.crt
    IPv4: listen on port 3000
    IPv6: listen on port 3000
    The negotiated next protocol: spdy/3
    [id=1] [ 17.456] send SETTINGS frame <version=3, flags=0, length=12>
              (niv=1)
              [4(0):100]
    [id=1] [ 17.457] recv SYN_STREAM frame <version=3, flags=1, length=108>
              (stream_id=1, assoc_stream_id=0, pri=3)
              :host: localhost:3000
              :method: GET
              :path: /README
              :scheme: https
              :version: HTTP/1.1
              accept: */*
              user-agent: spdylay/0.2.0
    [id=1] [ 17.457] send SYN_REPLY frame <version=3, flags=0, length=113>
              (stream_id=1)
              :status: 200 OK
              :version: HTTP/1.1
              cache-control: max-age=3600
              content-length: 15
              date: Sun, 13 May 2012 08:06:12 GMT
              last-modified: Tue, 17 Jan 2012 15:39:01 GMT
              server: spdyd spdylay/0.2.0
    [id=1] [ 17.467] send DATA frame (stream_id=1, flags=0, length=15)
    [id=1] [ 17.467] send DATA frame (stream_id=1, flags=1, length=0)
    [id=1] [ 17.468] stream_id=1 closed
    [id=1] [ 17.468] recv GOAWAY frame <version=3, flags=0, length=8>
              (last_good_stream_id=0)
    [id=1] [ 17.468] closed

Currently, ``spdyd`` needs ``epoll`` or ``kqueue``.

Shrpx - A reverse proxy for SPDY/HTTPS
++++++++++++++++++++++++++++++++++++++

The ``shrpx`` is a multi-threaded reverse proxy for SPDY/HTTPS.  It
converts SPDY/HTTPS traffic to plain HTTP.  It is initially developed
as a reverse proxy, but now it has other operation modes such as a
frontend forward proxy.  For example, with ``--spdy-proxy`` (``-s`` in
shorthand) option, it can be used as secure SPDY proxy with a proxy
(e.g., Squid) in the backend.  With ``--cliet-proxy`` (``-p``) option,
it acts like an ordinaly forward proxy but expects secure SPDY proxy
in the backend. Thus it becomes an adapter to secure SPDY proxy for
clients which does not support secure SPDY proxy. The another notable
operation mode is ``--spdy-relay``, which just relays SPDY/HTTPS
traffic to the backend in SPDY. The following table summarizes the
operation modes.

================== ========== ======= =============
Mode option        Frontend   Backend Note
================== ========== ======= =============
default            SPDY/HTTPS HTTP    Reverse proxy
``--spdy``         SPDY/HTTPS HTTP    SPDY proxy
``--spdy-relay``   SPDY/HTTPS SPDY
``--client``       HTTP       SPDY
``--client-proxy`` HTTP       SPDY    Forward proxy
================== ========== ======= =============

The ``shrpx`` supports configuration file. See ``--conf`` option and
sample configuration file ``shrpx.conf.sample``.

We briefly describe the architecture of ``shrpx`` here.  It has a
dedicated thread which listens on server sockets.  When it accepted
the incoming connection, it passes the file descriptor of the incoming
connection to one of the worker thread.  Each worker thread has its
own event loop and can handle many connections using non-blocking I/O.
The number of worker thread can be specified using the command-line
option. The `libevent <http://libevent.org/>`_ is used to handle
low-level network I/O.

Here is the command-line options::

    $ src/shrpx -h
    Usage: shrpx [-Dh] [-s|--client|-p] [-b <HOST,PORT>]
                 [-f <HOST,PORT>] [-n <CORES>] [-c <NUM>] [-L <LEVEL>]
                 [OPTIONS...] [<PRIVATE_KEY> <CERT>]

    A reverse proxy for SPDY/HTTPS.

    Positional arguments:
        <PRIVATE_KEY>      Set path to server's private key. Required
                           unless either -p or --client is specified.
        <CERT>             Set path to server's certificate. Required
                           unless either -p or --client is specified.

    OPTIONS:

      Connections:
        -b, --backend=<HOST,PORT>
                           Set backend host and port.
                           Default: '127.0.0.1,80'
        -f, --frontend=<HOST,PORT>
                           Set frontend host and port.
                           Default: '0.0.0.0,3000'
        --backlog=<NUM>    Set listen backlog size.
                           Default: 256
        --backend-ipv4     Resolve backend hostname to IPv4 address
                           only.
        --backend-ipv6     Resolve backend hostname to IPv6 address
                           only.

      Performance:
        -n, --workers=<CORES>
                           Set the number of worker threads.
                           Default: 1

      Timeout:
        --frontend-spdy-read-timeout=<SEC>
                           Specify read timeout for SPDY frontend
                           connection. Default: 180
        --frontend-read-timeout=<SEC>
                           Specify read timeout for non-SPDY frontend
                           connection. Default: 180
        --frontend-write-timeout=<SEC>
                           Specify write timeout for both SPDY and
                           non-SPDY frontends.
                           connection. Default: 60
        --backend-read-timeout=<SEC>
                           Specify read timeout for backend connection.
                           Default: 900
        --backend-write-timeout=<SEC>
                           Specify write timeout for backend
                           connection. Default: 60
        --backend-keep-alive-timeout=<SEC>
                           Specify keep-alive timeout for backend
                           connection. Default: 60
        --backend-http-proxy-uri=<URI>
                           Specify proxy URI in the form
                           http://[<USER>:<PASS>@]<PROXY>:<PORT>. If
                           a proxy requires authentication, specify
                           <USER> and <PASS>. Note that they must be
                           properly percent-encoded. This proxy is used
                           when the backend connection is SPDY. First,
                           make a CONNECT request to the proxy and
                           it connects to the backend on behalf of
                           shrpx. This forms tunnel. After that, shrpx
                           performs SSL/TLS handshake with the
                           downstream through the tunnel. The timeouts
                           when connecting and making CONNECT request
                           can be specified by --backend-read-timeout
                           and --backend-write-timeout options.

      SSL/TLS:
        --ciphers=<SUITE>  Set allowed cipher list. The format of the
                           string is described in OpenSSL ciphers(1).
        -k, --insecure     When used with -p or --client, don't verify
                           backend server's certificate.
        --cacert=<PATH>    When used with -p or --client, set path to
                           trusted CA certificate file.
                           The file must be in PEM format. It can
                           contain multiple certificates. If the
                           linked OpenSSL is configured to load system
                           wide certificates, they are loaded
                           at startup regardless of this option.
        --private-key-passwd-file=<FILEPATH>
                           Path to file that contains password for the
                           server's private key. If none is given and
                           the private key is password protected it'll
                           be requested interactively.
        --subcert=<KEYPATH>:<CERTPATH>
                           Specify additional certificate and private
                           key file. Shrpx will choose certificates
                           based on the hostname indicated by client
                           using TLS SNI extension. This option can be
                           used multiple times.

      SPDY:
        -c, --spdy-max-concurrent-streams=<NUM>
                           Set the maximum number of the concurrent
                           streams in one SPDY session.
                           Default: 100
        --frontend-spdy-window-bits=<N>
                           Sets the initial window size of SPDY
                           frontend connection to 2**<N>.
                           Default: 16
        --backend-spdy-window-bits=<N>
                           Sets the initial window size of SPDY
                           backend connection to 2**<N>.
                           Default: 16
        --backend-spdy-no-tls
                           Disable SSL/TLS on backend SPDY connections.
                           SPDY protocol must be specified using
                           --backend-spdy-proto
        --backend-spdy-proto
                           Specify SPDY protocol used in backend
                           connection if --backend-spdy-no-tls is used.
                           Default: spdy/3

      Mode:
        -s, --spdy-proxy   Enable secure SPDY proxy mode.
        --spdy-bridge      Communicate with the backend in SPDY. Thus
                           the incoming SPDY/HTTPS connections are
                           converted to SPDY connection and relayed to
                           the backend. See --backend-http-proxy-uri
                           option if you are behind the proxy and want
                           to connect to the outside SPDY proxy.
        --client           Instead of accepting SPDY/HTTPS connection,
                           accept HTTP connection and communicate with
                           backend server in SPDY. To use shrpx as
                           a forward proxy, use -p option instead.
        -p, --client-proxy Like --client option, but it also requires
                           the request path from frontend must be
                           an absolute URI, suitable for use as a
                           forward proxy.

      Logging:
        -L, --log-level=<LEVEL>
                           Set the severity level of log output.
                           INFO, WARNING, ERROR and FATAL.
                           Default: WARNING
        --accesslog        Print simple accesslog to stderr.
        --syslog           Send log messages to syslog.
        --syslog-facility=<FACILITY>
                           Set syslog facility.
                           Default: daemon

      Misc:
        --add-x-forwarded-for
                           Append X-Forwarded-For header field to the
                           downstream request.
        --no-via           Don't append to Via header field. If Via
                           header field is received, it is left
                           unaltered.
        -D, --daemon       Run in a background. If -D is used, the
                           current working directory is changed to '/'.
        --pid-file=<PATH>  Set path to save PID of this program.
        --user=<USER>      Run this program as USER. This option is
                           intended to be used to drop root privileges.
        --conf=<PATH>      Load configuration from PATH.
                           Default: /etc/shrpx/shrpx.conf
        -v, --version      Print version and exit.
        -h, --help         Print this help and exit.

For those of you who are curious, ``shrpx`` is an abbreviation of
"Spdy/https to Http Reverse ProXy".

Without any of ``-s``, ``--spdy-bridge``, ``-p`` and ``--client``
options, ``shrpx`` works as reverse proxy to the backend server::

    Client <-- (SPDY, HTTPS) --> Shrpx <-- (HTTP) --> Web Server
                            [reverse proxy]

With ``-s`` option, it works as secure SPDY proxy::

    Client <-- (SPDY, HTTPS) --> Shrpx <-- (HTTP) --> Proxy
                              [SPDY proxy]            (e.g., Squid)

The ``Client`` in the above is needs to be configured to use shrpx as
secure SPDY proxy.

At the time of this writing, Chrome is the only browser which supports
secure SPDY proxy. The one way to configure Chrome to use secure SPDY
proxy is create proxy.pac script like this::

    function FindProxyForURL(url, host) {
        return "HTTPS SERVERADDR:PORT";
    }

``SERVERADDR`` and ``PORT`` is the hostname/address and port of the
machine shrpx is running.  Please note that Chrome requires valid
certificate for secure SPDY proxy.

Then run chrome with the following arguments::

    $ google-chrome --proxy-pac-url=file:///path/to/proxy.pac --use-npn

.. note::

   At the time of this writing, Chrome 24 limits the maximum
   concurrent connections to the proxy to 32. And due to the
   limitation of socket pool handling in Chrome, it is quickly filled
   up if SPDY proxy is used and many SPDY sessions are established. If
   it reaches the limit, the new connections are simply blocked until
   existing connections are timed out. (See `Chrome Issue 92244
   <https://code.google.com/p/chromium/issues/detail?id=92244>`_). The
   workaround is make the number of maximum connections high, say, 99,
   which is the highest. To do this, you need to change so called
   Policy setup.  See `Policy Templates
   <http://dev.chromium.org/administrators/policy-templates>`_ for
   details how to change Policy setup on the platform you use.  The
   Policy name we are looking for is `MaxConnectionsPerProxy
   <http://dev.chromium.org/administrators/policy-list-3#MaxConnectionsPerProxy>`_
   For example, if you are using Linux, follow the instruction
   described in `Linux Quick Start
   <http://dev.chromium.org/administrators/linux-quick-start>`_ and
   create ``/etc/opt/chrome/policies/managed/test_policy.json`` file
   with the following content and restart Chrome::

       {
           "MaxConnectionsPerProxy" :99
       }

With ``--spdy-bridge``, it accepts SPDY/HTTPS connections and
communicates with backend in SPDY::

    Client <-- (SPDY, HTTPS) --> Shrpx <-- (SPDY) --> Web or SPDY Proxy etc
                              [SPDY bridge]           (e.g., shrpx -s)

With ``-p`` option, it works as forward proxy and expects that the
backend is secure SPDY proxy::

    Client <-- (HTTP) --> Shrpx <-- (SPDY) --> Secure SPDY Proxy
                     [forward proxy]         (e.g., shrpx -s or node-spdyproxy)

The ``Client`` is needs to be configured to use shrpx as forward proxy.

In this configuration, clients which do not support secure SPDY proxy
can use secure SPDY proxy through ``shrpx``. Putting ``shrpx`` in the
same box or same network with the clients, this configuration can
bring the benefits of secure SPDY proxy to those clients. Since the
maximum number of connections per server still applies in proxy
connection, the performance gain is not obvious. For example, if the
maximum number of connections per server is 6, after sending 6
requests to the proxy, client blocks further requests, which kills
performance which might be gained in SPDY connection.  For clients
which can tweak these values (e.g.,
``network.http.max-connections-per-server`` in Firefox), increasing
them may improve the performance.

With ``--client`` option, it works as reverse proxy and expects that
the backend is SPDY-enabled Web server::

    Client <-- (HTTP) --> Shrpx <-- (SPDY) --> Web Server
                     [reverse proxy]

For the operation modes which talk to the backend in SPDY, the backend
connections can be tunneled though HTTP proxy. The proxy is specified
using ``--backend-http-proxy-uri`` option. The following figure
illustrates the example of ``--spdy-bridge`` and
``--backend-http-proxy-uri`` option to talk to the outside SPDY proxy
through HTTP proxy::

    Client <-- (SPDY, HTTPS) --> Shrpx <-- (SPDY) --
                             [SPDY bridge]

            --===================---> SPDY Proxy
              (HTTP proxy tunnel)     (e.g., shrpx -s)

Examples
--------

The *examples* directory contains a simple SPDY client implementation
in C.

Python-Spdylay - Python Wrapper
-------------------------------

The library comes with Python wrapper ``python-spdylay``. See
``python`` directory.