Zero Downtime Deployment with Warp

Using warp, it is easy to perform zero downtime deployment with SO_REUSEPORT. This literate Haskell file creates a server for zero downtime deploys and the repo has utilities that restart it without failed requests.

Outline

• SO_REUSEPORT
• Start Warp with SO_REUSEPORT
• Reloading
• Setup
• Performance
• Design Analysis
• Thanks

SO_REUSEPORT

SO_REUSEPORT is an extension on newer versions of Linux and BSD (avoid OSX) that allows multiple sockets to bind to the same port. Additionally, Linux will load balance connections between sockets.

There is a downside to SO_REUSEPORT: when the number of sockets bound to a port changes, there is the possibility that packets for a single TCP connection will get routed to two different sockets. This will lead to a failed request. The likelihood is very low, but to prevent against this, we use a technique developed by Yelp.

Boring Haskell Import Statements

import Control.Exception
import Data.ByteString.Lazy.Char8 (pack)
import Network.HTTP.Types
import Network.Socket
import Network.Wai
import Network.Wai.Handler.Warp
import System.Posix.Process
import System.Posix.Signals
import System.Posix.Types

Start Warp with SO_REUSEPORT

bindSocketReusePort :: PortNumber -> IO Socket
bindSocketReusePort p =
  bracketOnError (socket AF_INET Stream defaultProtocol) close $ \sock -> do
    mapM_ (uncurry $ setSocketOption sock)
          [ (NoDelay  , 1)
          , (ReuseAddr, 1)
          , (ReusePort, 1) -- <-- Here we add the SO_REUSEPORT flag.
          ]
    bind sock $ SockAddrInet p $ tupleToHostAddress (127, 0, 0, 1)
    listen sock (max 2048 maxListenQueue)
    return sock

main :: IO ()
main = do
  -- Before we shutdown an old version of the server, we need to run health
  -- checks on the new version. The minimal health check is ensuring the new
  -- server is responding to requests. We return the PID in every request to
  -- verify the server is running. We grab the PID on load here.
  CPid processId <- getProcessID
  -- We create our socket and setup a handler to close the socket on
  -- termination, premature or otherwise.
  bracket
    (bindSocketReusePort 7000)
    close
    $ \sock -> do
       -- Before we start the server, we install a signal handler for
       -- SIGTERM to close the socket. This will start the graceful
       -- shutdown.
       installHandler sigTERM (CatchOnce $ close sock) Nothing
       -- Start the server with the socket we created earlier.
       runSettingsSocket defaultSettings sock $ \_ responder ->
         -- Finally, we create a single request for testing if the server is
         -- running by returning the PID.
         responder $ responseLBS status200 [] $ pack $ show processId

Reloading

The following code was copied from the Yelp blog post (link). It is also in the repo in the bin/setup-qdiscs file. All operations require sudo.

# Set up the queuing discipline
tc qdisc add dev lo root handle 1: prio bands 4
tc qdisc add dev lo parent 1:1 handle 10: pfifo limit 1000
tc qdisc add dev lo parent 1:2 handle 20: pfifo limit 1000
tc qdisc add dev lo parent 1:3 handle 30: pfifo limit 1000

# Create a plug qdisc with 1 meg of buffer
nl-qdisc-add --dev=lo --parent=1:4 --id=40: plug --limit 1048576
# Release the plug
nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --release-indefinite

# Set up the filter, any packet marked with "1" will be
# directed to the plug
tc filter add dev lo protocol ip parent 1:0 prio 1 handle 1 fw classid 1:4

iptables -t mangle -I OUTPUT -p tcp -s 127.0.0.1 --syn -j MARK --set-mark 1

Reload

Reloading a new version in production requires a dance with your process supervisor. However, the principle is similar even if the details are different.

1. Stop additional SYN from being delivered using

   sudo nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --buffer
   

2. Start a new version of the server and save the PID.

3. Release the plug and let the SYN packets flow.

   sudo nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --release-indefinite
   

4. Make requests to the health endpoint until the new PID is returned.

5. Stop SYN packets again.

6. Send SIGTERM to the other server processes so they will gracefully shutdown.

7. Release the plug again.

An example for demonstrating this process can be found in reload/Main.hs. The reload app creates a new server and shuts down all other instances. This is for demonstration purposes. In production, you will want to integrate reloading with your process supervisor.

Setup

The setup requires stack, ab, libnl-utils, Linux and, optionally gnuplot. The easiest way to test it is to use the Vagrant file, which will create a VM with everything installed. Vagrant can be downloaded here

Run with the following steps:

$ git clone https://github.com/jfischoff/reuse-port-example
$ cd reuse-port-example
$ vagrant up
$ vagrant ssh
$ cd /vagrant
$ stack setup
$ bin/test

bin/test will:

1. Build the project.
2. Setup the queuing disciplines.
3. Start the server.
4. Run ab.
5. Stop the server.
6. Start the reload every 100 ms.
7. Run ab.

Performance

Design Analysis

huptime was mentioned on the Yelp blog, but was not appropriate for their problem. However, I think it might solve many Haskell web server use cases. If anyone has any experience using huptime, let me know through the github issues of this repo or directly at @jfischoff.

Future Work

An alternative design for utilizing SO_REUSEPORT is to create a parent process that keeps a pool of sockets which its workers inherit on reload. This is essentially the design that nginx (nginx reload, nginx and SO_REUSEPORT) has. The primary advantage is that, in the typical case, we would not need to utilize the plug queuing discipline.

Changing the number of child processes will still be problematic, but we could utilize the plug queueing discipline approach Yelp developed.

Thanks