Why and how we use IPv6

[author: alex]

As some of you might know, we use IPv6 for 99% of our server infrastructure communications.

We were unable to completely phase out IPv4 due to the rest of the world still using it, but we’ve taken some pretty good steps forward.

In this blog post, I plan to explain what we’ve done, how we’ve done it, and our IPv6 plan for the future.

Why IPv6

Unless you’ve been living under a rock, you would know that IPv4 is not the future. Not only is the IPv6 address space much larger, but it has some neat and useful features which we want to take advantage of:

End-to-End communication between hosts, without NAT. I read somewhere in a Cisco guide a while back: “NAT is not PAT”. Pat is the co-founder of Unscramble. Pat is a cool guy. NAT is annoying. We don’t like NAT (no offence to anyone named Nat).
Easier IP allocation, since we’re not separating networks by /8, /16 or /24, we can be much more creative and flexible in our numbering scheme.
Native IPSec. Some of our servers are on unsecured networks. We apply mandatory IPSec over IPv6, to ensure all our end-to-end traffic is secure. Particularly between the load-balancers and web servers. This can be done over IPv4 too. I guess we just do it for bragging rights.
Multicast. Our nodes can join/leave a multicast group and get all data for that group. With IPv6 it’s the easiest thing. We don’t use it yet, but we’ll let you know when we do, and how.

How we did it

Unfortunately a few of the open source software we use only supported IPv4, so we started by patching software to support IPv6, and of course releasing the code.

Our approach has mostly been to enable dual-stack compatibility, but in some cases this was not necessary or useful in any way, so we completely replaced IPv4 with IPv6.

Here are some of our code changes:

Redis 2.6.10-ipv6 - There was no IPv6 support for Redis before 2.8 (unstable) so I created a patch to replace IPv4. We’ve been using it in production for quite some time without issues.
Redis-rb - I modified this Ruby gem to allow our Ruby apps to talk to Redis over IPv6.
StatsD-node client - We wanted to support IPv4 and IPv6, but not concurrently. It’s easy to change by simply setting a flag. This was the quickest change and was merged upstream by the Etsy team. Boom!
StatsD Ruby Gem - This change adds IPv6 support, but only one mode can work at a time (depending on the address of the server). The client tries to guess the address family of the server you’re trying to connect to, and it’ll use IPv4 or IPv6 based on the result.

Our servers

We run a mix of Debian Linux and FreeBSD servers, and both platforms support IPv6 out of the box. How lovely.

We didn’t remove IPv4 though. Unfortunately some Debian/FreeBSD package repos and source code repos *cough* GitHub *cough* are still only accessible over IPv4.

We also occasionally want to access certain servers remotely, and if we don’t happen to be on an IPv6-enabled network, then we’re locked-out.

Eventually I want to completely remove all IPv4 addresses from our servers, but this doesn’t seem feasible just yet, at least not until we have our own package/code repositories. For now we’re fine using public servers and *sigh* IPv4.

Other IPv6 stuff

99% of the services running on our servers only bind to an IPv6 address: redis, statsd, node, apache, nginx, unicorn, ipsec daemons, haproxy, etc. The only IPv4 listeners we have are some hosts with sshd, and haproxy which of course forwards requests to our web servers over IPv6.

What about you?

We want to turn this into a discussion. Perhaps our approach isn’t the best, but our quick fixes / patches allow us to move forward with technology instead of staying behind with the old. In my opinion that’s better than sitting around and waiting for someone else to do it for us.

What are you and your company doing to support, enable, use and promote IPv6?