Building The Net

Setting up IPv6 connectivity.

Back in December `09, my company ACT USA, began testing IPv6. These tests quickly advanced to our production environment. Over the last six months, I have been in the process of setting up native IPv6 connectivity for all our data centers. This connectivity is based on the dual stack model. This article attempts to cover the technology available, and the choices I made based on that technology.

Now, I know many good folks believe that IPv6 will never catch on, and another solution will sweep in and save us from having to move away from IPv4. Here is a quote from one of my other sites that sums up my position on IPv6 deployment:

“Many have cried of IPv4 depletion. They shout that it is just around the corner. I’m not going to argue the point of whether those who scream doom and gloom are sages, or the Internet equivalent of Chicken Littles, but I will argue this. We must all realize, that sooner or later, the great well of IPv4 address space will run dry. The question then will be, did we properly prepare?” — Stuart Sheldon

So, with that in mind…

Shortcuts for the Impatient

• Selecting your transition method
• Notes about IPv6 and security
  
• Choosing your hardware
• Recommended Deployment Strategy
  
• Closing Thoughts
  

Getting an IPv6 connection.

You would think with all the buzz regarding IPv4 depletion, that you could easily get IPv6 connectivity from your current upstream transport providers. This, unfortunately, is not always the case. I actually had to actively campaign with my upstreams to get them to seek IPv6 peers.

Depending on your size, you will either request IPv6 address space from your upstream, or try to get a direct allocation from your RIR. Most people will go to their upstream first. If your upstream does not support IPv6 yet, you can contact a “Tunnel Broker” and get hooked up that way. A tunnel broker is not your final solution, but it will bring you into the IPv6 world, and let you start testing.

When we were issued our initial IPv6 allocation from ARIN, I quickly found that none of our upstream transport providers supported IPv6. I searched around the Internet and found tunnelbroker.net, which is run by Hurricane Electric. They offer IPv6 tunnels and allow you to advertise your direct allocation of IPv6 address space via BGP free of charge. This connectivity allowed us to begin testing without delay. If you are an ISP / Hosting Provider, and have received you IPv6 allocation from your RIR, but do not have a transport provider that offers native IPv6 connectivity, I strongly recommend using tunnelbroker.net to get a leg up on IPv6 in your environment.

IPv6 Only Networks or Dual Stack?

Providers have two choices when it comes to IPv6 deployment:

• IPv6 Only Networks
• Dual Stack

The “IPv6 Only Networks” method is the concept of setting up segments of your network that only bind to the IPv6 protocol stack. The “Dual Stack” method will have you bind both the IPv4 and IPv6 protocol stacks to all hosts, devices, and routers on all segments of your network. Dual stacking protocols was very common in the 90s when most businesses were running IPX/SPX and TCP/IP at the same time, and used to be called “Dissimilar Networks on the Same Physical Media”.

For me, the choice was obvious, since our business model is heavier on the hosting side, dual stack just made sense. I believe that our hosting customers will want both IPv4 and IPv6 clients to be able to reach their public facing services. I also think that the “IPv6 Only Networks” model has some problems to solve before it will be viable.

Let’s take a quick look at both these methods…

IPv6 Only Networks.

The “IPv6 Only Networks” model seems to present a large set of problems on both sides (hosted services and end user termination). On the hosting side, investing in duplicate server hardware, at least for us, would not make a lot of sense (or dollars for that matter). And although there’s great IPv6 support for services like HTTP(S), DNS, SMTP, SNMP, IMAP, POP3, and other standard front end protocols, core services like NFS, MySQL, and other “supporting” protocols seem to have limited, or no IPv6 support. At least that seems to be the case for the versions that our clients commonly use.

Implementing the  “IPv6 Only Networks” model on the client side looks troublesome as well. The IPv6 protocol has some great new things built into it. Things like Router Discovery, Network Discovery, Stateless Auto Configuration, and all sorts of cool stuff that should make client configuration a real breeze. But, for some reason, DNS server settings are not usually accepted or advertised via these cool configuration methods. Stateful client configuration using the DHCPv6 protocol can deliver DNS, but it has limited support on many client systems, so it becomes rather difficult to get a client auto configured without manually setting the DNS server settings on it.

The third issue with the “IPv6 Only Networks” strategy is isolation. How does an IPv6 Only client get to IPv4 network services? The current solutions are based on the concept of NAT or Proxy systems located at and ran by the Carriers / ISPs. Here is the short list of solutions:

• Stateless IP/ICMP Translation – Obsolete
• NAT-PT – Obsolete
• NAPT-PT – Obsolete
• IPv4 NAT
• NAT64 / DNS64
• HTTP Proxy

So, let’s start by talk about the ideas that were eventually obsoleted by the community. Stateless IP/ICMP Translation requires a one to one NAT with an IPv4 address, so it was obsoleted. NAT-PT required lots of horsepower, and worked on the same basis of one to one IPv4 address maps, so it was obsoleted. NAPT-PT was NAT-PT with port translation. It fixed some of the address requirements, but didn’t really address the horsepower issue, so it was chopped as well.

So that leaves us three, IPv4 NAT, NAT64 / DNS64 and HTTP Proxy. Let’s look at these a bit closer.

IPv4 NAT.

The idea behind this is to setup an IPv6 tunnel to pass IPv4 traffic in. Then use regular old IPv4 NAT at the ISP level to deliver IPv4 only content. This seems to me like “NAT on NAT”. Not much more then Dual Stack with reserved IPv4 addresses. But, it is a viable solution if we really let IPv6 deployment lapse too long and run out if IPv4 addresses before we get transitioned.

NAT64 / DNS64.

This is the latest star of the 6 to 4 NAT models. It is based on the idea that if you only translate the IPv4 only sites, then you will have enough reduction in load, to manage all the translated connections. Basically, if you request a site that does not have a “AAAA” record, the DNS server rewrites the address to a “4in6” style address, which is then routed to a NAT server somewhere to translate the traffic. My opinion is that it will be slow and require way too much hardware to pull off for a large provider, but I’m not going to completely discount this. As they say, “It might have legs”…

HTTP Proxy

This would seem to be the most viable “IPv6 Only Networks” solution. Most of the traffic on the Internet is Email and HTTP. This solution would be the easiest to implement, and the lowest cost solution out there. Now, I know that the technically gifted folks reading this, are probably birthing kittens about now, but hear me out…

This is not a solution for your connectivity, this is one for your sweet old Aunt Betsy in Milwaukee. All dear old Betsy does, is get email from the kids, view pictures on the web, and maybe play a little bridge… For this kind of user, this connectivity will deal with 99.9% of the services they use.

It’s not perfect, but it’s temporary. Remember, these are all transition technologies, they won’t last forever, they are here until we get the servers up on IPv6. And, for the most part, that’s happening.

Dual Stack.

Dual Stack allows you to roll out IPv6 in small chunks. From the hosting side, it is relatively trivial to bring up a IPv6 stack next to your current IPv4 stack on most Operating Systems. Once the IPv6 stack is active, you can then configure and test your public services such as HTTP(s), SMTP, IMAP, POP3, etc, etc, etc… against IPv6. When you are convinced that these services are ready for public consumption, you can add a “AAAA” record for the host. All this, with no real client service disruption. This also buys you valuable time to figure out and fix those little bugs that seem to only show their pinchers when used in production.

Dual Stack from the client side brings the best of both protocol worlds to the host. If you have done your IPv6 homework, you’ve setup “Auto Configuration and Router Advertisement” on your router / gateway, so all you will need to do is tell the client to run IPv6. Since you are also running IPv4, you don’t have to worry about an IPv6 address for your DNS server. You will however need to have IPv6 enabled on your DNS server, but I’ll save that for another blog post. Now, by default, the client will look to see if there is an “AAAA” record, and if so, will connect to the service it is requesting via IPv6, if not, it will connect via IPv4. I’m simplifying this a bit, but it really is fairly straight forward… Here is the short list of Operating Systems that support IPv6 out of the box:

• Linux
• BSD
• MAC OSX
• Most other Unix Type OSs
• Windows XP
• Windows 7
• Windows 2003 Server
• Windows 2008 Server

Windows Vista has issues, but will work with a few tweaks… I understand there is a patch out there for Windows 2000, but if you are running 2000, IPv6 is probably the least of your pressing IT issues.

And the greatest advantage to Dual Stack over IPv6 Only Networks? You don’t need to figure out how to translate the IPv6 packets!

IPv6 and Network Security.

If you have been relying on NAT for your IPv4 security, you’re going to need to re-educate yourself about network and protocol level security. As of right now, there is no such thing as IPv6 to IPv6 NAT. And, there are no plans to implement it. Why you ask? Because IPv4 to IPv4 NAT was created to slow the exhaustion of the IPv4 address pool. It actually solved some issues, but created tons of problems for network application developers. Since there are more IPv6 addresses then grains of sand in the earth, IPv6 to IPv6 NAT is an unnecessary evil… All NAT protocols should be gone once IPv6 is fully adopted. And we should all be glad they’re gone!

An undesired side effect of the adoption of IPv4 to IPv4 NAT, was the sudden reliance on it for security. Although it does create a reasonably secure firewall by default, it also created a generation of network administrators that have blindly relied on it to secure their network assets.

The good news, is that securing IPv6 is reasonably simple with modern firewalls. You can create a simple IPv6 firewall that works a lot like a NAT firewall. Here is an example using Linux:

#!/bin/bash
ip6tables="/sbin/ip6tables"

outif="eth0"
inif="eth1"

# Lets Flush All the Rules
${ip6tables} -F

# We're discovering all the interfaces here
ipv6if=$(/sbin/ip link show \
	| egrep '^[1-9]' \
	| awk -F':' '{ print $2 }' \
	| sed 's/@.*$//g' )

# localhost should still pass everything
${ip6tables} -A INPUT -i lo -j ACCEPT

# Because ICMPv6 is interface specific, and required for any IPv6 connections
# we need to grant icmp6 on all interfaces. This for loop takes care of that
for i in ${ipv6if}; do
	${ip6tables} -A INPUT -p icmpv6 -i ${i} -j ACCEPT
done
${ip6tables} -A FORWARD -p icmpv6 -j ACCEPT

# Here we allow only IPv6 established connections in to the host
${ip6tables} -A INPUT -i ${outif} -m state \
	--state ESTABLISHED,RELATED -j ACCEPT

# We need to let the inside network access the firewall itself
${ip6tables} -A INPUT -i ${inif} -j ACCEPT

# Now here is where we setup our forwarding rules
${ip6tables} -A FORWARD -i ${inif} -j ACCEPT
${ip6tables} -A FORWARD -i ${outif} -m state \
	--state ESTABLISHED,RELATED -j ACCEPT

# And drop everything else coming in.
${ip6tables} -A INPUT -j DROP
${ip6tables} -A FORWARD -j DROP

This will pretty much reproduce the security level you get from IPv4 NAT firewall for your IPv6 traffic. You could then open up inbound ports for services just like you would do with port forwards:

# Open up tcp port 80 to 2001:db8::10. Insert above DROP rules.
${ip6tables} -A FORWARD -i ${outif} -p tcp --dport 80 \
	-d 2001:DB8::10 -j ACCEPT

Which would allow the world to access the web server on host 2001:DB8::10. Speaking of firewalls, maybe we should start talking about hardware compatibility…

Hardware Compatibility with IPv6.

Most hardware manufacturers will tell you that they are 100% IPv6 compatible… And, for the most part, they are telling the truth. The question they might have a little more trouble with would be if they are 100% IPv6 compatible with other vendor hardware? Another possible question, would be if their equipment obeys all the current IPv6 RFCs. I’ve included some software solutions that run on PC hardware as well…

Here is a quick summary:

Routers

Cisco Routers:

From what I can tell at the current moment, Cisco routers support IPv6 in their Advanced IP Services IOS tree. Since most of the Cisco gear ISPs run is pretty much maxed on installable flash and ram, they should have no problem getting the proper IOS version to work with their gear. Unfortunately,  most of the gear that is in place at end nodes is not capable of loading Advanced IP services level IOS bundles. They would require at minimum a flash and ram upgrade, and at worse, replacement.

Since Cisco pretty much sets the pace in enterprise and carrier networks, I guess it’s not up to them to make sure their gear “plays well with others”. So I’ll comment on their compatibility with other manufactures when I cover those manufactures.

As far as features and ability, like I said, everyone strives to talk to them, so we’ll cover this with each individual manufacturer. But here a the important ones to ISPs

• Router Advertisement
• Auto Configuration
• VLAN Support
• Secondary Interface Addresses
• BGP Multi Protocol
• OSPFv3
• ISIS
• IPv6 Access Lists
• IPv6 Prefix Lists
• IPv6 Route Map matches and sets
• PPP
• Multilink PPP
• HDLC
• And all the other settings and protocols considered standard

ImageStream Routers:

Many people have never heard of ImageStream, and that’s a shame. ImageStream has been building enterprise grade routers for over 15 years. They have placed themselves as a direct competitor to Cisco, which is a pretty big claim for a ‘little old company from Plymouth Indiana’. Their product line features industrial grade server hardware with installable WAN cards ranging from T1 to OC12 capibilities. Their routers are Linux based, and use all Open Source software to create their tightly packaged realtime router OS. ImageStream supports IPv6 natively, but to take advantage of all their current features, you’ll need to install their open beta version of their router OS.

ImageStream fully supports IPv6 both tunnelled and natively connected via ethernet with it’s stock router OS. To get PPP and Multilink PPP to work with a Cisco accross a WAN card, you’ll need to download and run the open beta version of their OS. As of this writing, IPv6 does not work accross HDLC.

Since ImageStream’s OS is based Open Source, it had no problems talking to our Linux based routers we use in the data centers. And, if you know the Cisco IOS commands, you will be right at home with the ImageStream OS. Here’s the short list of IPv6 features that are compatible with both Cisco and Linux:

• Router Advertisement
• Auto Configuration
• VLAN Support
• Secondary Interface Addresses
• BGP Multi Protocol
• OSPFv3 (Beta)
• IPv6 Access Lists (for routing only)
• IPv6 Prefix Lists (for routing only)
• IPv6 Route Map matches and sets
• IP6Tables
• Stateful IPv6 Firewalling
• PPP (beta with cisco)
• Multilink PPP (beta with cisco)
• Linux QOS
• Policy Routing

Just a note, all the routing daemons listed are provided by the Quagga Open Source Project.

Adtran:

I tried to get good usable data regarding Adtran routers. But it appears that for now, there is no solid statement from Adtran regarding IPv6.

Linksys:

Currently, there is no IPv6 support in any of the Linksys routers as configured from the factory. More in this under “Firewalls / Security Devices”.

Linux Generic:

Now, you didn’t think that I would forget to talk about Linux did you? I currently build generic Linux based routers using the Debian distribution, and have done extensive testing over the last 6 months. Linux currently supports IPv6 in all of it’s glory. It is fully compatible with Cisco when connected via ethernet. Here is the short list:

• Router Advertisement
• Auto Configuration
• VLAN Support
• Secondary Interface Addresses
• DHCPv6
• DNS Server
• BGP Multi Protocol
• OSPFv3 (Beta)
• IPv6 Access Lists (for routing only)
• IPv6 Prefix Lists (for routing only)
• IPv6 Route Map matches and sets
• IP6Tables
• Stateful IPv6 Firewalling
• Linux QOS
• Policy Routing

As I said, all of our testing was done against Debian, but you can safely bet that any modern Linux distribution can handle this. The specifications for the router were:

• 2.4ghz Xeon
• 4G Ram
• 250G Hard Drive
• 4 Port Intel GB Adapter (PCIe)

We use these in data centers when all we have are Ethernet links.

BSD Flavors:

BSD currently supports IPv6 in all of it’s glory. It is fully compatible with Cisco when connected via ethernet. Here is a list of features:

• Router Advertisement
• Auto Configuration
• VLAN Support
• Secondary Interface Addresses
• DHCPv6
• DNS Server
• BGP Multi Protocol
• OSPFv3 (Beta)
• IPv6 Access Lists (for routing only)
• IPv6 Prefix Lists (for routing only)
• IPv6 Route Map matches and sets
• Stateful IPv6 Firewalling
• QOS

Firewalls / Security Devices

Cisco ASA:

The ASA series products has IPv6 support, but at the time of writing, I was unable to evaluate one that actually had the right firmware to do IPv6. My understanding is that with the right software, the ASA security devices are fully capable to secure both Dual Stack and IPv6 Only Networks. And all IPv6 settings are available in their GUI configuration tool.

Juniper Netscreen:

The Netscreen product line that I am most familiar with is the 5GT. This product was discontinued in 2008, and to my knowledge, there is no IPv6 support for this product. Looking at the Juniper website, it appears that they have moved to an enterprise only product line, and claim to support IPv6 on all their current products via an add on license.

I am unaware of any compatibility issues with other devices at this time.

ImageStream:

ImageStream hardware is fully IPv6 capable as a security device. See above under Routers.

Linux Generic:

Linux is fully IPv6 capable as a security device. See above under Routers.

BSD Flavors:

BSD is fully IPv6 capable as a security device. See above under Routers.

Linksys, Netgear, DLink, and other “Home Routers”:

Sadly, at this time, none of the big home router manufacturers support IPv6. The good news is you can get IPv6 working on a large number of these devices if you load a third party Linux based firmware like DD-WRT or OpenWRT (WARNING! THIS VOIDS YOUR WARRANTY). Alas, even with third party software, you still need to configure IPv6 through the command line.

WAN Cards:

As of right now, the only PC based WAN card (T1/E1 DSU/CSU) that supports IPv6 that I have been able to find, are the cards that ImageStream sells both standalone and in their router products.

Switches:

This was a hit and miss. We use Procurve, 3com, and Cisco managed switches for layer two. All of there current layer two products from these vendors support IPv6 for management.

Hardware Summary.

It seems that only Cisco, and the open source community are showing a strong interest in IPv6. This is kinda disturbing to me. But, hey! Maybe this will give open source a boost!

Recommended Deployment Strategy.

Here’s a flowchart I use when talking to clients about IPv6 deployment:

From a deployment standpoint, regardless of what your plan is for actual deployment, whether it be “Dual Stack” or “IPv6 Only Networks”, this is the standard roadmap. Once you have IPv6 connectivity, you start getting your services setup. One important thing to note, you will want to get off of a 6in4 tunnel as quickly as you can. They work, but they are a bit buggy.

It’s also important to remember that if you start off with a tunnel broker, and you don’t have a direct IPv6 allocation, you will need to be prepared to renumber you IPv6 network when you finally get a real IPv6 connection.

Final Thoughts.

When I began this blog article, my goal was to put the important information for Enterprises, ISPs and Service Providers in one place to review and study before they embarked on their journey into IPv6 deployment. I learned a lot of very valuable things during our deployment, and if you are on the fence as to whether it’s time to jump in… Well, I encourage you to do so!

Oh, and if you want to talk to someone that’s been through an actual deployment, feel free to contact our offices.

Some important things I learned along the way:

6in4 tunnels are a stepping stone, not a solution…

I setup four 6in4 tunnels to Hurricane Electric, and it got us IPv6 connectivity and BGP peering, with our own IPv6 addresses months before we could have if we waited for native connectivity to be available. And it allowed us to flush out 90% of the bugs in both services and connectivity. But it could only bring us so far.

Connecting IPv6 across Ethernet is so much easier then other media types…

I had IPv6 up and routing with BGP in no time across tunnels and Ethernet links, but HDLC, PPP, MP-PPP, and all the rest were a real pain in the (well, you know). To be fair, most of the issues I ran into had to do with software bugs that were in the software because no one else has actually tried to use a particular feature… And to the vendors credit, when these things came up, they were quickly fixed. But some of the things I ran into were just weird.

The best example I can think of was that Cisco required a multihop statement in the neighbor configuration on a PPP link. It was a PPP link, where’s the additional hops?

Understand the differences between IPv4 and IPv6.

I made a statement in an earlier post back in January ’10, “IPv6 is not IPv4 with more addresses…” and that is the stone cold truth! There are some great things about IPv6 that will change the way you think about client deployment and supporting services. Learn the differences between the protocols.

If it isn’t enterprise level gear, then IPv6 probably isn’t fully cooked on it.

I found that if it wasn’t a “real” piece of network gear, then it either performed like crap, or didn’t work at all. I also found that without fail, only the latest versions of firmware seemed to actually support IPv6 properly even on the high end gear. How long has IPv6 been around? 15 years?

Get your transport in place, worry about the services later.

DNS, HTTP, SMTP, IMAP and all the other services are important, and you should consider getting those services working as a high priority, but that shouldn’t be where you start. Making IPv6 available to your customers, and getting your network setup to actually deliver those services should be at the very top of the project list.

Just Keep Swimming!

If the network doesn’t come up the first time, take a deep breath and try again. I finally realized that there are little to no “How-Tos” that talk about native connectivity. Do you know why? Because everyone is running tunnels, and the big boys meet on Ethernet ports. So, if you’re trying to get a PPP link on a Cisco to come up, and it won’t move IPv6 packets, or if you are banging your head against the wall because a BGP session isn’t coming up, and you’ve looked everywhere, don’t be afraid to call the manufacturer. It might just be a bug! And remember Dori’s advice, just keep swimming! You’ll get there, honest.

— Stu