how to make OpenVPN failover automatically

[johnsonx42]

I figured out how to make an active OpenVPN connection automatically failover to the backup WAN connection. (note: I did this for a client-to-site connection, but I see no reason it wouldn't work precisely the same way for a site-to-site connection)

First, this can only work in conjunction with the WAN Failover module. Because OpenVPN uses UDP, and UDP is stateless, the UDP packets coming from the server will only go out via the default gateway... so even though OpenVPN listens on all IP's, you can only actually establish a connection via the current default route. So requirement #1 is to have WAN Failover properly configured.

Next, as has been discussed in other threads here, putting multiple "remote" directives in the .ovpn file doesn't work. I don't know why it doesn't work, the OpenVPN documentation says it's supposed to work, but it doesn't. If you put in two, it only tries the first one and then gives up if it doesn't connect.

So, the solution is to use dyndns.org. They still give you one dynamic hostname for free, though you have to sign-up for a free trial of DynDNS Pro to get it, then cancel it within 14 days (though for the lousy 20 bucks I'll probably just tell the client to pay for it, just in case).

My client's UT public hostname is "firewall.domain.com", and I had a DNS "A" record pointing that to the public IP (and used that hostname in the OpenVPN configuration - very important). So I setup a DynDNS host "firewall-domain.dyndns.org", and then deleted the DNS "A" record and replaced it with a CNAME record pointing "firewall.domain.com" to "firewall-domain.dyndns.org".

(you could of course use the dyndns.org hostname directly, but this was an existing, already deployed configuration that also had a purchased SSL certificate, so the hostname had to stay the same)

The next trick is getting the DynDNS IP address updated quickly... it would be really slick if WAN Failover were to integrate the DynDNS updater and trigger it with every failover. But it doesn't (enhancement idea!).

In absence of that, the obvious solution was to put a DynDNS updater on a server behind UT.

I discovered that the official DynDNS Windows Updater program only checks the IP address every 10 minutes and can't be configured! That's useless for this purpose.

I found another dyndns updater that is command line based and can be fully configured called "inadyn" (it's multi-platform - installed it on a windows 2k8 box). It can check as often as I want it to, down to every millisecond... that would be excessive of course, so I set it to check the IP every ten seconds. I installed it as a service with the config in the registry. See: http://dyn.com/support/clients/linux/. I used the command line:

--background -u username -p password --update_period 10000 --alias firewall-domain.dyndns.org --log_file c:\\inadyn\\inadyn_srv.log --verbose 1

So now everything is complete. The OpenVPN client DOES do a new DNS lookup when retrying the connection - it is designed to work with dynamic IP's on both ends which is effectively what we have.

With the default settings in the OpenVPN client, an active VPN connection does successfully fail-over to the backup connection in a little over 2 minutes. You can make it do it in less time if you modify your OpenVPN configuration file; there's a line that says "keepalive 10 120". If you change that to "keepalive 10 60" it'll failover in about 1 minute. It probably wouldn't be wise to go much less than that: if it tries to reconnect too soon before everything else is ready, it'll just fail and then you'd have to reconnect manually. Presumably failovers will be rare, so the ~2 minute wait shouldn't be a big issue.

If the VPN connection is initiated after a failover has already happened, then the VPN will immediately connect to the backup with no delay.

Enjoy.

Here's the openvpn log of a failover and fail-back:

Sat Feb 04 00:33:22 2012 OpenVPN 2.2.1 Win32-MSVC++ [SSL] [LZO2] built on Jul  1 2011
Sat Feb 04 00:33:22 2012 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Sat Feb 04 00:33:23 2012 LZO compression initialized
Sat Feb 04 00:33:23 2012 UDPv4 link local: [undef]
Sat Feb 04 00:33:23 2012 UDPv4 link remote: 108.58.xxx.xxx:1194
Sat Feb 04 00:33:56 2012 [server.does.not.exists] Peer Connection Initiated with 108.58.xxx.xxx:1194
Sat Feb 04 00:33:58 2012 TAP-WIN32 device [Local Area Connection* 18] opened: \\.\Global\{7C3B5638-9C8E-40DA-9914-1A6EFBE50DA8}.tap
Sat Feb 04 00:33:58 2012 Notified TAP-Win32 driver to set a DHCP IP/netmask of 172.16.0.5/255.255.255.252 on interface {7C3B5638-9C8E-40DA-9914-1A6EFBE50DA8} [DHCP-serv: 172.16.0.6, lease-time: 31536000]
Sat Feb 04 00:33:58 2012 Successful ARP Flush on interface [23] {7C3B5638-9C8E-40DA-9914-1A6EFBE50DA8}
Sat Feb 04 00:34:03 2012 Initialization Sequence Completed

 ----I forced a failover at about 01:02, at which point data through the tunnel stopped----

Sat Feb 04 01:04:06 2012 [server.does.not.exists] Inactivity timeout (--ping-restart), restarting
Sat Feb 04 01:04:06 2012 SIGUSR1[soft,ping-restart] received, process restarting
Sat Feb 04 01:04:08 2012 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Sat Feb 04 01:04:08 2012 Re-using SSL/TLS context
Sat Feb 04 01:04:08 2012 LZO compression initialized
Sat Feb 04 01:04:08 2012 UDPv4 link local: [undef]
Sat Feb 04 01:04:08 2012 UDPv4 link remote: 108.12.xxx.xxx:1194
Sat Feb 04 01:04:12 2012 [server.does.not.exists] Peer Connection Initiated with 108.12.xxx.xxx:1194
Sat Feb 04 01:04:14 2012 Preserving previous TUN/TAP instance: Local Area Connection* 18
Sat Feb 04 01:04:14 2012 Initialization Sequence Completed

   ---- I brought the primary WAN back a little before 01:06---

Sat Feb 04 01:07:47 2012 [server.does.not.exists] Inactivity timeout (--ping-restart), restarting
Sat Feb 04 01:07:47 2012 SIGUSR1[soft,ping-restart] received, process restarting
Sat Feb 04 01:07:49 2012 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Sat Feb 04 01:07:49 2012 Re-using SSL/TLS context
Sat Feb 04 01:07:49 2012 LZO compression initialized
Sat Feb 04 01:07:50 2012 UDPv4 link local: [undef]
Sat Feb 04 01:07:50 2012 UDPv4 link remote: 108.58.xxx.xxx:1194
Sat Feb 04 01:07:52 2012 [server.does.not.exists] Peer Connection Initiated with 108.58.xxx.xxx:1194
Sat Feb 04 01:07:54 2012 Preserving previous TUN/TAP instance: Local Area Connection* 18
Sat Feb 04 01:07:54 2012 Initialization Sequence Completed

(yes, the primary and backup WANs do both start with 108; one is from the cable company, the other is a phone company DSL)

[johnsonx42]

oh, by the way, I found a cool way to simulate a primary WAN failure in a risk-free way, on a server that is 3000 miles away in the middle of the night. I just temporarily pointed the ping rule for the primary WAN at my own UT here, then put in a packet filter to drop ICMP Echo packets; if I enable the filter, the remote UT thinks the primary WAN has failed and fails over to the backup WAN... then I disable the filter, pings come back and the remote UT changes back again.

[paulgossett]

When you say "proper WAN failover configuration", can you be a little more specific? I know there are four different ways to configure the failover test, Ping, ARP, HTTP, DNS, so which method are you referring to? We originally set ours up with Ping testing, but I have the feeling that isn't what you are referring to.

Also, if somebody pulls the plug on my WAN1 connection, will the method you mention below redirect the OpenVPN UDP traffic to the WAN2 interface?

Thanks.

[johnsonx42]

"proper WAN failover configuration" just means that you've set it up such that it works. Perhaps using such explicit wording implies more than I meant it to. All I wanted to be clear on is that you do need WAN Failover - this won't work if you just setup two WAN connections without the Failover service.

Yes, if you pull the plug on WAN1, both existing and new OpenVPN sessions will move to the WAN2 interface. With default settings, existing sessions take about 2 minutes to time-out, retry and re-connect. New sessions will connect to the active WAN interface immediately.

[sharrisonUK]

Good post - thanks.
Just to add that I have had reasonable success using dnsmadeeasy over the past year or so for configuring inbound wan failover with static IPs. They also offer dynamic dns as well... There's a modest charge - about $30-40 pa but depends on what you need - the connectivity tests are configurable to a fine degree and can flip the necessary public DNS records quickly.

[johnsonx42]

I'm glad to hear you've found this helpful.

A comment about dnsmadeeasy: For this particular application, I do not think an external DNS failover solution would be a good fit though it would probably work most of the time.

The key here is that WAN Failover modifies the default route, and therefore the interface that OpenVPN will work on, based on which interface IT THINKS is up based on the tests it runs. It doesn't matter which interface is ACTUALLY up. So you could easily have a situation where WAN Failover has decided, for whatever reason, that it should switch to your secondary WAN interface, but an external DNS failover monitor thinks your primary WAN is ok. Even if WAN Failover on Untangle is wrong and shouldn't have failed over, the deed has been done and OpenVPN will no longer function on the primary WAN. The external DNS failover service would have no way to tell that this has happened - all it can know is which interface is up from it's point of view. Likewise the reverse would be true - if the DNS Failover service thinks your primary WAN is down when it isn't, it will switch the IP address and break your OpenVPN clients.

Having the IP monitoring come from a box behind Untangle ensures that the IP address of the currently active interface is the one that gets used at all times.

The ideal solution would of course be if WAN Failover would integrate the dynamic DNS updater and trigger it with every WAN change, but in absence of that having an always-on host behind Untangle keep the IP updated is the best I can think of. I did consider just loading the linux version of inadyn on to the Untangle box itself, but decided not to hack things that way just yet.