Router:Linux - Debian - With L2TP Fallback
DSL lines are not perfect, and line resyncs are guaranteed to happen at the most inconvenient moment. Fortunately AAISP provide L2TP for subscribers, but the L2TP takes routing precedence over your DSL line - you therefore can't simply leave your L2TP up and running as a fallback. This means you need to script the process of bringing this interface up upon PPPoE failure.
I investigated PFSense (My current firewall solution) and RouterOS, but neither had very satisfactory scripting abilities. Linux, on the other hand, had no such problem.
In this solution, I have chosen to place a Linux solution acting only as a router in front of my PFSense box. This kept a fairly nice GUI for day to day management, but provided it with a more robust way to connect to AAISP via a Linux router, all of which is hosted on an ESXi box. The obvious cost is an extra hop, which requires a bit of extra CPU and causes a bit of extra latency. When measured I found PFSense to need around 10x the CPU of the linux box (Likely SNORT has a lot to do with this, and there will also be a little extra vSwitch load - SR-IOV is supported if you're desperate to avoid this) and an extra 1-2ms latency.
Throughout this guide, I'll likely refer to your PFSense box. While a few config tweaks will be needed, the guide should apply to more or less any firewall.
If you wanted to use your linux box as a full firewall, you would likely have to script updates to the firewall rules based on interface up / down. I have not looked into this.
This guide provides an example configuration for Debian Buster.
- a Linux machine with Debian Buster already installed and three network interfaces - one for the connection to the modem, one for the connection to the alternative path router, and one for the connection to your LAN. The machine should be ready for internet facing duties, that means strong password!
- a ADSL or FTTC modem, or a fibre ONT (for FTTP) (as appropriate for your connection)
- an alternative path router of some type. In my case I am using a Huawei B315 4G router (With ethernet)
- a firewall which is already configured and working
- at least an extra IPv4/32 routed down your line, otherwise you'll struggle a little with routing - you could either look to see if a bridge configuration is possible, double NAT, or moving the NAT boundary from the firewall to the router. In my example I have a /29, so this isn't an issue
A note on routing
IPv6 routing is trivial in this case, I've simply used 1/65536th of the /48 we're all given for a /64 handoff within the public address range. There are two options for IPv4 ranges. You can either assign a /29 to the handoff interface between the Linux router and the firewall. This would be the "Proper" way to do things, but would waste 3 public IP addresses, which are a precious resource in this day and age. I have instead used RFC1918 private address space as a handoff between the devices, and routed the public /29 block to the private IP of the firewall. The firewall can then be configured with these as virtual addresses, with all 8 addresses kept as usable addresses.
Assumptions and example setup
- ens256 is plugged directly into your modem or ONT
- ens161 will be attached to your PFsense firewall
- ens224 will be attached to your alternative
As the names change from installation to installation, I strongly advise you to copy this wiki page, then do a find and replace. You will end up with extra troubleshooting otherwise!
Before you do anything, you'll probably want to connect your linux box to the internet, update it and make sure it has all the tools you need
apt-get update apt-get install ppp pppoe xl2tpd iproute2 tcpdump net-tools
You may also want open-vm-tools, if you're virtualised
Build network interface configuration
Edit /etc/network/interfaces. This was my final config, you may want to leave the up and pppoe configs until these have been completed
# The loopback network interface auto lo iface lo inet loopback
# PFSense-Linux Handoff auto ens161 allow-hotplug ens161 iface ens161 inet static address 192.168.0.1 netmask 255.255.255.0 network 192.168.0.0 broadcast 192.168.0.255 up /bin/ip route add 220.127.116.11/29 via 192.168.0.2 dev ens161 iface ens161 inet6 static address 2001:db8:0::1 netmask 64 up ip -6 route add 2001:db8::/48 via 2001:db8:0::2 dev ens161 # Linux - 4G Router Handoff auto ens224 allow-hotplug ens224 iface ens224 inet static address 192.168.8.3 netmask 255.255.255.0 network 192.168.8.0 broadcast 192.168.8.255 up /bin/ip route add 18.104.22.168/32 via 192.168.8.1 dev ens224 # Bring up ens256 (AAISP PPPOE) auto ens256 iface ens256 inet manual # Bring up AAISP PPPOE auto aaisp iface aaisp inet ppp provider aaisp pre-up /sbin/ip link set ens256 up dns-nameservers 2001:8b0::2020 2001:8b0::2021
Setting up pppd for PPPoE (DSL)
First I built the PPPoE config in order to connect via DSL (Primary path)
pppd uses several different configuration files:
- /etc/ppp/options - default settings for pppd
- /etc/ppp/peers/aaisp - the configuration file for settings specific to connecting to A&A
- /etc/ppp/chap-secrets - the location where your A&A line password is stored
/etc/ppp/options should be left as-is - we will not change this file.
This file contains the settings that are used to configure your connection to A&A:
user your-username-here plugin rp-pppoe.so ens256 noipdefault defaultroute hide-password lcp-echo-interval 1 lcp-echo-failure 4 noauth persist maxfail 0 mtu 1492 noaccomp default-asyncmap +ipv6 ipv6cp-use-ipaddr ifname pppoe-aaisp
Each line in this file sets a different setting:
- user your-username-here - this line sets the username that pppd will use to log in. Replace "your-username-here" with your A&A line username. It will be in the form "firstname.lastname@example.org"
- plugin rp-pppoe.so ens256 - tells pppd to load the PPPoE plugin, and to use the network interface "ens256" to connect
- noipdefault - tells pppd not to try and guess an IP to use, but instead to use the IP explicitly given by A&A
- defaultroute - automatically set the PPP connection as your default route (for IPv4 only)
- hide-password - hides your password when logging authentication packets
- lcp-echo-interval 1 - send a LCP echo to A&A once every second
- lcp-echo-failure 4 - automatically drop the connection after 4 failed LCP echoes (This is slightly faster than the standard config, and seems to strike a reasonable balance)
- noauth - don't require A&A to send authentication details
- persist - automatically reconnect if the connection drops
- maxfail 0 - sets the number of consecutive failed connection attempts before pppd gives up. Setting this to 0 means that pppd will retry forever
- mtu 1492 - sets the max MTU for packets inside the PPP connection - 1492 is a "safe" value for PPPoE on most hardware. Some modems will be able to use "baby jumbo frames" (RFC 4638). See the "Using a full 1500 MTU" section for more details.
- noaccomp - disables address/control compression
- default-asyncmap - disables the negotiation of an asyncmap - forces all control characters to be escaped
- +ipv6 - enable IPv6 support
- ipv6cp-use-ipaddr - use your IPv4 address as the local identifier for IPv6CP
- ifname pppoe-aaisp - renames the PPP connection from an automatically generated name (such as ppp0) to pppoe-aaisp - this makes further configuration easier!
This file contains the password that is used to connect to A&A.
# Secrets for authentication using CHAP # client server secret IP addresses email@example.com * YourLinePasswordGoesHere
Replace "YourLinePasswordGoesHere" with the password for your A&A connection.
Making IPv6 work with pppd
Out of the box, you'll notice that you can't access the internet using IPv6.
This is because pppd doesn't create a default route for IPv6. We can force it to do this by creating another file.
Create /etc/ppp/ipv6-up.d/0000-defaultroute, and enter the following contents:
#!/bin/bash /sbin/ip -6 route add default dev $1
chmod 755 /etc/ppp/ipv6-up.d/0000-defaultroute
This file will now be run every time your PPP connects, and will automatically create an IPv6 default route!
Before you proceed, you should test your ppp configuration.
pppoe -I eth0 -A
This should produce some output similar to the following:
Access-Concentrator: acc-aln1.ry Got a cookie: 79 f0 19 2c d3 ec ae 4b 04 75 ee 8a 30 76 a6 ea AC-Ethernet-Address: a0:f3:e4:34:5f:8f
If something is wrong, you will probably get:
pppoe: Timeout waiting for PADO packets
If you get this error message, check your configuration matches the examples above. If you're still stuck, contact A&A support.
Now try to actually connect:
pon aaisp tail -n 20 /var/log/messages
This should produce output like the following:
Jul 15 22:05:45 router pppd: Plugin rp-pppoe.so loaded. Jul 15 22:05:45 router pppd: pppd 2.4.6 started by thomas, uid 0 Jul 15 22:05:45 router pppd: PPP session is 522 Jul 15 22:05:45 router pppd: Connected to 00:03:97:1c:80:02 via interface eth0 Jul 15 22:05:45 router pppd: Renamed interface ppp0 to pppoe-aaisp Jul 15 22:05:45 router pppd: Using interface pppoe-aaisp Jul 15 22:05:45 router pppd: Connect: pppoe-aaisp <--> eth0 Jul 15 22:06:32 router pppd: CHAP authentication succeeded Jul 15 22:06:32 router pppd: CHAP authentication succeeded Jul 15 22:06:32 router pppd: peer from calling number 00:03:97:1C:80:02 authorized Jul 15 22:06:32 router pppd: local IP address <your WAN IP address here> Jul 15 22:06:32 router pppd: remote IP address 22.214.171.124 Jul 15 22:06:32 router pppd: local LL address fe80::5893:5ee6:a435:8672 Jul 15 22:06:32 router pppd: remote LL address fe80::0203:97ff:fe19:8000
If it does, then your pppd configuration works perfectly! You should test some IPv4 and IPv6 services to make sure everything is working (ipv4.google.com and ipv6.google.com are a good choice) Run the following to disconnect, and do the rest of the configuration:
Enabling IP forwarding
To tell our Linux router to actually forward traffic, you must first enable IP forwarding in /etc/sysctl.conf.
Look for this section in /etc/sysctl.conf:
# Uncomment the next line to enable packet forwarding for IPv4 #net.ipv4.ip_forward=1 # Uncomment the next line to enable packet forwarding for IPv6 # Enabling this option disables Stateless Address Autoconfiguration # based on Router Advertisements for this host #net.ipv6.conf.all.forwarding=1
Uncomment the two lines starting with "net":
# Uncomment the next line to enable packet forwarding for IPv4 net.ipv4.ip_forward=1 # Uncomment the next line to enable packet forwarding for IPv6 # Enabling this option disables Stateless Address Autoconfiguration # based on Router Advertisements for this host net.ipv6.conf.all.forwarding=1
This will reload /etc/sysctl.conf - applying our changes.
PFSense (Or any alternative firewall) now needs two or three pieces of configuration
- Instead of using PPPoE directly, it should now be configured with a network interface with a static IP (192.168.0.2 in this example) and a static gateway (192.168.0.1). I also disabled "Block RFC1918", although looking back I'm not sure if this is actually needed - we should see no inbound connections using the handoff address from this interface...
- PFSense will, by default perform IPv4 natting using the interface addresses. If you don't reconfigure this, then you'll try to connect to the internet using private addresses, and won't get very far. You need to configure NAT to use your /32 or one of your /29 addresses for all outbound traffic - you can do this using Firewall -> NAT -> Manual
- PFSense might also use the pppoe interface address for inbound rules, in which case you'll need to update the rules (And probably anything pointing to them, as the PPPoE public IP address now belongs to your Linux router)
If correctly configured, you should be able to connect to the internet from machines behind the firewall. If not, you should probably ping at various points along the chain and see where the problem lies. Don't forget to test both IPv4 and IPv6. The following commands may be of help
ip route ip -6 route ping 126.96.36.199 ping6 2001:4860:4860::8888 tcpdump -ni ens<n>
Now time to do what you're actually here for, configure L2TP. Debian Buster has all the kernel models you need, and you installed xl2tpd earlier, so you should be good to configure it now. There are a couple of modifications from [L2TP_Client], namely use of the IP address rather than FQDN - you won't be able to resolve the address when the PPPoE interface is down, and you need to route this address to your backup path anyway, so even if DNS were to allow an update, IP routing wouldn't... I've also used the ifname command, which provides more consistent naming than ppp<n>.
- Install xl2tpd and pppd on your Linux router.
/etc/xl2tpd/xl2tpd.confto contain the following:
lns = 188.8.131.52
require authentication = no
pppoptfile = /etc/ppp/options.aaisp
/etc/ppp/options.aaispcontaining the following (obviously change the name and password to match your L2TP login details):
noauth ifname ppp-aaisp-l2tp
- Create the xl2tpd control file:
mkdir -p /var/run/xl2tpd
- Start the xl2tpd service (for systemd, use service command for older RC systems):
systemctl start xl2tpd
- Tell the daemon to connect to aaisp:
echo "c aaisp" > /var/run/xl2tpd/l2tp-control
This should give you a new PPP device which encapsulates the L2TP connection. You'll likely find that the packets are being sent up the DSL line (because that's still linux's default route) and return via the L2TP tunnel (Because AAISP route down the L2TP tunnel in preference, if the L2TP is up). Don't worry for the moment, we'll be doing clever stuff with routing shortly.
PPP IF Up/Down scripts
Scripting brings all of this configuration together. Both PPPoE and L2TP are managed by pppd, and pppd fires scripts when interfaces go up and down. So we just need to add some scripts... Technically there is an IPv4 and an IPv6 script which executes, each has parameters specific to the network stack, but we only care about the interface names, so can use one script for both stacks.
Let's modify the /etc/ppp/ipv6-up.d/0000-defaultroute script we created earlier for PPPoE
#!/bin/bash #/sbin/ip -6 route add default dev $1 /bin/logger $1 is up if [ $1 = "pppoe-aaisp" ]; then /bin/logger "AAISP PPPoE online: Shutting down L2TP and deleting routes" echo "d aaisp" > /var/run/xl2tpd/l2tp-control /sbin/ip route del default dev ppp-aaisp-l2tp scope link /sbin/ip -6 route del default dev ppp-aaisp-l2tp scope link /bin/logger "AAISP PPPoE online: Adding routes" /sbin/ip route add default dev pppoe-aaisp scope link /sbin/ip -6 route add default dev pppoe-aaisp scope link fi if [ $1 = "ppp-aaisp-l2tp" ]; then /bin/logger "AAISP over L2TP circuit is online; adding routes" /sbin/ip route add default dev ppp-aaisp-l2tp scope link /sbin/ip -6 route add default dev ppp-aaisp-l2tp scope link fi
and create a second script called /etc/ppp/ipv6-down.d/0000-defaultroute - don't forget to chmod 755
#!/bin/bash /bin/logger $1 is down if [ $1 = "pppoe-aaisp" ]; then /bin/logger "AAISP PPPoE is offline! Initiating reundant link!" echo "c aaisp" > /var/run/xl2tpd/l2tp-control /bin/logger "AAISP down: Removing routes" /sbin/ip route del default dev pppoe-aaisp scope link /sbin/ip -6 route del default dev pppoe-aaisp scope link fi if [ $1 = "ppp-aaisp-l2tp" ]; then /bin/logger "AAISP over L2TP circuit is offline; removing routes" /sbin/ip route del default dev ppp-aaisp-l2tp scope link /sbin/ip -6 route del default dev ppp-aaisp-l2tp scope link fi
Your linux router will get the line /32 address, and sshd will listen on 0.0.0.0, hence will accept ssh connections from the internet. You'll find very quickly /var/log/auth.log fills with automated password attempts. I recommend altering /etc/ssh/sshd_config - either restrict it to binding on the internal IP only
ListenAddress 192.168.0.1 ListenAddress 2001:db8:0::1
or configure for ssh key authentication with no password fallback.
Now it's time to test. I recommend pulling the lan connection to your DSL model in order to test that failover works. Other than the latency for your connection going up significantly (at least for 4g failover), you should see able 7 seconds of disruption when the link goes down, and 3 seconds when the link comes up. Don't forget to do a reboot of the linux box and check everything comes up correctly.
You'll find the log files in /var/log/messages