DSL lines are not perfect, and line resyncs are guaranteed to happen at the most inconvenient moment. Fortunately AAISP provide L2TP for subscribers, but the L2TP takes routing precedence over your DSL line - you therefore can't simply leave your L2TP up and running as a fallback. This means you need to script the process of bringing this interface up upon PPPoE failure.

I investigated PFSense (My current firewall solution) and RouterOS, but neither had very satisfactory scripting abilities. Linux, on the other hand, had no such problem.

In this solution, I have chosen to place a Linux solution acting only as a router in front of my PFSense box. This kept a fairly nice GUI for day to day management, but provided it with a more robust way to connect to AAISP via a Linux router, all of which is hosted on an ESXi box. The obvious cost is an extra hop, which requires a bit of extra CPU and causes a bit of extra latency. When measured I found PFSense to need around 10x the CPU of the linux box (Although there will also be a little extra vSwitch load - SR-IOV is supported if you're desperate to avoid this) and an extra 1-2ms latency.

Throughout this guide, I'll likely refer to your PFSense box. While a few config tweaks will be needed, the guide should apply to more or less any firewall.

If you wanted to use your linux box as a full firewall, you would likely have to script updates to the firewall rules based on interface up / down. I have not looked into this.

This guide provides an example configuration for Debian Buster.

Prerequsites

a Linux machine with Debian Buster already installed and three network interfaces - one for the connection to the modem, one for the connection to the alternative path router, and one for the connection to your LAN. The machine should be ready for internet facing duties, that means strong password!
a ADSL or FTTC modem, or a fibre ONT (for FTTP) (as appropriate for your connection)
an alternative path router of some type. In my case I am using a Huawei B315 4G router (With ethernet)
a firewall which is already configured and working
at least an extra IPv4/32 routed down your line, otherwise you'll struggle a little with routing - you could either look to see if a bridge configuration is possible, double NAT, or moving the NAT boundary from the firewall to the router. In my example I have a /29, so this isn't an issue

A note on routing

IPv6 routing is trivial in this case, I've simply used 1/65536th of the /48 we're all given for a /64 handoff within the public address range. There are two options for IPv4 ranges. You can either assign a /29 to the handoff interface between the Linux router and the firewall. This would be the "Proper" way to do things, but would waste 3 public IP addresses, which are a precious resource in this day and age. I have instead used RFC1918 private address space as a handoff between the devices, and routed the public /29 block to the private IP of the firewall. The firewall can then be configured with these as virtual addresses, with all 8 addresses kept as usable addresses.

Assumptions and example setup

IPv4 routing setup used in this example

IPv6 routing setup used in this example

ens256 is plugged directly into your modem or ONT
ens161 will be attached to your PFsense firewall
ens224 will be attached to your alternative

As the names change from installation to installation, I strongly advise you to copy this wiki page, then do a find and replace. You will end up with extra troubleshooting otherwise!

Linux Setup

Before you do anything, you'll probably want to connect your linux box to the internet, update it and make sure it has all the tools you need

apt-get update
apt-get install ppp pppoe xl2tpd iproute2 tcpdump net-tools

You may also want open-vm-tools, if you're virtualised

Build network interface configuration

Edit /etc/network/interfaces. This was my final config, you may want to leave the up and pppoe configs until these have been completed

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# PFSense-Linux Handoff
auto ens161
allow-hotplug ens161
iface ens161 inet static
       address 192.168.0.1
       netmask 255.255.255.0
       network 192.168.0.0
       broadcast 192.168.0.255
       up /bin/ip route add 123.123.123.123/29 via 192.168.0.2 dev ens161
iface ens161 inet6 static
       address 2001:db8:0::1
       netmask 64
       up ip -6 route add 2001:db8::/48 via 2001:db8:0::2 dev ens161
# Linux - 4G Router Handoff
auto ens224
allow-hotplug ens224
iface ens224 inet static
       address 192.168.8.3
       netmask 255.255.255.0
       network 192.168.8.0
       broadcast 192.168.8.255
       up /bin/ip route add 90.155.53.19/32 via 192.168.8.1 dev ens224
# Bring up ens256 (AAISP PPPOE)
auto ens256
iface ens256 inet manual
# Bring up AAISP PPPOE
auto aaisp
iface aaisp inet ppp
        provider aaisp
        pre-up /sbin/ip link set ens256 up
        dns-nameservers 2001:8b0::2020 2001:8b0::2021

Setting up pppd for PPPoE (DSL)

First I built the PPPoE config in order to connect via DSL (Primary path)

pppd uses several different configuration files:

/etc/ppp/options - default settings for pppd
/etc/ppp/peers/aaisp - the configuration file for settings specific to connecting to A&A
/etc/ppp/chap-secrets - the location where your A&A line password is stored

/etc/ppp/options should be left as-is - we will not change this file.

/etc/ppp/peers/aaisp

This file contains the settings that are used to configure your connection to A&A:

user your-username-here
plugin rp-pppoe.so ens256
noipdefault
defaultroute
hide-password
lcp-echo-interval 1
lcp-echo-failure 4
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap
+ipv6
ipv6cp-use-ipaddr
ifname pppoe-aaisp

Each line in this file sets a different setting:

user your-username-here - this line sets the username that pppd will use to log in. Replace "your-username-here" with your A&A line username. It will be in the form "example@a.1"
plugin rp-pppoe.so ens256 - tells pppd to load the PPPoE plugin, and to use the network interface "ens256" to connect
noipdefault - tells pppd not to try and guess an IP to use, but instead to use the IP explicitly given by A&A
defaultroute - automatically set the PPP connection as your default route (for IPv4 only)
hide-password - hides your password when logging authentication packets
lcp-echo-interval 1 - send a LCP echo to A&A once every second
lcp-echo-failure 4 - automatically drop the connection after 4 failed LCP echoes (This is slightly faster than the standard config, and seems to strike a reasonable balance)
noauth - don't require A&A to send authentication details
persist - automatically reconnect if the connection drops
maxfail 0 - sets the number of consecutive failed connection attempts before pppd gives up. Setting this to 0 means that pppd will retry forever
mtu 1492 - sets the max MTU for packets inside the PPP connection - 1492 is a "safe" value for PPPoE on most hardware. Some modems will be able to use "baby jumbo frames" (RFC 4638). See the "Using a full 1500 MTU" section for more details.
noaccomp - disables address/control compression
default-asyncmap - disables the negotiation of an asyncmap - forces all control characters to be escaped
+ipv6 - enable IPv6 support
ipv6cp-use-ipaddr - use your IPv4 address as the local identifier for IPv6CP
ifname pppoe-aaisp - renames the PPP connection from an automatically generated name (such as ppp0) to pppoe-aaisp - this makes further configuration easier!

/etc/ppp/chap-secrets

This file contains the password that is used to connect to A&A.

# Secrets for authentication using CHAP
# client      server   secret                      IP addresses
example@a.1   *        YourLinePasswordGoesHere

Replace "YourLinePasswordGoesHere" with the password for your A&A connection.

Making IPv6 work with pppd

Out of the box, you'll notice that you can't access the internet using IPv6.

This is because pppd doesn't create a default route for IPv6. We can force it to do this by creating another file.

Create /etc/ppp/ipv6-up.d/0000-defaultroute, and enter the following contents:

#!/bin/bash
/sbin/ip -6 route add default dev $1

Now run:

chmod 755 /etc/ppp/ipv6-up.d/0000-defaultroute

This file will now be run every time your PPP connects, and will automatically create an IPv6 default route!

Testing pppd

Before you proceed, you should test your ppp configuration.

Firstly, run:

pppoe -I eth0 -A

This should produce some output similar to the following:

Access-Concentrator: acc-aln1.ry
Got a cookie: 79 f0 19 2c d3 ec ae 4b 04 75 ee 8a 30 76 a6 ea
AC-Ethernet-Address: a0:f3:e4:34:5f:8f

If something is wrong, you will probably get:

pppoe: Timeout waiting for PADO packets

If you get this error message, check your configuration matches the examples above. If you're still stuck, contact A&A support.

Now try to actually connect:

pon aaisp
tail -n 20 /var/log/messages

This should produce output like the following:

Jul 15 22:05:45 router pppd[23049]: Plugin rp-pppoe.so loaded.
Jul 15 22:05:45 router pppd[23050]: pppd 2.4.6 started by thomas, uid 0
Jul 15 22:05:45 router pppd[23050]: PPP session is 522
Jul 15 22:05:45 router pppd[23050]: Connected to 00:03:97:1c:80:02 via interface eth0
Jul 15 22:05:45 router pppd[23050]: Renamed interface ppp0 to pppoe-aaisp
Jul 15 22:05:45 router pppd[23050]: Using interface pppoe-aaisp
Jul 15 22:05:45 router pppd[23050]: Connect: pppoe-aaisp <--> eth0
Jul 15 22:06:32 router pppd[23050]: CHAP authentication succeeded
Jul 15 22:06:32 router pppd[23050]: CHAP authentication succeeded
Jul 15 22:06:32 router pppd[23050]: peer from calling number 00:03:97:1C:80:02 authorized
Jul 15 22:06:32 router pppd[23050]: local  IP address <your WAN IP address here>
Jul 15 22:06:32 router pppd[23050]: remote IP address 81.187.81.187
Jul 15 22:06:32 router pppd[23050]: local  LL address fe80::5893:5ee6:a435:8672
Jul 15 22:06:32 router pppd[23050]: remote LL address fe80::0203:97ff:fe19:8000

If it does, then your pppd configuration works perfectly! You should test some IPv4 and IPv6 services to make sure everything is working (ipv4.google.com and ipv6.google.com are a good choice) Run the following to disconnect, and do the rest of the configuration:

poff aaisp

Enabling IP forwarding

To tell our Linux router to actually forward traffic, you must first enable IP forwarding in /etc/sysctl.conf.

Look for this section in /etc/sysctl.conf:

# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1

Uncomment the two lines starting with "net":

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
net.ipv6.conf.all.forwarding=1

Now run:

sysctl -p

This will reload /etc/sysctl.conf - applying our changes.

Configuring PFSense

PFSense (Or any alternative firewall) now needs two or three pieces of configuration

Instead of using PPPoE directly, it should now be configured with a network interface with a static IP (192.168.0.2 in this example) and a static gateway (192.168.0.1). I also disabled "Block RFC1918", although looking back I'm not sure if this is actually needed - we should see no inbound connections using the handoff address from this interface...
PFSense will, by default perform IPv4 natting using the interface addresses. If you don't reconfigure this, then you'll try to connect to the internet using private addresses, and won't get very far. You need to configure NAT to use your /32 or one of your /29 addresses for all outbound traffic - you can do this using Firewall -> NAT -> Manual
PFSense might also use the pppoe interface address for inbound rules, in which case you'll need to update the rules (And probably anything pointing to them, as the PPPoE public IP address now belongs to your Linux router)

Testing

If correctly configured, you should be able to connect to the internet from machines behind the firewall. If not, you should probably ping at various points along the chain and see where the problem lies. Don't forget to test both IPv4 and IPv6. The following commands may be of help

ip route
ip -6 route
ping 8.8.8.8
ping6 2001:4860:4860::8888
tcpdump -ni ens<n>

L2TP

Now time to do what you're actually here for, configure L2TP. Debian Buster has all the kernel models you need, and you installed xl2tpd earlier, so you should be good to configure it now. There are a couple of modifications from [L2TP_Client], namely use of the IP address rather than FQDN - you won't be able to resolve the address when the PPPoE interface is down, and you need to route this address to your backup path anyway, so even if DNS were to allow an update, IP routing wouldn't... I've also used the ifname command, which provides more consistent naming than ppp<n>.

Install xl2tpd and pppd on your Linux router.
Edit /etc/xl2tpd/xl2tpd.conf to contain the following:
[lac aaisp] lns = 90.155.53.19 require authentication = no pppoptfile = /etc/ppp/options.aaisp
Create /etc/ppp/options.aaisp containing the following (obviously change the name and password to match your L2TP login details):
+ipv6 ipv6cp-use-ipaddr name xyz@a.X password Your_xyz@A.X_password noauth ifname ppp-aaisp-l2tp
Create the xl2tpd control file:
mkdir -p /var/run/xl2tpd touch /var/run/xl2tpd/l2tp-control
Start the xl2tpd service (for systemd, use service command for older RC systems):
systemctl start xl2tpd
Tell the daemon to connect to aaisp:
echo "c aaisp" > /var/run/xl2tpd/l2tp-control

This should give you a new PPP device which encapsulates the L2TP connection. You'll likely find that the packets are being sent up the DSL line (because that's still linux's default route) and return via the L2TP tunnel (Because AAISP route down the L2TP tunnel in preference, if the L2TP is up). Don't worry for the moment, we'll be doing clever stuff with routing shortly.

PPP IF Up/Down scripts

Scripting brings all of this configuration together. Both PPPoE and L2TP are managed by pppd, and pppd fires scripts when interfaces go up and down. So we just need to add some scripts... Technically there is an IPv4 and an IPv6 script which executes, each has parameters specific to the network stack, but we only care about the interface names, so can use one script for both stacks.

Let's modify the /etc/ppp/ipv6-up.d/0000-defaultroute script we created earlier for PPPoE

#!/bin/bash
#/sbin/ip -6 route add default dev $1
/bin/logger $1 is up
if [ $1 = "pppoe-aaisp" ]; then
       /bin/logger "AAISP PPPoE online: Shutting down L2TP and deleting routes"
       echo "d aaisp" > /var/run/xl2tpd/l2tp-control
       /sbin/ip route del default dev ppp-aaisp-l2tp scope link
       /sbin/ip -6 route del default dev ppp-aaisp-l2tp scope link
       /bin/logger "AAISP PPPoE online: Adding routes"
       /sbin/ip route add default dev pppoe-aaisp scope link
       /sbin/ip -6 route add default dev pppoe-aaisp scope link
fi
if [ $1 = "ppp-aaisp-l2tp" ]; then
       /bin/logger "AAISP over L2TP circuit is online; adding routes"
       /sbin/ip route add default dev ppp-aaisp-l2tp scope link
       /sbin/ip -6 route add default dev ppp-aaisp-l2tp scope link
fi

and create a second script called /etc/ppp/ipv6-down.d/0000-defaultroute - don't forget to chmod 755

#!/bin/bash
/bin/logger $1 is down
if [ $1 = "pppoe-aaisp" ]; then
       /bin/logger "AAISP PPPoE is offline! Initiating reundant link!"
       echo "c aaisp" > /var/run/xl2tpd/l2tp-control
       /bin/logger "AAISP down: Removing routes"
       /sbin/ip route del default dev pppoe-aaisp scope link
       /sbin/ip -6 route del default dev pppoe-aaisp scope link
fi
if [ $1 = "ppp-aaisp-l2tp" ]; then
       /bin/logger "AAISP over L2TP circuit is offline; removing routes"
       /sbin/ip route del default dev ppp-aaisp-l2tp scope link
       /sbin/ip -6 route del default dev ppp-aaisp-l2tp scope link
fi

Security

Your linux router will get the line /32 address, and sshd will listen on 0.0.0.0, hence will accept ssh connections from the internet. You'll find very quickly /var/log/auth.log fills with automated password attempts. I recommend altering /etc/ssh/sshd_config - either restrict it to binding on the internal IP only

ListenAddress 192.168.0.1
ListenAddress  2001:db8:0::1

or configure for ssh key authentication with no password fallback.

Finishing off

Now it's time to test. I recommend pulling the lan connection to your DSL model in order to test that failover works. Other than the latency for your connection going up significantly (at least for 4g failover), you should see able 7 seconds of disruption when the link goes down, and 3 seconds when the link comes up. Don't forget to do a reboot of the linux box and check everything comes up correctly.

You'll find the log files in /var/log/messages