Linux upload bonding using multipath routing

This is from a post on the newsgroup:

On 1 Jun 2011, Nix uttered the following: > > I think I can figure out if the PPP line is up by telnetting in, doing > > 'ip ifconfig', and looking for my own IP address in its 'wanif0' section > > (or, to be lazy, just looking for my own IP address anywhere in its > > output). > > > > What a kludge, though. Don't need any of that. It's installed and working, at last. The software changes took only half an hour to get right, nothing beside the month and a half it took to get the second line installed (thank *you* BT).

What I did:

Getting routing working was easy. Make sure packets for your public IP addresses (other than the two specific to each router) are being delivered to both lines in the control pages (which they are by default). Obviously, assign a reachable IP address to the LAN side of your router (if you use src routing it needn't be a public one, but that's a bit tricksy to set up) in the routers' configuration pages.

Assign a public IP address to your firewall's outgoing network interfaces to each router: you can use the same address if you like (I think), though it affects very little:

 ip addr add 81.187.191.129/29 dev adsl
 ip addr add 81.187.191.133/29 dev bdsl

Set up host routes pointing to the routers' IP addresses:

 ip route add to 81.187.191.130 dev adsl
 ip route add to 81.187.191.132 dev bdsl

Then set up a multipath default route:

 ip route add default nexthop via 81.187.191.130 dev adsl \
                      nexthop via 81.187.191.132 dev bdsl

You can use the 'weight' parameter to transmit different amounts of packets down each if they are different speeds:

 ip route add default nexthop via 81.187.191.130 dev adsl weight 9 \
                      nexthop via 81.187.191.132 dev bdsl weight 10

will transmit 10% more traffic down 'bdsl' than down 'adsl'.

As the command suggests, this works at the routing table level, so an outbound connection to a given host will tend to use one ADSL link until its routing table entry expires (controlled by /proc/sys/net/ipv*/route/gc_* parameters: see e.g. <http://mailman.ds9a.nl/pipermail/lartc/2002q4/005296.html> for more).

But that's not quite enough. If an ADSL link goes down but the Ethernet connection remains up (which is by far the most common case, of course: 1m of network cabling has much less chance of failure than the link all the way to AAISP), the kernel will not notice, and will keep hurling outgoing connections at the dead route: so your incoming connections keep working, but half your outgoing ones stall forever.

Julian Anastasov has kernel patches at <http://www.ssi.bg/~ja/#routes> that fix this. Unfortunately the rollup patch http://www.ssi.bg/~ja/routes-*.diff causes utter disaster, thousands and thousands of

Jun 20 20:04:18 fold warning: : [ 262.593396] martian source 192.168.16.20 from 192.41.162.30, on dev adsl Jun 20 20:04:18 fold warning: : [ 262.633643] ll header: 00:00:24:cb:c6:a1:00:23:f8:95:b7:bf:08:00

and an effectively-dead connection. This is because the 'improved' rp_filtering in this patch makes it illegal for a packet which left via gateway A of a multipath route to get a reply via gateway B (which is precisely what AAISP will do, of course, though it probably is invalid for the common case of two connections via different ISPs with distinct IP addresses). So with the rollup patch installed, pretty much every other packet is dropped!

You can fix this by installing only the first two patches of the series, i.e. 00_static_routes*.diff and 01_alt_routes*.diff, and *not* installing the third, 05_nf_reroute*.diff. No router configuration changes are needed: all that this code needs is for the router to hand back host/net unreachable errors on routing failure (which by default, when the link goes down, the Zyxels do).

With that, it works! If a link goes down, existing connections fall over to the other one, and new connections do as well, after a delay. When the link comes back up, new connections start to use the new link after the route cache expires (normally in between 30 and 300s). (Existing connections will use it only if they have an idle period long enough that their route cache entry expires as well. This is normal networking behaviour.)

-- NULL && (void)