Linux upload bonding using multipath routing

This is from a post on the newsgroup:

On 1 Jun 2011, Nix uttered the following:
> > I think I can figure out if the PPP line is up by telnetting in, doing
> > 'ip ifconfig', and looking for my own IP address in its 'wanif0' section
> > (or, to be lazy, just looking for my own IP address anywhere in its
> > output).
> >
> > What a kludge, though.
Don't need any of that. It's installed and working, at last. The
software changes took only half an hour to get right, nothing beside the
month and a half it took to get the second line installed (thank *you* BT).

What I did:

Getting routing working was easy. Make sure packets for your public IP
addresses (other than the two specific to each router) are being
delivered to both lines in the control pages (which they are by
default). Obviously, assign a reachable IP address to the LAN side of
your router (if you use src routing it needn't be a public one, but
that's a bit tricksy to set up) in the routers' configuration pages.

Assign a public IP address to your firewall's outgoing network
interfaces to each router: you can use the same address if you like (I
think), though it affects very little:

  ip addr add 81.187.191.129/29 dev adsl
  ip addr add 81.187.191.133/29 dev bdsl

Set up host routes pointing to the routers' IP addresses:

  ip route add to 81.187.191.130 dev adsl
  ip route add to 81.187.191.132 dev bdsl

Then set up a multipath default route:

  ip route add default nexthop via 81.187.191.130 dev adsl \
                       nexthop via 81.187.191.132 dev bdsl

You can use the 'weight' parameter to transmit different amounts of
packets down each if they are different speeds:

  ip route add default nexthop via 81.187.191.130 dev adsl weight 9 \
                       nexthop via 81.187.191.132 dev bdsl weight 10

will transmit 10% more traffic down 'bdsl' than down 'adsl'.

As the command suggests, this works at the routing table level, so an
outbound connection to a given host will tend to use one ADSL link until
its routing table entry expires (controlled by
/proc/sys/net/ipv*/route/gc_* parameters: see e.g.
<http://mailman.ds9a.nl/pipermail/lartc/2002q4/005296.html> for more).


But that's not quite enough. If an ADSL link goes down but the Ethernet
connection remains up (which is by far the most common case, of course:
1m of network cabling has much less chance of failure than the link all
the way to AAISP), the kernel will not notice, and will keep hurling
outgoing connections at the dead route: so your incoming connections
keep working, but half your outgoing ones stall forever.

Julian Anastasov has kernel patches at <http://www.ssi.bg/~ja/#routes>
that fix this. Unfortunately the rollup patch
http://www.ssi.bg/~ja/routes-*.diff causes utter disaster, thousands and
thousands of

Jun 20 20:04:18 fold warning: : [  262.593396] martian source 192.168.16.20 from 192.41.162.30, on dev adsl
Jun 20 20:04:18 fold warning: : [  262.633643] ll header: 00:00:24:cb:c6:a1:00:23:f8:95:b7:bf:08:00

and an effectively-dead connection. This is because the 'improved'
rp_filtering in this patch makes it illegal for a packet which left via
gateway A of a multipath route to get a reply via gateway B (which is
precisely what AAISP will do, of course, though it probably is invalid
for the common case of two connections via different ISPs with distinct
IP addresses). So with the rollup patch installed, pretty much every
other packet is dropped!

You can fix this by installing only the first two patches of the series,
i.e. 00_static_routes*.diff and 01_alt_routes*.diff, and *not*
installing the third, 05_nf_reroute*.diff. No router configuration
changes are needed: all that this code needs is for the router to hand
back host/net unreachable errors on routing failure (which by default,
when the link goes down, the Zyxels do).

With that, it works! If a link goes down, existing connections fall over
to the other one, and new connections do as well, after a delay. When
the link comes back up, new connections start to use the new link after
the route cache expires (normally in between 30 and 300s). (Existing
connections will use it only if they have an idle period long enough
that their route cache entry expires as well. This is normal networking
behaviour.)

-- NULL && (void)