Jump to content

This is the support site for Andrews & Arnold Ltd, a UK Internet provider. Information on these pages is generally for our customers but may be useful to others, enjoy!

TalkTalk and DSCP and 19 second latency: Difference between revisions

← Older edit
Content deleted Content added
AA-Andrew (talk | contribs)
autoreview, Bureaucrats, editor, Interface administrators, reviewer, Administrators
12,629 edits
AA-Andrew (talk | contribs)
autoreview, Bureaucrats, editor, Interface administrators, reviewer, Administrators
12,629 edits
mNo edit summary
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
<indicator name="Front">[[File:Menu-document.svg|link=Category:Technical Documents|30px|Back up to the Technical Documents category]]</indicator>
<indicator name="Front">[[File:Menu-document.svg|link=Category:Technical Documents|30px|Back up to the Technical Documents category]]</indicator>

'''Updates:'''
2021-12-07 Page created
2021-12-08 Updated as FireBrick has software update to change/remove DSCP field
2021-12-09 Updated with reply from TalkTalk
2021-12-13 Updated with reply from Aruba
Further updates from TalkTalk expected middle of the week
Further updates from TalkTalk expected in the new year
2022-02-01 Update from TalkTalk (below) - saying they have fixed it, but our pings still show latency...
2022-02-24 TalkTAlk confirm that further investigation is underway and we've given details of a staff line to test with


This page is about in interesting problem that was reported to us, by a customer, in December 2021. We have written this up so that this page can be found if other people are seeing a similar problem.
This page is about in interesting problem that was reported to us, by a customer, in December 2021. We have written this up so that this page can be found if other people are seeing a similar problem.
Line 9: Line 19:


1408 bytes from x.x.x.x: icmp_seq=986 ttl=63 time='''19084.589 ms'''
1408 bytes from x.x.x.x: icmp_seq=986 ttl=63 time='''19084.589 ms'''



===Video:===
===Video:===
Line 117: Line 128:
* Using other wireless devices (I can repro the problem with the "live view" of some Nest Cameras and with web-based Stadia on a Chromebook)
* Using other wireless devices (I can repro the problem with the "live view" of some Nest Cameras and with web-based Stadia on a Chromebook)
* Dumping packets on the WAN interface of the FB2900 (I've confirmed that the FB2900 itself isn't introducing the extra latency)
* Dumping packets on the WAN interface of the FB2900 (I've confirmed that the FB2900 itself isn't introducing the extra latency)
*Swapping the modem from a HG612 to a Technicolor in bridge mode


===...that did make a difference===
===...that did make a difference===
* Connecting the phone, running Stadia, via wired Ethernet (high latency goes away because the problematic QoS marking has gone, DSCP field = 0)
* Connecting the phone, running Stadia, via '''wired Ethernet''' (high latency goes away because the problematic QoS marking has gone, DSCP field = 0)
* Setting a special feature on AAISP and the FireBrick - 'IP over LCP' - this sends the IP traffic as control frames. (high latency goes away). This IPoLCP is a niche feature and has been used in the past to help diagnose problems in back-haul networks: eg: https://www.revk.uk/2015/02/congestion-case-study.html
* Setting a special feature on AAISP and the FireBrick - ''''IP over LCP'''' - this sends the IP traffic as control frames. (high latency goes away). This IPoLCP is a niche feature and has been used in the past to help diagnose problems in back-haul networks: eg: https://www.revk.uk/2015/02/congestion-case-study.html
* Changing the DSCP value - only packets marked CS6 (192 to 195) are affected. Using values higher or lower and the latency goes away
* '''Changing the DSCP value''' - only packets marked CS6 (192 to 195) are affected. Using values higher or lower and the latency goes away
⚫
*'''Getting the FireBrick 2900 to set the DSCP field to 0''' - this feature was added to the alpha release of FireBrick software on 8th December 2021


=== Things that were not tried ===
=== Things that were not tried ===
*Migrating the line to BT back-haul - this would have fixed the problem for the customer, but would not have fixed the problem in the TalkTalk network or the Aruba access point. Being engineers - we like to fix problems!
*Migrating the line to BT back-haul - this would have fixed the problem for the customer, but would not have fixed the problem in the TalkTalk network or the Aruba access point. Being engineers - we like to fix problems!
*Changing the DSCP setting on the Aruba Instant-On Access Points - there is no setting, the DSCP field is being set automatically.
*Changing the DSCP setting on the Aruba Instant-On Access Points - there is no setting, the DSCP field is being set automatically.

⚫
*Getting the FireBrick 2900 to strip the DSCP field - this feature hasn't been added to the FireBrick. ...yet.


== Further tests ==
== Further tests ==
With A&A having a lively IRC channel, we asked customers to try our ping test to see who far spread the problem was, we found out that:
With A&A having a lively IRC channel, we asked customers to try our ping test to see how far spread the problem was, we found out that:
* All AAISP TalkTalk VDSL lines tested showed latency
* All AAISP TalkTalk VDSL lines tested showed latency
* All AAISP TalkTalk ADSL lines tested showed latency
* All AAISP TalkTalk ADSL lines tested showed latency
Line 135: Line 148:
* No AAISP Ethernet lines showed latency
* No AAISP Ethernet lines showed latency


We were also able to test on non-AAISP TalkTalk lines:
Further, we were also able to test on non-AAISP TalkTalk lines:


'''Another TalkTalk partner, like us:'''
'''Another TalkTalk partner, like us:'''
Line 148: Line 161:
round-trip min/avg/max/std-dev = 17.830/33807.255/48113.126/15886.310 ms
round-trip min/avg/max/std-dev = 17.830/33807.255/48113.126/15886.310 ms


So, seems this is a problem within TalkTalks's UK network, probably affecting all TalkTalk ADSL and VDSL lines in the UK.
So, seems this is a problem within TalkTalks's network, probably affecting all TalkTalk ADSL and VDSL lines in the UK.


=Deep Packet Inspection Concerns=
=Deep Packet Inspection Concerns=
Line 164: Line 177:


There are seemingly two faults here:
There are seemingly two faults here:
# Aruba adding the DSCP field - which is trying to be helpful, it's not configurable, and so in this case it's an unhelpful feature - if it used the DSCP value intended for 'voice' then we'd not have this problem.
# Aruba adding the DSCP field - which whilst trying to be helpful is seemingly not configurable, and so in this case it's an unhelpful feature - if it used the DSCP value intended for 'voice' then we'd not have this problem. The customer has opened a support query with Aruba regarding this - Aruba's reply is below.
# (in our opinion) TalkTalk should not be looking at the DSCP field
# (in our opinion) TalkTalk should not be looking at the DSCP field. AAISP are taking this up with TalkTalk.

The customer has opened a support query with Aruba regarding this. AAISP are taking this up with TalkTalk.


==Fault raised with TalkTalk==
==Fault raised with TalkTalk==


===December 7th===
===December 7th 2021 ===
A&A got in touch with TalkTalk directly by emailing TalkTalk's escalations department and our Service Manager. (There was no point in reporting an individual line fault via the normal channels for broadband fault.)
A&A got in touch with TalkTalk directly by emailing TalkTalk's escalations department and our Service Manager. (There was no point in reporting an individual line fault via the normal channels for broadband fault.)

===December 9th===
TalkTalk are still investigating and are hoping to get back to us next week. They are assuring us on the point about packet inspection, that their policy remains the same in that they are not inspecting traffic in any way and that nor do they have the means to do so.

=== February 1st 2022 ===

Update from TalkTalk (below) - saying they have fixed it, but our pings still show latency...


''
The core engineering team have investigated and found that the config for the CoS classifier required adjustment - after discussing and planning the steps, a planned engineering work was scheduled, due to our change freeze it took a little longer to put in place, this has now completed and they request if you can kindly re-test and keep us updated.''

sudo ping -i 0.02 -Q 192 -s 1400 -c 200 X.X.X.X
PING X.X.X.X (X.X.X.X) 1400(1428) bytes of data.
1408 bytes from X.X.X.X: icmp_seq=1 ttl=63 time=8.25 ms
...
1408 bytes from 81.187.81.187: icmp_seq=200 ttl=63 time='''3084 ms'''
--- X.X.X.X ping statistics ---
200 packets transmitted, 200 received, 0% packet loss, time 675ms
rtt min/avg/max/mdev = 8.001/1466.857/3145.106/924.943 ms, pipe 70





'''To be continued....'''
'''To be continued....'''

==Ticket open with Aruba==
Apparently These Aruba Access Points do have the ability to open a CLI on the device and disable DSCP.
However, it's not so simple and Aruba advise against it because the CLI requires an interactively-generated token
from their support staff, and changing the setting back would require another support call.

Current Solution: Customer is currently using the FireBrick feature to set the DSCP field to 0.
Retrieved from "http://support.aa.net.uk/TalkTalk_and_DSCP_and_19_second_latency"