TalkTalk and DSCP and 19 second latency: Difference between revisions

← Older edit

TalkTalk and DSCP and 19 second latency (view source)

Revision as of 11:46, 28 February 2022

2,314 bytes added , 28 February 2022

m

no edit summary

AA-Andrew

autoreview, Bureaucrats, editor, Interface administrators, reviewer, Administrators

12,270

edits

Revision as of 11:38, 8 December 2021 (view source) AA-Andrew (talk \| contribs) m (→‎...that did make a difference) ← Older edit		Latest revision as of 11:46, 28 February 2022 (view source) AA-Andrew (talk \| contribs) mNo edit summary
(14 intermediate revisions by the same user not shown)
<indicator name="Front">[[File:Menu-document.svg\|link=Category:Technical Documents\|30px\|Back up to the Technical Documents category]]</indicator> '''Updates:''' 2021-12-07 Page created 2021-12-08 Updated as FireBrick has software update to change/remove DSCP field 2021-12-09 Updated with reply from TalkTalk 2021-12-13 Updated with reply from Aruba Further updates from TalkTalk expected middle of the week Further updates from TalkTalk expected in the new year 2022-02-01 Update from TalkTalk (below) - saying they have fixed it, but our pings still show latency... 2022-02-24 TalkTAlk confirm that further investigation is underway and we've given details of a staff line to test with This page is about in interesting problem that was reported to us, by a customer, in December 2021. We have written this up so that this page can be found if other people are seeing a similar problem. 1408 bytes from x.x.x.x: icmp_seq=986 ttl=63 time='''19084.589 ms''' ===Video:=== * Using other wireless devices (I can repro the problem with the "live view" of some Nest Cameras and with web-based Stadia on a Chromebook) * Dumping packets on the WAN interface of the FB2900 (I've confirmed that the FB2900 itself isn't introducing the extra latency) Swapping the modem from a HG612 to a Technicolor in bridge mode ===...that did make a difference=== Setting a special feature on AAISP and the FireBrick - ''''IP over LCP'''' - this sends the IP traffic as control frames. (high latency goes away). This IPoLCP is a niche feature and has been used in the past to help diagnose problems in back-haul networks: eg: https://www.revk.uk/2015/02/congestion-case-study.html * '''Changing the DSCP value''' - only packets marked CS6 (192 to 195) are affected. Using values higher or lower and the latency goes away '''Getting the FireBrick 2900 to ~~strip~~set the DSCP field to 0''' - this feature ~~hasn't been~~was added to the alpha release of FireBrick. ~~...yet.~~software on 8th December 2021▼ === Things that were not tried === Migrating the line to BT back-haul - this would have fixed the problem for the customer, but would not have fixed the problem in the TalkTalk network or the Aruba access point. Being engineers - we like to fix problems! Changing the DSCP setting on the Aruba Instant-On Access Points - there is no setting, the DSCP field is being set automatically. ▲Getting the FireBrick 2900 to strip the DSCP field - this feature hasn't been added to the FireBrick. ...yet. == Further tests == With A&A having a lively IRC channel, we asked customers to try our ping test to see ~~who~~how far spread the problem was, we found out that: * All AAISP TalkTalk VDSL lines tested showed latency * All AAISP TalkTalk ADSL lines tested showed latency * No AAISP Ethernet lines showed latency WeFurther, we were also able to test on non-AAISP TalkTalk lines: '''Another TalkTalk partner, like us:''' round-trip min/avg/max/std-dev = 17.830/33807.255/48113.126/15886.310 ms So, seems this is a problem within TalkTalks's UK network, probably affecting all TalkTalk ADSL and VDSL lines in the UK. =Deep Packet Inspection Concerns= There are seemingly two faults here: # Aruba adding the DSCP field - which iswhilst trying to be helpful, ~~it's~~is seemingly not configurable, and so in this case it's an unhelpful feature - if it used the DSCP value intended for 'voice' then we'd not have this problem. The customer has opened a support query with Aruba regarding this - Aruba's reply is below. # (in our opinion) TalkTalk should not be looking at the DSCP field. AAISP are taking this up with TalkTalk. ~~The customer has opened a support query with Aruba regarding this. AAISP are taking this up with TalkTalk.~~ ==Fault raised with TalkTalk== ===December 7th 2021 === A&A got in touch with TalkTalk directly by emailing TalkTalk's escalations department and our Service Manager. (There was no point in reporting an individual line fault via the normal channels for broadband fault.) ===December 9th=== TalkTalk are still investigating and are hoping to get back to us next week. They are assuring us on the point about packet inspection, that their policy remains the same in that they are not inspecting traffic in any way and that nor do they have the means to do so. === February 1st 2022 === Update from TalkTalk (below) - saying they have fixed it, but our pings still show latency... '' The core engineering team have investigated and found that the config for the CoS classifier required adjustment - after discussing and planning the steps, a planned engineering work was scheduled, due to our change freeze it took a little longer to put in place, this has now completed and they request if you can kindly re-test and keep us updated.'' sudo ping -i 0.02 -Q 192 -s 1400 -c 200 X.X.X.X PING X.X.X.X (X.X.X.X) 1400(1428) bytes of data. 1408 bytes from X.X.X.X: icmp_seq=1 ttl=63 time=8.25 ms ... 1408 bytes from 81.187.81.187: icmp_seq=200 ttl=63 time='''3084 ms''' --- X.X.X.X ping statistics --- 200 packets transmitted, 200 received, 0% packet loss, time 675ms rtt min/avg/max/mdev = 8.001/1466.857/3145.106/924.943 ms, pipe 70 '''To be continued....''' ==Ticket open with Aruba== Apparently These Aruba Access Points do have the ability to open a CLI on the device and disable DSCP. However, it's not so simple and Aruba advise against it because the CLI requires an interactively-generated token from their support staff, and changing the setting back would require another support call. Current Solution: Customer is currently using the FireBrick feature to set the DSCP field to 0.