CQM Graphs

The FB6000 L2TP router provides us with Continuous Quality Monitoring (CQM). This allows us to track the quality of each and every connection in great detail. The router itself produces the graphs in real time, and can also provide csv files with accurate data for each graph.

Customers and Staff can view these graphs in near real time (updated every 100 seconds), and can view historical graphs.

How does it work?

Our router sends an LCP echo (like a ping) every second while a line is active. Your router replies. We track how long it takes for each reply to arrive, and how many are lost. These results are collated into 100 second samples and shown as a graph like the one on this page. The graph shows us lots of information about the line, and gives a history covering the last 24 hours.

What information is on the graph?

CQM Graphs

Each column (pixel) represents 100 seconds of samples. The hour of day is shown at the bottom, and the day and date shown next to midnight in the graph. There is additional text superimposed on the graph such as a circuit ID. There are 8 pieces of data shown for each 100 second sample as follows:-

A note on Tx/Rx download/upload

Graph Key

As stated in the image above, the dark red is customer uploading, and the darker green is customer downloading. The key on th right does label these as Red=rx Green=tx. As the graph is created at the AAISP side, the Tx/Rx is in relation to AAISP, ie, Red is Rx, what AAISP has received from the customer.

The colours may vary and we even allow users to configure colours individually (useful for people with colour blindness, etc). It is also possible for us to show graphs with only some of the fields which can make some information clearer.

Pins

Customers (and staff) are able to add 'Pins' to graphs. This is useful for customers to add notes that staff see, for example you can add a pin saying 'Plugged router into the test socket'. The note will be shown when you hover your mouse over the pin.

Adding a pin to a graph

Pins are added on the Usage page. Simply click on the graph where you'd like to add a pin and enter in the details.

ICMP Graphs

Optionally, Staff can enable ICMP ping graphs, these will graph a normal ping to the customer WAN IP address. These tend to have a salmon colour background:

An graph created from ICMP Pings to a WAN address

Sometimes a normal CQM graph may have a salmon background - this would usually be when the line is logged in to our test LNS.

CQM Examples

Lots of Upload

Lots of upload at the end of the day

(Not an ADSL Fault.) The example above show a line with occasional short uploads causing spikes in peak latency, and then a sustained upload starting at around 6pm and causing high latency (queue in the router). At 8pm there was more upload filling the link causing higher latency still and some loss (normal when the link is full). This is normal. Also see: Packet Loss

Redcare

This is an example of a line where the solution was redcare changes

BT Core Network

Line showing latency where there is almost no usage. This is an example of a BT core network problem affecting two lines.

Congestion within BT

This is down to congestion within the BT core network, this could be at the Exchange, BRAS, or the VP. It would be seen on multiple lines.

Only Dropping during the day

(Probably not an ADSL Fault.) If a line is dropping during the day, and maybe just Monday to Friday, then it's probably not going to be an upstream problem. This could be caused by interference of bad wing on site. Check things like the phone line and extensions. Put the router in master socket and to unplug all other phones. Maybe change filter.

Heavy Packetloss

Also see Packet Loss

This is showing more than 50% packet loss with no usage. A problem!

This line has loss due to a Battery Contact Fault. A copper line test suggest that this should be reported to Openreach as a fault.

Faulty Switch on the LAN

(Not an ADSL Fault.) This latency and loss was caused by a fault Netgear switch on the LAN side of a Vigor router! The switch had failed to the point where it wouldn't talk gigabit but would talk 100M unreliably. Guess it was maxing router CPU perhaps. It's unknown how this affected the Vigor, but perhaps the switch was faulty enough to upset the Vigor causing it to delaying or not replying to the LCP echos. In this case unplugging the LAN side of the router would show a normal looking graph, indicating the fault is somehow caused by something on the LAN.

Other Examples

Examples of things that are a fault
	Dripping Blood Packet loss on an idle line is always bad news, even if only 1% (one red dot at the top is 1%). If this happens on several lines at the same times of day this usually means congestion on the back-haul to the exchange or elsewhere in the network. If it is steady loss all day it can mean a fault on a line card or interference on the line. Also see: Packet Loss		Long grass Some issues can be subtle, affecting latency on idle lines like this with occasional peaks. Even low levels of jitter like this can disrupt VoIP and can indicate a wide problem such vendor equipment bugs or the start of back-haul congestion. We have seen some routers (Asus) behave like this, the line is clean when swapping the router.
	Speed bumps Latency humps like this are normally a sign of back-haul congestion. They normally show on all lines on an exchange or BRAS at once at the same time of day, and only show at certain (busy) times. The clue here is the consistently high latency even when little or no traffic flowing.		Going off line is shown in purple, and this is often associated with packet loss (red). Where a line has occasional drops they are shown as purple lines. However in some case a line can deteriorate over a period of time, staying on line less and less until solid purple (off line). On the live graphs a line that is currently off line has a red square in the bottom right corner where as a line that is on-line has a green square.
	A congested link in the back-haul can show as loss or latency humps on an idle line, but in some cases, like this, it can show only where there is traffic on the line. The symptoms are latency spikes like this (green spikes) even when there is very low usage. Compare to normal graph below.

Examples of things that are not a fault
	A normal graph shows no packet loss (no red at the top) and little of no change in latency when downloading or uploading at low rates. Compare to congestion graph above.		This graph has a saw tooth latency pattern. This can sometimes be just in green (peak) latency or in blue (minimum) latency or some combination of the two. It can be regular like this or change frequency. This is just the way this specific make of router responds to LCP echos and is not actually affecting traffic.
	Some makes of router can also cause the occasional LCP echo not to be answered until the next one, causing 1 second latency spikes to appear making green lines like this. This is just the way the router handles LCP echos and is not a problem as such.		This graph shows very high latency initially, with blue filling the graph (500ms) much of the time and green the rest of the time. However the red line at the same time shows the uplink is being run at full capacity so the latency is just down to the queue in the router. Later the uplink rate is reduced and the latency drops away to a low level with some small peaks. During the high upload you will also notes a few red dots at the top (packet loss) also caused by the upload.
	Purple on the graph is off line, and this can be short blips if a line loses sync or longer. Notes (pins) are often added to graphs to explain why a line is off line if we know, especially when we are investigating a fault. The notes on this graph told when the BT engineer arrived and left.		Regular drops repeating every day eg Central Heating causing drops. View a week of these graphs Here there is interference as the central heating goes on, the drops are regular - twice a day, at the time the heating goes on. This actually highlighted a fault in the central heating for this customer as the graphs showed no drops for 2 days running!

Regular evening drops

Historical graphs are kept which makes it easy to see patterns is line problems. Here we can clearly see that the line is being affected by something causing it to drop in the evenings. This could be something local causing interference and is less likely to be a PSTN problem. This could be something electrical being used in the evenings, such as putting the dishwasher on after dinner. Knowing that a line drops regularly at a specific time will help diagnose the source of the problem for example, checking what electrical devices are used in the evening.

Other Information

Rate limit line - red dot at bottom of graph

BT run a rate limiter for your line at the BRAS. It limits traffic to a set level (typically whole Mb/s) that is below your line sync speed and can change over time depending on your line. We track this BRAS rate and set the CQM system to show this on the graph. It can only gets recorded per hour so shows the highest it was - with a dotted line for the lowest in the hour. You can ask us to limit your line below the BRAS rate - what is shown is the rate we have on the line for you service..

We normally rate limit the line at the same level as the BRAS, i.e. we are not trying to be the bottleneck. We can limit at a lower level if you ask us to. If there is a burst of traffic on your line that would exceed the BRAS rate and mean BT drop packets at the BRAS, then we drop the packet first. By us detecting the burst exceeding the BRAS rate we can (a) show this on the graph as a red dot at the bottom and (b) we can allow small packets such as those used for VoIP to go though when large packets would be dropped (we set a different latency limit for small and large packets). This may still be dropped by the BRAS but typically this has the overall effect of VoIP working well on a full line.

Tracking the BRAS rate is also important for load balancing multiple lines and attack detection.

Here is an example where the download is hitting the limit, and causing the line to drop sync - this may be to do with the router crashing. The graph shows red at the very bottom, indicating the download hitting the limit of the line. At the same time there is a green line near the top which indicated the download, but at the same time there is packetloss (red at the top) and line drops (purple).

Why only 24 hours of history?

Actually, each day, we archive the previous days graphs. This allows us to see a history of a line which is invaluable for intermittent faults or repeat faults.

Why all lines, not just those with faults?

We have to send LCP echos anyway to tell if a line goes off, so we may as well record the data. Sending one every second means we have very detailed information. When a fault does happen we have a full history already available, and this greatly speeds up the process of getting faults fixed. Another important point is that we can immediately see when a fault affects multiple lines as we are monitoring them already. We can even spot faults that our customers have not noticed yet.

Monitoring all lines also allows us to identify core network issues in carriers like BT and get them fixed.

How is this information used?

We make the information available to customers to see their own lines, and dealers (e.g. IT consultants) to help their own clients. We also make the information available to BT in the event of a fault on a line to help them understand the nature of the fault. The graphs allow us to identify many types of faults immediately and so ensure they are fixed more quickly.

I see grey bands on my graphs

In addition to the graphs from the FireBrick we add some extra information. This includes pins on the graph we can add to lines or groups lines to show know issues or events. We also add grey bands to mainland UK graphs which show night and twilight based on your postcode. This allows us to see any light related issues, such as automated security lights with a fault that affects the broadband. It also looks cool.

I pay for usage - what is this monitoring costing me?

The usage based tariffs are metered based on IP traffic. They do not include LCP traffic at all, so these LCP echos are not costing you anything. They do use some of your bandwidth. One ATM cell (53 bytes) is sent each second. On an 8M line this is 0.005% of your downlink bandwidth. Even with limited uplink such as 250Kb/s, only a tiny fraction of a percent of the bandwidth is used for the monitoring.

I have a firewall - will this stop the graphs working?

LCP traffic is not firewalled, and does not go on to your network. Your modem/router must reply to the LCP echo. There are some unusual makes of router that are slow at replying and generate interesting latency graphs which do not reflect the actual delay on the line, but these are rare and easy for our engineers to recognise so as not to mistake for a fault.

Can you really ping every line every second?

The FireBrick FB6000 platform is designed to send and accurately time each LCP packet even when fully loaded. So, yes we can.

CQM Graph options

The control pages have a number of options to allow the graphs shown to be refined. These are shown simply as 8 input boxes. If all empty then the standard graph is show for lines. You can however input control strings in to these to define up to 8 separate graphs to be displayed instead. This can be particularly useful if you are colour blind, for example. It also allows simplification of graphs which may be rather cluttered, e.g. separating usage from latency in to separate graphs.

Colour controls

The colour controls use a letter followed by a colour setting. The letter defines which of the 8 traces to show on a graph, and the colour control defines what colour to show.

Each of the 8 boxes is for a graph. Fill in the colour controls from the left box. You can have one graph, two graphs, three graphs, etc. Each graph can have one or more of the 8 traces shown. It is possible to have 8 graphs each with one of the traces on it if you wish.

It is also possible to change the background and text colours.

Key	Trace
m	Min latency
a	Ave latency
x	Max latency
s	Sent
f	Failed
o	Off line
d	Downlink rate
u	Uplink rate
I	Graticule colour
i	Axis colour
g	Background colour
G	Border background
w	Text colour

Example

For example, if you simply complete the first box as 'max' then you will only get one graph, and it will only have the Min, Ave and Max latency shown. If you complete the second box as sfo, then there will be two graphs shown, the second containing only Sent, Fail and Off-line.

To change the colours you include (rrggbb) format colour hex codes after the letter, so if you set a box as m(000000) then that graph will show only the Min latency and show it in solid black.