In this article, we tackle one of the biggest issues facing broadband users today, known as ‘bufferbloat’. While this term will be rather meaningless to most people, its devastating effects are very widely noticeable. With bufferfloat, other devices or applications running on your network can disproportionately wreak havoc with delay-sensitive applications you may be using, such as online gaming, web browsing and voice telephony (e.g. Skype).
Those with high-speed connections are not immune. With an increasing use of video streaming (e.g. from YouTube and Netflix) and an increasing number of Internet-connected devices in a typical home, bufferbloat will continue to get worse. According to expert Jim Gettys, “bufferbloat is responsible for much of the poor performance seen in the Internet today”. So, if you’re struggling to understand why your upgrade to superfast or ultrafast broadband connection has not given the performance boost you had hoped for, or expected, bufferbloat may well be the root cause.
We first discuss what bufferbloat is, and explain how you can measure it on your own broadband connection. We then discuss how you can successfully, and cheaply, address bufferbloat to substantially improve the quality of your network.
What is bufferbloat and how do you notice it?
Bufferbloat is a term coined by Jim Gettys, a US computer programmer, to describe a phenomenon that occurs on most home networks that substantially degrades performance. After scanning many Internet forums of people complaining about the performance of their broadband connections, many are perfectly describing the symptoms of bufferbloat. Bufferbloat is an acute problem for online gamers. Many people assume that the solution to their broadband woes is to increase the speed of their connection but high-speed connections can suffer dramatically from bufferbloat too. If you sort out your bufferbloat problem, you may be amazed how good your current broadband service is.
Bufferbloat is essentially latency (or delay) under load and refers to the problem where bandwidth-intensive applications (such as video streaming, file transfers, online backups and software downloads) can result in jitter and large increases and/or spikes in the latency (ping) of other applications being used at the same time, causing their performance to significantly degrade. This is because critical small data packets that have to be transferred in a timely manner for applications to work well (e.g. VoIP packets, DNS look-ups and TCP ACK acknowledgements) can be trapped in the buffers of network devices behind much larger packets associated with streamed video and file transfers. These delays can wreak havoc in a number of highly-noticeable ways:
- online games can suffer from unacceptable lag
- web browsing can become sluggish, and less responsive
- delay-sensitive applications such as video and audio telephony (e.g. Skype) can become severely degraded (e.g. with video freezing, audio drop-outs and poor quality)
- remote login (e.g. to a corporate network) can be unreliable.
I first noticed the effects of bufferbloat several years ago after upgrading from an ADSL broadband connection (with a download speed of 5 Mbps) to a VDSL2 fibre broadband connection (with download connection speed of 80 Mbps). I had assumed that the faster connection would be much better, and so it proved when I was alone in the house. However, when my daughter got home from school, I noticed a substantial degradation in performance across a number of applications caused by her accessing YouTube videos. With relatively low average bit rates on YouTube, I had assumed that my new broadband connection would be sufficient to support multiple simultaneous applications with good quality; however, this was not the case. YouTube was making a disproportionate impact on my broadband experience. For services such as web browsing, my previous 5 Mbps ADSL connection without YouTube running was giving a much better surfing experience than my 80 Mbps fibre broadband connection with YouTube running, because of bufferbloat. I laughed when I read the paper ‘Bufferbloat: Dark Buffers in the Internet’ by Jim Gettys and Kathleen Nichols, which describe how “The Internet is slow today, Daddy” became a frequent refrain in the Gettys household, with the children oblivious to their father’s uploading of a large file to his work server.
Bufferbloat is getting worse in many households, fuelled by the take-up of video streaming
The widespread availability of superfast broadband in the UK, with 95% of homes now able to get download speeds of 30 Mbps or above, is helping to fuel the adoption of a range of broadband-delivered, bandwith-intensive services to an ever-increasing number of Internet-enabled devices in the home. These trends are inevitably making bufferbloat much worse. Many people who have upgraded to superfast (or ultrafast) broadband in the hope that simply increasing broadband speed would solve their performance issue may be bitterly disappointed.
Internet forums are awash with online gamers who are struggling to achieve lag-free gaming while others within the household use streamed video-on-demand services such as Netflix and YouTube. In recent blog articles, we have reported on the rapid take-up of streamed video services in the UK, including Netflix and YouTube, and these can massively degrade performance of other applications as a result of bufferbloat. As outlined in our recent article Superfast Broadband Fuels Streamed Video-on-Demand Services, around half of UK households had subscribed to at least one subscription video-on-demand service such as Netflix or Amazon Prime Video by August 2019. Online streaming providers are enhancing picture and sound quality with a rapid evolution to Ultra HD 4k content with next-generation Dolby Atmos sound, increasing bit rates and consuming a greater proportion of a broadband connection. Netflix recommends a steady internet connection of 25 Mbps to stream Ultra HD content. In May 2019, Netflix introduced significant audio improvements, increasing bit rates by up to 233%. As discussed in our article BBC iPlayer Ultra HD (4k) Demonstrates Amazing Picture Quality, the BBC is set to position its online iPlayer service as being the main way to access its programmes and has been actively trialling streaming of Ultra HD programmes with data rates of 36 Mbps for live broadcasts. In November 2019, Apple introduced its Apple TV+ streaming service with average bit rates of 25 Mbps according to our Review of Apple TV+: Apple’s Streamed Service Leads the Pack in Picture Quality. Disney launches its 4k Disney Plus streamed service in March 2020.
In its ‘Media Nations: UK 2019’ report, published in August 2019, Ofcom reported that young adults (aged 16 to 34) spend over an hour on YouTube per day, and almost all adult internet users watch YouTube videos.
Video streaming is just one of the ‘culprits’ affecting performance with bufferbloat, and we could equally mention other culprits such as streamed audio services, online backup services and software updates. There has been significant growth in music streaming services, with subscription revenues increasing by 34% in real terms in 2018 according to Ofcom. 41% of the audio time of young people (aged 15 to 24) is spent on music streaming services and less than a third of their time is spent listening to radio. Increased broadband speeds have allowed streaming providers to increase bit rates. Amazon launched its HD music streaming service in September 2019, offering music in CD- and better-than-CD quality. The average bit rate for better-than-CD quality titles is nearly 4 Mbps.
The impact of bufferbloat is also made more significant as a result of an increasing number of devices in the home. Now desktop PCs, laptops, smartphones, tablets, smart TVs, set-top boxes, game consoles and streaming boxes are all typically connected to the home broadband connection and could be making high bandwidth demands, even if only performing software updates!
Latency is often more important than speed
An article by Computer World entitled ‘The Hidden Cause of Slow Internet and How to Fix It’ discusses the widespread conviction that the way to eliminate any deterioration in performance is to add bandwidth. This may, however, be fundamentally flawed as latency and speed are very different things. Internet forums are full of people who have upgraded to superfast and ultrafast broadband but are struggling with online gaming lag. I asked a friend with an excellent FTTP connection to record the ping to a website while running an online speed test. Ping spiked from its usual 12 ms measurement to over 200 ms. The connection was rated by the DSL Reports speed test (discussed below) as ‘grade C’. Our experience is shared by many others online. According to DSL Reports, the majority of broadband connections have been rated as grade C or worse for bufferbloat, which means that the increased round trip delay when the connection is fully utilised is more than 60 ms.
Many services that are used often are more sensitive to latency than speed and it’s not the obvious ones such as voice and video telephony (e.g. Skype) and online gaming. Latency can also wreak havoc with web browsing, for example, which can become an unresponsive, stuttering nightmare. In general, relatively low broadband speeds can deliver excellent performance with web browsing, online gaming and voice telephony provided that the latency and jitter are low. In his blog, Jim Gettys says “today, we often pay for much more bandwidth than needed for our broadband connections just to reduce bufferbloat’s effects; most applications are more sensitive to latency than bandwidth”. He adds, “Having a great TV experience, at the cost of good phone or video conversations with others is not a good trade-off.”
We absolutely agree that the fantastic services enabled by faster services, such as streamed video-on-demand, should not degrade other delay-sensitive services that do not consume significant bandwidth, such as web browsing, voice telephony and online gaming. However, as shown below, you can have your cake and eat it by minimising bufferbloat on your own network. This means that you can take advantage of all the bandwidth-intensive services that are available to you while maximising performance for delay-sensitive services. It also means that you can potentially save money by avoiding upgrading to faster broadband services if you do not need them. Investment in a new router, for example, that minimises bufferbloat could pay for itself quickly if you save money on your broadband connection.
How to measure bufferbloat on your network
It currently is a challenge to find online tools that measure bufferbloat, despite its critical importance. Most online speeds tests that measure latency/ping (such as Ooka) measure ping when nothing is being downloaded or uploaded i.e. when the broadband connection is not fully loaded. Currently, the most comprehensive speed test for the measurement of bufferbloat is provided by DSL Reports. You can measure your bufferbloat here:
The speed test continually checks latency from the start to the finish of the test, not just at the beginning (like most other speed tests). If the latency rises substantially during the download or upload section, this indicates a high degree of bufferbloat when the connection is fully loaded. Usefully, DSL Reports provides a grade for bufferbloat – between A+ and F. Grades are awarded according to the increase in round trip delay (averaged across downlink and uplink) when the connection is fully utilised. Grade A+ is awarded to connections where the average increase in latency is less than only 5 ms, which is an outstanding result. The screenshot above shows out speed test results after we had implemented our solution for bufferbloat (described below). Before, we were receiving a score of C.
Grade A is awarded to connections with average increased latency of less than 30 ms. Grade B is awarded for average latency of less than 60 ms.
The majority of tests have been awarded a Grade C or below. According to DSL Reports, this is because most equipment and software currently in use have bufferbloat issues. Grade C corresponds to an average latency increase of less than 200 ms. Grade D is awarded for an average increased latency of below 400 ms, while the lowest Grade F corresponds to an average increased latency of more than 400 ms. There is no E grade.
The introduction of grading by DSL Reports has definitely raised the profile of bufferbloat as an issue, and has stimulated competition among online gamers and enthusiasts to maximise and publish their performance grades.
Another way to measure bufferbloat is to use a do-it-yourself approach. First, run an application to measure ping on an ongoing basis. We find that pinging a site like www.bbc.co.uk or www.google.co.uk provides a low measurement. With this, you’ll be able to determine the no-load or minimal-load ping. Then, run an online speed test such as our Broadband Speed Test. Assess any increase to the no-load/minimal-load ping value.
Based on data from DSL Reports, most connections suffer an increased latency of 60 ms or above, allowing you to judge your own connection. To get the equivalent of an A+ rating from DSL Reports, you need to achieve an increased latency of less than 5 ms. Good luck!
Address bufferbloat by upgrading to a router incorporating Smart Queue Management
Now that you have identified the extent of bufferbloat on your own network (using the tests above), we consider how to reduce this. In this article, we concentrate on solutions that are relatively easy to implement by non-experts. In general, the most effective way to combat bufferbloat is to implement so-called Smart Queue Management (SQM) in your router. Ideally, your router should be running the most effective SQM algorithms that specifically address bufferbloat, such as FQ-CoDel or the more advanced CAKE (Common Applications Kept Advanced). These control how traffic is managed on your network and, in particular, ensure that the small packets that are so critical to latency-sensitive applications are not delayed in buffers behind packets for video-on-demand, for example.
According to Dave Taht, Co-Founder of Bufferbloat.net, there are numerous Smart Queue Management systems in existence with varying degrees of effectiveness and with different trade names (such as Adaptive QoS from ASUS, StreamBoost from Qualcomm, Anti-Bufferbloat from NETDUMA and Evenroute’s IQrouter software).
(1) Ubiquity ER-X and ER-4 routers provide excellent performance with bufferbloat
Perhaps one of the easiest solutions – and the one we have selected – is to purchase a new router incorporating Smart Queue Management that is known to work well with bufferbloat. A number of highly-regarded routers are available that do just this, including the Ubiquity EdgeRouter ER-X and more powerful ER-4 and the Evenroute IQrouter.
We have purchased the Ubiquity EdgeRouter ER-4 for this purpose because it is powerful enough to support fast broadband connections without CPU or memory limitations. It is a very popular router online and among online gamers and incorporates the FQ-CoDel algorithm. We will provide help pages for this router so that you can set it up easily shortly. Our tests below are for the included firmware, but note (as discussed below), it can be reflashed with OpenWrt firmware, which supports both FQ-CoDel and CAKE Smart Queue Management algorithms.
Broadband users with connections that do not exceed 100 Mbps could consider the substantially cheaper EdgeRouter ER-X, although you may feel the more powerful ER-4 is more futureproof if you expect to move to ultrafast broadband (>100 Mbps) in the not-too-distant future. Both these devices are significantly cheaper than many specialist, so-called ‘gaming routers’ many of which do not successfully mitigate bufferfloat. Internet forums are full of people who have invested in very expensive gaming routers to discover that these have not addressed their bufferfloat issues.
We absolutely love our ER-4, and our 80 Mbps fibre broadband connection does not tax the router in the slightest in terms of CPU and memory usage. The ER-X and ER-4 have received some criticism online for being relatively hard to set up. It is certainly true that they offer a powerful combination of options, and it is possible to implement extremely complex quality of service processes. However, both routers provide simple wizards to set up a basic connection, which will work for most people. They also offer a very simple ‘Smart Queue Management’ option that can be ticked and will make a huge difference to bufferbloat without any advanced changes. For those who just want something that works quickly, this is the best option. However, for those who want to tweak their connection, there are many options to play with. Basic options allows you to apply Smart Queue Management to upload traffic, download traffic or both. Your own latency testing will help you determine which of these to select. You are expected to enter your upload and download traffic rates. Here, you can experiment with different values to assess their effects. Beyond this, there are advanced options (which we will not consider here).
Our experience with using the ER-4 on our broadband connection has been that the simple settings have completely mitigated bufferbloat. For example, our connection now receives an impressive A+ for bufferbloat on the DSLReports speed test. We will experiment further with settings and update this page. No-load pings of 8 ms never increase beyond 10 ms when the network is fully loaded. Whatever is running, including speed tests, video-on-demand services and file downloads, all devices seem as responsive with Skype and web browsing than the do when there is no network load. It really is truly impressive. Given that the ER-X can be purchased for less than £60, this is a highly cost-effective way to dramatically improve your broadband connection. Once you experience what your broadband network should work like, there’s no going back!
In general, there is a small price to be paid in terms of maximum connection speed (i.e. capacity) to substantially reduce bufferbloat. For the simple settings we used, maximum throughout was 67 Mbps rather than 74 Mbps achieved without Smart Queue Management – a 9% reduction. There was a similar reduction in upload speed.
Broadband users with very high speed connections may have more flexibility to minimise bufferbloat. Most bufferbloat solutions can be turned on and off as and when required so some online gamers may choose to implement bufferbloat minimisation only at times when they are playing games, preferring to get the greatest broadband speeds at other times. Those benefiting from the highest speeds may prefer to implement bufferbloat mitigation on a continuous basis. As long as the maximum speed of your connection comfortably exceeds the minimum speed requirement of the services you wish to use (e.g. streamed video-on-demand), then we recommend prioritising full-load latency reduction for an amazing experience. We believe Smart Queue Management is a no-brainer for any user with a broadband connection delivering 40 Mbps or above.
For those who currently use a WiFi router (combining router and Wifi functionality), adding a dedicated router such as the ER-4 is relatively straightforward. Most WiFi routers can easily be reconfigured through their menus to act as WiFi access point. In our configuration, we have gone one stage further and use individual boxes for modem (the Billion 8800NL modem router acting as as modem only), the ER-4 router and the Synology RT2600 WiFi router (acting as a WiFi access point). Such an approach gives maximum flexibility and allows you to choose best-in-class boxes and upgrade individual elements. Furthermore, by splitting functionality across the CPU and memory capabilities of multiple boxes, the risk that the CPU and memory of a multi-purpose, single device will limit network performance may be minimised. In our case, the ER-4 is able to dedicate significant resources to implementing Smart Queue Management even with very high broadband speeds (several hundred Mbps) leaving processing of other aspects (for example, WiFi operation) to the processor and memory of another device.
With the ER-X and ER-4 routers it is possible to reflash them with OpenWrt software, described below. Dave Taht says, “I do love Ubiquity EdgeRouter hardware, but I reflash it immediately with OpenWrt nowadays, which, after you do that and install the Graphical User Interface, has full support for both FQ-CoDel and CAKE.”
(2) Reflash an existing router with the OpenWrt firmware
For many broadband users, the most straightforward, and risk-free way to address bufferbloat is to purchase a new router, as described above. We have shown that this approach is capable of delivering A+ bufferbloat performance (as measured by DSLReports) while achieving very high throughputs. However, another option available for enthusiasts and more expert users is to reflash an existing router with the OpenWrt firmware, which incorporates Smart Queue Management to address bufferbloat. You will need to check if OpenWrt is available for your particular router model here:
OpenWrt has become popular among enthusiasts because, compared with the stock firmware that comes with some routers, OpenWrt can be more stable and offer more advanced features, such as Smart Queue Management. However, please be aware that, if you attempt reflashing of your router’s firmware, you do so at your own risk. Also, bear in mind that OpenWrt is unlikely to transform the performance of an old router with limited CPU and memory. The processing requirements of Smart Queue Management means that maximum throughputs when it is operational may be significantly lower than the maximum speed (>100 Mbps) for ultrafast broadband connections.
OpenWrt has a page describing its Smart Queue Management and how to install its bufferbloat packages here:
Current versions of OpenWrt have both FQ_CoDel and CAKE built in. CAKE adds extra functionality over FQ-CoDel, aimed at improving performance yet further. Owners of the Uniquity ER-X and ER-4 routers (discussed previously) have the opportunity of using FQ-CoDel with stock firmware, or reflashing their router with OpenWrt to use FQ-CoDel or CAKE.
If you are not happy with the performance you are currently getting with your broadband connection, please visit our Increase Broadband Speed Guide and More Tips pages. In our experience, a tiny number of issues – namely bufferbloat and WiFi set-up issues – are responsible for the vast majority of the problems experienced by broadband users. For example, as well as minimising bufferbloat, we recommend that you connect as many devices using an Ethernet cable as possible (particularly smart TVs and video streaming devices) and optimise your WiFi set-up using our WiFi Optimisation Guide. Changing your DNS server settings may substantially improve performance and reliability too.