Testing Internet Speeds

Recently, a few of our customers have upgraded their internet connections. Wanting to test the shiny new connections and demonstrate that they were money well spent, they went to speedtest.net and were disappointed to discover that the reported speeds were unexpectedly low when the traffic was routed through their Opendium Iceni web filter. We ended up fielding a few support calls relating to this, and after some investigation discovered that the low speeds being reported were largely due to flaws in the speedtest.net application.

There appear to be a number of reasons for the misleading speedtest.net results: Firstly, the application appears to have a couple of different methods of testing throughput, and silently switches between them depending on various network conditions. By preference, it tests throughput by transferring large amounts of raw data on TCP port 8080, but if that fails, it falls back to using HTTP on port 80. In testing, the throughput we measured using the port 8080 test was significantly higher than the HTTP test, given otherwise identical network conditions. It was not clear to us whether the under performance of the HTTP test was due to a fault in the speedtest.net client, or the speedtest.net servers.

We also noted that the data transferred for the HTTP tests were random noise - this is good practice because random noise can't be compressed, so any data compression on the connection would not artificially inflate the measured throughput. Unfortunately, this good practice doesn't seem to be used on the port 8080 test - there, a repeating text string is used as the test data, which would be highly compressible. Therefore, if any data compression is used on the internet connection, we would expect the port 8080 test to produce artificially high throughputs, perhaps even exceeding the physical connection's maximum throughput.

Customers were making tests on completely unrestricted connections, and comparing them to tests made on their normal networks which are filtered by Iceni. The logical conclusion was that any reduction in the throughput was due to a bottleneck caused Iceni. However, connections on TCP port 8080 are typically not allowed on the filtered networks, causing the speedtest.net application to silently switch to the under performing HTTP test, irrespective of any interaction with the proxy / web filter.

We also showed that the speedtest.net application miscalculates upload speeds when testing over HTTPS. We ran speedtest.net tests whilst independently measuring the internet connection throughput. Speedtest.net reported upload speeds of around 90Mbps, whilst our independent measurements only showed a throughput of 50Mbps.

Most people using speedtest.net will be on unfirewalled home connections, so the tests will automatically be carried out using the port 8080 mode. It's likely that speedtest.net simply haven't tested the HTTP mode properly since it is used less frequently, and this has resulted in several bugs not being noticed. We believe that our research has demonstrated significant failings both with the speedtest.net application itself, and with the overall testing methodologies employed. As the speedtest.net service produces grossly inaccurate results in some cases, we conclude that it should not be used.

Simplistic throughput testing can be achieved by using the test download files provided by thinkbroadband. However, throughput testing is just one tool in measuring the performance of an internet connection, and the results must be interpreted in context. A single large download made by a single workstation in no way reflects the normal usage patterns you would expect to see on a multi-user network. In particular, in some environments a single stream will achieve a much lower throughput than multiple concurrent streams (to their credit, speedtest.net do use multiple streams); on the other hand, the overheads associated with connection setup and teardown are significantly higher when handling the typical short-lived web requests that most web browsing creates, compared to a few large downloads used by throughput testing.