Introduction

Connection speed is a key factor when selecting which VPN to use. In fact, when NordVPN asked their users about the most important features when choosing a VPN provider in the fall of 2020, the survey showed that VPN connection speed was the second-most important factor (97% of the 5248 survey participants listed “Internet speed when VPN is turned on” as “important”, including the 82% who listed it as “very important”).

These results come as no surprise. Network speeds are improving throughout the world, and users accustomed to high internet speeds will notice when a VPN is slowing them down. According to the , the average U.S. download speed in August 2021 was 199 Mbps, 27% increase over a year compared to 156.61 Mbps in August 2020.

That is why it is crucial to have a reliable, transparent, verifiable way to test different providers’ VPN speed performance. A robust open-source tool would let VPN users make accurate and educated judgements about the speeds they can expect with different VPN service providers. For VPN providers, this tool would represent an objective standard against which they can test, polish and improve their code and infrastructure – and compete with other providers. Our goal is to provide that tool for the general public.

The problem

VPN speed testing tools and methods already exist. However, there is no solution that sufficiently addresses all of the following concerns:

• Transparency: Users, reviewers, and VPN providers all need to understand how VPN speeds are measured. A lack of transparency can make it difficult to identify mistakes and trust the test results.

• Reliability and verification: The testing solution needs to work consistently across different providers’ networks, across years of VPN development, and it needs to be able to verify its own results. It must also provide the means to achieve sufficient data scales for statistically significant results.

• Universal application and usership: The solution needs to work for a wide range of different VPN providers. By becoming a widely adopted solution, it will.

When different reviewers rely on different methods and testing environments, they get different results. When this is the case, it makes it very difficult for users to meaningfully compare VPN speeds.

It’s not surprising that this problem exists. After all, testing VPN speed is quite complicated due to the number of variables.

Current VPN speed testing issues

Before we can solve the problem of providing standardized, verifiable VPN speed testing, let’s review the issues with testing as it is performed today. Here are just a few that we’ve observed:

• Insufficient volume: In most cases the VPN tests are being run manually which substantially limits the number of tests that can be reasonably performed for the review, thus, causing the results to be less accurate. This can be solved by automated testing.

• Irregular testing frequency: Different providers may be compared at different times, which is a significant issue. Such tests do not take into account the possibility that some VPN providers may have inconsistent performance based on the time of day - it is not unlikely that some providers may not have enough network capacity to provide unthrottled VPN service during peak times. This can also be solved by automated testing.

• Inconsistent network conditions: Some tests were being performed over WiFi or even on public networks with no assurance of a stable WiFi connection. Although home WiFi should generally be much more stable than public networks, it still isn’t as stable as a cable connection, and all of this can impact the final test results. This is especially true in condo areas where a huge number of WiFi networks are configured to use the same band and can cause interference - even with Bluetooth devices.

As a recent study by University of Chicago researchers shows, even LTE-Licensed Assisted Access (LAA) networks may have a drastic impact on WiFi network performance. Therefore, all tests should be conducted over ethernet connections. This eliminates WiFi performance inconsistencies that could impact the test results.

Finally, the tests must be performed on high-throughput networks. Otherwise, the client network may be the bottleneck preventing the test from identifying maximum VPN performance. The network should have at least as much throughput as the highest-throughput servers that most VPN providers can offer. Given that many VPN providers are expanding their 10 Gbps networks, the tests should also be performed on networks of 10 Gbps or greater if these are available. Otherwise, the conducted test will not be able to account for VPN providers that can provide speeds exceeding those of the local network.

• Browser limitations: Another issue we see is that tests are usually run through an internet browser. This introduces a new variable when running a test as the internet browser and its extensions consume part of the device's CPU resources. In fact, research by DebugBear shows that even a single extension may occupy CPU resources for over a quarter of a second.

VPN performance is highly dependent on the CPU being fast enough to encrypt all network traffic. Therefore, VPN speed tests should be performed in a controlled environment where varying resource usage by the browser cannot influence the results. Luckily, the most commonly used tool to test network performance, Ookla speedtest, offers a CLI client that can be used from a Terminal or PowerShell window.

• Inadequate CPU load considerations: Inadequate CPU resources or inadequate distribution may impact test results. Software running in the background can have an even higher impact on network speeds than browser extensions. Some of the biggest culprits of computer resource usage are cloud storage and anti-virus programs, both of which constantly run in the background and should be switched off when performing a test.

Even without competing software or processes, general CPU usage while running a speedtest over VPN has to be monitored. A VPN is capable of maxing out the CPU or one of its cores when using a VPN protocol that does not support multithreading like OpenVPN. Here are some CPU usage examples when running Ookla speedtest while connected to different services over different protocols:

• This image shows the CPU load while running VPN using OpenVPN protocol. A single CPU core takes the brunt of the CPU’s load while other cores remain under relatively low load. That is because, as of the publishing of this paper, OpenVPN does not support multithreading. Moreover, that single CPU core’s load is at almost 100%, potentially causing the CPU to become a bottleneck when running a speed test. Let’s compare that to the CPU load while running the speedtest over a proprietary protocol, provided by VPN A. We can see that the CPU workload is shared relatively evenly among all the cores. A similar CPU load spread can be observed using a proprietary protocol, developed by VPN B as well.

The design process

The prevalence of these issues showed us that we needed a publicly available solution to correctly test the performance of various VPN providers. That’s why we’ve developed speed test software available to anybody interested in testing the impact different VPN providers have on network performance.

To achieve these goals, the tool must:

• Address the issues that we currently see with VPN provider comparisons;

• Be easily accessible and usable;

• Offer customization to fit different testing scenarios;

• Work as open-source software so that anybody could add functionality for use cases that we may have missed.

Transparency is also key when objectively testing the performance of competing VPN providers. This is why we chose to release the source code on a widely-used software development and version control platform – GitHub. By distributing open-source code, we’ve ensured that absolutely anyone can reproduce and verify someone else’s results and examine the code for biases.

Having the source code hosted on GitHub makes the tool easily accessible via terminal or web interface, whichever option is preferred by the party installing the software. Additionally, the software can also be easily updated using Git to pull the updates from the master or other branches that may have features added by third parties that have not yet been merged into the main build of the software.

On release, our VPN provider network performance testing tool already has a multitude of features that should be useful when comparing different VPNs. First of all, however, we should point out the VPN providers supported upon release. Support for others can be added* but here are the ones included right now:

• NordVPN;

• ExpressVPN;

• SurfShark;

• PIA;

• PureVPN;

* Adding a provider requires creating a personal branch of the tool’s project on GitHub and updating the code. Our developers can later review the code and merge it into the master branch, making it available for anyone.

We chose to use the Ookla speed test as part of our tool. Here’s why:

• It can be used via CLI;

• It is one of the most popular tools for testing network performance that most of us are familiar with;

• Because Ookla allows third parties to configure speedtest servers on their own infrastructure, it allows them to have speedtest servers all around the world.

The last point – third-party configuration capabilities – is great for us as our speed test tool can be used to test VPN servers in numerous cities. However, this approach also has its drawbacks. Because Ookla servers can be freely hosted on your own infrastructure, quality control is diminished. Some servers running Ookla may not have sufficient network performance for high-speed tests.

The question of where to host the tool is part of a broader decision – whether it should be hosted on a VPS or a dedicated server (while it can also run on a personal computer, we recommend setting it up remotely so that no other software running on the device can influence the results). Both options have their own pros and cons. A VPS is an easily accessible service with relatively low costs. However, it is likely to have limits on resources like network bandwidth. On the other hand, dedicated servers probably won’t impose any significant restrictions on your network usage (having a shared network is an exception), but the costs and management complexity are much higher.

For most VPN providers, it would probably be easy to run speed tests on servers hosted on their own infrastructure. However, this could potentially skew test results. If a provider were to host speed test servers at the same DCs as their infrastructure servers, they could potentially gain an unfair advantage. Although the Ookla speed test does allow users to select specific servers, using their recommended servers feature reduces the chances that tests will be run to the same DC. Moreover, automatic server selection allows for speed tests that better simulate real-world VPN client VPN speed test practices.

This is why we’ve chosen to go with a VPS to provide a standardized client with a publicly available baseline for VPN provider comparison.

In regards to how VPN clients usually use VPN service, it should also be considered how they select VPN servers to connect to. Generally, there are 3 options to connect to VPN or 2 depending on the VPN provider:

• Leaving server selection completely up to VPN application;

• Selecting specific country/region/city to connect to but not selecting specific VPN server;

• Selecting specific VPN server to connect to (this option may not be supported by some VPN providers);

A vast majority of VPN clients use one of the first 2 options as these should generally provide the best VPN performance. VPN applications will then try connecting you to the least loaded servers that are closest to you if you have not specified the location of the VPN server you want to connect to. However, when connecting to specific VPN servers it is possible that the selected server is more heavily loaded which may impact the overall performance of the VPN. Considering that most users are not using the third option, nor are we recommending it when looking for the best performance, our speed test tool is also configurable to either connect to specific countries VPN servers are located at or leave the server selection up to VPN provider.

Lastly, while network download speeds might be the most important factor in many cases, they aren’t the only factor affecting internet usage experience while connected to VPN. Upload performance, latency, jitter are all important as well and can in fact be measured using Ookla speed test client. All these measurements are stored alongside download speed and can be reviewed when investigating the results.

Using the tool

To set up and run the tool, please refer to the readme file provided on Github. What follows are guidelines and best practices for achieving quality tests and getting the most out of the tool’s features.

Location features

The tool includes a number of features that should help accurately measure the network throughput of a VPN. For example, the common_cities setting lets you control what VPN server locations should be tested. Say we want to compare speeds to VPN servers in France. NordVPN has servers in Paris and Marseille. However, other VPN providers might only have servers in Paris. In this case, whenever the common_cities value is set to true, only VPN servers in Paris will be tested. This lets us choose between testing all servers in the country or only testing servers in common VPN server locations. This option is great when we want to compare different providers more accurately (comparing Paris speeds to Paris speeds, instead of Paris and Marseille to Paris). However, if the common_cities value is set to false, then the speed test tool does not specify the city of the server to connect to and lets VPN applications select what is supposedly the best server for the client in that country. For example, if we run a test for VPN performance when connecting from Europe to the USA, most of the tests will connect to VPN servers on the east coast if common_cities is set to false. However, if it is set to true, the cities the VPN servers are located in should be distributed more equally and include an even number of servers on the west coast, too.

Test frequency

Testing servers in the same cities is not enough to ensure an accurate VPN performance test. Having the speed test run continuously is equally if not even more important. We recommend running tests for 3 days with at least 1 test an hour. This will ensure that no less than 72 tests are completed, providing a significant amount of data to draw conclusions. We recommend running the tests for at least 3 days because we have observed that the measured results may fluctuate from day to day. However, when aggregated over 3 days, the results are much less prone to fluctuation, providing a more accurate overview of VPN performance over that period of time. For cases where the speed test tool has to run for multiple days, we recommend configuring it to run until manually stopped. However, there may be cases where it is more optimal to have the speed test tool to run for a limited time only. For example, if the tool is configured on a VPS with a bandwidth limit, keeping it running for longer periods of time may impose bandwidth overage charges or restrictions. Even a single test may use up to 7 GB of data. In cases like that, the client should be configured to operate for a limited number of cycles. This can be done using the mode setting in the configuration file. If the value is set to continuous, the tests will run until manually stopped. Alternatively, the once value can be used to stop tests once the entire cycle is complete.

Test regularity

Nevertheless, constantly running tests is not enough for a meaningful comparison between VPN providers. Ensuring an equal number of measurements across all the set testing conditions is crucial as well. This is important when configuring the tool to test multiple different server locations, VPN providers, and VPN protocols. Consider a test configuration being run every 3 minutes targeting 4 VPN providers, 10 different server locations, and 2 different VPN protocols. A single cycle to test all of the possible cases would take 4 hours (240 minutes). The configuration being tested cannot be selected randomly, as otherwise, it may fail to gather significant data to make a comparison. Because of this, our tool is designed to ensure that an even number of tests is performed for all set testing conditions.

Test environment

As mentioned before, VPN speeds may be affected by various software running on your device. Not only that, even other VPNs may cause issues with performance or even prevent you from connecting. Because of this, we’ve designed the speedtest tool on a Docker. A new Docker image is created for each test so that the test results are not affected by the system-level changes that some VPNs may cause. Moreover, the Docker image has no unnecessary software running on it either. That means that all the resources of the machine can be dedicated to the VPN being tested. Additionally, the Ookla CLI speedtest client mentioned earlier is compatible with Debian Linux distribution, which we can run on Docker.

Test server hosting

The tool uses VPSs from 2 different cloud hosting providers, Vultr and Linode, to ensure that the results are valid if there is no significant difference between the speeds from both VPS providers.

The VPS VM specifications at both Linode and Vultr are as follows:

• 2 CPU Cores;

• 80 GB Storage;

• 4 GB RAM.

Using these hosting services also lets us run tests from multiple different locations for a more detailed analysis of VPN provider speeds. Here is a list of locations we can run tests from:

• Sydney, Australia;

• Toronto, Canada;

• Paris, France;

• Frankfurt, Germany;

• Tokyo, Japan;

• Amsterdam, Netherlands;

• Singapore;

• Stockholm, Sweden;

• London, UK;

• Atlanta, US;

• Chicago, US;

• Dallas, US;

• Los Angeles, US;

• Miami, US;

• New York City, US;

• San Francisco, US;

• Seattle, US.

If you’d like to see the results before setting the test up for yourself, we upload weekly speed tool test results here.

Considerations

While using third party tools like Ookla is great for their accessibility and simplicity of implementation, they also introduce some issues stemming from limited control over aspects of the tool itself. Here are some things to consider when using it:

IP address geolocation inconsistencies

Firstly, the most common issue we observed is the inaccuracy of their GeoIP database. As far as we can tell, Ookla uses a MaxMind database to identify the client’s location based on the IP address the client is running their speedtest from. While MaxMind is one of the most accurate GeoIP databases out there, it still takes time for them to update the IP information if it is incorrect. It then also takes time for Ookla to update their own database.

This issue is actually relatively common among VPN services. Due to the ever decreasing availability of IPv4 addresses, VPN providers (or the data centers they run their servers at) can experience issues when expanding their server counts. There may not be sufficient IPs available with the correct GeoIP information for their chosen location. In this event, a new subnet gets announced by the DC the VPN server will be hosted at. That new subnet may have previously been announced in another city or even country, and this old, inaccurate information may still be reflected by various GeoIP databases like MaxMind. Fortunately, Ookla’s recommended speed test server is usually located in the same country as the selected VPN server, which can reduce the magnitude of these potential inaccuracies.

However, this issue becomes more prevalent when we start looking at city information. There are a considerable number of IPs that have incomplete GeoIP information. For instance, an IP may have information that it was announced in the US, but the city information is missing. In that event, Ookla recommends speed test servers located closest to the center of the country. In the US, for example, Ookla suggests testing the internet speeds to their server in Kansas when the IP is missing city information. We can actually use this knowledge to see which VPN providers have IPs with possibly inaccurate GeoIP information on MaxMind. In the table below, we’ve extracted tests performed 10/01 – 10/29, 2021, where VPN servers in the US were chosen and Ookla servers in Kansas were selected:

Inconsistent test server speeds

Though rare, there may be cases when the selected Ookla server’s speed does not exceed 1Gbps speeds or even less. Unfortunately, Ookla does not show which of its speed test servers are on 10 Gbps or faster internet connections and may occasionally select an underperforming test server. This issue can be seen when we check VPN A speeds in Frankfurt using their proprietary protocol from servers hosted with M247 (results from the speed test probes in the UK, NL, FR, DE):

As the image shows, the median download speeds with the lg.fra2-c.fdcservers.net and speedtest.avrx.de Ookla servers are way above 1Gbps. However, the speeds to speedtest.sewan-group.de are considerably slower. In this case, the SEWAN speedtest server has failed to provide sufficient bandwidth to properly test VPN speeds. Luckily, cases like that are relatively rare and can usually be ignored – especially if the median is used to analyze the results.

VPS CPU load resource availability

Since the speed test tool runs on a VPS, some of the resources have to be shared with other systems on neighbouring VPSs. The speed test tool uses a considerable amount of CPU and bandwidth resources in intervals and, unfortunately, VPS providers’ fair usage policies may limit the resources assigned to the VPS running the speed test. This issue can manifest itself as a steep decline in performance when analysing the results of a singular VPS that the speed tests are running on. Here is an example of one of our test servers getting rate limited:

Also, note that the speed tests are running on a Linux system. Network speed and hardware resources are the main factors affecting the VPN performance, so the tests should reflect similar performance on different operating systems, but at the kernel level, networking stack differences may result in a slightly better performance on Linux than other OSs.

All these cases should be considered outliers and they are not exclusive to VPN speed testing. They impact all real-world data. The outliers will be observations that significantly differ from other observations (they could be drastically higher or lower). Outliers can strongly impact the average on smaller data samples, which is why we recommend removing the top and bottom 3 percent of data (6 percent in total) when tests are run for at least 3 days. Smaller amounts can be excluded for shorter time periods. This task can easily be handled by our speed test tool when generating a report for all of the data acquired during the testing period.

Report evaluation

All the accumulated data can be converted into a detailed visualized report. This can be done using the vpnspeed report command. The aforementioned data outlier removal feature can also be done using this same command by adding an –outliers flag. For example, the vpnspeed report –outliers 0.03 command, where 0.03 indicates the percentage of worst/best data to ignore, would generate a report discarding the top and bottom 3% of data.

The data report will not only remove the outliers, but, as mentioned above, will also generate images showing the number of tests per hour, the arithmetic mean (further – mean) and median download speed results, percentile graphs, statistical significance tests, and median winner results.

When reviewing the test results, we should first review the hourly histograms to make sure that test counts for each hour are the same or similar (there can be some deviations). If the number of tests at a specific time is drastically lower, this may result in inaccurate test results that are biased towards the time of the day when most of the results were gathered. The image below shows an expected distribution of tests with occasional increases or decreases in the number of tests that were completed. These deviations are perfectly fine and should be expected.

Test distribution over time

Two other types of graphs generated by the report are mean bar and line graphs. The mean bar graph shows the average download speed grouped by VPN provider and VPN protocol, while the line graph also groups the data by the day the tests were performed on. This lets us see VPN performance deviations on a day-to-day basis.

Lastly, the most detailed graph generated by the report is the percentiles graph. This shows us VPN performance stability. The bars on this graph distribute measurements across four groups: 5th-25th, 25th-50th, 50th-75th, and 75th-95th percentiles. This graph helps you understand what kind of speed can be expected from each provider. Unlike mean or median graphs that show average or 50th percentile download speeds respectively, the percentile graph can be used to visualize the stability we can expect from the tested VPN providers. The narrower the area for each percentile group, the higher the likelihood of getting similar speeds through multiple VPN connection attempts.

The raw data can also be used to evaluate the results as well. For instance, we can see minimum and maximum values as well as mean and median. Mean shows us the average value of speed tests while median provides the middle value from the sample. The median is less affected by outliers and skewed data and represents a threshold where half of results fall above the median and half fall below.

Moreover, the raw data can be used for a more advanced data analysis like the Wilcoxon-Mann-Whitney test. The Wilcoxon-Mann-Whitney test is a nonparametric test that allows two groups to be compared without assuming that the values are normally distributed. It tests whether one variable tends to have values higher than the other by detecting differences in shape and spread as well as differences in medians.

The Wilcoxon-Mann-Whitney test is performed by pairing and comparing VPN providers. If the difference shows significance, then we pick the winner of the pair using the median (higher median wins). Otherwise, the result will be marked as undefined (no statistical difference in that pair). Finally, we count how many times each VPN provider has won in the selected location. The one with the maximum wins is declared the winner. Each statistical test requires at least 20 speed tests to be completed. In addition, we calculate the winner graph for speed medians. This graph finds winners by taking only the median speed in specific locations into account and then presents the percentage of wins from the total number of locations. Median and statistical significance graphs will have different results. Consider a scenario where one VPN provider has a median download speed of 700 Mbps and the other’s is 705 Mbps. There would be no significance between them because the difference between medians is very small. However, the median winner graph will consider the first VPN as the winner. The image below shows the percentage of Wilcoxon-Mann-Whitney test and median winners per tested location.

The Conclusion

By releasing the source code of the speed test and comparison tool, we want to invite all parties to participate in creating a universal, unbiased and transparent standard to measure the performance of different VPN service providers. The tool has not yet been reviewed by a broader audience, so we welcome all constructive criticism on the code and/or methodology that will help make the tool better. Hopefully, the widespread adoption of a single methodology will lead to a more equal competition ground for VPN providers and will help users make informed decisions.