TL;DR

We have recently enabled a new download location for Premium Members that can potentially increase download speeds for any users who might have slower speeds than expected with their Premium Membership. If you're a Premium Member and you think you should be getting faster speeds then please try the "CDN" download location, which now defaults to this new provider.


The long version

If you’ve had the (mis)fortune of taking the time to read my year in review post recently then you’ll be aware that we move a fair amount of data to our users - we estimate around 60PB per year - through our download servers.

We are also storing a large amount of data - we’ve just ticked over 40TB of mod files and this is increasing at around 1.5TB per month currently as the number of mods we host increases and the size of the mods authors are making are growing.

How we store and serve this amount of data on a budget is a challenge we're always tackling as we continue to provide a "free" service to our users. Free is in quotes because, as you know, there are ads on the site and these generate revenue used for paying for the services we provide. We don't sell user data or anything of that ilk, so our two revenue streams are straight-up advertising and Premium Memberships.

To help us manage our file serving needs we created our own mini-CDN, of sorts, a few years ago which uses bare-metal servers in 7 different locations around the world; Paris, Amsterdam, Prague, Chicago, Los Angeles, Miami and Singapore. These servers are kitted out with NVMe drives and each server has 20gbps of connectivity. While we average around 25gbps of total traffic (across all our servers) on any given week, the recent Fallout Frontier release proved the servers could handle these theoretical maximums without a sweat as our Chicago server alone managed to briefly hit that 20gbps mark during the initial hours of the mod's release. This custom-built CDN system is significantly cheaper to own and run for us than any CDN services that are available online. The caveat to this is that while our users in North America, Europe and parts of Asia and Oceania will have good download speeds from our servers, users from other areas of the world can potentially have poorer download speeds especially if their local ISP's infrastructure has bad connectivity to our CDN.

In layman's terms, the difference between good connectivity and bad connectivity is like the difference between a good sat nav for your car and a bad sat nav. A good sat nav knows when you are travelling, when and where traffic can bottleneck at different times of the day, where any roadworks are, where any newly constructed roads are and will direct you from A to B in the most efficient way possible which will significantly decrease your travel time and increase the speed at which you get to your destination. A bad sat nav does not know any of these things and can potentially take you through horrific traffic conditions, road works, bottlenecks and not make use of the newly built roads meaning it can take you significantly longer to get to your destination. Thus, good connectivity ensures our data packets reach your PC in the most efficient, fastest way possible ensuring good download speeds. Bad connectivity can send our data packets via bad cabling, overloaded networks and a myriad of other negative networking issues meaning you get considerably slower download speeds.

To clarify, we do not cap our Premium service in any way, shape or form. Whatever speed a user gets from our download servers is the maximum uninhibited speed possible between our server and your computer - if it's slow, it's almost certainly because the traffic is not being routed well by either our ISP or your ISP but, unfortunately, that is not something we have control over to fix.

An example of this is here in the UK where on a basic 75Mbit home connection if I try to download from our CDN node in Los Angeles, some 5,400 miles away, I can reach download speeds of 6MB/second without any issues. However, some of our users in Brasilia, a similar distance to LA, will get far lower speeds than this to the same LA location. This is due to the connectivity between the ISPs in the UK compared to the connectivity of some ISPs in Brazil. Essentially, there is more cabling and better management of the network between North America and Europe than there is between North America and South America. Unfortunately, this is something which is completely out of our control. The solution to this problem is to hope and pray the connectivity between North America and Brazil improves (which involves major ISPs laying more cables to provide more bandwidth and/or for them to improve how they route their traffic) OR for us to requisition more servers that are more local to South America. However, hardware costs, data centre costs and bandwidth in many areas of the world can oftentimes be many magnitudes more expensive than it is in North America and Europe meaning it's not financially viable to do this. For example, the bandwidth costs with our provider to South America or Asia costs 223% more than it does from Europe. From Australia it's 312% more. Unfortunately it is simply not sustainable for us to set up additional nodes in these locations.

As our user base increases, so do the number of users who are becoming Premium Members, and it is becoming an increasingly sore spot for me that some users have not been getting good download speeds when paying for our Premium service, irrespective of whose fault it is.

To remedy this we recently decided to spend a not-insignificant sum of money on an additional CDN service specifically for Premium Members. The CDN we have settled on contains 51 download location nodes, considerably more than the 7 nodes within our home-made CDN. Node locations include 2 in South America, 3 in the Middle East and 15 in Asia and Oceania. You cannot pick the specific download location within this CDN, the CDN will pick the node nearest to you, however, our existing CDN locations are still available for you to choose if you so wish. While we would love to roll this out to normal users too it simply isn't financially viable, so our "old" CDN will remain the service used for non-Premium users (and any Premium Members who wish to use it).

To put it into perspective, we are predicting this new service will increase our file serving costs by 62% per year, and that's just serving Premium Members, so it is not a decision I have taken lightly, and it is not something that would have been possible until more recently as our user base has increased.

If your download location is currently set as the default "CDN" option then you are using our new download service. If you have previously manually set your download location (either in your site preferences or by manually selecting a download location from the download window on a file page) then you will need to choose the "CDN" option when you next attempt to download in order to access this new download service. This new service is for Premium Members only so if you are not a Premium Member you will not have access to these options.

As a part of this update, we are decommissioning the bare metal Signapore CDN node that we currently have (which was Premium-member only) and this should go down in a few week's time. The new CDN service we offer should provide as good as if not better speeds than this node for all our Premium Members in the Asia and Oceania areas and thus it's an added expense for no gain.

We put this new CDN service live last week and since doing so we have not received a single complaint about download speeds, which I'm naturally extremely happy about. I'd be interested to hear from any Premium Members who have seen an improvement in their download speeds on the site since our new service went live. If you're a Premium Member and you're still getting slower than acceptable speeds on our servers please read our help article and, if possible, provide us with the feedback requested via our forums so we can try and help you troubleshoot the issue.

Thank you for your support.