Server Updates, Anycast, Steps going forward

[imasonaz]

Many of you may have noticed some changes lately, some downtime, and some updates, and I figured it was time we explain what all has been happening in HG, what it means for you all, and where we go from here.

First, you've probably noticed the new fourms, we upgraded our forums to Invision Community 4.6 from 4.4, which was last updated around 2019.
This brought some challenges in finding updated and working plugins, theme changes, and other look, feel, and management changes. We hope you enjoy the new layout and thank you for your patience while we work out all the other minor kinks around this.

Second, we have been doing major back-end consolidation and overhaul of our servers. HG leases 4 physical servers, collocates 2 HG-owned servers, and leases a handful of various VPSs.
Part of our forum update was moving to a new web host - we consolidated all our web servers into 1 collocated HG-owned (donated) dual-processor server. This gives us a ton more flexibility in what we're able to do, reduces our costs in infrastructure, and allows for potential future growth. Additionally, we've been working to consolidate game servers. We've worked to move CSGO Jailbreak to our Ryzen server, as well as working to move our "active" live servers to Ryzen and winding down our less active servers so we can concentrate on making a couple really high quality servers in each division. In the future, this will mean us retiring one OVH server, and since CSGO Jailbreak was on a Chicago server by itself, we have now been able to cancel that server as well. This will bring our physical servers down to 1 leased, and 2 HG-owned, with no VPSs.

Lastly, we need to talk about Anycast - that word that many of you have heard, but may not understand or know how it affects you.
We have previously partnered with GFL to add a number of our servers to Anycast, and have seen huge success with that project. While this may be a little controversial, we think that it is best for HG, our players, and the servers we manage to invest more in this and concentrate our live servers on our Anycast network. For those who aren't completely aware, Anycast is kind of a spoke and hub - we have a number of smaller servers around the world that act as responders for our main server. These smaller servers cache the reply from the hub server to send to clients when they are looking for a game. The side effect of this is that when a player is searching for a new server, our servers pop up quickly at the top of their list, because those near-by servers reply much faster than a server across the planet. This often has the side effect of users thinking our server has lower ping for them, but in reality they might be connecting from Australia to the US. While this isn't entirely ideal, we have seen the results, and taken notice of other communities doing this, while we need to compete with them to keep up and keep our servers from being empty at the bottom of the server search.
With that understood, we plan to move from GFL's Anycast to our own that we own and manage, which will allow for things like implementing our own customer DDoS solution, as many DDoS solutions utilize Anycast as well. We have been working to raise funds to start this, as well as reviewing our infrastructure and number of servers to narrow down which would benefit most from being moved to Anycast.

 

Here's some light Anycast reading:

https://www.cloudflare.com/learning/cdn/glossary/anycast-network/
https://en.wikipedia.org/wiki/Anycast
https://www.keycdn.com/support/anycast
https://gflclan.com/forums/topic/36839-what-is-gfls-network/

Overall, we think these changes should improve your experience in using our platforms and servers, while keeping our costs down and continuing to bring in fresh players to our servers.

As always, I'm happy to hear any questions, concerns, or feedback you might have about any of this.

[FERAK]

cool

[myes]

thanks for letting us know what's been going on recently can't wait for the future of hg

[ToxicTurd]

Thanks for informing us! 

 

[Snowyamur]

Thanks for informing us imason. Appreciate the transparency regarding this

[Zeused]

15 hours ago, imasonaz said:

The side effect of this is that when a player is searching for a new server, our servers pop up quickly at the top of their list, because those near-by servers reply much faster than a server across the planet. This often has the side effect of users thinking our server has lower ping for them, but in reality they might be connecting from Australia to the US. While this isn't entirely ideal, we have seen the results, and taken notice of other communities doing this, while we need to compete with them to keep up and keep our servers from being empty at the bottom of the server search.

hg shouldnt be using anycast if it fakes ping to get higher on the server browser. idk much about anycast but it sounds a bit scummy imo

Edited August 11, 2021 by Zeused

[SPlink :)]

34 minutes ago, Zeused said:

hg shouldnt be using anycast if it fakes ping to get higher on the server browser. idk much about anycast but it sounds a bit scummy imo

1. We've been using anycast for a very long time, I feel like this ship has sailed

2. It's about keeping HG competitive because we know for sure other gaming communities are doing it

3. We launched one of the biggest divisions a short time ago because of anycast (Rust), and without it, I doubt it'd have the success that it has had. 

Eddiecam has informed me that point 4 has not worked, so this point has been omitted.

[Vesta]

29 minutes ago, SPlink said:

1. We've been using anycast for a very long time, I feel like this ship has sailed

2. It's about keeping HG competitive because we know for sure other gaming communities are doing it

3. We launched one of the biggest divisions a short time ago because of anycast (Rust), and without it, I doubt it'd have the success that it has had. 

Eddiecam has informed me that point 4 has not worked, so this point has been omitted.

just throwing this out there. As a player, if I join a server that says it has really low ping, but when I join it performs like garbage, I'm alot less likely to come back to that server (obviously I dont speak for everyone though)

[Jeri :)]

2 minutes ago, Vesta said:

just throwing this out there. As a player, if I join a server that says it has really low ping, but when I join it performs like garbage, I'm alot less likely to come back to that server (obviously I dont speak for everyone though)

Considering we created the Rust division from nothing with players consistently coming back week after week with Anycast I would disagree. Also a lot of these players are going to have a shit connection on the vast majority of  servers and really are just clicking on populated ones that pop on their screen first. 

[Zeused]

2 minutes ago, Jeri said:

Considering we created the Rust division from nothing with players consistently coming back week after week with Anycast I would disagree. Also a lot of these players are going to have a shit connection on the vast majority of  servers and really are just clicking on populated ones that pop on their screen first. 

i think rust was successful because of how well the rust division managed and moderated the server. the rust division probably wouldve done just fine, with or without anycast

[Zeused]

1 hour ago, Sit. said:

Lol no. HG rust division got built up because around 10 people played for like 2 wipes, anycast did the rest. The only moderator was spawn, who wasn’t around 24/7. It’s okay to question the morality of HG’s decision to use anycast, but the results speak for themselves.

anycast just makes people find the server. if the server wasn't fun, people wouldnt play on it. the tf2 anycast server has been dead since day 1, and even some of the csgo servers have been getting lower population (granted this could be because summer is almost over lol). obviously anycast makes a huge difference, but hg should be more focused on improving the actual server rather than using methods to fake ping. it just gives hg a bad name in general

 

2 hours ago, SPlink said:

1. We've been using anycast for a very long time, I feel like this ship has sailed

2. It's about keeping HG competitive because we know for sure other gaming communities are doing it

3. We launched one of the biggest divisions a short time ago because of anycast (Rust), and without it, I doubt it'd have the success that it has had. 

Eddiecam has informed me that point 4 has not worked, so this point has been omitted.

I am positive a lot of players/leadership are not aware that anycast fakes ping. i remember thinking the low ping was an issue when anycast was used in tf2, and now i am kind of disappointed that it was intended. just because other communities fake ping doesn't make it justifiable. facepunch has taken down/blacklisted servers before for ping spoofing... even if valve and facepunch don't pay attention to community servers much anymore, hg still shouldn't be doing this

Edited August 11, 2021 by Zeused

[Seymour Butz]

 

[imasonaz]

46 minutes ago, Zeused said:

 

1. It doesn't fake ping
2. Anyone NA will not notice any difference between the ping they see and their experience in the server
3. I'd strongly argue the issues with the 2fort server were due to other things that we didn't do right
4. Yes, our goal is to be at the top of the server list. There's a huge number of game server (especially in rust) that do this.

My secondary purpose in this post is to talk about how we plan to stick to high-quality servers our players love - and that will make players stay for the content, even if the ping might be slightly higher than they hoped.

We have some control over where we are shown with low ping by selecting where we put Anycast servers, and as we move to our own servers, we plan to look at where we would like to target most. We have no plans on moving away from Anycast at this time, so please try to keep the conversation fact-based and constructive.

[Eric]

7 hours ago, Sit. said:

Lol no. HG rust division got built up because around 10 people played for like 2 wipes

Shoutout td. Always supporting the boy spawn : )

[Zeused]

5 hours ago, imasonaz said:

1. It doesn't fake ping
2. Anyone NA will not notice any difference between the ping they see and their experience in the server
3. I'd strongly argue the issues with the 2fort server were due to other things that we didn't do right
4. Yes, our goal is to be at the top of the server list. There's a huge number of game server (especially in rust) that do this.

My secondary purpose in this post is to talk about how we plan to stick to high-quality servers our players love - and that will make players stay for the content, even if the ping might be slightly higher than they hoped.

We have some control over where we are shown with low ping by selecting where we put Anycast servers, and as we move to our own servers, we plan to look at where we would like to target most. We have no plans on moving away from Anycast at this time, so please try to keep the conversation fact-based and constructive.

 

 

On 8/10/2021 at 3:08 AM, imasonaz said:

 The side effect of this is that when a player is searching for a new server, our servers pop up quickly at the top of their list, because those near-by servers reply much faster than a server across the planet. This often has the side effect of users thinking our server has lower ping for them, but in reality they might be connecting from Australia to the US.

 

you say its not faking ping, but in the original post you talk about how people far away from the server still get low ping. isn't showing the incorrect amount of ping in someone's browser literally the definition of faking ping? sorry if i came off as a bit aggressive, I am just trying to learn more about anycast and the ping issue, and it kind of grinds my gears since hg is possibly abusing this. if the ping was relative to the physical server, then there really wouldn't be a moral issue, but from my POV it seems like hg will be using anycast just for the low pings. here is a good read i found for anyone interested: https://docs.google.com/document/d/1-5f_TlJEki_qJYvZvlZ37QdsDambYIpEgPOUMP2RK9A/edit

gfl has been blacklisted by facepunch for ping spoofing before according to this thread https://gflclan.com/forums/topic/59681-anycast-expansion-plansapproaches/

hopefully i am just wrong about everything and hg will have normal ping from the physical server. not like i can really test it myself lol

Edited August 11, 2021 by Zeused

[Liloz01]

Facepunch don't give a fuuuuuuck anymore about A2S_INFO caching... they used to; not anymore now that all of the biggest communities do it, also they never had a problem with "Anycast" they had a problem with "A2S_INFO caching"... I'll touch on that a little bit later (because it annoys me when people talk as if they are one in the same). As for other publishers, Valve never gave a fuck, when have they about community servers? There's an incredibly, incredibly low chance that HG will become blacklisted because it would mean also blacklisting (almost) every large community out there (there's some exceptions like creator-owned/backed servers that are huge but do not use caching).

"Anycast" and "A2S_INFO caching" are not exactly synonyms... No, Anycast does not inherently introduce "fake ping" in the server browser, A2S_INFO caching does. In this context, "fake ping" refers to a mismatch between server browser "ping" and latency to the game server when you are connected and playing due to the PoP from which A2S_INFO is received being located far away from the game server. The document quoted below explains this in far greater detail.

On 8/11/2021 at 11:22 AM, Zeused said:

https://docs.google.com/document/d/1-5f_TlJEki_qJYvZvlZ37QdsDambYIpEgPOUMP2RK9A/edit

Yep... it's scummy - but it's basically necessary nowadays. I'm sure HG higherups hold reservations about using this technology, as did we. Heck, GFL was one of the main proponents of getting servers who use A2S_INFO caching as a form of "legitimate ping spoofing" blacklisted off the face off the earth by Valve back in the day (https://gflclan.com/work/projects/misc/contact-facepunch-and-valve-regarding-anycast-in-garrys-mod-r24/). 

In short: an anycast network with many PoPs in different locations with A2S_INFO caching enabled is, with very few exceptions, a necessary evil right now. 

Edit: What is the specs of this (quoted below) machine? I'm curious what you got donated as a dual processor machine haha.

On 8/10/2021 at 11:08 AM, imasonaz said:

1 collocated HG-owned (donated) dual-processor server

 

Edited August 12, 2021 by Liloz01

[KB]

I have nothing intelligent to add to this conversation other than it looks...

[imasonaz]

On 8/11/2021 at 6:20 AM, Liloz01 said:

Edit: What is the specs of this (quoted below) machine? I'm curious what you got donated as a dual processor machine haha.

I donated a Dell PowerEdge r420 with dual Xeon E5-2470, 128GB RAM and 6x500GB SSD in RAID 6+1. HG bought the RAM and we split costs on the SSD's. It's not the fastest out there, but it runs ESXi and all our services just fine with 8c/16t x2

[Emboss19]

I don't know if this matters much, but I hope that tf2 trade 1 server doesn't get perm shut down... I spend almost every day in that server and have been playing with friends there for a while. While I am excited for the future of hg, I hope this doesn't affect that server.

[imasonaz]

34 minutes ago, Emboss19 said:

I don't know if this matters much, but I hope that tf2 trade 1 server doesn't get perm shut down... I spend almost every day in that server and have been playing with friends there for a while. While I am excited for the future of hg, I hope this doesn't affect that server.

Trade 1 has been gone for months now....