Should We Update the ZS Server?

[Snowyamur]

In the past few days during the ZS Spooky Holidays Event, the HeLLsGamers Zombie Survival server has been experiencing more lag than ever. These fluctuations are a normal, yet unfortunate, part of the server, however, they've become more apparent as players have complained numerous times about how bad the lag on the server is.

 

In light of this ongoing situation and the possibility of false belief, I would like to put a few minds at rest on this issue, what it actually is, and what it's true cause is, with permission from @Thomasjosif to do so.

 

The Problem

For many players who've been around on the HG ZS server for quite a while, many of you have likely known that lag existed. It's not afraid to show up on the server, and when it does, it makes a server-wide impact that results in complaints, players leaving, memes, and jokes about HG not doing anything about it.

 

We currently are working on the situation, but it's not as simple as we initially perceived. It turns out that the issue involves not our server box, but in actuality, parts of the code and scripting that's integrated into the ZS game mode, as well as how it works with Garry's Mod itself. Some snag in the coding results in the server being incapable of handling large amounts of activity, and that results in it negatively-reacting to such large influxes of activity.

 

However, some may ask:

Isn't it the server's CPU that's causing the problem? The server only runs on one CPU, and code and scripts are processed through it, so wouldn't updating the CPU solve the problem?

 

Turns out, it's not what we initially believed.

 

The Reality and The False Belief

There has been an ongoing rumor that I, personally, didn't know existed, and that involves how HG's servers run only on one CPU processing core. From my talk with @Thomasjosif, this turns out not to be the case. As of last year on a similar thread made by @ToF, @Sirius confirmed that our servers are running on more than one processing core; the servers are actually running on 12 cores. This is quite a huge monetary expenditure because maintaining all 12 cores and running them day-and-night takes power and money to keep them in operation, thus HG is actually investing a lot of money into what we enjoy doing every day.

 

The chart below is a 2017 report on the processing core activity for running most of HG's servers:

Based on these results, not much CPU is being used. According to the chart, less than 50% of the server box's CPU is being used to run all servers, and the ZS server is no exception. Because of this, the CPU is not entirely at fault for what we initially believed to be the reason the ZS server lags too much.

 

Additionally, information from the following thread below further proves this case as to the frequent lag spikes:

As of now, The False Belief of our servers running on one CPU core is now settled, and to put it at rest, we must consider that there is a more underlying issue with the ZS server, and that has to be with its code. That''s the true problem and cause of why the ZS server frequently lags; the code becomes more and more incompatible with how Garry's Mod operates their servers, and because of this, it's becoming difficult to handle greater influxes of activity without crash.

 

The Solution and the Long-Term Resolve

For those of us who care about the ZS server and want to, in some way, see it resolved, improved, and fixed, we have a few options.

 

It's possible that receiving donations and monetary contributions could allow us to look for professional help on this situation and hope to get it resolved, but the help that may be required at best, given what we might have available, is to have people who wish to see this problem resolved contribute to the programming-aspect of our ZS server and work with Forrest to resolve this problem.

 

Alongside this, in-terms of long-term improvement of the server, people can choose to contribute as game designers, who will work with both me and Forrest on providing balanced, higher-quality updates that integrate smoothly with ZS's current mechanics, utilities, and operations.

 

If neither option is chosen, players can continue to play on our server, despite the lag, and that supports the ZS server itself nonetheless.

 

However, something must be done about this, or we lose the server that many have played on for years.

Edited December 24, 2018 by Snowyamur

[Aeux Deus]

i'd love to contribute any way shape or form, already submitted my answer, and when does the contributions start?

[Snowyamur]

@Aeux Deus

 

Currently, we're planning on contributions. As of when, they start, that's to be announced (TBA) when we have the plans set up and ready to go.

[Aeux Deus]

1 minute ago, Snowyamur said:

@Aeux Deus

 

Currently, we're planning on contributions. As of when, they start, that's to be announced (TBA) when we have the plans set up and ready to go.

Sounds Great, i'll be waiting, ill get paid the 1st of january then i might donate 30-40 $

[Snowyamur]

Quote

Sounds Great, i'll be waiting, ill get paid the 1st of january then i might donate 30-40 $

@Aeux Deus

 

I would suggest holding off on donating directly through the HeLLsGamers Store for now because it's likely we might do a Kickstarter or GoFundMe approach with this. It's likely; I'm not sure if this is the guaranteed approach we'll take.

[Thomasjosif]

Nothing has changed from this thread ~a year ago

 

[Lady Luck]

Would love to help with design, but me an forrest never see eye to eye xD. (not counting the times i one hit killed him and got a zombie or boss nerfed)

Ima stick to just being here until steam banns this ip and i move to my next alt =D

[Snowyamur]

In light of what Thomasjosif said and the talk we've had, the information above has been changed and updated accordingly to reflect concrete information gathered and to put the "one-CPU running" rumor to rest.

[ForrestMarkX]

It's not the code, HGs serverside code has been optimized through the roof and is roughly 4% to 8% faster than the latest ZS codeset code (Sunrust doesn't lag at all with 70 players but HG runs like shit at 60). The CPU usage chart shows global CPU usage and not singular core usage during times of high load.

Edited December 24, 2018 by ForrestMarkX

[Snowyamur]

@ForrestMarkX

Okay, so who do I believe here? The Programmer or the Executive Council? We need to place this conflict at rest, because if we don't, it's going to compound, and when problems compound, they become permanent dilemmas that are out of our power to resolve.

[ForrestMarkX]

All the profiling I've done ingame doesn't tell lies. Everytime the server lags into oblivion it isn't network choke it's cpu choke which means the CPU the server is running on is not fast enough to handle the work load. I will not attempt to further this discussion, the code is optimized as far as humanly allowable, there is nothing more on my side that is possible to do.

Edited December 24, 2018 by ForrestMarkX

[Snowyamur]

I'm just going to say here and now, but there are problems that are unsolved, and unless we do something about them, they're going to compound and get much worse. I'm willing and trying to do something to help, but it seems I can only do so much with what power I have and with what I'm given.

[Lady Luck]

You could go to where the server is and add a block of cpu s to it xD.

And yes, this is a joke. Like Wifi cables.

Edited December 24, 2018 by Lady Luck

[ForrestMarkX]

There 2 things needed for a 70+ player server to run good at 33 tickrate.

1. A CPU that has a high rated single core performance (HG CPU is ranked around #60 to #80)

2. A fast processor speed (Single core speed works since gmod is single threaded) (I was told the CPU is 2.8GHz so I'm assuming all the cores run at this speed) (2.8GHz is nowhere near fast enough to handle 33 tickrate at 70+ players)

Solutions:

1. Upgrade the CPU to a better rated single core performance CPU or overclock a single cpu core to 4.5GHz then force gmod to run on that core only

2. Lower the server player count

3. Lower the server tickrate

Edited December 24, 2018 by ForrestMarkX

[Snowyamur]

Quote

There 2 things needed for a 70+ player server to run good at 33 tickrate.

1. A CPU that has a high rated single core performance (HG CPU is ranked around #60 to #80)

2. A fast processor speed (Single core speed works since gmod is single threaded) (I was told the CPU is 2.8GHz so I'm assuming all the cores run at this speed) (2.8GHz is nowhere near fast enough to handle 33 tickrate at 70+ players)

Solutions:

1. Upgrade the CPU to a better rated single core performance CPU or overclock a single cpu core to 4.5GHz then force gmod to run on that core only

2. Lower the server player count

3. Lower the server tickrate

I think this could work. Can we get support for this solution please?

[ATB]

i mean what happened to the 2000 we had donated to perp? We never worked on perp or had any money actually put to use so where is it.....how about we use that money to upgrade or downgrade the servers to a lower pop like 60 which is what we used to have it at back then.  What makes it even more funny is that the servers used to run actually well back then with 70 people now we can barely run at 60 as the server is lagging while we speak,  because clearly we can't ask sunrust zs to help out with using their code for optimization unless someone is willing to approach them with offers and trade to have good optimization like they do. 

Edited December 26, 2018 by ATB

[Papa John]

Not that I particularly play Garry's Mod anymore, but this discussion has been an ongoing one almost since we started Zombie Survival. At the end of the day, Garry's Mod was never designed to work with this many players. They updated to the Orange Box engine and some of the subsequent updates but Garry's been upfront about the fact that this wasn't really a long-term solution. Forrest can start removing more resource intensive features like particle effects if you want marginal increases in FPS but nothing is going to fix it. Even getting a top of the line, single thread, commercial CPU likely wouldn't solve the issue.

tl;dr If you're looking for peak performance in a game, you're not going to find it in a title created in 2006.

[Snowyamur]

Quote

Not that I particularly play Garry's Mod anymore, but this discussion has been an ongoing one almost since we started Zombie Survival. At the end of the day, Garry's Mod was never designed to work with this many players. They updated to the Orange Box engine and some of the subsequent updates but Garry's been upfront about the fact that this wasn't really a long-term solution. Forrest can start removing more resource intensive features like particle effects if you want marginal increases in FPS but nothing is going to fix it. Even getting a top of the line, single thread, commercial CPU likely wouldn't solve the issue.

tl;dr If you're looking for peak performance in a game, you're not going to find it in a title created in 2006.

@Papa John

It doesn't hurt to try to improve though, does it? I mean, we can take a chance and a risk at trying to improve something old, and who knows? If it improves, great for us. If it doesn't, at least we tried something instead of giving up and saying, "Garry's Mod is an old game title; improving any software attributes that it uses up won't improve a user's performance on one of the game's servers."

 

It's worth a shot.

[Papa John]

Quote

It doesn't hurt to try to improve though, does it? I mean, we can take a chance and a risk at trying to improve something old, and who knows? If it improves, great for us. If it doesn't, at least we tried something instead of giving up and saying, "Garry's Mod is an old game title; improving any software attributes that it uses up won't improve a user's performance on one of the game's servers."

It's worth a shot.

@Snowyamur

At this point it's not really a matter of trying, we've done that repeatedly over the years. It hasn't worked. To put it in context:

• 2013
• 2016
• 2017

Naturally the number of people talking about it in-game always eclipsed those that took the time to make a thread about it on the forums. With enough time and money, anything conceivably is possible, but HG doesn't have either as it would be a colossal waste of resources for negligible returns. But what do I know, maybe we were just incompetent for each of the numerous attempts we've made throughout the years.

[Snowyamur]

Quote

@Snowyamur

At this point it's not really a matter of trying, we've done that repeatedly over the years. It hasn't worked. To put it in context:

• 2013
• 2016
• 2017

Naturally the number of people talking about it in-game always eclipsed those that took the time to make a thread about it on the forums. With enough time and money, anything conceivably is possible, but HG doesn't have either as it would be a colossal waste of resources for negligible returns. But what do I know, maybe we were just incompetent for each of the numerous attempts we've made throughout the years.

@Papa John

Thank you for responding and providing contextual evidence that this has been an ongoing issue for a few years.

Based on what you provided, all that I can see that we've tried were

• reducing the server population
• updating the network strength of the server (I'm not sure how you guys accomplished this)
• reworking some features on ZS to produce less lag

From these, I don't see anything that shows that we tried updating the CPU of our servers specific to Garry's Mod, let alone all servers, unless that's what you guys have done for updating the network strength. If we haven't made an attempt to update the CPU to reduce the lag on the Garry's Mod Zombie Survival server, why not try now?

I'm well-aware that this would cost money, and I'm aware that this may/ may not work, but again, why not give it a try? If it didn't work in the end, it wasn't an entire waste of money because we would all walk away knowing that upgrading CPUs can't resolve frequent lag fluctuations on online gaming servers, and this is technical knowledge. Also, we would still have a spare and powerful CPU in-case one of our server's CPUs blew out.

I don't think we've attempted updating the CPU, and according to @ForrestMarkX, who's our main programmer and developer for Garry's Mod, he's our only direct and possibly most reliable source of information on this. I'm only suggesting this because I care about the HeLLsGamers community, and I want to see things improved, but most importantly, I want to see ongoing problems be resolved at last so that we may move on to further ventures.

[Papa John]

Quote

Thank you for responding and providing contextual evidence that this has been an ongoing issue for a few years.

Based on what you provided, all that I can see that we've tried were

• reducing the server population
• updating the network strength of the server (I'm not sure how you guys accomplished this)
• reworking some features on ZS to produce less lag

From these, I don't see anything that shows that we tried updating the CPU of our servers specific to Garry's Mod, let alone all servers, unless that's what you guys have done for updating the network strength. If we haven't made an attempt to update the CPU to reduce the lag on the Garry's Mod Zombie Survival server, why not try now?

I'm well-aware that this would cost money, and I'm aware that this may/ may not work, but again, why not give it a try? If it didn't work in the end, it wasn't an entire waste of money because we would all walk away knowing that upgrading CPUs can't resolve frequent lag fluctuations on online gaming servers, and this is technical knowledge. Also, we would still have a spare and powerful CPU in-case one of our server's CPUs blew out.

I don't think we've attempted updating the CPU, and according to @ForrestMarkX, who's our main programmer and developer for Garry's Mod, he's our only direct and possibly most reliable source of information on this. I'm only suggesting this because I care about the HeLLsGamers community, and I want to see things improved, but most importantly, I want to see ongoing problems be resolved at last so that we may move on to further ventures.

@Snowyamur

Unless something has changed recently you need to understand we don't actually own these servers, we rent them from cloud hosting providers. While this varied based on the server, based on what I can recall from my time here we were using the company Linode for our Garry's Mod server (As some context I used to be a Division Leader for Garry's Mod and headed up ZS development before I brought Forrest on after he was recommended by someone). Now if you take a look at the pricing page you'll notice that they tell you how many CPU cores you get and you don't have the option to alter the clock speed of those CPUs. This is because, for the most part, modern software design prefers to scale using the number of cores over any single core's clock speed (i.e. multi-threaded computing). Now for Garry's Mod, this more or less doesn't work, as the LUA interpreter in Garry's Mod only can operate on one core. Linode isn't going to buy a new CPU just for us, it doesn't make any financial sense for them to do so. So we would have to move over to a larger provider like AWS and get something like their EC2 z1d Instances which at a minimum cost around $200 /month for the lowest spec instance (Garry's Mod probably still requires Windows to run the server). What HG can and cannot afford is a Homer question, but if we "try it out" and it works but we can't afford it, then what did we actually accomplish? Building your own server, in addition to the upfront cost,  requires that you pay around $200 /month collocating in an existing data center, so that is out as a possibility. I'm not trying to be pessimistic and I appreciate that you want to help the server, but its a lot easier to say than to do.