Severe Unknown Server Problem Cs:s

[ELK]

Hi again HeLL's Gamers. I am not a professional server host. In fact I have never hosted an online public server. I learn what I can about source multiplayer networking to give myself an advantage while gaming, and to help server owners if I can. You are having a very strange jitter problem on your CS:S servers.

 

I played on 2 servers. Both had the problem so I assume they all do.

 

[HG] 24/7 Office Deathmatch | 100 Tick | gameME | HeLLsGamers.c

67.228.245.2:27015

 

[HG] 24/7 Dust2 Deathmatch | 100 Tick | gameME | HeLLsGamers.co

67.228.181.72:27015

 

I played on two 100 tick servers not owned by HG and they run fine.

 

 

 

This is why it is strange:

Area 5 of net_graph 4 states that all servers including your own have a frame rate of 100. please view: https://developer.va...2_Network_Graph

The connection between my computer and your sever is experiencing no loss or choke and is uploading 100 commands per second and downloading 100 updates per second, both flawlessly.

My ping stays around 30 milliseconds. Also I am hardwired to my router.

I have a solid 144 frames per second on my computer.

My interpolation time is set to 20 milliseconds to match the 100 rate of the server. The interpolation time, as seen in area 9 of the network graph, never exceeds 20 milliseconds. (THIS IS NOT LERP: [interpolation time] It is lines behind and above the blue area on the very bottom of the network graph)

I have lag compensation enabled.

 

Input prediction is enabled via cl_predict 1

Input prediction smoothing is disabled but there are no prediction errors, as shown by cl_showerror 1

V

V

V

This information basically means my computer and the server are running perfectly. The server and my computer both see me in the correct position and no alterations are being made to my position other than the controls I use.

 

Why is it jittering?

 

 

 

Servers must be running a high precision event timer in order to achieve less than 15 millisecond frame time (66.666... fps [in order to achieve above 66 fps]). The server is getting 100 fps. This means the high precision event timer must be running. Double check if the high precision event timer is running regardless.

 

I am tired and I have a beer so I don't want to think the effect of these next things through. I am saying: I have no idea the effect they would cause. I just don't wish to contemplate their effects in this moment.

 

Are interrupt moderation and flow controlled enabled on the network adapter of the server. These settings are known to cause negative performance in gaming. (do server network adapters even have these settings lol)

 

Are there any other network adapter settings (which can be found in device manager in the properties of the network adapter being used to host the server) which could be affecting the server?

 

Are there any windows related network settings that could be affecting the server? Please view: https://www.speedgui...-8-10-2012-5821

 

Is there problem along the very short route from my computer to your server?

 

Ping statistics for 67.228.245.2:

Packets: Sent = 3872, Received = 3870, Lost = 2 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 14ms, Maximum = 71ms, Average = 20ms

Control-C

^C

C:\Users\MATT>tracert 67.228.245.2

 

Tracing route to 2.f5.e443.ip4.static.sl-reverse.com [67.228.245.2]

over a maximum of 30 hops:

 

1 <1 ms <1 ms <1 ms 192.168.1.1

2 11 ms 7 ms 7 ms 216.45.68.65

3 6 ms 7 ms 7 ms 216.45.95.65

4 22 ms 16 ms 22 ms v503.core1.dal1.he.net [216.66.76.161]

5 19 ms 17 ms 19 ms de-cix.dfw.us.ibm.com [206.53.202.9]

6 20 ms 19 ms 17 ms ae1.dar02.sr01.dal01.networklayer.com [173.192.1

8.213]

7 21 ms 20 ms 19 ms po2.fcr04.sr05.dal01.networklayer.com [66.228.11

8.218]

8 21 ms 19 ms 19 ms 2.f5.e443.ip4.static.sl-reverse.com [67.228.245.

2]

 

Trace complete.

 

C:\Users\MATT>

 

 

 

 

 

I can't pinpoint the problem. Threw some wild guesses. Could you look into the problem? Is there any addition information I could provide or is there any way you want me to help?

 

 

 

 

Attached File: demo of jittering problem.

Example of the directory to place: \Steam\steamapps\common\Counter-Strike Source\cstrike

Enter without quotes into console of counter-strike:source: "playdemo jitter"

jitter.dem

Edited June 24, 2017 by ELK

[.lsd]

I watched your demo, and I have noticed this jitteriness in the server as well, once the servers get their daily restart it is completely resolved, but as the day goes on starting maybe 8 hours after the restart, the jitteriness starts and gets progressively worse, hit reg gets worse with the jitteriness as well.

 

Perhaps the server is getting bloated as the day goes on, memory leaks, etc

 

I hope this gets resolved.

Edited June 24, 2017 by LSD-25

[ELK]

Taking a closer look at the demo I noticed that the death rag dolls do not jitter. This problem is so strange <.>. server running 100 tick fine. Not a problem with world map physics, because the ragdoll collides fines. There are no prediction errors.

 

Death rag dolls do not interact with dynamic objects. (I believe there are no dynamic objects on your servers)

Death rag dolls are not lag compensated.

 

Other player's or bot's weapons jitter with no respect to the direction they are facing. The other player's (or bot idk) sniper was doing the scoping and un-scoping animation quickly back and forth when he was only trying to scope in.

 

This is regarding prediction of weapons ( I'm not sure if this just applies to just your own weapon. I would guess no because the server should only send data that the weapon was fired, then the client can animate rather than the server sending the animation request/data )

If you add functionality to weapons and forgot to follow the steps above ( see https://developer.va...wiki/Prediction ), than an entity may jerk around or animate strangely. You can use cl_predictionlist and cl_pdump to debug this. Changing cl_pred_optimize can also help somtimes.

Yesterday I tested cl_pred_optimize values of 0 and 2 ( - Optimize for not copying data if didn't receive a network update (1), and also for not repredicting if there were no errors (2). )

These debug commands require sv_cheats 1: cl_predictionlist & cl_pdump

cl_predict 0 requires sv_cheats 1

 

I guess to why other players jitter (but not my own movement) could be lag compensation spazzing out (almost like choosing random values)

I am unable to test the server with cl_lagcompensation 0 right now because the server is not jittering.

lag compenstaion and prediction are closely linked

sv_showlagcompensation requires sv_cheats 1

 

 

 

If a bad enough memory leak was affecting the server it wouldn't hold 100 fps. I don't know the networking well enough to tell you if and how server bloating would occur.

 

The wiki does say: The -tickrate command line parameter is not available on CSS, DoD S, TF2, L4D and L4D2 because changing tickrate causes server timing issues.

but I always thought they only said that because: A Source server running with tickrate 100 generates about 1.5x more CPU load than a default tickrate 66.

Maybe the tickrate is causing server timing issues. I don't know why it would run fine at first then get progressively worse.

 

An obvious and easy solution would be to restart the server more often, but if it can be pinpointed and resolved maybe the hitreg would be better.

 

List of tests:

cl_predictionlist*

cl_pdump*

sv_showlagcompensation*

cl_predict 0*

cl_pred_optimize 1

cl_lagcompensation 0

Source SDK has extra debugging console commands. I may have to run a test server (if that's even possible) @ 100 tick with bots for 8~24 hours to see if I can replicate the jittering. ( I have slow internet and the SDK is not downloaded )

*requires sv_cheats 1

Edited June 25, 2017 by ELK

[KiLLeR SeNSaTioN™]

I watched your demo, and I have noticed this jitteriness in the server as well, once the servers get their daily restart it is completely resolved, but as the day goes on starting maybe 8 hours after the restart, the jitteriness starts and gets progressively worse, hit reg gets worse with the jitteriness as well.

 

Perhaps the server is getting bloated as the day goes on, memory leaks, etc

 

I hope this gets resolved.

 

I have always noticed this, and it clears up after the daily 5pm CT restart. I think it's a memory leak.

 

I also noticed that it helps to close my Steam Client and restart it daily. odd.

[.lsd]

servers restart at 7am eastern time

[ELK]

Source dedicated server (which hosts CS:S servers) is 32 bit. If it exceeded 4gb it would crash.

 

If the server doesn't have enough memory and it's leaking into the virtual memory it would cause bottlenecking. The server was running 100 tick @ 100 fps flawlessly (atleast under the load while I was playing)

 

It's not a memory leak.

 

but why not check the memory usage on the server anyways?

[Ruthless.Assassin]

It's not a server usage issue, we already have the stats on that.

[ELK]

I realized I can run debugs using demo. Can enable sv_cheats 1. Lots of info. Still looking into the cause.

[loadedmind]

Ruthless> Can you please provide the stats for those of us not privy to view? Also, this is one of the main reasons I was attempting to glean the data I've requested in the past regarding the CS:S Virtual Server. I'd be more than willing to go through whatever screening process may be necessary to ensure my legitimacy. As a concerned HG member, it pains me, knowing what I know, not being able to verify certain system information to help isolate/rule out whatever issues may be happening.

 

ELK> Your tracert proves the connection between your host and the HG host is well within the threshold and is actually better than average, but I sincerely appreciate you providing that info. This is a case where too much info is never a bad thing. To answer your question regarding capabilities of the NIC, you can absolutely change flow control settings via ethtool (I believe the server is running Ubuntu). Unfortunately, there isn't enough evidence to rule out whether flow control is the issue. The path I'd pursue first (path of least resistance) is to see whether the hosting provider has other similarly spec'ed hardware providing CS:S hosting for other groups. If not, perhaps the much-needed additional information can be provided to help us help them. Additionally, I'd also like to verify other hardware specs of the machine hosting CS:S and, if I remember correctly, whether hosting additional content on the server (which I believe I remember was the case) may be causing unexpected phenomena. The issue is that this is all pure conjecture until more information can be made known regarding server hardware, network info from the server to the 'Net, etc. Since I do Linux system administration for a living, I can definitely help with the list of commands that would provide this level of information to help resolve/rule out whatever may be causing the jitter.

 

For starters, assuming sar has been installed and enabled, it'd be good to get some trending info. This, along with iostat, iowait, a "while true" script to track Steam's PID activity over a period of time:

Ex. while true; do printf "`date` : " >> /var/log/css_resourcemonitor.txt; top -bn1 | awk 'IGNORECASE = 1;/steam*/; {print $9}' >> /var/log/css_resourcemonitor.txt ; sleep 30; done

Show NIC stats: ethtool -S DEVNAME (ex eth0) - Note: this command can be used to change speed/duplex, but exercise caution, aka test on dev before rolling to prod

free -go along with smem (need to install from source - provides more accurate memory map of Linux servers than free)

 

This is just a start, but should help get a glimpse of what's happening with the server at the point where registry starts becoming an issue. It might be worthwhile to setup a non-pub server purely for the purposes of testing if sufficient players can join to reproduce the problem. These people would need to be a bit patient in order to allow folks to "tail -f" live.

Edited June 27, 2017 by loadedmind

[.lsd]

Changing the map to the current map will rectify the issue 100% of the time.

 

This will not kick the server population.

 

example: sm_map cs_office

Edited June 28, 2017 by LSD-25

[ELK]

loadedmind I don't think there is anything wrong with the server hardware. It's running 100 updates/s and it never dips below.

I don't think flow control or any other network setting is problem because there is no choke or loss.

 

I think the problem lies within cs:s itself.

 

I've ran a few more debugs today and still have not found the issue.

 

 

 

on the bright side I found the command "net_showevents [0,1,2]"

- Dump game events to console (1=client only, 2=all).

 

I had mine set to 2 and when I launched the demo it told me the steam id of every player. (not sure if a setting of 1 would also do this)

 

This means that even if a player recording a demo of hacker does not type status in the console all steam IDs are still obtainable so the hacker can be banned.

Edited July 1, 2017 by ELK

[sweetbro]

It would be cool to know why this happens. To add to this, a server I frequent does the same thing if the map hasn't been changed for a long time.

[SherlockPwnz]

In my expertise with hosting servers in the past. On servers with mani admin or SM ragdolls takes a toll on the server and there is no proof. Maybe try shutting off ragdolls for a week and see what happens. I remember having this same issue and shut off ragdolls. Ragdoll effects hurts the CPU because it has to calculate the ragdoll trajectory on players. Then if you imagine the servers full with 32 players on each server that kills the cpu real bad.