vercidium
patreon
Lag Compensation and Packet Delay Detection
🕑 Added 2024-11-08 11:11:24 +0000 UTCThe purpose of networking is to connect players together, then make them forget they live on opposite sides of the world.
I live in Australia, and the majority of Sector's Edge players lived in America and Europe. If I wanted to enjoy playing my own game, it needed to have good networking.
This article covers the main networking issues I faced while making Sector's Edge:
Ping calculation + fluctuation
Lag compensation
Packet delay detection + remediation
Simulation + syncing
Part 2 will be shared next, which will cover:
Skeletal hit detection
Position smoothing
Map synchronisation
Implementing these features allowed players to have accurate hit detection and smooth gameplay with up to 400ms ping.
The Problem with Networking
Let’s say you and your friend are the only people in the world that use the internet. Even if you lived on opposite sides of the world, your packets would get to each other very quickly.
Then your brother starts watching a YouTube video, which means his data needs to share the same network lines as yours. Then your sister turns on the microwave, which messes with your Wi-Fi signal and causes packet loss.Then the rest of your country starts using the internet too.
When you send a packet to your friend on the other side of the world, you're competing with virtually everyone else who's using the internet. Routing, bandwidth, congestion, packet loss and physical distance all play a factor in how long it takes for a packet to reach its destination.
This makes it very difficult to play multiplayer games in real time.
Thankfully the overall bandwidth of the internet has increased over time. More routes and networking lines have been installed, meaning that if one route is congested, your packets can be redirected along a different route.
This means some of your packets might take the fast route, and some might get stuck in traffic. This means we can’t guarantee that your packets will arrive at their destination in the same order. We can’t even guarantee that all of your packets will survive the trip.
So how are multiplayer games possible at all?
Time Travel
When you play a multiplayer game with your friends, you will all connect to a server. Some of you might be close to the server, and some of you might unfortunately live in Australia.
But no matter how close you are to the server, there will always be some travel time for your packets. This means that when your friend walks past you, you’re actually looking at where they were in the past
That’s why you'll see strange things like players walking through walls:
This makes it difficult when you want to shoot them, because what you see on your screen is different to what the server sees, and is also different to what your friend sees.
Let's look at a simpler 2D platformer. When your character jumps, your game will send a ‘jump’ packet to your friend. Eventually your friend will get the packet, and they will see your character jump.
There are two non-negotiables here:
Packets will always take time to reach their destination. We can't avoid this
We can guess the amount of time it'll take for a packet to arrive, but can never know for sure
This means that unless you're standing still, you and your friend will never see each other in the same position. So how are you meant to attack each other accurately?
Lag Compensation
In older multiplayer games, you'd have to account for this travel time yourself. If an enemy was running in a straight line, you'd have to think "they're actually a bit further ahead than that, so I'll shoot a meter in-front of them". Your bullets will hit the enemy, but it looks strange because you're shooting at thin air.
The solution is to make the server do the time travel for you. When you tell the server you've shot your gun, the server needs to rewind time to figure out what you saw on the screen when you fired.
Since there's travel time between you and the server, the server needs to figure out how far back to rewind time.
The server can guess this by sending a ‘ping’ packet to you. Your computer then responds with a ‘pong’ packet, and when the server receives it, it’ll check how long has passed since the original ping packet was sent
This time is called your roundtrip time, and in games it’s called ping. The further away you are from the server, the higher your ping will be.
If you have 200ms ping, the server will collide your bullets against the positions of enemies 200ms ago. This means the server needs to keep a history of every player's positions for the past ~400ms - or the max lag you want to support for your players.
But you’ll be able to aim and shoot at your friend on your screen, and your bullets will collide with them. But every now and then, you’ll miss. Nothing is guaranteed in networking
Ping Fluctuation
We are assuming that it always takes 200ms for a packet to travel from the server to you and back.
In reality, this time fluctuates all over the place. That’s because some routes are longer than others, and we’re sharing the internet with millions of other people.
The first packet you send might take 100ms to arrive at the server, but the next might take 200ms. This means the server can’t rely on your ping to figure out what you saw on your screen at the time you fired your gun.
There’s another solution to this that doesn’t involve time at all. Right now, when you shoot your gun, your game sends a packet to the server containing this data:
When the server receives any packet, it first checks the MessageType, which is the first byte in the packet. In this case it's a Shoot packet, so the server will then fire a bullet in the direction you're looking (your pitch) and collide it against the position of enemies 200ms ago.
But your ping fluctuates and isn't reliable for collision detection. So we'll also include the position of a random player.
When the server receives this packet, it will look through the history of that enemy’s positions and find the position that best matches what we sent, and collide your bullet against that. The enemy's position is serving as a point-in-time snapshot of your current game state.
This will provide accurate collision for most cases, but there are times when enemies move in repeated patterns. It’s common in first person shooters to strafe left and right when firing your gun.
This means there will be multiple positions that match what we sent, and the server won’t know which to choose. So rather than sending the position of any random player, we’ll try to pick one that’s jumping or moving in a straight line.
In most cases you’ll be playing against a large number of players, and at least one of them will be moving in a straight line. If the server can’t find a good match, it can fall back to your ping or the result from the last time you shot your gun.
Since we are including the position of a random player in our Shoot packet, it doesn't matter how long it takes for our packet to arrive, and your hit detection will be accurate in 99.9% of cases (remember, nothing is guaranteed in networking).
Packet Delay
But we’ve only solved this for single-shot weapons. The second problem to solve is automatic guns, where things work a bit differently.
Since automatic guns fire so quickly, sending a 'shoot' packet each time you fire will get pretty ugly. Since your packets won't always take the same route to the server, your gun won't fire consistently. If 6 of your shoot packets are delayed and arrive at the server at the same time, the server would shoot your gun 6 times instantly. That's overpowered
To solve this we need to use the right packet type. In networking there’s 2 types of packets we can send:
Unreliable
Reliable
Unreliable packets are like your friend who never responds to messages or turns up on time. You can message him as much as you like, but it’s rare your messages will get to him. He'll also never let you know if he got your messages
Reliable packets are used for important information, like when giving someone specific instructions. You'll ask them to repeat them back to you to make sure they’ve got the information exactly right. If they don’t repeat it back correctly, you’ll say it again and repeat the process.
It would be terrible to play a game that only uses unreliable packets. Imagine shooting your gun, the packet gets lost, the server never fires your gun, and then you friend shoots you and you die.
So instead, we’ll send a Reliable packet to the server when we shoot. If the server doesn’t respond to us in time, we’ll send it again and again until it responds.
For single-shot guns like snipers it's fine if this packet is delayed, because it contains the position of an enemy at the time we shot.
But for an SMG with 30 bullets, sending 30 reliable packets wouldn’t work because reliable packets must also arrive in the same order. Each packet will have an Order number, and if the server receives packets 1 and 3, it’ll say “hold on a sec, I haven’t received packet 2 yet”.
This means your 3rd, 4th, 5th, etc. gunshots are delayed until the 2nd packet arrives. If you have 200ms ping to the server, that’s a long time for your gun to stall!
To solve this, we can also send packets as Ordered or Unordered.
Unordered Unreliable packets are for predictable data. For example, if I wanted my friend to tell me the time every second, and I stopped paying attention for a bit, I wouldn’t ask him to repeat all the times I missed, I’d just listen to the latest packet. If an old packet arrives later, I can tell it’s late because it’s an older timestamp.
Ordered Unreliable packets are for data that updates often. In every multiplayer game your game will constantly send the direction your character is looking to the server. It's not a big deal if one packet goes missing, because another packet will arrive shortly after it.
But if a missing packet turns up a second later, it shouldn't make your character temporarily flick back to where it was looking in the past. Using Ordered Unreliable packets fixes this, because each packet contains an Order number. This lets us ignore older packets that arrive late.
So when firing our automatic guns, we’ll use a combination of these types:
The first packet we send is a ‘start shooting’ packet, which is Reliable and Ordered
The next packets we send are ‘aiming’ packets, which are Unreliable and Ordered.
Your game will constantly send aim packets to the server, so that when the server starts firing your SMG, it can continuously fire it based on the direction you’re looking
If a few aim packets are lost, that doesn’t matter because 10ms later the server will receive the next packet. If an aim packet arrives late, the server will ignore it because the packets are sent with an Order number.
This prevents your aim from flicking back to where you were looking in the past, when an old packet arrives late. By using these packet types, we’ll be able to shoot our fast-firing SMG without issues.
But for a shotgun with huge recoil, you might notice this happening. See how the bullets damaged the wall above where we're aiming?
This is because of packet delay.
If you’re using a shotgun, each time you shoot (yellow line), your pitch will increase sharply and then fall back down smoothly
But since your Shoot packet was late, the server will shoot your gun slightly later, when your character is aiming higher near the peak of your recoil.
Note that the first time you shoot, your bullets will fire at the correct pitch. That's because your Shoot packet contains the pitch at the time you shot.
All your shots will go above the enemy's head, even though on your screen you shot them in the body every time.
We will solve this using timers. When the server and client connect to each other, they will each start a timer. These timers will never be in sync, but we can still use them to detect when a packet is delayed.
Every second, your client will send its timer's elapsed milliseconds to the server. The server will then check the difference between your elapsed time, and its elapsed time.
Let's say the server receives three timestamps, and notices that:
the first one is 300ms behind its timer
the second one is 300ms behind its timer
the third one is 350ms behind its timer
Looking at these time differences, we can see that the third packet is an extra 50ms behind the others. Therefore, we know that it took 50ms longer to travel from the client to the server than usual.
We don't know how long it took to travel from the client to the server, but we know that it took 50ms longer than usual.
All the server needs to do is keep track of the lowest time difference and use this as its benchmark.
To determine if a shoot packet is delayed, your game will also send the timer's elapsed milliseconds in the shoot packet:
When the server receives this packet, it'll check the difference between the timestamps. If the time difference is 400ms, the server knows your shoot packet was delayed by 100ms.
Currently the server keeps track of every player’s position over time. To fix this packet delay, it also needs to keep track of every player's pitch. Since your shoot packet was delayed by 100ms, the server will shoot your gun based based on where you were looking 100ms ago, and collide against the position of enemies an extra 100ms ago.
This lines up exactly with where you were looking on your screen each time your shotgun fired:
Now you can shoot your guns without worrying about packet loss or packet delay.
Client Simulation and Syncing
When throwing grenades in older FPS games you might notice a delay before the grenade actually appears. That’s because the server manages the positions of these objects, meaning you won't see your grenade until the server sends its position back to you.
If your game controlled this grenade, it send its position every frame to the server, and then tell the server where it should explode. However this makes it much easier for a hacker to say “I threw a grenade and it landed right at the enemy’s feet around the corner”.
So instead your game will send a ‘throw grenade’ packet to the server and the server will create the grenade and broadcast it to every other player. But the issue here is there’s a delay between your throw animation and when the grenade appears. This also makes it hard to adjust your aim when using physics-based weapons like grenade launchers.
The solution to this is to create a fake grenade on the client, and then sync it up to the positions the server sends us later (notice the little 'hop' in the grenade's position).
In a perfect world we wouldn’t need to sync it to the server. But your game and the server will likely run at different frame rates, and the results of physics simulations will be slightly different.
As your grenade flies through the air, these differences add up. The grenade might just scrape over a wall on your screen, but on the server it might bounce off the wall and land at your feet.
So when we start receiving the grenade’s position from the server, we need to compare it to where our grenade was in the past.
Then over time, we’ll interpolate more towards the positions we’re getting from the server, until eventually we’re not simulating the grenade on our client at all.
This might look weird for edge cases like this, where the grenade appears to snap back behind the wall.
But it’s much better than thinking the grenade is further away, when really it’s at your feet.
For most cases, you’ll never know the objects you throw are actually fake and synced up to the server half a second later. The game now looks and feels much more responsive.
Position Smoothing, Skeletal Hit Detection and Map Synchronisation
These three topics will be covered in Networking Part 2! I would like to get your feedback on this article and the code first, while I continue implementing and improving these features.
Demo + Code
If you're a Patron, you can access the code on the network project on the lagcomp branch on GitHub. It's a fully functioning demo with a server and two clients, all within the one executable.
To simulate lag between one of the clients and the server, I use clumsy to add lag on one of the client's ports (each client connects to the server on a different port).
I use 200ms lag and 30% packet loss chance:
To experience packet delay detection in real time, set NetworkConstants.AutomaticShootingRecoil to true. This will make your cursor jump up and down while shooting, to simulate huge shotgun recoil. You should see the projectiles travelling in the same trajectory on both clients:
That's all for today. Please feel free to comment or direct message me if any of these concepts don't make sense.
Next I'll start working on a YouTube video about raytraced audio. Expect some interesting 3D animations in this one!
Comments
Vercidium
Ahh I see, thank you I'll check out this video
Ramón
I'm referring to the variance in clients input latency: if several inputs are delayed (relative to the "average latency"), arrive suddenly at the same time and are processed immediately, they will appear to other players as if that player is teleporting. Overwatch has a small input buffer to hide it: https://youtu.be/W3aieHjyNvw (min. 26 onwards) and depending on the network conditions they resize it. Maybe it's not that important, and rarely happens so it makes sense removing it to reduce other players' "visual latency". Thanks for the answers.
Vercidium
Yes that’s right, no input buffer. What do you mean by client jitter? Yeah to keep bandwidth low the server broadcasts player/entity positions 60 times per second. For data that arrives at the server - e.g. a player’s jump packet - it is broadcast to all other players as soon as it arrives
Connor
Articles just as captivating as your videos. I particularly thought the "time travel" section was super fascinating. Really opens my eyes as to how much us actually going on behind the scenes to make a multilayer game work
Ramón
Wow, your engine is very well optimized because I can't imagine my server running at that tick rate. It makes sense, although, when you say you process packets as soon as they arrive, does it mean you don't have an input buffer to hide client's jitter? Also, I suppose you don't send server updates each 1/300 ms, but I'll wait for part 2 to see what you did. Thanks again, very useful information.
Vercidium
Yes I've never used a fixed tick rate. The server for Sector's Edge runs at ~300 FPS so that packets are processed as soon as they arrive. A fixed tick rate of 60Hz may add up to 16ms of extra latency, as the server would only handle incoming packets every 16ms
Ramón
Thank you, the tutorial is excellent. Just one question: when you say "But your game and the server will likely run at different frame rates, and the results of physics simulations will be slightly different." does that mean that you didn't adopt a fixed tick rate for both server and client? If so, what advantages does it have? because most tutorials and modern FPSeses recommend a 60Hz tick rate.
