This article is directed at people who want to run Internet game servers. The most common complaint I hear on servers is, of course, "laaaaaag!" Probably the second most common complaint is, "Player X is lagging us up. Kick him!" This article will show you the right way to address lag, and in the process you will learn why except for some "bug" in a games release or other programming annamoly, the latter statement is totally untrue.

How Servers Work

Prior to the release of Quakeworld, game servers and peer-to-peer hosts did their magic by calculating one world state or "server frame" at a time, for all clients simultaneously. In a sense, when you played one of those games, you were not really playing a real-time game, you were playing a turn based game in which each turn lasted for the "tic rate" of the server, usually a tenth of a second or less. Since each turn needed to be resolved for all clients, if one client's software or network was responding slowly, it slowed everyone else down as well. Some examples of early games that worked this way are Duke Nukem 3D, Descent, and Diablo.

Quakeworld brought some new magic into the gaming world. Each client was run in its own individual server frame, asynchronously from the others, i.e. the server dealt with each client's commands on an individual basis, using interpolation to resolve them relative to each other. This allowed each client to run at the speed that was best for them, without affecting the other clients. The bad news is that this method gives a distinct advantage to a client who can maintain low latency and fast throughput (the dreaded "LPB"); but the good news is that with this method no player can hold up other players due to high latency or slow throughput.

Most recently, Quake III and Half-Life have taken a further step by running clients asynchronously from the server as well as from each other. With Quakeworld technology clients could not slow down the server or each other, but the server or even the network could slow down clients by requiring them to run at a lower than optimal frame rate. This latest advancement means that now the server and client can each determine their optimum frame rates without affecting the other. Thus a fast client can run at his optimum speed without placing too high of a burden on either the server or the network, and conversely a slow server or slow network will not cause a client to slow down.

What all of this double talk means to players is that with any modern, popular game, there is no way that some other player can do anything to cause you to lag up. His frame rate does not affect yours. His ping does not affect yours. His packet loss does not affect yours. His data rate does not affect yours. You are totally independent from every other player on the server. You cannot blame other players when you lag up - it's either a problem at your end, or a problem with the server setup.

What Causes Lag

Why then does this misconception persist? Because many of us have been on a server, and seen it lag up when somebody joins, and seen the lag go away when that person leaves. Despite how it may look, it is not that person who is at fault. When you see that happen, it is not due to the player himself, it is due to the fact that either the server or the server's ISP did not have sufficient capacity to handle the extra load created by that player. In other words, the server admin did not properly limit the player capacity or player data rates on the server.

By most estimates, modern games spend about 70% of their time on graphics processing. This means that a dedicated server, which does not use graphics, only needs about 30% as much computing power as the client does. This quite literally means that most modern game servers can run satisfactorily on a 200 - 300 MHz PC with around 32 Meg of free memory. So unless your admin is trying to run a server on an old beater 486, it is unlikely that lag would ever be caused by insufficient processing power. When a server lags up due to exceeding its capacity, the problem is almost always due to insufficient network capacity, or bandwidth.

Bandwidth

So if you want to run a server, how can you avoid this problem? The most crucial consideration in setting up an Internet game server is to not exceed the upstream bandwidth available to you. If you do so, your players will experience high pings and packet loss. Staying within your upstream bandwidth allotment is the key to a smooth, low ping server.

Why is the upstream bandwidth so important? In typical asymmetrical networking schemes the available upstream bandwidth is significantly less than the available downstream bandwidth. This happens to be just the opposite of what a server needs. Just as clients normally receive a lot of data and send out very little, servers of all kinds normally have to send out a lot of data while receiving very little. With client/server games, the server has to send out large world updates to each client, but the server only has to receive relatively small command packets from each client. The amount of data that a game server sends out is typically two or three times as much as the amount of data that it receives.

Therefore the total upstream data rate must be limited to something less than the available bandwidth. Obviously, doing so requires first that you must be able to determine how much bandwidth is really available to you, and second that you must be able to prevent your server from exceeding it. The first condition can be determined experimentally. The second requires that you run a game that allows data rate control, and that you set it up appropriately.

A cap somewhere in your ISP's network usually determines the maximum bandwidth that is available. Depending on how the ISP is set up, the cap could be in your broadband modem, or it could be in the network head end. This means the cap may only apply to the data that you personally are sending, or it may apply to your entire node.

If the cap is on your individual line, as is typical for xDSL and some newer cable providers, you can test your bandwidth at pretty much any time of day and get consistent results. If on the other hand you are on an older cable network that does not use DOCSIS modems, in all likelihood your entire node shares the same bandwidth allotment. This means you will have to test your bandwidth repeatedly at different times of the day in order to determine how much is really available.

Testing

Testing your bandwidth is relatively simple. You need to have some file space available on one of your ISP's servers. It can be web space or FTP space, it doesn't matter as long as it is space to which you can upload a large file. Most ISP's include some free space for web sites, along with FTP access to that space. Check your ISP's home page if you are not sure.

Once you have the space, you also need to make sure that the space is close to the server that is going to run your game. If you are using your ISP provided web space, this is probably true, but if you are using space from some other provider, it may not be. To determine how far away the server is, get onto your game server PC and open up a console window. You need to know the domain name or IP address for the server that has the file space. Enter the following command on the game server:

tracert <file server name>

- or -

tracert <file server IP address>

For example, my home ISP's file server is named people.mw.mediaone.net, so I would enter:

tracert people.mw.mediaone.net

In return you should see a display showing you all of the hops between your game server and the file server, and the ping to each one. If the total number of hops is more than 5, try to find a closer server to test. We are trying to find a close server in order to minimize the effect of external traffic on the bandwidth measurement.

Next you will need to find a large file to transfer. It really doesn't matter if it is a text file or a binary file, but it does need to be a fairly large file in order to minimize the effect of caching and to get a good overall average. A file on the order of 5 to 10 Meg in size is usually good. Map or patch files that you have downloaded from the Internet are often about the right size.

Use an FTP client to transfer the file. Most operating systems provide a command line FTP client that will work just fine. If you are running Windows, you can use a product such as FTP Explorer or CuteFTP to transfer the file. Whatever client you use, make sure it is one which will report the transfer rate to you.

Either during or after the file transfer, you should get a message telling you the transfer rate in bytes/sec. This is the number we want. Watch the rate during the transfer if possible. You want to get an idea of how much it fluctuates.

You should try this test repeatedly, at different times of the day. It will change depending on your ISP's network load. If you are on cable it can fluctuate quite widely. If you are going to run a server 24/7 you need to know the minimum bandwidth that will always be available to you. Even if you are only going to run a server during the evenings you still need to know how the bandwidth varies from one evening to the next.

Interpreting the Results

If your ISP has told you that they use a rate cap, you will probably find that your measured rate is less than that. This is not necessarily a concern, as long as it is close. The rate cap your ISP uses is calculated based on the total number of bits per second that you are sending, including all the overhead tacked on by various network layers. The number that you measured, on the other hand, is based on the amount of data sent only, and will consequently always be smaller than the theoretical cap. In technical terms, your ISP cap determines bandwidth, but what you measured is data throughput.

That's okay for our situation, because as it happens, the rate settings in your game server are also based on data throughput, and do not take into account network overhead. So is the throughput that you measured during your file transfer directly applicable to your server rate setting? The answer is, "sort of."

There are two things that can cause this number to differ from the rate that is actually usable by your server. First, FTP uses TCP packets to send and receive data. Most game servers use UDP packets. UDP packets have smaller headers, and therefore are a more efficient transmission protocol. UDP allows higher throughput from the same bandwidth.

If that were the only difference we could probably jack the number up a little, but there is also an opposing effect. FTP, HTTP, and other file transfer protocols will try to send the largest possible packets in order to maximize transfer efficiency. Game servers on the other hand tend to send much smaller packets, in order to minimize the bandwidth required per player update. On any sort of broadband connection, the TCP packets used for file transfers are typically 1500 bytes. Game server packet sizes can grow that large also, but they are more typically in the range of 100 to 300 bytes. Smaller packets mean more space lost to communications overhead, which then reduces the throughput available to the server.

What can I say? It would be ridiculous to try and calculate exactly how far off our data throughput measurement is. Cross your fingers, knock on wood, sacrifice a virgin, and hopefully the two effects will cancel each other out. The measurements that you make using this method are close enough for our purposes. Besides, we automatically err on the conservative side when we apply these settings, since players do not run at their maximum allowed rate all of the time.

Now What Do I Do?

Now that you have an idea what your upstream throughput is, you need to set up your server in such a way that it will never exceed that number. The formula is simple enough:

(Number of Players) x (Maximum Rate per Player) < Throughput

Keep in mind that if you are planning on running more than one server on the same PC, you need to count the total number of players allowed on all servers. If you plan on using different rates on different servers, you are going to have to sum them all up individually.

The following table shows the variables that you need to adjust for some common game engines:

Controlling Server Bandwidth Game Player Count Rate Cap
Half-Life maxplayers sv_maxrate
Quake III sv_maxClients sv_maxRate
Unreal MaxPlayers MaxClientRate

Other Considerations

Be careful to set reasonable rate values. In general you would like to be able to provide an update rate to all players of at least 20 updates per second. For typical games this means that client performance will suffer if you restrict their connection rate to less than 3000 bytes per second. In tight situations, it is better to restrict a game to 10 players at 3000 bytes per second than to try and run 20 players at 1500 bytes per second.

On the other hand beware of allowing the rate to go too high. Some Admins will tell you that they have measured the actual data rate used by their players and found it to be approximately 3000 to 4000 bytes per second per client. But you can't count on this. Most likely they were measuring the bandwidth used by modem players, which is capped to that rate on the client side. A broadband client on the other hand can easily soak up 10000 bytes per second in Half-Life and as much as 25000 bytes per second in other games. This does not mean they are running at that rate all the time. But it does mean that when a broadband player walks into a crowded area where there is a lot of action going on, he can soak up all of the available bandwidth and lag up everybody else. This is exactly why you have to cap player data rates on the server end.

I have experimented with this extensively and I can honestly tell you that there is very little inconvenience to a broadband player if you cap him to a low rate. A low rate does not cause a low ping. Ping is a function of all of the network hardware between the player and the server, and is not directly affected by data rate. Ping can appear to be affected by data rate if the rate is set to an unreasonable value, or if any of the hops between the player and server are too heavily loaded. But under normal cir***stances there is no relationship between the two. I have personally played on servers where my rate was capped to 3000 bytes per second, and still pinged in the 40 millisecond range. It is perfectly reasonable to set a data rate cap for all clients that is somewhere in the range of 3000 to 5000 bytes per second.

Examples

Here are some examples for common types of networks. Keep in mind that the number of players used in the calculation represents the maximum number of players on all of the servers that the host machine is running, not just the number of players on one server.

ISDN, aDSL

Upstream bandwidth is capped at 128kbps for ISDN and for many of the less expensive aDSL plans. To make the calculations easy I assume a throughput of around 12000 bytes per second. Here are some possible settings that will work:

2 players x 6000 bytes / sec / player = 12000 bytes / sec

4 players x 3000 bytes / sec / player = 12000 bytes / sec

6 players x 2000 bytes / sec / player = 12000 bytes / sec
As you can see, such a line can support at most one server at a low data rate.

Cable, DSL

A common upstream cap for cable and DSL plans is 300kbps. This should allow close to 30000 bytes per second throughput.

4 players x 7500 bytes / sec / player = 30000 bytes / sec

8 players x 3750 bytes / sec / player = 30000 bytes / sec

10 players x 3000 bytes / sec / player = 30000 bytes / sec
Here again there is barely enough bandwidth for one server. You could run two small DM servers at a low data rate, but I wouldn't recommend it.

T1

A T1 line generally has a symmetrical bandwidth of 1.5 mbps. That will allow approximately 150 Kbytes / sec of throughput. That sounds like a lot, but it's not as much as you think. As you can see from the examples that follow, if you operate a server on a T1 line, but want to run large games or multiple servers, you still need a rate cap.

First lets look at what happens without a rate limit. With no limit Quake III players would be able to connect at a maximum of 25000 bytes / sec, UT players get a max of 20000 bytes / sec, and Half-Life players would be able to connect at a maximum of 10000 bytes / sec.

6 players x 25000 bytes / sec / player = 150000 bytes / sec, therefore in Quake III you should not run more than 6 players on a T1 line without using a rate cap.

7.5 players x 20000 bytes / sec / player = 150000 bytes / sec, therefore in Unreal Tournament you should not run more than 7 players on a T1 line without using a rate cap.

15 players x 10000 bytes / sec / player = 150000 bytes / sec, therefore in Half-Life you should not run more than 15 players on a T1 line without using a rate cap.

24 and 32 player games are common in many Half-Life team based mods. What size rate cap would we have to use in order to prevent lag on a T1 line for such large games?

(150000 bytes / sec) / (24 players) = 6250 bytes / sec / player

(150000 bytes / sec) / (32 players) = 4688 bytes / sec / player
The mighty T1 line, while great for small games, is apparently not an adequate solution for large internet games or multiple servers. If you are running a T1 based server, please note those numbers carefully!

What's The Problem?

When I have shown people these calculations in the past it has met with a lot of skepticism. Server admins just do not want to hear that their network line is not adequate for the number of clients that they want to support. Typical arguments include, "But I uploaded a file at a rate of 1.4 mbps at 3 am, so I think my cable is as good as a T1"; or "But I measured the data going out to clients and they only were using about 3 kbytes / sec each"; or "If I set the rate that low nobody will want to play"; and the ever popular "Data rate has nothing to do with ping". I actually already covered these issues, but in case you are not convinced let me go into a little more detail.

High Measured Server Bandwidth: Cable systems are generally set up so that a large number of users share all of the bandwidth of one node. This means that if nobody else is making heavy use of the network, it can appear like you have a huge amount of bandwidth. A T1 it's not, but typically there is 768kbps total bandwidth available per node. However you cannot count on that bandwidth being there all the time. All it takes is one guy running a Napster server to soak up all of the available bandwidth to himself. If you are on a cable line, there are two things you should do. First, find out what the theoretical upstream cap on your line is, and assume that this is the maximum you can count on. Secondly, repeat the upload test every day over a one week period during prime usage hours. The most bandwidth that you can count on having is the minimum of your upstream cap and the minimum rate from your testing. You might not like the results, but the alternative is having to listen to players constantly complaining about lag.

Low Client Rate Usage: Many times I have seen messages on bulletin boards to the effect that game clients only need 2 - 3 Kbytes / sec each. This is not true. It is typical that modem players have to limit their rates to this level, and therefore measuring a client's usage will often result in these numbers. But the fact is that servers are capable of sending more than this, and broadband clients are capable of using more than this. A broadband client can have his rate set as high as 25000 bytes/sec for some games. Of course that is only his rate cap, he is not using this rate all of the time. But a game is easily capable of sending that much data to a client. In a complex scene where there are many players, many weapons being fired, and many effects being triggered, the updates to clients can get quite large. Where an update might normally be in the range of 200 - 300 bytes, they can get as large as 1500 bytes each. If your server is sending 20 updates per second, several 1500 byte packets in a row would cause the data rate to spike up to 30 Kbytes / sec, which would require roughly 300 kbps of bandwidth per client. And don't forget that Quake III and Half-Life allow the client to request much higher update rates than I used in that example. The signs that this is happening are pretty straightforward. One sign is when everybody gets lagged up when a new player joins. The lag is not due to the new player, it is your fault for not limiting data rate appropriately. Another sign is if everyone gets lagged up when there is a big fight in the middle of an open area. This is not a problem with the map, it is your fault for not setting up your server properly. Listen to your players.

Game Quality: The fact that broadband players can set their data rates very high and get away with it has led many broadband players to believe that a high data rate is necessary in order for them to get a good ping. This is not true. Ping is not a measurement of the time it takes to transmit the data completely, it is a measurement of the time required to cross the network, added up in both directions. If you have had any physics, it's kind of like the difference between flow rate and speed. Okay, not the best analogy. What happens when you limit a player's rate is that some information will get dropped when the limit is exceeded. Games are normally programmed to provide reliable delivery of things like entity positions and critical messages such as damage amounts or flag captures, and less reliable delivery of things like explosion sounds, particle trails, decals, and so on. When the game server determines that it is about to exceed a player's rate limit, it will drop the less important information and send only those things that require reliable delivery. This can reduce packet size considerably. Since most servers run each client asynchronously, the server can also elect to reduce the frame rate or update rate for that particular client. But the long and the short of it is that if you set a rate cap that is no worse than what modem players are used to, the broadband player will receive approximately the same update rate and game quality as the modem players, but still at a much lower ping. Frankly, if you don't tell them they probably won't notice. I have experimented with low rate settings many times on cable servers and even at LAN games and nobody complained.

The Effect of Data Rate on Ping: This is a never ending argument, but like most of the stupid arguments we get into it's mostly a matter of definitions. Theoretically data rate has no effect on ping, they are two separate animals. However the reality of network play, as any modem player can tell you, is that trying to use too high of a data rate will increase lag. The reason is simple. Network latency is primarily a function of the number and speed of the hops between the client and the server. A hop is a router or gateway, which is a box with some computing power in it that has to look at each packet and forward it to the appropriate destination. The bandwidth that a hop can handle is therefore dependent on its processing power. When a hop gets near its bandwidth limit, it takes longer for a packet to get through that hop, therefore ping increases. Packets that take too long to transmit will exceed their TTL ("Time To Live" allotment) and be discarded, resulting in packet loss. Similarly, if your bandwidth is capped artificially you will experience lag and packet loss whenever you attempt to exceed that cap. It is therefore crucial that both clients and servers limit their data rates to reasonable levels.

Summary and Conclusion

Okay I admit that was far too big an article. But I wanted to make sure I got all of the relevant details in there, since this is such an emotional issue for both players and server admins. To summarize, here are a few simple steps that every server admin should take to prevent lag:

Use a game or mod that allows you to limit both the number of players and the maximum data rate that they are allowed to use. Avoid running games that don't, you are asking for trouble.

Measure the actual upstream data throughput available to you. Measure it many times throughout a one week period.

Look at your measurements and compare them to your ISP's upstream cap, converted to the same units. The lowest of all of these numbers is the maximum throughput that you can rely on having.

Set your servers up so that the total number of players allowed on all servers, multiplied by their rate cap, is less than the number determined in step 3.

Pretty simple, huh! Aren't you pissed that you had to read through that whole freaking article just to find these four simple steps? I know I would be. ;o) Consider the payoff though: if you do all of these steps correctly you should have smooth reliable gameplay at all times.

Isn't it worth a little effort to have that?

Well I dunno about the rest of you but reading all that would even make ME say the post is too long LOL.

Hope you all get somthing from this you can apply to your game and / or your game's server.

- June