The nuts and bolts of multiplayer performance in FSX...

Okay so I've been on something of a crusade when it comes to multiplayer framerates for quite awhile. Anyone who's flown online with me for more than, oh, five minutes, has no doubt heard me bitch about it, and well I haven't exactly made too many friends on this topic over at Avsim's FSX forum in recent debates, but eh, what can ya do? It matters to me because I haven't flown singleplayer for pleasure since October of 2006, not once. There is simply no going back. Not that I ever did much single player flying in FS9 either. It doubly matters to me because if I get less than 30 fps locked I get a little grumpy. ;)

Anyway in my quest to make the L-39 as lean as I possibly can I've been drawing on all my previous game art experience, but I've also been doing a lot of searching and reading on the peculiarities of FSX when it comes to the subject of performance. Torgo3000's blog, and the incredible well of knowledge provided by the very kind people who post at fsdeveloper.com in particular have been very enlightening to me. One of the things I've noticed though is that the issue of draw calls seems to be everyone's favourite whipping boy lately when trying to explain why one model performs better than another. I've seen quite a few developers on various forums pretty much state flat out that polygon counts just don't matter anymore, that it's all about the draw calls now. That kind of statement quite frankly worries me a bit because all of my previous game development experience tells me that just the opposite is true, and that's what I intend to talk about here.

By the way, if you're not interested in a somewhat technical little article about why your FPS always seems to head straight for the single digits in multiplayer then stop reading now! If you are though, or feel that the statement made by one Avsim forum member, that "FSX multiplayer flyers are little more than second class citizens in the eyes of most developers" is true, well, then please read on.

Personally I don't think they consider us second class citizens, rather I don't think most of them have really considered us much at all. There are happily some shining exceptions to that statement though, and not the least of which would be ACES themselves, in their valiant efforts to keep every kind of user happy at the same time.

So why should addon developers consider us you might ask? Well, because multiplayer, in my opinion, and no matter the genre, is *absolutely* the future of PC entertainment. If you're a diehard singleplayer flyer then you'll no doubt disagree with me, but multiplayer is the great feather in PC gaming's cap, and is something that the consoles have yet to even remotely compete with. Multiplayer experiences are growing in popularity at an insane pace, and singleplayer ones are starting to sit on the shelves in a big way. That trend has been going on for years now and is only getting stronger. If you don't believe me I'd recommend you look at the unbelievable popularity of a little game called World of Warcraft. ;) Yes, love it or hate it, the world's most popular and financially successful game contains no singleplayer mode whatsoever. Humans are social animals, give us a chance to play with others rather than play alone and we're drawn to it I guess.

However, on today's hardware, making something work efficiently in a multiplayer environment, massive or not, means dredging up some old school 8-bit era thinking and ingenuity when it comes to graphics, and doubly so when considering FSX content. I've been trying to do just that with my Albatros and in the process I've been doing a lot of comparing and testing, trying to find out just what is it that really kills framerate in FSX. I was pretty surprised at some of the results, so I hope you find what follows interesting.

Ok, so, the new whipping boy: draw calls. The easiest way to sum them up is that FSX needs to break airplanes into chunks when processing them and sending them off to the video card for rendering on your screen. Most game render engines try to make assumptions about parts of a model in order to avoid checking every single property for every single vertex, which is very time consuming and slows performance. If a number of polygons (triangles) share identical properties then it can batch process a whole pile of them together into one big draw call, rather than a bunch of small ones, thus using the available bus bandwidth more efficiently and resulting in higher framerates. That's the general idea anyway. Something about that idea has been bugging me though lately so I decided to do some investigating using a wonderful tool called DrawCallMonitor made by FS guru Arno Gerretsen.

Now, a couple of disclaimers. First, neither this tool nor the numbers I'm going to show from it can tell the whole story about a particular model's performance. There are many other factors that play into it such as the number of textures applied to a model, their sizes and bit depths, and about a million other little issues. However the three biggest players in FSX performance are draw calls, triangle counts, and texture vertices, and that's what the tool does deal with. I'll explain a bit about each later on. I'm also not counting the performance hit from virtual cockpits and gauges or anything else here. Everything that follows is strictly concerning external aircraft models in their highest LOD form (such as you encounter in multi-ship close formation flight), and only the most basic properties of them. However, having said that, everything I'm going to talk about also applies to virtual cockpit models.

Second, I have picked eight small aircraft from my hangar to compare, and in the interest of fairness and peace I've designated them with letters A through H. They range from single engine props to small fighters and warbirds, and one helicopter. The military aircraft involved in the comparison are in their clean configurations, ie: no weapons or fuel tanks, in order to keep things as fair as possible. In each case they are aircraft that I love to fly and have logged dozens or even hundreds of hours in. All are true FSX compiled aircraft and all are of extremely high visual quality both in shape and textures.

Okay, on with the data!

First up, draw call counts. Like I mentioned above, the aircraft is broken into chunks based on parts of the aircraft that have identical properties. By that I mean materials really. Each time you change the material from one type to another, say from matte to chrome on the same object, you're forcing a new draw call. As far as I know FSX has a total budget of somewhere around 5000 draw calls (very system dependent), which includes your aircraft's VC, AI/multi traffic, clouds and scenery. The seemingly generally accepted theory is that the more draw calls a model makes the more demands on the hardware it has, and the slower the model performs... or does it?



Ok, so at first glance aircraft D should be the highest performer, and indeed it is, but, as you'll see in the stats to follow, D has a lot more going for it than just a low number of draw calls. Based on draw calls alone Aircraft G might seem to be the worst performer of the bunch. Ah, but just wait, the plot thickens, or thins, or whatever it is that plots do...

Next, polygon counts. This is pretty straightforward. An airplane is made up of triangles. The more triangles it has the more rounded and refined it looks from all angles. Although the concept of 'size' in computer graphics really has no meaning at all, it does matter in a simulation, since you're going to end up with really large aircraft next to really small ones fairly often. The bigger physically an aircraft is the more polygons it's ultimately going to need to make it look comparably smooth. That's why I've chosen only small size aircraft to compare. Okay, so how many triangles does each contain?



Okay, so now things look quite a bit different. Aircraft H which had a low draw call count turns out to have the greatest number of triangles and Aircraft G which has a high draw call count has only a slightly above average number of polygons. Now the numbers are going to be a little unreliable as all of these models employ visibility tags to show or hide parts of the aircraft at various times. So not all of the polys in each are displayed all the time, but most are.

So let's look at the final count, Texture Vertices. Now basically texture vertices are a combination of the number of vertices that make up the polygons of the aircraft and the number of independent uv coordinates attached to them, which can be shared or not depending on how the airplane is texture mapped. Without getting too technical, the number of texture vertices (when compared to the number of polygons) is a very very loose indication of how efficiently texture mapped a model is, how many vertices share their uv coordinates. The real things we're looking at here are the number of texture vertices a model has in total, but also the ratio of the texture vertices to its polygon count. Anyway I say it's a very loose indication because it doesn't tell the whole story, not by a long shot. Every aircraft has its own unique texture mapping problems, and every artist tends to tackle them differently. Some of those problems simply can't be avoided due to an object's shape. But here you go...



Now, for better or worse, this is where these aircraft all start to show their true colours I think. Based on my own experience in multiplayer with each aircraft it's the texture vertices graph which most accurately portrays their respective performance hits when flying in close formation.

In order to prove to you that I'm not loony (or maybe that I am) I've done a series of tests on each aircraft, using an identical flight situation in which the cpu/videocard load caused by all other elements of the sim are negated to the maximum extent possible. The tests are done using nothing but a camera platform in slew mode, no user aircraft is loaded. The test is over the middle of the Pacific Ocean under clear skies and all scenery settings are at minimum. Multiple copies of each aircraft are placed in a closely spaced array via FSRecorder tracks. The test literally is this: How many copies of each aircraft can I have on screen at once and still maintain 30 frames per second?

Below is a screenshot of what the test looks like. For the keen, only one of the tested aircraft contains LODs and textures with mip maps. In that case I compressed the view to force each copy on screen into its highest LOD and mip state.



And the results? An almost exact mirror reflection of the texture vertex and polygon counts. The fewer polygons and texture vertices an aircraft has the higher its real world performance. In the graph below Aircraft H is the one to look at especially since it had a below average number of draw calls but a very high poly count. If polygons don't matter anymore then I should have been able to get quite a few more H's on screen before suffering in performance. In real world use I usually can't even have ONE type H aircraft on screen without dropping to single digit FPS. Conversely aircraft G with its very high draw call count should cause a lot of performance damage but in real world play with high settings I can usually have at least 3 flying with me.



The bottom line? Texture vertices and polygon counts still seem to be quite a bit more important to performance than they're being given credit for lately.

Once you put virtual cockpits (which are often far more damaging, and to be fair, necessarily so, than external models), gauges, flight models, scenery and all that back into the equation you can pretty much knock two to eight planes off any of those formations if you're trying to keep 30 fps. In the cases of aircraft C and H you're looking at major reductions in scenery settings to get even one on screen along with your own aircraft and still keep a decent framerate.

So when comparing the stats against real world demonstrated performance it seems that draw calls aren't quite the big deal they've been made out to be, at least for aircraft. Do they matter? Definitely. But having even an incredibly low number of draw calls doesn't let you get away with murder in the polygon and texture vertex count departments. For scenery, which concerns thousands of objects at once rather than just the few which make up an aircraft, I can see this kind of test panning out very differently, but the poly count there still matters a lot too. For airplanes though it seems that texture vertices are killer #1, followed very quickly by total polygon count, and lastly by draw calls. Texture vertices are entirely dependent on poly count though. There are a lot of mapping tricks that can be used to get the texture vertex count way down, even with a high poly count, but the bottom line is that the more efficient the model is polygon wise to start with, the easier everything is on the CPU. Highly clever mapping techniques aside, when the poly count comes down so do the texture vertices for the most part, and the knock-on effect in performance improvement, as demonstrated by aircrafts A, D, and E, is simply enormous.

There is also another area where a high poly count alone can kill your performance and that's real time shadows. The ground shadows cast by aircraft in FSX are calculated straight off the polys themselves and the more of them you have the more performance you lose. In case you don't believe me just load up your favourite high end addon aircraft, turn off the aircraft ground shadows in your display settings and watch what happens to your framerate. ;) This alone, or any two pass render effect (water, bloom) makes a strong case for efficient modeling, or at the very least for the return of low-polygon proxy shadow models.

Where a low draw call count on a plane does seem to make a noticeable difference though is in a scene completely over-saturated with autogen. In that kind of situation it starts to show some advantage, but at that point an average user's FPS is so low that the sim is all but unplayable anyway.

Here's a final graph (I promise!) that shows all the stats together normalized. Poly count and Texture vertices are best viewed as a unit in how they relate to each other. When you compare those though to the draw call counts and each aircraft's actual resource demands in the sim, you see the story I've been trying to tell. :)

My system's specifications are important to this last graph:

Intel QX9650 Penryn (@4.0 ghz)
4 GB DDR2-1066
Nvidia 8800 Ultra 768mb (not overclocked)

(please click to enlarge, it's hard to read)



The other large performance killer that I haven't touched on above is the size and bit depth of the textures applied to each model, and these can have a dramatic effect. In the cases of the aircraft I tested, with the exception of aircrafts D and E, all have similar texture sizes and bit depths, ie: between five and eight 1024x1024 DXT5 base textures applied, as well as matching bump and specular maps, and a number of smaller subsidiary textures. Aircraft D has a much lower number of textures than the others and aircraft E has a much higher number. The interesting part there is that both D and E are high performers, with dramatically different texture loads, E using nearly 5x the texture space of D. This is mostly due to their mapping and low polygon counts I think, with the remaining difference coming down to D's exceptionally low number of draw calls.

By the way there's a plane in my hangar, a freeware one actually, that is 350+ draw calls, 141,000 polygons, and 230,000 texture verts. Oy! How many of those can I have on screen in multiplayer and keep 30 fps? Umm... none! :)

So why 30 FPS specifically? It's because I consider it the 'magic' number when it comes to flight model fidelity, especially concerning aerobatic and helicopter flight. Any less than that and fine, accurate control becomes difficult. 30 FPS is also magic because it's one of the few framerates that properly synchronizes with the standard (and rather lame I might add) refresh rate limit of LCD monitors, 60hz/FPS, and as such is relatively stutter free. The next highest available synchronized framerate is 60 FPS, and it's going to be a little while before we see that kind of performance as standard in FS I think. :)

In closing, it turns out that polygon counts and their associated texture vertices do actually matter, a heck of a lot!

If you're a developer and reading this you're probably thinking by now that I couldn't care less about what singleplayer flyers want, and you'd be right, I couldn't! Kidding of course, I do actually care, a little anyway. :) However, consider this: Both player types want performance, badly. You need only to browse any FSX forum for two minutes to see that, and if you build a plane for multiplayer flyers first, you suddenly make singleplayer flyers very happy too. In doing so you give them the ability to crank up their scenery settings by several notches, and singleplayer flyers, at least the low and slow crowd, aren't looking at a plane three feet off their wing like we do, they're mostly looking at the ground.

Is building stuff this way harder? Yes, and even after doing exactly that for EA and Namco for eight years I still find it challenging. Do you have to sacrifice quality to do so though? No. There are tons of low poly modeling and mapping tricks you can employ. Is it worth the extra effort? I certainly think so.

Anyway if you made it this far, cheers for reading. Sorry for the abstraction of the graphs, I did the best I could with them. :)

Less is more. (except frames per second of course)

Multi for life.

-lotus