Thursday, August 7, 2008

The nuts and bolts of multiplayer performance in FSX...

Okay so I've been on something of a crusade when it comes to multiplayer framerates for quite awhile. Anyone who's flown online with me for more than, oh, five minutes, has no doubt heard me bitch about it, and well I haven't exactly made too many friends on this topic over at Avsim's FSX forum in recent debates, but eh, what can ya do? It matters to me because I haven't flown singleplayer for pleasure since October of 2006, not once. There is simply no going back. Not that I ever did much single player flying in FS9 either. It doubly matters to me because if I get less than 30 fps locked I get a little grumpy. ;)

Anyway in my quest to make the L-39 as lean as I possibly can I've been drawing on all my previous game art experience, but I've also been doing a lot of searching and reading on the peculiarities of FSX when it comes to the subject of performance. Torgo3000's blog, and the incredible well of knowledge provided by the very kind people who post at in particular have been very enlightening to me. One of the things I've noticed though is that the issue of draw calls seems to be everyone's favourite whipping boy lately when trying to explain why one model performs better than another. I've seen quite a few developers on various forums pretty much state flat out that polygon counts just don't matter anymore, that it's all about the draw calls now. That kind of statement quite frankly worries me a bit because all of my previous game development experience tells me that just the opposite is true, and that's what I intend to talk about here.

By the way, if you're not interested in a somewhat technical little article about why your FPS always seems to head straight for the single digits in multiplayer then stop reading now! If you are though, or feel that the statement made by one Avsim forum member, that "FSX multiplayer flyers are little more than second class citizens in the eyes of most developers" is true, well, then please read on.

Personally I don't think they consider us second class citizens, rather I don't think most of them have really considered us much at all. There are happily some shining exceptions to that statement though, and not the least of which would be ACES themselves, in their valiant efforts to keep every kind of user happy at the same time.

So why should addon developers consider us you might ask? Well, because multiplayer, in my opinion, and no matter the genre, is *absolutely* the future of PC entertainment. If you're a diehard singleplayer flyer then you'll no doubt disagree with me, but multiplayer is the great feather in PC gaming's cap, and is something that the consoles have yet to even remotely compete with. Multiplayer experiences are growing in popularity at an insane pace, and singleplayer ones are starting to sit on the shelves in a big way. That trend has been going on for years now and is only getting stronger. If you don't believe me I'd recommend you look at the unbelievable popularity of a little game called World of Warcraft. ;) Yes, love it or hate it, the world's most popular and financially successful game contains no singleplayer mode whatsoever. Humans are social animals, give us a chance to play with others rather than play alone and we're drawn to it I guess.

However, on today's hardware, making something work efficiently in a multiplayer environment, massive or not, means dredging up some old school 8-bit era thinking and ingenuity when it comes to graphics, and doubly so when considering FSX content. I've been trying to do just that with my Albatros and in the process I've been doing a lot of comparing and testing, trying to find out just what is it that really kills framerate in FSX. I was pretty surprised at some of the results, so I hope you find what follows interesting.

Ok, so, the new whipping boy: draw calls. The easiest way to sum them up is that FSX needs to break airplanes into chunks when processing them and sending them off to the video card for rendering on your screen. Most game render engines try to make assumptions about parts of a model in order to avoid checking every single property for every single vertex, which is very time consuming and slows performance. If a number of polygons (triangles) share identical properties then it can batch process a whole pile of them together into one big draw call, rather than a bunch of small ones, thus using the available bus bandwidth more efficiently and resulting in higher framerates. That's the general idea anyway. Something about that idea has been bugging me though lately so I decided to do some investigating using a wonderful tool called DrawCallMonitor made by FS guru Arno Gerretsen.

Now, a couple of disclaimers. First, neither this tool nor the numbers I'm going to show from it can tell the whole story about a particular model's performance. There are many other factors that play into it such as the number of textures applied to a model, their sizes and bit depths, and about a million other little issues. However the three biggest players in FSX performance are draw calls, triangle counts, and texture vertices, and that's what the tool does deal with. I'll explain a bit about each later on. I'm also not counting the performance hit from virtual cockpits and gauges or anything else here. Everything that follows is strictly concerning external aircraft models in their highest LOD form (such as you encounter in multi-ship close formation flight), and only the most basic properties of them. However, having said that, everything I'm going to talk about also applies to virtual cockpit models.

Second, I have picked eight small aircraft from my hangar to compare, and in the interest of fairness and peace I've designated them with letters A through H. They range from single engine props to small fighters and warbirds, and one helicopter. The military aircraft involved in the comparison are in their clean configurations, ie: no weapons or fuel tanks, in order to keep things as fair as possible. In each case they are aircraft that I love to fly and have logged dozens or even hundreds of hours in. All are true FSX compiled aircraft and all are of extremely high visual quality both in shape and textures.

Okay, on with the data!

First up, draw call counts. Like I mentioned above, the aircraft is broken into chunks based on parts of the aircraft that have identical properties. By that I mean materials really. Each time you change the material from one type to another, say from matte to chrome on the same object, you're forcing a new draw call. As far as I know FSX has a total budget of somewhere around 5000 draw calls (very system dependent), which includes your aircraft's VC, AI/multi traffic, clouds and scenery. The seemingly generally accepted theory is that the more draw calls a model makes the more demands on the hardware it has, and the slower the model performs... or does it?

Ok, so at first glance aircraft D should be the highest performer, and indeed it is, but, as you'll see in the stats to follow, D has a lot more going for it than just a low number of draw calls. Based on draw calls alone Aircraft G might seem to be the worst performer of the bunch. Ah, but just wait, the plot thickens, or thins, or whatever it is that plots do...

Next, polygon counts. This is pretty straightforward. An airplane is made up of triangles. The more triangles it has the more rounded and refined it looks from all angles. Although the concept of 'size' in computer graphics really has no meaning at all, it does matter in a simulation, since you're going to end up with really large aircraft next to really small ones fairly often. The bigger physically an aircraft is the more polygons it's ultimately going to need to make it look comparably smooth. That's why I've chosen only small size aircraft to compare. Okay, so how many triangles does each contain?

Okay, so now things look quite a bit different. Aircraft H which had a low draw call count turns out to have the greatest number of triangles and Aircraft G which has a high draw call count has only a slightly above average number of polygons. Now the numbers are going to be a little unreliable as all of these models employ visibility tags to show or hide parts of the aircraft at various times. So not all of the polys in each are displayed all the time, but most are.

So let's look at the final count, Texture Vertices. Now basically texture vertices are a combination of the number of vertices that make up the polygons of the aircraft and the number of independent uv coordinates attached to them, which can be shared or not depending on how the airplane is texture mapped. Without getting too technical, the number of texture vertices (when compared to the number of polygons) is a very very loose indication of how efficiently texture mapped a model is, how many vertices share their uv coordinates. The real things we're looking at here are the number of texture vertices a model has in total, but also the ratio of the texture vertices to its polygon count. Anyway I say it's a very loose indication because it doesn't tell the whole story, not by a long shot. Every aircraft has its own unique texture mapping problems, and every artist tends to tackle them differently. Some of those problems simply can't be avoided due to an object's shape. But here you go...

Now, for better or worse, this is where these aircraft all start to show their true colours I think. Based on my own experience in multiplayer with each aircraft it's the texture vertices graph which most accurately portrays their respective performance hits when flying in close formation.

In order to prove to you that I'm not loony (or maybe that I am) I've done a series of tests on each aircraft, using an identical flight situation in which the cpu/videocard load caused by all other elements of the sim are negated to the maximum extent possible. The tests are done using nothing but a camera platform in slew mode, no user aircraft is loaded. The test is over the middle of the Pacific Ocean under clear skies and all scenery settings are at minimum. Multiple copies of each aircraft are placed in a closely spaced array via FSRecorder tracks. The test literally is this: How many copies of each aircraft can I have on screen at once and still maintain 30 frames per second?

Below is a screenshot of what the test looks like. For the keen, only one of the tested aircraft contains LODs and textures with mip maps. In that case I compressed the view to force each copy on screen into its highest LOD and mip state.

And the results? An almost exact mirror reflection of the texture vertex and polygon counts. The fewer polygons and texture vertices an aircraft has the higher its real world performance. In the graph below Aircraft H is the one to look at especially since it had a below average number of draw calls but a very high poly count. If polygons don't matter anymore then I should have been able to get quite a few more H's on screen before suffering in performance. In real world use I usually can't even have ONE type H aircraft on screen without dropping to single digit FPS. Conversely aircraft G with its very high draw call count should cause a lot of performance damage but in real world play with high settings I can usually have at least 3 flying with me.

The bottom line? Texture vertices and polygon counts still seem to be quite a bit more important to performance than they're being given credit for lately.

Once you put virtual cockpits (which are often far more damaging, and to be fair, necessarily so, than external models), gauges, flight models, scenery and all that back into the equation you can pretty much knock two to eight planes off any of those formations if you're trying to keep 30 fps. In the cases of aircraft C and H you're looking at major reductions in scenery settings to get even one on screen along with your own aircraft and still keep a decent framerate.

So when comparing the stats against real world demonstrated performance it seems that draw calls aren't quite the big deal they've been made out to be, at least for aircraft. Do they matter? Definitely. But having even an incredibly low number of draw calls doesn't let you get away with murder in the polygon and texture vertex count departments. For scenery, which concerns thousands of objects at once rather than just the few which make up an aircraft, I can see this kind of test panning out very differently, but the poly count there still matters a lot too. For airplanes though it seems that texture vertices are killer #1, followed very quickly by total polygon count, and lastly by draw calls. Texture vertices are entirely dependent on poly count though. There are a lot of mapping tricks that can be used to get the texture vertex count way down, even with a high poly count, but the bottom line is that the more efficient the model is polygon wise to start with, the easier everything is on the CPU. Highly clever mapping techniques aside, when the poly count comes down so do the texture vertices for the most part, and the knock-on effect in performance improvement, as demonstrated by aircrafts A, D, and E, is simply enormous.

There is also another area where a high poly count alone can kill your performance and that's real time shadows. The ground shadows cast by aircraft in FSX are calculated straight off the polys themselves and the more of them you have the more performance you lose. In case you don't believe me just load up your favourite high end addon aircraft, turn off the aircraft ground shadows in your display settings and watch what happens to your framerate. ;) This alone, or any two pass render effect (water, bloom) makes a strong case for efficient modeling, or at the very least for the return of low-polygon proxy shadow models.

Where a low draw call count on a plane does seem to make a noticeable difference though is in a scene completely over-saturated with autogen. In that kind of situation it starts to show some advantage, but at that point an average user's FPS is so low that the sim is all but unplayable anyway.

Here's a final graph (I promise!) that shows all the stats together normalized. Poly count and Texture vertices are best viewed as a unit in how they relate to each other. When you compare those though to the draw call counts and each aircraft's actual resource demands in the sim, you see the story I've been trying to tell. :)

My system's specifications are important to this last graph:

Intel QX9650 Penryn (@4.0 ghz)
4 GB DDR2-1066
Nvidia 8800 Ultra 768mb (not overclocked)

(please click to enlarge, it's hard to read)

The other large performance killer that I haven't touched on above is the size and bit depth of the textures applied to each model, and these can have a dramatic effect. In the cases of the aircraft I tested, with the exception of aircrafts D and E, all have similar texture sizes and bit depths, ie: between five and eight 1024x1024 DXT5 base textures applied, as well as matching bump and specular maps, and a number of smaller subsidiary textures. Aircraft D has a much lower number of textures than the others and aircraft E has a much higher number. The interesting part there is that both D and E are high performers, with dramatically different texture loads, E using nearly 5x the texture space of D. This is mostly due to their mapping and low polygon counts I think, with the remaining difference coming down to D's exceptionally low number of draw calls.

By the way there's a plane in my hangar, a freeware one actually, that is 350+ draw calls, 141,000 polygons, and 230,000 texture verts. Oy! How many of those can I have on screen in multiplayer and keep 30 fps? Umm... none! :)

So why 30 FPS specifically? It's because I consider it the 'magic' number when it comes to flight model fidelity, especially concerning aerobatic and helicopter flight. Any less than that and fine, accurate control becomes difficult. 30 FPS is also magic because it's one of the few framerates that properly synchronizes with the standard (and rather lame I might add) refresh rate limit of LCD monitors, 60hz/FPS, and as such is relatively stutter free. The next highest available synchronized framerate is 60 FPS, and it's going to be a little while before we see that kind of performance as standard in FS I think. :)

In closing, it turns out that polygon counts and their associated texture vertices do actually matter, a heck of a lot!

If you're a developer and reading this you're probably thinking by now that I couldn't care less about what singleplayer flyers want, and you'd be right, I couldn't! Kidding of course, I do actually care, a little anyway. :) However, consider this: Both player types want performance, badly. You need only to browse any FSX forum for two minutes to see that, and if you build a plane for multiplayer flyers first, you suddenly make singleplayer flyers very happy too. In doing so you give them the ability to crank up their scenery settings by several notches, and singleplayer flyers, at least the low and slow crowd, aren't looking at a plane three feet off their wing like we do, they're mostly looking at the ground.

Is building stuff this way harder? Yes, and even after doing exactly that for EA and Namco for eight years I still find it challenging. Do you have to sacrifice quality to do so though? No. There are tons of low poly modeling and mapping tricks you can employ. Is it worth the extra effort? I certainly think so.

Anyway if you made it this far, cheers for reading. Sorry for the abstraction of the graphs, I did the best I could with them. :)

Less is more. (except frames per second of course)

Multi for life.



Anonymous said...

You must have quite a decent system if you are complaining about less than 30 fps :)

I start complaining when I get less then 12 fps lol.

All in all multiplayer doesn't bug me performance wise that much, this could be because my hangar isn't that extensive yet and most add-ons get replaced with either default planes or spitfires and my last landing at Heathrow on a busy day on TheBigFSX's server wasn't much of an issue.

Apart from company online my main reason for not singleplayer would be performance, default traffic kills it. It's purely cpu dependant as i get the same fps killing with SP traffic even at a distance where nothing is rendered and it's purely simulated traffic on the cpu.

I find my biggest killjoy isn't my old-ish geforce 7600gt 256mb but my x2 4200+ (2.2 ghz dual core) when I set traffic on. Multiplayer traffic which is mainly GPU dependent doesn't seem to be that big a deal (yet)

Cheers mate nice read :)
If you could let me know how you get to a place where one complains about getting less then 30 fps instead of less then 12 (at busy airports/lot's of detail) I am all ears.

Luggage (LuggageFS)

Anonymous said...

ow my, shouldn't have drank that beer :)

Lotus / Ramasurinen said...

Heya Luggage. Yeah, multi won't effect you much if you're flying into heathrow etc because any addon airliners there are just being replaced by the default ones which are nearly as efficient as the Extra 300 is.

AI traffic is worse than multiplayer because the cpu has to not only display the aircraft themselves but calculate their flight paths correctly. That's a lot of numbers to crunch per frame. In multiplayer it's just moving the aircraft to wherever it's owner's computer tells yours that it should be. Such translations are cheap to do per frame processor wise.

And I bitch about the 30 fps thing because I consider that the minimum framerate for smooth formation flight. Anything less and there just isn't enough flight model fidelity available.

As for getting that 30 fps in the first place, I stay away from big airports and I've overlocked my machine to within an inch of its life. ;)

Fab208 said...

Nice read there, even if I didn't get some of it :P

I have my system set at 20 fps locked but I don't fly online as I've never found a server that I like or one that likes me (Gamespy keeps kicking me :( ) so I've never had to fly in formations.

I have to agree with Luggage that the AI Traffic does kill the system but I've landed at LAX with 70% (Above 70/80% and the traffic spawns on to its self and causes crashes I've read on a Devs blog) airliner traffic with enough frames to keep me happy.

I don't have much mulitplayer experience so I can't complain about frame rate, but when I have been online, the frames have always been locked at 20.

I can see I'm going to have to join your server sometime ;)


Dougie said...

A few extracts from the Alphasim forums:

'Polycount is absolutely irrelevant. A designer with the knowledge to put out optimised models with as few polys as possible is what counts! I think AS is leading that part of the commercial market. They seem to understand the importance of framerates!'

'Textures and gauges will have much more of a noticeable effect on FPS than any amount of polys.'

The EFA v2 runs just as well as any of our other releases in FSX. End of discussion RE Poly counts.

Then you have to ask the question, how well do your releases actually run in FSX? (Not including multiplayer) But ofcourse it was 'End of discussion RE Poly counts.'

Just thought you guy might want to see that. Because it seems like they have been proved very much wrong. :)

Lotus / Ramasurinen said...

Good eyes Dougie. I hadn't seen those comments specifically but it's statements like that which prompted me to do this exploration in the first place. Cheers!

Sundowner said...

Can you tell us what is that monster model you mentioned in the text ? The one with 141,000 polygons ?

BTW the AlphaSim... their models usually run bit faster than competition, but after their BlackHawk, I'm through with them. I will never buy anything from them again. First they release the Apache witch lacks parts of the cockpit, than they release the CH-53s, which cockpits are unreadable, and now the CH-60 in which something as simple, as attitude indicator, don't show true a/c attitude... gaaarh! I'm going to burn the CD's with them now... where's my lighter?

Lotus / Ramasurinen said...

I hear you Sundowner. Concerning Alphasim, from whom I have purchased eight aircraft, they are really all over the block when it comes to performance in multiplayer. Bugs aside, which every plane comes with initially, their high performers are the Starfighter, F-5, and to some degree the SR-71. The rest... kind of rough on frames in multi. They employ a lot of different modelers though so each aircraft is quite differently built from the next.

The freeware poly monster is the new Bristol Beaufighter which you can find on Avsim. It's a very nice airplane, quite pretty and lovely to fly, with 3d gauges as well. I like it a lot. It is however a nonstarter for multiplayer use unfortunately.

Luggage said...

Haha no wonder you get 30fps+ your cpu costs about as much as all the parts combined when I built this one last year :p

Found the system specs on your youtube account. :)

See you up there ! :)

Luggage said...

This may be of interest on the topic of aircraft performance.

The blogs are neat, giving some more insight behind the scenes.

Lotus / Ramasurinen said...

Thanks Luggage, and yeah I've read Phil Taylor's blog inside out. He's alluded to quite a bit to the issues that I raised in this post but I don't think anyone's ever tried to demonstrate their real world impact numerically and with examples before, which is why I felt compelled to.

Saw that you got your spitfire multiplayer problem sorted out too on the avsim forum. That explains why it works fine with me, and that is one strange strange bug! When I get a spare five minutes I'll have a look in the spit's gauges, I bet the problem lies there somewhere.

Vlad said...

I must really say...I give up! :O Too much numbers!!!

Glad the maker knows where all them numbers come into play!:)

kingsnake11 said...

Hi Mike
Just got back again and am in catch up mode. I have to say, nice detective work. Good job on the graphs and your analysis; which I think are bang on. Doing more advanced statistical analysis (such as fitting a categorical regression model) wouldn't alter your conclusions, however, it could point out what factor(s) or interactions are significant. I would hypothesize that there is an interaction between polycount number and draw calls. I also suspect that inefficient modelling of guages in the VC are a fps killer too.

Lotus / Ramasurinen said...

Hey Kingsnake. Yeah, I'm a graph noob I know. ;)

You're right, gauges can be brutal on fps, but only the 2D kind. They also cause massive amounts of stutter, it's just a ton of work to composite layers and layers of 2D images and then draw them into a texture that has to be updated in realtime. 3D gauges are the way to go whenever possible. Sadly things like MFDs just can't easily be done that way. That's why I didn't go into VC comparisons, although all the modeling and texture issues for externals apply just the same to those.

tutmeister said...

Way late to the party mate, but that's a good read for all potential modelers out there. Let's hope it helps increase the amount of multiplayer models.