Before continuing, let me state clearly that what I’m about to describe requires Visual Studio 2010 Professional. Those of you using Visual C# Express Edition won’t be able to use this technique.

Okay, now I’m going to explain what I did. In my sample solution, I reduced the build time by 25% on my dual-core laptop (from 38 seconds down to 29 seconds).

For my experiment, I started with the Ship Game starter kit from create.msdn.com. I chose this starter kit because it contained enough content that locking up the IDE on rebuilds would get annoying really fast.

Here’s what I did (short version; longer version at the end):

1. Split content into multiple content projects, all building to the same output directory.
2. Add a Game Library project to the solution that references all the content projects.
3. Add a VC++ Makefile project to the solution that builds the game library from (2) with multi-proc support enabled.

The key here is that I introduced a VC++ Makefile project to initiate the content build. This project type invokes arbitrary, user-defined commands, and it isn’t all broken like the C# projects. Specifically, it doesn’t lock up the UI thread when it executes long-running commands, and it correctly redirects all the command output to the Output Window, and reports errors in the Error List.

To have the makefile project build all my content projects in parallel, I set up the Build command to build the Game Library project using msbuild.exe with multi-processor support enabled.

Example Build Command:

“$(MSBuildBinPath)\msbuild.exe” /m:$(NUMBER_OF_PROCESSORS) “$(SolutionDir)Content_All\Content_All.csproj”

For Rebuild and Clean, I simply added /t:Rebuild and /t:Clean, respectively.

Note: VC# Express doesn’t support VC++ projects, which is why this won’t work for Express users.

So that’s what I did, and it works great. Content builds don’t lock up the IDE, and the Output Window displays the name of each asset as it is compiled (some reassuring feedback during long-running builds).

At this stage, my solution is set up to build content projects referenced by the Game Library project (Content_All) in parallel. When I tried it the first time, I only squeezed 2 seconds off the build time. The bottleneck in my build was the project using the normal mapping processor, so I added another project, and moved some of the models and dependencies over there. That brought more parallelism into the build and the overall build time dropped to 29 seconds (from 38).

I experimented a little with putting the audio in another project, and shuffling some of the models back and forth, but gave up before finding any additional gains. On a machine with more processors (my laptop has two), there is potential for more savings.

I should note that in order to split up the custom model processing into multiple projects, I had to include some of their dependencies twice. For example, both projects build the NormalMapping shader. The cost of the redundancy was offset by a greater savings from parallel processing.

The drawbacks to doing this are:

1. You basically need two extra projects in your solution to enable it, for each target platform.
2. You need to manually edit the Solution Configurations to make things build properly.
3. For Windows projects, the content won’t be recognized by ClickOnce publishing.

The first isn’t so bad, because you can hide the extra projects in a Solution Folder. The second is something many people don’t really understand, but should be part of any advanced developer’s tool kit.

Another thing I want to note is that I separated the assets that use the custom processor from the assets that use standard processors to reduce incremental build times when NormalMappingModelProcessor.dll is modified. The standard content won’t rebuild at all – and thanks to splitting the custom content into two projects, the custom content can rebuild fully in less time as well.

Angry grumbling: I had intended to provide my modified ShipGame solution as a downloadable sample, because the web site said it was under Ms-Pl. However, the actual ZIP file contains a license file that describes the “XNA Premium Content” license, which prohibits redistribution of unmodified software.

Before I go, let me leave you with this disclaimer: your mileage may vary. I do not plan to write out a click-by-click tutorial, but I added more detail below.

Here’s what I did (longer version):

1. Move ShipGame content project up to the solution folder, and rename it Content_Main.
2. Add new content project, named Content_Custom.
3. Delete the Content Reference from ShipGameWindows.
4. Add new Game Library project to the solution, named Content_All.
5. Change the output folder of Content_All to match the output folder of ShipGameWindows.
6. In Content_All, add references to Content_Main and Content_Custom.
7. Change the Content Root Directory of all content projects to “Content”.
8. Add new C++ Makefile project, named Content_BuildAllParallel, and configure the Build, Rebuild, and Clean commands (details below).
9. Move all the content that depends on NormalMappingModelProcessor into Content_Custom, along with any additional content it depends on.
10. In Content_Custom, add a reference to NormalMappingModelProcessor.
11. In Content_Main, remove the reference to NormalMappingModelProcessor.
12. Edit the solution configuration so that Content_All and NormalMappingModelProcessor are excluded from all solution builds. Make sure that Content_BuildAllParallel is built when the rest of the game is built.

I thought about the XNA Content Pipeline, and the convenience of reflection-based serialization. I thought about the run-time performance penalty you pay for that convenience, and suddenly realized that you can have the best of both worlds.

What if the reflection-based XNB serializer also generated source code for a ContentTypeReader? Then you’d have the convenience of automatic serialization at build time, and the performance of custom deserialization at run-time! The source code for all the generated content readers could be compiled by the content pipeline into an assembly and deployed along with the rest of the content. Then, when the content pipeline needs to read the types at run-time, it would use the (automatically-generated) custom readers instead of reflection.

Since I’m not sure I want to make this my keep-busy-while-sick project, I’m throwing the idea out there to see if anyone else wants to try it. If you try, but get stuck, I’ll be happy to help by answering questions about dynamic code gen or msbuild customization.

A couple references to get you started:

Content Pipeline

Code Generation and Compilation

The problem was introduced by a change in the C# project system on which Game Studio relies. The C# project system has a logger that captures output from the build and reports it to the Output Window. In VS 2010, the logger only updates the Output Window when the build is entirely complete. Until then, the rest of the VS UI is completely unresponsive. If your build takes more than a few seconds – say thirty – and you click on a menu or button in the hopes that something might happen, you’ll be rewarded by the IDE window turning white and a little pop-up from the system tray telling you that VS is busy.

That little pop-up also says to report the problem to Microsoft if it happens during normal use. Well, the other week I saw it one too many times and I did that. I reported the bug through Microsoft’s customer feedback portal. This portal provides an opportunity for customers to vote on issues and say whether they can reproduce the problem. The more customers impacted by a bug, the less likely a team is going to ignore it.

Sadly, the VS team responsible for this regression doesn’t seem to think it’s a big deal. When I reported the problem, I listed six ways it could be reproduced. One of those ways was to build content in an XNA game project. They immediately rejected the bug saying it was the XNA team’s problem. Well, it isn’t – and for the record, this problem was reported to them by another Microsoft team even before VS 2010 shipped. At the time, they determined it was a regression, and still decided not to fix it.

I reactivated the bug with a sample project that reproduces the hang without any hint of XNA anywhere at all. If having the IDE freeze up while you build content bothers you, please tell the team that’s responsible for fixing it. Vote on the Connect feedback for the root cause here:

https://connect.microsoft.com/VisualStudio/feedback/details/645579/builds-occur-on-the-ui-thread-in-vs-2010

To install and use the template, right-click on the solution node of a solution where you have an XNA Content project to debug, and select Add—>New Project… Then, in the Add New Project dialog, click Online Templates on the left-hand-side, and type “debugging” in the search box in the upper-right corner.

Select the XNA Content Pipeline Debugging template, and it will start the install wizard. The template is released under the Microsoft Public License. (When I submitted to the VS Gallery, I discovered that all submitted templates must be licensed under the Ms-PL.) If you don’t agree to the license, do not install or use the software.

After creating the project, there are two string constants that you must edit before you begin debugging. They are marked with comments, but I’ll describe it here as well.

Replace the highlighted value of this first field with the full path of the content project you want to debug:

    private const string ProjectToDebug = @”TODO_FullProjectPathGoesHere“;

Then replace the highlighted value of this second field with the relative path of the item you want to debug. The path should be relative to the content project, so that it matches the item name in that project.

    private const string SingleItem = @”TODO_RelativeItemPathGoesHere“;

Now when you want to debug, you set the DebugPipeline project to be the startup project and press F5. Or, even better, just right-click on the DebugPipeline project and select Debug—>Start new instance.

By default, this project is configured to build content for Windows using the Debug configuration and the HiDef graphics profile. These settings will be apparent in the Program.cs file where you can change them to meet your needs.

Changing the graphics profile to Reach is easy, and you might need to do this if your content project is part of a Windows Phone game.

HOWEVER, for now please don’t change the TargetPlatform (I just discovered a bug as I’m writing this… Damn!). I’ll update the template tomorrow to fix the problem.

If you have feedback or questions about the template, or want to review screenshots before downloading, follow this link to the template description page in the Visual Studio Gallery:

XNA Content Pipeline Debugging

The link above has a discussion page.

The build engine used by C# projects in Visual Studio is MSBuild, so it is natural that XNA Game Studio uses MSBuild to drive the XNA Content Pipeline. It’s important to note that MSBuild is both a technology and a product. The technology is the engine, and it is implemented in the .NET Framework’s Microsoft.Build namespaces. The product is msbuild.exe, which is a simple console application that serves as a driver for the engine.

When you build a C# project or an XNA Content project in Visual Studio, the IDE invokes the MSBuild engine directly, inside the IDE’s process. When you build a project on the command line with msbuild.exe, it invokes the same engine, but in a different process than the IDE.

Ideally, the MSBuild engine would not behave any differently when building in VS or in msbuild.exe. However, there are subtle differences, and it turns out they can have significant impact.

One such problem is that the threading models is not the same between these two processes. In VS, the engine is executed in a single-threaded apartment (STA), while in msbuild.exe, the engine is executed in a multi-threaded apartment (MTA). Usually, this doesn’t make any difference for build tools… But I have a custom XAML importer in my content pipeline, and in .NET Framework 4.0, XamlReader.Load throws an exception when it is called by an MTA thread. That means in GS 4.0, my XAML importer works fine in the VS IDE, but throws an exception when building from the command line! (Note: I’m not sure whether the breaking change occurred in the new version of XamlReader or msbuild.exe, but the same code did not throw an exception in GS 3.1.)

I found this problem while working on a Canvas processor for the imported XAML. The processor was incorrectly inverting some UI elements, but when I tried to debug it by launching msbuild.exe under the debugger, the build couldn’t even get that far! Argh!

My eventual solution was to create my own console application to host the MSBuild engine. I marked the entry point method with the STAThread attribute, and configured it to build my content project. With only a few lines of code, I had a console application that would load the content pipeline and build my content with the same threading model as XNA Game Studio in the VS IDE. Better yet, because it was a standard C# console application project, debugging that project allowed me to debug my content pipeline extensions without any special configuration of the IDE!

Following that discovery, I began converting my solution into a project template so that it’s easy to share, easy to use, and works equally well in all editions of Visual Studio, including VC# Express and VS Express for Windows Phone.

The template creates a console application project with code to host the MSBuild engine and build a content project that you specify. You choose the project by providing its full path, and then tweak the build configuration by editing values in the source code. This provides full control over which configuration, platform, and profile to build. In addition, there is a field to specify which specific item you want to debug – this feature means you can select particular items in a large content project without waiting for everything else to rebuild. When the template is complete, it will be available through the Online Templates feature in the New Project Dialog.

As an aside, I’m a little disappointed that it is harder to build content projects from the command line in GS 4.0 than ever before. Several build properties do not have defaults, so the project won’t build if you don’t set them explicitly. Worse, when they’re missing, the errors don’t tell you which properties to set! My new template should take care of all of that for you. I’ll post a follow-up as soon as it’s ready.

First, the fun stuff! I worked with another developer from my team, Eli, to make a collection of distinct mini-games inspired by an SNL short film called Giraffes (warning: it is hilarious!). We did mini-games so we could add more as we completed them, and cut anything that we didn’t finish. In the end, we only finished one mini-game each – but they are reasonably polished and fun, in our humble opinions.

The two mini-games were Drive-By and Vampire Blood. (View a complete gallery of screenshots here.)

Eli’s drive-by shooting game was loosely inspired by Paper Boy, with the idea that you shoot pedestrians as you drive down the street instead of delivering newspapers. You have to avoid collisions with the other cars, both on the road and parked at the side (which can block your bullets for the pedestrians), or shoot them to blow them out of your way. It’s a twin-stick, side-scrolling shooter where right trigger fires your uzi and the left trigger sounds your horn (la cucaracha).

My vampire game involves pressing A to flap and stay airborne, while you constantly run out of blood. To replenish your blood, you have to swoop down and bite the happy shoppers at Pike Place Market – but touching the ground ends the game, so you have to be careful! If you’re too good at the game, the A-Team shows up to shoot you down and save the market.

Aside from the mini-games themselves, we spent a fair amount of time on our menu screens. We had a Zune-inspired menu control, tons of animated transitions, and head-banging giraffes popping in and out of view to our heavy metal soundtrack.

Our game was one of maybe two app week games that didn’t use the Game State Management sample to build our game screens. Instead, we used the Ferpect SDK, which I upgraded to Game Studio 4.0 specifically so we could use it. The Ferpect SDK provides an integrated design view for game components in Visual Studio. Building game screens works a bit like building user controls with Windows Forms: you drag and drop components from the Toolbox onto the design surface, edit their properties in the Properties Window, and add event handlers by double-clicking. I fixed a few bugs and added a few features to the component model while we were at it.

While we were working on our game, occasionally Eli would ask me how to do something specific in the designer. Sometimes, the feature was there and he didn’t find it, sometimes it was there but broken, and sometimes it just wasn’t there. Whenever it wasn’t there, he got frustrated. I explained that the designer doesn’t stop you from doing anything – it might not do everything, but it doesn’t take anything away. If you can’t do something in the designer, go to the code view and just write the code!

The funny thing Eli told me is that if there was no design view, he wouldn’t have thought twice about writing the code himself. But because the designer made other things so easy, it made him mad if he had to do something manually and he seemed to try really hard to avoid it. Honestly, I’m not sure what to take away from that. It’s like being happy to use a nail gun for driving nails, but then being annoyed when you need a regular hammer to pull a bent nail with its claw. A nail gun might always make your work faster and easier, but psychologically, having to use a hammer once in a while gives you the impression the nail gun falls short. Hmm.

Aside from the Ferpect SDK upgrade, I also spent some time upgrading my copy of Box2D, then my Xaml2DImporter (never released because it’s still only halfway done), and finally my Content Pipeline Debugging tool (still only available for GS 3.1). Neither Eli nor I used Box2D in our games, but I was thinking about it for another mini-game concept. While working on all these things, I realized something important: If you develop tools instead of games, you get really screwed by our (XNA Game Studio) upgrades!

All of our documentation and forum help are focused on changes in the XNA Framework API. We never really mention that we also switched from .NET Framework 2.0 to .NET Framework 4.0, from VS 2008 to VS 2010, and from MSBuild 3.5 to 4.0. If you’re a tools developer, you are likely impacted by each of these things, but they’re all documented independently by their owning teams, along with any breaking changes or gotchas. Upgrading my tools from Game Studio 3.1 to 4.0 was a challenge at times. One outcome of that effort, though, is that I’ve learned a few things. More on that another time.

I just tested my assembly optimizer on Xbox 360 for the first time. Since the whole point is to improve runtime performance, I decided to run my optimizer on Box2D and then compare some of its most intensive tests.

Without optimization, the Box2D pyramid test maxed-out at 59 milliseconds per update for the physics simulation, without firing any bullets.

With optimization, the Box2D pyramid test maxed-out at 51 milliseconds per update for the physics simulation, without firing any bullets.

That was my first real-world test, with only naive inlining of Microsoft.Xna.Framework types. That means there’s lots of room to improve, since Box2D uses a lot of internal types and functions for its physics simulation. Oh, I suppose I should note I compiled that assembly without optimizations (that is, C# optimizations were disabled).

Sweet!

I’m continuing work on my MSIL assembly rewriter. My primary goal for this tool is to optimize the IL in ways that the NetCF JIT compiler can’t. One such deficiency is that the NetCF JIT does a poor job of inlining. Especially troublesome is its lack of ability to inline methods with value type parameters, return types, or local variables. That means just about every mathematical function that games use ALL THE TIME.

Last week, I got my tool working on operator overloads. Operator overloads are actually static methods where the arguments are passed by value. For small structures like vectors, the call overhead makes operators very expensive (read: slow) compared to performing operations on each individual component of the type.

That is to say that the following addition between vectors:

Vector2 result = v1 + v2;

Is more expensive than the following sequence of additions:

Vector2 result;

result.X = v1.X + v2.X;

result.Y = v1.Y + v2.Y;

Some of the overhead is copying v1 and v2 onto the stack, some is copying the return value off the stack into result, and some is the call itself (method calls are expensive on Xbox 360).

Straight inlining of static methods is really easy in principle, and it didn’t take long to get it working (not counting the two years of procrastination). I then added some analysis to the method bodies, to determine which parameters are modified. If they’re not modified, then the parameters can be replaced by the actual arguments rather than copies placed in new local variables. For the mathematical operators, that eliminates not only the method call overhead, but ALL of the copying overhead, too.

This weekend, I worked on inlining value type constructors. These are interesting because they’re instance methods and so the inlining tool needs to deal with “this” references. Replacing the “this” references turned out to be easy – but it got harder when I thought about inlining Xbox 360 functions.

Pretty much the whole point of this tool is to help performance on the NetCF. In particular, I want to use it on XNA Framework games for Xbox 360 and I’m currently working on XNA Framework 3.1. (Yes, I will eventually update to XNA Framework 4.0, but I don’t have a Windows Phone right now, and I do have an Xbox 360.)

Ah, there’s a catch! The XNA Framework 3.1 has different assembly identities for Xbox 360 than for Windows, and XNA Game Studio does not include full MSIL assemblies for Xbox 360. Instead, it provides the metadata assemblies only, which don’t contain any method bodies. Argh! I can’t inline what isn’t there!

Of course, last week I’d already started on a solution for it. It just got harder when I tried instance methods. My solution was to use custom attributes to mark methods as inlinable for calls to identical methods on surrogate types.

For example,

public struct InlinableVector2
{
    [InlinableMethod(typeof(Vector2))]
    public static Vector2 op_UnaryNegation(Vector2 value)
    {
        Vector2 vector;
        vector.X = -value.X;
        vector.Y = -value.Y;
        return vector;
    }

//…

In the example above, the op_UnaryNegation is declared as an inlinable method that replaces calls to Vector2.op_UnaryNegation (the minus operator).

It works great, and allows me to ensure that the methods are written according to the limitations of my assembly rewriter. For example, I don’t want to inline methods with multiple return statements. It’s way easier just to rewrite such a method by hand to have one return statement than it is to write a program to do it. I wouldn’t have control over the method implementation conforming to my tool’s limits if I could only inline the actual methods of Vector2.

It got tricky when I wanted to inline constructor calls. Instance methods have ‘this’ references and field references. My inlinable method’s ‘this’ isn’t the same type as the called method’s ‘this’. That shows up here and there in the IL, too, like in field references. Well, just before writing this blog entry, I got it working.

I’m quite pleased with my progress so far. Next up is passing value types by reference (it might already work, but I haven’t tried), and then rewriting calls with equivalent, but more efficient calls (eg, replace “m3 = Matrix.Multiply(m1, m2)” with “Matrix.Multiply(ref m1, ref m2, out m3)”).

All the optimizations this tool is performing could be done by hand in the source code. However, most of the more efficient patterns are more cumbersome to write and more difficult to read. Having a tool do it at the IL level means programmers can keep writing code the most productive way, but still get the benefit of tedious micro-optimizations done everywhere.

I originally started this project about two years ago, but lost interest and put it away also about two years ago. Recently, I dusted it off and started working on it again. Tonight, I have my first optimizations running. The project is an MSIL rewriter to help overcome some of the JIT compiler shortcomings on Xbox 360. In particular, the focus of this tool is inlining method calls – starting with operator overloads on XNA Framework value types (eg, Vector2, Vector3, Matrix, etc.).

At this point, the inlining is terribly dumb. It doesn’t eliminate any of the implicit copying done in passing value types to a function, but that can be improved. (At the moment, it becomes explicit copying.) The main advantages of operator inlining is elimination of the function call, and elimination of copying arguments to and from the stack. Well, I’ve got one of the two so far.

The tool accomplishes about the same thing as manually inlining your code at the source level. However, it’s automatic, so it’s not tedious and error-prone. Also, it allows you to write source code more naturally, performing math at the vector or matrix level instead of breaking things down into components. Here’s what happens when my MSIL rewriter inlines the Vector2.op_Addition call…

Original C# Source code:

public Vector2 Add2(Vector2 a, Vector2 b)
{
    return a + b;
}  

MSIL Generated by C# Compiler:

.method public hidebysig instance valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 Add2(valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 a, valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 b) cil managed
{
    .maxstack 2
    .locals init (
        [0] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 CS$1$0000)
    L_0000: nop
    L_0001: ldarg.1
    L_0002: ldarg.2
    L_0003: call valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::op_Addition(valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2, valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2)
    L_0008: stloc.0
    L_0009: br L_000e
    L_000e: ldloc.0
    L_000f: ret
}

MSIL after Rewriting:

.method public hidebysig instance valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 Add2(valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 a, valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 b) cil managed
{
    .maxstack 9
    .locals init (
        [0] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 CS$1$0000,
        [1] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 vector,
        [2] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 vector2,
        [3] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 vector3,
        [4] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 vector4,
        [5] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 vector5,
        [6] valuetype [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2 vector6)
    L_0000: ldarg.1
    L_0001: stloc.1
    L_0002: ldarg.2
    L_0003: stloc.2
    L_0004: ldloca.s vector3
    L_0006: ldloca.s vector
    L_0008: ldfld float32 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::X
    L_000d: ldloca.s vector2
    L_000f: ldfld float32 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::X
    L_0014: add
    L_0015: stfld float32 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::X
    L_001a: ldloca.s vector3
    L_001c: ldloca.s vector
    L_001e: ldfld float32 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::Y
    L_0023: ldloca.s vector2
    L_0025: ldfld float32 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::Y
    L_002a: add
    L_002b: stfld float32 [Microsoft.Xna.Framework]Microsoft.Xna.Framework.Vector2::Y
    L_0030: ldloc.3
    L_0031: stloc.s vector4
    L_0033: ldloc.s vector4
    L_0035: stloc.s vector5
    L_0037: ldloc.s vector5
    L_0039: stloc.0
    L_003a: ldloc.0
    L_003b: stloc.s vector6
    L_003d: ldloc.s vector6
    L_003f: ret
}

Reflector’s Decompiled C# After Rewriting:

public Vector2 Add2(Vector2 a, Vector2 b)
{
    Vector2 vector3;
    Vector2 vector = a;
    Vector2 vector2 = b;
    vector3.X = vector.X + vector2.X;
    vector3.Y = vector.Y + vector2.Y;
    return vector3;
}

As you can see, the rewritten MSIL has eliminated a function call. That’s really cool because the Xbox 360 JIT compiler won’t do that!

The MSIL rewriter can be incorporated into the Game Studio build process pretty easily, making operator inlining automatic. It also supports marking your own methods with an attribute so they will be inlined as well. I’m pretty excited to see it beginning to work! I guess that’s one good thing about this cold summer keeping me indoors.