Faster Content Builds That Don’t Lock-up Your IDE

Yesterday, I decided to explore what a sufficiently-motivated developer could do to improve content build times in XNA Game Studio 4.0. In a relatively short time, I was able to hack together a solution that distributes content builds across available processors, and doesn’t lock up the IDE (as described here). This means Visual Studio remains responsive while your content builds, you can track build progress in the Output Window, and the whole thing will finish faster.

Before continuing, let me state clearly that what I’m about to describe requires Visual Studio 2010 Professional. Those of you using Visual C# Express Edition won’t be able to use this technique.

Okay, now I’m going to explain what I did. In my sample solution, I reduced the build time by 25% on my dual-core laptop (from 38 seconds down to 29 seconds).

For my experiment, I started with the Ship Game starter kit from create.msdn.com. I chose this starter kit because it contained enough content that locking up the IDE on rebuilds would get annoying really fast.

Here’s what I did (short version; longer version at the end):

  1. Split content into multiple content projects, all building to the same output directory.
  2. Add a Game Library project to the solution that references all the content projects.
  3. Add a VC++ Makefile project to the solution that builds the game library from (2) with multi-proc support enabled.

The key here is that I introduced a VC++ Makefile project to initiate the content build. This project type invokes arbitrary, user-defined commands, and it isn’t all broken like the C# projects. Specifically, it doesn’t lock up the UI thread when it executes long-running commands, and it correctly redirects all the command output to the Output Window, and reports errors in the Error List.

To have the makefile project build all my content projects in parallel, I set up the Build command to build the Game Library project using msbuild.exe with multi-processor support enabled.

Example Build Command:

“$(MSBuildBinPath)\msbuild.exe” /m:$(NUMBER_OF_PROCESSORS) “$(SolutionDir)Content_All\Content_All.csproj”

For Rebuild and Clean, I simply added /t:Rebuild and /t:Clean, respectively.

Note: VC# Express doesn’t support VC++ projects, which is why this won’t work for Express users.

So that’s what I did, and it works great. Content builds don’t lock up the IDE, and the Output Window displays the name of each asset as it is compiled (some reassuring feedback during long-running builds).

At this stage, my solution is set up to build content projects referenced by the Game Library project (Content_All) in parallel. When I tried it the first time, I only squeezed 2 seconds off the build time. The bottleneck in my build was the project using the normal mapping processor, so I added another project, and moved some of the models and dependencies over there. That brought more parallelism into the build and the overall build time dropped to 29 seconds (from 38).

I experimented a little with putting the audio in another project, and shuffling some of the models back and forth, but gave up before finding any additional gains. On a machine with more processors (my laptop has two), there is potential for more savings.

I should note that in order to split up the custom model processing into multiple projects, I had to include some of their dependencies twice. For example, both projects build the NormalMapping shader. The cost of the redundancy was offset by a greater savings from parallel processing.

The drawbacks to doing this are:

  1. You basically need two extra projects in your solution to enable it, for each target platform.
  2. You need to manually edit the Solution Configurations to make things build properly.
  3. For Windows projects, the content won’t be recognized by ClickOnce publishing.

The first isn’t so bad, because you can hide the extra projects in a Solution Folder. The second is something many people don’t really understand, but should be part of any advanced developer’s tool kit.

Another thing I want to note is that I separated the assets that use the custom processor from the assets that use standard processors to reduce incremental build times when NormalMappingModelProcessor.dll is modified. The standard content won’t rebuild at all – and thanks to splitting the custom content into two projects, the custom content can rebuild fully in less time as well.

Angry grumbling: I had intended to provide my modified ShipGame solution as a downloadable sample, because the web site said it was under Ms-Pl. However, the actual ZIP file contains a license file that describes the “XNA Premium Content” license, which prohibits redistribution of unmodified software. :(

Before I go, let me leave you with this disclaimer: your mileage may vary. I do not plan to write out a click-by-click tutorial, but I added more detail below.

Here’s what I did (longer version):

  1. Move ShipGame content project up to the solution folder, and rename it Content_Main.
  2. Add new content project, named Content_Custom.
  3. Delete the Content Reference from ShipGameWindows.
  4. Add new Game Library project to the solution, named Content_All.
  5. Change the output folder of Content_All to match the output folder of ShipGameWindows.
  6. In Content_All, add references to Content_Main and Content_Custom.
  7. Change the Content Root Directory of all content projects to “Content”.
  8. Add new C++ Makefile project, named Content_BuildAllParallel, and configure the Build, Rebuild, and Clean commands (details below).
  9. Move all the content that depends on NormalMappingModelProcessor into Content_Custom, along with any additional content it depends on.
  10. In Content_Custom, add a reference to NormalMappingModelProcessor.
  11. In Content_Main, remove the reference to NormalMappingModelProcessor.
  12. Edit the solution configuration so that Content_All and NormalMappingModelProcessor are excluded from all solution builds. Make sure that Content_BuildAllParallel is built when the rest of the game is built.
About these ads

About badcorporatelogo

My name is Stephen Styrchak, and I work as a software developer in Seattle, Washington, USA.
This entry was posted in XNA Game Studio. Bookmark the permalink.

8 Responses to Faster Content Builds That Don’t Lock-up Your IDE

  1. jason says:

    very awesome, thanks for spending the time to write this up

  2. rcashie says:

    Good stuff,
    Looks like my initial post got lost somehow, anyway…

    I am thinking of pulling out the actual ‘content build’ by creating a dummy content project for the game project to reference. When the game is built, the dummy content project just copies ‘pre-built’ xnb files to the output directory. (Easily done for an asset by setting the build action to none and making sure it is set to ‘copy if newer’)

    The actual content build would be done by an external batch script that instructs msbuild to build the ‘real’ content project. Then as a post process it updates the dummy content project to keep it in sync.

    Think this would work?

  3. rcashie says:

    I see, so the configuration could affect the content output, makes sense. However, it looks like incremental builds don’t work too well when you switch configs.. Without changing any code or data, I tried this: Built release, built debug, then built release and debug again and it built the content everytime. Maybe something is wrong with my projects and/or configurations.

    @ For Xbox 360 games, the deploy step deploys all files in the project’s output folder

    If that’s the case then, I’ll just build my content offline and drop them into the build folders!

    Looks like the programmer for Fez was going doing the same thing I was about to do until someone pointed out what you just did!

    http://theinstructionlimit.com/things-you-should-know-beforewhile-making-an-xbox-xna-game

    Cheers for your replies man! Glad I posted here before implementing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s