Speed up C++ build times by means of parallel compilation

Everyone who has worked on a fair-sized C/C++ project surely knows these scenarios: sometimes it’s unavoidable to change or introduce a new #define or declaration that is used nearly everywhere. When you hit the ‘Build’ button next time, you’ll end up recompiling nearly the whole thing. Or you just came to the office, updated your working copy and want to start the day with a clean build.

The complexity of the C++ language in combination with the preprocessor makes compilation orders of magnitude slower, compared to modern languages such as C#. Precompiled headers help here a bit, but it’s not a solution to the problem inherent to the language itself, only a mere optimization. There are coding practices that help a lot, not only in making robust and maintainable software, but also helping to improve build times. They go along the way of “minimize dependencies between modules” or “#include only what you use directly”. There are also tools that visualize #include trees and help you identify hot-spots. These are all clever tricks, which I may discuss later. However, this article is about raw, brute force :) You just got a new, powerful, N-core workstation? Well, let’s get those cores busy…

C++ translation units (.cpp files) are independent during the compilation phase and indeed are compiled in isolation. Therefore, the speed of compilation scales almost linearly with the number of processors. Most IDEs and build tools nowadays come with an option to enable parallel compilation. However, this option is almost never enabled by default. I will show you how to enable parallel compilation in build systems with which I have some experience:

  • Makefiles (Linux and Windows)
  • Qt’s Qtcreator IDE (Linux and Windows)
  • MS Visual Studio, MSBuild

Makefiles – gnu make

Telling the make program to compile in parallel could not be simpler. Just specify the -j N (or –jobs=N) option when calling make, where N is the number of things you want make to run in parallel. Good choice is to use the number of CPU cores as N. Warning: if you use -j but do not specify N, make will spawn as many parallel jobs as there are targets in the Makefile. This is neither efficient nor desirable.

Makefiles on Windows – nmake, jom

On Windows, Visual Studio comes with its own version of the make program called nmake.exe. Nmake does not know the -j option and can’t do parallel jobs. Luckily, thanks to the Qt community, there is an open source program called “jom”, which is compatible with nmake and adds the -j option. You can download the source and binary from here: http://qt.gitorious.org/qt-labs/jom. Installation is very simple, just extract the .zip file anywhere, optionally add it to %PATH%. Use it like you would use nmake.

Qt’s Qtcreator

First, let me say that Qtcreator is a very promising cross-platform IDE for (not only Qt) C++ projects, completely free and open source. Not surprisingly, Qtcreator uses the Qt’s qmake build tool first to generate a Makefile from a project description .pro file. Then it simply runs make on the generated Makefile. Qtcreator allows you to pass additional arguments to the build commands: Projects -> Build Settings -> Build Steps -> Make -> Make arguments: here you can specify the -j N option.

Project build settings in Qtcreator on Linux.

Qtcreator on Windows

If you use Qtcreator on Windows, the story is almost the same with only minor differences. On Windows platforms Qtcreator uses the MinGW32 build toolchain. Unfortunately due to the way (bug) MinGW’s make works on Windows and the way Qt’s qmake generates Makefiles, the -j option doesn’t work. The reason why and various workarounds are described in this discussion. One easy way is to override the mingw32-make.exe and use jom.exe instead.

Project build settings in Qtcreator on Windows.

MS Visual Studio, MSBuild

Not surprisingly, the Visual Studio/C++ IDE uses a completely different build system than the GNU toolchain, called MSBuild (formerly VCBuild). If you only work within the IDE and do not wander into the command line world very often, you probably haven’t even bumped into this tool. Yet it is invoked behind the scenes whenever you press the build button. In short, the process is as follows: Visual Studio keeps the list of project source files, compiler and linker options in a .vc(x)proj file. At the start of each build, the MSBuild tool then crunches the .vcxproj file and outputs a list of commands for invoking the compiler, the linker and any other tools involved in the build process.

The MS Visual C++ compiler (cl) can compile multiple source files in parallel, if you tell it to using the /MP switch. It will then spawn as many parallel processes as there are installed CPU cores in the system. You can set this option conveniently from the IDE: Project -> Properties -> Configuration Properties -> C/C++ -> General -> Multi-processor Compilation: Yes (/MP). This option will be saved into the .vcxproj file, so multi-process compilation will be used regardless if you build in the IDE or from the command line.

Enable parallel compilation for a MSVC project.

Multiple simultaneous builds

In Visual Studio, you can go even a little further and tell the IDE to build multiple projects in parallel. To enable this, go to: Tools -> Options -> Projects and Solutions -> Build and Run: and set the maximum number of parallel project builds. When building a solution from the command line, pass this option to MSBuild: /maxcpucount[:n]. This can be useful, if your solution consists of many small, independent projects. If your solution contains just a single or a couple of big projects, you’ll probably do best with the /MP option only.

Setting maximum number of parallel builds.

In closing

Modern machines come with a lot of horsepower, the trend is that the number of CPU cores will be increasing. Why not leverage this and turn your workspace builds from a lunch break into “only” a coffee break? Parallel compilation speeds up the build process almost linearly with the number of CPU cores.

However compilation is only one part of the story. Then there’s linking. It is not uncommon that a project, which compiles in seconds, takes minutes to link. I will point you to some articles on how to speed up linking in my next post.

Ditching Facebook, moving to Twitter

A month ago, I had an interesting talk with my friend who refused to join Facebook or any other “social” media for that matter. In fact, I have a couple of friends outside any virtual friendship circles, but they are certainly a tiny minority. Until then, I was a devoted fan of FB (previously Orkut, Unister, Tagged and maybe some more), having a fairly large network of “friends”, being tagged on many photos, checking other people’s status many times every day. Perhaps, you could even say, I suffered a slight addiction.

My friend convinced me, that with FB I get no real value added to my life. On the contrary, I got:

  • a network of people (contacts), I don’t really care about,
  • my time consumed by constantly reading and watching pointless content posted by all those people,
  • irrational fear of losing touch with people, with whom I would not want to get in touch anyway,
  • fear of posting really interesting personal stuff, because the whole thing got too public.

I came to the resolution to close my FB account. Finally, it was the ‘waste of time for nothing in return’ argument that convinced me. After all, I consider myself somewhat a follower of zen principles, not in the religious sense, but more in the ‘gain by losing’ sense. As a result, I got a lot less distracted over the day and a lot more focused on important things.

I don’t want to only bash FB and such media. There is a great and proven marketing and promotional potential in there. I can also see the appeal to teenagers. But for a grown up guy who is not selling anything, there is little to none to gain. However, as a starting freelance IT pro I felt that I should maintain my online visibility. I’ve had my LinkedIn profile for a while, later added a blog and a personal web/portfolio page. The blog (even with my sparse posting habits) proved to be useful for starting a conversation about some of my side projects. Most recently, I opened a Twitter account. So far, I have found this combination of internet tools powerful, yet not obtrusive, only serving me when I have something to say.

So what shall I be tweeting about? I guess, like most of the people, semi-random short thoughts. I ike to share them when I learn something interesting and eye-opening, mainly from the area of my expertise: software engineering, coding practices, C++, graphics programming and lately the Qt framework. I will also tweet about project updates, like the QOF for Visual Studio and possibly others. I really like the 140 character limit, because it makes you think twice before you post.

What do you think; does FB (or Google+ or any other) give you something that is worth the while?

Quick Open File, now available for VS 2010 Beta 2

Those of you who embarked on the Visual Studio 2010 Beta 2 train, surely miss my Quick Open File plugin :) Well, good news for you: here it is.

It was not as straightforward to port it over to VS 2010 as I first thought it would be. The new VS IDE is now WPF based, but my plugin is Forms-based. The experience could be best described as a half-day trial and error, struggling to implement undocumented interfaces. Well, I guess that’s part of the beta experience…

Anyway, here it is and I hope you’ll like it. You can get the plugin at Visual Studio Gallery or at my site. Or better yet, open Visual Studio, go to Extension Manager, click at Online Gallery and type “Quick Open File” to the search box. This way you can install it directly from the IDE.

Quick Open File for Visual Studio – minor update

I found out that people come to my homepage mainly to download the Quick Open File for Visual Studio 2008 plugin. In fact there have been over 700 downloads since April-2009, when I first released it. This makes me quite happy because I’ve finally created something people find useful :)

As the name suggests, it’s a little utility for Visual Studio 2008 that allows you to find and open any file anywhere in the solution, no matter how deeply buried in the project structure. You just press Ctrl+K, Ctrl+O (of course, you can customize the shortcut key), type a few letters from the file name and hit Enter. And voila, your file is on the screen.

Quick Open File plugin window.

Today I released version 1.1 which adds the option to open the selected file in any other associated editor. The behavior is as follows:

  • Pressing Enter will open the selected file in the default editor Visual Studio has associated with the file type.
  • Pressing Shift+Enter will open the “Open With” dialog first where you can select in which editor to open the file.

You can find the new version of the plugin at Visual Studio Gallery, or directly at my site.

Finally on the web

Hi there!

As some of you may know, I try to pursue a freelance career for a change. So I decided to build myself a website where I can present my work in a convenient way. I’m happy that I finally managed to put it up into a state which I am not completely ashamed of :) I tried to summarize my project portfolio which consits of some of the more interesting school projects and other private software projects. The most interesting things (my diploma thesis, my work-in-progress 3D engine) are yet to come…

You can also find a picture gallery there where I mainly put my nature photos. I’ve used many of those photos as desktop backgrounds. Maybe some of them will inspire you too :)

Follow

Get every new post delivered to your Inbox.