17 replies to this topic

[Nick]

Hi all,

I'm planning on adding a large library to one of my project, but will only use a fraction of its functionality. In order to keep build times low I'd like to keep just the source files that are actually required for the functionality I use. Using a separately built library is not an option because the libary is updated regularly and I'm making some small changes to its source code.

So does anyone know of any tool to detect which files can be safely removed from the build? I'm using Visual C++. Note that a full rebuild takes tens of minutes and there are hundreds of files so removing files one by one is not an option. :surrender

Thanks!

Nick

[alphadog]

Do you use any kind of build manager? (ex: SCons, CMake or CruiseControl)

[alphadog]

Another (or supplementary) approach is to build faster:
1. Fastest drive setup (striped RAID on fast drives)
2. Fastest CPU
3. Parallel and/or distributed builds, like w/MPCL (parallel) or Incredibuild (basically a parallel and distributed Make), but it requires multiple systems that can be used for builds.

EDIT: A buddy of mine also pointed me to Unity Building.

[kvakvs]

So goal is keeping build times low.

Put third party libraries in separate solution.
Configure every project in that solution to put output LIB files to a single specific folder, like "linklib" in my example.
Build it once for release and for debug.
Link your main project with libs from this folder.
--
Don't bother with removing "unused" files. That will be much harder than you might think.

[Nick]

alphadog said:

Do you use any kind of build manager? (ex: SCons, CMake or CruiseControl)

No, and I'm not planning to. I love tapping F5 to build and run my application. :happy:

[Nick]

alphadog said:

1. Fastest drive setup (striped RAID on fast drives)

I already have a 10.000 RPM Raptor driver. I'm saving for an SSD...

Quote

2. Fastest CPU

Core i7. Check.

Quote

3. Parallel and/or distributed builds, like w/MPCL (parallel) or Incredibuild (basically a parallel and distributed Make), but it requires multiple systems that can be used for builds.

Actually I would love a compiler that uses all the power of my Core i7...

Anyway, while this is valid advice it's really no cure. I already have a really fast system so there's little left to improve. Really the issue I'm having is that it's a gigantic project and I'm only using a small portion of it (but it's fairly tightly interwoven).

[Reedbeta]

If the problem is there are too many files to remove them one-at-a-time, you could try the opposite approach. First build only the files that you know are needed, and see what external dependencies are not found. Search and add the appropriate files, and repeat. Of course if the whole thing is tightly interwoven you may not find a very small minimal set of files. You may have to delve into the library and manually tease apart your desired functionality from the rest.

[alphadog]

Although you have a good machine, you are not doing any striping (RAID)? The usual biggest immediate build issue is file IO, not CPU IO. Get another spindle on your files and it should help. Also, a third drive for swap helps also.

BTW, SSD may not help, esp. now and until the SSDs mature, since they write slow, esp. with lots of random, small file writes.

Parallelizing or distributing your builds can have a dramatic effect on time. But, you can only gain that by adding a layer to your build.

Another thought: RAM drive. Use something like SuperVolume to mitigate the effect of reboots or crashes on data loss.

[alphadog]

Or, brute-force the damn thing! We're coders, aren't we?

Write a script that:
a) moves a possible target file to a temp folder
runs a build
c) check if it errors, move it back

Loop ad nauseum overnight and don't look back. :sneaky:

[kvakvs]

Another idea is to keep project files on partition X, large data files (if you have) on partition Y, while your temporary build files are going to partition Z, not very large like under 50G, and defragmented with fast optimization on a daily (max weekly) basis. Check mydefrag.com.

[Nick]

kvakvs said:

Put third party libraries in separate solution.
Configure every project in that solution to put output LIB files to a single specific folder, like "linklib" in my example.
Build it once for release and for debug.
Link your main project with libs from this folder.

That's not so different from what I have right now. But like I said the problem is that the third party library gets updated quite often and I have to patch a few things every time.

Quote

Don't bother with removing "unused" files. That will be much harder than you might think.

Actually that's working quite well. I'm already using only about one third of the complete library and in the past few months the updates never needed any code outside of that part. Even if new dependencies pop up it's pretty easy to search for the needed source files and to include them in the build. But still, it's a huge library and any additional files I can remove would be helpful.

Anyway, I found a strategy that is workable. The compiler doesn't detect when I remove files (so I need a full rebuild to check whether it was needed or not), but it does detect when I empty the content. This way it only recompiles the (empty) file and relinks, which is a lot faster. It's still a tedious job for over a hundred files but it looks like it will only take a couple hours which is well worth it...

[Nick]

alphadog said:

Although you have a good machine, you are not doing any striping (RAID)? The usual biggest immediate build issue is file IO, not CPU IO. Get another spindle on your files and it should help. Also, a third drive for swap helps also.

No, I'm not using RAID. I don't think that's very useful. RAID increases sequential read and write speeds but compilation has to deal with lots of little files so it should benefit more from fast search times. RAID in fact adds more latency...

Quote

BTW, SSD may not help, esp. now and until the SSDs mature, since they write slow, esp. with lots of random, small file writes.

They write slow-er than they read, but I wouldn't call them slow. Recent SSD's beat Raptor drives on all fronts, and since the search time is a thousand times faster than a HDD I think even a cheaper SSD could outperform them at compilation.

Quote

Parallelizing or distributing your builds can have a dramatic effect on time. But, you can only gain that by adding a layer to your build.

I hope Visual Studio 2010 fixes that. And I also hope it puts my 6 GB of RAM to use.

Quote

Another thought: RAM drive. Use something like SuperVolume to mitigate the effect of reboots or crashes on data loss.

That's not a bad idea! Data loss is actually not a big issue since I'm using source control and I've already made it a habit to save my work as a patch on a USB stick regularly.

[kvakvs]

In every your project, open C++/Additional compiler options and add manually "/MP" switch there. It will utilize all your cores for build instead of just one core. Switch is undocumented and not really recommended for clean release builds, but might speed up things alot.

[Rofar]

I suggest removing a single file each time you build the project. So instead of spending a few hours working on it, spend a few minutes each time you need to do a build. Of course, this suggestion is probably easier said than done since you would more than likely have to choose the file to remove carefully and it may not be possible to remove just a single file due to dependency issues in other files (that you are not using and are already targeted for the remove list).

[Nick]

kvakvs said:

In every your project, open C++/Additional compiler options and add manually "/MP" switch there. It will utilize all your cores for build instead of just one core. Switch is undocumented and not really recommended for clean release builds, but might speed up things alot.

That's awesome! :yes: It really does reduce compilation time pretty significantly.

Do you know of any issues with it? A few sites report that it doesn't work properly when multiple files try to update .pdb files. But maybe that has been fixed with recent service packs?

[kvakvs]

Nick said:

That's awesome! :yes: It really does reduce compilation time pretty significantly.

Do you know of any issues with it? A few sites report that it doesn't work properly when multiple files try to update .pdb files. But maybe that has been fixed with recent service packs?

Did not notice any problems in my two solutions (one with third party libs about 10-12 libraries and another with 5 projects main executable). Well, except usual visual studio screw-ups with Intellisense, and random insignifficant trouble which seemingly were not related to having /MP enabled.

Also you can disable it for release builds, hope you don't do them 10 times everyday.

[JarkkoL]

You should look at the unity building alphadog was referring. It's a classic hack to significantly reduce build times (IIRC, we went down to ~quarter of the original full rebuild time of UE3 by converting to unity build). It helps with full rebuild performance, but if you have to compile only a single file frequently, then compilation is slower. It has also positive impact on link times.

[Alex.11]

You Start with each include file, and ensure that each include file only includes what is necessary to compile itself. Any include files that are then missing for the C++ files, can be added to the C++ files themselves.
For each include and source file, comment out each include file one at a time and see if it compiles.
It is also a good idea to sort the include files alphabetically, and where this is not possible.