Revamping makefiles

This article is half instructional, half documentational. In the past, the Minigrand project has used a fairly basic makefile that just gets the job done. However, we can use some of the more advanced features of make to speed build-time and increase extensibility and maintainability. In the guide I will go through the changes I made to the makefiles and the rationale for these changes. This guide assumes you have some knowledge of how make works. I mean, simple, cse 311 type stuff. For a good introduction, check google.

The Previous Implementation
The previous build system for the Minigrand09 project consisted on a single top level makefile that recursively calls make on subdirectories in the source tree to build subparts of the system. This is done so that a developer can build only the subpart that he or she is currently working on, without worrying about the state of the rest of the source tree. The recursive calls are made using shellscript. The code used to do this is:

The subpart makefiles are fairly straight forward as well. Here is an example from the MGPlanner module: (I left off the header to save space) If you've used make before, perhaps for some CmpSc classes, this should all look very familiar. The only minor wrinkle is that the source is built in two steps. This is done because we need to pass special options to the linker, because we are producing a shared library as opposed to a normal application.

Improvements
While these makefiles are fairly straightforward, there are a couple of drawbacks. First, there is a lot of duplicated code in the modules because each one has a full makefile. Second, the way makefiles are called recursively cannot detect when a build of a module fails. Lastly, there are a many areas where our makefile could be expressed more elegantly. Outside of the intrinsic value of being elegant, this allows our files to be more flexible and extensible. Lets start with the global makefile.

The Global Makefile
As mentioned above, there are a couple of reasons that we may want to change the way makefiles are called recursively:
 * 1) If building a module fails, the parent make never knows.  If at any point we have modules which depend on each other, they will expect the the build to be successful up to this point.
 * 2) Because all the recursive makes are inside the same target, they cannot be run in parallel with the (-j) option.  Our codebase is currently pretty small, but if you've got a multicore machine, you might as well use it...
 * 3) If any command line arguments are passed to the parent make, they are not passed on to the recursive makes.

Here is what we'll do instead. This version creates a target for each driver directory. Thus, make knows in which directory the build has failed. Also, because they're separate targets, they can be parallelized. Lastly, the use of $(MAKE) will call make with any command line arguments given to the parent make. The -C flag tells make to run in the specified directory.

Module Makefiles
A number of small changes were made to the individual makefiles. Alone, these changes may seem trivial, or even awkward because they use features of make that are less well known. However, when taken together these changes give us a much more flexible build system. I think you'll see the advantage at the end.

Use Standard Variables
Make was made for building source files. While it can be used in a more general sense, it has many features that assist in building sources. The simplest is conventional variables. Lets jump into an example. Using the example from MGPlanner given above, we get this: Notice that we didn't set the CXX variable. It is set by make to whichever C++ compiler is default on the system.

Source-Oriented Makefile
You may notice that in our original makefile, we define which files belong in our project by both defining both the .cpp files and the .o files which are generated from them. However, it should be obvious that after adding a couple of files to this project, keeping the lists in sync requires effort. Lets use make to take care of this.

This command simply replaced the '.cpp' extension on all the files in DRI_SRC with '.o', then assigned that list to DRI_OBJ.

Use Implicit Rules
Like I stated above, make has many features made specifically to help programmers. This is probably the most obvious one. Make knows internally how to get from a .cpp to a .o file (which are then built into the final application). Make has internal rules for a lot of common programming cases. For a list, check the make manual. By looking at the manual, we see that the implicit rule that applies is Now we see the advantage of using the standard variables above. To take advantage of this, we simply remove our own line compiling sources. Now our main rule is: Whats the advantage of doing this? Well, one of makes features is that it only recompiles files if they've been changed since the last compile. However, if you notice in the previous version of this makefile, we recompiled the source files all at once. With this change, we allow make to selectively recompile the files it needs, rather than everything.

Generate Dependencies Automatically
If you've read any of the documentation on make you've heard that only the files that have changed, and files that depend on those files, are recompiled. This is great, and a real lifesaver for large projects or lazy developers. However, there is a downside. You have to tell make which files depend on what. This is tedious, and can leave you with some devious errors if dependencies are introduced in the source but not matched in the Makefile. But computers are good at tedious things! And the compiler already checks through the source files to figure out what depends on what when the project is built. Let's have GCC generate the lists of dependencies for us. Note: This is taken almost directly from Managing Projects with GNU make  For now, consider this some make black magic. If you want to figure out what is actually going on, read the relavant chapter from the Managing Projects book. In the future, it may be worthwhile to implement some of the changes mentioned here.

Wrapping it all up
Lets review. After all these changes, our makefile looks like this: Now, I'll admit, this is a bit more complected than the file we started with. But we did gain some important features.
 * 1) When add files to the project, all we have to do is add the .cpp files to the 'sources' array.
 * 2) Automatic dependency generation is a major win, at least in my opinion.
 * 3) Because compilation and linking are done in two separate steps, we do not need to recompile every .cpp file when we just make a change to one of them.

Now, notice that only the top two lines change for each of the drivers. Keeping around redundant code is a recipe for a headache when we want to change something in the future, so lets pull out the common parts.

This leaves us with two files. First the driver specific makefile:

Second, the general makefile, which is located in the directory that contains the driver subdirectories. And thats what its all about.

Conclusion
What we've gained:
 * 1) Recursive make works intuitively
 * 2) Automatic dependency regeneration means less devious bugs from building off old *.o's
 * 3) Makefile's structure follows the style of many open source projects.  Not that that means its correct but rather that it is standard.
 * 4) Reduction of redundant code - Writing the makefile for a new driver is as simple as defining its name and main source files.

What we've lost (hey, I can argue both sides of my issues)
 * 1) Simplicity
 * 2) Obvious Correctness