Revamping makefiles
modified on 1 April 2010 at 21:17 - 593 views

From RoboWiki

Jump to: navigation, search

This article is half instructional, half documentational. In the past, the Minigrand project has used a fairly basic makefile that just gets the job done. However, we can use some of the more advanced features of make to speed build-time and increase extensibility and maintainability. In the guide I will go through the changes I made to the makefiles and the rationale for these changes. This guide assumes you have some knowledge of how make works. I mean, simple, cse 311 type stuff. For a good introduction, check google.

Contents

[edit] The Previous Implementation

The previous build system for the Minigrand09 project consisted on a single top level makefile that recursively calls make on subdirectories in the source tree to build subparts of the system. This is done so that a developer can build only the subpart that he or she is currently working on, without worrying about the state of the rest of the source tree. The recursive calls are made using shellscript. The code used to do this is:

# Project drivers
drivers:
	echo Building Drivers...
	for d in $(DRIVERDIRS); do (cd $$d; make); done

The subpart makefiles are fairly straight forward as well. Here is an example from the MGPlanner module: (I left off the header to save space)

# Include global project variables
include ../../makefile.inc
 
# Driver definitions
DRI_SRC = MGPlanner.cpp dstar/Dstar.cpp
DRI_OBJ = MGPlanner.o Dstar.o
DRIVERNAME = libMGPlanner.so
 
# Build all
all: $(DRIVERNAME)
 
# Build this driver
$(DRIVERNAME): $(DRI_SRC)
	g++ -fpic -g3 `pkg-config --cflags --libs playercore ` -c $(DRI_SRC)
	g++  -shared -rdynamic -o $(DRIVERNAME) $(DRI_OBJ)
 
# Make the test case of this driver
test:
	g++ -Wall -Wextra `pkg-config --cflags --libs gtk+-2.0` $(TST_SRC) -o Test
 
# Clean all
clean:
	rm -f *.o *.so

If you've used make before, perhaps for some CmpSc classes, this should all look very familiar. The only minor wrinkle is that the source is built in two steps. This is done because we need to pass special options to the linker, because we are producing a shared library as opposed to a normal application.

[edit] Improvements

While these makefiles are fairly straightforward, there are a couple of drawbacks. First, there is a lot of duplicated code in the modules because each one has a full makefile. Second, the way makefiles are called recursively cannot detect when a build of a module fails. Lastly, there are a many areas where our makefile could be expressed more elegantly. Outside of the intrinsic value of being elegant, this allows our files to be more flexible and extensible. Lets start with the global makefile.

[edit] The Global Makefile

As mentioned above, there are a couple of reasons that we may want to change the way makefiles are called recursively:

  1. If building a module fails, the parent make never knows. If at any point we have modules which depend on each other, they will expect the the build to be successful up to this point.
  2. Because all the recursive makes are inside the same target, they cannot be run in parallel with the (-j) option. Our codebase is currently pretty small, but if you've got a multicore machine, you might as well use it...
  3. If any command line arguments are passed to the parent make, they are not passed on to the recursive makes.

Here is what we'll do instead.

# Project drivers (call made recursively)
drivers: $(DRIVERDIRS);
$(DRIVERDIRS):
	@echo Building $@ Driver...
	$(MAKE) -C $@

This version creates a target for each driver directory. Thus, make knows in which directory the build has failed. Also, because they're separate targets, they can be parallelized. Lastly, the use of $(MAKE) will call make with any command line arguments given to the parent make. The -C flag tells make to run in the specified directory.

[edit] Module Makefiles

A number of small changes were made to the individual makefiles. Alone, these changes may seem trivial, or even awkward because they use features of make that are less well known. However, when taken together these changes give us a much more flexible build system. I think you'll see the advantage at the end.

[edit] Use Standard Variables

Make was made for building source files. While it can be used in a more general sense, it has many features that assist in building sources. The simplest is conventional variables. Lets jump into an example. Using the example from MGPlanner given above, we get this:

# Include global project variables
include ../../makefile.inc
 
# Driver definitions
DRI_SRC = MGPlanner.cpp dstar/Dstar.cpp
DRI_OBJ = MGPlanner.o Dstar.o
DRIVERNAME = libMGPlanner.so
includes = `pkg-config --cflags --libs playercore `
CXXFLAGS += $(includes)
CXXFLAGS += -fpic -g3  
 
# Build all
all: $(DRIVERNAME)
 
# Build this driver
$(DRIVERNAME): $(DRI_SRC)
	$(CXX) $(CXXFLAGS)-c $(DRI_SRC)
        $(CXX)  -shared -rdynamic -o $(DRIVERNAME) $(DRI_OBJ)
 
# Clean all
clean:
	rm -f *.o *.so

Notice that we didn't set the CXX variable. It is set by make to whichever C++ compiler is default on the system.

[edit] Source-Oriented Makefile

You may notice that in our original makefile, we define which files belong in our project by both defining both the .cpp files and the .o files which are generated from them. However, it should be obvious that after adding a couple of files to this project, keeping the lists in sync requires effort. Lets use make to take care of this.

DRI_SRC = MGPlanner.cpp dstar/Dstar.cpp
DRI_OBJ = $(DRI_SRC:.cpp=.o)

This command simply replaced the '.cpp' extension on all the files in DRI_SRC with '.o', then assigned that list to DRI_OBJ.

[edit] Use Implicit Rules

Like I stated above, make has many features made specifically to help programmers. This is probably the most obvious one. Make knows internally how to get from a .cpp to a .o file (which are then built into the final application). Make has internal rules for a lot of common programming cases. For a list, check the make manual. By looking at the manual, we see that the implicit rule that applies is

$(CXX) -c $(CPPFLAGS) $(CXXFLAGS)

Now we see the advantage of using the standard variables above. To take advantage of this, we simply remove our own line compiling sources. Now our main rule is:

# Build this driver
$(DRIVERNAME): $(DRI_SRC)
        $(CXX)  -shared -rdynamic -o $(DRIVERNAME) $(DRI_OBJ)

Whats the advantage of doing this? Well, one of makes features is that it only recompiles files if they've been changed since the last compile. However, if you notice in the previous version of this makefile, we recompiled the source files all at once. With this change, we allow make to selectively recompile the files it needs, rather than everything.

[edit] Generate Dependencies Automatically

If you've read any of the documentation on make you've heard that only the files that have changed, and files that depend on those files, are recompiled. This is great, and a real lifesaver for large projects or lazy developers. However, there is a downside. You have to tell make which files depend on what. This is tedious, and can leave you with some devious errors if dependencies are introduced in the source but not matched in the Makefile. But computers are good at tedious things! And the compiler already checks through the source files to figure out what depends on what when the project is built. Let's have GCC generate the lists of dependencies for us. Note: This is taken almost directly from Managing Projects with GNU make [1]

ifneq "$(MAKECMDGOALS)" "clean"
 
  -include $(dependencies)
 
endif
 
%.d: %.c
        $(CC) -M $(CPPFLAGS) $< > $@.$$$$;                      \
        sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@;     \
        rm -f $@.$$$$

For now, consider this some make black magic. If you want to figure out what is actually going on, read the relavant chapter from the Managing Projects book. In the future, it may be worthwhile to implement some of the changes mentioned here.

[edit] Wrapping it all up

Lets review. After all these changes, our makefile looks like this:

drivername	:= libMGPlanner.so
sources		:= MGPlanner.cpp
objects		:= $(sources:.cpp=.o)
dependencies	:= $(sources:.cpp=.d)
 
includes := $(shell pkg-config --cflags playercore)
libs := $(shell pkg-config --libs playercore)
CXXFLAGS += $(includes)
CXXFLAGS += -fpic -g3		# prep for shared library
LDFLAGS += -shared -rdynamic	# link as shared library
 
$(drivername): $(objects)
	$(CXX) $(LDFLAGS) -o $@ $^
 
# You may notice that there are no rules defined for *.cpp
# `make' will actually automatically know to get a *.o from
# its corresponding .cpp
 
.PHONY: clean
clean:
	$(RM) $(drivername) $(objects) $(dependencies)
 
## Keeping track of dependencies is hard, lets let g++ do it for us
ifneq "$(MAKECMDGOALS)" "clean" 
  -include $(dependencies) 
endif
 
%.d: %.cpp
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< | \
	    sed 's,\($*\.o\) *:,\1 $@: ,' > $@.tmp 
	mv $@.tmp $@

Now, I'll admit, this is a bit more complected than the file we started with. But we did gain some important features.

  1. When add files to the project, all we have to do is add the .cpp files to the 'sources' array.
  2. Automatic dependency generation is a major win, at least in my opinion.
  3. Because compilation and linking are done in two separate steps, we do not need to recompile every .cpp file when we just make a change to one of them.

Now, notice that only the top two lines change for each of the drivers. Keeping around redundant code is a recipe for a headache when we want to change something in the future, so lets pull out the common parts.

This leaves us with two files. First the driver specific makefile:

# Filename Minigrand09/Source/MGPlanner/makefile
drivername	:= libMGPlanner.so
sources		:= MGPlanner.cpp
 
include ../drivers.mk

Second, the general makefile, which is located in the directory that contains the driver subdirectories.

# Filename Minigrand09/Source/drivers.mk
objects		:= $(sources:.cpp=.o)
dependencies	:= $(sources:.cpp=.d)
 
includes := $(shell pkg-config --cflags playercore)
libs := $(shell pkg-config --libs playercore)
CXXFLAGS += $(includes)
CXXFLAGS += -fpic -g3		# prep for shared library
LDFLAGS += -shared -rdynamic	# link as shared library
 
$(drivername): $(objects)
	$(CXX) $(LDFLAGS) -o $@ $^
 
# You may notice that there are no rules defined for *.cpp
# `make' will actually automatically know to get a *.o from
# its corresponding .cpp
 
.PHONY: clean
clean:
	$(RM) $(drivername) $(objects) $(dependencies)
 
## Keeping track of dependencies is hard, lets let g++ do it for us
ifneq "$(MAKECMDGOALS)" "clean" 
  -include $(dependencies) 
endif
 
%.d: %.cpp
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< | \
	    sed 's,\($*\.o\) *:,\1 $@: ,' > $@.tmp 
	mv $@.tmp $@

And thats what its all about.

[edit] Conclusion

What we've gained:

  1. Recursive make works intuitively
  2. Automatic dependency regeneration means less devious bugs from building off old *.o's
  3. Makefile's structure follows the style of many open source projects. Not that that means its correct but rather that it is standard.
  4. Reduction of redundant code - Writing the makefile for a new driver is as simple as defining its name and main source files.

What we've lost (hey, I can argue both sides of my issues)

  1. Simplicity
  2. Obvious Correctness

[edit] References

  • Stack Overflow quick overview A quick run though of what you need to know to get your project up and running with make.
  • An Introduction to the UNIX Make Utility A more in-depth introduction.
  • The Make Manual - The make manual provides a decent, if dated, introduction to the make utility. It also goes into all the nitty gritty details. Read this if you want to realize just how ridiculous a utility make is. (ex. make has integration with 2 source control systems, including the precursor to CVS, which was the basis for SVN)
  • Managing Projects with GNU make, 3rd Edition by Robert Mecklenburg - Read this for a practical guide on how to effectively use make in your projects. From simple class projects to enterprise code in the millions of lines, this book explains how to use make in actual projects. Along the way it introduces the reader to many of makes most useful features, as well as providing examples of where they're used (something missing from the make manual).
Retrieved from "http://www.psurobotics.org/wiki/index.php?title=Revamping_makefiles"