Difference between revisions of "Make"

From HPC Wiki
Jump to navigation Jump to search
m
 
(16 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Make is a build automation tool that automatically builds arbitrary targets (in most cases program executables or libraries) from a list of prerequisites (which can be files or other targets). Make is controlled by reading files called Makefiles which specify how to derive the target. While make is a standard UNIX tool today in most cases people refer to the [https://www.gnu.org/software/make/ GNU make] implementation which provides a lot of extensions.
+
[[Category:HPC-Developer]][[Category:HPC-User]]
 +
Make is a build automation tool, that can build arbitrary targets (in most cases program executables or libraries) from a list of prerequisites (which can be files or other targets). Make is controlled by reading files called Makefiles, which specify how to derive a target. While make is a standard UNIX tool, today in most cases people refer to the [https://www.gnu.org/software/make/ GNU make] implementation, which has a lot of extensions.
 +
 
 +
 
 +
__TOC__
 +
 
  
 
The main benefits of using a build automation tool are:
 
The main benefits of using a build automation tool are:
 
* Automation :-) of repetitive command execution
 
* Automation :-) of repetitive command execution
* Consistent build results preventing errors
+
* Consistent build results preventing errors even for complex build chains
 
* Easier configuration for different tool chains
 
* Easier configuration for different tool chains
 
* Formulate complex build settings with possibly various targets from the same source tree
 
* Formulate complex build settings with possibly various targets from the same source tree
 
* Automatic matching of prerequisites
 
* Automatic matching of prerequisites
* Speed up compilation by rebuilding only changed source files targets and parallel build support
+
* Speed up compilation by rebuilding only changed source file targets and parallel build support
  
Any serious software development effort sooner or later has to use a automated build tool. While make is the standard build tool on Linux systems there are various alternatives available, sometimes standalone and sometimes as a frontend to make.
+
Any serious software development effort sooner or later has to use an automated build tool. While make is the standard build tool on Linux systems there are various alternatives available, sometimes standalone and sometimes as a frontend to make.
  
 
== Basic usage ==
 
== Basic usage ==
A Makefile consists at its core of rules:
+
A Makefile consists of rules:
  
 
<syntaxhighlight lang="make">
 
<syntaxhighlight lang="make">
Line 18: Line 23:
 
         recipe
 
         recipe
 
</syntaxhighlight>
 
</syntaxhighlight>
Each rule is made of the target, which is either a real build result as a executable or library or just a label without a build result. After colon is a list of prerequisites, make will search either for a file for every prerequisite or tries to match a rule to build it. Finally the recipe is a command or list of commands to generate the target. A recipe consists of any shell command but may also contain make variables or macros.
+
Each rule is made of the target, which is either a real build result as an executable or library, or just a label without a build result. After the colon there is a list of prerequisites. Make will search for every prerequisite either for a file or tries to match a rule to build it. Finally the recipe is a command or list of commands to generate the target. A recipe consists of any shell command but may also contain make variables or macros.
  
A simple Makefile might look like that:
+
This is an example for a simple Makefile:
  
 
<syntaxhighlight lang="make">
 
<syntaxhighlight lang="make">
Line 39: Line 44:
  
 
By calling make without any arguments it will search for any file called makefile or Makefile and pick the first target in the file (indicated in this example by myProg:).  
 
By calling make without any arguments it will search for any file called makefile or Makefile and pick the first target in the file (indicated in this example by myProg:).  
The default target (the first one in the file) is matched first. Next make searches for the prerequisites, in this example a list of object files to link the executable. Next it either tries to find the object file or execute a rule to build it. In our case there is an explicit rule to build every object file. In case the object file was already build or is a primary source file make will check if the prerequisites are newer than the build result and automatically rebuild this file. This works in a recursive fashion: A target depends on  prerequisites which themselves depend on prerequisites and so on. Make will automatically figure out which parts need to be executed in which order to get it right.
+
The default target (the first one in the file) is matched first. Next make searches for the prerequisites, in this example a list of object files to link the executable. Next it either tries to find the object file or execute a rule to build it. In our case there is an explicit rule to build every object file. In case the object file was already build or is a primary source file make will check if the prerequisites are newer than the build result and automatically rebuild this file. This works in a recursive manner: A target depends on  prerequisites which themselves depend on prerequisites and so on. Make will automatically figure out which parts need to be executed in which order to get it right.
  
 
Above example also contains a target which simply triggers a command execution: The clean target does not build anything but automates cleaning up target and intermediate build results.
 
Above example also contains a target which simply triggers a command execution: The clean target does not build anything but automates cleaning up target and intermediate build results.
Line 53: Line 58:
 
* Uses dedicated build result directories allowing to build multiple tool chain variants in the same source tree
 
* Uses dedicated build result directories allowing to build multiple tool chain variants in the same source tree
  
To understand all aspects of the make language is not easy. The good news is that one can pick up a generic makefile and use it benefiting from advanced features without the need to ever touch it. A complete overview of available make commands can be found [https://www.gnu.org/software/make/manual/make.html#Quick-Reference here].
+
To understand all aspects of the make language is not easy. The good news is that one can pick up a generic makefile and benefit from advanced features without the need to ever touch it. A complete overview of available make commands can be found [https://www.gnu.org/software/make/manual/make.html#Quick-Reference here].
  
 
<syntaxhighlight lang="make">
 
<syntaxhighlight lang="make">
Line 144: Line 149:
 
Parallel build can be enabled using the <code>-j</code> command line switch. Be aware that only source files in the same directory can be processed in parallel. Therefore it is favorable to place all source files in  one directory.
 
Parallel build can be enabled using the <code>-j</code> command line switch. Be aware that only source files in the same directory can be processed in parallel. Therefore it is favorable to place all source files in  one directory.
  
Debugging makefile errors appears difficult to beginners. It is important to understand that a makefile is not evaluated like a standard programming questions. Instead all instructions are read in and evaluated creating a state machine which is then executed. To track down errors the following make output functions are useful: <code>error</code>, <code>warning</code> and <code>info</code>. The error function will exit the makefile while warning will continue.  You can use those functions to output the expanded value of make variables  or recipes.  
+
Debugging makefile errors appears difficult to beginners. It is important to understand that a makefile is not evaluated like a standard programming language code. Instead all instructions are read in and evaluated creating a state machine which is then executed. To track down errors the following make output functions are useful: <code>error</code>, <code>warning</code> and <code>info</code>. The error function will exit the makefile while warning will continue.  You can use those functions to output the expanded value of make variables  or recipes.
  
 
== Common Pitfalls ==
 
== Common Pitfalls ==
  
 +
The most common error in make is the message ''No rule to make target xxx.''. One must understand that make tries for every prerequisite to either find a file or to match a rule that resolves it. Typical sources of reasons for this error are typos in generated filenames or missing rules that resolve the prerequisite. Use the error or info macros to output generated prerequisite and target strings.
 +
 +
== Working example ==
 +
 +
You can download a ready to use generic makefile for C/C++ and Fortran as well as mixed language projects including an example [https://github.com/RRZE-HPC/Code-teaching/releases/download/v1.0-demos/make-demo-1.0.zip here].
 +
 +
For a working real example project including a build configuration file you can have a look at ''The Bandwidth Benchmark'' [https://github.com/RRZE-HPC/TheBandwidthBenchmark teaching code].
  
 
== Links and more Information ==
 
== Links and more Information ==
 
* [https://www.gnu.org/software/make/manual/make.html#Rule-Introduction GNU make info pages] Probably the most complete and exhaustive documentation on make. Includes many examples.
 
* [https://www.gnu.org/software/make/manual/make.html#Rule-Introduction GNU make info pages] Probably the most complete and exhaustive documentation on make. Includes many examples.
 
* An in-depth discussion of automatic make dependency generation can be found [http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ here]
 
* An in-depth discussion of automatic make dependency generation can be found [http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ here]

Latest revision as of 15:50, 3 September 2019

Make is a build automation tool, that can build arbitrary targets (in most cases program executables or libraries) from a list of prerequisites (which can be files or other targets). Make is controlled by reading files called Makefiles, which specify how to derive a target. While make is a standard UNIX tool, today in most cases people refer to the GNU make implementation, which has a lot of extensions.



The main benefits of using a build automation tool are:

  • Automation :-) of repetitive command execution
  • Consistent build results preventing errors even for complex build chains
  • Easier configuration for different tool chains
  • Formulate complex build settings with possibly various targets from the same source tree
  • Automatic matching of prerequisites
  • Speed up compilation by rebuilding only changed source file targets and parallel build support

Any serious software development effort sooner or later has to use an automated build tool. While make is the standard build tool on Linux systems there are various alternatives available, sometimes standalone and sometimes as a frontend to make.

Basic usage

A Makefile consists of rules:

target … : prerequisites …
        recipe

Each rule is made of the target, which is either a real build result as an executable or library, or just a label without a build result. After the colon there is a list of prerequisites. Make will search for every prerequisite either for a file or tries to match a rule to build it. Finally the recipe is a command or list of commands to generate the target. A recipe consists of any shell command but may also contain make variables or macros.

This is an example for a simple Makefile:

myProg: main.o kbd.o command.o display.o
        cc -o myProg main.o kbd.o command.o display.o

main.o : main.c defs.h
        cc -c main.c
kbd.o : kbd.c defs.h command.h
        cc -c kbd.c
command.o : command.c defs.h command.h
        cc -c command.c
display.o : display.c defs.h buffer.h
        cc -c display.c
clean :
        rm edit main.o kbd.o command.o display.o

By calling make without any arguments it will search for any file called makefile or Makefile and pick the first target in the file (indicated in this example by myProg:). The default target (the first one in the file) is matched first. Next make searches for the prerequisites, in this example a list of object files to link the executable. Next it either tries to find the object file or execute a rule to build it. In our case there is an explicit rule to build every object file. In case the object file was already build or is a primary source file make will check if the prerequisites are newer than the build result and automatically rebuild this file. This works in a recursive manner: A target depends on prerequisites which themselves depend on prerequisites and so on. Make will automatically figure out which parts need to be executed in which order to get it right.

Above example also contains a target which simply triggers a command execution: The clean target does not build anything but automates cleaning up target and intermediate build results.

Advanced usage

The simple example does miss out on many benefits of using make. The following example is a generic makefile for a C/C++ software project making use of some advanced features of make providing the following benefits:

  • Automatic dependency tracking
  • Multiple tool chain specific build configurations
  • Generic build rules
  • Is based on naming conventions
  • Uses dedicated build result directories allowing to build multiple tool chain variants in the same source tree

To understand all aspects of the make language is not easy. The good news is that one can pick up a generic makefile and benefit from advanced features without the need to ever touch it. A complete overview of available make commands can be found here.

TAG = ICC

#CONFIGURE BUILD SYSTEM
TARGET	   = myProg-$(TAG)
BUILD_DIR  = ./$(TAG)
SRC_DIR    = ./src
MAKE_DIR   = ./
Q         ?= @

#DO NOT EDIT BELOW
include $(MAKE_DIR)/include_$(TAG).mk

VPATH     = $(SRC_DIR)
OBJ       = $(patsubst $(SRC_DIR)/%.c, $(BUILD_DIR)/%.o,$(wildcard $(SRC_DIR)/*.c))
OBJ      += $(patsubst $(SRC_DIR)/%.cc, $(BUILD_DIR)/%.o,$(wildcard $(SRC_DIR)/*.cc))
OBJ      += $(patsubst $(SRC_DIR)/%.cpp, $(BUILD_DIR)/%.o,$(wildcard $(SRC_DIR)/*.cpp))

CPPFLAGS := $(CPPFLAGS) $(DEFINES) $(INCLUDES) 

${TARGET}: $(BUILD_DIR) $(OBJ)
	@echo "===>  LINKING  $(TARGET)"
	$(Q)${LINKER} ${LFLAGS} -o $(TARGET) $(OBJ) $(LIBS)

$(BUILD_DIR)/%.o:  %.c
	@echo "===>  COMPILE  $@"
	$(Q)$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
	$(Q)$(CC) $(CPPFLAGS) -MT $(@:.d=.o) -MM  $< > $(BUILD_DIR)/$*.d

$(BUILD_DIR)/%.o:  %.cc
	@echo "===>  COMPILE  $@"
	$(Q)$(CXX) -c $(CPPFLAGS) $(CXXFLAGS) $< -o $@
	$(Q)$(CXX) $(CPPFLAGS) -MT $(@:.d=.o) -MM  $< > $(BUILD_DIR)/$*.d

$(BUILD_DIR)/%.o:  %.cpp
	@echo "===>  COMPILE  $@"
	$(Q)$(CXX) -c $(CPPFLAGS) $(CXXFLAGS) $< -o $@
	$(Q)$(CXX) $(CPPFLAGS) -MT $(@:.d=.o) -MM  $< > $(BUILD_DIR)/$*.d

tags:
	@echo "===>  GENERATE  TAGS"
	$(Q)ctags -R


$(BUILD_DIR):
	@mkdir $(BUILD_DIR)

ifeq ($(findstring $(MAKECMDGOALS),clean),)
-include $(OBJ:.o=.d)
endif

.PHONY: clean distclean

clean:
	@echo "===>  CLEAN"
	@rm -rf $(BUILD_DIR)
	@rm -f tags

distclean: clean
	@echo "===> DIST CLEAN"
	@rm -f $(TARGET)
	@rm -f tags

An example tool chain configuration looks like the following (the file is named include_ICC.mk):

CC  = icc
CXX = icpc
LINKER = $(CXX)

CFLAGS   = -O3 -xAVX  -std=c99 
CXXFLAGS = -O3 -xAVX
LFLAGS   =  -vec-report0
DEFINES  = -D_GNU_SOURCE
INCLUDES = 
LIBS     =

The above example makes use of a bunch of make special variables. The @ sign placed before an command suppresses the output of the command line on stdout, which is the default. To be able to switch between printing all commands and suppressing printing the makefile assigns the @ sign to the Q variable. Now either Q can be set to be empty in the makefile, or the Q can be overwritten from the command line: make Q=

The makefile uses make text functions to generate the list of prerequisites and so called pattern rules for building them. The clean targets are marked using the phony keyword. A phony target is one that is not really the name of a file; rather it is just a name for a recipe to be executed when you make an explicit request. Many C compilers support to output make dependency files (often with a .d file ending). This is used in above example, the files are then included using the -include $(OBJ:.o=.d) statement. The makefile works for any number of source files without change.

Tips and Tricks

Parallel build can be enabled using the -j command line switch. Be aware that only source files in the same directory can be processed in parallel. Therefore it is favorable to place all source files in one directory.

Debugging makefile errors appears difficult to beginners. It is important to understand that a makefile is not evaluated like a standard programming language code. Instead all instructions are read in and evaluated creating a state machine which is then executed. To track down errors the following make output functions are useful: error, warning and info. The error function will exit the makefile while warning will continue. You can use those functions to output the expanded value of make variables or recipes.

Common Pitfalls

The most common error in make is the message No rule to make target xxx.. One must understand that make tries for every prerequisite to either find a file or to match a rule that resolves it. Typical sources of reasons for this error are typos in generated filenames or missing rules that resolve the prerequisite. Use the error or info macros to output generated prerequisite and target strings.

Working example

You can download a ready to use generic makefile for C/C++ and Fortran as well as mixed language projects including an example here.

For a working real example project including a build configuration file you can have a look at The Bandwidth Benchmark teaching code.

Links and more Information

  • GNU make info pages Probably the most complete and exhaustive documentation on make. Includes many examples.
  • An in-depth discussion of automatic make dependency generation can be found here