GrabDuck

Building packages with configure | nairobi-embedded

:

Overview

Packages distributed at the source-level and maintained with the GNU Autotools contain a shell script named configure. It’s purpose is to configure the end user’s environment and generate makefiles so that make can find the required external dependencies and successfully build the package.

The most general build steps for a package are:

$ ./configure
$ make
$ [sudo] make install

The configure script accepts several options to control a package’s configuration, build and installation process. Some of these options are generically available while others remain specific to a given package. To view all the options available in a package, run:

$ configure --help

The more commonly used and generic options are covered in the next section. For more details on available options by the configure script, check out the official GNU Manual.

Invoking configure

Commonly used options

  • -h, --help
    Display help and exit.

  • -V, --version
    Display version information and exit.

  • -n, --no-create
    Dry-run. Shows you what the configure script would do but no output is generated.

  • --prefix=PREFIX
    Installs files in appropriate subdirectories of PREFIX.

The most commonly used option is the --prefix. The default value for PREFIX is /usr/local.

Building a package via configure may sometimes involve external dependencies (headers and libraries) that are not found in the standard (or toolchain) search path. In addition, the need to use a different compiler driver other than the system default often arises e.g. during cross-compilation.

Most configure scripts will include a subset or more of the following pertinent environment variables that can be used to override the choices made by configure or to help it find libraries and programs with non-standard locations or names.

$ ./configure --help
...
Some influential environment variables:
CC          C compiler command
CFLAGS      C compiler flags
LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
            nonstandard directory <lib dir>
LIBS        libraries to pass to the linker, e.g. -l<library>
CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
            you have headers in a nonstandard directory <include dir>
CXX         C++ compiler command
CXXFLAGS    C++ compiler flags
CCAS        assembler compiler command (defaults to CC)
CCASFLAGS   assembler compiler flags (defaults to CFLAGS)
CPP         C preprocessor
CXXCPP      C++ preprocessor
...

These variables are typically defined in the commandline along with configure's invocation. So, for example, if building a package against a staging directory which is in a non-standard location while using a cross-compiler, configure's commandline may resemble:

$ CC=arm-linux-gnueabihf-gcc ./configure --prefix=/usr CPPFLAGS="-I${STAGING}/usr/include" LDFLAGS="-L${STAGING}/lib -L${STAGING}/usr/lib"

Standard make Targets

The following is a list of commonly used and standard make targets in makefiles generated by configure scripts.

  • all
    Builds the binaries, related data files and executable shell scripts. As with "regular” Makefiles, this is the default target i.e. executed when a plain make command is invoked.

  • clean
    Removes all binaries and intermediate object files produced during a previous invocation of make.

  • distclean
    Removes program binaries and any other files generated by a previous invocation of make. It also removes any files that configure generated. Essentially, it returns the package directory to the state it was after its extraction.

  • install
    Installs the program binaries, related data files and executable shell scripts.

  • uninstall
    Uninstalls the program binaries, related data files and executable shell scripts from the binary, library, and data directories.

  • install-strip
    Like install, but strip the executable files while installing them.

  • check
    Executes any tests associated with the application.

  • dist
    Creates a redistributable tar archive (tarball) of the package.

  • distcheck
    Verifies that the distribution produced by make dist is complete and correct.

Refer to this link for more details on standard make targets in packages conforming to GNU standards.

Out-of-tree Builds

It is often desirable to perform builds from a location separate from the source tree. The files generated by configure will be located in this separate location with a few files probably getting copied from the source directory. Should anything go wrong during package build, a clean restart is is guaranteed by simply deleting all contents of this seperate location. The isolated source tree remains clean and untouched.

Here is an example with the GNU dbm library package:

$ KAZIDIR=${PWD}
$ wget -c ftp://ftp.gnu.org/gnu/gdbm/gdbm-1.11.tar.gz
$ tar xzf gdbm-1.11.tar.gz
$ mkdir {temp,install}
$ SOURCE=${PWD}/gdbm-1.11 ; TEMP=${PWD}/temp ; INSTALL=${PWD}/install
$ cd $TEMP
$ ${KAZIDIR}/gdbm-1.11/configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
...
$ ls 
autoconf.h  config.log     doc     libtool   po   stamp-h1
compat      config.status  export  Makefile  src  tests

$ make

Here, all build actions are now performed from within ${KAZIDIR}/temp, leaving the ${KAZIDIR}/gdbm-1.11 source tree prestine.

Package Installation Options

Use of --prefix

Running make install will result in the installation of the package under the location specified by the value against --prefix. The default value of --prefix is /usr/local.

This default behaviour may be undesirable for a number of reasons including:

  • possibility overwriting pre-existing installations.

  • the fact that /usr/local is, conventionally, a system directory and hence the need for sudo.

  • avoiding unecessary/growing pollution of the directory.

Changing this default behaviour is simply a matter of specifying a different value against the --prefix option in the commandline e.g

$ ./configure --prefix=/home/${USER}/tmp

Note that the value specified against --prefix will be created (upon make install) if it didn't exist (and if the execution context has the required priviledges).

This mechanism allows for the installation of the package in an arbitrary, non-standard location. So, continuing with the example in Section link

$ make distclean # clean previous build and config
$ ls
compat  doc  export  po  src  tests
$ for i in `ls` ; do ls $i ; done  # confirming all generated files deleted
$ 
$ ${SOURCE}/configure --prefix=${INSTALL}/usr
$ make [-jN]    
$ make install

should result in an installation similar to:

Prefix Only Install

Fine-tuning of the installation directories

By default, everything gets installed under the location specified via --prefix. However, there exist several options to fine-tune the placement of the various installation directories:

  • --prefix=PREFIX
    Install architecture-independent files in PREFIX (default value /usr/local).

  • --exec-prefix=EPREFIX
    Install architecture-dependent files in EPREFIX (default value PREFIX).

And for more fine-grained control run:

$ ./configure --help
...
Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  ...

where, the values in square brackets, e.g. [EPREFIX/bin], stand for the default installation.

Staging and DESTDIR

Staging refers to the process placing the installation of a package in a temporary location rather than installing it (for use) in the build system. Staging is especially useful when preparing the installation of packages for transfer into, say, an embedded runtime host.

The DESTDIR variable is available specifically for the purpose of staging. All makefiles generated by configure support DESTDIR in their install* and uninstall* targets. The value specified against DESTDIR gets prepended to the value specified against prefix.

So, re-building the GDBM library package of Section link:

$ cd $TEMP
$ make distclean
$ ${SOURCE}/configure --prefix=/usr
$ make [-jN]
$ make DESTDIR=$INSTALL install

should result in an installation that resembles:

Prefix Dest Install

The DESTDIR value (/home/siro/kazi/install) has been prepended to --prefix value (/usr). In other words, the installed files maintain their relative directory structure (i.e. with respect to --prefix) and any embedded file names will not be modified.

The location ${INSTALL} (i.e. /home/siro/kazi/install) points to the staging directory.

In summary:

  • --prefix=
    specifies a path with respect to the root of the system in which the installation will run.

  • DESTDIR=
    specifies the location (absolute path) of the staging directory.

--prefix vs DESTDIR

Note that the layouts of the installations in Section link and link are (apparently) identical despite the fact that different approaches were used in the installation process i.e.:

$ ${SOURCE}/configure --prefix=${INSTALL}/usr
$ make
$ make install

versus,

$ ${SOURCE}/configure --prefix=/usr
$ make
$ make DESTDIR=${INSTALL} install

For several packages, their installations may continue to work blissfully regardless of the installation approach. For such cases, use of DESTDIR may be preferable. By using DESTDIR, a user can repeatedly perform installations in different locations without the need for reconfiguring (i.e specifying a different --prefix) and recompiling each time.

However, for many other packages, the intended use of --prefix and DESTDIR matters and will affect the scripts, symlinks, etc in the resultant installation. This translates to runtime failure if these installation options were not used correctly.

Case Study

To help illustrate the intended use of these installation options, the following case study is presented:

Download the Netfiler iptables package
$ wget -c http://www.netfilter.org/projects/iptables/files/iptables-1.4.18.tar.bz2
$ tar -xjf iptables-1.4.18.tar.bz2 
$ mkdir {bad_staging,good_staging,temp}
$ ls
bad_staging  good_staging  iptables-1.4.18  iptables-1.4.18.tar.bz2  temp

$ KAZI_DIR=$PWD
$ TEMP=${PWD}/temp
$ BAD_STAGING=${PWD}/bad_staging
$ GOOD_STAGING=${PWD}/good_staging
$ SOURCE=${PWD}/iptables-1.4.18
Peform "good" staging
$ cd $TEMP
$ ${SOURCE}/configure --prefix=/usr
$ make [-jN]
$ make DESTDIR=$GOOD_STAGING install

Now, let's view the installation:

Good Staging

Notice the broken symbolic link ${GOOD_STAGING}/usr/bin/iptables-xml points to a non-existent /usr/sbin/xtables-multi. However, when the installation is eventually moved to the runtime host, the symlink will resolve correctly and work as exprected.

Perform "bad" staging

On the other hand, performing:

$ cd $TEMP
$ rm -r *
$ ${SOURCE}/configure --prefix=${BAD_STAGING}/usr
$ make [-jN]
$ make install

results in a "bad" installation in the staging directory.

Bad Staging

i.e. while the ${BAD_STAGING}/usr/bin/iptables-xml symlink resolves properly in the build machine, it will break upon the installation's transfer to the runtime host.

Summary

  • Generally, when staging, the value of --prefix should be a standard system directory e.g. /usr or /usr/local. Otherwise, problems due to non-existent, non-standard locations are very likely to arise upon transfer to the runtime host. In other words, if --prefix is not a standard location, then this almost always translates to "bad" staging - regardless of whether DESTDIR is used or not.

    Typically, staging is meant for preparing an installation for transfer to a different runtime host other than the build machine. Nevertheless, there are situations where you may want/need to stage for the build machine. For example, staging in a temporary location with no root priviledges. Then transfer to the standard system directory specified against --prefix when permissions are granted.

  • When

    • performing a direct local install (i.e. for use in the build machine rather than for staging)

    and

    • in a non-standard location

    then the "bad" staging case study above becomes a valid installation and vice versa. In this scenario, the use of DESTDIR, however tempting, should be avoided at all costs. See Appendix link for case studies on how the use of DESTDIR can affect runtime in this context.

  • If

    • the value specified against --prefix is a standard location

    and

    • DESTDIR is not specified at all

    then the installation qualifies as both a valid local (direct) install and a "good" staging. The installation can be used on the build machine or transferred to a different runtime host (of the same architecture and in a corresponding location) without any problems.

Cross-Compiling

Refer to Cross-Compiling via GNU onfigure for details on using configure in a cross-build set-up.

Resources

Appendix: More Case Studies on --prefix vs DESTDIR

These case studies aim to demonstrate further the use of --prefix and DESTDIR.

Crosstool-NG

Download Crosstool-NG sources

$ wget -c http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.17.0.tar.bz2
$ tar xjf crosstool-ng-1.17.0.tar.bz2
$ mkdir {temp1,temp2}
$ ls
crosstool-ng-1.17.0  crosstool-ng-1.17.0.tar.bz2  temp1  temp2
$ KAZI_DIR=$PWD
$ CROSSTOOL=${PWD}/crosstool-ng-1.17.0
$ TEMP1=${PWD}/temp1
$ TEMP2=${PWD}/temp2

Generate toolchain build utilities

The configure, make and make install sequence against the downloaded crosstool-NG sources only generates the ct-ng utility and miscalleneous toolchain build files/scripts. It does not result in the actual toolchain build. In other words, the sequence only generates a toolchain build tree from which all actual toolchain build actions will be done.

Staging the toolchain build tree

Staging the toolchain build tree is used to package the generated ct-ng utility and other toolchain build scripts for distribution.

$ cd $CROSSTOOL
$ ./configure 
$ make
$ make DESTDIR=$TEMP1 install

The resultant "packaging" will resemble:

crosstool-ng TEMP1
Local install of the toolchain build tree

In this case, the generated toolchain build utilities are installed in a non-standard location and for use from current the build machine.

$ cd $CROSSTOOL
$ make distclean
$ ./configure --prefix=${TEMP2}/usr/local
$ make
$ make install

The resultant installation will resemble:

crosstool-ng TEMP2
Configure the toolchain

Note that while the layout of both installations in the previous section are identical, the individual installations are meant for use in different contexts i.e. can't be used interchangably. For instance running:

$ cd $TEMP2
$ usr/local/bin/ct-ng menuconfig

results in the following ncurses based configuration panel:

crosstool-ng MENU

On other hand, an attempt to perform toolchain configuration from the staged directory fails with the following error:

$ cd $TEMP1
$ usr/local/bin/ct-ng menuconfig
usr/local/bin/ct-ng:26: /usr/local/lib/ct-ng.1.17.0/paths.mk: No such file or dir...
usr/local/bin/ct-ng:113: /usr/local/lib/ct-ng.1.17.0/config/config.mk: No such ...
usr/local/bin/ct-ng:114: /usr/local/lib/ct-ng.1.17.0/kconfig/kconfig.mk: No such ...
usr/local/bin/ct-ng:115: /usr/local/lib/ct-ng.1.17.0/steps.mk: No such file or dir...
usr/local/bin/ct-ng:116: /usr/local/lib/ct-ng.1.17.0/samples/samples.mk: No such ...
usr/local/bin/ct-ng:117: /usr/local/lib/ct-ng.1.17.0/scripts/scripts.mk: No such ...
make: *** No rule to make target `/usr/local/lib/ct-ng.1.17.0/scripts/scripts.mk'. Stop.

Comparing the ct-ng utilities of both installations:

$ cd $KAZI_DIR

$ cat temp2/usr/local/bin/ct-ng
...
# Paths and values set by ./configure
# Don't bother to change it other than with a new ./configure!
export CT_LIB_DIR:=/home/siro/ct-demo/temp2/usr/local/lib/ct-ng.1.17.0
...
# Paths found by ./configure
include $(CT_LIB_DIR)/paths.mk
...
include $(CT_LIB_DIR)/config/config.mk
include $(CT_LIB_DIR)/kconfig/kconfig.mk
include $(CT_LIB_DIR)/steps.mk
include $(CT_LIB_DIR)/samples/samples.mk
include $(CT_LIB_DIR)/scripts/scripts.mk
...

$ cat temp1/usr/local/bin/ct-ng
...
# Paths and values set by ./configure
# Don't bother to change it other than with a new ./configure!
export CT_LIB_DIR:=/usr/local/lib/ct-ng.1.17.0
...
# Paths found by ./configure
include $(CT_LIB_DIR)/paths.mk
...
include $(CT_LIB_DIR)/config/config.mk
include $(CT_LIB_DIR)/kconfig/kconfig.mk
include $(CT_LIB_DIR)/steps.mk
include $(CT_LIB_DIR)/samples/samples.mk
include $(CT_LIB_DIR)/scripts/scripts.mk
...

In other words, the staged crosstool-NG toolchain build tree needs to first get transferred to /usr/local in order to work as expected.

Building GRUB

Once again, the objective is to perform a local installation (for use from the build machine) and in a non-standard location (a non-system directory).

NOTE: Admittedly, the GRUB package is not the best candidate for an illustration. If used incorrectly, it can overwrite the bootloader of your system. So, preferably, these instructions should be performed from within a VM environment such as QEMU1:

Download GRUB and prepare for build and installation:

$ wget -c ftp://ftp.gnu.org/gnu/grub/grub-2.00.tar.xz
$ tar xJf grub-2.00.tar.xz 
$ mkdir {grub-valid-install,grub-invalid-install,grub-temp}
$ KAZI_DIR=$PWD
$ GRUB_SOURCE=${KAZI_DIR}/grub-2.00
$ GRUB_TEMP=${KAZI_DIR}/grub-temp
$ GRUB_VALID_INSTALL=${KAZI_DIR}/grub-valid-install
$ GRUB_INVALID_INSTALL=${KAZI_DIR}/grub-invalid-install
$ sudo apt-get install bison flex autoconf automake autotools-dev libtool gettext libdevmapper-dev

Perform a valid local install

$ cd $GRUB_TEMP
$ ${GRUB_SOURCE}/configure --prefix=${GRUB_VALID_INSTALL}/usr
...
*******************************************************
GRUB2 will be compiled with following components:
Platform: i386-pc
With devmapper support: Yes
With memory debugging: No
With disk cache statistics: No
efiemu runtime: Yes
grub-mkfont: No (need freetype2 library)
grub-mount: No (need FUSE library)
starfield theme: No (No grub-mkfont)
With libzfs support: No (need zfs library)
*******************************************************

$ make -j8
$ make install

Perform an invalid local install

$ cd $GRUB_TEMP
$ rm -r *
$ ${GRUB_SOURCE}/configure --prefix=/usr
$ make -j8
$ make DESTDIR=$GRUB_INVALID_INSTALL install

Create a dummy disk image for use with GRUB

$ cd $KAZI_DIR
$ dd if=/dev/zero of=dummy.img count=102400 bs=1024
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0.307808 s, 341 MB/s

$ sudo losetup /dev/loop0 dummy.img
$ sudo fdisk /dev/loop0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd6b3978a.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-204799, default 2048): <ENTER>
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-204799, default 204799): +80M

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 2): <ENTER>
Using default value 2
First sector (165888-204799, default 165888): <ENTER> 
Using default value 165888
Last sector, +sectors or +size{K,M,G} (165888-204799, default 204799): <ENTER>
Using default value 204799

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

$ sudo losetup -d /dev/loop0

Install kpartx if missing:

$ sudo apt-get install kpartx

then proceed to format the disk image partitions with a filesystem and then mount:

$ sudo kpartx -a dummy.img
$ sudo mkfs.ext3 /dev/mapper/loop0p1
$ sudo mkfs.ext3 /dev/mapper/loop0p2
$ sudo mkdir dummy
$ sudo mount /dev/mapper/loop0p1 dummy

GRUB installation in the disk image

The following attempt to install GRUB in the disk image fails with:

$ sudo ${GRUB_INVALID_INSTALL}/usr/sbin/grub-install --no-floppy --root-directory=${PWD}/dummy /dev/loop0

/home/siro/tmp/grub-invalid-install/usr/sbin/grub-install: 337: .: Can't open /usr/lib/grub/i386-pc/modinfo.sh

Running a diagnostic via strace:

$ sudo -s
# GRUB_INVALID_INSTALL=${PWD}/grub-invalid-install 
# strace ${GRUB_INVALID_INSTALL}/usr/sbin/grub-install --no-floppy --root-directory=${PWD}/dummy /dev/loop0  2>&1 | grep '/usr/lib/grub'

stat("/usr/lib/grub/i386-pc", {st_mode=S_IFDIR|0755, st_size=28672, ...}) = 0
open("/usr/lib/grub/i386-pc/modinfo.sh", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "Can't open /usr/lib/grub/i386-pc"..., 43Can't open /usr/lib/grub/i386-pc/modinfo.sh) = 43

# exit

reveals that grub-install was configured to search for its runtime dependencies from /usr rather than ${GRUB_INSTALL}/usr. Inspecting the corresponding grub-install script confirms this:

$ cat ${GRUB_INVALID_INSTALL}/usr/sbin/grub-install
...
prefix="/usr"
exec_prefix="${prefix}"
datarootdir="${prefix}/share"
...

So now, using grub-install from the "valid" installation:

$ sudo ${GRUB_VALID_INSTALL}/usr/sbin/grub-install --no-floppy --root-directory=${PWD}/dummy /dev/loop0
device-mapper: table ioctl failed: No such device or address
device-mapper: table ioctl failed: No such device or address
...
Installation finished. No error reported.

completes successfully (inspite of the slew of warnings being emitted). Now, examining:

$ sudo -s
# GRUB_VALID_INSTALL=${PWD}/grub-valid-install
# strace ${GRUB_VALID_INSTALL}/usr/sbin/grub-install --no-floppy --root-directory=${PWD}/dummy /dev/loop0  2>&1 | grep '/usr/lib/grub'
stat("/home/siro/tmp/grub-valid-install/usr/lib/grub/i386-pc", {st_mode=S_IFDIR|0775, st_size=20480, ...}) = 0
open("/home/siro/tmp/grub-valid-install/usr/lib/grub/i386-pc/modinfo.sh", O_RDONLY) = 3
openat(AT_FDCWD, "/home/siro/tmp/grub-valid-install/usr/lib/grub/i386-pc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/home/siro/tmp/grub-valid-install/usr/lib/grub/i386-pc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
...

# exit

reveals that this grub-install was indeed configured properly and searches for its dependencies from under ${GRUB_INSTALL}/usr. Checking the corresponding grub-install script:

$ cat ${GRUB_VALID_INSTALL}/usr/sbin/grub-install
...
prefix="/home/siro/tmp/grub-valid-install/usr"
exec_prefix="${prefix}"
datarootdir="${prefix}/share"
...

Clean up:

$ sudo umount dummy
$ sudo kpartx -d dummy.img
loop deleted : /dev/loop0

If need be, the disk image, dummy.img, may now be tested. For example, running the disk image against QEMU2.

$ qemu-system-i386 dummy.img

or

$ qemu-system-x86_64 dummy.img

should result in a GRUB prompt similar to:

Grub Prompt

See Making a QEMU Disk Image Bootable with GRUB for more info on installing GRUB in disk image files.

1. These instructions were actually performed from within a QEMU VM instance. Otherwise, if working from a physical machine environment, proceed with extreme caution [go back]

2. This part was performed from the VM host (build machine) environment [go back]