Chapter 2. Compiling code for Bering uClibc using Buildtool

Arne Bernin

Martin Hejl

Table of Contents

Introduction
Installation
Buildtool usage
Creating buildtool sources/packages
Buildtool Tips and FAQ

Introduction

What is buildtool?

Buildtool is a set of perl-scripts/packages to build lrp/leaf-packages from source. Everything in the build process is automated, from downloading the sources and applying patches to configuring/building the source and creating the lrp-package.

Why bother?

Granted, the initial setup of a new package takes some time and tweaking, much more so than simply doing the compile manually would. So, for a one time shot, a package that will never be updated or modified, integrating it into buildtool is probably not worth the hassle.

But if you have a package that's security related (and therefore likely to receive security updates regularly) or one that is still under heavy development and new revisions are released all the time, the time invested in buildtool will pay off very quickly - since usually, all that's needed to integrate a new version of the software are a couple of changes in the config-files.

Besides, creating lrp-packages by hand can be tedious and error prone. And if the developer has to spend less time assembling the package, he can use that time for testing the package (or creating new packages).

Installation

CVS checkout

You can check out buildtool from the sourceforge CVS Server. If you have a sourceforge user account use:

export CVS_RSH=ssh
cvs -z3 -d:ext:developername@leaf.cvs.sourceforge.net:/cvsroot/leaf \
  co src/bering-uclibc/buildtool

If you don't have an account use anonymous access to the repository:

cvs -d :pserver:anonymous@leaf.cvs.sourceforge.net:/cvsroot/leaf login

Just press return if you see the password prompt. After login checkout with:

cvs -z3 -d :pserver:anonymous@leaf.cvs.sourceforge.net:/cvsroot/leaf \
  co src/bering-uclibc/buildtool

Checking out an older version to build

If you want to use buildtool for compiling sources for an older Bering-uClibc release, you need to check out buildtool and the relevant sources using a release-name or a date, and adjust the buildtool configuration to no longer use cvs to fetch the sources

For example, if you want to check out the buildtool setup and sources that were used to build Bering uclibc 2.4.1, use these commands:

mkdir /PATH/TO/WORKAREA
cd /PATH/TO/WORKAREA
cvs -z3 -d :pserver:anonymous@leaf.cvs.sourceforge.net:/cvsroot/leaf \
 co -r Release-2_4_1 src/bering-uclibc

or (if you have a sourceforge developer account)

export CVS_RSH=ssh
mkdir /PATH/TO/WORKAREA
cd /PATH/TO/WORKAREA
cvs -z3 -d:ext:developername@leaf.cvs.sourceforge.net:/cvsroot/leaf \
 co -r Release-2_4_1 src/bering-uclibc

After the checkout is complete, adjust the server definitions in /PATH/TO/WORKAREA/src/bering-uclibc/buildtool/conf/sources.cfg to no longer use CVS for downloading the sources. For example:

<Server cvs-sourceforge>
         Type = file
         Serverpath = /PATH/TO/WORKAREA/src/bering-uclibc/apps
</Server>

<Server cvs-contrib-sourceforge>
         Type = file
        Serverpath = /PATH/TO/WORKAREA/src/bering-uclibc/contrib
</Server>

Perl

Buildtool requires Perl 5.6 (or newer). Additionally, you need the Perl package Config::General. This can either be installed the "traditional" way (download it from CPAN, untar, perl Makefile.PL, make, make test and (as root) make install), or by running (as root) perl -MCPAN -e 'install Config::General'

Initial configuration

Edit the file conf/buildtool.conf (using your favourite text editor). It should look similar to this:

# version number
Version = 0.6

###################################################
# files&directories
# Adjust to your needs
# note that the dir or file entries can be relative 
# (to the dir buildtool.pl resides in) or absolute
#
###################################################
#
# where to log 
Logfile = log/buildtoollog

# name of the conf file with the source/package definitions
GlobalConffile = conf/sources.cfg

# Name of the source Directory
Source_Dir = source

# where to log what we have installed/compiled
InstalledFile = conf/installed

# debugging
DebugToConsole = 0
DebugToLogfile = 1

# good if you want to develop files
# in here and not to overwrite buildtool any files
# this means, if a file exists and its size is > 0,
# it will not be overwritten unless you delete it!
OverwriteFiles = 0

# how the files we use for each package are named:

Buildtool_Makefile = buildtool.mk
Buildtool_Config = buildtool.cfg

# some directories, these will be deleted if you make a 
# 'buildtool.pl distclean'
Buildenv_Dir=log
Buildenv_Dir=staging
Buildenv_Dir=source
Buildenv_Dir=build
Buildenv_Dir=package


# Stuff needed for buildpacket ---
gzip_options=-9

# Name of the packager -  please put your real name in here 
# (so people will know who created the package)
Packager=anonymous

Normally, the defaults in this config file are fine. The only setting that should be changed is the setting for the Packager. Enter your name here (this entry will be shown in the .help-files when creating packages - and I guess it is a good idea to let the user know who assembled the package).

Please see section "buildenv not building with gcc 4.0" at the end of this chapter if you're using a distribution that comes with gcc 4.x as its default compiler.

Toolchain download/build

Note

Unless otherwise noted, you should never run buildtool as root. Errors in the makefile might otherwise screw up your host system (I've done it, trying to build ncurses...).

You will probably want to open a new xterm (or whatever terminal emulator you prefer), go to the buildtool root directory (which is buildtool/ starting from where you made the cvs checkout) and enter tail -f log/buildtoollog in it. Go back to the other xterm, change to the buildtool root dir and enter (hopefully on a fast internet connection - there'll be several large downloads): ./buildtool.pl build buildenv

Finally, (and this must be done as root) create a link /lib/ld-uClibc.so.0 pointing to $(Root_Dir)/staging/lib/ld-uClibc.so.0 (where $(Root_Dir) is what you defined above in buildtool.conf). If you forget that step, you will sooner or later get a configure error (openssh is one of those candidates), claiming that cc cannot create binaries. This happens because configure compiles a test program and tries to execute it. The test program expects the uClibc library to be in /lib, if it is not found the program can not be executed (and this position is hardcoded while compiling the buildenv and cannot be changed with ldconfig).

Buildtool usage

Options

usage: ./buildtool.pl command [pkgname]|[srcname] [...]

commands:
describe [pkgname]|[srcname]      shows description lines of package
[-f] source [pkgname]|[srcname]   downloads, unpacks and patches
                                  the wanted package/source
[-f] build [pkgname]|[srcname]    the same as source, but builds
                                  and installs sources/packages also
[-f] clean [pkgname]|[srcname]    make a 'make clean' in srcdir
[-f] dlclean [pkgname]|[srcname]  remove everything from dldir
[-f] srcclean [pkgname]|[srcname] make a 'make srcclean' in srcdir
distclean                         remove everything
maketar                           make a tar for distribution

The -f switch allows you to force building sourcing and cleaning
even if it seems the packages/sources are already installed or cleaned.

Getting information about the sources that can be built

Buildtool can only build sources/packages that it has a configuration for (see the next chapter on how to create such a configuration). To get a listing about the sources that are configured for buildtool type

 ./buildtool.pl describe

This will give you a list (and a short description) of all sources/packages that can be built. An example output looks like this:

 ./buildtool.pl describe
 
 The following packages and sources are available:
 
 Sources:
 ---------------------------------------------------------
 openssh          openssh
 tcp_wrappers     Wietse Venema's TCP wrappers library
 helloworld       classical example
 openssl          Secure Sockets Layer and cryptography 
                  libraries and tools
 zlib             zLib shared libraries
 linux            kernel source only package
 
 Packages:
 ---------------------------------------------------------
 buildenv         this holds everything to build cross build env
 kernel           kernel pseudo package

The difference between sources and packages

You might be a bit confused why in the above output there are Sources and Packages. The main reason for having sources and packages separately is that there are some sources out there that produce programs used in several packages. Or in other words, it sometimes makes sense to just build the sources (like util-linux for example) only once and create several packages out of the resulting binaries.

Downloading the sources for a package

Running

./buildtool.pl source somepackageName

will cause the sources of somepackageName to be downloaded (and before that, all its dependencies, if there are any that haven't been downloaded yet). After that, the makefile buildtool.mk for this source is executed for target source. This has the effect that the sources will be extracted, and patches (if there are any) are applied.

If all this is completed without error, the sources of somepackageName are ready to be compiled.

Compiling the sources for a package

Running

./buildtool.pl build somepackageName

will cause buildtool to actually compile the source and install it to the build directory (and all its dependencies). After this is completed successfully, all binaries required for the lrp package (linked against uClibc) reside in the build directory.

Creating buildtool sources/packages

Overview

For each package, 3 files control how it's built:

conf/sources.cfg

Defines the servers that the config files for the packages are downloaded from, and the packages/sources themselves as well as the dependencies of a given package. All other files that are part of a package and should be downloaded from somewhere have to be explicitly named in the buildtool.cfg which is specific to the package. The file conf/sources.cfg is the same for all packages/sources, and you can find it in the conf subdir in your Buildtool root directory.

buildtool.cfg

(package specific - will be downloaded with the information from conf/sources.cfg). Defines where to get the sources/patches and also has the definitions for the package itself (used by buildpacket).

buildtool.mk

(package specific - will be downloaded with the information from buildtool.cfg). Makefile for actually building the sources.

Step by step guide to creating a simple config

This section should give you a quick glance at how the configuration for a specific source is created (but it won't show the hours of fighting with some source because it refuses to build - you will experience that soon enough when you create your own package. I don't want to discourage you, but cross-compiling often is more difficult than building on the target system). For a more complete overview, please see the following sections of this chapter.

For this exercise, we will create a package called hdsupp.lrp, which contains utilities for preparing a harddisk or compact flash disk for running under Bering uClibc. This package will contain fdisk, syslinux and mkfs.msdos.

Because there is no combined tarball for syslinux, fdisk and mkfs.msdos, we will have to create 3 different source definitions for buildtool. We will start with syslinux, since that is the easiest one.

By the time you read this, hdsupp will already be part of the buildtool checkout. This means that the settings for syslinux, util-linux and dosfstools in conf/sources.cfg will already be there. You can decide to either just skip that section, or (for the sake of learning something) delete that part of conf/sources.cfg and enter the information again manually. The following sections will assume that buildtool knows nothing about syslinux, util-linux and dosfstools.

Creating the source definition for syslinux, util-linux, dosfstools

First of all we need to edit conf/sources.cfg to contain a source definition for syslinux, util-linux, dosfstools.

<Source syslinux>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = syslinux
  Description = Syslinux boot-loader plus an MBR
  <Requires>
    Name  = buildenv
  </Requires>
</Source>

<Source util-linux>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = util-linux
  Description = provides fdisk and mkfs.minix
  <Requires>
    Name  = buildenv
  </Requires>
</Source>

<Source dosfstools>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = dosfstools
  <Requires>
    Name  = buildenv
  </Requires>  Description = provides mkfs.dos
</Source>

For now, we'll just pretend that the buildtool.cfg file will be in the leaf CVS. In this section, we will create everything needed for syslinux - the other sources will be taken care of later.

Next, we need to create a directory called syslinux in source. Doing

mkdir source/syslinux

will do the job. This directory will contain buildtool.cfg, buildtool.mk and the syslinux sources.

Create a new file source/syslinux/buildtool.cfg and enter the following information.

<Server cvs-sourceforge>
  Type = viewcvs
  Name = leaf.cvs.sourceforge.net
  Serverpath = /leaf/src/bering-uclibc/apps
</Server>

<Server kernel.org>
  Type = http
  Name = www.kernel.org
  Serverpath = pub/linux/utils/boot/syslinux 
</Server>

<File buildtool.mk>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = syslinux
</File>

<File syslinux-2.01.tar.bz2>
  Server = kernel.org
  envname = SYSLINUX_SOURCE
</File>

The two server definitions tell buildtool which server it should download the files from. In this case it is the leaf-cvs repository for buildtool.mk and kernel.org for the syslinux sources. Next we define the file definitiones themselves - specifying HEAD for revision of buildtool.mk is convenient during development of a package (since the revision number can change often while one is trying to correct some bugs).

Note the envname parameter - with this definition in the cfg-file, buildtool will set an environment variable (named SYSLINUX_SOURCE in this case) containing the filename of the syslinux sources (syslinux-2.04.tar.bz2 in our example). This saves us from having to hardcode any version-specific information into the makefile. Specifying envname is not mandatory, but it keeps things simple for upgrading to a new version of a source.

Even though version 2.01 is not the most recent one, we will use that one for now since the following versions need mtools to be installed (mcopy and mattrib).

Creating the makefile for syslinux

Obviously, this guide will not be able to teach you how to write makefiles, or what to do when there are errors during build. We'll simply focus on creating a makefile that produces the binaries and conforms to what buildtool expects.

Create a new file source/syslinux/buildtool.mk and enter the following contents:

#############################################################
#
# syslinux 
#
#############################################################

include $(MASTERMAKEFILE)
SYSLINUX_DIR:=syslinux-2.01
SYSLINUX_TARGET_DIR:=$(BT_BUILD_DIR)/syslinux

source: 
  bzcat $(SYSLINUX_SOURCE) |  tar -xvf - 
  -mkdir $(SYSLINUX_TARGET_DIR)

build:
  export LANG=en_US  # Hack for Redhat 9
  export CC=$(TARGET_CC)
  $(MAKE) -C $(SYSLINUX_DIR) syslinux
  -$(BT_STRIP) $(BT_STRIP_BINOPTS) $(SYSLINUX_DIR)/syslinux
  $(MAKE) -C $(SYSLINUX_DIR) INSTALLROOT=$(SYSLINUX_TARGET_DIR) install
  cp $(SYSLINUX_DIR)/mbr.bin $(SYSLINUX_TARGET_DIR)/usr/bin/mbr.bin
  mkdir -p $(BT_STAGING_DIR)/usr/bin
  -cp $(SYSLINUX_TARGET_DIR)/usr/bin/syslinux $(BT_STAGING_DIR)/usr/sbin/
  -cp $(SYSLINUX_TARGET_DIR)/usr/bin/mbr.bin $(BT_STAGING_DIR)/usr/sbin/

clean: 
  rm -rf $(SYSLINUX_TARGET_DIR)
  $(MAKE) -C $(SYSLINUX_DIR) clean

As with all makefiles, make sure that the delimiter in front of the commands (like $(MAKE) -C $(SYSLINUX_DIR) clean above) is a tab and not spaces.

Creating the source definition for fdisk

On Debian, fdisk (and also mkfs.minix, which we will use for the hdsupp package as well) is part of the util-linux package. So, we use the sources that are luckily already provided in the bering uClibc CVS (no need to pretend they're in CVS this time).

Create a directory source/util-linux/ and a new file source/util-linux/buildtool.cfg and enter the following information.

<Server cvs-sourceforge>
  Type = viewcvs
  Name = leaf.cvs.sourceforge.net
  Serverpath = /leaf/src/bering-uclibc/apps
</Server>

<File buildtool.mk>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = util-linux
</File>

<File util-linux_2.11n.orig.tar.gz>
  Server = cvs-sourceforge
  envname = UTIL_LINUX_SOURCE
</File>

<File util-linux_2.11n-4.diff.gz>So
  Server = cvs-sourceforge
  envname = UTIL_LINUX_PATCH1
</File>

The makefile for util-linux

As with syslinux, we need to create a makefile. This time the source target will also include applying a patch

#############################################################
#
# util-linux
#
#############################################################

include $(MASTERMAKEFILE)
UTIL_LINUX_DIR:=util-linux-2.11n
UTIL_LINUX_TARGET_DIR:=$(BT_BUILD_DIR)/util-linux
export CC=$(TARGET_CC)
export CFLAGS=-I$(BT_STAGING_DIR)/include 
export LDFLAGS=-L$(BT_STAGING_DIR)/lib -L$(BT_STAGING_DIR)/usr/lib

source: 
  zcat $(UTIL_LINUX_SOURCE) |  tar -xvf - 
  zcat $(UTIL_LINUX_PATCH1) | patch -d $(UTIL_LINUX_DIR) -p1
  cd $(UTIL_LINUX_DIR) && ./configure
  perl -i -p -e 's,HAVE_SLANG=yes,HAVE_SLANG=no,' $(UTIL_LINUX_DIR)/MCONFIG
  perl -i -p -e 's,LIBSLANG=-lslang,LIBSLANG=,' $(UTIL_LINUX_DIR)/MCONFIG
  -mkdir $(UTIL_LINUX_TARGET_DIR)

build:
  $(MAKE) CC=$(TARGET_CC) -C $(UTIL_LINUX_DIR) misc-utils 
  $(MAKE) CC=$(TARGET_CC) -C $(UTIL_LINUX_DIR) disk-utils 
  $(MAKE) CC=$(TARGET_CC) -C $(UTIL_LINUX_DIR) fdisk fdisk
  -$(BT_STRIP) $(BT_STRIP_BINOPTS) $(UTIL_LINUX_DIR)/fdisk/fdisk
  -$(BT_STRIP) $(BT_STRIP_BINOPTS) $(UTIL_LINUX_DIR)/disk-utils/mkfs.minix
  -cp $(UTIL_LINUX_DIR)/fdisk/fdisk $(UTIL_LINUX_TARGET_DIR)/
  -cp $(UTIL_LINUX_DIR)/disk-utils/mkfs.minix $(UTIL_LINUX_TARGET_DIR)/
  mkdir -p $(BT_STAGING_DIR)/sbin
  -cp $(UTIL_LINUX_DIR)/fdisk/fdisk $(BT_STAGING_DIR)/sbin/
  -cp $(UTIL_LINUX_DIR)/disk-utils/mkfs.minix $(BT_STAGING_DIR)/sbin/
  
clean: 
  rm -rf $(UTIL_LINUX_TARGET_DIR)
  $(MAKE) -C $(UTIL_LINUX_DIR) clean

This one shows where things get a bit trickier - for some reason, the configure-script that comes with util-linux insisted on building cfdisk (which needs ncurses). So, after running configure, we remove the definitions that cause this from MCONFIG (this is what the two calls to Perl do).

Source definition for dosfstools

Dosfstools provides mkfs.msdos. Again we need to create a directory for the sources/configs first. So, create a directory source/dosfstools and a new file source/dosfstools/buildtool.cfg with the following content:

<Server cvs-sourceforge>
  Type = viewcvs
  Name = leaf.cvs.sourceforge.net
  Serverpath = /leaf/src/bering-uclibc/apps
</Server>

<Server debian>
  Type = http
  Name = ftp.debian.org
  Serverpath = debian/pool/main/d/dosfstools
</Server>

<File buildtool.mk>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = dosfstools
</File>

<File dosfstools_2.9.orig.tar.gz>
  Server = debian
  envname = DOSFSTOOLS_SOURCE
</File>

<File dosfstools_2.9-1.diff.gz>
  Server = debian
  envname = DOSFSTOOLS_PATCH1
</File>

Pretty much like the definitions before. Don't let yourself be fooled - despite the name ftp.debian.org, we can access this server using the http protocol.

Makefile for dosfstools

As one would expect the makefile for dosfstools looks just like the other ones.

#############################################################
#
# dosfstools
#
#############################################################

include $(MASTERMAKEFILE)
DOSFSTOOLS_DIR:=dosfstools-2.9
DOSFSTOOLS_TARGET_DIR:=$(BT_BUILD_DIR)/dosfstools
export CC=$(TARGET_CC)

source: 
  zcat $(DOSFSTOOLS_SOURCE) |  tar -xvf - 
  zcat $(DOSFSTOOLS_PATCH1) | patch -d $(DOSFSTOOLS_DIR) -p1
  perl -i -p -e 's,CC\s*=\s*gcc,#CC = gcc,' $(DOSFSTOOLS_DIR)/Makefile
  perl -i -p -e 's,PREFIX\s*=.*,PREFIX=$(DOSFSTOOLS_TARGET_DIR),' \
    $(DOSFSTOOLS_DIR)/Makefile
  -mkdir $(DOSFSTOOLS_TARGET_DIR)

build:
  export PREFIX=$(DOSFSTOOLS_TARGET_DIR)
  $(MAKE) CC=$(TARGET_CC) -C $(DOSFSTOOLS_DIR) 
  $(MAKE) CC=$(TARGET_CC) -C $(DOSFSTOOLS_DIR) install
  -$(BT_STRIP) $(BT_STRIP_BINOPTS) $(DOSFSTOOLS_TARGET_DIR)/sbin/mkdosfs
  mkdir -p $(BT_STAGING_DIR)/sbin
  -cp $(DOSFSTOOLS_TARGET_DIR)/sbin/mkfs.msdos $(BT_STAGING_DIR)/sbin/

clean:
  rm -rf $(DOSFSTOOLS_TARGET_DIR)
  $(MAKE) -C $(DOSFSTOOLS_DIR) clean

Finishing up hdsupp

Finally - almost there. Hdsupp will be a "wrapper package" - it doesn't really compile anything, it just makes sure that all sources it needs have been compiled, copies the binaries to the staging dir (this part could also be done in the individual .mk files) and creates the lrp package.

In the beginning of this guide, we haven't added a definition for hdsupp to conf/sources.cfg. We'll change that now. Hdsupp is special, since one of its functions is to make sure all of the sources have been compiled already - it has dependencies. In buildtool, we use the required tag to indicate dependencies. This tag lists all sources that must be built before hdsupp can be built. Notice that hdsupp is not specified as a source, but as a package (at this point, it makes no difference to buildtool, but it most likely will in the future - only packages will be able to produce lrp-files).

So, enter the following to conf/sources.cfg:

<Package hdsupp>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = hdsupp
  Description = Package providing tools for harddrives/CF disks
  <Requires> 
    Name = syslinux
    Name = util-linux         
    Name = dosfstools
  </Requires>  
</Package>
Hdsupp package definition

As before we need to create a directory for hdsupp containing buildtool.cfg and buildtool.mk. Create a new directory source/hdsupp, a file buildtool.cfg and enter the following information to buildtool.cfg:

<Server cvs-sourceforge>
  Type = viewcvs
  Name = leaf.cvs.sourceforge.net
  Serverpath = /leaf/src/bering-uclibc/apps
</Server>

<File buildtool.mk>
  Server = cvs-sourceforge
  Revision = HEAD
  Directory = hdsupp
</File>

As you can see, this one is extremely simple. Since hdsupp doesn't provide any sources, we only need to specify buildtool.mk.

This is not all we need to add here - there's more to get buildpacket to work. But this will be taken care of in the "Step by step" section of the buildpacket chapter. (TODO add link)

Hdsupp makefile

You may be wondering why we need a makefile at all, since hdsupp does not provide any sources. You are right - we don't need a makefile. But to keep buildtool happy (and to make sure that buildpacket can verify that hdsupp has actually been built), we will provide one that simply does nothing.

As before, create a new file called buildtool.mk in sources/hdsupp and enter the following contents:

#############################################################
#
# hdsupp
#
#############################################################

include $(MASTERMAKEFILE)

source: 
  # nothing to be done

build:
  # nothing to be done

clean:
  # nothing to be done

That's it. You can now enter ./buildtool.pl build hdsupp and (hopefully) buildtool will download and compile everything needed. If something goes wrong, have a look at the buildtool-logfile (usually the error-messages are pretty clear about what's wrong).

Buildtool reference

conf/sources.cfg

In this file you need to set two definitions - first, the server from which buildtool.cfg gets downloaded.

Sample:

<Server hejl>
  Type = http
  Name = lrp.hejl.de
  Serverpath = /devel
</Server>
<Package openssh>
  Server = hejl
  Directory = openssh
  Description = OpenSSH (ssh, sshd, ssh-key, sftp)
  <Requires> 
    Name = openssl
    Name = zlib             
    Name = tcp_wrappers
  </Requires>     
</Package>

The server section specifies that buildtool will use the url http://lrp.hejl.de/devel as a base to download the config for the package. This url consists of two parts: The Name of the server and the serverpath which will be appended to the servername. Don't use an url here, since the protocol section of the url will be automatically added by buildtool (depending on what is specified as "Type").

Possible entries in the Server section are:

Type

The type of the server (Currently, the supported download types are: http, ftp, viewcvs and file).

Name

The name of the server (only for http, ftp and viewcvs.) without any leading protocol (like http://). Note if you use viewvcs.cgi you have to add here the pathname of the cgi script (leaf.cvs.sourceforge.net for example)

Serverpath

The directory/pathname that comes after the servername (will be added there by buildtool). In case of a viewcvs server it will be added after the *checkout* that will be appended to the server's Name (*checkout* is needed to tell viewcvs to really download a file). You should not use http for files that are downloaded via viewcvs - even though this is possible, buildtool tries to fetch the correct version, even if there already is a newer version of the file in the repository (in that case, the download via http would fail). For type "File" this should contain a relative path to the file. This really only makes sense for files that are part of the CVS checkout.

Possible entries in the package/source section are:

Server

The name of the server to use (has to be defined as server in the server section, see above).

Directory

This is an additional directory name that will be appended to the serverpath. This makes it possible to define just one server for downloads from multiple directories.

Description

The string that is shown when entering ./buildtool.pl describe

Requires

The Requires section should contain a list of package names that must be built before building this package. The entries in here are put in line by line starting with the keyword Name.

The package section specifies which server to use (there has to be a corresponding server section) and which directory to download the config files from. So, in this example, buildtool.cfg for the package openssh will be downloaded from http://lrp.hejl.de/devel/openssh (this is defined by the url from the server section plus the directory defined by the package definition.

The requires section in this example defines that, before trying to build openssh, buildtool must build openssl, zlib and tcp_wrappers (and whatever packages they require).

buildtool.cfg

Contains two main sections. First of all the server and file specifications used for downloading the sources, and the package specification (which will be used by buildpacket). See the documentation on buildpacket for the package specification.

Example:

<Server hejl>
  Type = http
  Name = lrp.hejl.de
  Serverpath = /devel
</Server>
<Server debian>
  Type = http
  Name = ftp.debian.org
  Serverpath = /debian/openssh
</Server>
<File buildtool.mk>
  Server = hejl
  Directory = openssh
</File>
<File openssh_3.6.1p1.orig.tar.gz>
  Server = debian
  envname = OPENSSH_SOURCE                
</File>

<File openssh_3.6.1p2-2.diff.gz>
  Server = debian
  envname = OPENSSH_PATCH2                                
</File>

The server section follows the same rules as in conf/sources.cfg. At this point, all servers that are referenced by a file section, need to be defined in this file (even if they have already been defined in conf/sources.cfg). This may change in the future, so that servers need to be defined only once.

Each file section defines a file to download. These usually are source archives and patches. The url to download the file is serverURL/Directory/File-section-name. In our example, the three files that will be downloaded are: http://lrp.hejl.de/devel/openssh/buildtool.mk http://ftp.debian.org/debian/openssh/openssh_3.6.1p1.orig.tar.gz and http://ftp.debian.org/debian/openssh/openssh_3.6.1p2-2.diff.gz

For patches, you can use the envname option. The effect of this option is that the value given with this option will be used as an environment variable, containing the name of the patch-file (this way, if a new patch comes out, replacing an old one, the -mk file doesn't need to be changed).

buildtool.mk

buildtool.mk is the makefile that will be used to extract the source archive, apply the patches and compile the source. It must include the "mastermakefile" make/MasterInclude.mk. Luckily, the environment variable MASTERMAKEFILE contains the file name of this file (including the full path). In short, every buildtool.mk file should start with:

include $(MASTERMAKEFILE)
targets

The following targets need to be defined in each mk-file (these are the ones that are called by buildtool - of course you are free to define additional ones you need for building the source).

source

Should extract the source tarball, apply patches and run configure (if applicable).

An example mk file follows here:

# example makefile for buildroot
# include the master settings (like BUILD_DIR, ARCH...).
include $(MASTERMAKEFILE)

MYSRC_DIR=example-1.1.1
MYINSTALL_DIR=$(BT_BUILD_DIR)/example

# MYEXAMPLE_PATCH2 is set via the buildtool.cfg entry 

.source:
        tar xvzf example.src.tgz
        (cd $(MYSRC_DIR) ; patch -p1 < ../example-1.patch ;\
        patch -p1 < $(MYEXAMPLE_PATCH2)
        touch .source
source: .source
 
.configure:
        (cd $(MYSRC_DIR) ; ./configure --PREFIX=/usr \
        --SYSCONFDIR=/etc \
        --enable-example )
        touch .configure

.build: .configure
        (cd $(MYSRC_DIR) ; make all CC=$(TARGET_CC) LD=$(TARGET_LD) )
        (cd $(MYSRC_DIR) ; make install INSTALL_PREFIX=$(MYINSTALL_DIR))
        touch .build
build: .build

clean:
        make -C $(MYSRC_DIR) clean
        rm .build
        rm .configure

srcclean: clean
        rm -rf $(MYSRC_DIR) 
        rm .source

The reason for the dot-entries in this file is: you might have problems if you try to apply a patch a second time to an already patched source. Although this should normally not happen, if you are working on a package to include into buildtool, it often does. The trick with the dot-files prevents this (as long as the .source file is there, the source won't be unpacked again. If you want the source to be upacked/patched , make a ./buildtool.pl srcclean PACKAGENAME).

build

Should build the sources and install the binaries (to the directory defined by the BT_BUILD_DIR environment variable).

clean

This should clean up all files created during the compile

Environment variables

The following environment variables are defined when the mk file is run

BT_BUILDROOT

root dir of buildtool (where buildtool.pl resides)

BT_SOURCE_DIR

where the sources are

BT_BUILD_DIR/BT_BUILDDIR

the directory where the binaries (from sources or packages) get installed to

BT_STAGING_DIR

the directory where the binaries for the toolchaing get installed to

BT_LINUX_DIR

directory of the linux kernel

BT_PACKAGE_DIR

where to put finished packages

BT_PATCHTOOL

path to patchtool

BT_DPATCH

path to dpatch

BT_KERNEL_RELEASE

contains the kernel release if the file (BT_SOURCE_DIR)/linux/linux/include/linux/version.h exists, otherwise this variable is empty.

GNU_TARGET_NAME

something like i386-linux

GNU_HOST_NAME

something like i386-linux

TARGET_CC

compiler to use for compiling the binaries

TARGET_LD

linker to use for linking against uClibc

BT_STRIP

strip program to use for stripping binaries

BT_STRIP_LIBOPTS

strip options to use for stripping library files

BT_STRIP_BINOPTS

strip options to use for stripping executables

HOSTCC

host compiler to use to compile the first stage of the toolchain. If your default host gcc is version 4.x, you'll need to adjust that setting. See below for more info.

Directory structure

A short overview of the subdirectories you find when you checked out buildtool and have the buildenv finished:

build

This is the place where sources can install their files (something you normally do with a make install DESTDIR=someDir. From here you can take the files to build the packages. Use a subdir for every package.

buildtool

Internal directory , contains the perl modules/classes.

conf

Global Config files are in here (sources.cfg, buildtools.cfg, installed)

doc

you can find some documentation in here.

log

should contain at least one file: buildtoollog, which has all the messages in it you don't want to see on your terminal. Useful for debugging.

make

The place for the MasterMakefile which should be included by every other makefile.

package

In here the actual packages are built. Every package has its own subdir (openssl for example) in which the unpacked package can be found. The packed packages are stored in the package dir as archives (like openssl.lrp).

source

all sources are here. Sources means downloaded files as well as the unpacked and compiled source (which should install its files to the build directory described above, and maybe to the staging directory from there. One subdir for every source.

staging

This directory is primarily for internal use. The uclibc toolchain , the cross-compiler and all libraries are installed here.

tools

you will find some tools in here , like upx for compressing the kernel and (maybe) additional scripts.

Buildtool Tips and FAQ

What you find here

In this chapter you can find a collection of common problems you may run into if you are trying to compile applications with buildtool.

undefined reference to '__libc_start_main'

The Problem

This may occur if you use a Makefile that uses a $(CC) variable to compile stuff and you have not defined CC. It seems that in this case make uses the wrong compiler or libraries. I ran into the problem while trying to compile an application via compileShell.

Solution

add a CC=gcc when calling make , e.g.

  make all CC=gcc 

and everything should work fine

buildenv not building with gcc 4.0

The Problem

It seems that gcc 4.0 is not capable of building gcc 3.3.3. Unfortunatly there are a number of package sources right now, that refuse to build with a compiler newer than 3.4.

Solution

You need to install an additional compiler, normally you should be able to install gcc3.3 or gcc3.4 (at least for ubuntu and debian testing, this is true). After installing, edit make/MasterInclude.mk and set HOSTCC to your newly installed compiler:

HOSTCC=gcc-3.3

Adjust the line above to match whatever compiler actually comes with the package you installed (it might be called gcc33 or gcc-3.4 too, for example).