GrabDuck

init6/init_6

:

From old wiki http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree

This article describes how to reduce the Portage tree, using a squashfs image, to 30-40MB. A good alternative to having the portage tree on a separate partition. For a list of publicly available copies please see the bottom of this article.

Required Packages

You will need sys-fs/squashfs-tools which provides mksquashfs,

emerge -av squashfs-tools

And of course the aufs module:

emerge -av aufs3

Note: If you are using kernel 2.6 then emerge sys-fs/aufs2 instead. In case aufs2 complains about missing Symbols related to inotify, you can remove inotify-useflag from aufs as a workaround.

Note: Alternative you can use the "live" ebuild, aufs2-9999 or a kernel source package that includes aufs, like sys-kernel/zen-sources.

Kernel Configuration

Kernels based on sys-kernel/gentoo-sources include SquashFS by default.

# Linux Kernel Configuration: SquashFS Kernel Config
Device Drivers --->
  Block Devices --->
    <M> Loopback device support
File systems --->
  Miscellaneous Filesystems --->
    <M> SquashFS

It is of course possible to have these built-in, but we avoid a reboot when building these as modules.

Preparation

For a single squashed tree, you will need to move anything not part of the portage tree out of PORTDIR( /usr/portage ). This usally includes DISTDIR and laymans storage. Modify /etc/make.conf,

# File: /etc/make.conf
...
DISTDIR="/var/portage/distfiles"
...

Remove /usr/portage/distfiles,

rm /usr/portage/distfiles -rf

Note: The following step is only needed if you are using a layman version older than 1.2.0

Layman's configuration file is /etc/layman/layman.cfg. Here we move layman's storage directory to /var/portage, same directory as DISTDIR,

# File: /etc/layman/layman.cfg
...
storage   : /var/portage/layman
...

Move the layman storage directory to our new location,

mv /usr/portage/local/layman /var/portage/layman

init.d/squash_portage

Paste this into /etc/init.d/squash_portage,

File: /etc/init.d/squash_portage
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# /etc/init.d/squash_portage allows efficient compression of
# Gentoo portage arborescence
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs and sys-fs/aufs are necessary for read-write support.
#
# Author: Mathias Laurin <mathias_laurin@users.sourceforge.net>
# 2006-11-28, v.0.1.5(4)
# 2009-02-24, v.0.1.6(1) Weedy <weedy2887@gmail.com>
# 2009-03-20, v.0.1.7(1) j0inty <j0inty@stollfuss.net>
# 2009-07-10, v.0.1.8(1) j0inty
# 2009-09-01. v.0.1.9(1) nall <soir@fuzzysock.net>

extra_started_commands="sync"

source /usr/share/portage/config/make.globals
source /etc/make.conf
SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"
SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"
SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"
DEF_RW="/dev/shm/.portage-rw"
SQFS_OPTS="-force-uid portage -force-gid portage -no-duplicates"

depend() {
	need localmount modules
}


check_support() {
	if ! [ -x /usr/bin/mksquashfs ] ; then
		eerror "ERROR: sys-fs/squashfs-tools is not installed."
		return 1
	fi
	if ! [ -w /dev/loop0 ] ; then
		eerror "ERROR: loopback support is not available."
		return 1
	fi
	if ! [[ $(grep -s $'\taufs$' /proc/filesystems) ]] ; then
		eerror "ERROR: aufs filesystem support is not available."
		return 1
	fi
	if ! [[ $(grep -s $'\tsquashfs$' /proc/filesystems) ]] ; then
		eerror "ERROR: squashfs filesystem support is not available."
		return 1
	fi
	return 0
}

makeImage() {
	mksquashfs $PORTDIR $SQFS_NEW $SQFS_OPTS # 2>/dev/null
	retval=$?
	ln -sf $SQFS_NEW $SQFS_CUR
	eend $retval
}

sync() {
	ebegin "Syncing portage tree"
	eval $SYNC_CMDS
	#svc_stop; svc_start
	stop
	start
	eend 0
}

start() {
	check_support || return 1
	if [ -f "$SQFS_CUR" ]; then
		ebegin "SQFS-PORTAGE: Mounting read-only squashfs image"
		mount -rt squashfs -o loop,nodev,noexec $SQFS_CUR $PORTDIR
		retval=$?
		[ $retval -ne 0 ] && return $retval
	else
		if [ ! -f "/usr/portage/metadata/timestamp.chk" ]; then
			ebegin "SQFS-PORTAGE: $PORTDIR looks empty or corrupted, syncing"
			eval $SYNC_CMDS
		fi
		einfo "  $SQFS_CUR does not exist, creating"
		mkdir -p $SQFS_DIRNAME
		makeImage
		[ $? -ne 0 ] && eerror "ERROR: failed to create initial tree image"
		einfo "Clearing ${PORTDIR}"
		rm -r ${PORTDIR}
		mkdir ${PORTDIR}
		start
		eend 0
	fi

	ebegin "Mounting read-write with aufs"
	if [ ! $PORTAGE_RW ] ; then
		einfo "  mounted in tmpfs (RAM)"
		PORTAGE_RW="${DEF_RW}"
	fi
	[ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
	chmod 0750 $PORTAGE_RW
	chown portage:portage $PORTAGE_RW
	mount -t aufs -o nodev,noexec,br=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR
	eend $?

	if [ "$DISTDIR" == "/usr/portage/distfiles" ]; then
		mkdir -p /usr/local/distfiles 
		mount -o bind /usr/local/distfiles /usr/portage/distfiles
		ewarn "DISTDIR is currently inside the portage tree. It has been bind 
		mounted to keep the SquashFS image small."
	fi
}

stop() {
	ebegin "SQFS-PORTAGE: Stopping and unmounting"
	[ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
	if [ $(du -s --exclude=.w* $PORTAGE_RW | cut -f 1) -gt 4 ]; then
		einfo "  Changes detected, updating image."
		mv -f $SQFS_NEW $SQFS_OLD
		makeImage
		rm -f $SQFS_OLD
	else
		einfo "  No changes detected, skipping update."
		eend 0
	fi

	if [ "$DISTDIR" == "/usr/portage/distfiles" ]; then
		einfo "  Unmounting distfiles"
		umount /usr/local/distfiles
	fi;

	einfo "  Unmounting the tree"
	umount -t aufs  $PORTDIR
	umount -t squashfs $PORTDIR
	rm -rf $PORTAGE_RW
	eend 0
}

Make it executable,

chmod 755 /etc/init.d/squash_portage

conf.d/squash_portage

Paste this in the corresponding configuration file /etc/conf.d/squash_portage,

# File: /etc/conf.d/squash_portage
# /etc/conf.d/squash_portage

# SQFS_DIRNAME points to the directory that will contain the sqfs
# images, recommended value is /var/portage
SQFS_DIRNAME="/var/portage"

# Leave PORTAGE_RW empty for use with tmpfs, a ram-based filesystem,
# This is recommended unless you are short of RAM
PORTAGE_RW=""

# If you need more then just emerge --sync, or are using another
# package manager add them here. Example SYNC_CMDS="/usr/bin/layman -S; /usr/bin/eix-sync"
SYNC_CMDS="emerge --sync"

Usage

To make sure you always have a mounted portage tree:

rc-update add squash_portage default

Now it is time to initialize the tree. Simply start the service.

/etc/init.d/squash_portage start

Note: The following option is only recommended if you know exactly what you intend to do.

Preparation

For multiple squashed trees, you probably want to move DISTDIR (usually /usr/portage/distfiles) out of PORTDIR (usually /usr/portage), or else it'll take a very long time to create a portage sqfs image, without any significant size reduction.

What we'll describe here is an example configuration, for three trees, each of located in /var/portage/{portage,layman,local}, with it's original locations being /usr/portage, /usr/local/portage/layman and /usr/local/portage. One could achieve the same with only two trees by merging local with layman. But this would side effect you dependent of layman for your local tree, and our script configuration, to work without modifications.

First we make sure that /var/portage and sub-dirs does exist.

mkdir -p /var/portage/{portage,layman,local}

Remember that we want every tree in /var/portage, and, we also need layman to write it's overlays to /etc/make.overlays, instead of /usr/local/layman/make.conf. So, modify /etc/make.conf,

Note: Remember to remove the old "source /usr/local/portage/layman/make.conf" before entering the new one

# File: /etc/make.conf
source /etc/make.overlays
...
PORTDIR=/var/portage/portage
LOCALDIR="/var/portage/local"
DISTDIR="/var/portage/distfiles"
...

and /etc/layman/layman.cfg,

# File: /etc/layman/layman.cfg
...
storage   : /var/portage/layman
...
make_conf : /etc/make.overlays
...

Move /usr/portage/distfiles,

mv /usr/portage/distfiles /var/portage/distfiles

Or remove /usr/portage/distfiles,

rm /usr/portage/distfiles -rf

Move /usr/local/layman/make.conf,

mv /usr/local/layman/make.conf /etc/make.overlays

Modify /etc/make.overlays overlays to /var/portage/,

Note: Example configuration, you have to adapt your configuration

File: /etc/make.overlays
PORTDIR_OVERLAY="
/var/portage/layman/sunrise
/var/portage/layman/gentoo-china
/var/portage/layman/devnull
/var/portage/layman/kde-testing
/var/portage/layman/mozilla
/var/portage/layman/qting-edge
$PORTDIR_OVERLAY
/var/portage/local"

Before we create the images we need to split layman from local, doing:

mv /usr/local/portage/layman/ /usr/local/layman/

Now we have to create the sqfs images, since our script expect the images to be at /var/portage/, named as sqfs.${TREE}-current.sqfs, we proceed as following:

mksquashfs /usr/portage/ sqfs.portage-current.sqfs
mksquashfs /usr/local/portage/ sqfs.local-current.sqfs
mksquashfs /usr/local/layman/ sqfs.layman-current.sqfs

Since some system files need files at /usr/portage/, we have to symlink after we remove them. It's safer to move:

mv /usr/portage/eclass /usr/portage/eclass.old
mv /usr/portage/profiles /usr/portage/profiles.old

After we symlink:

ln -fs /var/portage/portage/eclass/ /usr/portage/eclass
ln -fs /var/portage/portage/profiles/ /usr/portage/profiles

Init script

You can now create and configure /etc/init.d/squash_portage, first of all create it:

File: /etc/init.d/squash_portage
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# /etc/init.d/squash_portage allows efficient compression of
# Gentoo portage arborescence
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs, sys-fs/squashfs-tools and sys-fs/aufs
# or sys-fs/aufs2 (recommended) are necessary for read-write support.
#
# Author: Mathias Laurin <mathias_laurin@users.sourceforge.net>
# 2006-11-28, v.0.1.5(4)
# 2009-02-24, v.0.1.6(1) Weedy <weedy2887@gmail.com>
# 2009-03-20, v.0.1.7(1) j0inty <j0inty@stollfuss.net>
# 2009-07-10, v.0.1.8(1) j0inty
# 2009-08-14, v.0.1.9(1) morris <mauricioarozi@gmail.com>
# 2009-09-01. v.0.1.9(1) nall <soir@fuzzysock.net>
# 2009-11-22, v.0.2.0(3) morris <mauricioarozi@gmail.com>

opts="sync"

depend() {
	need localmount modules
}

retuval() {
	retval=${1}
	eend ${retval}
	[ ${retval} -ne 0 ] && \
	ewarn "${2}" && \
	return 1
}

check_support() {
	if ! [ -x /usr/bin/mksquashfs ] ; then
		eerror "ERROR: sys-fs/squashfs-tools is not installed."
		return 1
	fi
	if ! [ -w /dev/loop0 ] ; then
		eerror "ERROR: loopback support is not available."
		return 1
	fi
	if ! [[ $(grep -s aufs /proc/filesystems) ]] ; then
		eerror "ERROR: aufs filesystem support is not available."
		return 1
	fi
	if ! [[ $(grep -s squashfs /proc/filesystems) ]] ; then
		eerror "ERROR: squashfs filesystem support is not available."
		return 1
	fi
}


sync() {
	source /etc/make.globals		# MUST source inside function, causes initramfs errors otherwise
	source /etc/make.conf
	ebegin "Syncing portage tree"
	if [ ${SYNC_CMD} ]; then
		eval ${SYNC_CMD}
	else
		[ -x '/usr/bin/emerge' ] && local SYNC_CMD='command emerge --sync'
		[ -x '/usr/bin/paludis' ] && local SYNC_CMD='command paludis --sync'
		[ -x '/usr/bin/eix' ] && local SYNC_CMD='command eix-sync'
		# make eix-sync work with layman overlays shound't be default
		# [ -x '/usr/bin/eix' ] && [ -x '/usr/bin/layman' ] && [ ! -f /etc/eix-sync.conf ] && `/bin/echo '*' > /etc/eix-sync.conf`

		eval "${SYNC_CMD}"
	fi
	retuval ${?} "Error: ${SYNC_CMD}"
	stop
	start
	eend 0
}

start() {
	source /etc/make.globals	# MUST source inside function
	source /etc/make.conf

	ebegin "Mounting read-only squashfs image(s)"
	check_support
	if [ ! -d "${SQFS_DIRNAME}" ]; then
		einfo "${SQFS_DIRNAME} does not exist, creating"
		mkdir -p "${SQFS_DIRNAME}"
		retuval ${?} "Error: mkdir -p ${SQFS_DIRNAME}"
	fi

	for i in ${SQFSS[@]}; do
		einfo "Mounting ${i}"
		mount -rt squashfs -o loop,nodev,noexec "${SQFS_DIRNAME}/sqfs.${i}-current.sqfs" "${SQFS_DIRNAME}/${i}"
		retuval ${?} "Error: mount -rt squashfs -o loop,nodev,noexec \"${SQFS_DIRNAME}/sqfs.${i}-current.sqfs\" \"${SQFS_DIRNAME}/${i}\""
		[ "${SQFS_DIST}" ] || \
		if [ `echo ${DISTDIR} | grep "${SQFS_DIRNAME}/${i}"` ]; then
			mkdir -p /usr/local/distfiles
			retuval ${?} "Error: mkdir -p /usr/local/distfiles"
			mount -o bind "/usr/local/distfiles" "${DISTDIR}"
			retuval ${?} "Error: mount -o bind /usr/local/distfiles ${DISTDIR}"
			ewarn "DISTDIR is currently inside of ${SQFS_DIRNAME}/${i} tree. 
			It has been bind mounted to keep the SquashFS image small."
		fi
	done; unset i rw

	einfo "Mounting read-write with aufs"
	for i in `seq 0 $[${#FSRW[@]}-1]`; do
		local RW=${FSRW[${i}]:-"/dev/shm/.${SQFSS[${i}]}-rw"}
		[ -d "${RW}" ] || einfo "Creating ${RW}" && mkdir -p "${RW}"
		retuval ${?} "Error: mkdir -p \"${RW}\"" 
		chmod 0750 "${RW}"
		retuval ${?} "Error: chmod 0750 \"${RW}\""
		chown portage:portage "${RW}"
		retuval ${?} "Error: chown portage:portage \"${RW}\""
	done; unset a i rw

	for i in ${SQFSS[@]}; do
		[ ${a} ] && a=$[${a}+1] || local a=0
		local RW=${FSRW[${a}]:-"/dev/shm/.${i}-rw"}
		einfo "${RW}"
		mount -t aufs -o "nodev,noexec,br:${RW}=rw:${SQFS_DIRNAME}/${i}=ro" aufs "${SQFS_DIRNAME}/${i}"
		retuval ${?} "Error: mount -t aufs -o \"nodev,noexec,br:${RW}=rw:${SQFS_DIRNAME}/${i}=ro\" aufs \"${SQFS_DIRNAME}/${i}\""
		einfo "${SQFSS[${a}]} mounted in ${RW}"
	done; unset a i rw
	eend ${?}
}

stop() {
	source /etc/make.globals	# MUST source inside function
	source /etc/make.conf

	check_support
	if [ "$RC_RUNLEVEL" != shutdown ]; then	# OpenRC timeout doesn't allow this kind of thing
		ebegin "Updating portage tree"
		for i in `seq 0 $[${#FSRW[@]}-1]`; do
			local RW=${FSRW[${i}]:-"/dev/shm/.${SQFSS[${i}]}-rw"}
			einfo "Syncing the tree ${SQFSS[${i}]}"
			if [ ! -z "`/bin/ls -A "${RW}" | /bin/grep -v .wh.`" ]; then
				einfo "Syncing..."
				local SOLD="${SQFS_DIRNAME}/sqfs.${SQFSS[${i}]}-old.sqfs"
				local SNEW="${SQFS_DIRNAME}/sqfs.${SQFSS[${i}]}-current.sqfs"
				local SS="${SQFS_DIRNAME}/sqfs.${SQFSS[${i}]}.sqfs"
				mv -f "${SNEW}" "${SOLD}"
				retuval ${?} "Error: mv -f \"${SNEW}\" \"${SOLD}\""
				[ -w "${SQFS_DIRNAME}" ] && \
				/usr/bin/mksquashfs "${SQFS_DIRNAME}/${SQFSS[${i}]}" "${SNEW}" ${SQFS_OPTS}
				retuval ${?} "Error: /usr/bin/mksquashfs \"${SQFS_DIRNAME}/${SQFSS[${i}]}\" \"${SNEW}\" ${SQFS_OPTS}"
				/bin/ln -fs "${SNEW}" "${SS}"
				retuval ${?} "Error: /bin/ln -fs \"${SNEW}\" \"${SS}\""
			else
				einfo "Nothing to do"
			fi
		done; unset a i retval
	fi

	ebegin "Unmounting the tree(s)"
		for i in ${SQFSS[@]}; do
		[ ${a} ] && a=$[${a}+1] || local a=0
		local RW=${FSRW[${a}]:-"/dev/shm/.${i}-rw"}
				umount -f -l -t aufs "${SQFS_DIRNAME}/${i}"
		retuval ${?} "Error: umount -f -l -t aufs \"${SQFS_DIRNAME}/${i}\""
		umount -f -l -t squashfs "${SQFS_DIRNAME}/${i}"
		retuval ${?} "Error: umount -f -l -t squashfs \"${SQFS_DIRNAME}/${i}\""
		/bin/rm -fr "${RW}"
		retuval ${?} "Error: /bin/rm -fr \"${RW}\""
		done; unset i retval

	eend 0
}

Make it executable,

chmod 755 /etc/init.d/squash_portage

Now to configure, create and edit the corresponding configuration file /etc/conf.d/squash_portage,

Note: Configuration file for example setup, remember to change it!

# File: /etc/conf.d/squash_portage
# /etc/conf.d/squash_portage

# Fill SQFS_DIST with anything if you don't mind to squash your distfiles
# It is recommended to leave blank
SQFS_DIST=""

# SQFS_DIRNAME points to the directory that will contain the sqfs images
SQFS_DIRNAME="/var/portage"

# Options used for ALL image creations
# Do not change unless you know what you are doing
SQFS_OPTS="-force-uid portage -force-gid portage -no-duplicates"

# Populate FSRW array with nulls ("", '', 0...) to use tmpfs, a ram-based filesystem,
# This is recommended unless you are short of RAM
FSRW=('' '' '')

# FS mount array, used in substitution: ${SQFS_DIRNAME}/${SQFSS[@]}
SQFSS=("portage" "local" "layman")

# If you need more than just emerge --sync, or are using another
# package manager add them here. Example SYNC_CMDS="/usr/bin/layman -S; /usr/bin/eix-sync"
SYNC_CMD="eix-sync"
# Note: eix-sync can run `layman -S` automatically if you run `echo '*' > /etc/eix-sync.conf` once

Check your initial setup running the script once.

/etc/init.d/squash_portage start

Now complete your setup restarting it, so it will create necessary symlinks while stopping.

/etc/init.d/squash_portage restart

To make sure you always have a mounted portage tree:

rc-update add squash_portage default

If everything went alright, you can now delete the all old trees content but /usr/portage/eclass and /usr/portage/profiles links from /usr/ if you want.

Usage

You can sync the trees with:

/etc/init.d/squash_portage sync

You can also start, stop, and restart it:

/etc/init.d/squash_portage start
/etc/init.d/squash_portage stop
/etc/init.d/squash_portage restart

Note: Restart is useful if you synced trees manually, and want to sync just the sqfs images automatically