From b57e69407776c587c05893e8969834b794f54c6c Mon Sep 17 00:00:00 2001 From: null Date: Mon, 14 Oct 2019 10:15:04 +0100 Subject: [PATCH] - This commit adds a basic distribution-specific loading feature for mkinitcpio configs, and an implementation that can use pacman to add the binaries for a specfic package to the ramdisk. - The motivation for this was to add support for repairing the XFS root or usr filesystem in case of a no-boot. - Both the general design and implementaion as well as the KaOS-specific code and motivating case and will probably need a good reviewing and possible modificaion as I did this in one very long night. - If the system drops into initramfs because of a dirty filesystem, the `fsck` suite of programs is added, by default, from the upstream arch implementation, that determines the root filesystem type and includes these as necessary. However, XFS is not the usual target with linux and the `fsck.xfs` utility cannot do much by itself - it refers the user to the `xfsprogs` command `xfs_repair`. - The motivation of this commit was to enable the addition of this binary into the ramdisk, so that the filesystem can be repaired during the boot process, rather than needing to chroot via e.g. a live CD - If KaOS moves to Illumos, the game will change significantly, but whilst using a linux kernel, the OS has a significant number of differences to most typical linuxen found today. So, I decided to aim for a way to add extra utilities to the ramdisk to help repairs and debugging, both for developers and end users, without doing too much manual work. - Based on the rolling-release model and the atomic level of KaOS packages, I wrote this to extract the binary executables from a given named package, and include these in the mkinitcpio image - The files are queried from *installed* pacman packages, and can be run against an arbitrary callback to filter them, based on directory location or filetype (links or not), for example. - Thus, if a package is added, that needs to be incorporated into the ramdisk, a line can simply be added in `/usr/lib/initcpio/distros/kaos` - This supporting pacman script may be of (some) use to other distros too, hence the aim to separate it form the motivation to write it: - the new hook `distro` was added, and this uses the output from lsb_release (added dependency) to obtain the current OS' ID. It then looks in the distros folder to find a file with this name, in lower-case - The distros folder isn't a .d-style folder: one file is sourced from it, and it is not expected to run everything in there; the OS' script may have helper scripts not to be executed directly. The idea also was to help tracing the OS' heritage: for example, `mint` coulde use functions from `ubuntu` which could use functions from `debian`, etc., so that upstream changes are incorporated, even if some are not executed, overwritten, or avoided. - this hook is added to the default configuration for any build, in `/etc/mkinitcpio.conf`; if the distribution-specific file is found, it is sourced, otherwise the hook is a no-op and the build continues harmlessly. - Note: the build process warns that a command failed, yet the process seems to function and the ramdisk works for me - TODO: - Add support for recursively getting a package and its dependencies? This could quickly explode the number of files included for, in many cases, no good reason - Work with shell scripts. The busybox provided shouldn't have a bash inside it, so it would be mecessary to see what sort of scripts can be included and what can't. I fear parsing the shebang may be too simple to work, and dependencies would need to be extracted somehow - Test for other filesystems - I don't have the partitions to do this now - Define a better way of telling which binaries form the package are 'important', i.e. required for the core system, and which are extras, not likely to be included. At the moment, the config for this search is a const in the file, adn importance is just taken as 'does it live in /{s,}bin?' --- mkinitcpio/PKGBUILD | 14 +++- mkinitcpio/kaos.diff | 162 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+), 3 deletions(-) create mode 100644 mkinitcpio/kaos.diff diff --git a/mkinitcpio/PKGBUILD b/mkinitcpio/PKGBUILD index b6dbfe65..a8a36917 100644 --- a/mkinitcpio/PKGBUILD +++ b/mkinitcpio/PKGBUILD @@ -7,17 +7,25 @@ arch=('x86_64') url="https://projects.archlinux.org/mkinitcpio.git/" license=('GPL') depends=('awk' 'mkinitcpio-busybox' 'kmod' 'util-linux' 'libarchive' 'coreutils' - 'bash' 'findutils' 'grep' 'filesystem>=2016.11' 'gzip' 'systemd') + 'bash' 'findutils' 'grep' 'filesystem>=2016.11' 'gzip' 'systemd' 'lsb-release') optdepends=('xz: Use lzma or xz compression for the initramfs image' 'bzip2: Use bzip2 compression for the initramfs image' 'mkinitcpio-nfs-utils: Support for root filesystem on NFS') backup=('etc/mkinitcpio.conf') source=("https://sources.archlinux.org/other/mkinitcpio/${pkgname}-${pkgver}.tar.gz" - 'init.diff') + 'init.diff' + 'kaos.diff') install=mkinitcpio.install md5sums=('287e2d7ac5de5d50eb4b0a3adbdc6ca8' - '5bce3b1cf3de654151d5522086431624') + '5bce3b1cf3de654151d5522086431624' + '16abae98ce1b3178f9dc5bc261292183') + + prepare() { + cd ${pkgname}-${pkgver} + + patch -p1 -i ${srcdir}/kaos.diff +} build() { cd ${pkgname}-${pkgver} diff --git a/mkinitcpio/kaos.diff b/mkinitcpio/kaos.diff new file mode 100644 index 00000000..4758dc8e --- /dev/null +++ b/mkinitcpio/kaos.diff @@ -0,0 +1,162 @@ +diff -iruN mkinitcpio-26.old/install/distro mkinitcpio-26.new/install/distro +--- mkinitcpio-26.old/install/distro 1970-01-01 01:00:00.000000000 +0100 ++++ mkinitcpio-26.new/install/distro 2019-10-14 09:42:22.000000000 +0100 +@@ -0,0 +1,7 @@ ++#!/bin/bash ++ ++file="${BASH_SOURCE%/*}/distros/$(lsb_release -i | sed 's/.*ID:\s*//' | tr '[:upper:]' '[:lower:]')" ++ ++[ -f "$file" ] && source "$file" # not -f, files don;t seem to have execute bit set ++ ++ +diff -iruN mkinitcpio-26.old/install/distros/add_pacman_binaries.sh mkinitcpio-26.new/install/distros/add_pacman_binaries.sh +--- mkinitcpio-26.old/install/distros/add_pacman_binaries.sh 1970-01-01 01:00:00.000000000 +0100 ++++ mkinitcpio-26.new/install/distros/add_pacman_binaries.sh 2019-10-14 06:43:34.000000000 +0100 +@@ -0,0 +1,53 @@ ++#!/bin/bash ++ ++# Source me, do not run directly ++# This is not a hook itself, it is a utility script intended ot be run from other hooks ++# add_binary and friends need to be provided from mkinitcpio ++ ++packageBinaries(){ ++ ++# Add all executable binaries from the given pacman package(s) to the init ramdisk ++ ++ packageBinariesMatch true "$@" ++ ++} ++ ++packageBinariesMatch(){ ++# As above, but match the binary files against a 'test' before adding ++# Test is a callback that is executed with the filepath as the first argumnt (quoted) ++# i.e. ++# packageBinariesMatch function package1 package2 ... ++# ++# calls `function pack1bin1` `function pack1bin2`, etc. ++# If function does not return false, the executable is added to the ramdisk ++# This allows use of egrep to match the binaries against a pattern: ++# ++# inSbin() { echo "$1" | egrep #^/sbin/# } ++# packageBinariesMatch inSbin mypackage ++# ++# or file to exclude symlinks, for example. ++ ++ test_function="$1" ; shift ++ ++ isBinary(){ ++ file -iL "$@" | grep 'x-executable; charset=binary' -q && echo -n "$@ " ++ } ++ ++ export -f isBinary ++ binaries_to_test="$(pacman -Ql "$@" | xargs -n1 -P0 bash -c 'isBinary $@' _)" ++ export -n isBinary ++ ++ for i in $binaries_to_test ; do #At this point, if for some reason we have spaces in the path, this will break ++ "$test_function" $i && add_binary $i ++ done ++} ++ ++#TODO: add ability to query packages recursively, for dependencies: this requires expac or pacgraph ++#packageBinariesDependencies(){ ++# ++#} ++ ++ ++#TODO: what about shell scripts rather than executables? We have ash, not bash (busybox) ++ ++ +diff -iruN mkinitcpio-26.old/install/distros/kaos mkinitcpio-26.new/install/distros/kaos +--- mkinitcpio-26.old/install/distros/kaos 1970-01-01 01:00:00.000000000 +0100 ++++ mkinitcpio-26.new/install/distros/kaos 2019-10-14 08:27:14.000000000 +0100 +@@ -0,0 +1,78 @@ ++#!/bin/bash ++ ++IMPORTANT_FILESYSTEM_ONLY=1 ++ ++build() { ++ ++ source "${BASH_SOURCE%/*}/add_pacman_binaries.sh" ++ ++ add_filesystem_tools(){ ++ local -A packages ++ ++ packages[xfs]="xfsprogs" ++ packages[ext4]= #None needed by default ++ packages[ext3]= ++ packages[ext2]= ++ #The below are untested: ++ ++ packages[zfs]= #TODO: add packages for zfs rootfs, if it will be supported in linux ++ packages[reiserfs]="reiserfsprogs" #FIXME: Don't have a partition to test this label output from the below. ++ ++ [ -z ${packages[$1]} ] && return || package=${packages[$1]} ++ ++ # Add 'important' binaries from the package: ++ if (( $IMPORTANT_FILESYSTEM_ONLY )); then ++ isImportant (){ ++ [[ "$1" =~ ^/s?bin.*$ ]] ++ } ++ echo "Adding 'important' binaries from $package package to support $1 filesystem repair" ++ packageBinariesMatch isImportant $package ++ else ++ # or add all? ++ echo "Adding all binaries from $package package to support $1 filesystem repair" ++ packageBinaries $package ++ fi ++ ++ ++ } ++ ++ ++ ### Add tools needed to repair the filesystem in case of a no-boot ++ ++ if (( ! fs_autodetect_failed )) && [[ $rootfstype$usrfstype ]]; then ++ if [[ $rootfstype ]]; then ++ add_filesystem_tools $rootfstype ++ fi ++ if [[ $usrfstype && $usrfstype != $rootfstype ]]; then ++ add_filesystem_tools $usrfstype ++ fi ++ else ++ #KaOS defaults ++ echo Using KaOS default filesystems of support for xfs and ext4 ++ add_filesystem_tools ext4 ++ add_filesystem_tools xfs ++ fi ++ ++ ++ ### Others ++# Example: ++# ++# packageBinaries \ ++# vim `#Example` \ ++# nano `#Example` ++ ++} ++ ++help() { ++ cat <