catmeow/cache-apt-pkgs-action
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Execute installation scripts feature, debug mode, and permission denied fix. (#65)

Browse source 
* Execute installation scripts and debug mode features. (#64)

* Provide the ability to call Debian package manager installation scripts (i.e. `*.[preinst, postinst]`).
* Introduce a debug mode that runs the scripts in verbose mode and uploads the logs for retrieval.
* Updated README to reflect new features and provided more info on how to use the action versions.

* Dev (#66)

* Fix permission denied error.

* Fix permission denied error. (#51)

* Remove compression from file caching. (#53)

* Draft of postinst support from issue #44.

* Remove bad option.

* Removed extraneous line.

* Cover no packages edge case when writing manifest.

* Fix postinst bugs and add docs to lib.

* Made cache directory variable and more refinements to postinst.

* Update deprecated option.

https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

* Rollback accidental commit of new postinst feature.

* Minor edit ands full install script execution FR.

* Fix execute_install_scripts message to show the right param name.

* Fix param check.

* Minor fix to doc.

* Upload action logs for debugging.

* Make artifact names unique.

* Add debug option.

* Update description.

* Debug package list issue.

* Rollback 76128c60a1

* Revert outputs set behavior to see if it fixes outputs issue in dev.

* Restore updated outputs behavior. So strange it is working when I revert.

* Fix bugs in install script execution.

* Add error suppression on file testing.

* Debug feature.

* Link to the issue that started the postinst troubleshooting.

* Describe action version usage.

* Fix package outputs command.
This commit is contained in:
Andrew Walsh 2022-11-23 22:24:00 -08:00 committed by GitHub
parent 276bbdc9f3
commit 51678ad913
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 268 additions and 42 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

44
README.md
View file

@ -16,10 +16,22 @@ This action is a composition of [actions/cache](https://github.com/actions/cache
Create a workflow `.yml` file in your repositories `.github/workflows` directory. An [example workflow](#example-workflow) is available below. For more information, reference the GitHub Help Documentation for [Creating a workflow file](https://help.github.com/en/articles/configuring-a-workflow#creating-a-workflow-file).
### Versions
There are three kinds of version labels you can use.
* `@latest` - This will give you the latest release.
* `@v#` - Major only will give you the latest release for that major version only (e.g. `v1`).
* Branch
* `@master` - Most recent manual and automated tested code. Possibly unstable since it is pre-release.
* `@staging` - Most recent automated tested code and can sometimes contain experimental features. Is pulled from dev stable code.
* `@dev` - Very unstable and contains experimental features. Automated testing may not show breaks since CI is also updated based on code in dev.
### Inputs
* `packages` - Space delimited list of packages to install.
* `version` - Version of cache to load. Each version will have its own cache. Note, all characters except spaces are allowed.
* `execute_install_scripts` - Execute Debian package pre and post install script upon restore. See [Caveats / Non-file Dependencies](#non-file-dependencies) for more information.
### Outputs
@ -74,6 +86,36 @@ jobs:
version: 1.0
```
## Cache Limits
## Caveats
### Non-file Dependencies
This action is based on the principle that most packages can be cached as a fileset. There are situations though where this is not enough.
* Pre and post installation scripts needs to be ran from `/var/lib/dpkg/info/{package name}.[preinst, postinst]`.
* The Debian package database needs to be queried for scripts above (i.e. `dpkg-query`).
The `execute_install_scripts` argument can be used to attempt to execute the install scripts but they are no guaranteed to resolve the issue.
```yaml
- uses: awalsh128/cache-apt-pkgs-action@latest
with:
packages: mypackage
version: 1.0
execute_install_scripts: true
```
If this does not solve your issue, you will need to run `apt-get install` as a separate step for that particular package unfortunately.
```yaml
run: apt-get install mypackage
shell: bash
```
Please reach out if you have found a workaround for your scenario and it can be generalized. There is only so much this action can do and can't get into the area of reverse engineering Debian package manager. It would be beyond the scope of this action and may result in a lot of extended support and brittleness. Also, it would be better to contribute to Debian packager instead at that point.
For more context and information see [issue #57](https://github.com/awalsh128/cache-apt-pkgs-action/issues/57#issuecomment-1321024283) which contains the investigation and conclusion.
### Cache Limits
A repository can have up to 5GB of caches. Once the 5GB limit is reached, older caches will be evicted based on when the cache was last accessed. Caches that are not accessed within the last week will also be evicted.

21
action.yml
View file

@ -11,11 +11,17 @@ inputs:
required: true
default: ''
version:
description: 'Version will create a new cache and install packages.'
description: 'Version of cache to load. Each version will have its own cache. Note, all characters except spaces are allowed.'
required: false
default: ''
execute_install_scripts:
description: 'Execute Debian package pre and post install script upon restore. See README.md caveats for more information.'
required: false
default: 'false'
refresh:
description: 'Option to refresh / upgrade the packages in the same cache.'
description: 'OBSOLETE, use version instead.'
debug:
description: 'Enable debugging when there are issues with action. Minor performance penalty.'
required: false
default: 'false'
@ -40,6 +46,8 @@ runs:
${{ github.action_path }}/pre_cache_action.sh \
~/cache-apt-pkgs \
"${{ inputs.version }}" \
"${{ inputs.execute_install_scripts }}" \
"${{ inputs.debug }}" \
${{ inputs.packages }}
echo "CACHE_KEY=$(cat ~/cache-apt-pkgs/cache_key.md5)" >> $GITHUB_ENV
shell: bash
@ -56,8 +64,17 @@ runs:
~/cache-apt-pkgs \
/ \
"${{ steps.load-cache.outputs.cache-hit }}" \
"${{ inputs.execute_install_scripts }}" \
"${{ inputs.debug }}" \
${{ inputs.packages }}
function create_list { local list=$(cat ~/cache-apt-pkgs/manifest_${1}.log | tr '\n' ','); echo ${list:0:-1}; };
echo "package-version-list=$(create_list main)" >> $GITHUB_OUTPUT
echo "all-package-version-list=$(create_list all)" >> $GITHUB_OUTPUT
shell: bash
- id: upload-logs
if: ${{ inputs.debug }} == "true"
uses: actions/upload-artifact@v3
with:
name: cache-apt-pkgs-logs%${{ inputs.packages }}%${{ inputs.version }}
path: ~/cache-apt-pkgs/*.log

28
install_and_cache_pkgs.sh Executable file → Normal file
View file

@ -3,6 +3,11 @@
# Fail on any error.
set -e
# Debug mode for diagnosing issues.
# Setup first before other operations.
debug="${2}"
test ${debug} == "true" && set -x
# Include library.
script_dir="$(dirname -- "$(realpath -- "${0}")")"
source "${script_dir}/lib.sh"
@ -11,7 +16,7 @@ source "${script_dir}/lib.sh"
cache_dir="${1}"
# List of the packages to use.
input_packages="${@:2}"
input_packages="${@:3}"
# Trim commas, excess spaces, and sort.
normalized_packages="$(normalize_package_list "${input_packages}")"
@ -40,8 +45,13 @@ log "done"
log_empty_line
log "Updating APT package list..."
last_update_delta_s=$(($(date +%s) - $(date +%s -r /var/cache/apt/pkgcache.bin)))
if test $last_update_delta_s -gt 300; then
sudo apt-fast update > /dev/null
log "done"
else
log "skipped (fresh by ${last_update_delta_s} seconds)"
fi
log_empty_line
@ -71,18 +81,20 @@ log_empty_line
installed_package_count=$(wc -w <<< "${installed_packages}")
log "Caching ${installed_package_count} installed packages..."
for installed_package in ${installed_packages}; do
cache_filepath="${cache_dir}/${installed_package}.tar.gz"
cache_filepath="${cache_dir}/${installed_package}.tar"
# Sanity test in case APT enumerates duplicates.
if test ! -f "${cache_filepath}"; then
read installed_package_name installed_package_ver < <(get_package_name_ver "${installed_package}")
log " * Caching ${installed_package_name} to ${cache_filepath}..."
# Pipe all package files (no folders) to Tar.
dpkg -L "${installed_package_name}" |
while IFS= read -r f; do
if test -f $f || test -L $f; then echo "${f:1}"; fi; #${f:1} removes the leading slash that Tar disallows
done |
xargs tar -czf "${cache_filepath}" -C /
# Pipe all package files (no folders) and installation control data to Tar.
{ dpkg -L "${installed_package_name}" \
& get_install_filepath "" "${package_name}" "preinst" \
& get_install_filepath "" "${package_name}" "postinst"; } |
while IFS= read -r f; do test -f "${f}" -o -L "${f}" && get_tar_relpath "${f}"; done |
sudo xargs tar -cf "${cache_filepath}" -C /
log " done (compressed size $(du -h "${cache_filepath}" | cut -f1))."
fi

142
lib.sh
View file

@ -1,20 +1,40 @@
#!/bin/bash
# Sort these packages by name and split on commas.
function normalize_package_list {
local stripped=$(echo "${1}" | sed 's/,//g')
# Remove extraneous spaces at the middle, beginning, and end.
local trimmed="$(echo "${stripped}" | sed 's/\s\+/ /g; s/^\s\+//g; s/\s\+$//g')"
local sorted="$(echo ${trimmed} | tr ' ' '\n' | sort | tr '\n' ' ')"
echo "${sorted}"
###############################################################################
# Execute the Debian install script.
# Arguments:
# Root directory to search from.
# File path to cached package archive.
# Installation script extension (preinst, postinst).
# Parameter to pass to the installation script.
# Returns:
# Filepath of the install script, otherwise an empty string.
###############################################################################
function execute_install_script {
local package_name=$(basename ${2} | awk -F\: '{print $1}')
local install_script_filepath=$(\
get_install_filepath "${1}" "${package_name}" "${3}")
if test ! -z "${install_script_filepath}"; then
log "- Executing ${install_script_filepath}..."
# Don't abort on errors; dpkg-trigger will error normally since it is
# outside its run environment.
sudo sh -x ${install_script_filepath} ${4} || true
log " done"
fi
}
# Gets a list of installed packages as space delimited pairs with each pair colon delimited.
###############################################################################
# Gets a list of installed packages from a Debian package installation log.
# Arguments:
# The filepath of the Debian install log.
# Returns:
# The list of space delimited pairs with each pair colon delimited.
# <name>:<version> <name:version>...
###############################################################################
function get_installed_packages {
install_log_filepath="${1}"
local install_log_filepath="${1}"
local regex="^Unpacking ([^ :]+)([^ ]+)? (\[[^ ]+\]\s)?\(([^ )]+)"
dep_packages=""
local dep_packages=""
while read -r line; do
if [[ "${line}" =~ ${regex} ]]; then
dep_packages="${dep_packages}${BASH_REMATCH[1]}:${BASH_REMATCH[4]} "
@ -30,7 +50,13 @@ function get_installed_packages {
fi
}
# Split fully qualified package into name and version.
###############################################################################
# Splits a fully qualified package into name and version.
# Arguments:
# The colon delimited package pair or just the package name.
# Returns:
# The package name and version pair.
###############################################################################
function get_package_name_ver {
IFS=\: read name ver <<< "${1}"
# If version not found in the fully qualified package value.
@ -40,17 +66,107 @@ function get_package_name_ver {
echo "${name}" "${ver}"
}
###############################################################################
# Gets the package name from the cached package filepath in the
# path/to/cache/dir/<name>:<version>.tar format.
# Arguments:
# Filepath to the cached packaged.
# Returns:
# The package name.
###############################################################################
function get_package_name_from_cached_filepath {
basename ${cached_pkg_filepath} | awk -F\: '{print $1}'
}
###############################################################################
# Gets the Debian install script file location.
# Arguments:
# Root directory to search from.
# Name of the unqualified package to search for.
# Extension of the installation script (preinst, postinst)
# Returns:
# Filepath of the script file, otherwise an empty string.
###############################################################################
function get_install_filepath {
# Filename includes arch (e.g. amd64).
local filepath="$(\
ls -1 ${1}var/lib/dpkg/info/${2}*.${3} 2> /dev/null \
| grep -E ${2}'(:.*)?.'${3} | head -1 || true)"
test "${filepath}" && echo "${filepath}"
}
###############################################################################
# Gets the relative filepath acceptable by Tar. Just removes the leading slash
# that Tar disallows.
# Arguments:
# Absolute filepath to archive.
# Returns:
# The relative filepath to archive.
###############################################################################
function get_tar_relpath {
local filepath=${1}
if test ${filepath:0:1} = "/"; then
echo "${filepath:1}"
else
echo "${filepath}"
fi
}
function log { echo "$(date +%H:%M:%S)" "${@}"; }
function log_err { >&2 echo "$(date +%H:%M:%S)" "${@}"; }
function log_empty_line { echo ""; }
###############################################################################
# Sorts given packages by name and split on commas.
# Arguments:
# The comma delimited list of packages.
# Returns:
# Sorted list of space delimited packages.
###############################################################################
function normalize_package_list {
local stripped=$(echo "${1}" | sed 's/,//g')
# Remove extraneous spaces at the middle, beginning, and end.
local trimmed="$(\
echo "${stripped}" \
| sed 's/\s\+/ /g; s/^\s\+//g; s/\s\+$//g')"
local sorted="$(echo ${trimmed} | tr ' ' '\n' | sort | tr '\n' ' ')"
echo "${sorted}"
}
###############################################################################
# Validates an argument to be of a boolean value.
# Arguments:
# Argument to validate.
# Variable name of the argument.
# Exit code if validation fails.
# Returns:
# Sorted list of space delimited packages.
###############################################################################
function validate_bool {
if test "${1}" != "true" -a "${1}" != "false"; then
log "aborted"
log "${2} value '${1}' must be either true or false (case sensitive)."
exit ${3}
fi
}
###############################################################################
# Writes the manifest to a specified file.
# Arguments:
# Type of manifest being written.
# List of packages being written to the file.
# File path of the manifest being written.
# Returns:
# Log lines from write.
###############################################################################
function write_manifest {
if [ ${#2} -eq 0 ]; then
log "Skipped ${1} manifest write. No packages to install."
else
log "Writing ${1} packages manifest to ${3}..."
# 0:-1 to remove trailing comma, delimit by newline and sort.
echo "${2:0:-1}" | tr ',' '\n' | sort > ${3}
log "done"
fi
}
get_installed_packages "/tmp/cache-apt-pkgs-action-cache/install.log"

16
post_cache_action.sh
View file

@ -12,20 +12,26 @@ cache_dir="${1}"
# Root directory to untar the cached packages to.
# Typically filesystem root '/' but can be changed for testing.
# WARNING: If non-root, this can cause errors during install script execution.
cache_restore_root="${2}"
# Indicates that the cache was found.
cache_hit="${3}"
# List of the packages to use.
packages="${@:4}"
# Cache and execute post install scripts on restore.
execute_install_scripts="${4}"
script_dir="$(dirname -- "$(realpath -- "${0}")")"
# Debug mode for diagnosing issues.
debug="${5}"
test ${debug} == "true" && set -x
# List of the packages to use.
packages="${@:6}"
if [ "$cache_hit" == true ]; then
${script_dir}/restore_pkgs.sh ~/cache-apt-pkgs "${cache_restore_root}"
${script_dir}/restore_pkgs.sh "${cache_dir}" "${cache_restore_root}" "${execute_install_scripts}" "${debug}"
else
${script_dir}/install_and_cache_pkgs.sh ~/cache-apt-pkgs ${packages}
${script_dir}/install_and_cache_pkgs.sh "${cache_dir}" "${debug}" ${packages}
fi
log_empty_line

22
pre_cache_action.sh Executable file → Normal file
View file

@ -1,5 +1,11 @@
#!/bin/bash
# Debug mode for diagnosing issues.
# Setup first before other operations.
debug="${4}"
validate_bool "${debug}" debug 1
test ${debug} == "true" && set -x
# Include library.
script_dir="$(dirname -- "$(realpath -- "${0}")")"
source "${script_dir}/lib.sh"
@ -10,8 +16,14 @@ cache_dir="${1}"
# Version of the cache to create or load.
version="${2}"
# Execute post-installation script.
execute_install_scripts="${3}"
# Debug mode for diagnosing issues.
debug="${4}"
# List of the packages to use.
input_packages="${@:3}"
input_packages="${@:5}"
# Trim commas, excess spaces, and sort.
packages="$(normalize_package_list "${input_packages}")"
@ -23,16 +35,18 @@ log "Validating action arguments (version='${version}', packages='${packages}').
if grep -q " " <<< "${version}"; then
log "aborted"
log "Version value '${version}' cannot contain spaces." >&2
exit 1
exit 2
fi
# Is length of string zero?
if test -z "${packages}"; then
log "aborted"
log "Packages argument cannot be empty." >&2
exit 2
exit 3
fi
validate_bool "${execute_install_scripts}" execute_install_scripts 4
log "done"
log_empty_line
@ -43,7 +57,7 @@ for package in ${packages}; do
if test ! "$(apt-cache show "${package}")"; then
echo "aborted"
log "Package '${package}' not found." >&2
exit 3
exit 5
fi
read package_name package_ver < <(get_package_name_ver "${package}")
versioned_packages=""${versioned_packages}" "${package_name}"="${package_ver}""

21
restore_pkgs.sh
View file

@ -3,6 +3,11 @@
# Fail on any error.
set -e
# Debug mode for diagnosing issues.
# Setup first before other operations.
debug="${4}"
test ${debug} == "true" && set -x
# Include library.
script_dir="$(dirname -- "$(realpath -- "${0}")")"
source "${script_dir}/lib.sh"
@ -13,6 +18,10 @@ cache_dir="${1}"
# Root directory to untar the cached packages to.
# Typically filesystem root '/' but can be changed for testing.
cache_restore_root="${2}"
test -d ${cache_restore_root} || mkdir ${cache_restore_root}
# Cache and execute post install scripts on restore.
execute_install_scripts="${3}"
cache_filepaths="$(ls -1 "${cache_dir}" | sort)"
log "Found $(echo ${cache_filepaths} | wc -w) files in the cache."
@ -31,12 +40,22 @@ log "done"
log_empty_line
# Only search for archived results. Manifest and cache key also live here.
cached_pkg_filepaths=$(ls -1 "${cache_dir}"/*.tar.gz | sort)
cached_pkg_filepaths=$(ls -1 "${cache_dir}"/*.tar | sort)
cached_pkg_filecount=$(echo ${cached_pkg_filepaths} | wc -w)
log "Restoring ${cached_pkg_filecount} packages from cache..."
for cached_pkg_filepath in ${cached_pkg_filepaths}; do
log "- $(basename "${cached_pkg_filepath}") restoring..."
sudo tar -xf "${cached_pkg_filepath}" -C "${cache_restore_root}" > /dev/null
log " done"
# Execute install scripts if available.
if test ${execute_install_scripts} == "true"; then
# May have to add more handling for extracting pre-install script before extracting all files.
# Keeping it simple for now.
execute_install_script "${cache_restore_root}" "${cached_pkg_filepath}" preinst install
execute_install_script "${cache_restore_root}" "${cached_pkg_filepath}" postinst configure
fi
done
log "done"