Add tool to compare built ipa and master ipa initramfs #1058

peppi-lotta · 2025-07-29T12:06:56Z

This PR introduces a script ipa_debug_tools.sh that provides a set of utilities to assist with debugging and comparing IPA (ironic-python-agent) initramfs builds. The script is particularly useful for identifying differences between locally built IPA images and the official master images.

This tool is meant especially for situations where built IPA size get over a certain limit limit and two file systems need to be compared.

🛠️ What's Included

set_up_debug_dirs
- Sets up a local debug environment under /tmp/debug-initramfs
- Builds the local IPA using jenkins/scripts/dynamic_worker_workflow/build_ipa.sh
- Downloads and extracts the master IPA image from upstream
compare_dir_sizes [path]
- Compares sizes of files/directories between the local build and master IPA
- Provides a colored, sorted table of size differences
- Highlights large discrepancies (10MB+, 1MB+, etc.)
- Lists files exclusive to either initramfs tree

compare_rpm_packages
- Extracts and compares installed RPMs between local and master IPA environments
- Outputs side-by-side view of mismatched or missing packages
- Helps identify unintended package changes

clean_up_debug_dirs
- Deletes all temporary directories and files created during debugging

metal3-io-bot · 2025-07-29T12:07:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tuminoid for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: peppi-lotta <peppi-lotta.saari@est.tech>

Rozzii

In general LGTM,
It is a very useful tool as I have mentioned offline.
I have one improvement proposal but it can be a later addition if you don't feel like implementing it now.

Rozzii · 2025-07-31T06:15:14Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


It would be nice if the upstream IPA URI and tar file name could be customized same as in IPA downloader, you could actually use IPA downloader here instead of your own download logic to keep it simple .

For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

I am fine either way, but the url and the name of the tar file has to be customizable because otherwise only a specific master on a specific base image can be tested. I suggested to use the downloader because that has all the customization options implemented but I am fine with doing it with the one liners too.

Rozzii · 2025-07-31T06:22:40Z

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

metal3-io-bot · 2025-07-31T06:22:45Z

@Rozzii: Overrode contexts on behalf of Rozzii: metal3-ubuntu-e2e-integration-test-main

In response to this:

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tuminoid

This is great addition to debug tools, thanks @peppi-lotta.

Some cleanup to do, some minor syntax things to fix and some suggestions to make it more readable and maintainable.

tuminoid · 2025-08-05T06:11:33Z

hack/ipa_debug_tools.sh

@@ -0,0 +1,227 @@
+#!/usr/bin/env bash
+


We should be running with set -eu by default.

tuminoid · 2025-08-05T06:12:15Z

hack/ipa_debug_tools.sh

+    export TEST_IN_CI="false"
+
+    # Build ipa with build_ipa.sh
+    cd "$CURRENT_SCRIPT_DIR/../jenkins/scripts/dynamic_worker_workflow" || exit


Suggested change

cd "$CURRENT_SCRIPT_DIR/../jenkins/scripts/dynamic_worker_workflow" || exit

cd "${CURRENT_SCRIPT_DIR}/../jenkins/scripts/dynamic_worker_workflow"

|| exit 1 not needed if we're running set -e mentioned earlier, cleans up the code.

tuminoid · 2025-08-05T06:12:40Z

hack/ipa_debug_tools.sh

+    # Unzip ironic-python-agent.initramfs
+    sudo mkdir -p /tmp/debug-initramfs/build-ipa-initramfs
+    sudo cp /tmp/dib/ironic-python-agent.initramfs /tmp/debug-initramfs/
+    cd /tmp/debug-initramfs/build-ipa-initramfs || exit


Suggested change

cd /tmp/debug-initramfs/build-ipa-initramfs || exit

cd /tmp/debug-initramfs/build-ipa-initramfs

tuminoid · 2025-08-05T06:19:00Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

tuminoid · 2025-08-05T06:19:09Z

hack/ipa_debug_tools.sh

+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz
+    cd master-ipa-initramfs || exit


Suggested change

cd master-ipa-initramfs || exit

cd master-ipa-initramfs

tuminoid · 2025-08-05T06:31:58Z

hack/ipa_debug_tools.sh

+#the built ipa and master.
+# Check if packages are a part of CentOS Stream in https://pkgs.org
+compare_rpm_packages() {
+    basedir1="${1:-/tmp/debug-initramfs/build-ipa-initramfs}" 


Same as above for descriptive dir and files names.

tuminoid · 2025-08-05T06:32:19Z

hack/ipa_debug_tools.sh

+    common_count=$(comm -12 "${compfile1}" "${compfile2}" | wc -l)
+    echo -e "\nNumber of common packages: ${common_count}\n"
+    # Extract base package names and join for fuzzy matching
+    awk -F'-[0-9]' '{print $1}' "${compfile1}" | sort > /tmp/build-ipa-base.txt


Same suggestion for tmpfile names.

tuminoid · 2025-08-05T06:32:49Z

hack/ipa_debug_tools.sh

+        lfull=$(grep -m1 "^${left}-" "${compfile1}" || echo "")
+        rfull=$(grep -m1 "^${right}-" "${compfile2}" || echo "")
+        # Skip printing if both lines are non-empty and exactly the same
+        if [[ -n "${lfull}" && "${lfull}" == "${rfull}" ]]; then


Suggested change

if [[ -n "${lfull}" && "${lfull}" == "${rfull}" ]]; then

if [[ -n "${lfull}" ]] && [[ "${lfull}" == "${rfull}" ]]; then

tuminoid · 2025-08-05T06:34:02Z

hack/ipa_debug_tools.sh

+function_name="$1"
+shift
+
+# Check if function exists and call it


Why not use standard argument parsing here, much more readable and reliable?

tuminoid · 2025-08-05T06:34:39Z

hack/ipa_debug_tools.sh

+# Usage:
+#   ./ipa_debug_tools.sh clean_up_debug_dirs
+# -----------------------------------------------------------------------------
+clean_up_debug_dirs() {


Cleaning up all the temp files should be done here, and the cleanup should be trapped function so it gets called even if we exit mid run.

metal3-io-bot requested review from Rozzii and smoshiur1237 July 29, 2025 12:07

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 29, 2025

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 1a30ce3 to 4c2b687 Compare July 30, 2025 06:07

Add tool to compare built ipa and master ipa initramfs

9e9d4b4

Signed-off-by: peppi-lotta <peppi-lotta.saari@est.tech>

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 4c2b687 to 9e9d4b4 Compare July 30, 2025 06:15

Rozzii reviewed Jul 31, 2025

View reviewed changes

Rozzii added this to Metal3 - Roadmap Jul 31, 2025

Rozzii moved this to MISC WIP in Metal3 - Roadmap Jul 31, 2025

tuminoid reviewed Aug 5, 2025

View reviewed changes

	cd "$CURRENT_SCRIPT_DIR/../jenkins/scripts/dynamic_worker_workflow" \|\| exit
	cd "${CURRENT_SCRIPT_DIR}/../jenkins/scripts/dynamic_worker_workflow"

	cd /tmp/debug-initramfs/build-ipa-initramfs \|\| exit
	cd /tmp/debug-initramfs/build-ipa-initramfs

	if [[ -n "${lfull}" && "${lfull}" == "${rfull}" ]]; then
	if [[ -n "${lfull}" ]] && [[ "${lfull}" == "${rfull}" ]]; then

Add tool to compare built ipa and master ipa initramfs #1058

Are you sure you want to change the base?

Add tool to compare built ipa and master ipa initramfs #1058

Uh oh!

Conversation

peppi-lotta commented Jul 29, 2025

🛠️ What's Included

Uh oh!

metal3-io-bot commented Jul 29, 2025

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

Rozzii Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rozzii commented Jul 31, 2025

Uh oh!

metal3-io-bot commented Jul 31, 2025

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Rozzii Jul 31, 2025 •

edited

Loading