BM2EC  Filr, and all of these appliances do use zypper for updates.  usually via command similar to: zypper --non-interactive --quiet patch --with-interactive --no-recommends

I think we may need to look at the /etc/zypp/zypp.config for the entry handling multiversion kernel.  Should only allow for 3 by default and auto clean the old ones if memory serves.  This is a default setting on SLES 15 based systems.  May be something wrong with it though as

multiversion.kernels = latest,latest-1,running

This is supposed to be handling the auto cleanup, but something seems amiss.  I noticed I have 10 kernels installed on my 24.4 filr which is not right.

Update:  The issue is it appears all the Filr related appliances are missing the purge-kernels service.  This is the service that runs the zypper purge command to cleanup old kernels when new ones are installed.
I've opened a case and notified Filr team that every Filr 5+ appliance they built is affected by this service missing.

Here's a script that will search for kernel-default*, grab the latest 3 to keep, then list the older ones to remove.  It will ask before removing the list.

#!/bin/bash

# Function to get all related kernel packages
get_kernel_packages() {
    local kernel_version="$1"
    rpm -qa | grep -E "kernel-default.*${kernel_version##kernel-default-}"
}

# Get list of installed kernels, sorted
KERNELS=$(rpm -q kernel-default | sort -V)

# Count total kernels
KERNEL_COUNT=$(echo "$KERNELS" | wc -l)

# Keep only the last 3 kernels
if [ $KERNEL_COUNT -gt 3 ]; then
    # Calculate how many to remove
    REMOVE_COUNT=$((KERNEL_COUNT - 3))
    
    # Identify kernels to keep (last 3)
    KERNELS_TO_KEEP=$(echo "$KERNELS" | tail -n 3)
    
    # Collect kernels to remove
    KERNELS_TO_REMOVE=$(echo "$KERNELS" | head -n $REMOVE_COUNT)
    
    # Collect all related packages to remove
    PACKAGES_TO_REMOVE=()
    
    # Display information and collect packages
    echo "Kernel Cleanup Report:"
    echo "Total installed kernels: $KERNEL_COUNT"
    echo "Kernels to keep:"
    echo "$KERNELS_TO_KEEP"
    echo ""
    echo "Kernels to remove ($REMOVE_COUNT):"
    
    # Iterate through kernels to remove
    for kernel in $KERNELS_TO_REMOVE; do
        echo "Processing kernel: $kernel"
        
        # Find all related packages
        related_packages=$(get_kernel_packages "$kernel")
        
        echo "Related packages for $kernel:"
        echo "$related_packages"
        
        # Add to removal list
        PACKAGES_TO_REMOVE+=($related_packages)
    done
    
    echo ""
    echo "Total packages to remove:"
    printf '%s\n' "${PACKAGES_TO_REMOVE[@]}"
    
    # Prompt for confirmation
    read -p "Do you want to proceed with removing these packages? (y/N) " -n 1 -r
    echo    # move to a new line
    if [[ $REPLY =~ ^[Yy]$ ]]
    then
        # Bulk remove packages in a single zypper command
        zypper --non-interactive remove $(printf '%s\n' "${PACKAGES_TO_REMOVE[@]}" | tr '\n' ' ')
        echo "Kernel and related package cleanup completed."
    else
        echo "Kernel removal cancelled."
    fi
else
    echo "Not enough kernels to remove. Keeping all installed kernels:"
    echo "$KERNELS"
fi

Here's example output on my Filr server

./cleanup-kernels.sh
Kernel Cleanup Report:
Total installed kernels: 10
Kernels to keep:
kernel-default-5.14.21-150400.24.111.2.x86_64
kernel-default-5.14.21-150400.24.122.2.x86_64
kernel-default-5.14.21-150400.24.128.1.x86_64

Kernels to remove (7):
Processing kernel: kernel-default-5.14.21-150400.22.1.x86_64
Related packages for kernel-default-5.14.21-150400.22.1.x86_64:
kernel-default-5.14.21-150400.22.1.x86_64
kernel-default-devel-5.14.21-150400.22.1.x86_64
Processing kernel: kernel-default-5.14.21-150400.24.38.1.x86_64
Related packages for kernel-default-5.14.21-150400.24.38.1.x86_64:
kernel-default-5.14.21-150400.24.38.1.x86_64
kernel-default-devel-5.14.21-150400.24.38.1.x86_64
Processing kernel: kernel-default-5.14.21-150400.24.46.1.x86_64
Related packages for kernel-default-5.14.21-150400.24.46.1.x86_64:
kernel-default-5.14.21-150400.24.46.1.x86_64
kernel-default-devel-5.14.21-150400.24.46.1.x86_64
Processing kernel: kernel-default-5.14.21-150400.24.63.1.x86_64
Related packages for kernel-default-5.14.21-150400.24.63.1.x86_64:
kernel-default-devel-5.14.21-150400.24.63.1.x86_64
kernel-default-5.14.21-150400.24.63.1.x86_64
Processing kernel: kernel-default-5.14.21-150400.24.66.1.x86_64
Related packages for kernel-default-5.14.21-150400.24.66.1.x86_64:
kernel-default-5.14.21-150400.24.66.1.x86_64
kernel-default-devel-5.14.21-150400.24.66.1.x86_64
Processing kernel: kernel-default-5.14.21-150400.24.81.1.x86_64
Related packages for kernel-default-5.14.21-150400.24.81.1.x86_64:
kernel-default-devel-5.14.21-150400.24.81.1.x86_64
kernel-default-5.14.21-150400.24.81.1.x86_64
Processing kernel: kernel-default-5.14.21-150400.24.100.2.x86_64
Related packages for kernel-default-5.14.21-150400.24.100.2.x86_64:
kernel-default-5.14.21-150400.24.100.2.x86_64
kernel-default-devel-5.14.21-150400.24.100.2.x86_64

Total packages to remove:
kernel-default-5.14.21-150400.22.1.x86_64
kernel-default-devel-5.14.21-150400.22.1.x86_64
kernel-default-5.14.21-150400.24.38.1.x86_64
kernel-default-devel-5.14.21-150400.24.38.1.x86_64
kernel-default-5.14.21-150400.24.46.1.x86_64
kernel-default-devel-5.14.21-150400.24.46.1.x86_64
kernel-default-devel-5.14.21-150400.24.63.1.x86_64
kernel-default-5.14.21-150400.24.63.1.x86_64
kernel-default-5.14.21-150400.24.66.1.x86_64
kernel-default-devel-5.14.21-150400.24.66.1.x86_64
kernel-default-devel-5.14.21-150400.24.81.1.x86_64
kernel-default-5.14.21-150400.24.81.1.x86_64
kernel-default-5.14.21-150400.24.100.2.x86_64
kernel-default-devel-5.14.21-150400.24.100.2.x86_64
Do you want to proceed with removing these packages? (y/N)

Supported pointed me at this thread after I started seeing disk space issues on one of my Filr appliances.

In my case, I found the disk space being eaten up by snapper snapshots.  We're running 24.4.1.  I don't know what patch caused or changed this, but my Filr appliances are doing timeline snapshots now every hour on the hour.  I ended up removing all the snapshots and used this command to disable timeline snapshots:

snapper --config root set-config "TIMELINE_CREATE=no"

Matt

did you see this thread?  

 out of root disk space after 24.4 upgrade 

I just ran "zypper purge-kernels" and it got rid of all the extra kernel versions.  But it freed no actual space because the btrfs snapshots are enabled too.  As far as I can see, every Filr 5 appliance is time-bombed to run out of space using default settings.

I hope the Filr team is working on the entire issue, not just the kernel purging as that's only the tip of the iceberg.

this is a battle I just fought a month or so ago, see this thread:

 out of root disk space after 24.4 upgrade 

It looks like you've gotten to the same fixes I did.  It's still not truly fixed though, it'll still create zypper snapshots during every upgrade, and it still isn't going to purge old kernels on its own. 

No, I didn't see that thread, but I just read it and see I'm not alone. I did not know about 10% free disk space and the cron job that shuts down Filr! I couldn't figure out what the heck was shutting down Filr for me, but that was it! Support didn't mention that one to me.

I went ahead and ran the cleanup kernels script listed here.  It removed 7 kernels, but oddly, freed absolutely no disk space.

This is pretty ridiculous that we're having to deal with this on appliances.  I know I've been actively deleting snapper snapshots on Filr appliances for a long time.  Wouldn't take much to adjust Btrfs installation settings to disable them.  As someone mentioned in another thread, every Filr installation is time bombed to fail eventually!

Matt

Yeah, no space is initially freed by purging kernels, as they're all still in the zypper snapshot(s).  I ended up deleting all snapshots to get all my space back.  the zypper snapshots can't as readily be turned off, at least not in a way that won't come back after an update; somewhere in that thread I believe I linked to an article about turning off snapshots, but based on what it said about the zypper snapshots I didn't bother.

And yeah, I'm the one who made the time-bomb comment (though perhaps I'm not the only one)

They really need to straighten this out.

Development has confirmed 25.1 will bring the proper purge-kernels service that's missing in the appliances according to the case I opened on this last month.  it's not a massive amount of data cleaned up, but is a step towards better appliance snapshot and OS data management.

We really need to document better best-practices for btrfs maintenance and management on the appliances.   A lot of the various appliance issues across the various Novell/Microfocus/NetIQ/OpenText products come from software devs being asked to build appliances, which is something out of their wheelhouse.  If we help them, they can set better defaults than the out-of-box sles settings.

There is talk about next version of fresh appliance not using btrfs by default for root, but I don't think that's the correct direction to go into.  Proper defaults, or better documented best practices I think is a more sane approach providing a balance between the recovery safety built into btrfs snapshots and storage management. These are after all VM appliances so assumption is admins are taking frequent snapshots of them.  Main reason for btrfs snapshots is when admin is using the appliance scheduled auto updates and something goes wrong outside of usual hypervisor based snapshot, so we really don't need comprehensive long term rollback built into the appliances, just few most recent snaps triggered by updates. They may also need to expand the default root os partition size to better handle in-appliance btrfs snapshots.  Snapper policy is easily modified to facilitate this. 

I just looked and am also seeing that snapper timelines were somehow enabled recently.  This is set in /etc/snapper/configs/root
TIMELINE_CREATE="no"
I'm adding this to my case for them to fix.  This should never have been turned on.  Good catch.  Once those are cleaned, the services and timers can also be disabled as unnecessary overhead.

While we're there, I think a more reasonable setting for snapshot retention after updates vs the defaults of 15 and 10:

# Limit number of snapshots
NUMBER_LIMIT="5"
NUMBER_LIMIT_IMPORTANT="3"

I've been saying for awhile that OT should have 1 or multiple appliance devs to help standardize the appliances across product suites and departments.

This is likely why SMG and a few others recently dumped appliances and moved to just providing rpm installations on supported platforms.  Too much support and dev resources put towards appliance maintenance taking away from their actual product's software development.

Here's a VERY quick and dirty command to cleanup the timeline snapshots, but I don't know what repercussions this may have on older versions or overlap with zypper generated snaps.

Use at own risk.  Won't break a running system, but may break snapshot chain

snapper list | awk '/timeline/ {print $1}' | xargs -I {} snapper delete {}

Thank you all for the feedback and information.  I'll pass this along to the Filr engineering team.