Less space @FiLR-VA

I see less space at a FiLR-VA related to the FiLR partition ifself (not vastorage)

I found a lot of really old kernels and telemetry log in /opt/

WFT ...

1. It seems that you not use -> zypper to update the VA. So tell your own piece of code how to handle old kernels.

Or call zypper to update the system and let them clean up the old kernels.

2. Logfiles should be placed in /var/log (or /var/opt/ if it should be separated from system logs. In your case like: /var/opt/novell/filr/logs/telemetry/) and deeper (https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)

Nobody assumed that there are volatile files in /opt ...

... Ah OK, I found you has changed this meanwhile ... perfect! One step more if you had moved the old telemetry files from /opt into the new place or delete the old logs. (And another point if you create a entry in /etc/logrotate.d/ for this files.)

Tags:

  • Verified Answer

    +1  

       Filr, and all of these appliances do use zypper for updates.  usually via command similar to: zypper --non-interactive --quiet patch --with-interactive --no-recommends

    I think we may need to look at the /etc/zypp/zypp.config for the entry handling multiversion kernel.  Should only allow for 3 by default and auto clean the old ones if memory serves.  This is a default setting on SLES 15 based systems.  May be something wrong with it though as

    multiversion.kernels = latest,latest-1,running

    This is supposed to be handling the auto cleanup, but something seems amiss.  I noticed I have 10 kernels installed on my 24.4 filr which is not right.

    Update:  The issue is it appears all the Filr related appliances are missing the purge-kernels service.  This is the service that runs the zypper purge command to cleanup old kernels when new ones are installed.
    I've opened a case and notified Filr team that every Filr 5+ appliance they built is affected by this service missing.

    Rodney

    If you found this post useful, give it a "Like" or click on "Verify Answer" under the "More" button.   This helps others.

  • 0  

    Here's a script that will search for kernel-default*, grab the latest 3 to keep, then list the older ones to remove.  It will ask before removing the list.

    #!/bin/bash
    
    # Function to get all related kernel packages
    get_kernel_packages() {
        local kernel_version="$1"
        rpm -qa | grep -E "kernel-default.*${kernel_version##kernel-default-}"
    }
    
    # Get list of installed kernels, sorted
    KERNELS=$(rpm -q kernel-default | sort -V)
    
    # Count total kernels
    KERNEL_COUNT=$(echo "$KERNELS" | wc -l)
    
    # Keep only the last 3 kernels
    if [ $KERNEL_COUNT -gt 3 ]; then
        # Calculate how many to remove
        REMOVE_COUNT=$((KERNEL_COUNT - 3))
        
        # Identify kernels to keep (last 3)
        KERNELS_TO_KEEP=$(echo "$KERNELS" | tail -n 3)
        
        # Collect kernels to remove
        KERNELS_TO_REMOVE=$(echo "$KERNELS" | head -n $REMOVE_COUNT)
        
        # Collect all related packages to remove
        PACKAGES_TO_REMOVE=()
        
        # Display information and collect packages
        echo "Kernel Cleanup Report:"
        echo "Total installed kernels: $KERNEL_COUNT"
        echo "Kernels to keep:"
        echo "$KERNELS_TO_KEEP"
        echo ""
        echo "Kernels to remove ($REMOVE_COUNT):"
        
        # Iterate through kernels to remove
        for kernel in $KERNELS_TO_REMOVE; do
            echo "Processing kernel: $kernel"
            
            # Find all related packages
            related_packages=$(get_kernel_packages "$kernel")
            
            echo "Related packages for $kernel:"
            echo "$related_packages"
            
            # Add to removal list
            PACKAGES_TO_REMOVE+=($related_packages)
        done
        
        echo ""
        echo "Total packages to remove:"
        printf '%s\n' "${PACKAGES_TO_REMOVE[@]}"
        
        # Prompt for confirmation
        read -p "Do you want to proceed with removing these packages? (y/N) " -n 1 -r
        echo    # move to a new line
        if [[ $REPLY =~ ^[Yy]$ ]]
        then
            # Bulk remove packages in a single zypper command
            zypper --non-interactive remove $(printf '%s\n' "${PACKAGES_TO_REMOVE[@]}" | tr '\n' ' ')
            echo "Kernel and related package cleanup completed."
        else
            echo "Kernel removal cancelled."
        fi
    else
        echo "Not enough kernels to remove. Keeping all installed kernels:"
        echo "$KERNELS"
    fi

    Here's example output on my Filr server

    ./cleanup-kernels.sh
    Kernel Cleanup Report:
    Total installed kernels: 10
    Kernels to keep:
    kernel-default-5.14.21-150400.24.111.2.x86_64
    kernel-default-5.14.21-150400.24.122.2.x86_64
    kernel-default-5.14.21-150400.24.128.1.x86_64
    
    Kernels to remove (7):
    Processing kernel: kernel-default-5.14.21-150400.22.1.x86_64
    Related packages for kernel-default-5.14.21-150400.22.1.x86_64:
    kernel-default-5.14.21-150400.22.1.x86_64
    kernel-default-devel-5.14.21-150400.22.1.x86_64
    Processing kernel: kernel-default-5.14.21-150400.24.38.1.x86_64
    Related packages for kernel-default-5.14.21-150400.24.38.1.x86_64:
    kernel-default-5.14.21-150400.24.38.1.x86_64
    kernel-default-devel-5.14.21-150400.24.38.1.x86_64
    Processing kernel: kernel-default-5.14.21-150400.24.46.1.x86_64
    Related packages for kernel-default-5.14.21-150400.24.46.1.x86_64:
    kernel-default-5.14.21-150400.24.46.1.x86_64
    kernel-default-devel-5.14.21-150400.24.46.1.x86_64
    Processing kernel: kernel-default-5.14.21-150400.24.63.1.x86_64
    Related packages for kernel-default-5.14.21-150400.24.63.1.x86_64:
    kernel-default-devel-5.14.21-150400.24.63.1.x86_64
    kernel-default-5.14.21-150400.24.63.1.x86_64
    Processing kernel: kernel-default-5.14.21-150400.24.66.1.x86_64
    Related packages for kernel-default-5.14.21-150400.24.66.1.x86_64:
    kernel-default-5.14.21-150400.24.66.1.x86_64
    kernel-default-devel-5.14.21-150400.24.66.1.x86_64
    Processing kernel: kernel-default-5.14.21-150400.24.81.1.x86_64
    Related packages for kernel-default-5.14.21-150400.24.81.1.x86_64:
    kernel-default-devel-5.14.21-150400.24.81.1.x86_64
    kernel-default-5.14.21-150400.24.81.1.x86_64
    Processing kernel: kernel-default-5.14.21-150400.24.100.2.x86_64
    Related packages for kernel-default-5.14.21-150400.24.100.2.x86_64:
    kernel-default-5.14.21-150400.24.100.2.x86_64
    kernel-default-devel-5.14.21-150400.24.100.2.x86_64
    
    Total packages to remove:
    kernel-default-5.14.21-150400.22.1.x86_64
    kernel-default-devel-5.14.21-150400.22.1.x86_64
    kernel-default-5.14.21-150400.24.38.1.x86_64
    kernel-default-devel-5.14.21-150400.24.38.1.x86_64
    kernel-default-5.14.21-150400.24.46.1.x86_64
    kernel-default-devel-5.14.21-150400.24.46.1.x86_64
    kernel-default-devel-5.14.21-150400.24.63.1.x86_64
    kernel-default-5.14.21-150400.24.63.1.x86_64
    kernel-default-5.14.21-150400.24.66.1.x86_64
    kernel-default-devel-5.14.21-150400.24.66.1.x86_64
    kernel-default-devel-5.14.21-150400.24.81.1.x86_64
    kernel-default-5.14.21-150400.24.81.1.x86_64
    kernel-default-5.14.21-150400.24.100.2.x86_64
    kernel-default-devel-5.14.21-150400.24.100.2.x86_64
    Do you want to proceed with removing these packages? (y/N)

    Rodney

    If you found this post useful, give it a "Like" or click on "Verify Answer" under the "More" button.   This helps others.

  • 0  

    Supported pointed me at this thread after I started seeing disk space issues on one of my Filr appliances.

    In my case, I found the disk space being eaten up by snapper snapshots.  We're running 24.4.1.  I don't know what patch caused or changed this, but my Filr appliances are doing timeline snapshots now every hour on the hour.  I ended up removing all the snapshots and used this command to disable timeline snapshots:

    snapper --config root set-config "TIMELINE_CREATE=no"

    Matt

  • 0 in reply to   

    did you see this thread?  

     out of root disk space after 24.4 upgrade 

    I just ran "zypper purge-kernels" and it got rid of all the extra kernel versions.  But it freed no actual space because the btrfs snapshots are enabled too.  As far as I can see, every Filr 5 appliance is time-bombed to run out of space using default settings.

    I hope the Filr team is working on the entire issue, not just the kernel purging as that's only the tip of the iceberg.

  • 0 in reply to   

    this is a battle I just fought a month or so ago, see this thread:

     out of root disk space after 24.4 upgrade 

    It looks like you've gotten to the same fixes I did.  It's still not truly fixed though, it'll still create zypper snapshots during every upgrade, and it still isn't going to purge old kernels on its own. 

  • 0   in reply to 

    No, I didn't see that thread, but I just read it and see I'm not alone. I did not know about 10% free disk space and the cron job that shuts down Filr! I couldn't figure out what the heck was shutting down Filr for me, but that was it! Support didn't mention that one to me.

    I went ahead and ran the cleanup kernels script listed here.  It removed 7 kernels, but oddly, freed absolutely no disk space.

    This is pretty ridiculous that we're having to deal with this on appliances.  I know I've been actively deleting snapper snapshots on Filr appliances for a long time.  Wouldn't take much to adjust Btrfs installation settings to disable them.  As someone mentioned in another thread, every Filr installation is time bombed to fail eventually!

    Matt

  • 0 in reply to   

    Yeah, no space is initially freed by purging kernels, as they're all still in the zypper snapshot(s).  I ended up deleting all snapshots to get all my space back.  the zypper snapshots can't as readily be turned off, at least not in a way that won't come back after an update; somewhere in that thread I believe I linked to an article about turning off snapshots, but based on what it said about the zypper snapshots I didn't bother.

    And yeah, I'm the one who made the time-bomb comment (though perhaps I'm not the only one)

    They really need to straighten this out.

  • 0   in reply to   

    Development has confirmed 25.1 will bring the proper purge-kernels service that's missing in the appliances according to the case I opened on this last month.  it's not a massive amount of data cleaned up, but is a step towards better appliance snapshot and OS data management.

    We really need to document better best-practices for btrfs maintenance and management on the appliances.   A lot of the various appliance issues across the various Novell/Microfocus/NetIQ/OpenText products come from software devs being asked to build appliances, which is something out of their wheelhouse.  If we help them, they can set better defaults than the out-of-box sles settings.

    There is talk about next version of fresh appliance not using btrfs by default for root, but I don't think that's the correct direction to go into.  Proper defaults, or better documented best practices I think is a more sane approach providing a balance between the recovery safety built into btrfs snapshots and storage management. These are after all VM appliances so assumption is admins are taking frequent snapshots of them.  Main reason for btrfs snapshots is when admin is using the appliance scheduled auto updates and something goes wrong outside of usual hypervisor based snapshot, so we really don't need comprehensive long term rollback built into the appliances, just few most recent snaps triggered by updates. They may also need to expand the default root os partition size to better handle in-appliance btrfs snapshots.  Snapper policy is easily modified to facilitate this. 

    I just looked and am also seeing that snapper timelines were somehow enabled recently.  This is set in /etc/snapper/configs/root
    TIMELINE_CREATE="no"
    I'm adding this to my case for them to fix.  This should never have been turned on.  Good catch.  Once those are cleaned, the services and timers can also be disabled as unnecessary overhead.

    While we're there, I think a more reasonable setting for snapshot retention after updates vs the defaults of 15 and 10:

    # Limit number of snapshots
    NUMBER_LIMIT="5"
    NUMBER_LIMIT_IMPORTANT="3"

    I've been saying for awhile that OT should have 1 or multiple appliance devs to help standardize the appliances across product suites and departments.

    This is likely why SMG and a few others recently dumped appliances and moved to just providing rpm installations on supported platforms.  Too much support and dev resources put towards appliance maintenance taking away from their actual product's software development.

    Rodney

    If you found this post useful, give it a "Like" or click on "Verify Answer" under the "More" button.   This helps others.

  • 0   in reply to   

    Here's a VERY quick and dirty command to cleanup the timeline snapshots, but I don't know what repercussions this may have on older versions or overlap with zypper generated snaps.

    Use at own risk.  Won't break a running system, but may break snapshot chain

    snapper list | awk '/timeline/ {print $1}' | xargs -I {} snapper delete {}

    Rodney

    If you found this post useful, give it a "Like" or click on "Verify Answer" under the "More" button.   This helps others.

  • 0   in reply to   

    Thank you all for the feedback and information.  I'll pass this along to the Filr engineering team.