out of root disk space after 24.4 upgrade

after completing the 24.4 upgrade, I found the root / partition down to 10% free space which triggers /etc/cron.hourly/filr-diskcheck.sh to shutdown the filr service every hour.

the big consumers of space I found on / were /.snapshots and /usr/src.  I tried various snapper cleanup commands but nothing ever changed so I used "snapper rm" to manually delete a bunch of the older snapshots to free up about 10GB.

running "du -d 1 -h" in /usr/src I have:

23M ./filr
3.5M ./linux-5.14.21-150400.22-obj
1.2G ./linux-5.14.21-150400.22
4.0K ./linux-obj
0 ./packages
1.2G ./linux-5.14.21-150400.24.38
3.6M ./linux-5.14.21-150400.24.38-obj
1.2G ./linux-5.14.21-150400.24.46
3.6M ./linux-5.14.21-150400.24.46-obj
1.2G ./linux-5.14.21-150400.24.63
3.6M ./linux-5.14.21-150400.24.63-obj
1.2G ./linux-5.14.21-150400.24.66
3.6M ./linux-5.14.21-150400.24.66-obj
1.2G ./linux-5.14.21-150400.24.81
3.6M ./linux-5.14.21-150400.24.81-obj
1.2G ./linux-5.14.21-150400.24.100
3.6M ./linux-5.14.21-150400.24.100-obj
1.2G ./linux-5.14.21-150400.24.111
3.6M ./linux-5.14.21-150400.24.111-obj
1.2G ./linux-5.14.21-150400.24.122
3.6M ./linux-5.14.21-150400.24.122-obj
1.2G ./linux-5.14.21-150400.24.128
3.6M ./linux-5.14.21-150400.24.128-obj
12G .

/usr/src/linux is linked to linux-5.14.21.150400.24.128 which is dated the day of the 24.4 update so that's clearly the current one..

is it safe to delete the older ones?

  • 0  

    snapper list-configs
    Config | Subvolume
    -------+----------
    root   | /

    snapper --config root list

    snapper --config root delete # 

    example
    snapper --config root delete 399 - 401

    George

    “You can't teach a person anything, you can only help them to discover it within themselves.” Galileo Galilei

  • 0 in reply to   

    Thanks George, but I already got the .snapshots taken care of; I deleted every snapshot between the oldest pair and Oct 1st of this year. (oddly snapper list doesn't list the snapshot space on my Filr server, so I just deleted them all and ran df after every group - some freed up several GB, others freed up essentially nothing)

    I was asking about the source code directories in /usr/src; I'm guessing any other than the most current one is safe to delete, in fact I'm not sure why a Filr appliance would need a local source code repository at all 

  • 0   in reply to 

    what snapper list shows has to do with the width of the cli window.
    Another question is whether /usr/src is needed. Yes, for the running kernel there is always a symlinc in the corresponding source files. For kernels that have already been purged, the data is actually no longer needed.

    “You can't teach a person anything, you can only help them to discover it within themselves.” Galileo Galilei

  • 0 in reply to   

    I just ran zypper purge-kernels and it cleaned up 8 old kernels and their attendant source and devel packages.  that was 10GB of data, though no more space was actually freed up because all these kernels are still in the snapshots.  at least next time I run low on space I can purge snapshots and get this space back

    one wonders why none of the appliance upgrades ever purges old kernels and cleans up obsolete snapshots... surely I won't be the only one to encounter this after a few years of upgrades

  • 0   in reply to 

    My generation was one that thought carefully about what to do. 128 KB of memory on the board, a 5 1/4 inch floppy disk only had 1.2 MB of memory. That's why I can't understand why obsolete kernels, source files or whatever remain in the system when updating or patching appliances. This leads to bloated backups and, if necessary, to long-lasting restores. That's why I often take a broom to the customer's premises and in our systems to sweep out and clean up. George

    “You can't teach a person anything, you can only help them to discover it within themselves.” Galileo Galilei

  • 0  

    Now I have to check all my customers I have access to.

    I have checked my server. "/" is filled up to 94%.  Directory .snapshots is eating all the space.

    In my eyes this is a bug - I will open a case.


    Use "Verified Answers" if your problem/issue has been solved!

  • 0 in reply to   

    You must have changed or eliminated your /etc/cron.hourly/filr-diskcheck.sh which defaults to shutting down the filr services every hour if the / disk space falls below 10 percent.  I changed mine to 5% while troubleshooting this to stop the shutdowns.

    I think they implemented BTRFS for the / volume without considering what the default .snapshot settings would do.  Because it keeps every zypper snapshot and years of timeline snapshots, filling the volume is inevitable (or at least filling it more than 90% is inevitable... I'm not sure what the cleanup settings are, but I couldn't get snapper cleanup to do anything no matter what I told it; it would either do nothing or say it couldn't give me the space I asked for, yet when I deleted a few snapshots manually I easily freed more space than what I asked cleanup to give me).

    This is probably another consequence of there being no Appliance team any more, the product teams all have to manage their own appliances and things are getting missed.

  • 0   in reply to 

    A case is open ...


    Use "Verified Answers" if your problem/issue has been solved!

  • 0 in reply to   

    let me know if your snapper list command shows the space used by each snapshot?  mine doesn't, and no it's not due to my terminal width, I run my terminals as wide as the mississippi river.  Yet as George's screenshot shows, he gets space used for each snapshot

  • 0 in reply to   

    it appears Filr is the only appliance with snapshots enabled, at least of the ones I have installed (Filr, Zenworks, SMG, Service Desk); the others all say:

    -bash: snapper: command not found