This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Apache not loading after in-place OS Upgrade

I have an Access Manager 5.0.4 Gateway SERVICE running on RedHat 7.9.  We tried upgrading it in-place to RHEL 8.8.  After the upgrade, the Apache2 module won't load, get an SSL error:

[ssl:emerg] [pid 12945:tid 140493271382016] AH02564: Failed to configure encrypted (?) private key test:443:0>
[ssl:emerg] [pid 12945:tid 140493271382016] SSL Library Error: error:140C1041:SSL routines:SSL_SET_PKEY:malloc failure

From what I can tell, it is trying to use the default system libssl and libcrypto in /usr/lib64 instead of what gets installed in /opt/novell/ssllib.  In this case, it is trying to use 1.1.1k which fails (segfault).  I don't have another RHEL 8.8 box to look at, but looking at other working SLES 15 servers and another RHEL 7.9 server, it appears apache always loads libssl and libcrypto from /opt/novell/ssllib, not from the system library directory. So something in the upgrade messed that up, but I cannot figure out what.  

Any ideas?

FWIW, in-place upgrades of the admin console and identity server from RHEL 7.9 to RHEL 8.8 worked fine.

Matt

  • 0

    # objdump -p /opt/novell/apache2/sbin/httpd

    ...

      RUNPATH              /opt/novell/pcre/lib:/opt/novell/apache2/lib

    This means shared libraries by httpd and any module loaded by httpd (e.g. mod_ssl) will be searched for under /opt/novell/pcre/lib and /opt/novell/apache2/lib, respectively. Now, as to what shared libraries are being required by mod_ssl.so:

    # objdump -p /opt/novell/apache2/libexec/mod_ssl.so

    ...

    NEEDED libssl.so.1.0.0
    NEEDED libcrypto.so.1.0.0

    Do either of these exist in /usr/lib64 on RHEL 8?

  • 0   in reply to 

    NEEDED libssl.so.1.0.0
    NEEDED libcrypto.so.1.0.0

    Do either of these exist in /usr/lib64 on RHEL 8?

    Not supplied by RHEL, I searched and only found these:

    /opt/netiq/common/openssl/lib64/libssl.so.1.0.0
    /opt/novell/ssllib/libssl.so.1.0.0
    /opt/novell/openssl/lib/libssl.so.1.0.0

    and

    /opt/netiq/common/openssl/lib64/libcrypto.so.1.0.0
    /opt/novell/ssllib/libcrypto.so.1.0.0
    /opt/novell/openssl/lib/libcrypto.so.1.0.0

    Hmm.  In order to test my theory, I tried moving/renaming this:

    /usr/lib64/libcrypto.so.1.1.1k

    and tried to start the novell-apache2 service. When I did that, I got errors about libcrypto.so.1.1.1k missing.  So that is how I deduced it was trying to use the system provided crypto, but maybe that is not a correct assumption?

    On RHEL 7.9 though, they're a libcrypto.so.1.0.0 in /usr/lib64 and a libssl.so.1.0.0.  They are NOT symlinks and they differ form what is in /opt/novell/ssllib.

    Hmm, I wonder what would happen if I created a symlink on the RHEL 8.8 box in /usr/lib64 for the libcrypto and libssl in /opt/novell/ssllib?

    Matt

  • 0 in reply to   

    If you look at the systemd unit file for novell-apache2, you find:

    Environment="LD_LIBRARY_PATH=/opt/novell/ssllib:/opt/novell/openssl/lib"
    ExecStart=/opt/novell/apache2/sbin/httpd $ARGL

    I would modify ExecStart to run httpd under strace and take a look at the resulting output to determine which version of the libssl library is being used. You can possibly also try running httpd manually like so:

    # LD_LIBRARY_PATH=/opt/novell/ssllib:/opt/novell/openssl/lib strace -s256 -f -o /tmp/apache.tr /opt/novell/apache2/sbin/httpd

    then when it fails look at /opt/apache.tr and search for libssl or libcrypto.

    If you cannot run httpd manually, modify ExecStart with the strace command then "systemctl daemon-reload" and restart the novell-apache2 service.

  • 0   in reply to 

    Ok, so I did that and I grep'd apache.tr for libcrypto.  It almost seems like it is loading/using both of them:

    2561179 openat(AT_FDCWD, "/opt/novell/ssllib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 4
    2561179 openat(AT_FDCWD, "/opt/novell/ssllib/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    2561179 openat(AT_FDCWD, "/opt/novell/openssl/lib/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    2561179 openat(AT_FDCWD, "/lib64/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 5
    2561179 read(4, "1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 1"..., 1024) = 1024
    2561179 read(4, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcryp"..., 1024) = 1024
    2561179 openat(AT_FDCWD, "/opt/novell/ssllib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 7
    2561179 read(7, "fe9c290e000 r--p 002b5000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c290e000-7fe9c2912000 rw-p 002e1000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-"..., 1024) = 1024
    2561179 read(7, "7806 /opt/novell/nghttp2/lib/libnghttp2.so.14.22.0\n7fe9c7eb6000-7fe9c8127000 r-xp 00000000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /"..., 1024) = 1024
    2561179 read(7, "fe9c290e000 r--p 002b5000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c290e000-7fe9c2912000 rw-p 002e1000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-"..., 1024) = 1024
    2561179 read(7, "7806 /opt/novell/nghttp2/lib/libnghttp2.so.14.22.0\n7fe9c7eb6000-7fe9c8127000 r-xp 00000000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /"..., 1024) = 1024
    2561179 read(7, "fe9c290e000 r--p 002b5000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c290e000-7fe9c2912000 rw-p 002e1000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-"..., 1024) = 1024
    2561179 read(7, "7806 /opt/novell/nghttp2/lib/libnghttp2.so.14.22.0\n7fe9c7eb6000-7fe9c8127000 r-xp 00000000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /"..., 1024) = 1024
    2561179 read(15, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
    2561179 read(15, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024
    2561179 read(7, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
    2561179 read(7, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024
    2561179 read(7, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
    2561179 read(7, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024
    2561179 read(7, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
    2561179 read(7, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024

    I see both the novell and system ones being read/referenced here.  What does that tell us?

    Matt

  • 0 in reply to   

    Does running httpd on RHEL 7 also show more than one version of libssl being loaded?

  • 0   in reply to 
  • 0   in reply to   

    I saw that TID too when I was first troubleshooting this. Tried some variations of that but it made no difference unfortunately.

  • 0 in reply to   

    Did you symlink the libraries in /opt/novell/ssllib?

  • 0   in reply to 

    Just tested on a RHEL 7.9 box that hasn't been updated and no, it only loads the libssl and libcrypto from /opt/novell/ssllibs.  As a matter a fact, I see much fewer entries with libcrypto (only two I see in apache.tr), nothing like on the RHEL 8.8 box.

  • 0   in reply to 

    Tried that, no difference.