OBM 2022.05 upgrade to OA 12.27 issue

Problem after upgrade of OA from 12.22 to 12.27 on the 2022.05 OBM GW servers.  I already have a high priority case open on this, but I'd like to know if anyone has seen this. Every little bit helps.  2 DPs 2 GWs so it's a distributed environment on win2022 servers and the DPs are set as HA, Oracle DB, no LB.  We're having to upgrade OA to 12.27 on every server with an agent due to a CVE.  We're not prepared just yet for an OBM upgrade, which I know normally is the best way to upgrade OA on the OBM servers with the OBM install itself.  I upgraded to OA 12.27 on the DPs and the DPs start with no issues and OA looks good on both.  I upgraded to 12.27 on the GWs and every thing starts, but OA on the OBM servers and the managed nodes/agents starts buffering.  I reverted back to 12.22 on one GW to restore service.  It seems OA 12.27 runs fine until the GW is started then OA can't communicate.  I start seeing a lot of IO and SSL errors in the logs and when running opcagt it does not show a buffering status at the bottom of the output when ran on the GWs.  In our test environment we do not see this issue.  We only have 10 or so managed nodes/agents connected to our test environment, but in prod we have thousands.  Thank you.

  • 0  

    Hi Angil,

    The Operations Agent on the OBM servers cannot be independently upgraded as the agent ties in with the OBM software.

    You can open a support case to ask OpenText if they have Operations Agent updates available for your version of OBM.

    Thanks and regards,

    Tim

  • Verified Answer

    +1  

    Hello there,

    OBM 2022.05 can be upgraded to OA12.27 - please see https://docs.microfocus.com/doc/Operations_Agent/12.27/ReleaseSummary where it states:

    Co-existence with Operations Agent 12.27

    Component Version Upgrade to 12.27
    OBM 2018.11 or higher Supported

    And again it's mentioned in the "Co-existence with Operations Agent 12.27" https://docs.microfocus.com/doc/Operations_Agent/12.27/Upgrade.  At least to me, that suggests that OA12.27 and OBM 2022.05 is supported, though I've never run OBM 2022.05 and OA12.27 together.

    1. On the managed nodes, what happens if you run:

    /opt/OV/bin/bbcutil -ping https://<load_balancer|GW name/?

    And on the OBM GWs

    /opt/OV/bin/bbcutil -ping https://<load_balancer|GW name/ -ovrg server?

    Do both of these commands work?

    2. On the GW, if you run:

    # lsof -i :383 |grep LISTEN
    ovbbccb 196636 root 9u IPv6 274543 0t0 TCP *:hp-alarm-mgr (LISTEN)

    Do you get a similar output?

    3. Is WDE on the GWs running:

    # /opt/HP/BSM/opr/support/opr-support-utils.sh -ls |grep -A5 wde
    Service Name = wde
    Process Name = hpbsm_wde
    Process Status = STARTED
    Execution Order = 20
    Status Since = 01/03/2025 13:39:14
    ---------------
    ----------------------

    And then lsof -p on the process called hpbsm_wde, should return:

    # lsof -p `pidof hpbsm_wde` |grep LIST
    hpbsm_wde 709655 root 556u IPv6 4285402 0t0 TCP localhost:attachmate-g32 (LISTEN)
    hpbsm_wde 709655 root 558u IPv6 4285405 0t0 TCP localhost:8010 (LISTEN)
    hpbsm_wde 709655 root 562u IPv6 4286245 0t0 TCP *:29604 (LISTEN)
    hpbsm_wde 709655 root 568u IPv6 4276872 0t0 TCP localhost:29904 (LISTEN)
    hpbsm_wde 709655 root 652u IPv6 4305591 0t0 TCP localhost:8105 (LISTEN)
    hpbsm_wde 709655 root 668u IPv6 4295913 0t0 TCP localhost:45727 (LISTEN)
    hpbsm_wde 709655 root 672u IPv6 4299970 0t0 TCP *:27316 (LISTEN)
    hpbsm_wde 709655 root 675u IPv6 4301941 0t0 TCP localhost:37225 (LISTEN)
    hpbsm_wde 709655 root 677u IPv6 4301620 0t0 TCP localhost:45215 (LISTEN)
    hpbsm_wde 709655 root 678u IPv6 4297491 0t0 TCP localhost:33359 (LISTEN)
    hpbsm_wde 709655 root 680u IPv6 4301950 0t0 TCP *:21500 (LISTEN)

    In this case, the process ID of hpbsm_wde is 709655 (second column in the lsof output).  

    4. Then on the LB itself, please run bbcutil -ping 127.0.0.1/endpoint_name -ovrg server

    For example, you can do this:
    # echo $(pidof hpbsm_wde)
    709655

    # output=$(/opt/OV/bin/ovbbccb -reg | grep -B5 709655)

    # echo "$output" | awk '
    /BasePath/ {base_path=$1; sub("BasePath=", "", base_path)}
    /BindAddress/ {bind_address=$1; sub("BindAddress=", "", bind_address)}
    /Protocol/ {protocol=$1; sub("Protocol=", "", protocol)}
    /PID=/ {
    url = protocol "://" bind_address base_path
    system("/opt/OV/bin/bbcutil -ping " url -ovrg server)
    }
    '


    Should give you an output like this:
    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=12 ms


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=OMi appV=11.80.010.001 conn=0 time=66 ms
    tlsVersion=TLSv1.3


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=74 ms tlsVersion=TLSv1.3


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=69 ms tlsVersion=TLSv1.3

    5. What errors do you see in:

    /var/opt/OV/log/System.txt # on the GW

    /var/opt/OV/log/System.txt # on one managed node?

    I hope this helps...

  • 0

    I ran into similar in one of the sites. I should have investigated further but I thought it was one off situation. I was able to work it out by running the config wizard.

    OBM 2022.05 Vanilla OA 12.22 to 12.27.In my case it was a single server OBM install though(DPS and GW same server).

  • 0 in reply to   

    Thank you Duncan.  I'll go through what you've provided and requested.  I really appreciate the detailed reply.

  • 0 in reply to 

    Thank you Misaq.  I have already tried running through the configuration wizard, which did not help.  Stopped all OBM, ran config on primDP, secDP, then GWs.  Same issue when starting back up.  I'm going to look further into what Duncan provided and we'll see how this goes.  Thank you for the reply.

  • 0 in reply to 

    Duncan I'm still working my high priority case with opentext support.  No resolution just yet.  Many of the troubleshooting commands you gave are for Unix.  Our OBM servers are on windows.  Also we do not have a load balancer.

    Currently OA 12.27 is running with no issue on the GW.  After the GW is started System.txt on the GW shows lots of these errors....

    0: WRN: Thu Jan 23 11:57:55 2025: ovbbccb (13220/1328): (bbc-90) The incoming HTTPS client connection from host 00.00.00.00 failed due to the SSL error:
    1: WRN: Thu Jan 23 11:57:55 2025: ovbbccb (13220/1328): (sec.core-116) An SSL connection IO error has occurred. This may be due to a network problem or an SSL handshake error. Possible causes for SSL handshake errors are that no certificate is installed, an invalid certificate is installed, or the peer does not trust the initiator's certificate.

    ---------------------------------------------

    Before GW is started - OA on the GW itself.

    D:\HPBSM\opr\support>ovcert -status
    Status: Certificate is installed.

    D:\HPBSM\opr\support>opcagt

    dsisrv (DSI service) (7428) Running
    midaemon (measurement interface) (3448) Running
    misrv (measurement interface service) (2848) Running
    perfd (Real Time Metric Access Daemon) (15844) Running
    perfdsrv (Real Time Metric Access Service) (15824) Running
    ttd (transaction tracking) (3800) Running
    ttsrv (transaction tracking service) (3484) Running
    Mwecsrv (extended collection service) (6356) Running
    agtrep OV Discovery Agent AGENT,AgtRep (20092) Running
    hpsensor Compute Sensor AGENT,OA (15460) Running
    oacore Operations Agent Core AGENT,OA (19000) Running
    oahmon Agent Health Monitor AGENT,EA (15820) Running
    ompolparm OM Parameter Handler AGENT,EA (16508) Running
    opcacta OVO Action Agent AGENT,EA (13080) Running
    opcle OVO Logfile Encapsulator AGENT,EA (6220) Running
    opcmona OVO Monitor Agent AGENT,EA (20196) Running
    opcmsga OVO Message Agent AGENT,EA (19036) Running
    opcmsgi OVO Message Interceptor AGENT,EA (17084) Running
    opcwbemi OVO WMI Interceptor AGENT,EA (18500) Running
    ovbbccb OV Communication Broker CORE (13220) Running
    ovbbcrcp OV RC PROXY COREXT (3832) Running
    ovcd OV Control CORE (16100) Running
    ovconfd OV Config and Deploy COREXT (7836) Running

    Agent Health Status: Minor, Time:Wed Jan 22 15:21:41 2025
    Message Agent is not buffering.

    ---------------------------------------------

    After GW is started - OA on the GW itself.

    D:\HPBSM\opr\support>ovcert -status
    Status: Undefined (Certificate Client could not be contacted).

    Notice no buffering state

    D:\HPBSM\opr\support>opcagt

    dsisrv (DSI service) (7428) Running
    midaemon (measurement interface) (3448) Running
    misrv (measurement interface service) (2848) Running
    perfd (Real Time Metric Access Daemon) (15844) Running
    perfdsrv (Real Time Metric Access Service) (15824) Running
    ttd (transaction tracking) (3800) Running
    ttsrv (transaction tracking service) (3484) Running
    Mwecsrv (extended collection service) (6356) Running
    agtrep OV Discovery Agent AGENT,AgtRep (16056) Running
    hpsensor Compute Sensor AGENT,OA (18180) Running
    oacore Operations Agent Core AGENT,OA (4604) Running
    oahmon Agent Health Monitor AGENT,EA (8820) Running
    ompolparm OM Parameter Handler AGENT,EA (3744) Running
    opcacta OVO Action Agent AGENT,EA (4324) Running
    opcle OVO Logfile Encapsulator AGENT,EA (19380) Running
    opcmona OVO Monitor Agent AGENT,EA (10908) Running
    opcmsga OVO Message Agent AGENT,EA (12996) Running
    opcmsgi OVO Message Interceptor AGENT,EA (17284) Running
    opcwbemi OVO WMI Interceptor AGENT,EA (13540) Running
    ovbbccb OV Communication Broker CORE (13220) Running
    ovbbcrcp OV RC PROXY COREXT (8624) Running
    ovcd OV Control CORE (16100) Running
    ovconfd OV Config and Deploy COREXT (17872) Running


    D:\HPBSM\opr\support>

  • 0 in reply to   

    Hi Tim,

    I used to understand the same, but I found you can upgrade OA on the OBM servers without an OBM upgrade.  I sure it's best to upgrade OBM and let it upgrade OA, but timing to fix this CVE is not allowing me a complete upgrade of OBM.  Again we are on OBM 2022.05 and I'm trying to upgrade to 12.27 to fix CVE-2024-5532.  The SUMA shows that OA 12.27 is supported and co-exist with OBM 2022.05.

    https://portal.microfocus.com/s/article/KM000035731?language=en_US

    The upgrade on our test servers worked great and when starting the OBM gateways I see no issues.  It's in prod that OA has issues after starting the GW.  In prod OA 12.27 runs with no issue until the GW is started.

  • 0 in reply to 

    Did check the cert validation on the node(OBM) (before and after GW)

    Command should be same for windows

  • 0   in reply to 

    Hi Angil,

    If your Data Processing servers starting OK, you could try requesting new agent certificates for the Gateways.  It might be worth checking the asymmetric key length and hash algo settings on all 4 OBM servers and older certs are no longer compatible.  Maybe some of the default TLS settings have changed with the upgrade.

    Regards,

    Tim

  • 0 in reply to 

    Misaq,

    It shows OK for both commands before and after starting the GW.  You can see some more into I responded to Duncan that shows a difference with the ovcert -status and opcagt commands.