OBM 2022.05 upgrade to OA 12.27 issue

Problem after upgrade of OA from 12.22 to 12.27 on the 2022.05 OBM GW servers.  I already have a high priority case open on this, but I'd like to know if anyone has seen this. Every little bit helps.  2 DPs 2 GWs so it's a distributed environment on win2022 servers and the DPs are set as HA, Oracle DB, no LB.  We're having to upgrade OA to 12.27 on every server with an agent due to a CVE.  We're not prepared just yet for an OBM upgrade, which I know normally is the best way to upgrade OA on the OBM servers with the OBM install itself.  I upgraded to OA 12.27 on the DPs and the DPs start with no issues and OA looks good on both.  I upgraded to 12.27 on the GWs and every thing starts, but OA on the OBM servers and the managed nodes/agents starts buffering.  I reverted back to 12.22 on one GW to restore service.  It seems OA 12.27 runs fine until the GW is started then OA can't communicate.  I start seeing a lot of IO and SSL errors in the logs and when running opcagt it does not show a buffering status at the bottom of the output when ran on the GWs.  In our test environment we do not see this issue.  We only have 10 or so managed nodes/agents connected to our test environment, but in prod we have thousands.  Thank you.

Parents
  • Verified Answer

    +1  

    Hello there,

    OBM 2022.05 can be upgraded to OA12.27 - please see https://docs.microfocus.com/doc/Operations_Agent/12.27/ReleaseSummary where it states:

    Co-existence with Operations Agent 12.27

    Component Version Upgrade to 12.27
    OBM 2018.11 or higher Supported

    And again it's mentioned in the "Co-existence with Operations Agent 12.27" https://docs.microfocus.com/doc/Operations_Agent/12.27/Upgrade.  At least to me, that suggests that OA12.27 and OBM 2022.05 is supported, though I've never run OBM 2022.05 and OA12.27 together.

    1. On the managed nodes, what happens if you run:

    /opt/OV/bin/bbcutil -ping https://<load_balancer|GW name/?

    And on the OBM GWs

    /opt/OV/bin/bbcutil -ping https://<load_balancer|GW name/ -ovrg server?

    Do both of these commands work?

    2. On the GW, if you run:

    # lsof -i :383 |grep LISTEN
    ovbbccb 196636 root 9u IPv6 274543 0t0 TCP *:hp-alarm-mgr (LISTEN)

    Do you get a similar output?

    3. Is WDE on the GWs running:

    # /opt/HP/BSM/opr/support/opr-support-utils.sh -ls |grep -A5 wde
    Service Name = wde
    Process Name = hpbsm_wde
    Process Status = STARTED
    Execution Order = 20
    Status Since = 01/03/2025 13:39:14
    ---------------
    ----------------------

    And then lsof -p on the process called hpbsm_wde, should return:

    # lsof -p `pidof hpbsm_wde` |grep LIST
    hpbsm_wde 709655 root 556u IPv6 4285402 0t0 TCP localhost:attachmate-g32 (LISTEN)
    hpbsm_wde 709655 root 558u IPv6 4285405 0t0 TCP localhost:8010 (LISTEN)
    hpbsm_wde 709655 root 562u IPv6 4286245 0t0 TCP *:29604 (LISTEN)
    hpbsm_wde 709655 root 568u IPv6 4276872 0t0 TCP localhost:29904 (LISTEN)
    hpbsm_wde 709655 root 652u IPv6 4305591 0t0 TCP localhost:8105 (LISTEN)
    hpbsm_wde 709655 root 668u IPv6 4295913 0t0 TCP localhost:45727 (LISTEN)
    hpbsm_wde 709655 root 672u IPv6 4299970 0t0 TCP *:27316 (LISTEN)
    hpbsm_wde 709655 root 675u IPv6 4301941 0t0 TCP localhost:37225 (LISTEN)
    hpbsm_wde 709655 root 677u IPv6 4301620 0t0 TCP localhost:45215 (LISTEN)
    hpbsm_wde 709655 root 678u IPv6 4297491 0t0 TCP localhost:33359 (LISTEN)
    hpbsm_wde 709655 root 680u IPv6 4301950 0t0 TCP *:21500 (LISTEN)

    In this case, the process ID of hpbsm_wde is 709655 (second column in the lsof output).  

    4. Then on the LB itself, please run bbcutil -ping 127.0.0.1/endpoint_name -ovrg server

    For example, you can do this:
    # echo $(pidof hpbsm_wde)
    709655

    # output=$(/opt/OV/bin/ovbbccb -reg | grep -B5 709655)

    # echo "$output" | awk '
    /BasePath/ {base_path=$1; sub("BasePath=", "", base_path)}
    /BindAddress/ {bind_address=$1; sub("BindAddress=", "", bind_address)}
    /Protocol/ {protocol=$1; sub("Protocol=", "", protocol)}
    /PID=/ {
    url = protocol "://" bind_address base_path
    system("/opt/OV/bin/bbcutil -ping " url -ovrg server)
    }
    '


    Should give you an output like this:
    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=12 ms


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=OMi appV=11.80.010.001 conn=0 time=66 ms
    tlsVersion=TLSv1.3


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=74 ms tlsVersion=TLSv1.3


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=69 ms tlsVersion=TLSv1.3

    5. What errors do you see in:

    /var/opt/OV/log/System.txt # on the GW

    /var/opt/OV/log/System.txt # on one managed node?

    I hope this helps...

Reply
  • Verified Answer

    +1  

    Hello there,

    OBM 2022.05 can be upgraded to OA12.27 - please see https://docs.microfocus.com/doc/Operations_Agent/12.27/ReleaseSummary where it states:

    Co-existence with Operations Agent 12.27

    Component Version Upgrade to 12.27
    OBM 2018.11 or higher Supported

    And again it's mentioned in the "Co-existence with Operations Agent 12.27" https://docs.microfocus.com/doc/Operations_Agent/12.27/Upgrade.  At least to me, that suggests that OA12.27 and OBM 2022.05 is supported, though I've never run OBM 2022.05 and OA12.27 together.

    1. On the managed nodes, what happens if you run:

    /opt/OV/bin/bbcutil -ping https://<load_balancer|GW name/?

    And on the OBM GWs

    /opt/OV/bin/bbcutil -ping https://<load_balancer|GW name/ -ovrg server?

    Do both of these commands work?

    2. On the GW, if you run:

    # lsof -i :383 |grep LISTEN
    ovbbccb 196636 root 9u IPv6 274543 0t0 TCP *:hp-alarm-mgr (LISTEN)

    Do you get a similar output?

    3. Is WDE on the GWs running:

    # /opt/HP/BSM/opr/support/opr-support-utils.sh -ls |grep -A5 wde
    Service Name = wde
    Process Name = hpbsm_wde
    Process Status = STARTED
    Execution Order = 20
    Status Since = 01/03/2025 13:39:14
    ---------------
    ----------------------

    And then lsof -p on the process called hpbsm_wde, should return:

    # lsof -p `pidof hpbsm_wde` |grep LIST
    hpbsm_wde 709655 root 556u IPv6 4285402 0t0 TCP localhost:attachmate-g32 (LISTEN)
    hpbsm_wde 709655 root 558u IPv6 4285405 0t0 TCP localhost:8010 (LISTEN)
    hpbsm_wde 709655 root 562u IPv6 4286245 0t0 TCP *:29604 (LISTEN)
    hpbsm_wde 709655 root 568u IPv6 4276872 0t0 TCP localhost:29904 (LISTEN)
    hpbsm_wde 709655 root 652u IPv6 4305591 0t0 TCP localhost:8105 (LISTEN)
    hpbsm_wde 709655 root 668u IPv6 4295913 0t0 TCP localhost:45727 (LISTEN)
    hpbsm_wde 709655 root 672u IPv6 4299970 0t0 TCP *:27316 (LISTEN)
    hpbsm_wde 709655 root 675u IPv6 4301941 0t0 TCP localhost:37225 (LISTEN)
    hpbsm_wde 709655 root 677u IPv6 4301620 0t0 TCP localhost:45215 (LISTEN)
    hpbsm_wde 709655 root 678u IPv6 4297491 0t0 TCP localhost:33359 (LISTEN)
    hpbsm_wde 709655 root 680u IPv6 4301950 0t0 TCP *:21500 (LISTEN)

    In this case, the process ID of hpbsm_wde is 709655 (second column in the lsof output).  

    4. Then on the LB itself, please run bbcutil -ping 127.0.0.1/endpoint_name -ovrg server

    For example, you can do this:
    # echo $(pidof hpbsm_wde)
    709655

    # output=$(/opt/OV/bin/ovbbccb -reg | grep -B5 709655)

    # echo "$output" | awk '
    /BasePath/ {base_path=$1; sub("BasePath=", "", base_path)}
    /BindAddress/ {bind_address=$1; sub("BindAddress=", "", bind_address)}
    /Protocol/ {protocol=$1; sub("Protocol=", "", protocol)}
    /PID=/ {
    url = protocol "://" bind_address base_path
    system("/opt/OV/bin/bbcutil -ping " url -ovrg server)
    }
    '


    Should give you an output like this:
    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=12 ms


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=OMi appV=11.80.010.001 conn=0 time=66 ms
    tlsVersion=TLSv1.3


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=74 ms tlsVersion=TLSv1.3


    127.0.0.1/.../:
    status=eServiceOK coreID=5f25052a-f0c0-75e9-0cae-fa39433c17dc
    bbcV=12.27.005 appN=com.hp.ov.ow.SvcDscSvr appV=unknown version
    conn=0 time=69 ms tlsVersion=TLSv1.3

    5. What errors do you see in:

    /var/opt/OV/log/System.txt # on the GW

    /var/opt/OV/log/System.txt # on one managed node?

    I hope this helps...

Children