OBM 2022.05 upgrade to OA 12.27 issue

Problem after upgrade of OA from 12.22 to 12.27 on the 2022.05 OBM GW servers.  I already have a high priority case open on this, but I'd like to know if anyone has seen this. Every little bit helps.  2 DPs 2 GWs so it's a distributed environment on win2022 servers and the DPs are set as HA, Oracle DB, no LB.  We're having to upgrade OA to 12.27 on every server with an agent due to a CVE.  We're not prepared just yet for an OBM upgrade, which I know normally is the best way to upgrade OA on the OBM servers with the OBM install itself.  I upgraded to OA 12.27 on the DPs and the DPs start with no issues and OA looks good on both.  I upgraded to 12.27 on the GWs and every thing starts, but OA on the OBM servers and the managed nodes/agents starts buffering.  I reverted back to 12.22 on one GW to restore service.  It seems OA 12.27 runs fine until the GW is started then OA can't communicate.  I start seeing a lot of IO and SSL errors in the logs and when running opcagt it does not show a buffering status at the bottom of the output when ran on the GWs.  In our test environment we do not see this issue.  We only have 10 or so managed nodes/agents connected to our test environment, but in prod we have thousands.  Thank you.

  • 0 in reply to   

    I have already tried a remove and re-request of the cert on the broke GW, which did not help.  How do I check the asymmetric key length and hash also settings on the certs?

  • 0   in reply to 

    Hi Angil,

    You can run 'ovcoreid' to get the core id, then run 'ovcert -certinfo <coreid>'.

    You can also run 'ovconfget' to list out all the OA settings, or 'ovconfchg -edit' which will open the settings in notepad on Windows, where you can edit and save if you need to.  Most settings will take effect after you run 'ovconfchg' on its own, but some require an agent restart.

    It's worth looking for any differences with TLS and cipher settings too.

    Regards,

    Tim

  • 0 in reply to 

    You may have checked  just putting for ref. to check output of ovc command (to validate ovcs component)

    Also its not directly related but you can see keys and algo information

    Certificates missing after upgrade - Operations Bridge Manager

  • 0   in reply to 

    Hello there,

    This is how events flow between the managed node and the OBM server
    OA12 managed node --> ovbbccb --> wde(java) --> [JMS] --> opr-backend(java)

    1. OA12 managed node opcmsga process sends event to...
    2. ovbbccb, which is part of OA12 running on the Gateway on the OBM server. If it's a combined server, it's just the OBM server.
    3. ovbbccb passes the event to JMS, which is also know as the "bus", this sends the event to...
    4. opc-backend that processes the event.

    These are the same tests for Windows:
    1. This tests certificates that OA12 uses. These commands should not fail:
    "%ovinstalldir%\bin\win64\bbcutil.exe" -ping https://<<Gateway or DPS>>/
    "%ovinstalldir%\bin\win64\bbcutil.exe" -ping https://<<Gateway or DPS>>/ -ovrg server

    This tests that the managed node can connect to the OBM server. If this does not work, you have a problem with certificates.

    If you certificates have a problem, then talk to Opentext support about next steps.

    2. This checks if ovbbccb is listening:
    netstat -ano | findstr :383

    ovbbccb is the process that gets events from the managed nodes and passes them to WDE.

    3. This checks that WDE is running. This is the JVM that gets events from managed nodes.
    "%topaz_home%\opr\support\opr-support-utils.bat" -ls > some.file.txt

    4. This checks wde has registered with ovbbccb:
    "%ovinstalldir%\bin\win64\bbcutil.exe" -reg |findstr msgr
    BasePath=/com.hp.ov.opc.msgr/

    So, WDE registers where to send events with ovbbccb. This means ovbbccb can send the event to WDE, so the event can be sent to JMS (or the bus) and forwarded to opr-backend for processing.

    Bonus: run "%topaz_home%\opr\support\opr-jmsUtil.bat" and look if opr_gateway_queue has event buffered:

    C:\Program Files\HP\HP BTO Software>%topaz_home%\opr\support\opr-jmsUtil.bat
    Live Server: puegcsvm60395
    Last Alive: Fri Jan 24 09:19:35 CET 2025 (12 seconds ago)
    ==============================================================================================================================

    queue | total | buffered | delivering | memory | pages | consumers
    -----------------------------------------------------------------------------------------------------------------
    opr_action_launch_queue | 0 | 0 | 0 | 0 | 0 | 1
    recipient_notification | 0 | 0 | 0 | 0 | 0 | 1
    queue/alert_engine_alert | 0 | 0 | 0 | 0 | 0 | 1
    queue/alert_engine_notification | 0 | 0 | 0 | 0 | 0 | 1
    opr_gateway_queue | 1232619 | 12532 | 0 | 0 | 0 | 1
    failed_recipient_notification | 0 | 0 | 0 | 0 | 0 | 1

    So the above example shows there are 12532 buffered waiting to be processed by opr-backend. This means opr-backend has an issue processing events, which can be caused by many, many different reasons.


    5. System.txt
    This is %ovdatadirdir%\log\OpC\System.txt is the OA12 error log file. Please check for errors.


    Finally:
    One more thing you can try - restart the relevent processes::
    "%ovinstalldir%\bin\win64\ovc.exe" -kill
    "%topaz_home%\opr\support\opr-support-utils.bat" -stop wde

    "%ovinstalldir%\bin\win64\ovc.exe" -start
    "%topaz_home%\opr\support\opr-support-utils.bat" -start wde

    Then check steps 1-5.

    I hope this helps.

  • 0   in reply to   

    Finally, I'd really like to think that Opentext Support are working directly on your system at this point.