problem with StoreOnce Software deduplication version 24.3/24.4 - server side deduplication did't work properly with other gateways.

 Hi.

We recently updated cell server 11.03 -> 24.3.

Everything went well without any problems.


But we discover the following problem. media agent gateway after upgrading to version 24.3 did not work with sosd storage of the same version.


in debugging i found that the media agent didn't find some library

[ 5] 2024-09-27 13:37:46.843 ("/ma/dev/system_SODA.c $Rev$ $Date:: $:":18)
[100] *************** SKPrintf [START] ***************
[100] Tokenize the string...
[100] token [0] = ^G[%1$s] From: %3$s@%4$s "%6$s" Time: %2$s
%5$s

[100] token [1] = Critical
[100] token [2] = 09/27/24 13:37:46
[100] token [3] = MMA
[100] token [4] = hornet.sdab.sn
[100] token [5] = ^G[12:5] Internal error in ("%s":%d) => process aborted
This is an unexpected condition and is likely due to a
corrupted media or combination of circumstances
involving both this product and the operating system.
Please report this error to your post-sales
Support Representative.

[100] token [6] = /ma/dev/system_SODA.c $Rev$ $Date:: $:
[100] token [7] = 18
[100] token [8] = B2D gateway
[100] *************** SKPrintf [END] ***************
[ 5] [REPORT 3] in GetSodaLibrary
Marked: ("/lib/cmn/common.c $Rev$ $Date:: $:":Line:2221 in 'dlopen')
Code is:1007 SystemErr: libDeduplicationAPI.so: libomnissl.so.1.0.0: cannot open shared object file: No such file or directory
[Critical] From: MMA@hornet.sdab.sn "B2D gateway" Time: 09/27/24 13:37:46
[12:5] Internal error in ("/ma/dev/system_SODA.c $Rev$ $Date:: $:":18) => process aborted
This is an unexpected condition and is likely due to a
corrupted media or combination of circumstances
involving both this product and the operating system.
Please report this error to your post-sales
Support Representative.

# omnidownload -library DDSTOREDRAGON
NAME "DDSTOREDRAGON"
DESCRIPTION "DDSTOREDRAGON"
POLICY BackupToDisk
TYPE StoreOnceSoftware
REPOSITORY
"200b010a_66f688ec_b72d_003b"
"200b010a_66f69130_b72d_003d"
"200b010a_66f69a02_b72d_003f"
DIRECTORY
"\\dragon.sdab.sn\ddstore"
MGMTCONSOLEURL ""

# omnidownload -device DDSTOREDRAGON-hornet
NAME "DDSTOREDRAGON-hornet"
DESCRIPTION "DDSTOREDRAGON"
HOST hornet.sdab.sn
POLICY BackupToDisk
TYPE StoreOnceSoftware
POOL "DDSTOREDRAGON_MediaPool"
LIBRARY "DDSTOREDRAGON"
SERVERSIDEDEDUP
RESTOREDEVICEPOOL NO
COPYDEVICEPOOL NO

cell_info:-host hornet.sdab.sn -os "gpl x86_64 linux-3.10.0-862.el7.x86_64" -type "rhel 7.8" -core 24.3 -corepatch 24.3 -integ 24.3 -integpatch 24.3 -cmnutils 24.3 -cmnutilspatch 24.3 -da 24.3 -dapatch 24.3 -ma 24.3 -mapatch 24.3 -oracle8 24.3 -oracle8patch 24.3 -ts_core 24.3 -ts_corepatch 24.3 

cell_info:-host dragon.sdab.sn -os "gpl x86_64 linux-3.10.0-1160.90.1.el7.x86_64" -type "rhel 7.9" -core 24.3 -corepatch 24.3 -da 24.3 -dapatch 24.3 -ma 24.3 -mapatch 24.3 -integ 24.3 -integpatch 24.3 -pegasus 24.3 -pegasuspatch 24.3 -cmnutils 24.3 -cmnutilspatch 24.3 -vepa 24.3 -vepapatch 24.3 -cc 24.3 -ccpatch 24.3 -ts_core 24.3 -ts_corepatch 24.3 -StoreOnceSoftware 24.3 -StoreOnceSoftwarepatch 24.3

  • Suggested Answer

    0  

    I think I have seen something similar before. While this should not be needed, please push the SOS component to your gateway client "hornet" as a workaround. Please report back the result. Thanks.

  • 0 in reply to   

    hi. 

    I try this WA ( by installing StoreOnce software packet on the problem gw )  but it's did't helped. 

    Backup session starting from gw hornet hangin up by long term and do not writing any data on the sosd. 

    In debugs I did't find any information that will hepled with this issue ( may be it's due the luck of my skill's ).

    Such session hanging long time and did't abort by omniabort comand so I will used kill -QUIT by the apropriate process. 

    BTW backup that was started over owners sosd gateway running witout issue. 

  • 0   in reply to   

    I believe you mean that when the gateway on "dragon" is used then the backup runs fine?

    Do you maybe have a support case for this? It would be ggod to provide the debugs of the failing backup to support so they can verify what's going on. Feel free to let me know the caseID. I can take a look and monitor the case.

  • 0 in reply to   

    Hi. 

    >I believe you mean that when the gateway on "dragon" is used then the backup runs fine?

    Yes. Dragon is the server that host SOSD, and when its used as gw for backup - backup run without problem. 

    Opened related case in support portal. 

    case:02974179 

    waitng for url for uploading debugs log. 

  • 0 in reply to 

    Hi.

    I'v upload debugs log related to this case.

    name of archive failed-sosd.7z

  • 0   in reply to 

    I quickly checked those debugs and they are NOT showing the problem that you reported here. Regarding the problem reported her in this thread, if it still occurs then please share the output of

    find /opt/omni |grep libomnissl

    from the gateway system hornet.sdab.sn

    Regarding the debugs: Are there possibly system resource issues? Did the bma possibly crash? Anything you can find from the system logs that could correspond to the time of the reported problem?

  • 0 in reply to   

     Hi. 

    14:16 [/2][root@hornet]# find /opt/omni |grep libomnissl 
    /opt/omni/lib/libomnissl.so
    /opt/omni/lib/libomnissl.so.1.0.0
    /opt/omni/lib64/libomnissl.so
    /opt/omni/lib64/libomnissl.so.1.0.0
    14:27 [/2][root@hornet]#

    > Are there possibly system resource issues? 

    I dod't think so. We get the same problem on the production servers, which have grate memory resources and cpu's.

    forexamle configuration of oure production media agent.

    + Hostname = totoro.sdab.sn
    + OS = OracleLinux 7.9
    + CPUs = 80
    + Memory = 187.37 GiB

    On the gateway hornet I did't find any crush dump of bma or other dp binaries. I'v Trying to greping logs in /var/log/* with no results.

    I'll try another short with collecting debugs log set's - without interrupt backup session as I did last time.

    The main issue - backup starting without error, but do not send or write to sosd system. This problem was discovering on all  oure production and test system's (rhel oel 7,8,9). It's can hangin long time with freezing state. I will try to wait for some error.

     
     

  • 0 in reply to   
    Greetings 
    
    Finally I got the error but it is an expectation error. 
    
    Anyway - the backup session tried to write one file to sosd and hung for over 1 day and finally timed out.
    
    the debug log has been collected and uploaded to the portal. Hope this helps in some way to solve this problem.
  • Verified Answer

    +1  

    Hi Andriy,

    I had the same issue with DP 24.2 on a windows server.

    An Hot Fix has been developed to address the issue.

    Ask support team to give you HF  DP242_b174_OCTCR19Q2230770_HF1 updated for DP 24.3 and for linux server.

    Kind regards

    Dionisio Sciascia

    Although I am an OpenText employee, I am speaking for myself and not for OpenText.
    If you found this post useful, give it a “Like” or click on "Verify Answer" under the "More" button.

  • 0 in reply to   

    Hi.

    Thank you for your help in solving this problem.

    Sent a request to support to get this fix