This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Non-Software-Installer patches all not detected as applicable

This is forked from the discussion in /collaboration/zenworks/zpm/f/patchmanagement/201196/recent-patches-showing-not_applicable, per request from . I had replied (a week or three after the final other post in that thread) reporting that I am experiencing what from its symptoms seems to be the same issue, but based on that discussion it might actually be different under the hood.

To recap some of the information from there, and add more:

My organization is running ZENworks 2017 Update 4 on the servers, although nearly all of the clients are still running the 11.4.3 client.

For some undetermined number of months now, certain machines in our fleet have been failing to detect newly-available patches as being applicable - at least initially, and quite possibly permanently. This has been a minority of computers, but enough to be noticeable as a problem. The exception has been that patches which are of a "Software Installer" priority seem to be detected as applicable without issues - which is not useful to us, since we don't install software through such patches, because we need to be able to apply install-time customizations which are not possible with that method.

With the March patches, the problem has become apparently universal. It seems to be affecting every single computer we manage.

In the cases where I've examined this directly, it has usually seemed as if the *.ospx files which I gather contain the patch-applicability definitions have not been becoming updated. More on this below.

In cases where I observed this behavior on a virtual machine, nothing I did seemed to result in the patches becoming detected as applicable, but if I then restored a pristine image (built months prior to this problem having first manifested) into that same VM the resulting system - connected to a new, separate, workstation object in ZCC - would detect the patches without issues.

I have attempted to troubleshoot this by enabling the verbose logging described in https://support.microfocus.com/kb/doc.php?id=7014131 and examining the resulting log files for detection information regarding a specific patch (released on March 9th, more than two weeks prior to the date of the troubleshooting) which we expect to want to deploy. As far as I can tell, most of the log files do not contain any reference to that patch. debug*.log and *.state do mention it, but only to report that detection failed, without anything to indicate how that detection was carried out; none of the others seem to show any sign that the patch exists. Also, the "Discovery Agent" Registry location referenced in that KB item does not contain any readily identifiable reference to the patch in question.

On several of the machines where I've attempted to troubleshoot this, I've observed that the *.ospx files which seem to contain the patch-applicability definitions have had timestamps which predate the release of the patches which are not being detected. Nothing I've tried, that I recall, has been able to get these files to be updated; even deleting them and trying to verify the DAU bundle just results in the scan running without sign of errors, and without the files being put back into place.

On one machine (the one where I'm typing this), these files have timestamps from today and yesterday, but the detection-failure behavior is still occurring. When I ran the lm_detection_x64 verbose-logging command from the above-linked KB item, I noticed that files began to appear and disappear in the tmpScan\lm.detection.cache\ directory (which is normally empty, if present at all), with names corresponding to the *.ospx files but with the file extension '.lmd' - and these files had timestamps from February 23rd, one month ago yesterday, well prior to both the dates of the *.ospx files and the dates of the updated patches which are not being detected.

In the thread from which this one is forked, it was suggested that I download the MSU for the most recent applicable Servicing Stack Update and apply it on selected machines by hand, then re-run the scans and see whether anything had changed. Results were somewhere between negative and mixed, with a stopover at inconclusive; on one machine (for which only two non-Software-Installer patches were listed as applicable, neither of them cached) the update and re-scan changed nothing, on another (for which I had forgotten to check the exact set of available patches prior to the scan), after the scan there were patches shown as applicable which dated from as far back as July of 2020 and as recently as February, but the patches released in March still did not show as applicable.

(As I write this, it occurs to me that I did not rename the SYSTEM_HASH Registry entry on at least one of these computers prior to the re-scan. I do not know whether this might have affected the results.)

Any suggestions for things to try, or should I open a SR about this for more in-depth and hands-on analysis?

  • 0

    Just to confirm, have you tried deleting the DAU bundle, running zman ctc afterwards and then wait some time, and run the subscription update again?

    On an individual workstation, Did you try going to "Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Patchlink.com" in
    the registry and rename the "cache" folder/key there to something else like "cache1" or
    "cache2" prior to another patch scan?

     I think you're probably in the territory  where you may want an SR.

  • 0 in reply to 

     wrote:

    Just to confirm, have you tried deleting the DAU bundle, running zman ctc afterwards and then wait some time, and run the subscription update again?

    No; I wasn't aware of that as a path to be taken. I don't think I knew that there were included / supported ways to regenerate the DAU bundle, or that this might be an advisable option.

    I'll look for appropriate documentation, and assuming it looks reasonable, try this out.

    On an individual workstation, Did you try going to "Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Patchlink.com" in
    the registry and rename the "cache" folder/key there to something else like "cache1" or
    "cache2" prior to another patch scan?

    No. I considered it, but hadn't managed to find anything to indicate that doing this was safe; the KB item I mentioned only talks about renaming the SYSTEM_HASH entry from that key, not doing anything with the rest of the key.

    I'd be glad to try this, and will probably do so even ahead of trying the above.

     I think you're probably in the territory  where you may want an SR.


    I was planning on one, before I found the other thread, and I still do plan to file one if nothing else pans out, possibly as soon as this afternoon.

  • 0  

    What you want to do is work top-down to detect and resolve your issues.

    #1 - When was your last Subscription Poll?  What is Successful?  This is most likely fine, there should not be any systemic issues here.  But if failing, it does not fall into any known issues.  If it is, we can loop back.

    #2 - Is your DAU Bundle Correct? Specifically, make sure the OSPX files are referenced.  There can be known issues with older Zones where these are not downloading correctly.  Your zone definitely could meet those criteria.  If missing, we will need to rebuild your  Java Certificate Store.  (Details Later)

    OSPX.PNG

    #3 - If all is good at this point, you will need to verify your test device is assigned the DAU bundle.  By default, it is assigned to all devices when the Update in Step#1 runs.  If you are testing a device with a newly created device object, it may not have the assignment until the update runs unless it is manually assigned.  be sure to do that.  In general, there are not any known issues with devices getting assignments so long as the overnight downloads are successful.

    #4 - Make sure the device has the latest "SSU" update.  This is needed for proper detection and installation of "Microsoft Patches" but not Installers.  If the SSU is out of date, a patch will be flagged as "Not Applicable" because it is missing the Pre-req of a current SSU.  This can even impact the SSU itself, which is a MS bug that MS is working on addressing and impacts native MS patching solutions as well.  A "Workaround" patch has been created which is the "SSU" with "Workaround" in the name which installs using "Software Installer" calls instead of using an "Windows Patch" routine.

     

    --

    If you found this post useful, give it a “Like” or click on "Verify Answer" under the "More" button

    Be sure to "Like" My (and a few others) Cool Solutions below! 

    https://community.microfocus.com/members/craigdwilson/bookmarks

  • 0   in reply to   

    Edit: Alert - If you are using an ZCM Appliance, The steps below may impart port 9443 w/o additional steps.

    Avoid using the steps on an appliance: (Comment Added Dec 7th 2021)

    Fixing the JavaTrustore is something that only should be done if you KNOW it needs to be done.  ZCM 20.1 and later updates will actually rebuild it for you during the update to bring in the latest trusts with the latest Java.

    The following steps will presume Linux, but if you have Windows it should not be hard for you to translate.

    #1 - Get the "Passphrase" for your JavaStore from: 

    cat /etc/opt/novell/zenworks/security/zenCacertPassphrase.txt

    #2 - Export your Java Truststore to a text file.

    keytool -list -v -keystore /vastorage/etc/opt/novell/zenworks/security/zenCaCertStore -storepass xxxx > /tmp/certout.txt

    #3 - Search the certou.txt file for "The USERTRUST Network"

    cat certout.txt | grep 'The USERTRUST Network'

    There should be entries similar to

    Owner: CN=USERTrust RSA Certification Authority, O=The USERTRUST Network, L=Jersey City, ST=New Jersey, C=US

    Issuer: CN=USERTrust RSA Certification Authority, O=The USERTRUST Network, L=Jersey City, ST=New Jersey, C=US

    --

    If you found this post useful, give it a “Like” or click on "Verify Answer" under the "More" button

    Be sure to "Like" My (and a few others) Cool Solutions below! 

    https://community.microfocus.com/members/craigdwilson/bookmarks

  • Verified Answer

    0   in reply to   

    Edit: Alert - If you are using an ZCM Appliance, The steps below may impart port 9443 w/o additional steps.

    Avoid using the steps on an appliance: (Comment Added Dec 7th 2021)

    Assuming the OSPX files are MISSING from the DAU bundle "AND" 'The USERTRUST Network' is MISSING from your CA store, you will want to rebuild it following these steps.

    1. - novell-zenworks-configure -c MergeTruststore -Z
    2. - novell-zenworks-configure -c EnableJMX
    3. - novell-zenworks-configure -c ZenProbe
    4. - novell-zenworks-configure -c Start. Select the restart option to restart the services.

    When the above is finished, rerun the steps to verify it was added to your Java Truststore.

    It should have been, but it is possible that the Java with 17.4.0 is so old it may not have it, but I believe it does.

    After confirming the Java CertStore has the proper entries, do an "Update Now" on your Patch Subcription server.  When it is complete, verify the contents of your "DAU" bundles.

    --

    If you found this post useful, give it a “Like” or click on "Verify Answer" under the "More" button

    Be sure to "Like" My (and a few others) Cool Solutions below! 

    https://community.microfocus.com/members/craigdwilson/bookmarks

  • 0 in reply to   

     wrote:

    What you want to do is work top-down to detect and resolve your issues.

    #1 - When was your last Subscription Poll?  What is Successful?  This is most likely fine, there should not be any systemic issues here.  But if failing, it does not fall into any known issues.  If it is, we can loop back.

    Under Patch Management -> Status, the Signature Download shows Complete, and the Last Signature Download Time is 3:00 this morning.  (Last Patch Download is only a couple of minutes later. Last Patch Released On shows March 22nd, three days ago.)

    Under Configuration -> Patch Management -> Subscription Service Settings, Last Subscription Poll likewise shows 3:00 this morning, and Subscription Replication Status shows Complete.

    I don't know offhand where to look for a Successful status on this, although I think I do remember seeing one at least once before.

    If I go to Patch Management -> Patches, check the box for "Not Applicable", and hit Search, all the patches I'd expect do show up; they just show zero applicable devices. I interpret that as indicating that subscription updating was correct; the above timestamp information is just extra confirmation of that point.

    #2 - Is your DAU Bundle Correct? Specifically, make sure the OSPX files are referenced.  There can be known issues with older Zones where these are not downloading correctly.  Your zone definitely could meet those criteria.  If missing, we will need to rebuild your  Java Certificate Store.  (Details Later)

    OSPX.PNG

    This seems to be on the nose. Those OSPX file entries are not present in the DAU bundle. (I had examined that bundle previously, but had not had an example of "what it should look like" to compare against.)

    I take this to mean that I will need to follow the rebuild-certificate-store steps from a later reply.

    #3 - If all is good at this point, you will need to verify your test device is assigned the DAU bundle.  By default, it is assigned to all devices when the Update in Step#1 runs.  If you are testing a device with a newly created device object, it may not have the assignment until the update runs unless it is manually assigned.  be sure to do that.  In general, there are not any known issues with devices getting assignments so long as the overnight downloads are successful.

    Yeah, this bundle association is one of the first things I checked, and it's present normally.

    #4 - Make sure the device has the latest "SSU" update.  This is needed for proper detection and installation of "Microsoft Patches" but not Installers.  If the SSU is out of date, a patch will be flagged as "Not Applicable" because it is missing the Pre-req of a current SSU.  This can even impact the SSU itself, which is a MS bug that MS is working on addressing and impacts native MS patching solutions as well.  A "Workaround" patch has been created which is the "SSU" with "Workaround" in the name which installs using "Software Installer" calls instead of using an "Windows Patch" routine.

    This is not necessarily in place on some or all of the affected computers, but I suspect it's orthogonal to the issue being observed. We did deploy the February patches successfully to many (though not all) computers, and that would include the SSU. If this turns out to be an issue in addition, I can follow it up as a separate step once the primary issue is resolved.

  • 0 in reply to   
    There are entries mentioning 'The USERTRUST Network', but none of them mention 'USERTrust RSA Certification Authority' or 'Jersey'. They have CNs including 'UTN-USERFirst-Client Authentication and Email', 'UTN-USERFirst-Hardware', 'UTN-USERFirst-Object', and 'UTN - DATACorp SGC'.

    Is this sufficiently similar to qualify, or does it indicate that I probably need to rebuild this store?
  • 0   in reply to 

    Yes, I would go ahead and rebuild using all of the commands.  

    I've never seen that CAUSE an issue, but since this is a public forum.....I'm always nervous about folks just running stuff like that completely needlessly because they saw it fixed some issue for someone once

    --

    If you found this post useful, give it a “Like” or click on "Verify Answer" under the "More" button

    Be sure to "Like" My (and a few others) Cool Solutions below! 

    https://community.microfocus.com/members/craigdwilson/bookmarks

  • 0 in reply to   
    I entirely understand being cautious on that front!

    One quick additional question: we have three primary servers. Do these commands need to be run on each of them? Or on just one? Or do some need to be run on each primary, and others on just one?

    I've run through all the commands except the final restart of services on our first primary, and am waiting for confirmation that no one is in ZCC on that primary before I kick off that final step.
  • 0   in reply to 

    Just on the ZPM Primary should be sufficient, as that is the only one that will be reaching out to those servers.

    Note: If this all gets resolved, you may want to open an SR anyway to help plan out some upgrades.  ZCM 17.4.0 is EOL at the end of April, so it may not receive any more updates.

    ZCM 20.x has many nice new features.  ZCM 20.2 which will ship this summer is expected to add "Free Anti-Virus" if your ZCM License includes ZESM, which is part of the Suite.

    --

    If you found this post useful, give it a “Like” or click on "Verify Answer" under the "More" button

    Be sure to "Like" My (and a few others) Cool Solutions below! 

    https://community.microfocus.com/members/craigdwilson/bookmarks