I have been discussing the Identity Governance product in terms of interesting things I have noticed using it.

I discussed the product, how to enable logging, and the identified the main configuration tools in the first article.

In the second article I discussed the configutil.sh file that is used to manage the Identity Governance side of the product. (This is as distinct from the OSP part, which is configured with configupdate.sh as before).

In the third article I discussed some real world issues I ran into, collecting information from attributes that held multiple values when IDG expected single values and how this broke the Display Name attribute, which is unique and different than all the other attributes.

I have a couple of issues I ran into that are worth talking about.

Testing transformations:

If you look at the documentation there is a link to a document (PDF only) that offers some examples of ECMA transformations you can apply when trying to manage the incoming data. I discussed an example I ran into, in terms of importing data from attributes that have some multi valued instances into a data field expecting a single value..

I discussed some of the issues I ran into as I tried to get it to work in the previous article, but I had noticed in the PDF document on transformations, it mentioned on the second page a Transformation Testing tool.

https://www.netiq.com/documentation/identity-governance-30/pdfdoc/references/Collector-Data-Transformation.pdf

I was discussing this with a NetIQ engineer and was confident I had seen this tool, but could not remember where or how. However in writing my previous articles I noticed the file:
/opt/netiq/idm/apps/idgov/bin/transform-tester.sh

I ran it command line and nothing happened. I ran it on a system with X windows properly configured and I got the UI that is shown in the PDF document on Data Transformation.



I edited the file /home/gcarman/testing/transform1.txt and put in the ECMA from the last article:

if( inputValue[0]=='[' ) {
                        var jsonObj = JSON.parse(inputValue);
                        result=jsonObj[0];
                        } else {
                        result = inputValue;
        }
outputValue=result;

Then I provided the value of Geoffrey and it returned Geoffrey as expected.

I took the JSON array example I used in the previous article of:

["Geoffrey1","Geoff","Jeff"]

I modified the first value to make sure it was different than the previous test and it returned the value of Geoffrey1.

I actually tested the above ECMA using Designer's ECMA simulator to get it working the first time, since I had not gotten the Transform tester working initially. If you open an ECMAScript object in Designer (It is really a DirXML-Resource with a DirXML-ContentType attribute of text/ecmascript;charset=UTF-8 which Designer will open in the ECMA Script editor) it gives you the ECMA in the upper right window. The left side has all the function and variables so you can jump around within the ECMA. However most people miss there is a window at the bottom, with a command line prompt, so indicated by a greater than sign. "&grt;"

It turns out you can test your ECMA from that object in that lower window.

So I defined a function to wrap the ECMA and tested it until I got it to work.

That approach is a tiny bit more interactive, but this DaaS Transformation Test Utility is better for ensuring it will really work in Identity Governance.

There is also an Identity Governance Connector SDK tool available for download with the 3.0 product, but I have yet to get far enough along to talk about that tool. Still looking into it, and I shall report back when I get more insight.

Configutil won't run error:

I think that collecting errors and discussing them can be pretty helpful in terms of any product. I have this dream of finding a product that has all possible errors documented with examples. One day, one day my dream shall be fulfilled. Not any time soon though.

Thus my current approach of collecting errors I run into with any particular product and saving them to write into an article like this.

You can see some examples of those I have done for IDM drivers in the past:

• • https://www.netiq.com/communities/cool-solutions/error-codes-edirectory-driver-identity-manager-part-1/

• • https://www.netiq.com/communities/cool-solutions/series/active-directory-driver-error-messages/

• • https://www.netiq.com/communities/cool-solutions/series/error-codes-of-the-novell-identity-manager-driver-for-jdbc/

• • http://www.novell.com/communities/node/12152/error-codes-soap-driver-part-1

• • http://www.novell.com/communities/node/12158/error-codes-soap-driver-part-2

• http://www.novell.com/communities/node/9271/error-codes-sap-hr-driver-identity-manager-part-1

I highly recommend to everyone to collect errors and try to do articles like this, or blog or whatever. The goal is that when someone hits an error, and can Google it, they might find a hit if someone has written it down. You would be surprised how many messages I get by people thanking me for writing about a particular error for which they could find literally nothing else. I am glad to help, I just want others to do it as well, so when I search I find their articles.

This one was fun. Back when I was trying to fix my database issues, I thought some of the issue was that the several hundred lines of key value pairs in the global.properties file was missing in my ism-configuration.properties file. This was before I understood that Identity Governance was different than the Identity Manager User Application. In the IDM case, eDirectory acts as the common shared database that the used to share common information for a cluster. It is stored in an XML blob in an attribute on the configuration object under the AppConfig under the User App driver object.

Identity Governance however has a shared database that is always there, so it is used instead to store the common settings to all servers, and the ism-configuration.properties file is used to store local server specific settings.

But I thought that the reason I was failing to connect was missing the info so I copied all of global.properties into my ism-configuration.properties file. This did not specifically help and in the end had to be backed out as it caused more problems.

However in the process of backing out I edited the ism-configuration.properties file multiple times by hand.

After one of those edits, I started getting this, or variants of the error all over the place.

Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.netiq.ism.configuration.util.ConfigUtil.getConfiguration(ConfigUtil.java:341)
        at com.netiq.ism.configuration.util.ConfigUtil.main(ConfigUtil.java:192)
Caused by: java.lang.IllegalArgumentException: Key for add operation must be defined!
        at org.apache.commons.configuration.tree.DefaultExpressionEngine.prepareAdd(DefaultExpressionEngine.java:420)
        at org.apache.commons.configuration.HierarchicalConfiguration.addPropertyDirect(HierarchicalConfiguration.java:383)
        at org.apache.commons.configuration.AbstractConfiguration.addPropertyValues(AbstractConfiguration.java:423)
        at org.apache.commons.configuration.AbstractConfiguration.append(AbstractConfiguration.java:1271)
        at org.apache.commons.configuration.ConfigurationUtils.convertToHierarchical(ConfigurationUtils.java:252)
        at org.apache.commons.configuration.CombinedConfiguration$ConfigData.getTransformedRoot(CombinedConfiguration.java:943)
        at org.apache.commons.configuration.CombinedConfiguration.constructCombinedNode(CombinedConfiguration.java:790)
        at org.apache.commons.configuration.CombinedConfiguration.getRootNode(CombinedConfiguration.java:626)
        at org.apache.commons.configuration.HierarchicalConfiguration.fetchNodeList(HierarchicalConfiguration.java:958)
        at org.apache.commons.configuration.CombinedConfiguration.fetchNodeList(CombinedConfiguration.java:739)
	        at org.apache.commons.configuration.HierarchicalConfiguration.getProperty(HierarchicalConfiguration.java:344)
        at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1171)
        at org.apache.commons.configuration.AbstractConfiguration.getBoolean(AbstractConfiguration.java:671)
        at org.apache.commons.configuration.AbstractConfiguration.getBoolean(AbstractConfiguration.java:654)
        at com.netiq.ism.config.impl.ConfigurationImpl.getBoolean(ConfigurationImpl.java:229)
        at com.netiq.ism.config.ConfigurationFactory.(ConfigurationFactory.java:59)
        ... 2 more

I was seeing this when I tried to run configutil.sh, or when I started Tomcat and the WAR files started to deploy.

The catalina.out errors were pretty close to the same as the command line ones, which in hindsight should have pointed me back at the same root cause.

First off, when looking at a Java exception, look at the thread (main) which hints where in the code it is happening. Not useful in this case. Then look at the exception itself, and the first 'at' in the error stack.

at com.netiq.ism.configuration.util.ConfigUtil.getConfiguration(ConfigUtil.java:341)

This tells us the function getConfiguration failed. If we had access to the .java source file, it would be on line 341, which can be super helpful in tracking the issue down.

Then we get a 'Caused by:' line:

Caused by: java.lang.IllegalArgumentException: Key for add operation must be defined!

This should have been the giveaway. It actually did find a reference in the forums, but it was not helpful. Looking at some non-NetIQ sources and knowing the answer I see that the hints were there as well. But that is how it always is, isn't it? Hindsight is so much easier.

In the end, in my manual edits of the file, I had inserted a typo. Specifically I had inserted an equal sign at the beginning of a line. Thus there was a key of null, with a value that was com.netiq.something=SomeValue. Because the line was incorrectly entered as:

=com.netiq.something=SomeValue

This was a silly typo that looking at a long file was really hard to see, until you specifically start looking for it. The equal sign at the beginning was just a killer. Other special characters have the potential to cause the same issue, so take care when you need to use them.

Thus the two errors in both configutil.sh and in Tomcat loading, which have to read the config files, and throwing basically the same error should have pointed me at an issue in the config files.

Alas I did not initially see it. I did try using someone else's config file to run and it worked. Go back to mine, same error which is how we eventually figured out what was going wrong.

I think that some basic error checking in the tool would be mighty helpful to rule out bad typos like this. Perhaps even reporting the line with the error would be enough. Has the error included the line in the config file with the error i would have fixed this in seconds, instead of having to open an SR. (Now to be fair, I needed the SR for help with the database Instance on MS-SQL anyway.)

Duplicate value error:

This was interesting as an error, specifically since it was not shown in the catalina.2018-07-17.log file, the catalina log for the day, but instead was in the main catalina.out file.

To date, I had yet to see anything of value in the main catalina.out file. I am still not clear on what goes into the daily file, vs the main file. If you happen to understand what that is all about, please leave a comment on this article.

We saw this error go by during a Identity collection.

ERROR | Violation of UNIQUE KEY constraint 'ENTITY_ATTRIBUTE_JOIN_UNIQ'. Cannot insert duplicate key in object 'dbo.ENTITY_ATTRIBUTE_JOIN'. The duplicate key value is (PERMISSION, 7, 254).
 WARN | SQL Warning Code: 3621, SQLState: S0000
 WARN | The statement has been terminated.

In this case, I was able to trace the cause back to an issue I had to fix first, so once I had the problem identified, and fixed, and then found the error I knew what it was all about. As noted earlier, hindsight is so much easier than figuring it out the first time.

What happened was, in the identity source we were using, the username we told Identity Governance to import was uid, and it actually had some duplicate values. That is, there actually were several user objects with the same uid value.

This is pretty bad for many reasons, since in LDAP world uid is the uniqueID which as it is named, is meant to be unique and NOT have duplicate values.

Now finding examples of this on your own, with simply an LDAP browser is pretty hard. However, there is a super neat tool called Console 2 by a developer named Aleksandar Mujadin available at his web site: https://sneakycat.biz

If you get the latest version (3.61 as I write this) under the Extra menu there is a Non Unique Value finder.



I ran it on this tree, and it found a bunch, I trim down the output for simplicity down to 1 example.

#LDAP filter used: (&(uid=*))
#Base DN: dc=edu
#Number of objects found containing attribute uid: 59476
#Number of duplicate values found: 1
Attribute value;DN containing attribute value
0001A1;cn=0001A1,ou=ACTIVE,ou=PERSON,dc=acme,dc=edu
0001A1;cn=0001A1,ou=INACTIVE,ou=PERSON,dc=acme,dc=edu
#Total number of duplicate rows: 2

I have to say it ran in under a second, and it looked at 60,000 some odd objects, so this tool truly rocks!

This made tracking down the error cases really simple. Console2 is pretty darn awesome so I highly recommend you go get it, if you do anything with IDM.

Just for fun, back to my issue with multi valued attributes, how might you find those? Again, via a simple LDAP browser this is non-trivial. But again, Alekz included it in his amazing tool.

This time you go to the Extras menu, and select Value Count Report item, which opens the Multi Value Finder tool.



I ran it with sn,2 as the parameter, saying find me all sn attribute values with 2 or more values.

That found 362 out of 60,000 again, this time it took about 20 seconds to run, so again mighty fast all things considered. It generated an LDIF showing all the objects and values.

# This LDIF file was generated by the LDIF APIs. of Novell's Java LDAP SDK
version: 1

# C2 MultiValueFinder, looking for entries using the following LDAP filter: (&(sn=*))
# This file will contain entries found after filtering based on the value count specified.
# Looking for attribute: sn with minimum value count: 2
# sn: 2
dn: uid=HY0114T,ou=ACTIVE,ou=PERSON,dc=acme,dc=edu
sn: Smith
sn: Jones

Overall, a super useful tool, that makes some very hard things, trivial. What a pleasure to work with.

Anyway, that is about it for now. I will collect more errors and try to write them up as I run into them with Identity Governance.