I think that collecting errors and discussing them can be pretty helpful in terms of any product. I have this dream of finding a product that has all possible errors documented with examples. One day, one day my dream shall be fulfilled. Not any time soon though.

Thus my current approach of collecting errors I run into with any particular product and saving them to write into an article like this.

You can see some examples of those I have done for IDM drivers in the past:

• Error Codes of the eDirectory Driver for Identity Manager – Part 1

• Active Directory Driver Error Messages - Part 5
• Active Directory Driver Error Messages - Part 4
• Active Directory Driver Error Messages - Part 3
• Active Directory Driver Error Messages - Part 2
• Active Directory Driver Error Messages - Part 1

• Error Codes of the Novell Identity Manager Driver for JDBC: Part 1 of 4
• Error Codes of the Novell Identity Manager Driver for JDBC: Part 2 of 4
• Error Codes of the Novell Identity Manager Driver for JDBC: Part 3 of 4
• Error Codes of the Novell Identity Manager Driver for JDBC: Part 4 of 4

• Error Codes of the SOAP Driver - Part 1
• Error Codes of the SOAP Driver - Part 2

• Error Codes of the SAP HR driver for Identity Manager – Part 1

I highly recommend to everyone to collect errors and try to do articles like this, or blog or whatever. The goal is that when someone hits an error, and can Google it, they might find a hit if someone has written it down. You would be surprised how many messages I get by people thanking me for writing about a particular error for which they could find literally nothing else. I am glad to help, I just want others to do it as well, so when I search I find their articles.

Login for DB user failed:

This error came up when we changed the password on our database user. It turned out we had an @ (at) sign in our password which some of the tools did not like when scripting. (Watch out for dollar signs ($) especially when working with PowerShell). It was a matter of finding the encrypted password in the server.xml (/opt/netiq/idm/apps/tomcat/conf/server.xml) and there is a section with all the database connections. Each DB (igops, igara, wf, etc) all have a matching username and an associated password.

There is a tool in the /opt/netiq/idm/apps/idgov/bin/encrypt-password.sh that will generate the proper string value to insert into the nodes for each database. You can tell which DB and which user since the connect string has the database name and the username provided.

[SEVERE] 2018-07-13 09:56:49 com.netiq.iac.common.logging.IACLoggingUtils logExceptionError - [IG-DTP] Encountered unexpected error: Cannot open database "igops" requested by the login. The login failed. ClientConnectionId:6087d435-4b69-49c8-9839-4896807dcb07
[SEVERE] 2018-07-03 13:02:41 org.apache.catalina.core.StandardContext listenerStop - Exception sending context destroyed event to listener instance of class [com.netiq.iac.server.dtp.j2ee.ArcInitListener]
java.lang.IllegalStateException: com.netiq.ism.config.ConfigurationException: javax.naming.NamingException: Cannot create PoolableConnectionFactory (Login failed for user 'igops'. ClientConnectionId:15def13e-075f-4622-8582-46423d00b2b4)
        at com.netiq.ism.config.ConfigurationFactory.(ConfigurationFactory.java:71)
        at com.netiq.iac.common.j2ee.IacBaseInitListener.contextInitialized(IacBaseInitListener.java:87)
        at com.netiq.iac.server.dtp.j2ee.ArcInitListener.contextInitialized(ArcInitListener.java:95)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.netiq.ism.config.ConfigurationException: javax.naming.NamingException: Cannot create PoolableConnectionFactory (Login failed for user 'igops'. ClientConnectionId:15def13e-075f-4622-8582-46423d00b2b4)
        at com.netiq.ism.config.provider.DatabaseConfigurationProvider.getJndiDataSource(DatabaseConfigurationProvider.java:221)
        at com.netiq.ism.config.provider.DatabaseConfigurationProvider.getDataSource(DatabaseConfigurationProvider.java:197)
        at com.netiq.ism.config.provider.DatabaseConfigurationProvider.get(DatabaseConfigurationProvider.java:177)
        at com.netiq.ism.config.impl.ConfigurationImpl.addConfiguration(ConfigurationImpl.java:528)
        at com.netiq.ism.config.ConfigurationFactory.(ConfigurationFactory.java:68)
        ... 15 more
Caused by: javax.naming.NamingException: Cannot create PoolableConnectionFactory (Login failed for user 'igops'. ClientConnectionId:15def13e-075f-4622-8582-46423d00b2b4)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:856)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:173)
        at org.apache.naming.factory.ResourceLinkFactory.getObjectInstance(ResourceLinkFactory.java:152)
        at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:840)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:173)
        at org.apache.naming.SelectorContext.lookup(SelectorContext.java:163)
        at javax.naming.InitialContext.lookup(InitialContext.java:417)
        at com.netiq.ism.config.provider.DatabaseConfigurationProvider.getJndiDataSource(DatabaseConfigurationProvider.java:219)
        ... 19 more

Here the Caused by: line is helpful, that the login failed for the specific user.


Configutil won't run error:

This one was fun. Back when I was trying to fix my database issues, I thought some of the issue was that the several hundred lines of key value pairs in the global.properties file was missing in my ism-configuration.properties file. This was before I understood that Identity Governance was different than the Identity Manager User Application. In the IDM case, eDirectory acts as the common shared database that the used to share common information for a cluster. It is stored in an XML blob in an attribute on the configuration object under the AppConfig under the User App driver object.

Identity Governance however has a shared database that is always there, so it is used instead to store the common settings to all servers, and the ism-configuration.properties file is used to store local server specific settings.

But I thought that the reason I was failing to connect was missing the info so I copied all of global.properties into my ism-configuration.properties file. This did not specifically help and in the end had to be backed out as it caused more problems.

However in the process of backing out I edited the ism-configuration.properties file multiple times by hand.

After one of those edits, I started getting this, or variants of the error all over the place.

Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.netiq.ism.configuration.util.ConfigUtil.getConfiguration(ConfigUtil.java:341)
        at com.netiq.ism.configuration.util.ConfigUtil.main(ConfigUtil.java:192)
Caused by: java.lang.IllegalArgumentException: Key for add operation must be defined!
        at org.apache.commons.configuration.tree.DefaultExpressionEngine.prepareAdd(DefaultExpressionEngine.java:420)
        at org.apache.commons.configuration.HierarchicalConfiguration.addPropertyDirect(HierarchicalConfiguration.java:383)
        at org.apache.commons.configuration.AbstractConfiguration.addPropertyValues(AbstractConfiguration.java:423)
        at org.apache.commons.configuration.AbstractConfiguration.append(AbstractConfiguration.java:1271)
        at org.apache.commons.configuration.ConfigurationUtils.convertToHierarchical(ConfigurationUtils.java:252)
        at org.apache.commons.configuration.CombinedConfiguration$ConfigData.getTransformedRoot(CombinedConfiguration.java:943)
        at org.apache.commons.configuration.CombinedConfiguration.constructCombinedNode(CombinedConfiguration.java:790)
        at org.apache.commons.configuration.CombinedConfiguration.getRootNode(CombinedConfiguration.java:626)
        at org.apache.commons.configuration.HierarchicalConfiguration.fetchNodeList(HierarchicalConfiguration.java:958)
        at org.apache.commons.configuration.CombinedConfiguration.fetchNodeList(CombinedConfiguration.java:739)
	        at org.apache.commons.configuration.HierarchicalConfiguration.getProperty(HierarchicalConfiguration.java:344)
        at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1171)
        at org.apache.commons.configuration.AbstractConfiguration.getBoolean(AbstractConfiguration.java:671)
        at org.apache.commons.configuration.AbstractConfiguration.getBoolean(AbstractConfiguration.java:654)
        at com.netiq.ism.config.impl.ConfigurationImpl.getBoolean(ConfigurationImpl.java:229)
        at com.netiq.ism.config.ConfigurationFactory.(ConfigurationFactory.java:59)
        ... 2 more

I was seeing this when I tried to run configutil.sh, or when I started Tomcat and the WAR files started to deploy.

The catalina.out errors were pretty close to the same as the command line ones, which in hindsight should have pointed me back at the same root cause.

First off, when looking at a Java exception, look at the thread (main) which hints where in the code it is happening. Not useful in this case. Then look at the exception itself, and the first 'at' in the error stack.

at com.netiq.ism.configuration.util.ConfigUtil.getConfiguration(ConfigUtil.java:341)

This tells us the function getConfiguration failed. If we had access to the .java source file, it would be on line 341, which can be super helpful in tracking the issue down.

Then we get a 'Caused by:' line:

Caused by: java.lang.IllegalArgumentException: Key for add operation must be defined!

This should have been the giveaway. It actually did find a reference in the forums, but it was not helpful. Looking at some non-NetIQ sources and knowing the answer I see that the hints were there as well. But that is how it always is, isn't it? Hindsight is so much easier.

In the end, in my manual edits of the file, I had inserted a typo. Specifically I had inserted an equal sign at the beginning of a line. Thus there was a key of null, with a value that was com.netiq.something=SomeValue. Because the line was incorrectly entered as:

=com.netiq.something=SomeValue

This was a silly typo that looking at a long file was really hard to see, until you specifically start looking for it. The equal sign at the beginning was just a killer. Other special characters have the potential to cause the same issue, so take care when you need to use them.

Thus the two errors in both configutil.sh and in Tomcat loading, which have to read the config files, and throwing basically the same error should have pointed me at an issue in the config files.

Alas I did not initially see it. I did try using someone else's config file to run and it worked. Go back to mine, same error which is how we eventually figured out what was going wrong.

I think that some basic error checking in the tool would be mighty helpful to rule out bad typos like this. Perhaps even reporting the line with the error would be enough. Has the error included the line in the config file with the error i would have fixed this in seconds, instead of having to open an SR. (Now to be fair, I needed the SR for help with the database Instance on MS-SQL anyway.)

Duplicate value error:

This was interesting as an error, specifically since it was not shown in the catalina.2018-07-17.log file, the catalina log for the day, but instead was in the main catalina.out file.

To date, I had yet to see anything of value in the main catalina.out file. I am still not clear on what goes into the daily file, vs the main file. If you happen to understand what that is all about, please leave a comment on this article.

We saw this error go by during a Identity collection.

ERROR | Violation of UNIQUE KEY constraint 'ENTITY_ATTRIBUTE_JOIN_UNIQ'. Cannot insert duplicate key in object 'dbo.ENTITY_ATTRIBUTE_JOIN'. The duplicate key value is (PERMISSION, 7, 254).
 WARN | SQL Warning Code: 3621, SQLState: S0000
 WARN | The statement has been terminated.

In this case, I was able to trace the cause back to an issue I had to fix first, so once I had the problem identified, and fixed, and then found the error I knew what it was all about. As noted earlier, hindsight is so much easier than figuring it out the first time.

What happened was, in the identity source we were using, the username we told Identity Governance to import was uid, and it actually had some duplicate values. That is, there actually were several user objects with the same uid value.

This is pretty bad for many reasons, since in LDAP world uid is the uniqueID which as it is named, is meant to be unique and NOT have duplicate values.


Can't find JDBC JAR File:

When you define a database to use in your Identity Governance implementation you need to copy in a JAR file that provides the database specific JDBC driver class. In my case, I needed the MS SQL file, that Microsoft supplies. I got three different errors for it, that are all slightly different but worth recording, for anyone searching.

[WARNING] 2018-07-03 12:49:07 org.apache.naming.NamingContext lookup - Unexpected exception resolving reference
java.sql.SQLException: Cannot load JDBC driver class 'com.microsoft.sqlserver.jdbc.SQLServerDriver'
        at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createConnectionFactory(BasicDataSource.java:2145)
        at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2037)
        at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getLogWriter(BasicDataSource.java:1598)
        at org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.createDataSource(BasicDataSourceFactory.java:596)
        at org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.getObjectInstance(BasicDataSourceFactory.java:275)
        at com.netiq.tomcat.jdbc.pool.CustomBasicDataSourceFactory.getObjectInstance(CustomBasicDataSourceFactory.java:78)
        at org.apache.naming.factory.FactoryBase.getObjectInstance(FactoryBase.java:94)
        at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:840)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:827)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:173)
        at org.apache.catalina.core.NamingContextListener.addResource(NamingContextListener.java:1091)
        at org.apache.catalina.core.NamingContextListener.createNamingContext(NamingContextListener.java:661)
        at org.apache.catalina.core.NamingContextListener.lifecycleEvent(NamingContextListener.java:249)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:785)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:670)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:355)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:495)
Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createConnectionFactory(BasicDataSource.java:2137)
        ... 26 more

As usual the Caused by: line is helpful.

Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver

Could not find the class, that is needed.

It would be nice I suppose to have a table of the class names for the various JDBC classes we need to specify. Though since there are templates provided for the major database types, which preset the value I guess it is not that important. However sometimes you need to specify something different than one of the preconfigured templates and want to use the Generic JDBC template, and then you would need to know the proper class name.

[SEVERE] 2018-07-03 12:49:07 org.apache.catalina.mbeans.GlobalResourcesLifecycleListener createMBeans - Exception processing Global JNDI Resources
javax.naming.NamingException: Cannot load JDBC driver class 'com.microsoft.sqlserver.jdbc.SQLServerDriver'
        at org.apache.naming.NamingContext.lookup(NamingContext.java:856)
        at org.apache.naming.NamingContext.lookup(NamingContext.java:159)
        at org.apache.naming.NamingContextBindingsEnumeration.nextElementInternal(NamingContextBindingsEnumeration.java:117)
        at org.apache.naming.NamingContextBindingsEnumeration.next(NamingContextBindingsEnumeration.java:71)
        at org.apache.naming.NamingContextBindingsEnumeration.next(NamingContextBindingsEnumeration.java:34)
        at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.createMBeans(GlobalResourcesLifecycleListener.java:138)
        at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.createMBeans(GlobalResourcesLifecycleListener.java:145)
        at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.createMBeans(GlobalResourcesLifecycleListener.java:110)
        at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.lifecycleEvent(GlobalResourcesLifecycleListener.java:82)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:339)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:786)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:670)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:355)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:495)

This time it tells us that the JNDI interface cannot find the be found, which is slightly different, and has no Caused by: but is still obvious from the first line of the error. This is a good example of some of the diversity of Java errors you might see. It is frustrating sometimes to get a slightly different error which makes it hard to find via a Google search. Which is why I am including all three different errors I saw, for basically the same thing. Hopefully someone looking for this in Google will find one of these that matches their error.

[WARNING] 2018-07-03 12:49:08 org.apache.tomcat.util.scan.StandardJarScanner scan - Failed to scan [file:/opt/netiq/idm/apps/tomcat/lib/mssql-jdbc-6.4.0.jre8.jar] from classloader hierarchy
java.util.zip.ZipException: zip file is empty
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.(ZipFile.java:225)
        at java.util.zip.ZipFile.(ZipFile.java:155)
        at java.util.jar.JarFile.(JarFile.java:166)
        at java.util.jar.JarFile.(JarFile.java:130)
        at org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:60)
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
        at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:338)
        at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:288)
        at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1898)
        at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1126)
        at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5105)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

This time I had copied the proper file in, and during the load of the JVM it tried to load this JAR file, but it says it is empty, not because it is a 0 byte file, but instead because it was flagged with ownership of root, since I copied it in as root and forgot to do a chown novlua:novlua to the file.

Again there is no Caused by: line but the main error is pretty helpful.

Cluster cannot obtain lock:

[SEVERE] 2018-07-05 13:43:20 org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager manage - ClusterManager: Error managing cluster: Failure obtaining db row lock: Line 1: FOR UPDATE clause allowed only for DECLARE CURSOR. org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: Line 1: FOR UPDATE clause allowed only for DECLARE CURSOR. [See nested exception: com.microsoft.sqlserver.jdbc.SQL
ServerException: Line 1: FOR UPDATE clause allowed only for DECLARE CURSOR.]
        at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:157)
        at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:113)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3226)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3836)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.initialize(JobStoreSupport.java:3821)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:688)
        at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:580)
        at org.quartz.impl.StdScheduler.start(StdScheduler.java:142)
        at com.netiq.iac.persistence.service.JobScheduleService.(JobScheduleService.java:153)
        at com.netiq.iac.persistence.service.util.impl.BaseNotifyJob.(BaseNotifyJob.java:131)
        at com.netiq.iac.persistence.service.util.impl.DefaultEngine.(DefaultEngine.java:41)
        at com.netiq.iac.persistence.service.util.WorkflowInterfaceFactory.init(WorkflowInterfaceFactory.java:255)
        at com.netiq.iac.persistence.service.util.WorkflowInterfaceFactory.(WorkflowInterfaceFactory.java:224)
        at com.netiq.iac.server.j2ee.ArcServerInitListener.contextInitialized(ArcServerInitListener.java:157)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Line 1: FOR UPDATE clause allowed only for DECLARE CURSOR.
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:259)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1547)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:548)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:479)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7344)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2713)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:224)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:204)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:401)
        at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:82)
        at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:82)
        at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:96)
        ... 26 more

This was interesting. When we initially configured our instance of Identity Governance when the database was not contactable. We had permission issues on the empty DB, we had connectivity issues (instance of MS SQL but no custom listen port) and we did not properly complete the installation. We actually got it sufficiently running to load the WAR's but it did not properly run. The symptom that showed this error to be our fault was that in the database the table ISM_GLOBAL_CONFIG was empty and content was in ism-configuration.properties file instead.

I had noticed that global.properties had a ton of lines, with important configuration information that was not in the ism-configuration.properties file, as I expected based on my experience with the Identity Apps in IDM.

I was under a mistaken impression that support was able to clear up for me. The configuration in both IDM and IDG is actually stored in two places. One in the local nodes ism-configuration.properties file, with information specific to get this node up and communicating and working. Then there is a second set of data stored in a shared database that all the nodes of the potential cluster (even a cluster of one node is a cluster) is stored.

In the case of the IDM Identity Apps, they use the Identity Vaults eDirectory instance as that central database, since all nodes have to talk to it. That is why in configupdate.sh you cannot save, if the User Application object's DN is not valid and reachable. There is a configuration object with a large XML blob of text that holds the cluster node information that they all share. In the case of IDG they use the database itself and the ISM_GLOBAL_CONFIG table.

eDirectory being infinitely expandable and extensible makes it the obvious choice when you have to share data across many nodes. But since IDG is designed to work Active Directory as well, and those guys are so touchy about schema extensions, it is not worth the fuss and muss to deal with them.

There is a post install script that calls configutil.sh with a -script command which imports the settings in Global.properties into the database and we had missed that step.

Thus the error, "ClusterManager: Error managing cluster: Failure obtaining db row lock: Line 1: FOR UPDATE" the table and data it was looking at was empty. Once we fixed it by simply running the script that imports the settings it worked:

/opt/netiq/idm/apps/jre/bin/java" -Djava.util.logging.config.file="/opt/netiq/idm/apps/idgov/conf/logging.properties"  -Dcom.netiq.ism.config=/opt/netiq/idm/apps/tomcat/conf/ism-configuration.properties -jar /opt/netiq/idm/apps/idgov/lib/ig-configutil.jar -script "/opt/netiq/idm/apps/idgov/scripts/import-configs-init.script

The error was annoying since it repeated every 30 seconds or one minute, so it filled the logs and made it hard to read. Of course it did actually have the hint as to the root cause, which alas I only recognized after I had the solution in hand. How often is that the case for you? But it is good to look back and see where you could have figured it out sooner as it gives you insight into the processes and how things ought to work. And watching how they fail also can give you insight.

I did not actually realize it ran as a cluster of one, until I ran into this error. That actually makes sense and is a simpler config than having a clustered and non-clustered config. But I only noticed because of the error.

Error Codes for Identity Governance - Part 2