Wikis - Page

Creating Dynamic Groups in the IDM Engine

1 Likes

I was recently asked to help troubleshoot a perceived problem with Dynamic Groups.  While at first I thought the problem was non-working dynamic groups, in actuality the problem was less about Dynamic Groups on their own and more about IDM working with them, specifically creating them or setting their memberQuery attribute; the memberQuery attribute holds the dynamic part of the group, telling eDirectory how to find members of the group in a query-like way.

This problem had come up before ( https://forums.novell.com/showthread.php/483776-Re-Create-DynamicGroups-with-a-Driver ), but never needed solving by me with IDM, so with eDirectory's LDAP interface available it could always be solved by setting memberQueryURL (the LDAP view of eDirectory's memberQuery attribute) which somehow gets translated into eDirectory-speak on the backend via something I have never researched but which "Just Works" (tm) reliably.  Since a memberQueryURL can be set with a standard LDAP URI doing the same with IDM seemed likely to succeed since usually the eDirectory LDAP interface does not translate values of attributes as they go through it.  Had that been the case I probably never would have been called since that is exactly what was being tried with something like the following URI:

ldap://DC=user,O=company,DC=org??sub?(&(costCenter=123456))


After IDM writes this it would seem that the group should work for any matching objects, assuming nothing else is missing.  A common omission on dynamic groups is the dgIdentity attribute which sets the permissions that the group uses when executing its search.  For some simple filters this is not necessary, for example if all of the components of the filter are publicly-readable, but costCenter is not public, so in this case we would need to set the dgIdentity attribute to something that could read dgIdentity in the contexts setup for query by the dynamic group.  Setting that appropriately, though, still resulted in a dynamic group sans dynamic members.

We looked at the memberQueryURL attribute from LDAP as it had been set by IDM and it had a value like this:

memberQueryURL:: bGRhcDovLy9EQz11c2VyLE89Y29tcGFueSxEQz1vcmc/P3N1Yj8oJihjb3N0Q2VudGVyPTEyMzQ1NikpCg==


Those familiar with LDAP will recognize the double-colons at the end indicating that the following value, on however many lines, should be combined into one line and is then base64-encoded, meaning for us to have a chance of understanding it easily it must be decoded.  Linux has a base64 command built in so that's easy to do:

> echo bGRhcDovLy9EQz11c2VyLE89Y29tcGFueSxEQz1vcmc/P3N1Yj8oJihjb3N0Q2VudGVyPTEyMzQ1NikpCg== | base64 -d
ldap:///DC=user,O=company,DC=org??sub?(&(costCenter=123456))


The next troubleshooting step was to create a matching dynamicGroup via some supported method, for example using iManager.  Doing so allowed us to see that the query works, the context is right, the dgIdentity attribute's referenced object had rights to find things, and so on.  Dynamic member showed up nicely in both iManager and via queries to the group via LDAP, so clearly the problem was somehow on the interaction between IDM and the object as it was changing it.  As a relevant note, if IDM set the filter in a way we thought would work, but clearly did not, iManager somehow knew it was wrong and helpfully replaced the filter in its view to the administrator with a default (&(objectClass=*)) filter, which further-confused the issue because it was nothing at all like the filter we knew was successfully (per the IDM trace) written to the directory.  End-user tools, including those for administrators and particular any with Graphical User Interfaces (GUI), are often inferior to tools geared toward troubleshooting as they try to be helpful and fix things that are wrong.

While LDAP is a great tool for interacting with eDirectory, including many kinds of troubleshooting, it is still just one of multiple interfaces into the director; others include NCP and the view provided via HTTPS by iMonitor.  iMonitor specifically was designed around, as you may guess from the name, monitoring eDirectory; it is a troubleshooting tool from the start and that is its entire purpose still today.  Developed to fill a need within (at the time) Novell, it was eventually made part of the product because others would probably need it as well, and tools like DSBROWSE.NLM were only available on NetWare, not the newer Linux and Unix platforms available and rapidly growing in popularity.  For more on iMonitor, some documentation is here, or otherwise found as part of the eDirectory Administration Guide:  https://www.netiq.com/documentation/edirectory-9/edir_admin/data/b1gkpdzf.html

Accessing a dynamic group via iMonitor is a fascinating experience, well worth the effort; the only issue I had was when I set my working dynamic group to include everything in the tree, which caused iMonitor to return over a million members of my group, and that was painful for my browser to render.  Using a sensible group for troubleshooting is a good takeaway from that; worry about scale later and for now get it working with something simple.

The memberQuery attribute as viewed in iMonitor is a binary thing, not a nice string looking like an LDAP URI or filter.  This was the first big clue pointing out that eDirectory and iManager are doing something sneaky behind the scenes when setting this value on these objects.  The data I have from a working, but unrelated, dynamic group follows:

00000     00 00 00 00 02 00 00 00 2C 00 00 00 4F 00 3D 00     ........,...O.=.
00010     6E 00 6F 00 76 00 65 00 6C 00 6C 00 2E 00 74 00     n.o.v.e.l.l...t.
00020     3D 00 47 00 57 00 41 00 50 00 50 00 53 00 54 00     =.G.W.A.P.P.S.T.
00030     52 00 45 00 45 00 00 00 02 00 00 00 01 00 00 00     R.E.E...........
00040     00 00 00 00 07 00 00 00 12 00 00 00 75 00 6E 00     ............u.n.
00050     69 00 71 00 75 00 65 00 49 00 44 00 00 00 00 00     i.q.u.e.I.D.....
00060     10 00 00 00 74 00 65 00 73 00 74 00 30 00 30 00     ....t.e.s.t.0.0.
00070     2A 00 00 00     *...


A couple things should stand out from this.  First, most of it is not ASCII.  Second, the parts that are ASCII have null bytes between them, so this is using two bytes for every character, which is not what IDM was setting in there.  Third, there's a tree component in there; do not ask me why, as it seems completely redundant, but it's definitely there.  Fourth, there are no obvious equal signs within the filter portion, or ampersands to indicate multiple components are required to satisfy the LDAP filter's requirements, or a 'sub' to indicate this is a subtree search, etc..  In other words, this is a completely separate value that has a relationship to what we specified, but is definitely not identical.

Before moving on to how this was overcome, a brief aside on IDM may be useful.  IDM works at a very low level of eDirectory out of necessity.  This is why we have wonderful things like an event-driven identity management system.  This is why we can override password policies in the Identity Vault (IDV) if we must, or why we are able to synchronize passwords out to any system in the world that accepts passwords (because IDM can access the password unlike many things), or synchronize passwords in any other form we can generate from the user's original password value.  This is why IDM scales well entirely based on transactions per period of time rather than being limited by the size of the directory alone.  On the other hand, this is also why IDM is able to do things we may not want, such as skip the helpful parts of eDirectory and its LDAP interface that deal with the memberQuery attribute and its odd form as shown above.  The IDM engine's architecture is amazing, but with great power comes great responsibility; this should be more a theme for products coming from Micro Focus than from some comic book or movie series, as it is what we have always had from the company: products that will do anything and scale while doing it, but needing understanding.

Going back to the problem at hand, there are at least two ways to solve it; the easy way is probably to just use LDAP via a call from ECMAScript in the engine to set the value, but this requires a couple things that may not always be there, or which may have problems for one reason or another, and which certainly are going to be slower than using the engine natively: LDAP.  The LDAP interface on a particular engine box is not required to be operational for IDM to function.  Making the change would require a user with rights, preferably authenticating via TLS which means having the IDM engine trust the certificate presented by eDirectory or, terribly, ignoring security entirely.  As much as I love LDAP, this seemed like a challenge worth figuring out in policy, at least for this simple example.

The next task was to reverse-engineer what we could see from a working dynamic group and what actually showed up in the attribute value as shown by iMonitor.  In the example above I was setting uid=test00 and a base context of o=novell which shows at least two things right away: first, the attribute representation is the native eDirectory (not LDAP) name, or else this would have been uid=test00 and not uniqueID=test00; second, the context both includes the tree portion and is in Fully-Qualified Dot Notation (FQDN) and not LDAP (comma-delimited) notation.  Remember that behind all of this is eDirectory, not its LDAP interface, and we see that clearly here.

Some other conclusions reached right away: every character that is used throughout must be two bytes with a null as the second byte; I am not doing anything with double-byte character sets, but if I was maybe that would change.  In the meantime, whatever solution we have via IDM must make sure after each regular ASCII character is a null.

After that we have a problem on our hands: what is the rest of that stuff in there?  Null byte after null byte scattered through, a myriad of other non-printable characters, and no obvious sign of binary logic that should be there.  I felt the first step was to verify that IDM could, in fact, set something properly in my tree.  My attempt was to take the value there, code the entire thing into policy, and see if it could both be written to eDirectory and then be used by clients (iMonitor, iManager, LDAP) that wanted to get members based on that.  IDM has the ability to do byte-by-byte stuff with a 'char' token, so I spent way too much time creating just the eighty-eight char calls in order to duplicate that value exactly in a new dynamic group; yes, this is as ugly as it looks, because we're reverse-engineering and proving a theory before investing tons of time in something that may never work:

 <rule>
        <description>Create Dynanmic Group</description>
        <comment xml:space="preserve">Tricky, because IDM is low-level, and other interfaces (iManager, LDAP) show values that are directly able to be manipulated in normal ways without factoring in the pesky tree field.</comment>
        <comment name="author" xml:space="preserve">Aaron Burgemeister (aburgemeister@gmail.com, ab@a2btech.com)</comment>
        <comment name="version" xml:space="preserve">0.1.20170510162800</comment>
        <comment name="lastchanged" xml:space="preserve">2017-05-10T16:28:00</comment>
        <conditions>
            <and/>
        </conditions>
        <actions>
            <do-add-src-object class-name="dynamicGroup">
                <arg-dn>
                    <token-text xml:space="preserve">org\novell\group\dynamictest00</token-text>
                </arg-dn>
            </do-add-src-object>
            <do-add-src-attr-value name="memberQuery">
                <arg-dn>
                    <token-text xml:space="preserve">org\novell\group\dynamictest00</token-text>
                </arg-dn>
                <arg-value type="octet">
                    <token-base64-encode>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="2"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="44"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="79"/>
                        <token-char value="0"/>
                        <token-char value="61"/>
                        <token-char value="0"/>
                        <token-char value="110"/>
                        <token-char value="0"/>
                        <token-char value="111"/>
                        <token-char value="0"/>
                        <token-char value="118"/>
                        <token-char value="0"/>
                        <token-char value="101"/>
                        <token-char value="0"/>
                        <token-char value="108"/>
                        <token-char value="0"/>
                        <token-char value="108"/>
                        <token-char value="0"/>
                        <token-char value="46"/>
                        <token-char value="0"/>
                        <token-char value="116"/>
                        <token-char value="0"/>
                        <token-char value="61"/>
                        <token-char value="0"/>
                        <token-char value="71"/>
                        <token-char value="0"/>
                        <token-char value="87"/>
                        <token-char value="0"/>
                        <token-char value="65"/>
                        <token-char value="0"/>
                        <token-char value="80"/>
                        <token-char value="0"/>
                        <token-char value="80"/>
                        <token-char value="0"/>
                        <token-char value="83"/>
                        <token-char value="0"/>
                        <token-char value="84"/>
                        <token-char value="0"/>
                        <token-char value="82"/>
                        <token-char value="0"/>
                        <token-char value="69"/>
                        <token-char value="0"/>
                        <token-char value="69"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="2"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="1"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="7"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="18"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="117"/>
                        <token-char value="0"/>
                        <token-char value="110"/>
                        <token-char value="0"/>
                        <token-char value="105"/>
                        <token-char value="0"/>
                        <token-char value="113"/>
                        <token-char value="0"/>
                        <token-char value="117"/>
                        <token-char value="0"/>
                        <token-char value="101"/>
                        <token-char value="0"/>
                        <token-char value="73"/>
                        <token-char value="0"/>
                        <token-char value="68"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="16"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="116"/>
                        <token-char value="0"/>
                        <token-char value="101"/>
                        <token-char value="0"/>
                        <token-char value="115"/>
                        <token-char value="0"/>
                        <token-char value="116"/>
                        <token-char value="0"/>
                        <token-char value="48"/>
                        <token-char value="0"/>
                        <token-char value="48"/>
                        <token-char value="0"/>
                        <token-char value="42"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                        <token-char value="0"/>
                    </token-base64-encode>
                </arg-value>
            </do-add-src-attr-value>
        </actions>
    </rule>


Notice that the char token takes decimal, not hexadecimal, values which is why I have things like '44' instead of '2C' in there.  Thankfully a bit of scripting helped with that, and the end result was a dynamic group that was able to show members who had a uniqueID attribute value matching 'test00*' (thus test00, test000, test001, test0010, etc.).  With this test out of the way the next step was to try to make it flexible so that it could work in other trees, other context, with other attributes and other values.  While not wanting to use LDAP, I did switch over to ECMAScript for the remainder of the creation of the memberQuery attribute's value as it is a more-flexible language.

Let's compare the iMonitor memberQuery representation above with another dynamic group's memberQuery that works:

00000     00 00 00 00 02 00 00 00 2C 00 00 00 4F 00 3D 00     ........,...O.=.
00010     6E 00 6F 00 76 00 65 00 6C 00 6C 00 2E 00 74 00     n.o.v.e.l.l...t.
00020     3D 00 47 00 57 00 41 00 50 00 50 00 53 00 54 00     =.G.W.A.P.P.S.T.
00030     52 00 45 00 45 00 00 00 02 00 00 00 01 00 00 00     R.E.E...........
00040     00 00 00 00 07 00 00 00 12 00 00 00 75 00 6E 00     ............u.n.
00050     69 00 71 00 75 00 65 00 49 00 44 00 00 00 00 00     i.q.u.e.I.D.....
00060     10 00 00 00 74 00 65 00 73 00 74 00 30 00 30 00     ....t.e.s.t.0.0.
00070     2A 00 00 00     *...

00000     00 00 00 00 02 00 00 00 2C 00 00 00 4F 00 3D 00     ........,...O.=.
00010     6E 00 6F 00 76 00 65 00 6C 00 6C 00 2E 00 74 00     n.o.v.e.l.l...t.
00020     3D 00 47 00 57 00 41 00 50 00 50 00 53 00 54 00     =.G.W.A.P.P.S.T.
00030     52 00 45 00 45 00 00 00 02 00 00 00 01 00 00 00     R.E.E...........
00040     00 00 00 00 07 00 00 00 06 00 00 00 43 00 4E 00     ............C.N.
00050     00 00 00 00 10 00 00 00 74 00 65 00 73 00 74 00     ........t.e.s.t.
00060     30 00 30 00 2A 00 00 00     0.0.*...


A lot of things are the same, but some are definitely different, and those are parts we need to better-understand in order to create a solution that generates the memberQuery value dynamically.  The first four lines, in fact, are identical, and while some of that is the context and another part is the tree name, at least it means I can focus on the latter portion in order to get the attribute/value part worked out.  The first differing byte is 0x49 and it is a non-printable character (if translated to ASCII), and it is clearly not either the attribute name or value.  Since the filters were in all other ways identical, this was a clue to me that this may indicate how many bytes may be following as part of this query filter component.

Computers did not originally use delimiters to determine how far to read for a field of data, or a bunch of memory, because it was too expensive and too restrictive.  Input to computers started out in a fixed-width fashion, not a character-separated values (CSV) fashion, so you ended up with data input that would take exactly ten bytes for a field, then five bytes for the next, then twenty for the next, and so on.  CSV was a big change that required a lot more processing and introduced other complexities because of the need to escape delimiters when they were actually in values.  I relate all of this to indicate that byte-offsets are still used in a fashion more like fixed-width fields than the delimiters we humans like to see and use.  The performance limitations from decades ago are hilarious compared to the resources today in any computing device available; my Android-based phone, a Nexus 6P, has more processing power than all of the world combined a half-century ago; times have changed I guess.

In this case, the 0x12 and 0x06 values seen in the data at position 0x49 indicate how many bytes should be read for whatever is next.  "Whatever is next" in this case happens to be through the end of the attribute name, including its null bytes after each character.  For example, six bytes after 0x06 includes 0x00, 0x00, 0x00, 0x43, 0x00, 0x4E, 0x00, and eighteen bytes after 0x12 (0x12 == eighteen (18) in decimal) gives you the following:  0x00, 0x00, 0x00, 0x75, 0x00, 0x6E, 0x00, 0x69, 0x00, 0x71, 0x00, 0x75, 0x00, 0x65, 0x00, 0x49, 0x00, 0x44, 0x00

In both cases you can easily see that the offset includes through the end of the attribute name, whether 'uniqueID' or 'CN'.  Simple enough, the code that is created to generate this big memberQuery value will need to know the length of the attribute name and then set an offset prior to placing the attribute, with two bytes per character in its name, after it.

Next we have a few more 0x00 bytes (four of them) and then wee reach 0x10.  From the values above you cannot tell this is another offset because it is identical in both cases; in my testing I changed the value being used in the filter to test000* and suddenly 0x10 became 0x12, showing that it was another offset, this time for the attribute value instead of the attribute name.

After 0x10 (representing sixteen (16) in decimal) we have three 0x00 bytes again followed by the actual value involved: 0x74, 0x00, 0x65, 0x00, 0x73, 0x00, 0x74, 0x00, 0x30, 0x00, 0x30, 0x00, 0x2A, 0x00.

After all of that, we have a couple 0x00s to finish the value out.  I later found out that the entire value length must be evenly divisible by four, so I had to add a condition in my ECMAscript that took the total length of everything up until the end, see if it was evenly divisible by four, and if not pad the end with a couple more 0x00 bytes or else the whole thing would be invalid.  If all of this does not make you curious why it is encoded this way rather than just being a standard LDAP URI then you may not understand how I feel; it is probably a performance thing, or a native compatibility with eDirectory thing, or a need to pre-screen for certain incompatible combinations thing, but I would really like to know the specifics behind this implementation, having it work this way rather than parsing the LDAP filter on the fly.

All that analysis gets us the attribute name and attribute value figured out if implemented properly, but what about the context and tree name?  It turns out those are setup the same way.  The 0x2c that is in the first line of the memberQuery value (as shown in iMonitor) represents the distance, in bytes, to the end of the context that includes the tree and is in FQDN form (dotted, not commas).  0x2C is forty-four (44) bytes, which if you count down a couple rows (0x20) and then count twelve bytes (0x0C) to the right you end up at the end of the tree name after the context and tree are printed with their 0x00's after each character.  Simple enough, as long as we do the calculations properly.

There are some parts I have not worked out yet, and would invite others to test, specifically the very simple nature of the filter having one condition with an attribute and value portion and nothing else.  This would not work for things like the following or anything more complex:

#Users whose UID starts with 'x-adm'
(&(objectClass=inetOrgPerson)(uid=x-adm*))

#Users whose UID starts with either 'ab' or 'cd'
(|(uid=ab*)(uid=cd*))

#Users whose UID does NOT start with 'test'
(!(uid=test*))


I suspect some of the "static" bits, per my ECMAScript below, are actually hiding some of the details of AND/OR/NOT and those need to be worked out.  In the meantime, this works well for arguments sent in.

First, here's the policy that can be used to call it, though with your own context, attribute name, and filter as desired:

        <actions>
            <do-add-src-object class-name="dynamicGroup">
                <arg-dn>
                    <token-text xml:space="preserve">org\novell\group\dynamictest00</token-text>
                </arg-dn>
            </do-add-src-object>
            <do-set-local-variable name="attrValue" scope="policy">
                <arg-string>
                    <token-op-attr name="Description"/>
                </arg-string>
            </do-set-local-variable>
            <do-set-local-variable name="context" scope="policy">
                <arg-string>
                    <token-text xml:space="preserve">O=novell</token-text>
                </arg-string>
            </do-set-local-variable>
            <do-set-local-variable name="attrNameVar" scope="policy">
                <arg-string>
                    <token-text xml:space="preserve">uniqueID</token-text>
                </arg-string>
            </do-set-local-variable>
            <do-add-src-attr-value name="memberQuery">
                <arg-dn>
                    <token-text xml:space="preserve">org\novell\group\dynamictest00</token-text>
                </arg-dn>
                <arg-value type="octet">
                    <token-xpath expression="es:buildDynamicGroupMemberQuery(string($dirxml.auto.treename), string($context), string($attrNameVar), string($attrValue))"/>
                </arg-value>
            </do-add-src-attr-value>
        </actions>


IDM helpfully provides a Global Configuration Variable (GVC) with the tree name, so passing that GCV's value dynamically is easy.  All the user needs to specify is a context in dotted FQDN (not LDAP) format, so something like OU=adm.O=novell and NOT OU=adm,O=novell as the script will not try to correct an error of that kind, and the resulting dynamic group will NOT work.

The tall that actually calls the ECMAscript is this one:

                    <token-xpath expression="es:buildDynamicGroupMemberQuery(string($dirxml.auto.treename), string($context), string($attrNameVar), string($attrValue))"/>


The first parameter is the treename GCV, followed by something holding the desired context, followed by something holding the name of an attribute value (set however you see fit), followed by something holding the value to be used with that attribute, which in my case was 'test00*' including the asterisk on the end.  The ECMAScript method returns a big long base64-encoded string which is intended to be written back to the new dynamic group object as an octet/binary type of data.  If you use the action above, you may need to add a section that adds an identity with rights to execute the query involved; my query only used the publicly-readable uniqueID attribute (UID as seen by LDAP) so I needed no additional rights for it to work.

That part is not that unusual, and the real magic is happening in the ECMAscript.  The following has a lot of commented-out lines that I was using as I reverse-engineered the value, so feel free to strip those out, though I suspect they may help with further development as we figure out how to do other types of operations with a smarter version of this method in the future.

Also, ugly as it is, there are a lot of individual concatenations to the ByteArrayOutputStream object rather than appending a longer created byte array which would probably be marginally faster.  The reason I have not combined that is just like above: I plan on continuing this when I have a moment, or letting others do that and respond with their developments, and combining a bunch of possibly changing bytes together just makes one need to split them out again.  At the end of the day if this impacts performance enough to cause you a problem, you may be generating or modifying too many dynamic groups to be completely sane. 

importClass(Packages.com.novell.xml.util.Base64Codec);
importClass(java.io.ByteArrayOutputStream);
importClass(java.lang.System);
importClass(java.lang.Integer);

var JString = java.lang.String;

function buildDynamicGroupMemberQuery(tree, context, attrName, attrValue) {
  //00 00 00 00 02 00 00 00 2C 00 00 00
  //O = n o v e l l . t = g w a p p s t r e e
  //00 00 02 00 00 00 01 00 00 00
  //00 00 00 00 07 00 00 00 12 00 00 00
  //eDir-attr-name
  //00 00 00 00 10 00 00 00
  //eDir-attr-value
  //2A 00 00 00

  //Setup a default encoding if nothing is specified previously.
  encoding = System.getProperty("file.encoding")

  var byteData = new ByteArrayOutputStream();
  //outputStream.write(a);
  //outputStream.write(b);

  //Setting up the static part of the value.
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(2);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  //byteData.write(hex2dec("2C"));  //Some kind of byte offset for context/tree length.  For 18 characters (36 bytes), 2C (44) is shown.
  //Example: dc=org,t=gwappstree = 19 chars = 38 bytes = 40 offset = 0x28
  var offsetLength = ((3 tree.length context.length) * 2) 2;
  byteData.write(offsetLength);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);

  //return new JString(Base64Codec.encode(new JString(str).getBytes(encoding)));

  //Add in the context, then the tree.
  //byteData.write(new JString(context).getBytes(encoding));
  var byteArray = spliceBytesWithNulls(new JString(context).getBytes(encoding));
  byteData.write(byteArray, 0, byteArray.length);
  byteData.write(hex2dec("2E"));
  byteData.write(0);
  byteData.write(hex2dec("74"));
  byteData.write(0);
  byteData.write(hex2dec("3D"));
  byteData.write(0);
  //byteData.write(new JString(tree).getBytes(encoding));
  var byteArray = spliceBytesWithNulls(new JString(tree).getBytes(encoding));
  byteData.write(byteArray, 0, byteArray.length);

  //Add in more static stuff.
  byteData.write(0);
  byteData.write(0);
  byteData.write(2);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(1);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(7);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  //byteData.write(hex2dec("12")); //Represents the number of bytes to offset for the attribute name.
  //'uniqueid' = 8 chars = 16 bytes = 18 = 0x12
  var offsetLength = ((attrName.length) * 2) 2;
  byteData.write(offsetLength);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);

  //Add in the attribute name and, eventually, value.
  //byteData.write(new JString(attrName).getBytes(encoding));
  var byteArray = spliceBytesWithNulls(new JString(attrName).getBytes(encoding));
  byteData.write(byteArray, 0, byteArray.length);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  //byteData.write(hex2dec("10")); //Represents the number of bytes to offset for the value.
  //'test00*' = 7 chars = 14 bytes = 16 = 0x10
  var offsetLength = ((attrValue.length) * 2) 2;
  byteData.write(offsetLength);
  byteData.write(0);
  byteData.write(0);
  byteData.write(0);
  //byteData.write(new JString(attrValue).getBytes(encoding));
  var byteArray = spliceBytesWithNulls(new JString(attrValue).getBytes(encoding));
  byteData.write(byteArray, 0, byteArray.length);
  byteData.write(0);
  byteData.write(0);

  //The value needs to have a number of bytes that is evenly disible by four, so make sure that happens padding with nulls.
  if(byteData.size() % 4) {
    byteData.write(0);
    byteData.write(0);
  }

  return new JString(Base64Codec.encode(byteData.toByteArray()));
}

/**
 * Silly function to put null bytes between other bytes in a byte array.
 */
function spliceBytesWithNulls(origByteArray) {
  var newByteArrayStream = new ByteArrayOutputStream();
  for(var ctr0 = 0; ctr0 < origByteArray.length; ctr0) {
    newByteArrayStream.write(origByteArray[ctr0]);
    newByteArrayStream.write(0);
  }
  return newByteArrayStream.toByteArray();
}

/**
 * Convert a hex string into an integer.
 *
 * @param hexStr     hex string
 * @return intStr    int string
 */
function hex2dec(hexStr) {
  return Integer.parseInt(hexStr, 16);
}


That's it.  It seems like the need for IDM-created dynamic groups has not abated so I am posting this in the hope it will make this easier for others without needing to rely on LDAP for interactions, and hopefully to also improve upon it so that it can be more robust and complete in the future.  After a brief bit of digging in iManager's NPMs did not turn up anything really useful such as a method I could use to do the filter conversion for me, I gave up there; if others can find such a function, if only to be rewritten with it as "The Rulebook"" in hand, that would be appreciated; please post any developments in the comments section.

Once I get into it more, I suspect some of those non-null, non-off, and non-data bytes are probably indicative of the boolean logic that comes from the '&', '|', or '!' characters in an LDAP filter, or that somehow will indicate grouping of filter components when there are multiples at a certain level being AND'd or OR'd together.  Priorities (work, real life) are preventing me from continuing at the present, but with any luck I'll be back in a month or two to solve this more-completely.  With a bit more luck, one of you will solve it for me and post back in the Comments section and with your own Communities post.

If there are other interesting parts of eDirectory that seem to need to be disassembled, hearing about those in the comments section is also appreciated, particularly if it has a business case behind it.

Happy computing.

Labels:

How To-Best Practice
Comment List
  • Just to add to 's interpretation of this attribute I share the following notes I made while trying to understand this better. I had hoped to modify 's script for more flexibility, but ran out of time (along with my limited EcmaScript skills) and ended up implementing a separate LDAP driver to update this attribute as an interim fix. In any case the following may be helpful for those who wish to pursue this further. Building on the notation used by in his code, the '//' at the beginning of each hex line are a carry over from the commenting format. One item to be aware of, as was identified in the original article, is the strings are all Null terminated and padded, with additional Null's, to ensure each string ends on a full word boundary (or multiples of 4 bytes).

    Hopefully this is helpful information.

    Cheers,

    D

     // 00 00 00 00 { null - parameter start }  // 02 00 00 00 { Search Level } // 00 = Entry; // 01 = One Level; // 02 = Sub Levels  // 2C 00 00 00 { Search Base string length } null terminated (UTF-16) double byte char string  // 4F 00 3D 00  6E 00 6F 00  76 00 65 00  6C 00 6C 00 { Search Base string  //O = n o v e l l }  // 2E 00 74 00  3D 00 47 00  57 00 41 00  50 00 50 00 { string…  // . t = g w a p p }  // 53 00 54 00  52 00 45 00  45 00 00 00 { string…  // s t r e e }  // 02 00 00 00 { filter operator } 01 = OR; 02 = AND; 03 = NOT  // 01 00 00 00 { component count } follows filter operators 01 or 02; does not follow 03  // 00 00 00 00 { null - parameter start }  // 07 00 00 00 { component evaluation } 07 = Equals; 08 = Is After; 09 = Is Before; 0F = Present  // 12 00 00 00 { attr name string length} null terminated (UTF-16) double byte char string  // 75 00 6E 00  69 00 71 00  75 00 65 00  49 00 44 00 { eDir attr-name // u n i q u e I D }  // 00 00 00 00 { null - parameter start }  // 10 00 00 00 { attr value string length } null terminated (UTF-16) double byte char string  // 74 00 65 00  73 00 74 00  30 00 30 00  2A 00 00 00 { attr-value  // t e s t 0 0 * }     Filter Operator    // 01 00 00 00 { OR }  "|"     [followed by Count of Filter Components and then components]                        // 02 00 00 00  - example Count = 2 components to follow    (|(a=b)(c=d))  // 02 00 00 00 { AND }  "&"    [followed by Count of Filter Components and then components]                        // 03 00 00 00  - example Count = 3 components to follow   (&(a=b)(c=d)(e=f))  // 03 00 00 00 { NOT}  "!"    [followed by Filter Components]- see Not Operator Evaluation below     Filter Component Evaluation    // 07 00 00 00 { EQUALS }  "="    (a=b)   - followed by 2 components  // 08 00 00 00 { IS AFTER }  ">="    (a>=b)   - followed by 2 components
         [GreaterOrEqual in iManager]  // 09 00 00 00 { IS BEFORE }  "<="    (a<=b)   - followed by 2 components
         [LessOrEqual in iManager]  // 0A 00 00 00 { APPROX } "~="   (a~=b)   - followed by 2 components
         [Not in iManager]  // 0F 00 00 00  { PRESENT }  "=*"    (a=*)   - followed by 1 component     NOT Operator Evaluation    // 03 00 00 00   00 00 00 00   07 00 00 00 { NOT - Parm Start - EQUALS }   (!( a=b ))   - followed by 2 components  // 03 00 00 00   00 00 00 00   0F 00 00 00 { NOT - Parm Start - PRESENT }   (!(a=*))   - followed by 1 components      // 03 00 00 00   00 00 00 00   08 00 00 00 { NOT - IS AFTER }  "(!(>=))"    (!(a>=b))   - followed by 2 components
          [Less - Not in iManager]  // 03 00 00 00   00 00 00 00   09 00 00 00 { NOT - IS BEFORE }  "(!(<=))"    (!(a<=b))   - followed by 2 components
          [Greater - Not in iManager]      Filter Component Format    // 00 00 00 00 { Null - parameter start }  // 0x 00 00 00  { Filter Component Evaluation; 07 | 08 | 09 | 0A | 0F }  // xx 00 00 00 { String Length for Attr Name }  null terminated (UTF-16) double byte char string  // xx 00 xx 00   xx 00 xx 00 { Attr Name string } double byte char string with null termination of xx bytes - Length preceding  // 00 00 Optional { null padding double byte if string length is not / 4 - full word boundary }  // xx 00 00 00 Not present if Evaluation is 0F { String Length for Attr Value }  // xx 00 xx 00   xx 00 xx 00 Not present if Evaluation is 0F { Attr Value string }
  • AB,

    I needed something working now; so I chose the route of creating a temporary service account to authenticate as LDAP, write the memberqueryurl; and so far, this is working really well. I'll have to come back to this on my next round of updates. I'd really like to get it up and running as it would save some steps.

    --Aaron
  • The baes64 output is, presumably, coming from LDAP, right? I would expect the LDAP view of the attribute to be as you described, assuming things are fine. It may be useful to create the same group via iManager to see how it looks there, comparing both the LDAP memberQueryUrl attribute and the memberQuery attribute as seen in iMonitor to see how they differ. From there the code could be modified to handle your situation correctly.

    Feel free to start a thread in the forums with trace output and then the LDAP and iMonitor output and we may be able to troubleshoot it there.
  • AB,

    I would really benefit if I could get this code working: I need to update a query to include today's date in it once a day just before a job runs; however, something has changed, and the end result is something that does appear to be proper base64 - but when I run the result I get past the base64 -d on Linux, I get something that's not quite ascii:

    [root@triviredir drivers]# echo AAAAAAIAAAAmAAAAbQB5AGMAbwBuAHQAZQB4AHQALgB0AD0AbQB5AHQAcgBlAGUAAAACAAAAAQAAAAAAAAAHAAAAEgAAAGEAdAB0AHIAbgBhAG0AZQAAAAAAFAAAAGEAdAB0AHIAdgBhAGwAdQBlAAAAAAA= | base64 -d
    &mycontext.t=mytreeattrnameattrvalue

    When I expected to see something that started with ldaps:// . . .Any thoughts? Has something changed in Java 8 that would cause the encryption to be different? I'm generating the above value using this call when running Rhino from a Windows workstation:

    java.lang.System.out.println(buildDynamicGroupMemberQuery("mytree", "mycontext", "attrname", "attrvalue"));

    --Aaron
  • Thanks for the reply; I have not worked out the other details yet (pesky "work" requiring my time) but will get back to it, hopefully this month. If you find new clues please feel free to update here.
  • AB,

    This is amazing. I've wanted this functionality for IDM job scopes for years now: this gives us the ability to deploy scoped jobs into a tree without requiring additional scope configuration (which is a step I tend to forget) and minimizes the amount of redundant authentication info in the driver just to give me ldap filter support. Thank you for your work on this.

    I'm a little concerned though; as I assume with the next update to the LDAP server, the byte order and/or format may change, correct? Also, I'd love to hear if you' have done any work on this in the past month as I'm going to need the remaining LDAP filter meta-characters for what I'm doing now.

    I wonder if IDM will natively support LDAP queries: that would completely solve my issue too (with having redundant credentials in the driver), or having to read it off of the driver object and the user name from security equals.

    --Aaron
  • Cool stuff that sounds like you had a ton of fun digging into it!
Related
Recommended