Can you write an UTF-8 string containing "Flexed Biceps" directly to eDirectory via LDAP? If you retrieve that attribute again, do you also get UTF-8 encoded representation of the two unicode surrogates?

Just tried the LDAP writes. And yes, the same problem is there are well. I wasn't able to write that unicode character using either UTF-8 or UTF-16 (LE or BE) encoding to the eDirectory. The only syntax that worked was the UTF-16 surrogate pair further encoded as UTF-8. Here are the used LDAP attribute modify byte buffer values and the operation results:

1. Plain UTF-8 (as per LDAP v3 RFC)
   • b'\xF0\x9F\x92\xAA'
   • NDS error: syntax violation (-613)
2. Plain UTF-16 BE
   • b'\xD8\x3D\xDC\xAA'
   • Write succeeds, but reading the attribute value afterwards results in different result: b'\xd6\xbd\xdc\xaa'
3. Plain UTF-16 LE
   • b'\x3D\xD8\xAA\xDC'
   • NDS error: buffer too small (-119)
4. UTF-16 surrogate pair further encoded as UTF-8
   • b'\xED\xA0\xBD\xED\xB2\xAA'
   • Write succeeds and the read result is the same.
   • This double encoded version is the only one that "works", but LDAP strings shouldn't be double encoded

I have also raised a support case about this, trying to get through the first line now...

I was wondering if python-ldap (which is not new) could be the problem, not handling these strings (data) correctly. In general eDirectory LDAP like to handle "unknown" data as base64.

How does the data look like if you use ldapsearch?

python-ldap handles LDAP data purely as byte buffers, encoding/decoding is left to the caller. In the tests, I also used pure byte buffers to make sure the byte representation of the data was according to the UTF-8/16 formats.

LDAP protocol on the other hand does not know anything about base64. LDIF is another thing of course, where base64 is used for the values which are binary or contain "exotic" characters.

I also tried other tools, like ldapsearch and Apache directory studio, with similar results. Ldapsearch returns the value double encoded (UTF-8 string, which contains further UTF-16 surrogate pairs). Here's a sample result from ldapsearch (base64 value):
7aC97bKq

Apache directory studio shows the returned surrogate pair character as unicode replacement character, since those characters shouldn't be there. That also breaks directory studio functionality, since it cannot delete the attribute value (tries delete with the replacement chars --> value does no exist in directory).

NetIQ IDM seems to have worked around the issue, since it is able to store higher plane unicode characters into eDirectory and ever retrieve them later on. And I think IDM is actually using LDAP nowadays, instead of NCP (Actually, I might be wrong here, looking at the connections of the ndsd process, there are both ldaps and NCP connections).

Humm, I've had a couple of surprises with python-ldap in the past, which is why I was wondering.

The LDAP Connections are done by tools and ldapsearch, everything should be using ncp - it would be surprising if they changed the engine to use LDAP as it would add a layer of abstraction (and make operations slower).

While Unicode is pretty cool and solves many problems we had with codepages, but it introduces just as many problems as we still do not have a proper way to handle all of them (just my humble opinion).

With python-ldap I've used normalize to handle extended characters, though I doubt it would be helpful for you.

stackoverflow.com/.../how-does-unicodedata-normalizeform-unistr-work

Yes, I have came across unicode string combined characters and normalization before, but that's another story. :)

I think it was just the IDM Designer which was changed to use LDAP instead of NCP at some version. But IDM engine internal functionality and directory communication may of course be something else. Glad to see that the unicode strings seem to work there, at least.

IMO, unicode is a great thing, but also brings some challenges. But it's actually pretty great to be able to enter all kinds of "characters" into strings.

See if you can get a bug opened, at least to get engineering to look at it - and comment on this (and hope that you'll get an answer).

I do suspect that you might get a "working as designed".