Hello Thorsten,

1) What you downloaded was the RESTGitHub (Account or Permission) Collector Template

2) At this time, we do not offer a Generic set of REST Collectors (Identity, Account, or Permission).  This is on our Roadmap and an aspect we are investigating.
What we provide are specific collectors that utilize REST.

Is it possible to utilize SCIM as we do provide a Generic set of SCIM collectors?

I will review the information about OData as that might be an aspect to look at regarding the ability to have a Generic set of REST collectors.

Sincerely,
Steven Williams
Principal Enterprise Architect
OpenText Cybersecurity

Hello Steven,

thanks for your fast answer!

I modified the getRestResource() function in a way it does not tray to run a paged search, and I replaced the users schema with the one which would work in my case.

token = '/opx2_user?$select=class,object_number,name,comment,list_of_groups'

The $select limits the properties/attributes returned for each object!

Running a collection I can see the REST Response code in catalina.out presented in the following way:

{"@odata.context":"https://<serverFQFDN>/odata/$metadata#opx2_user(class,object_number,name,comment,list_of_groups)","value":[
{
"@odata.id":"https://<serverFQFDN>/odata/opx2_user(448911)",
"class" : "USER",
"object_number" : 448911,
"name" : "PROJEKTLEITER",
"comment" : "Projektleiter",
"list_of_groups" : "PROJECT_MANAGER,ALL,P_PREMIER"
},
{
"@odata.id":"https://<serverFQFDN>/odata/opx2_user(196911)",
"class" : "USER",
"object_number" : 196911,
"name" : "SUPPORT",
"comment" : "Support User",
"list_of_groups" : "ALL,P_APP"
},
{
"@odata.id":"https://<serverFQFDN>/odata/opx2_user(1669821)",
"class" : "USER",
"object_number" : 1669821,
"name" : "N10005",
"comment" : "Freier Dienstag TEst",
"list_of_groups" : "ADMIN,ALL,P_FADM"
}
]
}

But IG does not "translate" this to Accounts, and therefore the collection does not import any object.

I still believe, there is a way by further manipulating the gerRestRessource() function - but I am not 100% sure how - and even if I get it to work, if I need to implement a replacement for the paged search!

Kind regards

Thorsten 

Hello Steven,

I applied a few more changes to the getRestResource() function used in the GitHub REST collector and finally I got the users collected running a test collection!

I changed the function in the way, it only sends one REST request, and iterates once through the JSON provided in the resulting response. In this case, there is a Content.value property providing an array in the JSON object of all users/accounts!

I use the for-loop within the function to setup user properties of the result JSON object for each value found!

As far as I interpret the original code of the function it used a link in the first REST response to run additional REST requests for each individual object, but I guess this is not needed, is it?

In my PoC I am not using a paged search, but request all objects at once - I hope this would not be a problem, especially since I am requesting only a few properties of users/Accounts!

I believe, with some more additional time spend on further changes of this function it would not be a problem to use the $size and &skip parameters to "simulate" a paged search, but still the question is, if this is mandatory?

Kind regards

Thorsten

Hi,

the OpenText professional services team in the US did develop a custom collector using an OData REST API for for MS Dynamics CRM.

Hi Norbert,

this sounds great, and would possibly save me some gray hair developing such a collector from scratch ;-)

Is there any way to work with this collector, or will it be available in the next on-prem release of IG 4.x?

Kind regards

Thorsten

Hi Thorsten,

please reach out to Alexander Adam regarding how to license that component.

Norbert

Hi Norbert,

I will forward this information to my customer - but since I did some progress in my lab, I will spend some more time on further tweaking the function contained in the GitHub Collector.

So far, I am able to read all meaningful account data in one request. The next stage will be to implement a kind of paged search in case there are hundreds or thousands pf accounts to be imported ;-)

Kind regards

Thorsten

Hello Steven,

as written before, I modified the getRestResource() to request all users or groups in one response. I further modified the code to add an outer loop utilizing the $top and $skip odata parameters. The end of the loop is defined by a variable providing the maximum results provided by the REST request.- For this purpose, odata offers the $count parameter. 

var objectsFound = ((helpperFunctions.httpClient(connectionParm, 'GET','/opx2_user?$count', '{}')));

According to the debug data offered in catalina.out the REST request is working, but the function terminates because the response is not provided in JSON.

[INFO] 2024-04-15 15:18:00.398 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Connection Framework execution info message: 'Executing the request to: sparkassen-versicherung-dev.planisware.live/.../opx2_user
[INFO] 2024-04-15 15:18:00.398 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Connection Framework execution info message: 'Executing the request to: sparkassen-versicherung-dev.planisware.live/.../opx2_user
[FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Response Body: 48
[FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Response Body: 48
[FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Http Response Code :200
[FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Http Response Code :200
[FINE] 2024-04-15 15:18:00.506 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Response is a not a valid JSONObject or JSONArray
[FINE] 2024-04-15 15:18:00.506 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Response is a not a valid JSONObject or JSONArray
[FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Received request to unload service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7
[FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Received request to unload service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7
[FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Decremented load count on service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7, load count: null
[FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Decremented load count on service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7, load count: null
[SEVERE] 2024-04-15 15:18:03.306 [com.netiq.iac.persistence.dcs.dce.thread.TestDataCollectionServiceThread] [IG-DTP] Data collection from DaaS failed

is there any way to force the helpperFunction() to accept anything in this case? 

The only other option would be to request all objects, or find a way to exit the outer loop in case no more objects are returned in one iteration ...

Kind regards

Thorsten

HI,

since I cannot use the REST Request from the helpperFunction() to work with a string in the REST response, I changed the outer loop to while-loop like:

while (responseArray.length > 0)

Within this loop I run several times the REST Request with the URL parameters $top and $skip. top is set to the page size and skip is incremented in each loop by the page size.

This is working as expected, and finally I end with a resource JSON object containing all users returend.

{
"resource":[
{
"id":123,
"name":"INTRANET",
"comment":"INTRANET ",
"list_of_groups":"MODULES,ALL",
"user":{
"@odata.id":"FQDN/odata/opx2_user(123)",
"class":"USER",
"object_number":5625,
"name":"INTRANET",
"comment":"INTRANET ",
"list_of_groups":"MODULES,ALL"
}
},
{
"id":1234,
"name":"P_PLW_TEMPLATE_USER",
"comment":"Template for profile P_PLW",
"list_of_groups":"P_PLW",
"user":{
"@odata.id":"FQDN/alt/plw/localhost:8400/odata/opx2_user(1234)",
"class":"USER",
"object_number":196111,
"name":"P_PLW_TEMPLATE_USER",
"comment":"Template for profile P_PLW",
"list_of_groups":"P_PLW"
}
}
]
}

This JSON code is simplified, but I double-checked the code on Catalina with the expected results, and found no problems.

The code even matches my first attempt, requesting all users at once (no pagination with top & skip).

But DAAS is not happy with the result in this case:

[FINEST] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Execution of Javascript completed......
[FINEST] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Execution of Javascript completed......
[FINE] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Result returned by script must contain token
[FINE] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Result returned by script must contain token

I suspect, this error is coming up, since during the last iteration of the while-loop the REST Response returned an empty Array! Or did I miss something else?

I can see in catalina,  that the while loop is run several times with a new URL used in each iteration, and different objects contained in the response JSON.

Due to the fact the JSON.stringify(result) I put to the debug code is as expected as well, I guess the logic of the new function is correct ....

Can anybody provide some ideas, please?

BTW: Is there any way to ease the process of testing a collector? In my case I have to delete the collector, import a new collector template and setup a new collector? This is quite hard, if I apply only some changes to the function ;-)

Kind regards

Thorsten

Finally, I found, the JSON response object needs a "token" element

For my understanding this contains the URLs for the paged search  - in my case I believe this is not needed, therefore I chose no value for the token element.

After adding this, the previous error is gone, but the import script is run infinitely! The only way to get further is to stop the collection!

Can anybody tell me, what value shall be used for token to stop the looping?

In the working collector the code reads:

result['token'] = helpperFunctions.getNextPageURL(connectionParm, JSON.stringify(httpRes['Response_Headers']));

But I guess there is a hard coded value to tell the collector to stop reading ....

BTW: this is my current function:

function getRestResource(connectionParm, token, pageSize) {

var helpperFunctions = Java.type('com.netiq.daas.restconnector.internal.HelperFunctions');

var skip = 0;

var size=pageSize;

var token ='/opx2_user?$select=class,object_number';

var respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

var httpRes = JSON.parse(respString);

var responseArray = httpRes.Content.value;

var maxObjects = responseArray.length;

helpperFunctions.debug('--------------------------------------------------------------');

helpperFunctions.debug('Page Size: ' + pageSize);

helpperFunctions.debug('MaxObjects: ' + maxObjects);

helpperFunctions.debug('--------------------------------------------------------------');

token ='/opx2_user?$select=class,object_number,name,comment,list_of_groups&$top='+ size + '&$skip=' + skip;

userResource = [];

respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

httpRes = JSON.parse(respString);

responseArray = httpRes.Content.value;

helpperFunctions.debug('--------------------------------------------------------------');

helpperFunctions.debug('Response Array: ' + responseArray);

helpperFunctions.debug('--------------------------------------------------------------');

while (responseArray.length > 0){

  for (var i = 0; i < responseArray.length; i++) {

if (responseArray[i]['class'] === 'USER') {

var userObject = {};

var id = responseArray[i]['id'];

userObject['id'] = responseArray[i]['object_number'] ? responseArray[i]['object_number'] : '';

userObject['name'] = responseArray[i]['name'] ? responseArray[i]['name'] : '';

userObject['comment'] = responseArray[i]['comment'] ? responseArray[i]['comment'] : '';

userObject['list_of_groups'] = responseArray[i]['list_of_groups'] ? responseArray[i]['list_of_groups'] : '';

userObject['user'] = responseArray[i];

helpperFunctions.debug('User Data ' + userObject);

userResource.push(userObject);

}

}

skip = skip + size;

token ='/opx2_user?$select=class,object_number,name,comment,list_of_groups&$top='+ size + '&$skip=' + skip;

respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

httpRes = JSON.parse(respString);

responseArray = httpRes.Content.value;

helpperFunctions.debug('--------------------------------------------------------------');

helpperFunctions.debug('Response Array: ' + responseArray);

helpperFunctions.debug('UserResource Array: ' + userResource);

helpperFunctions.debug('--------------------------------------------------------------');

}

var result = {};

result['resource'] = userResource;

helpperFunctions.debug('UserResource Array: ' + JSON.stringify(result));

return JSON.stringify(result);

}