Generic REST Collector supporting odata

I am considering to connect an application offering an odata compliant REST API to IG (3.7.3), and I am wondering if it is possible - and/or suggested - to modify the shippir REST collector for GiHub.

Unfortunately, without modification the REST GitHub Collectors cannot be used for this challenge, since they have some "hard" coded stuff in them.

After exporting the collector I found only the following two functions:

  • getRestResource(connectionParm, token, pageSize)
  • getNextPageURL(response) 

I believe, the responsibility of the first one is to implement the REST request and compose the JSON object representing the collection.

The code seams to utilize a paged search (token = '/users?per_page=' + pageSize;) which can not work for two reasons when working with odata:

1) the schema (/users) is not correct

2) odata does not support paged searches in this way.

The schema can be possibly simply set to the one used in the odata implementation, but the paged search has to be implemented differently.

 In addition to these findings, the first function uses some "HelpperFunctions" provided by com.netiq.daas.restconnector.internal.HelperFunctions!

Can anybody provide some information regarding the HelperFunctions class, and the structure of the final JSON object, IG is able to process?

In case a paged search can not be implemented for odata, is there a technical limitation on the data received in one REST response?

To replace a paged search odata offers the parameters &size and $skip - so a paged search could be replaced by a loop running several requests with in increasing value in the $skip parameter, if needed!

I am not sure, regarding the purpose of the function getNextPageURL(response)  - is this needed, or does this need to be adapted as well?

Kind regards

Thorsten

Parents
  • 0  

    Hello Thorsten,

    1) What you downloaded was the RESTGitHub (Account or Permission) Collector Template

    2) At this time, we do not offer a Generic set of REST Collectors (Identity, Account, or Permission).  This is on our Roadmap and an aspect we are investigating.
    What we provide are specific collectors that utilize REST.

    Is it possible to utilize SCIM as we do provide a Generic set of SCIM collectors?

    I will review the information about OData as that might be an aspect to look at regarding the ability to have a Generic set of REST collectors.

    Sincerely,
    Steven Williams
    Principal Enterprise Architect
    OpenText Cybersecurity

  • 0 in reply to   

    Hello Steven,

    thanks for your fast answer!

    I modified the getRestResource() function in a way it does not tray to run a paged search, and I replaced the users schema with the one which would work in my case.

    token = '/opx2_user?$select=class,object_number,name,comment,list_of_groups'

    The $select limits the properties/attributes returned for each object!

    Running a collection I can see the REST Response code in catalina.out presented in the following way:

    {"@odata.context":"https://<serverFQFDN>/odata/$metadata#opx2_user(class,object_number,name,comment,list_of_groups)","value":[
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(448911)",
    "class" : "USER",
    "object_number" : 448911,
    "name" : "PROJEKTLEITER",
    "comment" : "Projektleiter",
    "list_of_groups" : "PROJECT_MANAGER,ALL,P_PREMIER"
    },
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(196911)",
    "class" : "USER",
    "object_number" : 196911,
    "name" : "SUPPORT",
    "comment" : "Support User",
    "list_of_groups" : "ALL,P_APP"
    },
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(1669821)",
    "class" : "USER",
    "object_number" : 1669821,
    "name" : "N10005",
    "comment" : "Freier Dienstag TEst",
    "list_of_groups" : "ADMIN,ALL,P_FADM"
    }
    ]
    }

    But IG does not "translate" this to Accounts, and therefore the collection does not import any object.

    I still believe, there is a way by further manipulating the gerRestRessource() function - but I am not 100% sure how - and even if I get it to work, if I need to implement a replacement for the paged search!

    Kind regards

    Thorsten 

Reply
  • 0 in reply to   

    Hello Steven,

    thanks for your fast answer!

    I modified the getRestResource() function in a way it does not tray to run a paged search, and I replaced the users schema with the one which would work in my case.

    token = '/opx2_user?$select=class,object_number,name,comment,list_of_groups'

    The $select limits the properties/attributes returned for each object!

    Running a collection I can see the REST Response code in catalina.out presented in the following way:

    {"@odata.context":"https://<serverFQFDN>/odata/$metadata#opx2_user(class,object_number,name,comment,list_of_groups)","value":[
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(448911)",
    "class" : "USER",
    "object_number" : 448911,
    "name" : "PROJEKTLEITER",
    "comment" : "Projektleiter",
    "list_of_groups" : "PROJECT_MANAGER,ALL,P_PREMIER"
    },
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(196911)",
    "class" : "USER",
    "object_number" : 196911,
    "name" : "SUPPORT",
    "comment" : "Support User",
    "list_of_groups" : "ALL,P_APP"
    },
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(1669821)",
    "class" : "USER",
    "object_number" : 1669821,
    "name" : "N10005",
    "comment" : "Freier Dienstag TEst",
    "list_of_groups" : "ADMIN,ALL,P_FADM"
    }
    ]
    }

    But IG does not "translate" this to Accounts, and therefore the collection does not import any object.

    I still believe, there is a way by further manipulating the gerRestRessource() function - but I am not 100% sure how - and even if I get it to work, if I need to implement a replacement for the paged search!

    Kind regards

    Thorsten 

Children
  • 0 in reply to 

    Hello Steven,

    I applied a few more changes to the getRestResource() function used in the GitHub REST collector and finally I got the users collected running a test collection!

    I changed the function in the way, it only sends one REST request, and iterates once through the JSON provided in the resulting response. In this case, there is a Content.value property providing an array in the JSON object of all users/accounts!

    I use the for-loop within the function to setup user properties of the result JSON object for each value found!

    As far as I interpret the original code of the function it used a link in the first REST response to run additional REST requests for each individual object, but I guess this is not needed, is it?

    In my PoC I am not using a paged search, but request all objects at once - I hope this would not be a problem, especially since I am requesting only a few properties of users/Accounts!

    I believe, with some more additional time spend on further changes of this function it would not be a problem to use the $size and &skip parameters to "simulate" a paged search, but still the question is, if this is mandatory?

    Kind regards

    Thorsten