This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

REST Driver and Pagination (Link Header with rel="next")

Hi!

I´m working on an IDM integration of an existing REST API using IDM 4.8.1 and  the REST driver 1.1.1.0. The driver has to provide user and group entitlements with value from the application.

The API has an endpoint to get the list of available permissiongroups, but as result of a GET request the API returns only a limited number (currently 100) of objects plus a Link header with rel="next" to get the next.

Level 7 Trace shows the returned header (with anonymized URLs):

[09/15/20 14:27:48.668]:RESTDRIVER PT:RESTDRIVER: poll
[09/15/20 14:27:48.669]:RESTDRIVER PT:RESTDRIVER: Custom: preparing GET to apiserver.domain/.../permissiongroups
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Resetting headers
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Setting the following HTTP request properties:
Authorization: <content suppressed>
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Accept:application/json
[09/15/20 14:27:48.671]:RESTDRIVER PT:RESTDRIVER: Did a HTTP GET with 0 bytes of data to apiserver.domain/.../permissiongroups
[09/15/20 14:27:48.671]:RESTDRIVER PT:RESTDRIVER: *******************************************************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: **********************LOGGING REQUEST******************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: *******************************************************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: Request URL :apiserver.domain/.../permissiongroups
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Http Method : GET
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Sending http request with below headers :-
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Authorization: <content suppressed>
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Accept: application/json
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: ***************************END**************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ********************************************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ***********************LOGGING RESPONSE*****************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ********************************************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: Http response code : 200
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: Http response status : HTTP/1.1 200 OK
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Getting http response with below headers :-
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Date: Tue, 15 Sep 2020 12:27:40 GMT
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Server: Apache
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Expires: 0
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Cache-Control: no-cache, no-store, must-revalidate
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Pragma: no-cache
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Content-Length: 11508
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Link: <apiserver.domain/.../permissiongroups rel="next"
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Keep-Alive: timeout=3, max=10000
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Connection: Keep-Alive
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Content-Type: application/json
[09/15/20 14:27:48.945]:RESTDRIVER PT:RESTDRIVER: Sending http response with body :-
[09/15/20 14:27:48.945]:RESTDRIVER PT:RESTDRIVER: {"liste":[{..... <the first 100 records> }]}

The API does not allow to force a complete result set, or a required number of results in the request, so I have to follow the Link.

I could not find any information on how to get the REST driver to follow the links to get the complete data.

Would be grateful for any ideas...

best regards,
Thomas

PS: This is also an issue from publisher poll request.

 

 

Parents
  • 0  

    Currently the REST driver returns the request headers with the response instead the ones send by the server. (I opened an SR for this). So to get the value for Link respose header, you'll probably have to look at writing a DocumentModifier: https://www.netiq.com/documentation/identity-manager-48-drivers/generic_rest/data/bvpan6k.html

     

  • 0   in reply to   

    While this seems like a daunting task, it turns out, with basic Java skills you can do this.

     

    What happens is there is a skelton file in the Driver Dev kit, with the various classes you can add to the SOAP or Delim text drivers.

    They are basically a class definition that takes the document as input, and then returns your modified version. (For the most part).  So your code goes inside and does what is needed.  Now offhand, I have no idea how to get access to the headers... Nor how to write to them, but seems doable.

  • 0   in reply to   

    You mean the "Document Handling" class stuff?

    I am not sure that  it will be possible to get the http headers after the XDS document is recreated and send to your custom document handler.

  • 0 in reply to   

    Thank you for your answeres.

    Currently I understand:

    - the REST drriver shim itself does NOT handle paged results (thats what i initially expected)

    - the Link header from the response is not passed to the event ans is therfore not available in policies

    - it is not sure, if a Java extension can get to the Link header

    So it looks like the REST driver possibly is not the way to go...

    I could look into writing my complete own driver shim or look into the new Invoke REST Endpoint action (whitch lacks the PATCH method at least according to the documentation).

    Any other ideas?

    best regards,
    Thomas

     

     

  • 0 in reply to 

    Thomas, Norbert’s SR should allow handling pagination (albeit painfully) through Document Modifiers in the short term.

    As for supporting pagination natively, I would recommend that you log an ideas exchange request. We are actively looking at enhancing the capabilities of the REST driver, and this should get addressed with the next release of the driver.

    Regards,
    Abhishek

  • 0 in reply to 

      abs.  the pagination functionality is must-have  built in netiq REST driver, i would vote for this for sure, mostly REST api today if done (professionally) does pagination, example Microsoft Graph and other oDATA apis, so yes, it would be still possible to do work-around with current version,  but NetIQ REST driver should be smart enough to discover REST api and support pagination out of the box. 

    one of the work around could be:

     

    1) Issue query from subscriber command transformation

    2) Output transformation should implement command 

    3) Publisher input return data from REST api , as result set with pagination back to subscriber command

    4) Subscriber command transformation recieves result set, parse data, and  submit new command with new pagination link

     

    this would work, if the response of first query is not so big enough, so smart idea would be to issue query with filters possible to reduce the result-set received from service.

     

    but yes, it requires some experience and knowledge of netiq.

     

    Regards

    /Maqsood

     

     

     

     

     

     

     

  • 0   in reply to 

    Different REST Systems handles pagination in different ways. But the way the pagination is handled is often returned in the response header returned by the system. Currently the driver does not provide a way to access the response header.

    We will implement this shortly where the headers returned in response will also be part of the response like <response-headers> as part of <driver-operation-data>. This will give the access (for example link in this connected system) in the policy channel and pagination could be achieved by writing policies based on the <response-headers>

Reply
  • 0   in reply to 

    Different REST Systems handles pagination in different ways. But the way the pagination is handled is often returned in the response header returned by the system. Currently the driver does not provide a way to access the response header.

    We will implement this shortly where the headers returned in response will also be part of the response like <response-headers> as part of <driver-operation-data>. This will give the access (for example link in this connected system) in the policy channel and pagination could be achieved by writing policies based on the <response-headers>

Children
  • 0 in reply to   

    There is probably another problem with that even if the response header will be available in the policies soon.

    The codemap refresh results are only available in the input policies, so the query for the additional bunches of results will have to be done from there, but a query from an itp policy back to the application does not flow through the otp policies. Therefore no conversion from XDS to JSON an so on is done and the shim does nothing. 

    Do i miss something?

    best regards,
    Thomas

  • 0   in reply to 

    The Code-Map refresh supports query-ex, so it should be possible to use the "Link https://...." if you could get your hands on it.

    I do not know how to do query-ex with the code-map refresh, just know that it's supported.

     

  • 0 in reply to   

    Thank you, query-ex is a good idea!

     I´ve tested with an entitlement marked as query-ex supported. 

    The codemap refresh then injects a query-ex token, but it does not make any difference for the current REST driver 1.1.1 --> the query result looks like from a normal query

    The REST driver documentation 1.1.1 also does not contain any information about query-ex so it probably does not support query-ex currently (besides the header part).

     

  • 0   in reply to 

    SO when you have issues with CMR, like timeouts, takes too long, runs too often, what I suggest is in the ITP catch the query for CMR.  Then change it to something you know will return and fast. I like the __driver_identitication_query__ since all drivers have to support it, handled in the shim, no network/connection traffic.

    Then tag it with op-data so you can catch the __driver_identification_response__ in the OTP.  Then clone in your values to the <instance> document.

    There was a recent thread where someone had CMR fail since it did not finish inside a hard coded 10 min window so it kept running.  David Gersic and I discussed a solution he worked on (we are colleagues) where he used a Job to do the CMR's data collection task on a better controlled schedule, and then simply stored the <instance> XML in a text blob on an object. Then when catching the CMR query response in the OTP, clone the contents of that object into the results, and it is fast and controlled.

    Similarly here, you could likely create a Job and catch the <trigger> event in the Sub-ETP so ITP applies to the queries, do the work, store it somewhere, and then when CMR happens and you catch it in the OTP, replace the results with the pre-collected data.

     

    Agreed, would be nice if it worked as it should, but you can work around it if it does not.

  • 0   in reply to 

    I though so, somehow you need to return the token (Link) to UA which then should use that for the next query - complicated. And as long as you can't get your hand on the return header values, there is no way to make this work.

  • 0 in reply to   

    Hi!

    This is how I solved it:

    I intercept the permissiongroup entitlement query in an OTP and covert it into a query for class "__driver_identification_class__" like the account entitlement query does and as Geoffry suggested.

    In an ITP policy before NETQRESTJSON-itp-JSONtoXDS I use a custom Java class based on Apache HTTP client to query the values from the rest service in a "while Link header is set" loop without using the REST driver´s shim.

    The results are converting into a REST driver compatibel JSON format using ECMA code and injected into the <status> element so that it looks like the "normal" response oft the REST driver´s shim:

    <nds dtdversion="3.0">
    <source>
    <product build="20200821_0139" version="1.1.1.0">Identity Manager REST Driver</product>
    <contact>NetIQ Corporation.</contact>
    </source>
    <output>
    <status event-id="0" level="success" type="driver-general">
    <driver-operation-data class-name="permissiongroup" command="query" event-id="0">
    <response>
    <header content-type="application/json"/>
    <value message="OK" status="200">{"totalResults":456,"results":[{...}]
    </value>
    </response>
    </status>

     

    The existing rules in NETQRESTJSON-itp-JSONtoXDS then convert the JSON query response to a list of instances.

    That is what codemap refresh needs.

     

    The downside is that I have this custom Java code and two ways of talking to the REST API.

    Thank you for all your suggestions
    Thomas

     

     

     

     

     

     

     

  • 0   in reply to 

    A pretty cool solution, I must say.

    Just shows that we need a pure http client driver from MF, just a simple driver which can do transport. I know about Steffan's driver, but MF over engineered their REST driver...

  • 0   in reply to   

    Response headers are now supported in REST driver version 1.1.2.0:

    [12/07/20 14:44:25.895]:PAM-CC ST:            PAM-CC: ********************************************************
    [12/07/20 14:44:25.895]:PAM-CC ST:            PAM-CC: ***********************LOGGING RESPONSE*****************
    [12/07/20 14:44:25.896]:PAM-CC ST:            PAM-CC: ********************************************************
    [12/07/20 14:44:25.896]:PAM-CC ST:            PAM-CC: Http response code : 200
    [12/07/20 14:44:25.896]:PAM-CC ST:            PAM-CC: Http response status : HTTP/1.0 200 OK
    [12/07/20 14:44:25.897]:PAM-CC ST:            PAM-CC: Getting http response with below headers :-
    [12/07/20 14:44:25.897]:PAM-CC ST:            PAM-CC: Set-Cookie: pum_rest_auth=eJzj8...
    [12/07/20 14:44:25.901]:PAM-CC ST:            PAM-CC: Content-Type: application/json
    [12/07/20 14:44:25.902]:PAM-CC ST:            PAM-CC: Content-Length: 38918
    [12/07/20 14:44:25.902]:PAM-CC ST:            PAM-CC: Sending http response with body :-
    [12/07/20 14:44:25.906]:PAM-CC ST:            PAM-CC: {"vrm":"4.0.0","status":200,"message":"OK"}
    [12/07/20 14:44:26.044]:PAM-CC ST:            PAM-CC: **********************END*****************************
    [12/07/20 14:44:26.045]:PAM-CC ST:            PAM-CC: Response code and message: 200 OK
    [12/07/20 14:44:26.047]:PAM-CC ST:            SubscriptionShim.execute() returned:
    [12/07/20 14:44:26.048]:PAM-CC ST:
     
    <nds dtdversion="3.0"> <source> <product build="20201022_1154" version="1.1.2.0">Identity Manager REST Driver</product> <contact>NetIQ Corporation.</contact> </source> <output> <status event-id="0" level="success" type="driver-general"> <driver-operation-data class-name="Policies" command="query" dest-dn="" event-id="0" src-dn=""> <response> <url-token/> <header/> <response-header Content-Length="38918" Content-Type="application/json" Set-Cookie="pum_rest_auth=eJzj8...; secure"/> <value message="OK" status="200">{"vrm":"4.0.0","status":200,"message":"OK"}</value> </response> </driver-operation-data> </status> </output> </nds>
  • 0   in reply to   

    Download lnk for the new shim  update with the HEader support:

     

    https://download.microfocus.com/Download?buildid=jjXMIB4Tqu4~

     

    Readme:

    https://www.netiq.com/documentation/identity-manager-48-drivers/RESTDriver1.1.2readme/data/RESTDriver1.1.2readme.html

     

    Says fixes are:

    • ALM 231647 - REST shim version 1.1.0.0 breaks processing of binary content types.
    • ALM 235207 - REST shim is unable to access the HTTP response headers.
    • ALM 280236 - REST driver shim must retry when the "java.net.SocketTimeoutException: Read timed out" error occurs.

    I would say #2 in that list is what Norbert is referring to.  Curious to know more about the first one though.  (No ALM access, alas).

     

     

     

     

  • 0   in reply to   

    Hi Geoffry,

    in some cases, the new logging added in 1.1.0.0 did consume the HTTP body so that is was not available to be returned as value anymore.