Idea ID: 2875473

GraphQL Client Driver Shim

Status: New Idea

GraphQL has become a valuable tool for working with the rather complex APIs that modern microservice applications often have.

It has been increasingly adopted as a preferred query language for working with APIs. Thousands of companies like  Audi, Airbnb, Coursera, Expedia, DailyMotion, GitHub, Facebook, Twitter, The New York Times all use GraphQL 

GraphQL returns data from a single endpoint in a way that is a better fit for the way that IDM drivers process data, rather than forcing IDM to query collate and deduplicate data from many REST endpoints.

There are specific deficiencies in the REST driver as it ships today that mean that it cannot be 100% used as a basis for a GraphQL integration. (I have tried, mostly succeeded, but it is not perfect!)

In particular, GraphQL rolls it's own error reporting and does not always follow the RESTful logic of reporting errors via HTTP Status Codes.

It should not be particularly difficult to implement a GraphQL Client - driver that builds upon the REST driver shim.

This could be a great addition to the REST/SCIM/SOAP family of "tools" drivers.

  • Definitely, specialized SHIM, which will take care of all driver's complexities is the right way to go.

    If I will need to have it available right now, I will use SneakyCat CLE driver with GraphQL PowerShell as a quick (and dirty) option.

  • Thank you for the clarification. I also always build my only logic for the xds2json / json2xds - faster and cleaner that way.

    Looks like GraphQL is another one of thous "good idea, not perfect implementation", I say that we need a "transport" shim, which basically only talks http, which you feed with driver-operation-data, which does not filter anything out on the response. I am yet to understand why they haven't provided us with something like that. That would allow you to do what you need.

    Another thing would be to use the official GraphQL java llibrary (https://github.com/graphql-java/graphql-java) and build a driver shim wrapper around that - but only to the point where you build the driver-operation-data for the shim to consume.

    I would be nice if they could build you a working GraphQL shim.

  • For reference I have NEVER used the xds2json functions that ship with the REST shim in any production solution. They are quite simply too rigid and unreliable for real-world usage.

    I absolutely do not want a REST shim plus xds2graphql java library solution as that would continue the negative trend of NetIQ providing us with libraries that are not real-world battle tested or functional in any other scenario than the one presented in the documentation (a JSON structure that I have never encountered in the wild). Libraries that go unused and unloved because they were never really what we asked for, so we worked around them.

    Also, the assumption that GraphQL uses http as transport is not really true.

    GraphQL is totally agnostic to the transport layer, it is purely a query language used for remote client-server communications.

    Yes, today most implementations run over http, but this does not have to be the case. GraphQL can and often does run over WebSockets or even in theory some other network transport layer.

    Also: roundtripping xds2graphql is a solved problem for my purposes.

    I have already solved the translation of XDS documents to equivalent GraphQL documents for queries (or updates). Also processing the JSON responses that are returned and transforming these to XDS again.

    We use the existing tools available to us for this: DirXML-Script, ECMAScript, ability to call Java from ECMAScript.

    For basic XPath/XSLT querying and restructuring of JSON (which GraphQL also uses) - we use JMESPath, in our experience, this is a very performant and reliable.

    For roundtripping XML to JSON we use Jackson (which ships with IDM these days).

    I am not talking about having made a GraphQL client with all possible bells and whistles, only just enough to be able to pass relevant IDM operations to the transport layer and expect a reasonably usable response in return.

    So these problems are for my needs (at least) are already solved.

    ---

    The problems that remain lie with the polling model, but mostly with the authorization and error handling logic.
    Particularly these relate to how the REST shim has made lots of assumptions in the way it implements the transport layer for us.

    Here we do need to talk about how GraphQL interacts with the underlying transport layer (which is generally http, but could also be another transport layer entirely).

    First the error handling logic.

    GraphQL errors are fundamentally different from REST errors. Because it is transport agnostic, one can no longer strictly rely on status codes from the transport level.

    Instead, it is expected that the client inspect the returned body also and determine whether the errors listed there are cause to discard or retry the response.

    One can receive partial "success" data, where the errors indicate which sub-queries failed, one can encounter a network error or a query validation error (which generally means no results were returned at all).

    In the REST shim, almost all error handling occurs in the shim, it is tied explicitly to the transport layer (hard coded logic that relates to given HTTP status codes) and cannot really be overridden at an early stage (except for the HTTP status codes to retry on). There is currently no way to code a java extension that can inspect both the returned content and the transport information (http headers) at the same time to be able to make a properly informed decision as to how to handle errors. By the time the response is returned to DirXML policy, it is often too late to be able to take control over the error handling in a meaningful manner.

    This is not an issue specific to GraphQL, over time we have seen a lot of edge cases in implementing solutions on top of this shim. In the vast majority of scenarios I know of, if the REST shim could just expose via an extension that permitted an override the default http status handling, then we could support more services that don't match the very rigid assumptions of what HTTP status codes should be returned in specific scenarios.

    Regarding Authorization- this is a bit more niggling and really folds back into the error handling again and to assumptions made in the implementation of the transport layer that are not friendly to GraphQL.

    We have an example where the same OAUTH2 token that is issued when the user authenticates is reused by the GraphQL Server to run the sub-queries in the underlying backend. Apparently this is not an unusual scenario.

    A single GraphQL query can initiate many back-end sub-queries (could be database queries or REST queries) and rather than implement any authentication or authorization work in the GraphQL layer, in many cases the authorization HTTP header is simply passed right through to the underlying REST queries. The returned queries are aggregated and merged into the format as indicated in the GraphQL client's request. 

    However, it is not uncommon that ones authorization token has expired and needs to be refreshed/reissued. Normally in REST this would return a 401 error and the IDM shim will handle the renewal of the token/auth for you.

    GraphQL doesn't do this, it almost always returns HTTP 200 and expects you to look at the body of the response to understand whether an error actually occurred. This is totally incompatible with the assumptions made by the REST shim's hard-coded transport layer logic. With the ability to hook into error handling logic, we could solve this also.

    Finally: polling - with Rest shim, one can only define a series of URIs that are registered by the shim and polled using GET specific interval (determined by a shim param)

    The number of URIs is limited to 10, which is tiny. I found ways around this limit, but it becomes hacky.

    Also, whilst technically per spec - one can query via GraphQL using a GET, not all GraphQL servers actually support this and even if they do, with more complex and longer queries, one can easily trigger a 414 URI Too Long HTTP status response.

    The best practice is thus to send queries via a POST request, with application/json Content-Type and the GraphQL query specified in the request body. (something that is totally incompatible with the way the REST shim handles polling).

  • Would it not be "dangerous" to ask for a driver which does something very specific with a rest interface?

    Meaning GraphQL is using http as transport, which could be done with the REST driver, what is lacking is the xds2graphql which could be done in a library - but then we could end up with something as was done in the REST Drivers xds2json which is very specific to something which is almost SCIM.

    I would like to see something from MF/OT which is flexible so that it can be configured in such away that we do not need to modify the output to fit the GraphQL end-point.