New Python Executor server is introduced in HCMX 2023.05.
Switch from using an interpreter to using a new Python Executor that is based on a Python server with framework and library to evaluate all python-based pressions through REST API calls.
This feature will be applied on the Cloud Slang project and will affect all the workers within OO Central, OO RAS and OO Workflow Designer.
Each worker will have its own python executor server configured and running.
There will not be an option to use the python executor server from another worker
No changes are made to OO components or APIs.
No changes are made to the OO DB.
Main motivation for this architectural feature change is to dramatically improve performance for CloudSlang based flows which are affected by the slow evaluation of python-based expressions with the previous installed Python interpreter.
On average, the performance increase is about 95% with the selected 3rd party Python framework and server.
When I upgraded the OO RASes from 2020.02 to 2023.05, I did not choose to use the python executor, so I thought it wouldn't be configured. However, it created a "python-executor.properties" file and starts up. What would be the cleanest way to stop this happening, please? I think it may be causing issues with JVM memory errors.
2023-11-20 23:43:30,564 [WrapperSimpleAppMain] (ExternalPythonExecutionEngine.java:49) INFO - python timeout strategy: scheduled-executor 2023-11-20 23:43:34,805 [WrapperSimpleAppMain] (PythonExecutorConfigurationDataServiceImpl.java:55) INFO - Reading properties from: python-executor.properties 2023-11-20 23:43:34,820 [WrapperSimpleAppMain] (PythonExecutorLifecycleManagerServiceImpl.java:171) INFO - A request to start the Python Executor was sent 2023-11-20 23:43:36,941 [WrapperSimpleAppMain] (PythonExecutorLifecycleManagerServiceImpl.java:213) INFO - Starting Python Executor on port: 8002 2023-11-20 23:43:36,957 [WrapperSimpleAppMain] (PythonExecutorLifecycleManagerServiceImpl.java:230) INFO - Waiting to start 2023-11-20 23:43:40,235 [WrapperSimpleAppMain] (PythonExecutorLifecycleManagerServiceImpl.java:234) INFO - Python Executor was successfully started
even if the performance optimizations are very tempting, I would currently advise not using the new Python-Executor.
Why is it currently not recommended to use the Python executor?
After updating to OO 2023.05, we discovered that incorrect values were returned as input or output when executing the same workflows at the same time. Specifically, we are using the Cloudslang function “cs_json_query”. The response from support: “Switch back to the classic Python-Interpreter”.
And the problem here is not a small one: It's about data integrity in the ongoing workflows. Incorrect values are used - depending on the purpose of the workflow, this can have serious consequences.
We suspect the cause is the use of Uvicorn, but we don't yet know a real solution. However, the use of Uvicorn does not seem to be thread or transaction safe.