SDK client.push_workitem not working in python while in callback

Issues and feature requests
, ,
1

Hi there, I have a self deployed OpenIAP instance and am trying to get the basic code interactions with the python SDK working.

I’m basing my code off of pythontest/test.py at main · openiap/pythontest · GitHub using a really simple script just to get some basic workitem interaction.

Basically if I run the code in a sync way like in the transient connections it works correctly:

client = Client()
client.connect()
signin_result = client.signin()
workitem = client.push_workitem( name="python test no files", wiq="rustqueue", payload="{}")

But if I run it like the persistent code example style, it never pushes the work item, seemingly locking within the lib at TRACE tokio-runtime-worker trace: openiap_clib: push_workitem_async called

client = Client()
def handle_event(event, count):
    if event["event"] == "SignedIn":
        workitem = client.push_workitem( name="python test no files", wiq="rustqueue", payload="{}")
try:
    client.connect()
    signin_result = client.signin()
    client.on_client_event(callback=handle_event)
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    client.info("Shutting down...")
except Exception as e:
    client.error(f"Error: {str(e)}")
finally:
    client.disconnect()

Loading into the package lib I can see it’s never passing

self.lib.push_workitem_async(self.client, byref(req), cb)
self.trace("push_workitem_async called")
event.wait()

So I know you’ve stated years ago in other threads @allan_zimmermann that python was a weak point for you, but this is some of your published test code, so either something has fundamentally changed or I’m not doing a python thing that’s necessary to make async code work when nested in callbacks.

I’m not advanced enough to dig further into self.lib.push_workitem_async so any guidance here would be appreciated. I’m just trying to use the SDK to push some workitems and then build another agent that reads said items at another stage for processing (I’ve found code examples and can work around for now using transient methods, but I’d rather have a cleaner persistent connection that does all my work here queuing within callbacks and listeners)

2

Hey
I’m a little rusty on the Python SDK; maybe @Rutger_van_Schoonhov can help here?
But I’m 99% sure that you cannot do an async call from a callback with the OpenIAP SDK right now. So, in the main thread, you are free to use sync or async functions, but callbacks (that run in a different thread) always have to be synchronous…

3

This is a deadlock, not a bug in your Python. The callback you pass to on_client_event is invoked on the SDK’s internal dispatcher thread (the tokio worker you see in the TRACE line). client.push_workitem() is a blocking wrapper: it fires push_workitem_async and then does event.wait() until the SDK delivers the response — but the response is delivered by the same dispatcher thread that is currently stuck inside your callback. It’s waiting on itself, which is why it hangs exactly at event.wait() and why the identical call works fine in the transient/sync example (there the main thread blocks and the SDK thread is free to answer).

Rule of thumb: never make a blocking SDK call from inside an SDK callback. Keep the callback tiny — record the event and return immediately — and do the blocking work on the main thread. Your example, minimally reworked:

import queue

client = Client()
events = queue.Queue()

def handle_event(event, count):
    events.put(event)  # never block in here

try:
    client.connect()
    client.signin()
    client.on_client_event(callback=handle_event)
    while True:
        event = events.get()  # main thread consumes
        if event["event"] == "SignedIn":
            client.push_workitem(name="python test no files",
                                 wiq="rustqueue", payload="{}")
except KeyboardInterrupt:
    client.info("Shutting down...")
finally:
    client.disconnect()

Same persistent connection, but now push_workitem blocks the main thread while the SDK’s dispatcher thread stays free to deliver the response. (If you’re on asyncio instead, the equivalent bridge is loop.call_soon_threadsafe(…) from the callback plus the SDK’s async variants.)