I’m able to use OpenIAP Agent to run a python script that performs RPA actions and it uses pyautogui library. It works great when I select it locally though the openIAP agent that is connected with a user that I have created, however when I try to run it from openIAP, I get errors, because it is looking for a display I guess. Any ideas how can I just activate the local agent from openIAP? Would love to just activate the local RPA script and let it perform all the necessary mouseclicks and keyboard actions. The local agent performs a specific action, in this case, makes an appointment on a calendar. Any assistance is greatly appreciated! Thank you!
Hi, could you share the error shown on openIAP?
Sure! Here is what I see when I try to run the agent:
undefined
Child process 2597 exited with code 1
KeyError: ‘DISPLAY’
File “”, line 679, in getitem
~~~~~~~~~~^^^^^^^^^^^
_display = Display(os.environ[‘DISPLAY’])
File “/home/openiapuser/.local/lib/python3.11/site-packages/mouseinfo/init.py”, line 223, in
import mouseinfo
File “/home/openiapuser/.local/lib/python3.11/site-packages/pyautogui/init.py”, line 246, in
import pyautogui
File “/home/openiapuser/.openiap/runtime/rwg2fro4uij9uonjw41me/main.py”, line 1, in
Traceback (most recent call last):
Child process started as pid 2597
Child process 2593 exited
Requirement already satisfied: protobuf in /home/openiapuser/.local/lib/python3.11/site-packages (5.28.3)
Defaulting to user installation because normal site-packages is not writeable
Child process started as pid 2593
Child process 2496 exited
Successfully installed Pillow-11.0.0 grpcio-1.67.0 mouseinfo-0.1.3 openiap-0.0.37 protobuf-5.28.3 pyautogui-0.9.54 pygetwindow-0.0.9 pymsgbox-1.0.9 pyperclip-1.9.0 pyrect-0.2.0 pyscreeze-1.0.1 python3-Xlib-0.15 pytweening-1.2.0 websockets-13.1
Installing collected packages: pytweening, python3-Xlib, pyrect, pyperclip, pymsgbox, websockets, pygetwindow, protobuf, Pillow, mouseinfo, grpcio, pyscreeze, openiap, pyautogui
Successfully built pyautogui pygetwindow pyscreeze pytweening mouseinfo pymsgbox python3-Xlib pyperclip pyrect
Stored in directory: /home/openiapuser/.cache/pip/wheels/c4/e9/fc/b7a666dd4f9a3168fb44d643079b41d36ddab52f470707e820
Created wheel for pyrect: filename=PyRect-0.2.0-py2.py3-none-any.whl size=11180 sha256=16a6885a746dbf03e43415de06f195e35868cfa353eb45ba12ef388f3c570ea9
Building wheel for pyrect (setup.py): finished with status ‘done’
Building wheel for pyrect (setup.py): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/e8/e7/56/591cb88ba1783b38c40d584026e766aac9c3a048e34128ce8b
Created wheel for pyperclip: filename=pyperclip-1.9.0-py3-none-any.whl size=11005 sha256=430b929ed970625a722d66b208a3250676a2fc1125adfcc19d1d64d368fd9d59
Building wheel for pyperclip (setup.py): finished with status ‘done’
Building wheel for pyperclip (setup.py): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/58/d2/1b/c2247396a6c5fbb1900ccf1ec3b73482d428b137dcdf1d99de
Created wheel for python3-Xlib: filename=python3_xlib-0.15-py3-none-any.whl size=109501 sha256=67f5040b24b5134550160e1979bc39e53846b0db2ae8427b6a27c00b525b4f20
Building wheel for python3-Xlib (setup.py): finished with status ‘done’
Building wheel for python3-Xlib (setup.py): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/85/92/63/e126ee5f33d8f2ed04f96e43ef5df7270a2f331848752e8662
Created wheel for pymsgbox: filename=PyMsgBox-1.0.9-py3-none-any.whl size=7406 sha256=029a9f422f0b48a1064cf91c9a0abc562e52b973f0f3f39bd7012fdd8a0457f0
Building wheel for pymsgbox (pyproject.toml): finished with status ‘done’
Building wheel for pymsgbox (pyproject.toml): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/20/0b/7f/939ac9ff785b09951c706150537572c00123412f260a6024f3
Created wheel for mouseinfo: filename=MouseInfo-0.1.3-py3-none-any.whl size=10891 sha256=af441d6054196235df5b1b865ab68f8eb0db1221649e89a2373f057dbaaa9923
Building wheel for mouseinfo (setup.py): finished with status ‘done’
Building wheel for mouseinfo (setup.py): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/db/81/dc/0d61a3c9614f288e057ab63924e2a49edbeed4ffc916dcda1e
Created wheel for pytweening: filename=pytweening-1.2.0-py3-none-any.whl size=8010 sha256=3bc7332d0ab484cfb0354df3ed73c00cc3b798b3509ffa3a9debef74735138cf
Building wheel for pytweening (setup.py): finished with status ‘done’
Building wheel for pytweening (setup.py): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/cd/e3/dd/267b393d8e8f607e47194942740d080d9bfd835cd4375a3de1
Created wheel for pyscreeze: filename=PyScreeze-1.0.1-py3-none-any.whl size=14365 sha256=e8cb91c7fad3dc7e84d737a31312e98a0df41da88577c305417a9f32b0c38da5
Building wheel for pyscreeze (pyproject.toml): finished with status ‘done’
Building wheel for pyscreeze (pyproject.toml): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/07/75/0b/7ca0b598eb4c21d43ba4bcc78a0538dfcf803a5997da33bc19
Created wheel for pygetwindow: filename=PyGetWindow-0.0.9-py3-none-any.whl size=11064 sha256=c6e2815f665115a910c0b76e0720564ea5eb03dabb0c36671ca87f73dcbf49b7
Building wheel for pygetwindow (setup.py): finished with status ‘done’
Building wheel for pygetwindow (setup.py): started
Stored in directory: /home/openiapuser/.cache/pip/wheels/95/dc/b1/fe122b791e0db8bf439a0e6e1d2628e48f10bf430cae13521b
Created wheel for pyautogui: filename=PyAutoGUI-0.9.54-py3-none-any.whl size=37575 sha256=2bbbc9c09d80ee1e4b6d570277b0ac7802cdd3d566811bdd20c31ab2982f0448
Building wheel for pyautogui (pyproject.toml): finished with status ‘done’
Building wheel for pyautogui (pyproject.toml): started
Building wheels for collected packages: pyautogui, pygetwindow, pyscreeze, pytweening, mouseinfo, pymsgbox, python3-Xlib, pyperclip, pyrect
Preparing metadata (setup.py): finished with status ‘done’
Preparing metadata (setup.py): started
Downloading pyperclip-1.9.0.tar.gz (20 kB)
Collecting pyperclip
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 57.7 MB/s eta 0:00:00
Downloading pillow-11.0.0-cp311-cp311-manylinux_2_28_x86_64.whl (4.4 MB)
Collecting Pillow>=9.3.0
Preparing metadata (setup.py): finished with status ‘done’
Preparing metadata (setup.py): started
Downloading PyRect-0.2.0.tar.gz (17 kB)
Collecting pyrect
Preparing metadata (setup.py): finished with status ‘done’
Preparing metadata (setup.py): started
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.8/132.8 kB 20.4 MB/s eta 0:00:00
Downloading python3-xlib-0.15.tar.gz (132 kB)
Collecting python3-Xlib
Preparing metadata (setup.py): finished with status ‘done’
Preparing metadata (setup.py): started
Downloading MouseInfo-0.1.3.tar.gz (10 kB)
Collecting mouseinfo
Preparing metadata (setup.py): finished with status ‘done’
Preparing metadata (setup.py): started
Downloading PyGetWindow-0.0.9.tar.gz (9.7 kB)
Collecting pygetwindow>=0.0.5
Preparing metadata (pyproject.toml): finished with status ‘done’
Preparing metadata (pyproject.toml): started
Getting requirements to build wheel: finished with status ‘done’
Getting requirements to build wheel: started
Installing build dependencies: finished with status ‘done’
Installing build dependencies: started
Downloading pyscreeze-1.0.1.tar.gz (27 kB)
Collecting pyscreeze>=0.1.21
Preparing metadata (setup.py): finished with status ‘done’
Preparing metadata (setup.py): started
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 171.2/171.2 kB 25.5 MB/s eta 0:00:00
Downloading pytweening-1.2.0.tar.gz (171 kB)
Collecting pytweening>=1.0.4
Preparing metadata (pyproject.toml): finished with status ‘done’
Preparing metadata (pyproject.toml): started
Getting requirements to build wheel: finished with status ‘done’
Getting requirements to build wheel: started
Installing build dependencies: finished with status ‘done’
Installing build dependencies: started
Downloading PyMsgBox-1.0.9.tar.gz (18 kB)
Collecting pymsgbox
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.8/164.8 kB 26.4 MB/s eta 0:00:00
Downloading websockets-13.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (164 kB)
Collecting websockets
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 25.1 MB/s eta 0:00:00
Downloading grpcio-1.67.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)
Collecting grpcio
Child process started as pid 2496
**** Running pip install
So, this exact script works when I run it manually on my virtual box via openIAP agent application. However, when I call it from openIAP and try to run the agent using this package, it fails. I don’t have anything fancy in the script, just some python code that has an openIAP instance and packed/pushed using openIAP VS code assistant
The node agent will run as a service and not have access to the screen, mouse, and keyboard. On Linux (maybe also macOS, not sure), it will use xvfb to run inside a virtual desktop. On Windows, nothing like that is possible, so here you can use the openrpa HD service to spawn an assistant instead.
1 Like
Possibly, I’m not using the right approach? I want to have the capability of mouseclicks and gui navigation using python by an unattended bot that is activated by openIAP.
If approach is correct, would I just create a new workflow in OpenRPA, then create a sequence that opens the OpenIAP application with an Open Application activity and then add another step to the openrpa sequence that clicks on my script within the agent? Or, is there an activity within openRPA that can interact with the OpenIAP agent application?
You have 3 options.
- On Windows, you can install OpenRPA … when you say unattended, that normally means running inside an RDP session without user interaction handled by a High-Density service.
From OpenRPA, you can useInvoke Codeto run Python code (there are two installers, one for Python 3.11.4 and one for 3.7.3).
You say you need to be able to do mouse clicks, this requires a desktop, so
- On Docker/Kubernetes/Linux, you can use/install nodeagent GitHub - openiap/nodeagent. This service will allow you to schedule or run ad hoc packages written in either C/C++, Rust, Microsoft .NET, Python, or Node.js. The Docker images with Chrome have xvfb installed. If you run it yourself on a Linux machine, you need to install that as well. Then the nodeagent will run any packages with xvfb, and then you can do mouse/keyboard/screen automation in there (so unattended).
- Or you can install the assistant and run it inside a user session. This is the “GUI” version of the nodeagent, so this will also allow you to run packages written in either C/C++, Rust, Microsoft .NET, Python, or Node.js. But since it’s running inside a user’s desktop, you can do whatever you want. If you NEED this to be unattended, the only complete solution we have is combining the HD service (made for OpenRPA) to run the assistant … and in that case, you need to decide if you prefer using OpenRPA as the running or a node agent.
Both OpenRPA and the nodeagent ( / assistent ) can be controlled from openIAP
Thank you very much, Allen. I was able to get it to work by creating a workflow in openRPA that opens the OpenIAP “gui” assistant program and clicks on the package, then getting it to run from OpenIAP via a work item. It’s a little rough around the edges, so now I’ll setup some steps to verify where I am in the bot sequence so it doesnt go haywire. I appreciate all the help, hope this helps someone else as well!
If you are processing “units” of work, you should be using work items and work item queues. If you are controlling a flow from an agent (your code or a workflow in Node-RED), you should be using message queues. So in your Python code, you call registerqueue and either wait for your other agent to send a message or wait for OpenFlow to notify you about new work items and pop them using pop_workitem. This makes it clean and easy to scale. Using OpenRPA to run a package in the assistant is definitely a messy setup that is bound to break at some point.
1 Like
So, I could make the bot in Python even though I am using pyautogui, I would just need to set it to await a message or work item? Thus, I won’t need openRPA? Or, should I do everything in openRPA?
I tried doing everything in python, but was unsuccessful. I attempted to write the entire script in Python, bypassing the OpenRPA setup, and was able to connect to openIAP. The issue arose as to how to accept and process work items.
I would like to send it a payload of {
“APPT_DATE”: “11082024”,
“APPT_TIME”: “11:00 AM”,
“FIRST_NAME”: “John”,
“LAST_NAME”: “Doe”,
“EMAIL”: “john.doe@example.com”
}
as these are variables that I am using in the automation script. Received Error, Error message
Cannot read properties of undefined (reading ‘DownloadFile’).
I examined the openiap/pythontest repo, but this example was illustrating working with files. The basic function I am trying to do is have a local bot scheduling an appointment on the calendar (using GUI functions, mouseclicks, keyboard, etc.) when receiving a JSON with the appointment details from openIAP.
Here is an example in Python on how you could consume a queue. When you get a message, start processing items.
So you add your PyAutoGUI code inside the __ProcessWorkitem function.
Then publish that as a package in OpenFlow (using the vc.code extension). Make sure you set “daemon”: true in package.json.
In OpenFlow, create a queue that matches the name you want to use (in the example “pyagent”) and make sure the queue has itself selected under “amqp queue” after you create it.
This will make OpenFlow send a message to that queue whenever there are items in the workitem queue that are ready to be popped.
Then under agents, find the assistant you are running on the client where you want to run your PyAutoGUI code. Edit it and schedule your package to run on that agent. (If the agent is running as a different user than the one you used to publish the package, go to entities, select agents collection, find your package, and add the assistant user to the package with read and invoke permissions.)
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.