Populate queue from nodered

[Mateo_Corigliano]

Hello people, hope you are very well!

I’m trying to populate a queue from nodered with a form in openflow and then run it in openrpa. I did not find tutorials about it since the queues in all the examples are populated from openrpa (like on Workitems · open-rpa/openrpa Wiki · GitHub or youtube). Seeing that nodered has this possibility I would like to follow this path.

For example, in a first field of the form, place a web page and in a second field, another web page. That payload will be added to a queue as a workitem to be executed. Then, when executing the queue (manually or by calendar) take the workitem and openrpa execute the opening of the two web pages.

So, create the following flow in node red…

Populate step:

Play queue step:

If the input does not start from a workitem taken in “pop workitem”, the form correctly and directly passes the arguments to openrpa. But something is modified in the payload structure when the flow starts from a workitem so it doesn’t seem to recognize the parameters directly and I have to put a function between them.
I tried to put the following, but obviously it is not correct since although it executes it at the moment of “update workitem” I receive the following error “server error: unexpected end of JSON input”…
Function 1:

msg.payload = 
{ 
    url : msg.payload.payload.url,
    url2: msg.payload.payload.url2,
    _id : msg.payload.payload._id,
    submit: msg.payload.payload._id,
    submitbutton: msg.payload.payload.submitbutton
    
    };
return msg;

Then the workitem remains in a pending state even though the web pages have been opened.
What is the correct way to do this?

Thank you so much for your time!

[Allan_Zimmermann]

Hey
I have edit’ed your post, to make the code more readable.

I really like this question and your solution to it, but I don’t think that will work ( or I’m not completely understanding what you are trying to do )

The “workflow in” node will get activated every time the user press a button or a field is auto submitting, there for adding pop/push in there is generally not a good idea unless it’s right after completing the form instance.
Normally you return a form with no data on first activation, this add an instance id to the database, and every activation after this, will keep form state in the database, and ensure we have access to all of that in nodered. Then when the user submit/completes the form, you complete the workflow instance, and do something with the result.

So please correct me if this is not what you intended. I suggest something like this.

This will ensure we only create the workitem once the workflow is complete, and if you need additional logic, for looking up data from external resources or do more complex form validation, you can do that, and ensure only one workitem gets added. How ever this will not allow the user to see the result of processing, since we are completing the form, once we have the data.
There are two solutions to this, a bad one ( that could break ) and a more solid one, that will require a little more work.

• We could change the “form completed” to go into “processing” instead of completing, and then add logic that checks once the work item has completed and then add the result payload and set the state to completed.
  This can/will break if the user closed the webpage or navigates away, And if the user tries to come back AFTER the result has been set, the workflow will be hard to find, since it will be completed.
  And depending on the implementation it will also break/stop if nodered is restarted doing the flow.
• So we normally notify the user in other ways ( send email, text message, slack message etc. ) or we create a new form instance, with the result and some kind of confirmation ( you use the “Assign” node for this ) We can use queue “on success” for that.

For running the work item on the robot. There are diffidently cases where using the RPA node would make sense, mostly when you need a reply right away, but with workitems we try to keep it transactionaly safe, so here we should let openflow take of that. We do that by updating the workitem queue to allow openflow to send notification to the robot(s) when there is workitems to be processed.

Then in a workflow on the robot you add the exception catching logic, so openflow can retry when needed, and you don’t get workitems stuck in pending state.
That is what you need a workflow like this for

One thing i often see, is people “chain” the queues, for each part of the process.
So once the robot is done processing the workitem, you set a “success” queue on the workitem queue. This new queue is then responsible for notifying the user about a new workflow has been assigned to them.

This way it becomes very easy to report on workflows that is waiting on the user to acknowledge they have seen the result from the robot. Or what is more likely, you would push the failed items to this queue, to notify people about failed workitems, that they can then re-submit after updating the payload.

[Allan_Zimmermann]

A few notes.

• I experimented with the idea of using exchanges instead of queues on the webpage, to allow one or more uses to work on the same form (or have it open multiple times in different browser), that will also fix part of the above issue, but no one is using it, so that feature will most likely get removed again
  

• I have long been working on a complete rewrite of the openflow web interface and doing that time, i have long wanted to re-write the code for handling web forms, to be a workitem queue instead. This will save many of the users who use a flow similar to explained above much simpler, and it will make the database a lot cleaner too. My fear is I cannot make this backward compatibility, and i REALLY don’t like making breaking changes.

• You can control a form using an agent too. Here is an example of doing that
  GitHub - openiap/formagent

[Mateo_Corigliano]

Thanks as always Allan for the prompt and kind reply,

What I’m actually trying to do is to avoid handling the queue from openrpa in order to manage it completely in nodered/openflow and have the robot/openrpa only handle the executions. So the following that you recommend is exactly what I want to avoid:

Perhaps you have some reason why it is not advisable to manage it from node red?

That is why I am trying to pop workitem from nodered but I did not find any example that performs the flow in this way (populate a queue from nodered , then call the item from nodered as well and send it to openrpa to work on it).

The example I provided is as simple as possible for scalability: A form with two fields and a submit button. First field where a web is placed, second field where another web is placed. When sending, a workitem is loaded with the two websites to be opened. The queue is now populated with a workitem.

When executing the queue, it will read the only item that is pending and send it to Openrpa to open the two web pages, openrpa only now comes into play to open the web and then notify nodered that the job has been completed successfully, then node network will be able to mark the item as completed.

That flow is the one that I successfully replicated in the diagram sent above except that the item fails to be marked successfully and I can’t find the correct way to send the data from nodered to openrpa. Here it is critical to highlight that the data is not transmitted in the same way when the payload is sent directly from a form to openrpa as if it were sent from a workitem read to openrpa.

[Allan_Zimmermann]

Ah, well I guess nodered could be used for that as well.
But what confuses me is the workflow in node in the middle. You need to separate the form logic from the openrpa/workitem logic, or this will not be resilient to outages ( nodered restarting, the api nodes restarting, etc. )
If you want to interact with the robot in the same logic that handles the webpage, you should be using the RPA node, and not workitems.

If you want, I don’t mind taking a short 15- minutes online meeting talking about this, you might have a need I don’t quite understand, that is worth creating a video about

[Mateo_Corigliano]

good idea Allan, I will be contacting you on private next week to coordinate a call.
thanks a lot