Feature request: Nuget feed in OpenFlow?

Hi,

Was wondering if it would be feasible to add a nuget feed to OpenFlow for OpenRPA usage.
Since it already needs to authenticate to OpenFlow, that would save quite a bit of setup for attended end users for private feed scenarios.

That is a two-part thing.

  • OpenRPA: You can already do that. I have not made a UI for it, but other products do (like Visual Studio) have one. You simply edit C:\Users\%USERNAME%\AppData\Roaming\NuGet\NuGet.Config and add whatever feeds you need.
  • OpenFlow: I cannot remember what version numbers there are, but there is the “current” (3, I think) and one before (2). With version 2, everything is handled using OData-like interfaces to support searching and so on. In version 3, they also added support for simply generating a JSON file with all the information. If I remember correctly, OpenRPA needs both to support searching, and that is going to be a lot more work.
    I think I would lean toward just using one of the open-source NuGet servers out there in that case.
1 Like

Let me elaborate on the use case, as I think we’re not aligning.
Will not support is of course still a valid response, just want to make sure we’re talking about the same thing.

The use case is for streamlining attended end-user setup.
When using proprietary packages within OpenRPA (data models, crunching, API abstractions and things like).
When a new attended user is added, aside of setting up OpenRPA, they also need to set up the nuget feeds.
This is fine when we’re on the same domain, as we can use internally hosted feeds so the whole thing is just one script away in the config.
The issue pops up when the user is outside our domain. Due to policies and so on, we obviously can’t just put the packages in an open feed (nuget, myget etc.), and private feeds on those services come with other sets of compliance questions etc.
Yes, of course you can set up a nuget server on your own, but that means a separate auth for the user (and one more thing to host).

The feature request was for adding a nuget feed with the OpenFlow, as the user needs to authenticate there anyway to get the projects, so fetching those project dependencies from a different feed feels a bit off.

That said, if this is considered out of scope, that’s fine, but it would be wrong not to ask, as I can’t imagine we’re an isolated use case here (or maybe we are, and most setups do not involve writing own .dll’s for OpenRPA?).

I see. Yeah, having a NuGet feed in OpenRPA would make that easier.
You are correct, I’ve always pushed very hard for people to use agents or Node-RED for API calls and not OpenRPA, so not a lot of people use NuGet.
To be honest, most people are using Python either from inside OpenRPA or agents.

I know it’s not ideal … but …
OpenRPA simply tries to load all DLLs inside the extensions folder, so you don’t NEED NuGet to handle it. You could use OpenFlow to host the DLLs, then create a workflow that checks what DLLs it has access to and then download them. That way you keep it inside OpenFlow and use the permissions. But you need to run a workflow that checks and then copies the files into the extension folder.
A little hacky, I know.

I have super high hopes for adding git to OpenRPA. I guess we could look into also using that for distributing dll’s for the extension folder ?

API wrappers is one thing, but when working with on-prem DB’s being able to quickly whip out a model, instead of always working with datatables and fixing bazillion typos all over the place, is a godsend.

That won’t work if you have dependencies. And unless it’s pure DTO’s, sooner or later it will have dependencies. Even some DTO’s for us have dependencies (json serializations, datatable bi-directional handling etc.).

Let’s pause this for now if the demand is low, I’ll see if I can figure some low-maintenance solution outside of the stack for now.

Actually the OpenFlow “hack” could work by utilizing existing Nuget functionalities of OpenRPA, but the workflow would need to be more complicated to set up . Pseudo code:

string localNugetPath = ""; //some directory in either OpenRPA folder or Documents
if (!Directory.Exists(localNugetPath) 
{
     // no local cache
     Directory.Create(localNugetPath);
}
var nugetConfig = File.ReadAsXml(%appdata%/Nuget/Nuget.Config); // handle as xml (needs some boilerplate)
if (!nugetConfig.HasSource(localNugetPath)) // handling tbd
{
     nugetConfig.AddSource(localNugetPath); // handling tbd
}

var localPackages = Directory.EnumerateFiles(localNugetPath); // get what was cached locally already
var remotePackages = OpenFlow.ListFiles(collection: nugets); // not sure about the syntax from the top of my head, but you know what I mean here I hope
var packagesToDownload = remotePackages.Except(localPackages);
foreach (var packageToDownload in packagesToDownload)
{
    var downloaded = OpenFlow.GetFile(packageToDownload);
    File.Move(downloaded, localNugetPath);
}
StartProcess(waitForExit: false) -> restorepackages.ps1 {path: openrpa.extensions.folder}:
Process.Kill(openrpa);
Directory.Clean(path);
Process.Start(openrpa); // restore happens automatically on startup for imported projects

Or in short:

  • Ensure a known-location folder for package store exists
  • Ensure that folder is a nuget source
  • Check what’s there
  • Check what’s available in OpenFlow → nuget file collection (I think that’s possible?)
  • Download what’s not there
  • Use powershell “out of process trick” to clean up extensions and restart OpenRPA

This would mean that there would be an “overkill” on packages that someone has access to, but doesn’t use, so it could be improved, but something like that should work.

Thoughts?

Agree

there a plenty of ways to do it.
I think i would prefer using a combination of powershell and git, but using c# and invoke would as you suggest works too.

I’m just throwing ideas around to try to figure something out :wink:

I’m not going to go into ps scripting at almost midnight, but attached is a simple workflow that does the first part (before extensions clean-up/restart).
Rudimentarily tested.
PackageRestore.json (21.2 KB)

Needed to use VB since C# scripting in OpenRPA is having some namespace conflicts on my dev (which I’m not going to debug now, probably just a double import of something between the workflow and defaults of InvokeCode).
Do note that it very slightly messes up formatting of the nuget.config, so be careful when deleting the lines as you can invalidate the xml:

To test, upload some random .nupkg to OpenFlow → Files. It only looks at files matching “*.nupkg”.
“Clean” run:

Re-run:

Even without the clear extensions → restart part, it can still help to set up an “OpenFlow nuget feed” (in very big quotes :wink: ) for end users, so that’s something.

Off to sleep, will test this more tomorrow with some new end users (we’ll be onboarding a few more this week anyway, so… guinea pigs incoming :wink: )

EDIT:
Added a bit clearer logs, so it’s visible that it doesn’t just count but actually compares :wink:

2 Likes

ahhh, smart … adding a custom nuget repo and then just dumping the nupkg files into it …
That could also work for a more permanent solution, added into openrpa at some point.
i feel so stupid i missed that in your former post … Guess I’m tired too :smiley:
Love how fast you made something that works with that

It was middle of the night, after all :slight_smile:

Here’s a “final” version, although I’d still advise backing up extensions before testing.

PackageRestore.json (29.1 KB)

Put the .ps1 script in the OpenRPA folder (can’t attach .ps1 files, but it’s embedded near the end of the workflow in a commented out invoke code) and adjust the name in the workflow if needed. I used “CleanExtensionsAndRestartOpenRPA.ps1” in the script, but that can obviously be adjusted. So can the path, but it seemed like a natural place to have it.

There’s also a commented out -NoExit args version, so the errors in ps1 don’t just fly past if something is wrong.

Hopefully someone will find it useful. I think we’ll use this (or something very similar) for our out-of-domain attended users. HD Robots could actually use that too, I guess, until there’s a built-in way.

1 Like

Ok, this doesn’t work for the full intended purpose.

It does the local nuget feed nicely, and the extensions clean up with restart, BUT openrpa still thinks the packages are installed, so it doesn’t do an automatic restore. The workflows simply are treated as invalid (can’t be loaded due to missing dependency).

The “Update” button there actually is useful as a “restore” when the extensions are empty, ironically, as it then “updates” to the same version.
But it still leaves it up to the user (or robot maintainer) to do it.

So at best this helps to do a nuget feed sync to a local folder from OpenFlow. Kinda useful, and solves this topic, but not the package update issue.
I’ll need to think more about this case and see if we can solve it some other way. The issue of dependency reload, as mentioned, is tied to the single-process execution, so all of this is just becoming one giant hack around it.

Will tinker a bit more around it, but I’m starting to lean into just cleaning the board and seeing how else (other parts of the stack maybe?) can we fulfill the requirements. But damn if it isn’t a hard thought to forego custom packages in RPA processes :frowning:

(and now I see first hand why people are not using custom libraries in OpenRPA…)

I think my last reply in the other post is related to this

I think we should have a “shared” session and look at this.
I cannot easily get 2 windows machines running, so if you can have visual studio and a robot open and a second machine with a robot, then we could troubleshoot this together, so you can get a working solution.
If you are up for that, then I am too ( you know my calendly link, so you can find a free slot and then throw me a team’s invite )

Yeah, the topics are closely related.
I’ll see what I can find in the evening and schedule if needed.