4 weeks ago, I decided to learn Rust and decided to recreate all SDKs for OpenIAP using Rust as a backend as a way to learn it.
The reason was to enforce a consistent experience across all programming languages. I had hoped using protobuf “was enough,” but time/skill/language limits ended up making it far from a perfect experience across the different languages (Python does not support stream, so upload/download was not working. .NET was horribly behind since I was never really touching it, Node.js and the browser version are always “out of sync,” etc.).
I can normally “wing it” when learning a new programming language, but after 2 days with Rust, I was ready to kill anyone who talked to me. Goddamn, Rust was giving me a hard time, but after watching and reproducing every example in The Rust Book, I restarted my journey with Rust.
I was sure I would only take 1-2 weeks to get a “basic” version up and running using gRPC, and then I could start adding WebSocket, REST, TCP, and named pipes.
I was wrong… It has been an extremely fun journey, and I learned a lot along the way, but I’m almost done creating a Rust SDK that also works as a “backend” for a Node.js, Python, and .NET SDK (source code here).
I’m going away for a few weeks so I cannot work on it for a while, but I would love if someone would have a look at it and test it out.
I have finally managed to add all the proto buff functions to rust.
I have added unit testing for all, and slowly started on adding documentation.
The “core”/most used functions are also added to nodejs/python and dotnet, but i still need to add all of them.
I have pushed an update for all 4 languages as version 0.0.4
Once i have added all functions to all languages I will start work on adding support for WS/REST/TCP and named pipes … but for now, this is still gRPC only.
So I finally had some time to test the new api for dotnet
First thing that jumps out is that unless I’m doing something weird, the api seems to live within the top level global namespace, which makes it awkward for intellisense. Would probably be nice to wrap into some namespace for ambiguity error avoidance (especially since some of the names are by necessity very simple and common, like Client).
After reworking the older project to use the new one, when running from local the channel gets closed after 60 seconds, same as the old one. Not sure if it is some setting in our OpenFlow, but it’s very consistent on that.
And I haven’t found the OnDisconnect equivalent, so that makes it a bit harder to handle this time.
Checking queues...
Retry attempt 1 after delay of 250ms after exception: Client+ClientError: Pop workitem failed: ClientError("Custom Error Client Error channel closed")
I’ll try to test with a published agent, and later on against the app.openiap.io (probably the best test to see if it’s just a config issue), and will update here.
Aside of that, looks nice so far and the functionality as is seems to work. After I get the disconnection sorted out, will get it up and running as a background agent for a couple of days to check stabilities.
I find it very weird, you keep having those 10 minute disconnects.
We should investigate that a little more.
I like your idea of testing toward app.openiap.io … if you also have the problem there, the problem most likely is “in your end” and is not traefic or openflow.
Ok, so I checked some more things and:
When running locally against app.openiap.io it does not disconnect after the 60s, so it must be some configuration that differs.
If I publish the same agent openflow, it doesn’t initialize the client:
// my openflow
2024-09-10T09:13:02.182932Z INFO connect: openiap_client: Connecting to http://api
Unhandled exception. Client+ClientCreationError: Connection failed: ClientError("Failed to connect: transport error")
// app.openiap.io
Unhandled exception. Client+ClientCreationError: Connection failed: ClientError("Failed to connect: transport error")
2024-09-10T10:51:04.222362Z INFO connect: openiap_client: Connecting to http://api-grpc
Looks like a routing issue when running from docker?
Also a bit weird that the api endpoint is different.
Also as a sidenote - looks like when supplying the jwt for authorization, the Signin call is not needed? Seems to be “working fine” with just the .Connect() without the .Signin() call.
wow, thank you for putting time into it.
Sorry, i have not gotten to testing in docker yet, I’m working on adding telemetry now.
your logs look correct. On docker-compose install it should be connecting to host “api” but on port 50051
on kubernetes it should be connecting to host api-grpc and port 80 … both “insecure” since the tls layer is adding using traefik
The plan was to start working on adding websocket support, after telemetry, but i guess looking into docker is better ?
Ok, that’s useful to know - I can work with that
I worked around it with hardcoding the connection address to http://api:50051 (can parametrize around it for local runs), so feel free to set the priorities as you see fit
It’s not pretty, but works.
So far when running as a published agent within my openflow, the connection seems stable (still polls every 1s on queues). I’ll let it run for some time on mocked functionality and see if it’s stable.
EDIT:
The agent is running for…~20h right now, with 2 queue checks every second (~72k check iterations x2 queues). I don’t see any hiccups in the logs, but we have the reconnection handling, so even if there were some small ones it didn’t interrupt.
So while not the most wholesome test ever, it looks to be working better then the old API.
And it just stayed like that, looks like the build didn’t trigger. And no info on why it restarted.
Previous restart-on-uncaught-error went fine:
2024-09-11T13:11:15.783706931Z Iteration: 84222, 2024-09-11T13:11:15.7836148+00:00
2024-09-11T13:11:15.783754110Z Checking queues...
2024-09-11T13:11:15.789584242Z Nothing, waiting...
2024-09-11T13:11:16.789899922Z Iteration: 84223, 2024-09-11T13:11:16.7897166+00:00
2024-09-11T13:11:16.790210597Z Checking queues...
2024-09-11T13:11:16.808555109Z Process terminated. A callback was made on a garbage collected delegate of type 'openiap!Client+PopWorkitemCallback::Invoke'.
2024-09-11T13:11:17.529973655Z create client
2024-09-11T13:11:17.594138717Z Setting up server stream api:50051
2024-09-11T13:11:17.646151780Z connected!
2024-09-11T13:11:17.997916981Z package myagent-newapi
[build info follows and things start]
This might be more related to the dotnet6 image than the api itself, but it’s a bit hard to distinguish.
Which brings a question - how to monitor agents “automatically”?
The first one, i need to know where it’s “hanging” to have an idea on what might be wrong and hot to fix it.
If you don’t mind A LOT of debug information you can call
client.enabletracing("openiap=trace", "new");
There is a “bug” (?) with this function. Rust will not allow me to update the logging settings, so you can only call this function once, every call after that will have no effect.
Then if you also add a lot of verbose logging yourself in your code, so we can see if it stops in your or my code ? you can use my client.verbose and client.trace if you want an easy way to enable/disable log messages.
This is an error in my code. I need to use variables for the callbacks, so they do not get garbage collected. But luckily it also crashes the process, so it will restart when this happens, but i will ofc get this fixed.
a little info about enabletracing. This sets up a tracing filter ( this can also be set by using environment variable RUST_LOG )
all “my” code will be within the openiap space, so by using openiap=LEVEL you only get tracing information for that. If you want all you could simply use LEVEL. level can be either info, warn, debug, trace. The second parameter is for span’s, it defines if it should log every time it enters/exit a function, this can be empty to log nothing, valid values are new, enter, exit, close, active or full.
