Support for OTEL agent instrumentation and LangFuse

[alessandrocornacchia]

It would be very useful to have agent's executions neatly displayed in a dashboard and not just logged to .json for which manual inspection and filtering is cumbersome.

Do the folks plan to add support for LLM observability dashboard such as LangFuse?

[HacksonClark]

We have wandb, but I don't think it's what you're looking for.

We don't have any plans to support something like LangFuze, but the project is open source and we welcome external contributions!

[gaganso]

Hi @alessandrocornacchia , nice to see you here! We agree that it would be a great feature to add.

We have an industry collaborator who has added observability to understand agent trajectories. They have plans to push this feature but I will confirm and get an ETA on this. If that doesn't work out, like Jackson said, it would be great if you would like to contribute to the repo.

We can track the progress using this issue.

[alessandrocornacchia]

Thanks @HacksonClark and @gaganso for your prompt replies.

I am planning to add this with my team, though I cannot guarantee about the timeline. Once we have some progress, I'll follow up here.

[HacksonClark]

@alessandrocornacchia Sounds awesome! Feel free to reach out if you need any help.

[gaganso]

Hi @alessandrocornacchia, curious to know the progress and if there is anything we can do to help merge the changes!

[srmanda-cs]

Hi @gaganso @HacksonClark

Do you think I could work on this? Integrating this benchmark with an LLM observability dashboard? There's been no activity on this issue for about 7 months now with zero commits, I could have a proto ready within few weeks or a month.

[HacksonClark]

Hi @gaganso @HacksonClark

Do you think I could work on this? Integrating this benchmark with an LLM observability dashboard? There's been no activity on this issue for about 7 months now with zero commits, I could have a proto ready within few weeks or a month.

Yes! Please work on it.

[srmanda-cs]

SGTM! I'll come back to y'all with a plan about all the options we have, and what would be the simplest and least intrusive way to implement it.

[srmanda-cs]

This is gonna be a MAJOR cross cutting heavy change. Mainly because the current implementation of agents inside clients/ doesn't have access to the session_id, pid, step count, etc which these dashboards need for observability. We would first have to tackle issue #146 anyways, because otherwise we would have to make modifications separately to every client's code. Before that we would have to build some tests on top of issue #19 to test and build CI for each of the agents along with the no-op scenario to make sure we are not introducing any regressions into any of the clients. Plus, we would have to make sweeping changes across orchestrator/, session, and all of the clients, and llm.py

LangFuse would most need to be added to poetry. I wouldn't know if the changes would introduce any regressions until I actually write the code, but I can guarantee you the scope is a lot larger than it appears at first glance @HacksonClark

My personal take on this, is that the project is not yet ready for this kind of deep integration, at least not yet. I would recommend we focus on the cleanup, some other QoL changes, and any other things you want to see improved in the codebase before we tackle this.

[HacksonClark]

@srmanda-cs

My personal take on this, is that the project is not yet ready for this kind of deep integration, at least not yet. I would recommend we focus on the cleanup, some other QoL changes, and any other things you want to see improved in the codebase before we tackle this.

I agree!

[gaganso]

@srmanda-cs , great points, thank you! More broadly, I think AIOpsLab should provide primitives for users to build well-instrumented clients (the less instrumentation code, the better). Looking at a single client that implements the generic client/agent interface you recently added, and thinking about what generic instrumentation hooks/interfaces it needs, might be a good starting point for the design. We could think of this in phases (starting with session level and moving towards llm-call level)

Thinking about what do we want to observe - agent trajectory (action sequence, observations, decisions), LLM calls (tokens, latency, model), environment data (application telemetry which could be pulled from prometheus/jaeger, k8 state), etc., and what do we want to do with the observation - compare agents, improve agents, etc., would also be a good way to start designing. The granularity or unit of collection, where does responsibility of AIOpsLab end, etc are also worth considering.

At some point we could also snapshot the current client code as v1 for benchmarking/basellines and port them to the generic client with instrumentation built in.

[srmanda-cs]

A solid way to start would be to write tests and CI for the clients. Once that is done, we can be a bit more bold and confident in trying to generalize the interface as we can point out when we're introducing regressions. Post that, we can think about what data we want to be shared from the session to the client and expose the data we wanna observe. I'm willing to see this through with both your guidance. If it's okay with y'all, and you think this is a good place to start I can open an issue for the tests.