Opik for Observability and Optimization: Feedback Loops for Better AI Applications

As work on AI applications – agents, tool uses, external memory modules, chain-of-thought and related methods – becomes more sophisticated, the need for evaluating results and putting the results of evaluations to good use becomes more acute. What’s been emerging is a feedback loop involving observation and optimization.

Opik is a popular open source platform which gives you all the tools you need for LLM observability (logging, debugging, evaluations, experiments), enabling end-to-end AI Engineering through development, testing, and production. Plus there’s an open-source SDK for agent and prompt optimization. Open a browser-based dashboard for your application and see how the evaluations of LLM integrations have run on different sessions. At last count, Opik includes 52 integrations with popular Python tooling for AI (we counted) plus more for Ruby, TypeScript, .NET, and Java. Then the agent optimizers include GEPA, HRPO, and more, and can be extended.

If you step back and look in big picture terms, this feedback loop is essentially the same used in reinforcement learning, where agents learn and improve through feedback based on how they interact with their environment.

This is a real world example of a math-guided design. This feedback mirrors the dopamine signaling in animal brains which regulates emotional and motivational behavior through “rewards” given by neurotransmitters.

Claire Longo is a Lead AI Researcher at Comet AI, working on Opik. Senzing has been using Opik in our AI tutorials, and Claire’s talks about leveraging math-driven thinking to build better AI caught our attention, especially her analogies which help explain relatively complex mathematical approaches in terms of real world examples. Graphs come into play here, since context from knowledge graphs can help guide agents toward better outcomes, and also the “competency questions” used for knowledge graph construction feed nicely into generating evaluations to use for LLM observability. Let’s talk optimization with Claire!