Tracing, logging and monitoring AI systems and apps in production.
No, LLMs cannot currently carry a pager or fully handle incident response on their own. Despite unlimited tokens and proximity to researchers, reliability teams still rely heavily on human intervention for complex reasoning.
▶ 2:54
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Most vendors accept OpenTelemetry data but immediately convert it into their own proprietary internal formats, destroying the original semantic tags and context. Dash Zero avoids this by keeping the data strictly in the OpenTelemetry format throughout the platform.
▶ 6:31
Why Most Observability Platforms Won't Survive the AI Agent Era
Humans naturally respond to conversational cues within 300 milliseconds. If an AI response takes longer than 500 milliseconds, users will immediately notice, and delays over a second will cause them to hang up. Therefore, keeping latency extremely low is critical for usability.
▶ 2:55
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Subjective tasks lack clear deterministic pass-fail criteria like string matching or symbolic verification. When we replace a deterministic grader with a non-deterministic LLM judge, we introduce massive noise and fluctuations. This makes it very difficult to draw reliable conclusions from our evaluation results.
▶ 1:22
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
LLMs excel at superhuman signal gathering by querying metrics, reading logs, and finding needles in haystacks. They can run parallel PromQL queries rapidly and process thousands of log lines without getting tired.
▶ 11:02
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Because OpenTelemetry is an open-source and openly documented standard, AI models are trained on its text-based syntax and semantics by default. This allows models to parse spans, logs, and status codes directly without needing custom parsers.
▶ 11:02
Why Most Observability Platforms Won't Survive the AI Agent Era
The standard pipeline streams audio chunks through an agent orchestrator to three sequential components. First, a speech-to-text system converts audio to text, which is then fed into an LLM to determine tool calls and responses. Finally, a text-to-speech model turns the text output back into audio chunks for the user.
▶ 4:36
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
You start by partnering with subject matter experts to create clear quality guidelines using binary or Likert scores. Then, you extensively calibrate this process by taking multiple human evaluation samples for the same data and updating guidelines based on disagreements. This iterative process acts as your North Star for measuring and improving task quality over time.
▶ 4:32
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Instead of opening a blank chat, engineers should pass the alert's infrastructure-as-code expression directly to the AI agent. Instructing the agent to be curious allows it to quickly check endpoints, regions, and historical traffic patterns.
▶ 30:16
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
While individual traces can be dropped directly into models, large-scale anomaly detection requires exposing backend features like triage tools via Model Context Protocol servers. This lets autonomous AI agents query millions of data points and receive precise correlated results.
▶ 15:27
Why Most Observability Platforms Won't Survive the AI Agent Era
The primary metrics are quality, measured by word error rate, and latency, measured by time to complete a transcript. State-of-the-art models typically achieve around a 6% word error rate on open benchmarks. Getting this right is vital because downstream LLMs and TTS models will inherit any transcription errors.
▶ 5:25
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
You can use a statistical concept called power analysis to estimate the required volume of data. This estimation looks at factors like your desired effect size, the observed noise in resampled completions, and your target significance level. Using these factors ensures you have enough data to confirm model performance differences are legitimate rather than noise.
▶ 6:19
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
You can instruct the model to use trace IDs to follow a request across disparate logging systems and clouds. Claude builds a coherent timeline in its context, helping identify where timeouts or errors occurred.
▶ 31:52
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Static chat interfaces merely output a final answer or rigid artifact like an image, preventing further user manipulation. An interactive workflow allows developers to modify parameters, filter data, and continuously collaborate with the agent to refine troubleshooting results.
▶ 22:27
Why Most Observability Platforms Won't Survive the AI Agent Era
Older batch models like Whisper were trained on 30-second audio clips, requiring complex chunking and padding workarounds for streaming. Newer streaming-native models use specialized encoders trained with short look-ahead times and cached activations. This allows them to process small audio steps without redoing heavy computations.
▶ 7:25
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Evaluation datasets should focus heavily on diversity and realism to accurately reflect production tasks. They also need to be difficult and discriminative so they can effectively differentiate between various model capabilities. Achieving this accuracy usually requires a combination of model and human-based filtering over time.
▶ 7:18
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
LLMs frequently confuse correlation with causation when interpreting system graphs and metrics. For instance, when a KV cache blows up and traffic doubles alongside errors, the model wrongly diagnoses a capacity problem rather than a cache failure.
▶ 22:52
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
If observability tools fail to provide value inside their own platforms, they risk being commoditized into mere databases. To survive, they must empower all users to troubleshoot via agents and integrate broader external context sources like Jira or Notion.
▶ 26:25
Why Most Observability Platforms Won't Survive the AI Agent Era
LLM streaming latency is measured by time to first token, and you generally want to hit 200 to 300 milliseconds. To meet this latency budget while maintaining sufficient intelligence for tool calling, models typically need to be in the 8 to 30 billion parameter range. Going larger blows the latency budget, while going smaller hurts reasoning capabilities.
▶ 8:44
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
LLM judges rely on high-level quality guidelines to evaluate solutions similarly to how humans do across various tasks. Rubrics use a task-specific checklist of individual items that are separately verified by an LLM judge to make the process more objective. Both serve as automated proxies to scale evaluation when human review becomes a bottleneck.
▶ 8:39
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
SRE automation can be mapped to self-driving car levels ranging from assistance to full autonomy. Most mature systems operate around level two with automated observability, while human engineers still handle mitigations and fixes.
▶ 28:33
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Agent-first design shifts the primary interface away from traditional charts and numbers toward textual status descriptions and direct agent interaction. While charts were built for human visual weaknesses, agents perform deep data analysis directly on underlying data. Users now review agent summaries and investigate flagged anomalies instead of scanning raw dashboards.
▶ 41:25
Why Most Observability Platforms Won't Survive the AI Agent Era
Performance is evaluated using time to first audio and the real-time factor, which measures how many seconds of audio can be produced per second of processing time. Quality is also critical and is best assessed by directly listening to audio samples rather than relying solely on objective metrics. You typically want a real-time factor under one to avoid buffering issues.
▶ 9:50
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
You perform meta-evaluation by measuring the agreement between your automated judge and golden data from trusted human evaluations. You can evaluate this using correlation metrics, binary accuracy, or by testing if judge outputs correlate causally with real production metrics like user abandonment. You should aim for a judge-human agreement level that matches the agreement seen between different humans.
▶ 9:58
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
LLMs can aggregate Slack channels and meeting transcripts to generate postmortem drafts using standard templates. However, because they struggle with root causes and company history, teams must review them and insert human to-do items.
▶ 35:50
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Troubleshooting complex microservice architectures traditionally requires expert knowledge held by only a few people in an organization. AI agents bridge this gap by possessing a wider system scope and guiding regular developers through the right troubleshooting steps. This reduces reliance on tribal knowledge when key experts leave the enterprise.
▶ 46:34
Why Most Observability Platforms Won't Survive the AI Agent Era
Running models in distant data centers introduces network latency that adds up quickly in real-time systems. Co-locating your models and agent orchestrator in the same data center can drop network latency from 75 milliseconds down to 5 milliseconds. This yields a massive 30% reduction in total latency for an already optimized setup.
▶ 13:33
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
You can apply a kitchen sink of statistical methods to establish tight confidence intervals and report error bars alongside your results. Techniques like power analysis help determine benchmark size, while resampling model and judge rollouts helps reduce variance. This ensures that reported performance improvements reflect actual capability gains rather than random noise.
▶ 12:02
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Automated fixes remove the feedback loop that helps engineers learn how systems behave during failures. Senior engineers risk scar tissue atrophy, while junior engineers miss out on training from handling routine incidents.
▶ 39:49
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Pure speech-to-speech models process audio natively without converting it to text first, preserving tone and emotion while enabling full duplex communication. However, they currently struggle with complex instruction following and reliable tool calling in production. Developers often resort to prompt engineering before eventually falling back to pipeline architectures.
▶ 15:10
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
You can track explicit metrics like thumbs-up and thumbs-down feedback or preference ratings commonly seen in chatbots. Alternatively, you can use implicit metrics like session length and abandonment rates to gauge performance. These production quality signals help you evaluate whether your deployed agent is functioning as intended.
▶ 13:43
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Jevons paradox states that increased development efficiency lowers costs and causes software consumption to rise instead of fall. This ever-growing system complexity leads to more microservice interactions, more interesting failures, and ultimately more incidents.
▶ 41:46
Anthropic Reliability Engineer EXPOSES the Limits of AI SRE
Tool calling evaluations assess whether the generated function call structure is correct and whether the output is feasible for the use case. Teams often aim for near 100% structural correctness in their tool calls. To achieve this with smaller LLMs constrained by latency budgets, developers fine-tune models on use-case-specific data.
▶ 17:35
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
A/B testing allows you to measure production metrics and gain a relative sense of how your new agent performs compared to existing solutions. Instead of pushing the model to all users at once, you can vet its quality safely in a live environment. Power analysis can help determine the necessary duration and sample size for running these tests.
▶ 14:11
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Guardrail models and classifiers can be introduced right before the main LLM to route requests, or placed at the end of the generation step to screen responses. While adding these components introduces extra latency and management overhead, catching issues before the TTS model runs is essential. If a policy violation slips through, you cannot take back spoken words.
▶ 20:13
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
You can continuously monitor production to identify concerning performance trends and uncover novel failure cases or use cases. By feeding these real-world failure examples back into offline evaluations and human reviews, you ground your eval pipeline in reality. This creates a continuous improvement cycle where every part of the framework is regularly refined.
▶ 15:02
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
The thinker-talker pattern uses a small LLM to instantly generate a filler response like letting the user know it is thinking. Simultaneously, it issues a tool call to a larger, more capable model with better instructions and guardrails. This pattern helps hide latency and improves response quality when your main model is too slow for real-time conversation.
▶ 21:34
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Future speech-to-speech models eliminate traditional text-to-text and text-to-speech intermediate steps, rendering standard component-level evals obsolete. Instead, teams must rely on transcription sidecars for auditability and shift toward longer, full duplex conversational metrics. Most of these comprehensive evaluations will happen on top of the base inference API.
▶ 23:32
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
No answer mentions that.