top of page

semarx

Structural drift monitoring for LLM pipelines

Measures how much each prompt shapes the response it draws, and how much each response shapes the next prompt. Raw prompts and responses only: no labels, no reference data, no predefined rules. Blind test: you send prompt and response logs that include conversations you already know went off track, unlabeled; we return where we see drift; you score us against your own records, before any commitment.

Diagram of what Semarx measures in an LLM pipeline: how much each prompt shapes the response, and how much each response shapes the next prompt
Comparison for LLM pipelines: usual monitoring checks the quality of each answer with reference answers, rules or a judge model; the LLM information monitor checks whether answers still build on what came before, needs only the prompts and responses, and reports when the link between turns weakened.
Comparison for LLM pipelines: usual monitoring checks the quality of each answer with reference answers, rules or a judge model; the LLM information monitor checks whether answers still build on what came before, needs only the prompts and responses, and reports when the link between turns weakened.

Results

  1. Two loop-level metrics: communication closure (does what the pipeline returns at one turn match what it faces next?) and conditional action contribution (how much does a message resolve the reply that follows?).

  2. Pairing signal: swapping a response for one from another turn removes 87-92% of the measured contribution, with surrounding prompts untouched.

  3. Direction: in human-LLM conversations one side dominates in 57 of 58 conversations; human-human dialogue is balanced.

  4. Data: 2,171 human-human, 58 human-LLM and 8 LLM-LLM dialogues.

  5. ​Per stage: retrieval, memory, tools and handoffs can each carry their own reading. A blind test on your own traffic is the natural next step.

Is that for you?

Chatbots and assistants, agent chains and tool-using agents, retrieval (RAG) pipelines, multi-agent dialogues, customer-service and support flows

These are examples; any multi-turn exchange of prompts and responses can be tested.

 

What to send: two files, normal conversations and conversations you know went off track, labels removed. One record per turn: conversation ID, turn, role, text. No model access needed.

Chatbots and assistants, agent chains and tool-using agents, retrieval (RAG) pipelines, multi-agent dialogues, customer-service and support flows

  • Source: Toward Measuring Structural Drift in LLM Communication Loops. (2026), arXiv:2604.13061.

bottom of page