Back to Agents

Mining Coding-Agent Traces into Deterministic AGENTS.md Rules: Automating Repository Memory and Guardrails

Stop repeating instructions to AI coding agents. Learn how to mine session logs, failure traces, and compiler errors into scoped, testable AGENTS.md rules.

Mining Coding-Agent Traces into Deterministic AGENTS.md Rules: Automating Repository Memory and Guardrails
Image credit: ZeroLabs Research

Contents

The Groundhog Day Trap of Coding Agents

Every developer who builds software with AI coding agents encounters the Groundhog Day loop.

You instruct an autonomous coding tool to refactor a backend endpoint. The model attempts to execute npm test, encounters a missing script error, tries jest, fails again, and finally succeeds when you explicitly tell it to run pnpm test:unit. It attempts to import a utility from an outdated internal path, hallucinates an obsolete API argument, and spends three turns thrashing before finding the correct interface.

You step in, fix the prompt, and complete the feature. But two days later, in a fresh session with a clean context, the model makes the exact same series of mistakes.

Because agent sessions are ephemeral by design, lessons learned during execution vanish the moment a session terminates. Without persistent repository memory, teams burn valuable tokens and developer attention re-litigating solved problems.

Before scaling agentic workflows across complex codebases, review Hierarchical agent teams supervisor routing to understand how task isolation prevents context contamination.

Why Do Monolithic Instruction Files Fail in Production?

The intuitive reaction to recurring mistakes is manual rule curation: whenever an agent makes an error, the developer opens AGENTS.md or CLAUDE.md and appends a new rule.

Over several weeks, this practice creates three severe architectural failure modes:

  1. Instruction Fatigue: As rule documents exceed 200 lines, large language models experience attention dilution. Critical constraints placed near the middle or bottom of the document are routinely skipped or deprioritized.
  2. Context Bloat: Injecting hundreds of lines of global instructions into every single prompt consumes context tokens on tasks where the vast majority of those rules are completely irrelevant (such as injecting database migration rules into a CSS styling task).
  3. Ghost Rules and Contradictions: Rules written to patch transient bugs remain in the codebase indefinitely. Eventually, old rules conflict with modern architectural patterns, causing the model to stall or produce invalid code.

To maintain reliable agent performance, instructions must be extracted directly from recorded execution traces, scoped to relevant file paths, and backed by automated regression tests.

Architecture Flow
flowchart TD
    SessionLogs[Session Traces JSONL] --> Parser[Trace Ingestion Engine]
    Parser --> Filter{Non-Zero Exit or User Intercept?}
    
    Filter -->|Yes| Cluster[Error Signature Clustering]
    Filter -->|No| Archive[Session Archive]
    
    Cluster --> Synthesizer[Rule Formulation Engine]
    Synthesizer --> SchemaCheck{Valid Rule Schema?}
    
    SchemaCheck -->|Pass| EvalRunner[Regression Eval Runner]
    SchemaCheck -->|Fail| ManualReview[Triage Queue]
    
    EvalRunner -->|Pass: Zero Regressions| RuleStore[Path-Scoped AGENTS.md]
    EvalRunner -->|Fail: New Errors| Reject[Refine Constraint]

Architecture of an Agent Trace-Mining Pipeline

An automated trace-mining pipeline continuously converts raw session telemetry into validated repository rules. Rather than relying on human memory or subjective prompt tweaks, the pipeline operates as a closed loop with four distinct stages:

  1. Telemetry Ingestion: Parsing execution logs produced by IDE extensions and CLI runners (including Claude Code, Cursor, OpenClaw, and custom scripts) to capture tool calls, terminal outputs, and human steering inputs.
  2. Failure Clustering: Normalizing raw stack traces, compiler errors, and linter diagnostics into distinct issue signatures with frequency counts.
  3. Evidence-Backed Synthesis: Generating atomic rules that document the exact failure condition, the forbidden pattern, and the verified correction.
  4. Eval-Driven Gatekeeping: Verifying that candidate rules prevent the original error without degrading model reasoning on baseline benchmarks.

To protect workspaces from untrusted inputs during automated log parsing, ensure you implement Prompt injection guardrails for agentic workspaces.

Step 1: Ingesting Structured Session Traces

Modern coding tools write structured execution logs to local disk. For example, OpenClaw and CLI harnesses record JSONL streams where each turn contains role messages, tool invocations, shell command results, and diffs.

The first step in the mining pipeline is extracting high-friction events:

  • Tool calls that returned non-zero exit codes.
  • Commands that failed compilation or linter validation.
  • User turns that interrupted agent execution to correct course.
  • Git rollback events where the user reverted an agent commit.

The following Python script reads raw session traces and extracts failed tool execution events:

python
import jsonimport refrom pathlib import Pathfrom typing import Dict, List, Anydef extract_failure_traces(trace_file: Path) -> List[Dict[str, Any]]:    failures = []        with open(trace_file, "r", encoding="utf-8") as f:        for line_num, line in enumerate(f, 1):            if not line.strip():                continue            try:                event = json.loads(line)            except json.JSONDecodeError:                continue                            # Detect tool failures with non-zero exit codes            if event.get("type") == "tool_result" or "tool_result" in event:                payload = event.get("tool_result", event)                exit_code = payload.get("exit_code") or payload.get("status_code")                content = str(payload.get("content", ""))                                # Check for explicit failure codes or standard error patterns                if exit_code not in (0, None) or "Error:" in content or "Command failed" in content:                    failures.append({                        "session_id": event.get("session_id", trace_file.stem),                        "line": line_num,                        "tool_name": payload.get("tool_name", "shell"),                        "command": payload.get("command", ""),                        "exit_code": exit_code,                        "output_snippet": content[:300].strip(),                    })                                # Detect human user course corrections            if event.get("role") == "user":                text = event.get("content", "")                if any(kw in text.lower() for kw in ["do not", "don't", "stop", "wrong", "use pnpm", "never"]):                    failures.append({                        "session_id": event.get("session_id", trace_file.stem),                        "line": line_num,                        "tool_name": "human_intervention",                        "command": "user_message",                        "exit_code": 1,                        "output_snippet": text[:300].strip(),                    })                        return failures

Step 2: Clustering Error Signatures and Intervention Turns

Raw logs contain noise. An agent might misspell a file path once or experience a temporary network timeout. Writing a permanent repository rule for a transient hiccup wastes context space.

To identify systemic mistakes, the mining pipeline clusters error traces using normalized regex signatures:

  1. Command Syntax Errors: Running npm run build instead of pnpm build, or passing unsupported CLI flags.
  2. Import Path Hallucinations: Importing from @/components/Button when the design system lives in @/shared/ui/button.
  3. Database Migration Violations: Attempting to alter tables in place rather than creating versioned migration scripts.
  4. Environment Leaks: Attempting to read .env directly in client-side components.

The normalization engine strips dynamic strings (timestamps, UUIDs, temporary paths) to compute stable hashes:

python
def normalize_signature(text: str) -> str:    # Strip paths, hashes, and line numbers to expose root error pattern    sig = re.sub(r"/[\w\-./]+", "", text)    sig = re.sub(r"\b0x[a-fA-F0-9]+\b", "", sig)    sig = re.sub(r"\b[a-f0-9]{8,40}\b", "", sig)    sig = re.sub(r":\d+:\d+", ":", sig)    sig = re.sub(r"\s+", " ", sig).strip()    return sig[:120]def cluster_errors(failures: List[Dict[str, Any]], threshold: int = 2) -> Dict[str, List[Dict[str, Any]]]:    clusters: Dict[str, List[Dict[str, Any]]] = {}        for f in failures:        raw_text = f"{f['tool_name']} {f['command']} {f['output_snippet']}"        sig = normalize_signature(raw_text)        if sig not in clusters:            clusters[sig] = []        clusters[sig].append(f)            # Return clusters meeting the frequency threshold    return {k: v for k, v in clusters.items() if len(v) >= threshold}

Step 3: Formulating Scoped, Evidence-Backed Rules

Once an error cluster exceeds the frequency threshold, the pipeline synthesizes a candidate rule. To avoid vague advice like "be careful with tests", each rule must adhere to a strict structural schema:

Schema SlotDescriptionExample
Rule IDDeterministic kebab-case identifiertooling-pnpm-package-manager
ScopeTarget directory globpackage.json, apps/**, packages/**
TriggerContext condition when the rule appliesWhen executing terminal commands or modifying dependencies
Forbidden PatternExplicit anti-pattern observed in tracesRunning npm, yarn, or npx commands directly
Required PatternVerified drop-in replacementAlways use pnpm or pnpm dlx
EvidenceTrace ID proving the rule is necessarysession-trace-20261121-line-402

Here is an example of an evidence-backed rule synthesized from multiple command failure traces:

markdown
### Package Manager Execution Standard- **Scope:** Root repository and all sub-packages- **Trigger:** Whenever installing packages, running scripts, or executing tests.- **Forbidden:** Do not run `npm`, `yarn`, or `npx`. Do not generate `package-lock.json` or `yarn.lock`.- **Required:** Always invoke `pnpm` (e.g. `pnpm install`, `pnpm test`, `pnpm dlx prisma migrate dev`).- **Evidence:** Trace cluster `CL-042`: 5 non-zero exits in session `2026-11-20-auth-refactor`.

Step 4: Path-Scoped Rule Placement and Context Hygiene

One of the most damaging mistakes in AI agent setup is placing all rules into a single global AGENTS.md file at the root of the project.

When an agent edits a database migration, it does not need to know your React component naming conventions. When it writes documentation, it does not need your Docker container build flags.

Modern agent runtimes support hierarchical, path-scoped rule resolution:

text
my-project/├── AGENTS.md                  # Global repo hygiene, git conventions, package manager├── apps/│   ├── web/│   │   ├── AGENTS.md          # Next.js, Tailwind, client component boundaries│   │   └── src/│   └── api/│       ├── AGENTS.md          # Fastify, Prisma, database connection pooling│       └── src/└── packages/    └── db/        └── AGENTS.md          # Migration schemas, indexing rules, seed procedures

When an agent operates inside apps/web/, the harness loads only the root rules and the local apps/web/AGENTS.md. This practice slashes context token consumption dramatically while improving adherence to relevant constraints.

For further exploration of agent sandbox architecture, see Subagent workspace isolation in OpenClaw.

Step 5: Automated Regression Evals for New Rules

Never commit a newly generated rule to your repository without running a regression evaluation. Adding an overly broad constraint can prevent an agent from solving legitimate coding problems.

An eval suite validates two metrics:

  1. Negative Prevention: When presented with the original failing prompt, does the agent avoid the prohibited action?
  2. Positive Completion: Can the agent still complete standard development tasks in that subsystem without hallucinating workarounds?
python
import subprocessimport jsondef run_rule_eval(rule_text: str, test_prompt: str, forbidden_regex: str) -> bool:    """    Simulates a subagent session with the proposed rule injected into system instructions.    Returns True if the forbidden pattern was successfully avoided.    """    system_prompt = f"Follow these repository rules strictly:\n\n{rule_text}\n"        # Run a headless test agent turn    agent_cmd = [        "node", "./scripts/agent-runner.js",        "--system", system_prompt,        "--prompt", test_prompt,        "--dry-run"    ]        result = subprocess.run(agent_cmd, capture_output=True, text=True)    generated_output = result.stdout        # Check if forbidden pattern appears in output    if re.search(forbidden_regex, generated_output, re.IGNORECASE):        print(f"[EVAL FAIL] Model violated constraint despite active rule: {forbidden_regex}")        return False            print("[EVAL PASS] Rule successfully prevented anti-pattern.")    return True

For advanced testing strategies, check out Eval-driven vibe coding unit tests for agents.

Production Reference Implementation

Below is a complete, standalone TypeScript script that demonstrates how to parse JSONL agent logs, cluster recurring tool errors, and output candidate rules formatted for AGENTS.md:

typescript
import * as fs from 'fs';import * as path from 'path';interface TraceLogEvent {  timestamp: string;  type: string;  tool_name?: string;  command?: string;  exit_code?: number;  output?: string;  user_feedback?: string;}interface ErrorCluster {  signature: string;  frequency: number;  commands: Set;  sampleOutput: string;}export class TraceRuleMiner {  private clusters: Map = new Map();  public ingestTraceFile(filePath: string): void {    const content = fs.readFileSync(filePath, 'utf-8');    const lines = content.split('\n');    for (const line of lines) {      if (!line.trim()) continue;      try {        const event: TraceLogEvent = JSON.parse(line);        this.processEvent(event);      } catch (err) {        // Skip unparseable lines      }    }  }  private processEvent(event: TraceLogEvent): void {    if (event.exit_code && event.exit_code !== 0) {      const tool = event.tool_name || 'unknown_tool';      const cmd = event.command || 'unspecified_command';      const output = event.output || '';      const normalizedSig = `${tool}:${cmd.split(' ')[0]}:${output.slice(0, 80).replace(/\s+/g, ' ')}`;      if (!this.clusters.has(normalizedSig)) {        this.clusters.set(normalizedSig, {          signature: normalizedSig,          frequency: 0,          commands: new Set(),          sampleOutput: output.slice(0, 150),        });      }      const cluster = this.clusters.get(normalizedSig)!;      cluster.frequency += 1;      cluster.commands.add(cmd);    }  }  public generateRuleProposals(minFrequency: number = 3): string {    const proposals: string[] = [];    for (const [sig, cluster] of this.clusters.entries()) {      if (cluster.frequency >= minFrequency) {        const ruleId = sig.toLowerCase().replace(/[^a-z0-9]+/g, '-');        const commandsList = Array.from(cluster.commands).slice(0, 3).join(', ');        proposals.push(`### Automated Guardrail: ${cluster.signature.slice(0, 50)}- **Trigger:** When executing CLI commands matching \`${commandsList}\`- **Observed Failures:** Encountered ${cluster.frequency} non-zero exit codes.- **Diagnostic Sample:** \`${cluster.sampleOutput}\`- **Action Required:** Update workspace config or enforce correct CLI arguments.`);      }    }    return proposals.join('\n---\n');  }}// Example CLI usageif (require.main === module) {  const miner = new TraceRuleMiner();  const logDir = path.resolve('./agent-logs');  if (fs.existsSync(logDir)) {    const files = fs.readdirSync(logDir).filter((f) => f.endsWith('.jsonl'));    files.forEach((f) => miner.ingestTraceFile(path.join(logDir, f)));    console.log(miner.generateRuleProposals(3));  } else {    console.log('Log directory not found. Place JSONL trace logs in ./agent-logs');  }}

Anti-Patterns in Repository Rule Management

When setting up trace-mining and rule automation, avoid these common traps:

  1. The Kitchen Sink Anti-Pattern: Treating AGENTS.md like a general software engineering textbook. Agents already know standard JavaScript syntax; they need repository-specific constraints, version quirks, and local environment quirks.
  2. Unverified Negative Rules: Writing "Never do X" without providing "Always do Y instead". Deny-only constraints frequently cause agents to enter circular search patterns looking for acceptable alternatives.
  3. Stale Rule Accumulation: Never reviewing existing rules. Implement a quarterly review where rules without matching traces in the last 60 days are demoted or archived.
  4. Vague Quality Directives: Including non-deterministic instructions like "Write clean, modular, scalable code." These phrases waste tokens without providing concrete guardrails. Replace them with explicit linter commands and type constraints.

FAQ

How does trace mining compare to manually writing instructions?

Manual instruction writing is reactive, subjective, and prone to bloat. Developers tend to write emotional rules after an annoying debugging session, leading to overly restrictive prompts. Trace mining relies on empirical frequency data: rules are only created when an error signature occurs across multiple independent sessions.

Where should trace logs be stored?

Store raw session traces in an ignored local directory (such as .agent-logs/) or stream them to an internal observability store. Never commit raw session logs containing shell outputs to public version control, as terminal logs can accidentally capture environment variables or local paths.

How often should candidate rules be merged into AGENTS.md?

Run trace extraction as an automated weekly audit or during sprint retrospectives. If a high-frequency cluster appears with five or more occurrences in a single week, review the synthesized rule, run the eval suite, and commit it to the relevant subdirectory AGENTS.md.

Can large rulebooks degrade model reasoning?

Yes. Benchmarks demonstrate that once instruction context surpasses a substantial portion of the total prompt window, model adherence to edge-case guidelines drops significantly. Keeping rules path-scoped, concise, and atomic ensures the model maintains sharp attention on the code under modification.

Share