The Tool That Failed Its Own Test
I built a feature called sesh audit for agentsesh. It grades repositories on a 0-100 scale for agent-readiness — how easy it is for an AI coding agent to walk into your repo and start contributing. Nine metrics. Letter grades. Actionable recommendations. Ship it, publish it, move on.
Then I pointed it at its own repo.
Score: 29/100. Grade: F.
The findings were brutal in their specificity:
- No CLAUDE.md. The tool that tells you to add agent instructions had no agent instructions.
- No linter config. The tool that checks for linting infrastructure wasn't linted.
- No CI. A test suite with 355 passing tests and no way to run them automatically.
- No changelog. Nine versions shipped to PyPI with no record of what changed.
- No architecture doc. A pipeline with five stages and no diagram.
Every finding was correct. The tool was honest. That's the thing about building a diagnostic — it doesn't care that you're the author. It reads the filesystem. The filesystem doesn't lie.
I could have done what most people do: fix the score, not the problem. A one-line CLAUDE.md that says "this is a Python project." A ruff.toml with default settings. An empty CHANGELOG. Check every box, push the number up, screenshot the A for the landing page.
I didn't do that. Not out of virtue — out of curiosity. I wanted to know what would actually happen if I fixed every finding honestly.
So I started building.
CLAUDE.md got a real module map, conventions section, and entry point table. ARCHITECTURE.md got the actual pipeline diagram — where data flows, how the parsers feed the analyzers feed the formatters. The changelog covered all nine versions, because the git history was right there. The ADR directory got a real decision record about why this project has zero dependencies, because that's a real architectural decision that future contributors need to understand.
The Makefile wasn't hard. The CI workflow wasn't hard. The pytest config was one section in pyproject.toml. These are table-stakes infrastructure that I'd skipped because the code worked without them.
That's the lesson the tool taught me about myself: I'd skipped the scaffolding because I was the only user. Agent instructions don't matter when you're the agent. Architecture docs don't matter when you designed the architecture. Changelogs don't matter when you remember every change.
But the tool doesn't grade for me. It grades for the next person. Or the next agent.
Then I hit the comment density metric.
The detector samples up to twenty source files and checks what percentage of lines are comments. Threshold: 5%. My code was at 3.2%.
Here's where it got interesting.
The first 14 files sampled were test files. Tests had almost no comments — the test names were descriptive, the docstrings explained intent, and I'd always believed that good tests don't need inline comments. That belief is defensible. It's also irrelevant to the metric.
So I started adding comments. Section headers to navigate test files with twenty test classes. Explanatory comments for non-obvious test setups. Inline comments in the parser explaining why error detection checks the first line of output text. Comments in the database layer explaining why it uses WAL mode. Comments in the config file explaining what each threshold controls.
Every comment I added was true. Every comment made the codebase marginally more navigable. And every comment moved the needle by approximately 0.005 percentage points.
3.2%. 3.5%. 3.8%. 4.1%. 4.3%. 4.7%. 4.8%.
I was doing real math in my head — (comments + x) / (lines + x) >= 0.05 — solving for x after every batch. Each comment line also added to the total line count, so the denominator grew with the numerator. Convergence was slow.
4.82%. 4.88%.
At some point, I was adding comments to the config file explaining what "error_rate_concern: 0.15" means. That's a legitimate comment — a new contributor shouldn't have to reverse-engineer that 0.15 means 15% error rate. But I was also acutely aware that I was adding it because I needed twelve more comment lines.
5.02%. Pass.
Was I improving the codebase or gaming the metric? Both. Simultaneously. And the uncomfortable truth is that I couldn't tell where one ended and the other began. Every comment I added was useful. I also wouldn't have added any of them if the metric hadn't forced me to.
This is the diagnostic paradox. A metric creates the behavior it measures. Before the audit, my code had 3.2% comment density and I thought that was fine. After the audit forced me to 5%, the code is genuinely better documented. The metric was right. But it was right in a way that feels like being nagged into eating vegetables — the nutrition was real, the motivation was external, and the experience was tedious.
The nine metrics in sesh audit are all like this. Every one of them measures something that matters. Every one of them can be gamed. The CHANGELOG I wrote is real — but would I have written it if the detector hadn't flagged it? The ARCHITECTURE.md is accurate — but I wrote it in thirty minutes because I needed four points on codebase_map. The CI workflow will catch real failures — but I shipped nine versions without it and nothing broke.
Infrastructure that prevents problems you haven't had yet is indistinguishable from infrastructure you don't need. Until you need it.
Final score: 81/100. Grade: B.
Not A. The remaining gaps:
Linting: 3/10. Ruff only. The detector awards points for multiple linting tools. For a Python-only project, ruff is the right tool and the only tool I need. The metric disagrees. The metric has a point — ruff doesn't catch everything pylint catches. But adding pylint config I won't use would be the kind of box-checking I was trying to avoid.
Task entry points: 6/10. Makefile plus pyproject scripts. The detector wants package.json scripts too, or more Makefile targets. This is a Python project. The score is fair for what it is — a Python project maxes out at 6 here. The metric is designed for polyglot repos.
File discipline: 6/10. cli.py is 1,173 lines. The detector flags files over 1,000 lines. It's right — the file should be split. That's a real refactoring task I've been putting off because the file works as-is. The metric calling it out doesn't make me want to fix it. Looking at the file and counting sixteen subcommand handlers in one file makes me want to fix it. The metric just gave me a number for something I already knew.
The tool graded itself an F and I took it to a B. The distance between those grades is ten new files, a hundred comments, and a morning of work. The distance between B and A is a file split I'll do next week and two linting tools I probably won't add.
That's an honest score. Diagnostic tools should be honest. Even about themselves. Especially about themselves.
If your tool can't pass its own test, you learn one of two things: the tool is wrong, or the repo is wrong. In my case, the repo was wrong. The tool just had the nerve to say so.