Hub · everything on one subject · 11 articles · 6 repos & apps
Loop Engineering and Self-Improving Systems
A loop is how something gets better without anyone having to remember to improve it: try, measure honestly, keep what helped, undo what hurt, and go again. The trick is a referee who is not the player, so the score cannot flatter itself.
In engineering terms: Observe, Evaluate, Control (OEC) at every scale; an external judge instead of self-report; regression rollback and a quality floor that only rises (a ratchet); cost and iteration budgets; and gates that say UNMEASURED instead of faking a pass.
It is the spine of everything else I build. The same loop runs a codebase, a team review and a research program, and the repos below are that loop as open source.
Loop engineering (7)
Why a loop needs an outside referee, and how to build one that catches you.
- Loop Engineering: Build the Loop That Catches You
2026-08
Prompt engineering asks what to say; loop engineering asks how a system finishes the job — and how it convicts you when you are the one who got it wrong. The treehouse model, twelve laws, and four times my own gates caught their author.
- Loop Engineering Is an External Referee, Not a Retry
2026-09-11
Episode 4 of a series comparing the ways to engineer an AI agent — prompt, context, harness, loop and graph engineering. This one is loop engineering: its history from self-consistency (2022) to the self-improving agents of 2026, who drives it, and the rule that decides whether a loop converges or drifts — a loop improves only when the scorer sits outside the generator, and every accepted step must be a measured non-regression. It reconciles the Reflexion and Self-Refine gains with the finding that models cannot self-correct reasoning without external feedback: the gains appear where the feedback is a test, a compiler or an oracle. Ships a free Colab lab (no API key, stdlib only) whose 20-seed run puts a self-judged loop at 127.4 true while it reports 158.0, against 149.2/149.2 for the oracle-gated one, plus two apps built from nothing: a study-plan refiner where 0 of 413 accepted steps lowered the external score, and a test-driven repair loop that goes 30 to 42.6 of 60 green with rollback and 30 to 9.6 without, with a CI gate that exits 1 when rollback is removed.
- OEC Loops at Every Scale: How a Codebase (and a Team) Keeps Improving Without Being Reminded
2026-10-26
Observe, evaluate, control, at every scale at once: a unit test, a quality floor that only rises, an outside review, a weekly loop and dated tripwires. Built on a codebase that graded itself from 65 to 97, then needed a live run and a reviewer to catch eight bugs its tests missed (fixed: 98), with a dimension ladder from a single number to a fractal.
- Stop Generating Tools. Start Engineering Loops.
2026-06-16 · LinkedIn
The bug hiding in every AI-built tool Ask an AI to build you a CLI for some API and it will. You skim the output, it looks plausible, you ship it.
- Claude Code, Codex, and Hermes Agent all shipped a /goal command this year...but I am still failing my goal
2026-05-27 · LinkedIn
Claude Code, Codex, and Hermes Agent all shipped a command this year. The idea: give the agent a precise objective, and it executes until the evaluator confirms it
- I Built a System That Catches My AI Agent Slacking Off. Then It Started Teaching Me About Myself.
2026-02-24 · LinkedIn
A story about idle detection, self-improvement protocols, and why the code you write for machines accidentally becomes the best therapy you never paid for. Last Tuesday at 2 AM, my AI agent had been "running" for six hours straight.
- I Built a System That Optimizes AI Agents. Then Realized the Marketing Had Zero Feedback Loops.
2026-02-24 · LinkedIn
How the architecture that catches lazy AI agents turned into a marketing analytics engine — with zero dependencies and uncomfortable self-awareness. I
Referees, gates and governors (4)
The measuring half: a rubric that could not fail, a governor, and the one assumption that caps a guarantee.
- I Scored 100% on My Own Checklist. That Was the Bug.
2026-08-21
A webinar summary listed sixteen things a mature AI agent loop must have and verified none of them against real software, so I encoded it as a machine-checkable rubric and pointed it at my own engine. It scored 23 out of 23. That perfect score was the bug: I had written the questions after an hour inside the codebase, and never once wrote one I expected to come back red. What I had built was a well-tested mirror. So I reread the lecture only for the claims I had skipped, chose them by a hostile criterion — include it if I expect it to fail — and the honest score became 25 of 28 with the gate red. One miss was worth fixing on the spot: a cross-run success rate, where two lines decide whether the number is honest (a run still in flight is excluded from the denominator, and a run halted by the safety gate counts as a failure, because a rate that forgave safety blocks would reward being stopped for being dangerous). Three stayed red with a reason recorded beside them and deliberately no way to turn a reason into a pass. Three more are declared gaps whose probes fail the moment the gap silently closes. The mechanism is selection at authoring time, and the principle is that a checklist which cannot fail its author is not an assessment, it is a mirror.
- The Governor
2026-08-24
Watt shipped feedback in 1788. Maxwell asked the harder question in 1868 — under what conditions does the correction settle instead of oscillating? Routh made it decidable in 1877. Agentic loops ask "did it converge?", which is a weaker question, and cannot name gain, dead time or stability at all. A grade sequence like C, A, C, A is not a broken evaluator; a control engineer reads it on sight as too much gain for the dead time.
- The Assumption Ledger: a Safety Guarantee Is Capped by the One Thing Nobody Checked
2026-09-19
An assumption ledger is the list of things a safety claim silently depends on, each marked proven, watched at runtime, or merely hoped for — and the claim's guarantee is then capped at what its weakest assumption supports. One unchecked item sets the ceiling however many sound ones surround it, which is why "we used a model predictive controller, therefore it is safe" fails: the stability theorem is about the model, and nobody checked the model against the robot. The five-rung ladder from anecdote to certified mapped onto the cooking ladder, the capping rule and why assumptions combine by worst while evidence combines by best, what happened when the rule ran against 32 curated safety claims (30 capped or refused, and why that number is not a survey), the two that survived, three catches worth your time including Doyle's 1978 two-page demolition of optimal-means-safe, the blank comparator row in every agent loop, and a 30-minute build with its CI threshold.
- The AI Review Loop Report: Eleven Silent Days, One Chat Card, and a 24% Ceiling
2026-09-24
A field report on an AI review loop. The reviewer in a factory quality codebase was out of budget for eleven days; it posted "unmeasured" on every pull request, exactly as designed, and nobody acted on it (0 of 51 labelled pull requests auto-merged, 47 never reviewed). The fix routes each verdict to the team chat as one card, with a separate grey alarm when the reviewer itself is broken, and keeps approval on GitHub. Six design decisions and why, what the reviewer caught in ten rounds on its own repairs, a measured 24% ceiling on auto-merge that replaced a 90% goal, and the next moves tied to the Visionboard's bets, each with a dated falsifier.
Repos and live apps (6)
- loop-engineering-anything
A loop orchestrator that turns any target into a self-improving, agent-native CLI by driving generate, judge, refactor, re-judge to convergence.
- eval-anything
The evaluation operating system: Observe, Evaluate, Control for foundation models, agents and business objectives.
- flywheel
A closed-loop self-improvement substrate: flywheel(X) = X, self-improved.
- rsi-os
A community resource on recursive self-improvement and automated AI research: papers, tools, people, labs and roadmaps.
Open the live app → - longevity-loop
The loop applied to aging science, in public: falsifiable question, open data, public verifier, honest write-up, share, compound. No wet lab required.
- Destination Playground (live demo)
Play the loop: write a destination, watch a gate rule options out, choose, and see a quality floor that only rises.