Post
47
A coding agent in 5 files, plus a skeptic that catches fake fixes.
Coding agents are everywhere now, but almost nobody checks whether they actually fixed the bug — or just gamed the test. Tell one "make the failing test pass" and sometimes it fixes it; sometimes it edits the test, hardcodes the value, or stubs the function, then reports done . As people wire agents into CI and auto-merge, "tests are green" and "code is correct" quietly stop being the same claim.
So I built two things: a small coding agent from scratch (5 readable files — the loop, real tools, context compaction, a permission gate), and — the point — an independent **skeptic** that catches the fake fix.
The core idea will feel familiar if you do ML: you can't evaluate on the data you trained on. The skeptic runs a *hidden contract oracle* — it checks the code's behavior on inputs the agent never saw. A fix that games the visible test fails the held-out one, deterministically, no LLM needed. (A model judge reads the full diff on top as a second opinion, not the guarantee.)
Works with anything OpenAI-compatible — a HF Inference Endpoint / TGI (point OPENAI_BASE_URL at it), OpenRouter, or local Ollama/vLLM. Honest about limits: on your own repo you supply the held-out check, and the judge layer is model-sensitive (small models miss cheats AND false-reject real fixes — the README says so plainly).
Tests + a reproducible cheat-catch eval run in CI with no key. Feedback welcome — especially new cheats it should catch.
https://github.com/Saivineeth147/skeptic
Coding agents are everywhere now, but almost nobody checks whether they actually fixed the bug — or just gamed the test. Tell one "make the failing test pass" and sometimes it fixes it; sometimes it edits the test, hardcodes the value, or stubs the function, then reports done . As people wire agents into CI and auto-merge, "tests are green" and "code is correct" quietly stop being the same claim.
So I built two things: a small coding agent from scratch (5 readable files — the loop, real tools, context compaction, a permission gate), and — the point — an independent **skeptic** that catches the fake fix.
The core idea will feel familiar if you do ML: you can't evaluate on the data you trained on. The skeptic runs a *hidden contract oracle* — it checks the code's behavior on inputs the agent never saw. A fix that games the visible test fails the held-out one, deterministically, no LLM needed. (A model judge reads the full diff on top as a second opinion, not the guarantee.)
Works with anything OpenAI-compatible — a HF Inference Endpoint / TGI (point OPENAI_BASE_URL at it), OpenRouter, or local Ollama/vLLM. Honest about limits: on your own repo you supply the held-out check, and the judge layer is model-sensitive (small models miss cheats AND false-reject real fixes — the README says so plainly).
Tests + a reproducible cheat-catch eval run in CI with no key. Feedback welcome — especially new cheats it should catch.
https://github.com/Saivineeth147/skeptic