A few months ago we launched Freckle 2.0 as a headless agent tool. Today we’re announcing Freckle’s MCP server, with open beta starting at the end of the week.

Switching wasn’t the hard part. It was confidence: how do you move to a completely new architecture and trust that every feature made it across without regressing?

This series is about the platform we built for that, and where it goes next: a product that improves performance on its own, with autonomous updates and self evaluation.

Better than before, but how would we know?

We want a product that gets better on its own: agents propose changes, test them, keep what helps. First obstacle: defining what “better” meant, because when we launched we weren’t measuring anything.

We had a CLI, a 14,000-word skill, and hand judgment on every change. We knew the skill was a burden. But we didn’t know whether any change helped, mapping CLI traces back to each request was a hassle, and agent slop was making the skill longer and harder to maintain with every edit.

Here’s an example of what eye-balling misses. We changed the skill to make the agent estimate the cost before every run. It was one added line: before spending credits, tell the user the maximum cost (price per row times the number of rows) and get approval. Reasonable, and it looked fine. Then the runs got slower. The agent needed a row count it had no way to get, so it paged through the whole dataset, rewrote its own parser when it broke, and blew out its context. Thirty minutes later it reported success. A small change, a dumb consequence, and nothing in our process to catch it. Today the estimate is 50 times faster than before, and we know that because we measure it on every new version we release.

My tech lead at SambaNova (an absolute legend, by the way) reduced practical testing to one line: you can’t fix what you can’t measure. Measure first, fix next. Find better ways to measure later.

Measuring means many metrics. A VP doesn’t judge a department on one number, because one number gets gamed. We’ll cover metrics in part 2.

What we built

We built an autoresearch platform. An idea comes in from a customer call, a Slack message or a hunch, and we queue it. From there the platform runs itself: it makes several attempts at the change, runs evals on each in parallel against a full software mock of Freckle, reads the traces and metrics, and raises PRs for the tool, and/or updates its own harness, along the way. Nothing touches production while we experiment.

The whole experiment platform and mocked Freckle was vibe-coded, end to end, in 1 week. Early on, new tests broke the harness a dozen times a day, and AI fixed it each time. Nothing off the shelf moves that fast.

Call it autoresearch for a product. A vibe goes in. Many candidate changes compete, the measurements pick the winners, and what comes out is an improvement that holds across the board, not just on the case someone happened to try.

Getting started

To run your first eval on a Claude subscription:

claude setup-token                       # once; prints a token
export CLAUDE_CODE_OAUTH_TOKEN=<token>

claude -p "<a real customer ask>" \
  --model <cheap model> \
  --mcp-config mcp.json --strict-mcp-config \
  --output-format stream-json --verbose \
  --max-turns 40

claude setup-token                       # once; prints a token
export CLAUDE_CODE_OAUTH_TOKEN=<token>

claude -p "<a real customer ask>" \
  --model <cheap model> \
  --mcp-config mcp.json --strict-mcp-config \
  --output-format stream-json --verbose \
  --max-turns 40

claude setup-token                       # once; prints a token
export CLAUDE_CODE_OAUTH_TOKEN=<token>

claude -p "<a real customer ask>" \
  --model <cheap model> \
  --mcp-config mcp.json --strict-mcp-config \
  --output-format stream-json --verbose \
  --max-turns 40

On a ChatGPT subscription:

codex login                              # once
mkdir -p runs/1 && cp ~/.codex/auth.json runs/1/
CODEX_HOME=runs/1 codex exec --json --skip-git-repo-check \
  -m <cheap model> -c model_reasoning_effort=low \
  "<a real customer ask>"

codex login                              # once
mkdir -p runs/1 && cp ~/.codex/auth.json runs/1/
CODEX_HOME=runs/1 codex exec --json --skip-git-repo-check \
  -m <cheap model> -c model_reasoning_effort=low \
  "<a real customer ask>"

codex login                              # once
mkdir -p runs/1 && cp ~/.codex/auth.json runs/1/
CODEX_HOME=runs/1 codex exec --json --skip-git-repo-check \
  -m <cheap model> -c model_reasoning_effort=low \
  "<a real customer ask>"

--strict-mcp-config and a fresh CODEX_HOME keep your own tools out. run.jsonl has every call: count turns, time it, check what got saved. That’s an eval. Parallel runs hit subscription limits; API keys don’t.

Next

Part 2 is about what we run and what we look for:

• How to cut the price and token cost of a run until you can afford hundreds a day.
• How to keep quality high while reducing the number of evals run.
• How to reduce the maintenance cost of skills and documentation.
• The agent’s reasoning comes back empty now, what else can you read?
• And more.