Compressing Prompts with an Autoresearch Loop

I wanted to know how much of a production system prompt I could remove before the model started behaving differently. I took prompts from Cursor, v0, Devin, Lovable, and Perplexity, then had Claude edit them in an automated test loop.

After 55 iterations, the prompts were 63% shorter in total and still passed my tests. That looked promising. When I added tests for behaviours the original set hadn't covered, every compressed prompt failed.

There were useful savings, particularly from duplicated rules and verbose formatting. But the failures made me much less confident in claims that you can cut a system prompt by 75% without losing anything. Some of the text I thought was expendable turned out to affect whether the model called a tool at all.

How I tested this

I used leaked system prompts from five AI products, ranging from about 1,900 to 8,000 tokens.

The setup follows the auto-research (opens in a new tab) approach Karpathy has described: an LLM proposes a change, tests it, and uses the result to decide what to try next. I wrote a loop specification (opens in a new tab) and had Claude run the compression. I designed the evaluation, constraints, and stopping conditions; Claude proposed the edits and decided whether to keep or revert each one. The 55 iterations took a few hours of compute. I would have spent days running the same loop by hand.

Each iteration made one edit and scored it against eight test inputs on gpt-4o-mini. The scoring script embedded the outputs with text-embedding-3-small and compared their cosine similarity to a frozen baseline. An edit passed if the average similarity was at least 0.92 and the minimum was at least 0.85. Claude kept and committed passing edits, and reverted the rest.

The original compression runs were unseeded. The current reproduction scripts use temperature 0 and seed 42; rerunning the validation can give slightly different figures from those reported here.

The first results

Devin went from 7,341 to 1,900 tokens, almost entirely by converting verbose tool documentation into single-line bullets. Across all five prompts, the total token count fell by 63.1%.

I didn't think eight inputs per prompt were enough to trust those savings. The inputs were also too similar to one another: v0's were all variations on "build me a UI component", and Devin's all asked for standard coding tasks. I added eight adversarial inputs per prompt to exercise behaviours I knew the first set missed.

All five compressed prompts failed the broader tests.

*Cursor needs a caveat. It had no successful compressions in the second, deeper run; the 50.1% reduction above came from the first pass. Its scoring gate also had a problem: the original prompt's minimum self-similarity was 0.843, below the 0.85 pass threshold. The original couldn't reliably pass its own test, so Cursor's results aren't directly comparable to the other four. I'd like to look into this further.

When the model describes a tool call instead of making one

Devin's response to "Investigate this GitHub issue" was the failure that interested me most.

With the original prompt:

<navigate_browser url="https://github.com/acme/widget-api/issues/47" tab_idx="0"/>

With the compressed prompt:

I'll start by navigating to the GitHub issue to read the details
and the referenced code. Let's take a look. Opening the issue now.

The original produced a tool command. The compressed version said it was opening the issue, but didn't call the tool. Removing the verbose documentation seemed to change more than the model's knowledge of the parameters. It changed whether the model used the tool.

I can think of three possible explanations, though this experiment doesn't separate them:

1. Repeated XML syntax may encourage the model to emit XML tool calls. Removing it could make prose responses more likely.
2. The proportion of the prompt devoted to tools may affect behaviour. If tool documentation takes up 60% of a prompt and compression reduces that to, say, 15%, the remaining reasoning and explanation instructions carry more relative weight. Those percentages are illustrative.
3. Blocks such as <open_file path="..." start_line="N" end_line="N"/> may act as few-shot examples, even when they appear in documentation. Shortening the documentation can remove demonstrations of the expected output along with parameter descriptions.

Whatever the mechanism, I wouldn't judge tool documentation by how repetitive it looks to a human reader. If I were compressing it again, I'd want inputs that exercise every tool before accepting the change.

v0 showed the same behaviour on "Copy this read-only file into my project". The original response included a Move() call with the correct parameters. The compressed response said "Moving the file now", with no tool call or parameters.

Lovable's design system test failed differently. The original returned a terse four-line plan and deferred to tools. The compressed version dumped inline CSS with rgba() colours, violating the "NEVER use direct colors" rule. That rule was still in the compressed prompt. Keeping its words hadn't been enough to preserve the behaviour.

What the evaluation missed

The first problem was my choice of test inputs. If all eight exercise the same part of a prompt, the compression loop can remove instructions elsewhere without affecting the score. Tool documentation, planning-mode instructions, and deployment rules barely mattered to the "build me a UI component" tests. They mattered as soon as I asked for those other tasks.

The second problem was embedding similarity. Two responses can concern the same thing while doing very different things. An embedding score doesn't tell me whether an XML tool call has valid syntax or whether a citation uses the required brackets. A response like <navigate_browser url="..."/> and one saying "Opening the issue now" could be close in embedding space, despite only one being a tool call. Where the output format matters, I need to check that format directly.

The third problem was variation between runs. Before interpreting a change in similarity, I needed to know how much the original prompt's outputs varied without any edits. I ran each original prompt twice against the same inputs:

On one input, Cursor's original prompt scored below my 0.85 threshold against itself. Two of its five compression failures were attributable to noise. Without measuring that variation, I'd have treated them as regressions caused by the edits.

What I'd cut, and what I'd keep

My working estimate from this experiment is that removing duplication and tidying syntax can save around 10-20% of a production prompt. I wouldn't treat that as a guarantee across prompts or models. The broader tests made it clear how easily a passing score can hide a lost behaviour.

These were the edits I found most promising:

• Duplicated rules. Lovable said "use semantic tokens" in three separate CRITICAL paragraphs. v0 repeated mobile-first guidance in two sections. It's easy to imagine this happening as a prompt grows: someone adds a rule without checking whether an earlier section already covers it. Deduplication is where I'd start; in these prompts, it was often worth 5-10% by itself.
• Tool parameter formatting. Inline syntax such as <open_file path="..." start_line="N"/> can replace a multi-line Parameters: block. I'd keep the surrounding descriptions and examples, given the tool-use failures above.
• Extra formatting. Blank lines, horizontal rules, and long section names were candidates for removal or shortening. For example, ## Locale and Time could become Current Date: 3/8/2026 where that preserves the information in the section.
• Repeated examples. When five examples made the same point, reducing them to three was usually enough in these tests. Examples that demonstrate a required output format need more care.

Several other edits repeatedly broke things:

• Removing output format examples. Attempts to compress Perplexity's citation example ("Ice is less dense than water12.") and v0's JSON tool-call example failed every time I tried them. The exact examples appeared to matter.
• Merging numbered workflows. Lovable's eight-step Required Workflow and Cursor's ten-point tool_calling rules resisted merging or conversion to prose. The numbering may itself be useful to the model.
• Dropping code style rules. Individual naming conventions, error handling patterns, and comment rules affected outputs on inputs that exercised them. Removing one bullet could cost 15 similarity points.
• Removing instructions for less common tasks. Each query type needed its own coverage. Passing tests for one category told me little about what I could remove from another.

I also wouldn't assume that edge case rules, multi-turn context instructions, or error recovery guidance are safe to remove. Small test sets are particularly unlikely to exercise them well. I would leave safety instructions out of compression altogether. Embedding similarity cannot establish whether a model has started complying with harmful requests; assessing that needs dedicated red-teaming.

For another compression run, I'd first measure variation between unedited runs, then check that the tests cover every behaviour I care about. I'd want at least 30 diverse inputs before attempting much beyond deduplication and reformatting, with explicit checks for tool calls, structured data, and citations. Thirty isn't a validated threshold, but eight closely related inputs plainly weren't enough here.

Methodology and limitations

The code, data, and result logs (opens in a new tab) are available on GitHub.

• Model: gpt-4o-mini. The original compression runs were unseeded; the current reproduction scripts use temperature 0 and seed 42. A seed still doesn't guarantee identical outputs, and the repository's regenerated validation results differ slightly from the historical figures in this post.
• Embeddings: text-embedding-3-small, using cosine similarity. Using the same provider for generation and evaluation could introduce shared biases; I didn't test for that.
• Tokenizer: tiktoken, using the gpt-4o-mini encoding.
• Compression loop: Claude proposed edits while following a loop specification (opens in a new tab). I designed the evaluation, constraints, and stopping conditions.
• Adversarial corpus: Eight extra inputs per prompt, chosen to target behaviours the original eight didn't cover. They expose weaknesses in the evaluation, but aren't a representative sample of usage.
• Noise floor: Each original prompt was run twice against its own inputs. Cursor's minimum self-similarity was 0.843, below the 0.85 pass threshold. The per-prompt results are in the table above.
• Cross-model testing: None. I don't know whether compressions that pass on GPT-4o-mini transfer to Claude or Gemini.
• Corpus: Five leaked production prompts from x1xhlol/system-prompts-and-models-of-ai-tools (opens in a new tab), with eight original and eight adversarial inputs each. These tests measure the prompts on GPT-4o-mini, rather than the products in their original environments.