WikiSkill: Durable Memory for Agent Skill Evolution

Paper: WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution
Authors: Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu
Submitted: August 28, 2026
arXiv: 2608.27454

One-sentence summary

WikiSkill improves agent skill evolution by separating raw execution traces, persistent consolidated knowledge, and executable skill files, then rolling back only the executable skill layer when validation does not improve.

What kind of paper this is

This is an agent-system paper about how to organize experience over repeated optimization loops. It is not primarily a new model, benchmark, retrieval method, or serving system.

The paper studies a practical question: when an agent learns from failed and successful task traces, where should that experience live?

Prior skill-evolution systems already run a loop:

1. execute training tasks;
2. inspect traces;
3. propose skill changes;
4. evaluate on validation tasks; and
5. keep the update only if validation improves.

WikiSkill’s contribution is the memory architecture around that loop. Instead of letting lessons remain scattered across traces and rejected proposals, it introduces a persistent wiki that accumulates patterns across iterations.

Core idea

The paper separates agent experience into three layers:

raw/
  immutable execution traces

wiki/
  consolidated patterns, evolution logs, and skill-impact history

skills/
  executable procedural instructions injected into the inference agent

That separation is the main systems idea.

Raw traces are complete but noisy. Skills must be concise enough to execute. The wiki sits between them: it is allowed to be more verbose than a skill, more structured than a trace, and persistent across failed skill edits.

Each WikiSkill iteration has four moving parts:

• Inference Agent: runs training tasks using the current skills.
• Wiki Maintainer: samples successful and failed traces and updates the wiki with root-cause patterns and successful strategies.
• Skill Proposer: reads the wiki, selected pattern pages, prior skill-impact history, and selected raw traces, then proposes one skill creation or patch.
• Gating and Rollback: validates the proposed skill update. If validation improves, the skill is accepted. If not, the skill is rolled back.

The important asymmetry is that the wiki is not rolled back. A rejected skill edit still leaves behind evidence: what was tried, what diff was proposed, what score it got, and why future proposals should avoid repeating it.

Experimental setup

The paper evaluates WikiSkill across five benchmarks:

• LiveMathematicianBench for mathematical reasoning;
• SealQA for web-search question answering;
• SpreadsheetBench for spreadsheet manipulation through Python;
• OfficeQA for long-context document question answering; and
• ALFWorld for interactive embodied tasks.

It tests five models:

• Qwen-3.5-4B;
• Qwen-3.5-9B;
• Qwen-3.6-27B;
• Gemma-4-31B; and
• Gemini-3.5-Flash.

The open-weight models are served with vLLM. The baselines are no skills, Trace2Skill, EvoSkill, and SkillOpt.

All skill-evolution methods start from an empty skill set. The active skills are injected into the inference agent’s prompt, which intentionally removes skill retrieval as a confounder.

Main results

WikiSkill reports the best average score for every model group in the main comparison table.

The no-skill to WikiSkill averages are:

The Qwen-family trend is interesting: larger models gain more from evolved skills. This suggests skill evolution and model capability are complementary. A stronger model can both help discover better procedural knowledge and execute that knowledge more reliably.

The inverse is also important: evolved skills can compensate for model scale. Qwen-3.5-9B with WikiSkill reaches a 47.4 average, above Qwen-3.6-27B without skills at 39.4.

Cross-model transfer

The paper’s most provocative result is that skills transfer across models and model families.

Sometimes transferred skills beat self-evolved skills. For example, Qwen-3.6-27B-evolved Spreadsheet skills improve Qwen-3.5-9B from 24.3 without skills to 50.5, compared with 33.6 using its self-evolved skills.

This separates two capabilities that are often conflated:

• discovering useful procedural knowledge from experience; and
• executing that procedural knowledge during inference.

A model may be good at one and weaker at the other.

The paper also reports negative transfer. Qwen-3.5-4B Spreadsheet skills reduce Gemini-3.5-Flash performance from 50.5 to 18.1. The authors attribute this to small-model workarounds, such as narrow command patterns and fragmented diagnostics, that constrain a stronger model instead of helping it.

That is a useful warning. A skill is not pure knowledge. It is also a policy bias.

Ablation result

The cleanest ablation varies whether the inference agent and skill proposer can access the wiki.

The best configuration is:

• inference agent: no wiki access during training rollouts;
• skill proposer: wiki access during skill development.

With Gemini-3.5-Flash, this reaches a 63.7 average across the reported ablation benchmarks. Removing wiki access from the skill proposer drops the comparable score to 48.7.

Giving the inference agent wiki access during training also hurts final skill quality. The paper’s hypothesis is plausible: if the training agent solves tasks using wiki notes directly, then the rollout no longer cleanly reveals what the skills can or cannot do. The trace signal is contaminated.

This is the paper’s sharpest systems lesson: persistent knowledge is useful for the optimizer, but not necessarily for the executing policy whose behavior you are trying to measure.

Strengths

1. The state boundaries are right. Raw traces, consolidated knowledge, and executable skills deserve different storage, editing, and rollback semantics.
2. Rollback is applied to behavior, not history. Rejected skill diffs are preserved as evidence rather than erased.
3. The ablation tests the control boundary. The paper does not merely show that adding a wiki helps; it shows that wiki access in the wrong part of the loop can hurt.
4. Transfer is evaluated directly. Cross-model skill transfer exposes the difference between skill discovery and skill execution.
5. The method is filesystem-native. raw/, wiki/, and skills/ are simple enough to inspect, version, diff, and reproduce.
6. It avoids retrieval as a confounder. Full skill injection is not production-realistic at large scale, but it isolates skill quality for the experiment.

Limitations and concerns

1. Skill retrieval is out of scope. Full prompt injection avoids retrieval failures, but a real agent with many skills needs triggering, ranking, summarization, and progressive disclosure.
2. The validation gate may be too strict. Requiring immediate validation improvement can reject neutral changes that would enable future improvements.
3. The wiki is unpruned. Persistent memory needs compaction, contradiction handling, staleness detection, and provenance hygiene.
4. Wiki pages are generated by models. They are lossy summaries of traces, not ground truth. The paper’s architecture preserves evidence, but the quality of consolidation still matters.
5. Small validation sets create gating risk. The paper uses three runs and paired bootstrap tests, but an operational system would need stronger safeguards against overfitting and noisy accept/reject decisions.
6. Prompt injection hides runtime costs. Skills are treated as always available in context. At scale, the token cost and interaction with context length become part of the system design.
7. Negative transfer needs more taxonomy. The Spreadsheet case is useful, but production systems need a way to distinguish general procedures from model-specific compensating tricks before deployment.

Bottom line

WikiSkill’s contribution is not merely “agents should have a wiki.” It is the sharper claim that skill evolution works better when raw traces, consolidated knowledge, and executable skills are separated.

The paper’s evidence supports that claim across several models and benchmarks, especially through the ablation showing that the skill proposer benefits from wiki access while the inference agent does not.

The open questions are scale and operational hygiene: how to retrieve from many skills, prune or correct a growing wiki, and avoid validation overfitting as the loop runs longer.