TLDR
We reproduced the paper’s knowledge integration experiments and extended them to additional models. Our findings are:
- The results are reproducible.
- SEAL consists of two steps: (A) Self-editing — the model generates edits based on new external information and is fine-tuned on them. (B) Reinforcement learning (RL) — the model is trained to become a better self-editor by optimizing its edit quality through RL.
- For instruction-tuned models, almost all improvements came from Step A. Step B added little benefit despite being far more expensive.
- Even with a 3× speedup from our Ray-based reimplementation, Step B remains slow because each reward computation requires fine-tuning.
- Using a strong external editor (e.g., GPT-5) achieves comparable results at a fraction of the cost, making it a practical alternative to RL-based training.
In short, self-editing alone provides most of the benefit, and strong external editors can outperform RL training in both efficiency and cost.
SEAL in Two Steps
The Goal
The goal of SEAL (Self-Adapting Language Models) is to enable models to internalize new knowledge into their parameters.
Step A — Self-editing
To incorporate new knowledge, the model generates self-edits—training examples that can be used for fine-tuning.
For example, suppose the news reads:
Apple launches iPhone 17.
If we ask an outdated model “What is the latest iPhone?”, it might still answer “iPhone 16.”
With self-editing, the model itself generates a new training pair:
Q: What is the latest iPhone?
A: iPhone 17
Fine-tuning on such edits updates the model so it answers correctly in the future.
Step B — Training the model to be a better self-editor through RL
Step A assumes the model can already generate useful edits, but SEAL makes an important assumption: models are initially not strong enough at generating good self-edits on their own. We will revisit this assumption later.
To address this, SEAL introduces Step B: training the model to improve its ability to generate effective edits through RL.
For intuition, suppose we prompt the model to sample self-edits from the same news article. We might sample two different versions using two different random seeds:
- Version 1:
Q: What is the news about? A: Apple. - Version 2:
Q: What is the latest iPhone? A: iPhone 17.
If we fine-tune on Version 1, the model still cannot answer “What is the latest iPhone?” correctly.
If we fine-tune on Version 2, it now answers correctly.
SEAL uses the outcome after fine-tuning as the reward. Version 1 would receive reward = 0 because the fine-tuned model couldn’t answer correctly and Version 2 reward = 1. The training loop then nudges the model toward producing more Version 2–style edits, effectively making it a better self-editor.
How does this RL work in practice?
Although described as reinforcement learning, the procedure functions more like supervised training on successful self-edits. The loop is roughly:
- Generate multiple candidate self-edits.
- Fine-tune the model separately on each and evaluate performance.
- Keep the edits that improve performance; discard the rest.
- Fine-tune the model on the retained edits.
In practice, this process looks like: sample → fine-tune (incorporate new knowledge) → evaluate → filter → meta-fine-tune (learn to teach itself better).
Key distinction:
- Step A: Directly injects new knowledge into the model via self-edits.
- Step B: Optimizes the model’s ability to generate higher-quality self-edits.
Our Faster Reimplementation in Ray
While the original paper provided code, we found training to be very slow: even with only 50 examples, one RL round took 3 hours on two L40 GPUs. Running 10 RL rounds required roughly 30 hours per experiment.
For each text input, the pipeline generates multiple rollouts, and each reward computation involves several rounds of fine-tuning (three per rollout in the original implementation).
To accelerate experiments, we reimplemented the training framework in Ray, enabling parallel SFT runs and flexible worker scheduling. The figure below illustrates our pipeline:
By fully utilizing GPUs, we achieved a 3× speedup, reducing total training time from 30 hours to about 10 hours per run.
Are the Results in the Original Paper Reproducible?
Yes, at least for the knowledge incorporation setting.
| Model | Adapting using Vanilla Model | Adapting using Trained Model | Improvement |
|---|---|---|---|
| Qwen 2.5 3B-Base (Original Paper) | 31.93% | 36.98% | 5.05% |
| Qwen 2.5 3B-Base (Our Reproduction) | 31.31% | 38.91% | 7.60% |
Note: The original paper did not perform checkpoint selection. We selected the best checkpoint across RL rounds. Due to the lack of a predefined validation set, we used the test set for checkpoint selection—a limitation that could be improved in future work.
Is Training the Model to Be a Better Self-Editor Really Necessary?
Our key finding is that as models become stronger (e.g., instruction-tuned rather than base models), the benefit from RL-based self-editor training becomes smaller.
In the plot above, the red bars (without RL) and yellow bars (with RL) show minimal differences for instruction-tuned models.
The following table shows the same results numerically:
| Model | Adapting using Vanilla Model | Adapting using Trained Model | Is-Instruction-Tuned | Improvement |
|---|---|---|---|---|
| Qwen 2.5 3B-Base (Original Paper) | 31.93% | 36.98% | No | 5.05% |
| Qwen 2.5 3B-Instruct (Ours) | 54.11% | 56.47% | Yes | 2.36% |
| Qwen 3 0.6B-Instruct (Ours) | 40.55% | 43.53% | Yes | 2.98% |
| Qwen 3 4B-Instruct (Ours) | 64.78% | 67.15% | Yes | 2.37% |
| Llama 3.2-1B-Instruct (Ours) | 37.98% | 38.91% | Yes | 0.93% |
Our Recommendation
From the above results, most improvements come from self-editing the model rather than from the expensive RL training. Our first recommendation is to prompt the model to generate self-edits directly, instead of training it to become a better self-editor.
We also ran an additional experiment showing that using an external editor can achieve comparable or even better results at a fraction of the cost. For example, the figure above shows the results of using GPT-5 to generate edits with low reasoning effort, which still perform very well overall. Moreover, using GPT-5 is significantly cheaper than the RL training process.
Training one round of RL required two L40S GPUs for one hour, costing $0.88 (the lowest available rate). Ten rounds cost $8.80. In contrast, directly calling the GPT-5 API used 44,496 input tokens and 143,673 output tokens, costing only $0.746 with batch requests.
This demonstrates that using strong LLM APIs can be a highly effective and cost-efficient alternative to RL training for most use cases.