Paul Oamen

(Paper Review) Self-Adapting Language Models

October 15, 2025 (3m ago)58 views

MIT's SEAL demonstrates that language models can use reinforcement learning to generate their own optimal training data. A 7B model trained with SEAL outperforms synthetic data from GPT-4.1 (47.0% vs 46.3%) because it optimizes specifically for "what helps me learn" rather than "what sounds good."

Self-Adapting Language Models: Teaching AI to Learn Like Humans Do

Original paper: Zweiger et al. (2025)

Right now, if you want to teach ChatGPT, Claude, or any language model something new, you have two options: either cram the information into the conversation (that's the context window), or you retrain the model on that new data. Both work, but here's what doesn't happen: the model doesn't strategically think about how to transform that information to learn it most effectively.

Imagine you're learning something new - say, quantum mechanics. You don't just read the textbook over and over exactly as written. You probably make flashcards, draw diagrams, explain concepts in your own words, or generate practice problems. You actively transform the information into formats that help YOU learn better. Different people use different strategies because we're all optimizing for our own understanding.

AI models don't do this. When you finetune them on a medical textbook, they learn from the textbook exactly as written. They don't think "maybe I should break these complex concepts into simpler implications" or "let me generate some practice questions to test my understanding." They're passive learners, not strategic ones.

And this limitation is about to become a major problem

There's research projecting that by 2028, we'll have completely exhausted human-generated text for training AI [1]. Every book, every article, every website, every scientific paper - all of it will have been used. After that point, if AI systems want to keep improving, they'll need to generate their own high-quality training data. They'll need to become active participants in their own learning process.

This is where a new paper from MIT comes in. "Self-Adapting Language Models" (SEAL) [2] demonstrates something remarkable: AI models can learn to generate their own optimal training data through reinforcement learning, and when they do, they can actually outperform much larger models at creating useful training content.

Here's how SEAL works (and why it's clever)

Let me walk you through a concrete example. Say you want to teach a model about the Apollo space program. Here's what happens:

Traditional approach:

  1. Feed the model a passage about Apollo
  2. Finetune directly on that passage
  3. Test if it learned the facts

SEAL's approach:

  1. Feed the model a passage about Apollo
  2. The model generates its own "self-edit" - maybe it writes: "The Apollo program faced political opposition from Kennedy's science advisor Jerome Wiesner" → generates implications like "Space programs can face internal government resistance" and "Presidential advisors can influence major technological initiatives"
  3. The model finetunes on these self-generated implications (not the original passage)
  4. Test how well it learned
  5. Use that test result as feedback to improve how it generates self-edits next time
Self-Edit Example: Data TransformationINPUT PASSAGE"The Apollo program faced opposition from Kennedy's scienceadvisor Jerome Wiesner, who favored robotic missions overmanned spacecraft flights..."TransformSELF-EDIT (Generated Implications)1. Space programs can face internal government resistanceEven major national initiatives may encounter opposition frominfluential advisors and policymakers.2. Presidential advisors can influence major technological initiativesScience advisors to presidents play significant roles in shapingnational policy on space exploration and technology development.3. Scientific expertise doesn't always align with political prioritiesTechnical experts may disagree with political leaders about thebest approach to achieving national goals.Model finetunes on these implications, not the original passage

It's a two-loop system:

Think of it as teaching the model to be a good student, not just a student. A good student doesn't just consume information, they develop study strategies that work for them.

SEAL's Two-Loop ArchitectureOUTER LOOP: Reinforcement Learning"Learn HOW to generate good self-edits"INNER LOOP: Supervised Learning"Learn FROM the self-edits"Self-EditFinetuneEvaluateReward SignalUpdate Self-EditGeneration Policy

The results are genuinely impressive

They tested SEAL on two very different tasks:

Task 1: Knowledge Incorporation Can the model learn facts from passages well enough to answer questions WITHOUT having the passage available anymore? (This tests: did you actually internalize this, or are you just pattern matching?)

Wait, hold on. A 7B parameter model, after learning how to generate good training data for itself, beats GPT-4.1 (a much larger, more capable model) at generating useful training content? Yes. Because SEAL optimizes specifically for "what helps ME learn" rather than "what sounds generally good."

Task 2: Few-Shot Learning Give the model just 3 examples of a reasoning puzzle and see if it can figure out the pattern. But here's the twist: the model also has to decide HOW to learn from those examples. Which data augmentations should it use? (Rotations? Reflections? Transformations?) What learning rate? How many training epochs?

In other words, the model is configuring its own learning pipeline.

Why this matters more than it might seem

Let me zoom out for a moment. There's a fundamental question about the future of AI: what happens when we run out of human data to train on?

One path is to just generate synthetic data using larger models. But that's expensive and doesn't necessarily produce data optimized for learning. It's like having a professor write your study guide - it might be accurate, but it's not personalized to how YOU learn best.

SEAL demonstrates a different path: models that learn to generate training data optimized for their own improvement.

Traditional Learning vs. SEAL LearningTraditional ApproachNew PassageDirect FinetuningEvaluationSEAL ApproachNew PassageGenerate Self-Edit(implications & explanations)Finetune on Self-Edit(not original passage)EvaluationRL Optimization(improve strategy)

And the framework is general. Right now it's knowledge incorporation and few-shot learning, but the same principle could apply to:

The vision is models that consume new information (research papers, technical docs, domain knowledge) and autonomously generate explanations and implications optimized for their own learning, then use those to update their weights, then use their enhanced capabilities to generate even better self-edits. It's a virtuous cycle of self-improvement.

Now let's talk about the hard parts (and why they're interesting)

The authors are admirably honest about limitations. Let me walk through the key challenges, but I want to use them to teach you about some fundamental problems in AI.

Challenge 1: Catastrophic Forgetting

Here's something you might not know about neural networks: when you train them on new stuff, they tend to forget old stuff. This is called catastrophic forgetting [3], and it's one of the biggest differences between human and artificial learning.

Let me explain why this happens. When you train a neural network, you're adjusting billions of parameters (weights). These weights encode everything the model "knows." When you train on new data, you're moving those weights to optimize for the new task. Problem is, moving the weights to be good at the new thing often makes them worse at the old thing.

Concrete example: Train a model to translate English → French (it gets good). Then train it on English → German. Result: It gets better at German but worse at French. The weights that encoded French translation got overwritten.

This is exactly what happens with SEAL when you do sequential self-edits. Figure 6 in the paper shows it clearly: as the model learns new passages, performance on earlier passages degrades.

Catastrophic Forgetting in Sequential Updates100%75%50%25%0%Performance on Earlier Tasks123456Number of Sequential UpdatesPerformance Degradesas model learns new tasks

Why don't models like ChatGPT or Claude have this problem?

Great question! These models were trained on massive, diverse datasets all at once. In each training batch, the model saw a mix of everything - code, then history, then physics, all jumbled together. The weights learned patterns that work across ALL domains simultaneously because they were optimized on everything together from the start.

But SEAL is trying to do continual learning - keep adapting to new information over time, like a human does throughout their life. That's much harder.

Possible solutions:

The reward shaping approach seems most practical near-term: maintain a validation set from previous tasks and include retention in the reward: . The challenge is computational - you're now evaluating on both new and old tasks for every self-edit. But if you solve the computational cost problem (I'll get to that), this becomes tractable.

Longer term, the null-space constraint idea is fascinating: identify directions in parameter space that preserve previous task performance while allowing updates for new tasks. You get adaptation without interference. Recent continual learning research on orthogonal gradient descent [4] could integrate here.

Challenge 2: The Computational Cost is Brutal

To compute the reward for a single self-edit, you have to:

  1. Actually finetune a model on that self-edit
  2. Evaluate it on downstream tasks
  3. This takes 30-45 seconds per self-edit

One training round with 50 passages, 5 self-edits each, 3 evaluation runs for variance = 750 finetuning cycles = six hours on two H100 GPUs.

Compare this to normal reinforcement learning for language models:

SEAL's reward computation is literally 10,000× more expensive.

Why this teaches us something important:

This reveals a fundamental tension in AI research. The "true" reward signal - actual performance after finetuning - is what makes SEAL's optimization meaningful. It's what enables the model to discover genuinely better learning strategies. But it's prohibitively expensive to compute at scale.

They experiment with a proxy reward (using GPT-4 to judge edit quality based on rubrics). This cuts training from 6 hours to 5 minutes, with only slight performance drop (45.6% vs 47.0%). This is promising but raises interesting questions about reward hacking and proxy alignment.

A potential solution:

Train a learned proxy reward model. Think of it like AlphaGo's value network [6] - a smaller, efficient model that predicts "will this self-edit lead to good downstream performance?" You train this proxy using data from expensive true reward evaluations, then use the cheap proxy for most training, with periodic true reward calibration.

The research question becomes: what's the optimal balance between proxy efficiency and ground truth fidelity? How often do you need true reward evaluation to keep the proxy aligned?

Challenge 3: You Need Labeled Evaluation Data

Currently, every passage needs associated questions, every task needs test queries. This prevents training SEAL on unlabeled corpora.

Why this matters:

Most of the world's information doesn't come with test questions attached. If SEAL can only learn from (passage, questions) pairs, its applicability is limited.

Possible solutions:

Use perplexity on held-out text as a proxy: if self-edits help the model better predict related content from the same domain, that's evidence of effective learning, no questions needed.

For few-shot learning, use consistency-based rewards: if the model produces identical predictions for augmented versions of the same task (rotations, reflections) after adaptation, that suggests robust learning even without ground truth.

Challenge 4: RL Algorithm Instability

They use ReSTEM [7] (filtered behavior cloning) because PPO [8] and GRPO were unstable. But why?

Here's what I think is happening:

The reward for any self-edit depends on the current model's parameters, which are constantly changing. This creates what's called a non-stationary reward landscape - the ground is shifting under your feet while you're trying to climb the hill.

In normal RL, the environment is fixed. In SEAL, the "environment" (the model's ability to learn from self-edits) is changing with every update. PPO's trust region constraints might not be enough when the underlying reward function itself is shifting.

Potential solutions:

Use more aggressive on-policy learning with shorter horizons - keep the policy close to the current reward landscape. Or add regularization that prevents rapid policy divergence, smoothing the non-stationarity. A hybrid might work best: ReSTEM for initial training when rewards are most unstable, then PPO once the policy converges.

What I'd want to see next (and what it would teach us)

Scaling experiments: They test 3B and 7B models with hints of compounding benefits (1.75× vs 2.04× relative improvement). Do these advantages persist at 30B, 70B parameters? This would tell us if SEAL becomes more powerful as base model capability increases - which would be a very good sign.

Cross-method comparisons: Why does Generative Adapter [9] (hypernetwork approach) dominate in single-passage (66.8% vs 47.0%) but collapse in continued pretraining (28.0% vs 58.2%)? Understanding this would reveal fundamental trade-offs in how we parameterize weight updates.

Less structured domains: Knowledge incorporation on SQuAD [10] and few-shot learning on ARC [11] are clean, well-defined tasks. What about open-ended generation, creative writing, dialogue? Can the model learn useful self-editing strategies when success is ambiguous?

Integration with continual learning techniques: Combine SEAL with elastic weight consolidation [12], progressive networks, or PackNet. These methods were designed for the forgetting problem SEAL faces.

The bigger picture: why this research matters

SEAL demonstrates something we didn't have proof of before: AI models can learn to optimize their own learning process through reinforcement learning on self-generated data. This isn't theory, it's working code producing measurable improvements.

A 7B model learns to generate training data better than GPT-4.1. A 1B model learns to autonomously configure its learning pipeline with 72.5% success versus 20% without that capability. These aren't marginal gains, they're qualitative shifts in what models can do.

The technical execution is solid, the experimental design is rigorous, and the limitations are clearly documented. This is early-stage research with real constraints around computational cost, training scale, and catastrophic forgetting. But these feel like engineering challenges with plausible solution paths rather than fundamental blockers.

What matters most is the paradigm shift

Models shouldn't be passive consumers of training data. They should be active architects of their own learning process, strategically transforming information to maximize improvement. This mirrors how human learning actually works.

As we approach 2028 and the exhaustion of human-generated text, progress will depend on models' capacity to generate high-quality training signals for themselves. SEAL provides concrete evidence this is possible, and that optimization for learning utility beats general capability for synthetic data generation.

If you're working on model training, synthetic data generation, continual learning, or thinking about long-term AI trajectories, this paper deserves your attention. It won't give you production-ready systems today, but it will change how you think about what's possible and what problems need solving.

And for curious readers who want to understand modern AI better, SEAL offers a window into some of the field's most important open problems: continual learning, catastrophic forgetting, reward design, and how to build systems that genuinely improve themselves over time.

That's what makes this research exciting. It's not just about better benchmarks. It's about fundamentally rethinking how AI systems learn.

References

[1] P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, and M. Hobbhahn, "Will we run out of data? Limits of LLM scaling based on human-generated data," arXiv preprint arXiv:2211.04325, 2024.

[2] A. Zweiger, J. Pari, H. Guo, E. Akyürek, Y. Kim, and P. Agrawal, "Self-Adapting Language Models," in Proc. 39th Conf. Neural Information Processing Systems (NeurIPS), 2025.

[3] M. McCloskey and N. J. Cohen, "Catastrophic interference in connectionist networks: The sequential learning problem," in Psychology of Learning and Motivation, vol. 24, pp. 109-165, Academic Press, 1989.

[4] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, "Efficient lifelong learning with A-GEM," in Proc. Int. Conf. Learning Representations (ICLR), 2019.

[5] L. Ouyang et al., "Training language models to follow instructions with human feedback," in Proc. 36th Conf. Neural Information Processing Systems (NeurIPS), 2022.

[6] D. Silver et al., "Mastering the game of Go with deep neural networks and tree search," Nature, vol. 529, no. 7587, pp. 484-489, 2016.

[7] A. Singh et al., "Beyond human data: Scaling self-training for problem-solving with language models," Trans. Machine Learning Research, 2024.

[8] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017.

[9] T. Chen, H. Fang, P. Xia, X. Liu, B. Van Durme, L. Zettlemoyer, J. Gao, and H. Cheng, "Generative Adapter: Contextualizing language models in parameters with a single forward pass," in Proc. 13th Int. Conf. Learning Representations (ICLR), 2025.

[10] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine comprehension of text," in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), pp. 2383-2392, 2016.

[11] F. Chollet, M. Knoop, G. Kamradt, and B. Landers, "ARC prize 2024: Technical report," arXiv preprint arXiv:2412.04604, 2025.

[12] J. Kirkpatrick et al., "Overcoming catastrophic forgetting in neural networks," Proc. National Academy of Sciences, vol. 114, no. 13, pp. 3521-3526, 2017.