Social Twitter May 29
Most people training agentic LLMs with RL right now have a silently broken training loop and have no...
By @ClementDelangue
88 score
AI Analysis
Clement Delangue of Hugging Face explains a subtle bug in agentic LLM RL training: re-tokenizing decoded outputs between turns causes silent gradient errors, with the fix being to never re-encode decoded tokens (Token-In, Token-Out), referencing a team deep-dive.
Most people training agentic LLMs with RL right now have a silently broken training loop and have no idea.
Here's the trap: single-turn RL works beautifully. Clean curves, sane rewards, everything converges. Then you add tools so the model can act mid-rollout, and things get weird. Loss spikes for no reason. Eventually a shape-mismatch error.
The culprit: every time you parse the model's output to detect a tool call, then re-tokenize the updated conversation for the next turn, you're rolling t
reinforcement learningagentic LLMstokenizationtraining infrastructure