Category intelligence

Research Briefing — February 6, 2026

564 current items analyzed and ranked.

Executive synthesis

Research Summary

Today's research spans foundational AI evaluation, training failure modes, safety limitations, and theoretical insights into optimization.

First Proof, authored by Fields medalists and Abel Prize winners including Martin Hairer, introduces 10 unpublished math problems to benchmark AI on genuine mathematical reasoning — a landmark evaluation effort.
Chunky Post-Training identifies a systematic failure mode where LLMs learn spurious correlations from distinct data chunks, with practical implications for post-training pipelines.
Compound Deception in Elite Peer Review reveals ~100 AI-hallucinated citations across ~1% of NeurIPS 2025 accepted papers, exposing a serious integrity gap in top-tier review.
Phantom Transfer demonstrates that data poisoning attacks persist even when the exact method is known and full paraphrasing defenses are applied — a strong negative result for data-level safety.

PACE challenges the scaling hypothesis in iterative DPO for math reasoning, showing diminishing returns from aggressive Best-of-N exploration. Steven Byrnes's analysis of LLM algorithmic progress critically reexamines widely-cited ~8-month halving-time estimates, arguing they conflate distinct improvement sources. Alignment Verifiability formalizes why behavioral evaluation cannot distinguish truly aligned models from strategically compliant ones. Steering Externalities shows benign activation steering (e.g., for JSON output) inadvertently degrades safety guardrails. An exactly solvable model demonstrates SGD has no intrinsic preference for flat minima, challenging the flatness-seeking narrative. Finally, a first general uncertainty quantification framework for LLM agents moves beyond single-turn QA toward reliable agentic deployment.

Key Themes

AI Safety & Alignment · 12Academic Integrity and LLM Hallucination · 1Agentic AI & Multi-Agent Systems · 16AI Safety & Adversarial Attacks · 19AI Safety and Alignment · 15LLM Post-Training & Alignment · 8Reasoning Models & RLHF/RLVR · 7LLM Evaluation & Benchmarking · 14GRPO and RLVR for LLM Reasoning · 4Deep Learning Theory & Optimization · 12

Primary evidence

Top Ranked Signals

Research arXiv (Artificial Intelligence) Feb 6

First Proof

By Mohammed Abouzaid, Andrew J. Blumberg, Martin Hairer, Joe Kileel, Tamara G. Kolda, Paul D. Nelson, Daniel Spielman, Nikhil Srivastava, Rachel Ward, Shmuel Weinberger, Lauren Williams

82 score

AI Analysis

A distinguished group of mathematicians shares 10 unpublished research-level math questions to benchmark current AI systems on genuine mathematical research, with encrypted answers to prevent contamination.

arXiv:2602.05192v1 Announce Type: new Abstract: To assess the ability of current AI systems to correctly answer research-level mathematics questions, we share a set of ten math questions which have arisen naturally in the research process of the authors. The questions had not been shared publicly until now; the answers are known to the authors of the questions but will remain encrypted for a short time.

Mathematical ReasoningLLM EvaluationBenchmarking

Research arXiv (Machine Learning) Feb 6

Chunky Post-Training: Data Driven Failures of Generalization

By Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price

78 score

AI Analysis

Identifies 'chunky post-training' as a failure mode where LLMs learn spurious correlations from distinct data chunks during post-training. Introduces SURF (detection) and TURF (mitigation) pipelines. Includes John Schulman and Collin Burns as authors.

arXiv:2602.05910v1 Announce Type: new Abstract: LLM post-training involves many diverse datasets, each targeting a specific behavior. But these datasets encode incidental patterns alongside intended ones: correlations between formatting and content, narrow phrasings across diverse problems, and implicit associations arising from the discrete data curation process. These patterns are often invisible to developers yet salient to models, producing behaviors that surprise their creators, such as re

LLM Post-TrainingAI SafetyData QualitySpurious Correlations

Research arXiv (Artificial Intelligence) Feb 6

Phantom Transfer: Data-level Defences are Insufficient Against Data Poisoning

By Andrew Draganov, Tolga H. Dur, Anandmayi Bhongade, Mary Phuong

75 score

AI Analysis

Presents 'Phantom Transfer,' a data poisoning attack that remains effective even when the exact poisoning method is known and defenses like full paraphrasing are applied. Demonstrates the attack works across models including GPT-4.1, suggesting data-level defenses are fundamentally insufficient against sophisticated poisoning.

arXiv:2602.04899v1 Announce Type: cross Abstract: We present a data poisoning attack -- Phantom Transfer -- with the property that, even if you know precisely how the poison was placed into an otherwise benign dataset, you cannot filter it out. We achieve this by modifying subliminal learning to work in real-world contexts and demonstrate that the attack works across models, including GPT-4.1. Indeed, even fully paraphrasing every sample in the dataset using a different model does not stop the

AI SafetyData PoisoningAdversarial ML

Research arXiv (Artificial Intelligence) Feb 6

Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025

By Samar Ansari

75 score

AI Analysis

Analyzes 100 AI-generated hallucinated citations that appeared in 53 NeurIPS 2025 accepted papers (~1% of all accepted papers), developing a five-category taxonomy of citation hallucination failure modes.

arXiv:2602.05930v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in academic writing workflows, yet they frequently hallucinate by generating citations to sources that do not exist. This study analyzes 100 AI-generated hallucinated citations that appeared in papers accepted by the 2025 Conference on Neural Information Processing Systems (NeurIPS), one of the world's most prestigious AI conferences. Despite review by 3-5 expert researchers per paper, these fab

AI SafetyAcademic IntegrityLLM HallucinationScientific Publishing

Research arXiv (Computation and Language) Feb 6

PACE: Defying the Scaling Hypothesis of Exploration in Iterative Alignment for Mathematical Reasoning

By Jun Rao, Zixiong Yu, Xuebo Liu, Guhan Chen, Jing Li, Jiansheng Wei, Xiaojun Meng, Min Zhang

73 score

AI Analysis

Challenges the scaling hypothesis in iterative DPO for mathematical reasoning, showing that aggressive Best-of-N exploration yields diminishing returns and policy collapse. Proposes PACE (Proximal Alignment via Corrective Exploration) as alternative.

arXiv:2602.05370v1 Announce Type: new Abstract: Iterative Direct Preference Optimization has emerged as the state-of-the-art paradigm for aligning Large Language Models on reasoning tasks. Standard implementations (DPO-R1) rely on Best-of-N sampling (e.g., $N \ge 8$) to mine golden trajectories from the distribution tail. In this paper, we challenge this scaling hypothesis and reveal a counter-intuitive phenomenon: in mathematical reasoning, aggressive exploration yields diminishing returns and

AlignmentReinforcement LearningMathematical ReasoningLLM Training

Research arXiv (Artificial Intelligence) Feb 6

Towards Reducible Uncertainty Modeling for Reliable Large Language Model Agents

By Changdae Oh, Seongheon Park, To Eun Kim, Jiatong Li, Wendi Li, Samuel Yeh, Xuefeng Du, Hamed Hassani, Paul Bogdan, Dawn Song, Sharon Li

72 score

AI Analysis

Presents the first general formulation of uncertainty quantification (UQ) for LLM agents beyond single-turn QA, proposing an 'uncertainty reduction' perspective instead of 'uncertainty accumulation.' Argues interactive agents can actively reduce uncertainty through environmental interaction.

arXiv:2602.05073v1 Announce Type: new Abstract: Uncertainty quantification (UQ) for large language models (LLMs) is a key building block for safety guardrails of daily LLM applications. Yet, even as LLM agents are increasingly deployed in highly complex tasks, most UQ research still centers on single-turn question-answering. We argue that UQ research must shift to realistic settings with interactive agents, and that a new principled framework for agent UQ is needed. This paper presents the firs

AI SafetyUncertainty QuantificationAgentic AILanguage Models

Research arXiv (Artificial Intelligence) Feb 6

Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models

By Chen Xiong, Zhiyuan He, Pin-Yu Chen, Ching-Yun Ko, Tsung-Yi Ho

72 score

AI Analysis

Discovers 'Steering Externalities' where benign activation steering vectors (e.g., for compliance or JSON output) inadvertently erode safety guardrails in LLMs, increasing jailbreak risk. This is a surprising and practically important finding for model deployment. Co-authored by Pin-Yu Chen from IBM Research.

arXiv:2602.04896v1 Announce Type: cross Abstract: Activation steering is a practical post-training model alignment technique to enhance the utility of Large Language Models (LLMs). Prior to deploying a model as a service, developers can steer a pre-trained model toward specific behavioral objectives, such as compliance or instruction adherence, without the need for retraining. This process is as simple as adding a steering vector to the model's internal representations. However, this capability

AI SafetyActivation SteeringLLM Alignment

Research arXiv (Artificial Intelligence) Feb 6

Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluation

By Igor Santos-Grueiro

72 score

AI Analysis

Formally analyzes alignment verifiability in LLMs, showing that behavioral evaluation cannot distinguish truly aligned models from strategically compliant ones under partial observability. Introduces normative indistinguishability concepts.

arXiv:2602.05656v1 Announce Type: cross Abstract: Behavioral evaluation is the dominant paradigm for assessing alignment in large language models (LLMs). In practice, alignment is inferred from performance under finite evaluation protocols - benchmarks, red-teaming suites, or automated pipelines - and observed compliance is often treated as evidence of underlying alignment. This inference step, from behavioral evidence to claims about latent alignment properties, is typically implicit and rarel

AI SafetyAlignmentFormal AnalysisLLM Evaluation

Research arXiv (Artificial Intelligence) Feb 6

Learning to Inject: Automated Prompt Injection via Reinforcement Learning

By Xin Chen, Jie Zhang, Florian Tramer

72 score

AI Analysis

Proposes AutoInject, an RL framework for automated prompt injection attacks that generates transferable adversarial suffixes, successfully compromising frontier systems including GPT 5 Nano and Claude Sonnet 3.5 using only a 1.5B parameter model.

arXiv:2602.05746v1 Announce Type: cross Abstract: Prompt injection is one of the most critical vulnerabilities in LLM agents; yet, effective automated attacks remain largely unexplored from an optimization perspective. Existing methods heavily depend on human red-teamers and hand-crafted prompts, limiting their scalability and adaptability. We propose AutoInject, a reinforcement learning framework that generates universal, transferable adversarial suffixes while jointly optimizing for attack su

AI SafetyPrompt InjectionAdversarial AttacksRed Teaming

Research arXiv (Machine Learning) Feb 6

Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model

By Yizhou Xu, Pierfrancesco Beneventano, Isaac Chuang, Liu Ziyin

72 score

AI Analysis

Provides an exactly solvable analytical model showing that SGD has no intrinsic preference for flat minima, but rather prefers minimal gradient fluctuations, with data distribution determining whether flattening or sharpening occurs during training.

arXiv:2602.05065v1 Announce Type: new Abstract: A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there have been conceptually opposite pieces of evidence regarding when SGD prefers flatter or sharper solutions during training. In this work, we partially but causally clarify the flatness-seeking behavior of SGD by identifying and exactly solving an analytically solvable mode

Optimization TheorySGDLoss LandscapeDeep Learning Theory

Research arXiv (Machine Learning) Feb 6

Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold

By Ye He, Yitong Qiu, Molei Tao

72 score

AI Analysis

Provides a theoretical characterization of how diffusion models generalize beyond memorization by proposing a 'log-density ridge manifold' concept. The inference process is described as a reach-align-slide process around this manifold, offering quantitative understanding of generated distributions.

arXiv:2602.06021v1 Announce Type: cross Abstract: When a diffusion model is not memorizing the training data set, how does it generalize exactly? A quantitative understanding of the distribution it generates would be beneficial to, for example, an assessment of the model's performance for downstream applications. We thus explicitly characterize what diffusion model generates, by proposing a log-density ridge manifold and quantifying how the generated data relate to this manifold as inference dy

Diffusion ModelsGeneralization TheoryMachine Learning Theory

Research arXiv (Computation and Language) Feb 6

Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systems

By Ziyuan Yang, Wenxuan Ding, Shangbin Feng, Yulia Tsvetkov

72 score

AI Analysis

Studies safety risks in multi-LLM collaboration systems by engineering malicious models and measuring their impact across routing, debate, and merging systems. Finds severe impacts especially on reasoning (7.12%) and safety (7.94%) performance.

arXiv:2602.05176v1 Announce Type: new Abstract: Language models (LMs) are increasingly used in collaboration: multiple LMs trained by different parties collaborate through routing systems, multi-agent debate, model merging, and more. Critical safety risks remain in this decentralized paradigm: what if some of the models in multi-LLM systems are compromised or malicious? We first quantify the impact of malicious models by engineering four categories of malicious LMs, plug them into four types of

AI SafetyMulti-Agent SystemsAdversarial AttacksModel Collaboration