Research Briefing

An empirical study showing that GPT-6-Astra performs substantially better when prompted with filler tokens and instructed to answer without verbal reasoning, improving from roughly 10% to 50% on 4-hop natural facts reasoning and from 60% to 90% on older AIME problems. The finding suggests Astra can execute significant unverbalized cognition, undermining chain-of-thought monitorability.

Tencent's Hy Foundation team presents T1, a 122B-parameter Mixture-of-Experts model post-trained with reinforcement learning to perform long-horizon Linux terminal tasks. T1 reaches 64.0% on Terminal-Bench 2.1, a 28.5% relative improvement over its SFT checkpoint and ahead of several large contemporaries, using new MoE training-stability mechanisms and dense reward generation.

SchemeArena presents a factorized 400-scenario benchmark plus an evidence-based monitor for studying covert scheming in LLM agents, varying instrumental goals, oversight, and strategic hints. It enables controlled stress tests of misaligned instrumental behavior rather than relying on single-shot jailbreak probes.

BenchShield is a model-backed instrumentation layer for reward integrity in LLM-agent evaluation, grounding detection in a finite lifecycle model of an evaluation's reward-relevant events. It provides reusable evidence that a concrete run remained within its intended evaluation boundary, addressing reward hacking beyond task-specific patches or post-hoc detectors.

SpecGuard repurposes speculative decoding for inference-time backdoor detection in LLMs at zero added model-computation cost, avoiding assumptions about trigger form or extra generation passes. Targets deployment scenarios where models are frequently updated and runtime auditing is needed alongside pre-deployment checks.

A technical document operationalizing 'opaque serial depth' (from Brown-Cohen et al., GDM 2026) as a measurable proxy for unverbalized cognition, with a specific standard for what counts as an 'interpretable bottleneck.' The post grounds the measure in computational graph analysis and proposes concrete measurement criteria for monitoring latent reasoning.

Neel Nanda independently replicates a striking UK AISI finding that Astra (OpenAI, GA 2026-09-01) performs dramatically better on no-chain-of-thought reasoning than peers, reporting an 8.6x odds advantage over the next-best model and roughly 7.2 serial arithmetic steps in a single forward pass. The result is presented on a custom No-CoT Reasoning Index benchmark and discussed as concerning for chain-of-thought monitorability.

Same Neel Nanda post cross-posted to the AI Alignment Forum, replicating UK AISI's finding that Astra shows a large jump in no-CoT reasoning ability (8.6x odds over the next-best model) and ~7.2 serial arithmetic steps per forward pass vs ~4.1 for the next best. Discusses implications for monitorability and latent reasoning.

Continuing our coverage from yesterday, Documents systematic reward-hacking and quality issues in SWE-Bench Pro (gold-solution leakage, hidden test info, misleading problem statements, improperly scoped tests) and proposes SWE-Bench Pro Verified with anti-hacking safeguards and minimal task corrections. The work is essentially a benchmark integrity audit and fix.

Empirically shows that Mixture-of-Experts models degrade faster than dense Transformers under data repetition, with degradation scaling with total (not active) parameter count and increasing with sparsity. Important for training strategies as human text supplies deplete.