EAGLE makes AI agents act faster by tackling one of their biggest bottlenecks. Speculative tool calling is already a major unlock for agentic systems. EAGLE takes it further by accelerating the inference behind every tool call. Instead of waiting for the model to finish writing a code block, you parse it as it streams and fire each tool call the moment its line completes. On a 7-call agent turn (6 independent + 1 dependent, simulated 2s tools), the first tool call using EAGLE fires 3 times faster compared to vanilla GLM. Two different things get you there: Time to First Token (TTFT) (3x faster): you start emitting sooner Throughput (11x faster): you clear the first statement sooner Time-to-first-tool-call is just TTFT + the decode of that first statement. Eagle cuts both halves.

This is a great step forward for open-source models. The cost advantage makes it possible to scale inference and verification in ways that would be expensive with closed models. The tradeoff is latency. Verification can require dozens of inference passes per task. While most can run in parallel, some must run sequentially, making end-to-end latency multiple times longer than a single generation. With EAGLE, we can accelerate those inference passes and bring the end-to-end time faster than a single standard generation, while keeping the benefits of self-verification.

AI hardware has advanced rapidly over the past few years, delivering unprecedented compute for training and inference. As AI systems continue to scale, achieving higher performance increasingly depends on both more capable hardware and more efficient software. Speculative decoding is a prime example. Rather than requiring new hardware, it unlocks more performance from existing accelerators by fundamentally changing how inference is executed. A lightweight draft model predicts multiple future tokens, which are then verified in parallel by the target model. By reducing sequential decoding steps and improving hardware utilization, speculative decoding enables substantially higher throughput from the same underlying compute. This reflects a broader shift in AI infrastructure. Software is redefining AI performance by unlocking more from increasingly powerful hardware. As accelerators continue to evolve, advances in inference algorithms will be just as critical as advances in the hardware itself, together defining the next generation of AI systems.

Can increasing draft model size improve speculative decoding performance? Intuitively, the answer should be yes. Larger draft models generally produce more accurate predictions, leading to higher acceptance rates during verification. Conventional wisdom suggests that this should translate directly into higher throughput. Our recent experiments on GPT-OSS-120B suggest the relationship is more nuanced. We evaluated EAGLE draft models with 1, 4, and 16 layers and observed that increasing draft model size does improve prediction accuracy, particularly for medium-difficulty tokens. However, the corresponding gains in acceptance rate are often limited, especially once acceptance approaches saturation. At the same time, larger draft models introduce additional drafting overhead. This creates a fundamental trade-off: acceptance rate increases with draft model size, but so does the cost of generating drafts. When draft trees are enabled, the effect becomes even more apparent. Across many token distributions, all draft models already achieve very high acceptance rates, leaving little room for additional gains from larger architectures. The implication is straightforward. Speculative decoding is not an optimization problem for acceptance rate alone. It is an optimization problem for end-to-end throughput. In many cases, reducing draft overhead can be more effective than increasing draft model complexity. Sometimes, less is more.

Speculative decoding transforms the inference process from sequential token generation into a draft-and-verify paradigm. A draft model proposes future tokens. A target model verifies them in parallel. Inference speedup is ultimately determined by acceptance. This video explains the the mechanics of speculative decoding and explain how EAGLE improves acceptance rates to deliver state-of-the-art inference acceleration.

EAGLE introduced a simple but important idea: future tokens should not be predicted independently. This same architectural principle is reflected in DeepSeek-V3's Multi-Token Prediction (MTP) design, which combines the MTP training paradigm with an EAGLE-style causal drafting architecture. Instead of relying on parallel LM heads, EAGLE's autoregressive drafting architecture conditions each draft token on previously drafted tokens, explicitly modeling dependencies within the draft sequence. This architectural change improves draft quality by allowing the model to reason over future-token relationships rather than predicting each position in isolation. What's particularly interesting is how this idea has evolved beyond speculative decoding. Meta's Multi-Token Prediction (MTP) introduced the concept of training additional heads to predict multiple future tokens during pre-training. DeepSeek-V3 combines this training paradigm with EAGLE's autoregressive drafting approach, replacing independent future-token prediction with causal drafting. The result is a notable example of training and inference research influencing one another. An architectural idea originally developed for inference acceleration is now incorporated into frontier model architectures themselves.

EAGLE-3 speculative decoding has now been integrated natively into llama.cpp. The implementation adds support for EAGLE-3's encoder-decoder speculative decoding architecture, with reported inference speedups of 2–3×. This brings one of the most widely adopted speculative decoding algorithms from large-scale serving environments to local and consumer hardware deployments. Start running EAGLE-3 in llama.cpp: github.com/ggml-org/llama.cp…

Glad to see EAGLE-3 powers up Minimax and vLLM.

Today, @MiniMax_AI unveiled MiniMax M3, bringing together frontier coding performance, agentic capabilities, native multimodality, and a 1M-token context window in a single open-weights model. We're proud to see MiniMax M3 powered by an open-source speculator built by the @inferact team on top of Hongyang Zhang's EAGLE-3 algorithm from Eagle Corp, helping bring faster, production-grade inference to one of the most capable open models released today. Inferact/MiniMax-M3-EAGLE3 is a 3.3B-parameter draft model trained with TorchSpec’s online speculative decoding framework, combining FSDP training with live vLLM target inference to learn directly from MiniMax-M3 responses and hidden states while preserving the base model’s exact token distribution. Served end-to-end with vLLM, it leverages EAGLE-3’s shared embedding and LM head architecture to deliver efficient, production-grade inference acceleration at scale. Explore the open-source EAGLE-3 speculator powering MiniMax M3 on @huggingface: huggingface.co/Inferact/Mini…

EAGLE-3 introduces two key innovations: Training-Time Testing (TTT) and multi-level feature fusion. By removing the feature prediction constraint used in previous EAGLE versions and leveraging semantic features across multiple layers, EAGLE-3 achieves higher acceptance rates, faster generation, and lossless performance. The result? - 5.6× faster than vanilla decoding (13B) Compared to EAGLE-1, EAGLE-3 delivers a 1.8× speedup on the 13B model, with the future EAGLE-4 release expected to further improve decoding efficiency. *Inference on the video conducted on 2x RTX 3090 GPUs at fp16 precision using the Vicuna 13B model.

Today, EAGLE powers some of the industry’s most formative AI infrastructure companies and teams. With EAGLE 3.1, we’re taking another major step toward delivering a core piece of the fastest possible inference stack that exists, open to all. By improving hidden-state feedback stability and mitigating attention drift across deeper decoding steps, EAGLE 3.1 significantly improves long-context acceptance length and serving robustness in real-world inference environments. We are thrilled to collaborate with vLLM @vllm_project and TorchSpec @lightseekorg on advancing the next generation of inference acceleration infrastructure.