Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

On August 12, 2026, Alibaba’s Qwen team released Qwen3.8-2.4T-A95B. This is the first time a Qwen-Max-class model has been made available as open weights. With 2.4 trillion total parameters (95 billion activated per token), a hybrid linear-plus-full-attention architecture, and native context up to 262K tokens (extensible to 1M), Qwen3.8 targets the most demanding agentic and reasoning workloads. These include multi-step coding, long-horizon planning, and autonomous tool use.

Open weights models give you full control. Data stays within your infrastructure, inference behavior can be customized, and there are no per-token API fees at scale. The trade-off is operational: hosting a 2.4T-parameter model requires purpose-built GPU infrastructure and an optimized serving stack.

In this post we show how to deploy Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod using vLLM on a ml.p6-b300 instance (8× NVIDIA B300 Blackwell Ultra GPUs). We cover the full path from cluster provisioning to an OpenAI-compatible endpoint, including vLLM configuration for NVFP4 quantization, built-in reasoning, tool calling, and native Multi-Token Prediction (MTP) speculative decoding.

This is the second post in our series on deploying open trillion-parameter models on Amazon SageMaker HyperPod. For the first post covering Kimi K3, see Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS.

Qwen3.8-2.4T-A95B at a glance

Qwen3.8-2.4T-A95B (the open-weight release of Qwen3.8-Max) is the largest and most capable model in the Qwen family. The following is a summary of the key architectural details relevant to deployment.

Architecture

The hybrid attention design is key to efficient long-context inference. Gated DeltaNet layers (69 of 92) use linear attention with a bounded recurrent state, replacing the growing KV-cache with a fixed-size memory. Gated Attention layers (23 of 92) use full quadratic attention for high-fidelity token interactions. This 3:1 ratio keeps both compute and memory bounded as context scales toward 1M tokens. This is a critical property for agentic workloads that accumulate tool outputs, code, and reasoning traces across many turns.

The fine-grained MoE distributes capacity across 512 small experts rather than a few large ones, improving routing efficiency and specialization. Only approximately 95B parameters are active per forward pass, so serving costs track activated parameters, not the full 2.4T.

Capabilities and reasoning control

Qwen3.8 is designed for agentic execution: multi-step coding, autonomous tool use, long-horizon planning, and complex research workflows. It includes built-in reasoning controls through the reasoning_effort parameter (low, medium, high), so developers can trade compute for reasoning depth per request. Dial up for hard multi-step problems and dial down for high-throughput tasks.

Model weights and quantization

The open weights are published on Hugging Face in the standard Transformers format. Community quantizations include MXFP4 and NVFP4 (W4A4), which compress the model to approximately 1.2 TB, fitting on a single 8-GPU node with B300 Blackwell Ultra GPUs.

Benchmark highlights

According to the vendor’s benchmarking results, Qwen3.8-2.4T-A95 shows particular strength in research workflows (PaperBench 93.0), instruction following (IFBench 82.8), and terminal-based coding (86.6). It performs comparably with leading frontier models across most categories, with remaining headroom on harder repository-level tasks (SWE-bench Pro) and general tool use (Toolathlon). For organizations evaluating self-hosted alternatives to proprietary APIs, these results position Qwen3.8-2.4T-A95 as a credible frontier-class option, particularly for coding agents and research pipelines.

Why Amazon SageMaker HyperPod for large MoE inference

Deploying a 2.4T-parameter model is not only a GPU problem. It requires orchestration that handles model download, container scheduling, health monitoring, autoscaling, and node failures without manual intervention. Amazon SageMaker HyperPod is purpose-built for this class of workload.

EKS-orchestrated clusters. HyperPod clusters use Amazon Elastic Kubernetes Service (Amazon EKS) as the control plane. You get the full Kubernetes landscape (kubectl, Helm charts, custom resource definitions), while AWS manages the underlying infrastructure lifecycle: networking, storage, GPU driver installation, and the NVIDIA device plugin.

Inference Operator. The HyperPod Inference Operator (installed automatically or as an EKS Add-on) provides a single custom resource definition (CRD), InferenceEndpointConfig, that declaratively specifies your model, container image, GPU resource requests, and vLLM launch arguments. The operator handles:

• Model weight download (from Hugging Face Hub, Amazon Simple Storage Service (Amazon S3), or Amazon FSx).
• Container scheduling and GPU allocation.
• Health checks and readiness gates.
• Rolling updates and endpoint lifecycle management.
• Autoscaling through KEDA with Amazon CloudWatch or Prometheus metrics.

Reserved capacity with Flexible Training Plans. The ml.p6-b300.48xlarge instance type requires reserved capacity. Flexible Training Plans provide committed GPU reservations that can be allocated directly to your HyperPod cluster. There’s no contention with on-demand pools and no cold-start capacity risk.

Resilience. HyperPod continuously monitors node health and automatically replaces degraded nodes. For sustained inference workloads running 24/7, this alleviates the operational overhead of manually detecting and recovering from hardware failures.

Additional inference features (Inference Operator v3.x):

• Disaggregated Prefill and Decode (DPD) – separates prefill and decode onto distinct GPU pools for predictable per-token latency under concurrent load.
• Inference data capture – log inputs/outputs at the endpoint, load balancer, or pod level.
• Local NVMe model deployment – load weights from node-local storage to reduce cold-start latency.
• Amazon Route 53 DNS management – automatic custom domain records for your endpoints.

In short: You write a YAML manifest describing what to deploy. HyperPod handles how to run it reliably at scale.

Infrastructure sizing: Matching hardware to the model

The p6-b300 instance

The ml.p6-b300.48xlarge provides the compute density required for single-node serving of Qwen3.8:

Why NVFP4 quantization

At BF16 precision, Qwen3.8’s 2.4T parameters require approximately 4.8 TB of memory for weights alone, exceeding a single 8-GPU node. NVFP4 (W4A4) quantization compresses weights to approximately 4 bits per parameter, bringing the total weight footprint to approximately 1.2 TB. This fits comfortably within the 2.1 TB of aggregate GPU memory on a p6-b300 instance, leaving headroom for KV-cache and activations.

Memory budget

A rough breakdown for a single p6-b300 node:

The hybrid attention architecture is a key advantage here: the 69 DeltaNet layers maintain a fixed-size recurrent state regardless of context length, unlike traditional models where KV-cache grows linearly with every layer. Only the 23 full-attention layers contribute to context-dependent memory growth.

Throughput expectations

Reference numbers from NVIDIA’s Day-0 benchmarks on GB300 NVL72 (FP8, 72 GPUs): >4K tokens/sec/GPU, >350 tokens/sec/user. A single 8-GPU p6-b300 node with NVFP4 will deliver proportionally lower aggregate throughput but remains well-suited for production inference workloads with moderate concurrency.

Capacity procurement

The ml.p6-b300.48xlarge instance type isn’t available on-demand. You must procure capacity through a Flexible Training Plan, a committed reservation of GPU availability for your HyperPod cluster. Set the target Availability Zone to match your plan’s allocation when configuring the instance group.

vLLM configuration deep dive

This section details the vLLM serving parameters for Qwen3.8 on a single p6-b300 node. The configuration is informed by the vLLM recipe for Qwen3.8 on B300 (NVFP4).

Base serving command

The full vllm serve invocation:

Key flags explained:

• --tensor-parallel-size 8 – shards the model across all 8 B300 GPUs.
• --quantization nvfp4 – activates NVIDIA FP4 (W4A4) quantization so the 2.4T model fits in 2.1 TB of GPU memory.
• --load-format fastsafetensors – uses accelerated weight deserialization for faster cold-start.
• --trust-remote-code – required for Qwen3.8’s custom modeling code on Hugging Face.
• --enable-prefix-caching – reuses computed KV-cache across requests that share prompt prefixes. Critical for multi-turn agentic conversations where the system prompt and conversation history repeat.
• --moe-backend auto – lets vLLM select the optimal MoE dispatch kernel for the hardware.

Reasoning (thinking mode)

The --reasoning-parser qwen3 flag extracts reasoning content from the model’s <think>...</think> output blocks. Key behaviors:

• Qwen3.8 reasoning is enabled by default – no extra flag needed on the model side.
• The API response separates reasoning_content (the thinking trace) fromcontent (the final answer).
• To disable thinking per-request, pass extra_body={"chat_template_kwargs": {"enable_thinking": False}} in the client call.
• Structured output (guided_json ,guided_regex ) works alongside reasoning – the structured output engine constrains only thecontent field.

Tool calling (function calling)

The --enable-auto-tool-choice and --tool-call-parser qwen3 flags enable OpenAI-compatible function calling:

• Supports tool_choice values:auto ,required ,none , and named functions.
• Tool calls are parsed from the content field only — thereasoning_content is not parsed for function calls. This means the model can reason aboutwhich tool to call, then emit the structured call separately.
• When tool_choice="auto" andstrict: true is set on a tool definition, vLLM enforces schema-constrained decoding for tool arguments, facilitating valid JSON output.

Speculative decoding (native MTP)

The --speculative-config '{"method":"mtp","num_speculative_tokens":1}' flag enables Multi-Token Prediction using Qwen3.8’s built-in draft heads:

• Qwen3.8 was trained with MTP – lightweight draft heads are bundled in the model weights. No separate draft model download or configuration is required.
• The draft head predicts the next N tokens in parallel, then verifies them in a single forward pass. Accepted tokens skip individual decode steps, increasing throughput.
• num_speculative_tokens: 1 is the safe starting point. Increase to 2–3 for throughput-sensitive workloads once you’ve validated that the acceptance rate remains high (monitor through vLLM’s/metrics endpoint).
• MTP adds minimal latency overhead on the draft step because the heads reuse the model’s existing hidden states.

Deployment walkthrough on SageMaker HyperPod

The complete deployment manifests and scripts used in this post are available in our GitHub repository.

Prerequisites

Before deploying the model, you need a running HyperPod cluster with p6-b300 capacity:

1. Create a HyperPod cluster with EKS orchestration. In the Amazon SageMaker AI console, navigate toHyperPod Clusters , then chooseCreate . ChooseOrchestrated by Amazon EKS and eitherQuick Setup (default networking and IAM) orCustom Setup with an existing virtual private cloud (VPC) and subnets. Make sureUse default Helm charts and add-ons is selected so the Inference Operator is installed automatically.
2. Provision a Flexible Training Plan. Under the instance group configuration, selectTraining plan as the capacity source. Create or attach a plan coveringml.p6-b300.48xlarge with the instance count and duration you need. Set the Target Availability Zone to match the plan’s allocation.
3. Add a p6-b300 worker group. Add an instance group withml.p6-b300.48xlarge and at least 1 instance. Wait for the cluster to reachActive state with healthy GPU nodes.
4. Verify access. Confirm you can reach the cluster:

InferenceEndpointConfig manifest

Apply the following InferenceEndpointConfig to deploy Qwen3.8 with the vLLM configuration:

This manifest is also available in the GitHub repository.

Applying and monitoring

Apply the manifest:

Monitor the deployment progress:

The deployment proceeds through these stages: model download (approximately 1.2 TB from Hugging Face, time depends on network bandwidth) then weight loading (fastsafetensors deserialization to GPU memory) then health checks pass and endpoint ready. On a fresh deployment with no cached weights, expect 15–30 minutes for the full sequence. Subsequent restarts with local NVMe caching are significantly faster.

After the endpoint shows Ready, you can send requests to the service:

Inference in action: Calling the endpoint

After the endpoint is ready, it exposes an OpenAI-compatible API. You can use the standard OpenAI Python SDK, curl, or another HTTP client (note that in this deployment example the endpoint isn’t exposed to the public internet).

Basic chat completion (with reasoning)

To control reasoning depth per request, pass reasoning_effort:

For streaming responses, check for the reasoning_content attribute on each chunk’s delta:

Tool calling example

With strict: True set on the tool definition, vLLM makes sure the arguments conform to the JSON schema. No post-validation is needed.