Upgrade AI infra: Huawei OceanStor M900 offers 64PB KV cache to slash latency

Huawei has released a new storage system designed to help large artificial intelligence models run faster and cheaper by sharing memory resources across a cluster. The company announced the OceanStor M900 at its HUAWEI CONNECT 2026 event in Shanghai, positioning it as a solution for organizations moving toward more complex AI interactions that require larger context windows and longer conversations.

In plain English

When AI models process information, they need to remember previous parts of a conversation to make sense of new inputs. This memory requirement grows significantly as conversations become longer or more complex. Traditional computer memory is fast but expensive and limited in size. Standard storage drives are cheaper and hold more data, but they are too slow for AI processors to wait for without losing efficiency. Huawei’s new system attempts to bridge this gap by creating a middle ground that offers high capacity with low delay, allowing AI systems to access vast amounts of cached data quickly.

The background

As organizations shift from simple AI queries to autonomous agents and complex reasoning tasks, the demand on infrastructure changes. Processing large amounts of context places heavy pressure on memory capacity. Keeping AI accelerators supplied with data is becoming just as critical as raw computing power. The key-value cache, which stores information generated during inference so models can reuse previous computations, grows considerably as context windows expand. GPU and system memory provide fast access but are constrained by cost and capacity, while traditional storage introduces latency through CPU-mediated data paths.

What changes now

Huawei introduced the OceanStor M900 Context Memory Storage to support AI inference across hyperscale datacenters and its Ascend SuperPoD infrastructure. The platform adds a dedicated Context Memory Storage layer that extends the key-value cache into a shared, multi-tier storage pool. Using Huawei's UnifiedBus high-speed interconnect, the system allows KV cache resources to be shared across a cluster. Huawei claims the platform offers capacity of up to 64PB and provides substantially more context per NPU.

The system uses an innovative ASU disk that integrates CPU, networking, and storage controller functions. Combined with native KV semantics, this design allows NPUs to access stored cache data in a single hop, avoiding conventional protocol conversion and CPU forwarding. According to Huawei, this approach reduces access latency to 60 microseconds. The company states this could potentially double token throughput and halve the time to the first token in typical AI coding workloads.

Huawei also highlights cost and durability benefits. Its KV-aware adaptive storage technology manages data placement according to cache lifecycles. The company claims endurance of up to 24 drive writes per day, which is 16 times that of conventional SSDs. Huawei argues this durability could reduce replacement and maintenance costs over time for large-scale deployments.

Key terms explained

System administrators should understand that AI inference relies heavily on caching previous computations to maintain context. When these caches exceed the fast memory attached to processors, performance drops. Teams usually evaluate storage solutions based on how quickly data can move from disk to processor. Check whether your current infrastructure can handle the increased data movement required by larger context windows. Ensure your storage architecture can scale without introducing significant latency that would leave expensive accelerators idle while waiting for information.

What to do and how to stay safe: Huawei

• Review your current AI inference workloads to identify if memory constraints are limiting context window sizes or causing latency spikes.
• Evaluate whether your existing storage architecture can support the increased data throughput required by growing key-value caches.
• Monitor the performance of your AI accelerators to determine if they are frequently waiting for data from slower storage tiers.
• Plan for potential infrastructure upgrades that prioritize low-latency data access paths between storage and processing units.

Step-by-step guide: Stop AI Chatbots Leaking Sensitive Business Data

General security guidance from the Vector Update newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is the OceanStor M900?

It is a Context Memory Storage platform introduced by Huawei at HUAWEI CONNECT 2026 to support AI inference in hyperscale datacenters.

How much capacity does the platform offer?

Huawei claims the OceanStor M900 provides KV cache capacity of up to 64PB across a shared cluster.

What performance improvements does Huawei claim?

The company states the system reduces access latency to 60 microseconds and can double token throughput in typical AI coding workloads.