How do you divide eight NVIDIA B200 GPUs among several virtual machines without breaking NVLink connectivity or allocating the same accelerator twice? NAVER's engineering team describes a practical answer: replace count-only Kubernetes device-plugin allocation with Dynamic Resource Allocation (DRA), model the NVIDIA Fabric Manager's permitted GPU partitions as schedulable devices, and pass the selected hardware into Kata Containers virtual machines.
This is part three of NAVER's four-part engineering series on container-style operations with VM-style isolation. The source is NAVER D2's Korean technical article on Fabric Manager partitions and DRA. The detailed original text was provided for this analysis; the webpage itself was not independently accessible during preparation. All benchmarks below are NAVER-reported measurements, not tests reproduced by NetContentSEO.
Why requesting any two GPUs is not enough
In a multi-GPU training or inference workload, accelerators exchange activations and gradients. Collective operations such as NCCL all-reduce can become bottlenecks if the selected GPUs cannot use their intended high-bandwidth interconnect. NAVER's B200 node contains eight GPUs connected through NVSwitch, with NVIDIA Fabric Manager controlling which allowed combinations form an NVLink fabric partition.
A fabric partition is not the same as a disk partition or NVIDIA MIG. Rather than splitting one GPU into smaller slices, it groups specific physical GPUs into an NVLink-connected unit. When a two-GPU partition is activated, those two GPUs can communicate through the configured fabric without being linked into another tenant's GPU partition. Two active partitions cannot overlap on the same physical GPU.
The actual partitions reported on NAVER's B200 node
| Partition type | Available combinations | Physical GPU grouping |
|---|---|---|
| 8 GPU | 1 | 1–8 |
| 4 GPU | 2 | 1–4; 5–8 |
| 2 GPU | 4 | 1–2; 3–4; 5–6; 7–8 |
| 1 GPU | 8 | One partition for each GPU |
These fifteen configurations are alternative partitions, not fifteen independently usable GPU groups. Their physical GPU identifiers are Fabric Manager identifiers, not PCI-address ordering. The permitted combinations are hardware-dependent and must be discovered from each node rather than copied blindly to another server. For example, GPUs 1 and 2 form an allowed two-GPU partition in this environment, while GPUs 2 and 3 do not.
Why the traditional Kubernetes device plugin failed
The conventional device-plugin workflow exposes an integer resource such as nvidia.com/gpu: 4. Kubernetes schedules a Pod based on a node's available GPU count; the node's kubelet and plugin then select specific devices. But a quantity cannot encode which four GPUs belong to a valid Fabric Manager partition.
NAVER tested three approaches. First, requesting four passthrough GPUs with the NVIDIA sandbox device plugin could select an incompatible set, causing CUDA initialization to fail with cudaErrorSystemNotReady (802). Second, specifying CDI device identifiers through Pod annotations did not cause kubelet to attach the requested GPUs. Third, a forked plugin using GetPreferredAllocation() could recommend a suitable grouping, but kubelet could silently fall back to ordinary allocation if the recommendation was unavailable. The device-plugin API also lacked a corresponding release notification that the custom allocator could use to maintain reliable partition state.
The lesson is broader than this GPU model: when the validity of a resource depends on its relationship to other resources, counting devices is insufficient. The scheduler needs to understand device identities, attributes and mutual exclusion before choosing a node and allocation.
How Kubernetes DRA changes the allocation model
Dynamic Resource Allocation provides named devices with attributes and a claim-based lifecycle. A DeviceClass describes the type of resource, ResourceSlice publishes a node's available devices, and a ResourceClaim represents a workload's request and allocation. A ResourceClaimTemplate allows an administrator to package the request so that users only need to reference a predefined GPU tier.
The important change is where selection happens. With the older device-plugin pattern, kubelet makes the specific selection on the chosen node. With DRA, the scheduler can evaluate device properties and constraints while allocating resources. In NAVER's implementation, each Fabric Manager partition becomes one DRA device with attributes such as GPU count (size) and partition identifier (fmPartitionId). A Common Expression Language selector can request a partition whose size == 8, for example.
Preventing overlapping partitions with SharedCounters
Publishing all fifteen partition alternatives creates an obvious danger: a four-GPU partition containing GPUs 1–4 overlaps with the two-GPU partitions containing 1–2 and 3–4, and with four one-GPU alternatives. They cannot be allocated simultaneously.
NAVER used DRA SharedCounters, with one counter of capacity one per physical GPU. Every partition device declares consumption of one unit from each GPU it includes. When the scheduler allocates the four-GPU partition covering GPUs 1–4, those four counters are exhausted, so no overlapping partition remains eligible. The separate four-GPU partition covering GPUs 5–8 can still be allocated. This scheduling design depends on the DRAPartitionableDevices feature gate in the described environment.
The custom driver checks again during NodePrepareResources to guard against conflicting prepared claims. It does not rely on a last-minute heuristic to choose a non-overlapping combination after scheduling.
Why NAVER built its own DRA driver
Existing public DRA drivers did not match the deployment. NAVER's host GPUs were bound to vfio-pci rather than exposed through a conventional host NVIDIA driver and NVML discovery path. A reviewed NVIDIA DRA driver relied on NVML, while the dranet project was designed to move network interfaces into Pod network namespaces rather than pass through VFIO-bound NICs. NAVER also needed to coordinate GPUs with InfiniBand NICs sharing their PCIe switches, a topic expanded in part four.
Its custom node driver queries Fabric Manager for available partitions, publishes DRA devices and translates selected physical GPU identifiers into PCI addresses and VFIO device references. On claim preparation, it activates the partition and writes a claim-specific CDI specification. On release, it removes the CDI specification and deactivates the partition.
Reliability details matter: the driver does not publish partition devices if Fabric Manager is unavailable or returns an empty list; it refreshes partition and NIC discovery every sixty seconds. On restart, it reads existing claims to restore expected active partitions. It warns rather than deactivating a partition that may have been activated by another component. The source also warns that restarting Fabric Manager while workloads are active can terminate CUDA work and clear active partitions.
The Kata Containers cold-plug problem
Scheduling a GPU is not the same as making it appear inside a VM. NAVER found that upstream Kata's shim read device-plugin allocations from kubelet's PodResources API but did not consume the DRA DynamicResources entries needed for this workflow. Although containerd later supplied VFIO details through CDI, that information arrived after the VM's initial boot, too late for the required cold-plug attachment.
NAVER modified its Kata shim to read DRA allocations by claim and connect GPU and NIC devices in a deterministic order to virtual root ports before guest boot. That ordering also helps the team calculate guest PCI addresses for NCCL topology information in the subsequent part of the series. The engineers proposed the attachment work upstream as Kata Containers PR #13323 for the Go runtime and PR #13420 for runtime-rs. These are reported proposals; the source does not establish that the patches are merged and available in every stable Kata release.
Benchmarks: 2, 4 and 8 GPUs versus bare metal
After implementing allocation and VM attachment, NAVER ran four VMs concurrently on one eight-GPU B200 server: two one-GPU guests, one two-GPU guest and one four-GPU guest. It then measured NCCL all-reduce bus bandwidth for the two- and four-GPU partitions. An eight-GPU VM was tested separately. All comparisons used 8 GiB messages and the same GPU groupings in a conventional bare-metal container configuration.
| Configuration | Kata VM bus bandwidth | Bare-metal container | VM shortfall |
|---|---|---|---|
| 2 GPUs, with 1+1+2+4 VMs active | 623.5 GB/s | 629.7 GB/s | 0.98% |
| 4 GPUs, with 1+1+2+4 VMs active | 680.7 GB/s | 683.3 GB/s | 0.38% |
| 8 GPUs, one VM | 835.9 GB/s | 839.7 GB/s | 0.45% |
Source: NAVER D2, part three. These values are NCCL's reported all-reduce bus bandwidth, not end-to-end training throughput, application tokens per second or an independent measurement of all VM overheads. Percentages are calculated from the reported figures.
The 8 GiB all_reduce_perf output for the eight-GPU VM also reported 837.9 GB/s bus bandwidth at that message size, with no out-of-bounds values and a successful test. The 835.9 GB/s figure above comes from the separate comparative benchmark table; these should not be treated as contradictory measurements of an identical run.
What the numbers do—and do not—prove
These results show that, under the reported conditions, a correctly partitioned and attached GPU group can preserve almost all of its NVLink collective communication bandwidth inside a Kata VM. They do not prove that training jobs, model serving, cold starts or storage will universally run within one percent of bare metal. Those workloads involve CPU scheduling, PCIe and network topology, I/O, kernel-launch latency and VM startup costs beyond intra-node NVLink.
Indeed, part four of NAVER's series reports additional work needed for InfiniBand NIC pairing, GPUDirect RDMA, SPDK-backed NVMe storage and vCPU pinning, along with larger overheads in some latency-sensitive workloads. The two installments complement rather than replace each other.
What this means for AI infrastructure teams
The most important insight is that GPU scheduling is becoming topology-aware resource orchestration. A cloud platform may have eight physical GPUs, but the useful allocation units are the permitted fabric groups. A scheduler that understands only resource counts can waste capacity or launch a configuration that cannot initialize correctly. Claims, device attributes and shared counters allow administrators to represent the hardware's actual constraints.
For multi-tenant AI platforms, DRA also creates a cleaner separation between an administrator's hardware policy and a developer's workload specification. Developers can request a predefined GPU tier without understanding Fabric Manager partition IDs, PCI addresses or VFIO attachment. The complexity still exists, but it is centralized in a platform layer that can be tested and audited.
For teams building local AI servers with Ollama, the lesson is conceptual rather than a recipe to copy. Most small single-GPU installations do not need NVSwitch fabric partitions or custom DRA drivers. The principles—understand device topology, avoid double allocation, make resource ownership explicit and test real workload performance—remain relevant as systems grow into shared multi-GPU infrastructure.
A practical validation plan
- Discover actual Fabric Manager partitions on every node rather than assuming physical GPU numbering or universal groupings.
- Represent each allowed partition as a named resource with the correct size and identifiers.
- Prevent overlapping allocations with scheduler-visible constraints and confirm release behavior.
- Verify that DRA claims are consumed before VM boot and that VFIO devices appear in the guest.
- Check partition recovery and failure behavior after driver restart without interrupting active jobs.
- Run NCCL tests for each supported partition size and compare identical message sizes and GPU groupings.
- Benchmark real inference and training workloads separately from collective communication microbenchmarks.
- Track custom runtime patches and their upstream status before treating the implementation as portable.
Frequently asked questions
What is a Fabric Manager GPU partition?
It is an allowed group of physical GPUs that NVIDIA Fabric Manager configures for NVLink communication through NVSwitch. It is not the same as dividing a GPU into MIG slices.
Why was a Kubernetes device plugin insufficient?
The standard count-based allocation did not guarantee a permitted NVLink grouping or reliably express mutual exclusion between overlapping partitions.
What does DRA add?
It allows the scheduler to select named devices based on attributes and constraints, with a claim lifecycle for preparing and releasing resources.
What are SharedCounters used for?
They prevent two partition devices that include the same physical GPU from being allocated at the same time.
Did Kata match bare metal?
For the reported intra-node NCCL all-reduce bus-bandwidth tests, the VM measurements were within one percent of bare-metal containers. That does not generalize to every AI workload.
Is NAVER's driver available as an ordinary upstream configuration?
The article describes a custom driver and internal Kata shim modifications. It references upstream pull requests, but it does not establish universal out-of-the-box support.
Conclusion
NAVER's third engineering installment illustrates why GPU virtualization is as much a scheduling problem as a hardware passthrough problem. By treating Fabric Manager's permitted NVLink groups as DRA devices and enforcing non-overlap with SharedCounters, the team made it possible to share a B200 node among multiple isolated Kata VMs while retaining near-bare-metal collective bandwidth in the tested configurations. The result is compelling evidence for topology-aware scheduling—not a promise that virtualization has no cost.
Primary source: NAVER D2 — “컨테이너처럼 쓰고, VM처럼 격리하기 3편 - Fabric Manager 파티션과 DRA를 이용한 GPU 할당”. See also part four on InfiniBand, NVMe, vCPU placement and bare-metal performance. The detailed Korean text of part three was provided as source material; the webpage could not be independently accessed during preparation. Barry Schwartz: NOT CHECKED.