How workload flexibility can improve access to AI capacity

AI infrastructure capacity depends on available supply and the constraints attached to each workload. A request that fixes execution time, machine configuration, zone and accelerator allocation can use only capacity that satisfies all four conditions. Google Cloud recommends relaxing some conditions where the workload permits. Google Cloud sells the cloud infrastructure and services described here, so it has a commercial interest in customers adopting these mechanisms.

AI capacity depends on workload constraints

Each workload has requirements and preferences. An online service may need immediate execution because customers are waiting, while an offline fine-tuning job may tolerate a delayed start. A particular VM generation or zone may be a preference rather than a requirement for correct execution. Platform teams can expose acceptable alternatives to infrastructure software while keeping genuine constraints fixed.

This distinction changes the planning question. Teams need to determine which execution windows, machine families, sizes, zones and accelerator allocations affect correctness, performance or business deadlines. They can then rank acceptable alternatives across the remaining dimensions. Google Cloud characterizes this approach as a way to address overprovisioning, stranded capacity and bottlenecks when AI systems and conventional applications share infrastructure.

The same reasoning applies when provisioning fails. More accelerator capacity may still be required, especially when workload constraints cannot move. In other cases, widening the approved execution window, hardware set, location or accelerator allocation gives infrastructure more ways to satisfy demand. The architecture decision is which constraints must remain fixed and which can become controlled variables.

Four variables can broaden available capacity

Google Cloud groups those choices around time, hardware configuration, location and accelerator quantity. Each variable gives a scheduler more acceptable outcomes when an application can tolerate the change.

Time is the first choice. Google Cloud’s Dynamic Workload Scheduler is designed to provide access to scarce resources including GPUs, TPUs and some CPU-based virtual machine families. Its calendar mode targets work requiring resources at a fixed time for a defined period. Flex-start lets latency-tolerant work wait for available capacity; Google Cloud gives batch processing, model training and offline fine-tuning as examples.

For platform teams, the distinction is operational. A job with a fixed business deadline may still have freedom over its exact start time. Encoding that freedom gives the provider a wider window to satisfy the request. Teams therefore need to classify which jobs may wait and how much delay their deadlines permit.

Hardware configuration is the second choice. Google Cloud recommends instance flexibility in managed instance groups and bulk virtual machine creation for Compute Engine users. Operators can provide several compatible options, allowing the system to choose an approved configuration based on real-time availability. The architecture task becomes defining which machine configurations can run the workload correctly.

That policy can also accommodate hardware refresh. A ranked list can prioritize newer virtual machine generations while retaining older generations as fallbacks. This works only where those older generations continue to meet the application’s requirements.

Location is the third choice. Google Cloud recommends using multiple zones within a region where a workload can tolerate that placement. Approving several zones lets provisioning systems consider capacity across a larger placement set. A single-zone requirement remains appropriate when the application genuinely depends on it.

Google Kubernetes Engine custom ComputeClasses combine hardware and location choices. Google Cloud says platform teams can define acceptable combinations of machine families, sizes, performance ratios, pricing models and zones, then rank them by priority. If the preferred node configuration is unavailable, Google Kubernetes Engine can proceed to another approved option. The platform can then select among configurations that application and infrastructure teams have already accepted.

Fallback also needs a recovery policy. Google Cloud’s active migration moves workloads toward higher-priority configurations when those configurations become available, providing one such mechanism in Google Kubernetes Engine. A temporary fallback can then remain temporary when preferred capacity returns. Teams still need to decide which workloads should move and which configurations qualify as preferred.

Accelerator quantity is the fourth choice. Google Cloud highlights dynamic resource allocation in Google Kubernetes Engine as a way for applications to request part of a GPU or TPU, including requirements expressed through memory or core counts. The platform then assigns a matching portion of the hardware. Where isolation and performance requirements permit sharing, this can align accelerator allocation more closely with the application’s resource requirement.

Google Cloud presents this mechanism as a response to accelerators being underused when allocated in large blocks. Its business effect depends on each organization’s workload shapes, performance requirements and sharing constraints. Executives can assess it by measuring accelerator utilization and workload performance in their own estate. Those measurements can show whether finer allocation changes the economics of a particular workload.

Other dependencies can constrain the same workload. Google Cloud says Compute Engine can use Spot virtual machine capacity signals and favor machine types expected to provide longer uptime with lower pre-emption risk; pre-emption means the provider can reclaim that Spot capacity. For storage, Google Cloud recommends baseline disk defaults for many short-lived instances and Hyperdisk when data disks need to survive associated virtual machines across machine generations. These recommendations extend substitution beyond the initial compute choice.

Guaranteed capacity serves predictable demand

Some demand patterns justify securing resources in advance. Google Cloud explicitly recommends doing so for predictable demand such as product launches, migrations, offline training runs and seasonal traffic peaks. Dynamic Workload Scheduler’s calendar mode addresses fixed-time, defined-duration requirements. Google Cloud also advises customers to consider committed spending models that offer discounts while preserving flexibility across machine types and locations.

The decision depends on predictability and workload tolerance. A migration that must run during a scheduled window can justify advance capacity planning. An unexpected surge caused by a news event or market change calls for a different response because its timing is uncertain. Predefined fallback configurations and broader placement choices give infrastructure more approved options in that case.

Delay-tolerant workloads create another operating pattern. Batch processing or an offline training run with a flexible start time can use flex-start. Spot virtual machines may also be candidates where applications can withstand pre-emption risk. The workload owner therefore needs to specify acceptable uncertainty across start time, continuity and hardware configuration.

Reservations and fallback policies address distinct risks and can coexist in one platform. Reservations secure capacity for predictable requirements. Fallback policies broaden the resources that can satisfy a workload when preferred capacity is unavailable. Different applications, or different phases of one application, can require different combinations of the two.

The operating model is built around acceptable outcomes

This approach requires policies that define acceptable execution outcomes. Platform teams can rank machine families and sizes, approve zones, identify workloads that can queue, define where accelerator sharing is safe and specify acceptable pricing models. These decisions determine how much choice infrastructure software has when preferred capacity is unavailable. They also move key decisions out of the incident window and into architecture and platform policy.

Custom ComputeClasses illustrate that operating model. A platform team can encode combinations of machine family, size, performance ratio, pricing model and zone centrally. Application teams can then consume policies whose fallback choices have already been approved. Scarcity triggers predefined substitution instead of forcing teams to invent alternatives under operational pressure.

Application architecture sets the boundary of that freedom. A workload coupled to a specific machine generation gives the scheduler fewer acceptable choices. The same applies when an application requires one zone or an entire accelerator. Platform and application teams therefore need to identify such coupling and decide whether changing it justifies the engineering effort.

Some constraints will remain fixed because correctness, performance or business deadlines require them. Workloads tied to a particular machine family, machine type, zone or execution window have fewer substitution options. Applications that cannot tolerate accelerator sharing or Spot pre-emption also narrow the available choices. In those cases, capacity planning has to respect the fixed requirement and secure resources around it.

The business decision can be tested workload by workload. Teams can identify the constraints required for correctness, performance and deadlines, encode approved alternatives, and measure utilization, provisioning behavior and workload performance in their own environment. This turns flexibility into an explicit engineering policy with observable results and gives executives a concrete basis for deciding where additional supply remains necessary.

Key highlights

• AI capacity depends on workload constraints: Leaders should distinguish fixed requirements from preferences across execution time, hardware, location and accelerator allocation. Relaxing nonessential constraints can give infrastructure more ways to satisfy demand.
• Four variables can broaden available capacity: Platform teams should define approved flexibility across time, hardware configuration, location and accelerator quantity. Google Cloud mechanisms can then use those alternatives when preferred capacity is unavailable.
• Guaranteed capacity serves predictable demand: Reserve capacity for workloads with known timing and fixed requirements, while using fallback policies for less predictable demand. The two approaches address different capacity risks and can be combined.
• The operating model is built around acceptable outcomes: Encode fallback choices before capacity becomes scarce, including approved hardware, zones, scheduling delays and sharing policies. Measure provisioning, utilization and workload performance to determine where flexibility works and where additional supply remains necessary.