Voir en français
Menu

AI

Scalable GPU infrastructure for running generative AI models

The context

A generative AI SaaS vendor lets its users run specialised models trained for their domain. The problem with this kind of product is economic before it is technical: GPUs are expensive, demand is very uneven, and infrastructure sized for the peak is financially untenable when it idles overnight.

What was needed was to absorb bursts of inference, genuinely scale back down when demand falls away, stay predictable on latency — and make the resulting mass of output usable, since it runs into millions of images and is worth nothing if it cannot be found again.

What we did

We built the execution platform and the control plane that sizes it.

Ten-second control loop driving several GPU Auto Scaling groupsA control loop runs every ten seconds. It reads business and application metrics, computes the GPU capacity actually required, adjusts each Auto Scaling group accordingly and maintains the set of pre-warmed models, then starts again. Capacity is spread over several groups: Spot in more than one GPU type and availability zone, with an on-demand group as fallback.Loop every 10 secondsAuto Scaling groupsBusiness and application metricsRequired GPU capacity computedCapacity groups adjustedPre-warmed models maintained10 sSpot, GPU type A — zone 1Spot, GPU type B — zone 2On-demand — fallback capacity… further zones and types
Managed target tracking reacts in minutes, to infrastructure metrics. The capacity actually needed is a function of application metrics, so the loop computes it directly — and spreads demand over several groups, because Spot capacity is not interchangeable between GPU types or zones.
  • A request intake that is not sized like the GPUs. Inference requests arrive through API Gateway and are queued by Lambda, so accepting work and doing work scale independently. A burst that arrives faster than the fleet can grow gets absorbed by the queue instead of being refused at the door.
  • Spot GPU capacity first. It is the only lever that changes the order of magnitude of the cost on this kind of workload. The price is that capacity can be reclaimed at any moment: interruption is handled as a normal lifecycle event, not as a failure.
  • Several Auto Scaling groups rather than one. Spot capacity is not fungible: a GPU type being unavailable in one zone says nothing about its availability elsewhere. Spreading demand over several groups, by GPU type and by zone, with an on-demand group as fallback, is what makes Spot sustainable in production rather than merely opportunistic.
  • Sizing recomputed every 10 seconds. Managed scaling thinks in minutes and in infrastructure metrics; the GPU capacity actually required follows from business and application metrics — queued requests, throughput per model, the shape of the work in flight. So the loop computes the required capacity directly, group by group, and adjusts.
  • Pre-warmed model management. The same loop decides how many instances of each model to keep ready to serve, based on observed demand. That is what avoids paying model load time on a user’s request, without keeping the whole catalogue resident.
  • Indexing inference results at scale, spread across S3 for the artefacts, DynamoDB for the metadata and OpenSearch for the index, with automatic enrichment of the generated content through Rekognition. Indexing sits on the inference path, not in an overnight batch.
  • KNN search over millions of images, on vectors produced by embedding models served from SageMaker and indexed in OpenSearch.
From inference request to KNN search over millions of imagesRequests arrive through API Gateway, are queued by Lambda, and are dispatched to GPU workers running in parallel, each holding a pre-warmed model, and how many of them exist is decided by the sizing loop. Inference results are indexed at scale across S3, DynamoDB and OpenSearch, which then serves KNN search over millions of images, returned to users.Inference and indexing chainRequest intake — API Gateway, queued by LambdaGPU worker 1pre-warmed modelGPU worker 2pre-warmed modelGPU worker 3pre-warmed model…as load requiresResults indexed — S3, DynamoDB, OpenSearchKNN search across millions of imagesReturned to users
Indexing is part of the inference path, not a batch job bolted on afterwards: results become searchable as they are produced, which is what makes KNN search over millions of images answer in a usable time.

The outcome

The vendor opens the product to more users without the bill following the same curve, and a reclaimed Spot instance no longer shows up in the service. The accumulated mass of results has become a searchable asset instead of a file store.

The main lesson concerns the sizing loop. We started with managed scaling on infrastructure metrics: it always arrives too late, because GPU utilisation describes what has already happened. The requirement, by contrast, is known in advance — it is written in the work queue. A ten-second loop that reads it beats a managed policy that ignores it.

The second lesson is less flattering: Spot buys nothing until it is committed to fully. A single Spot Auto Scaling group is a production incident waiting for its moment.

All case studies

Talk to us directly

Write to us directly. No salesperson, no qualification call. One of us two replies, within 24 hours.

contact@onescale.io

We reply within 24 hours, and it is one of us who replies.

Based in Lyon, working with clients in France, across Europe and internationally.