AI Compute Infrastructure: GPUs, Networking and the Build vs Rent Decision
The GPUs get the headlines, and the invoices. But an AI build succeeds or fails on the whole stack being in balance: compute, network and storage, designed together. Get any one of them wrong and the most expensive hardware you own sits idle. This is the honest guide to the compute and network side of an AI build, from people who spent years selling and architecting this infrastructure from the vendor side.
Ask a vendor what you need to build AI and the answer is a pile of GPUs. They are the glamorous part and the biggest line on the quote, so the whole conversation gravitates to them. That is a mistake. GPUs only earn their cost when they are busy, and whether they stay busy is decided far more by the network feeding them and the storage behind that than by the badge on the accelerator. A cluster is a system, and it runs at the speed of its weakest link. This guide walks the compute and the network honestly, and then the decision that sits on top of both: whether to build it yourself or rent it.
You are not buying GPUs, you are buying utilised GPUs. An idle accelerator is the most expensive thing in the building, and accelerators go idle when the network cannot move data between them fast enough or the storage cannot feed them. Every decision here comes back to one question: will this keep the GPUs working, at this scale, without paying for capability the workload will never use.
The GPUs: the point, and the risk
The GPU is the compute engine, and it is genuinely where the money goes, whether you own the cards or rent them by the hour. The trap is buying for the wrong job. Training a large model from scratch, fine tuning an existing one, and serving inference in production are three different workloads with three different appetites. Training wants many GPUs working in concert and is hungry for interconnect. Fine tuning is far more modest and often runs on a handful of cards. Inference is about latency and cost per query, not raw cluster scale. Buy a frontier training cluster to serve a modest inference workload and you have spent spectacularly on capability you will never light up. Size the compute to the job you actually have, and to the one you can realistically see, not to the slide.
The network: the part people forget
This is where most first builds go wrong. For a single GPU or a small node the network barely matters. The moment you train across many GPUs, they have to exchange enormous amounts of data with each other constantly, and the fabric that carries that east west traffic becomes as important as the cards themselves. A fast set of GPUs on a slow interconnect is a slow cluster, because they spend their time waiting for each other rather than computing. You have paid for accelerators and bought yourself idle time.
The practical choice is between InfiniBand and high performance Ethernet running RoCE, which is remote direct memory access over Ethernet. InfiniBand has been the traditional home of large scale training, with very low, very predictable latency, around a microsecond for small messages. It commanded most of the AI cluster market as recently as 2023. The trade is cost and specialism: it runs meaningfully higher per port and needs dedicated skills that most enterprise network teams do not already have. RoCE closes much of the gap. Properly engineered it delivers most of InfiniBand's training throughput, in the region of 85 to 95 percent for clusters of a few hundred to around a thousand GPUs, on Ethernet your team already knows how to run. By 2025 Ethernet had become the more common choice for new back end AI networks, and hyperscalers proved RoCE at genuine scale, Meta among them training on tens of thousands of GPUs over standard Ethernet.
For frontier scale training where every microsecond of latency compounds, InfiniBand still has the edge and often justifies its premium. For the large majority of enterprise AI, at hundreds of GPUs rather than tens of thousands, well engineered RoCE Ethernet gets you most of the performance on infrastructure your team can actually operate, and that operational reality is worth more than a benchmark. The wrong answer is to ignore the fabric entirely and discover the bottleneck after the GPUs arrive.
Power, cooling and density: the constraint on paper plans
Here is the reality that stops more AI builds than budget does. A traditional enterprise rack was provisioned for something like 6 to 15 kilowatts. A modern dense GPU rack can draw far more: an H100 generation rack sits around 40 kilowatts, current designs exceed 80, and the newest go beyond 100, with some approaching 140 kilowatts in a single rack. Most existing data centre halls simply cannot deliver that power or remove that heat. Above roughly 50 kilowatts a rack, air cooling runs out of road and liquid cooling stops being optional. That means direct to chip cooling or rear door heat exchangers, and facilities work that has a lead time measured in months, not days. If you are planning an on premises build, the honest first question is often not which GPU, it is whether the building can power and cool it at all.
Build vs rent: own, cloud, or specialist provider
Once you know what you need, you have three broad ways to get it, and none is right for everyone. Owning it on premises means you buy the GPUs, the fabric and the facilities, and you get the lowest cost per hour once the kit is running hard, plus full control and data locality. The catch is a large capital outlay, the power and cooling problem above, and the risk of owning depreciating hardware if utilisation disappoints. Renting from a hyperscaler cloud gives you speed and elasticity, nothing to build and the ability to stop, but the per hour cost is high and it stacks up fast under sustained load, and you inherit egress and lock in considerations. Specialist GPU providers, the newer wave of dedicated AI clouds, sit in between, often cheaper per GPU hour than the hyperscalers for raw training capacity, with their own trade offs on ecosystem, support and commitment terms.
Rent when demand is spiky, early or uncertain, and you value speed over unit cost. Own when utilisation is high and steady and the maths on cost per hour turns in your favour, and you can actually house the kit. Many serious builds end up hybrid: rent to start and to burst, own the steady baseline. The mistake is defaulting to one because it is familiar rather than because the workload profile points there.
The stack has to be balanced
The theme running through all of this is balance. A GPU estate is only as fast as the network moving data between the accelerators and the storage feeding data into them. Overspend on GPUs and starve the fabric, and you have bought idle time. Build a fast fabric and put the data on storage that cannot keep up, and you have moved the bottleneck rather than removed it. Compute, network and storage are one design decision, not three purchases. The storage half of this is a discipline of its own, and our guide on the right storage platform for AI works through it, with a separate guide on on premises AI storage for regulated data where the data cannot leave. Design the three together or one of them will quietly throttle the other two.
How C4C helps
This is exactly where our data centre and AI experience meet, and it is a decision that punishes getting the balance wrong. We spent years on the vendor side, architecting and selling this infrastructure, so we know how the compute, the fabric and the facilities really behave under AI load rather than how the datasheet reads, and where a quote is sized to sell rather than to fit. We will profile your actual workload, training, fine tuning or inference, size the GPUs, the network and the storage to keep the accelerators busy without overpaying, pressure test whether your building can power and cool the plan, and work the build versus rent maths on your real utilisation rather than a vendor's assumption. Independent, with no hardware line to defend and no quota to hit. If the ambition is still forming, the honest first question is often the data, which is where our data readiness guide starts.
Planning an AI build?
Tell us what the AI needs to do, the scale you are planning, and whether you are leaning towards owning or renting. We will give you an evidence based view of the compute, the network and the storage that actually fit, sized to keep your GPUs busy without paying for capability you will never use, and an honest read on whether to build it or rent it. Independent, with no hardware to sell. We have architected and sold this infrastructure from the inside.
Prefer email? Reach us directly at hello@c4cgroup.co.uk.
Frequently asked questions
Should we build our own AI infrastructure or rent GPU cloud?
It depends on your utilisation and your certainty. Rent when demand is spiky, early or uncertain and you value speed and the ability to stop over unit cost. Own when utilisation is high and steady, the maths on cost per hour turns in your favour, and your building can actually power and cool the hardware. Many serious builds end up hybrid, renting to start and to burst while owning the steady baseline. The mistake is defaulting to whichever is familiar rather than the one your real workload profile points to.
How many GPUs do we actually need?
Size to the workload, not to the slide. Training a large model from scratch wants many GPUs working together, fine tuning an existing model often runs on a handful, and inference is about latency and cost per query rather than cluster scale. The most common and expensive error is buying a frontier training cluster to serve a modest inference job, which lights up a fraction of what you paid for. Profile the job you actually have and the one you can realistically see, then size to that.
Does the network really matter for AI, or is it all about the GPUs?
It matters enormously the moment you train across many GPUs. They have to exchange large amounts of data with each other constantly, so a fast set of accelerators on a slow interconnect becomes a slow cluster, because they spend their time waiting rather than computing. For a single GPU the network barely registers, but for multi GPU training the fabric is as important as the cards. Ignore it and you discover the bottleneck after the expensive hardware has already arrived.
InfiniBand or Ethernet for an AI cluster?
InfiniBand offers the lowest and most predictable latency and still has the edge for frontier scale training, but it costs more per port and needs specialist skills most enterprise teams do not already have. High performance Ethernet running RoCE closes much of the gap, delivering roughly 85 to 95 percent of InfiniBand's training throughput for clusters of a few hundred to around a thousand GPUs, on networking your team already knows how to run. For most enterprise AI, well engineered RoCE Ethernet is the pragmatic choice; for the largest training clusters, InfiniBand often still justifies its premium.
Why do expensive GPUs end up sitting idle?
Because something upstream cannot keep them fed. GPUs only earn their cost when they are busy, and they go idle when the network cannot move data between them fast enough or the storage cannot deliver data quickly enough, or when checkpoints stall a training run. It is almost always a balance failure: money went on accelerators while the fabric or the storage was undersized. The fix is to design compute, network and storage together so no single link throttles the rest.
Can our existing data centre handle AI hardware?
Often not without work, and this stops more builds than budget does. A traditional rack was provisioned for around 6 to 15 kilowatts, while a dense GPU rack can draw 40, 80, or beyond 100 kilowatts. Above roughly 50 kilowatts a rack, air cooling runs out and liquid cooling becomes mandatory, which means direct to chip or rear door cooling and facilities work with a lead time in months. Before choosing a GPU, the honest first question is whether the building can power and cool it at all.