Behind every AI inference
is a network
Training built the case for GPUs. Inference is building the case for the network moving data between them, the cloud, and everyone waiting on an answer.
For years, the AI infrastructure conversation centered on one question: where will we find enough compute? GPUs, power, cooling, data center capacity.
Those questions haven't gone away. But as AI moves from training models to putting them to work for millions of users through inference, a second question is moving to the forefront: where will the data move, and how fast can it get there?
Inference doesn't end when a model is built. It happens every time someone uses it, an answer, an image, a recommendation, an autonomous agent completing a task. And every one of those requests has to travel: from the model, across a network, to the person or system waiting on the other end. As inference scales, AI becomes not only a compute challenge, but a data movement challenge.
The data behind the shift
Hover a stat for its sourceTraining and inference ask different things of infrastructure
Training concentrates compute. Inference has to reach people, applications and other AI systems, often in real time. That difference is reshaping how AI infrastructure gets planned.
Training
- Concentrates massive compute into GPU clusters
- Runs over days, weeks, or months
- Can often tolerate geographic distance
- Priority: raw processing power working together
Inference
- Interacts with users, apps, and other AI in real time
- Runs continuously, every time a model is used
- Latency, route diversity, and location matter directly
- Priority: capacity, diversity, and network optionality
AI infrastructure is no longer one massive campus. It's becoming distributed.
Instead of concentrating everything in centralized compute campuses, the industry's next phase connects infrastructure across many locations, closer to where inference demand actually originates.
AI grids
NVIDIA and telecom operators including AT&T, Spectrum, and Akamai are building "AI grids": geographically distributed, interconnected AI infrastructure that runs inference closer to users across existing network footprints.
Read more →Campus-as-a-computer
Google is decoupling its data center network into distinct domains under a "campus as a computer" philosophy, designed for the scale, low latency, and predictability inference workloads demand.
Read more →Net-new network buildout
Dell'Oro Group projects front-end data center networking opportunities will grow at nearly 40% CAGR, driven in large part by the shift from training toward inference and agentic workloads.
Read more →Power determines whether GPU infrastructure can operate. Cooling determines how densely it can operate. Networking determines where that intelligence can go.THE ROLE OF NETWORKING IN THE NEXT GENERATION OF AI INFRASTRUCTURE
Power
Determines whether GPU infrastructure can operate at all.
Cooling
Determines how densely that infrastructure can run.
Networking
Determines where the intelligence built on top of it can actually go.
Compute and connectivity, meeting at one point on the network
A model may be trained in one location. The data behind it may sit somewhere else. The enterprise deploying it may operate across regions. And the user asking the question could be thousands of miles away. Every inference request is a journey between those points.
NJFX's carrier-neutral Cable Landing Station campus in Wall Township, New Jersey brings subsea systems, terrestrial fiber, network operators, and data center infrastructure together at a strategically positioned point on the U.S. East Coast, giving that journey more options for how it happens.
Through SecureWAVES, customers and network operators are adding and upgrading equipment to move terabits of data across terrestrial and subsea networks, building the capacity behind the next generation of AI-driven data movement.
The AI infrastructure race started with compute. It won't end there. More GPUs require more power. Higher densities require more advanced cooling. And more inference will require bigger, faster, more resilient network pipes.
Sources & further reading
- Cisco AI Impact on Wide Area Networks 2026 — token growth, flow duration, and upstream traffic data cisco.com →
- Cisco Fiscal 2026 Q4 results — $9.3B in hyperscaler AI infrastructure orders marketbusinessnews.com →
- Dell'Oro Group Front-end networking market forecast — inference and agentic AI driving ~40% CAGR prnewswire.com →
- NVIDIA AI grids — distributed, interconnected AI infrastructure announced at GTC 2026 blogs.nvidia.com →
- Google Cloud "Campus as a computer" — data center and network architecture built for AI cloud.google.com →
