Thanks to the artificial intelligence boom, new data centers are springing up as quickly as companies can build them. This has translated into huge demand for power to run and cool the servers inside. Now concerns are mounting about whether the U.S. can generate enough electricity for the widespread adoption of AI, and whether our aging grid will be able to handle the load.

“If we don’t start thinking about this power problem differently now, we’re never going to see this dream we have,” said Dipti Vachani, head of automotive at Arm. The chip company’s low-power processors have become increasingly popular with hyperscalers like Google, Microsoft, Oracle and Amazon— precisely because they can reduce power use by up to 15% in data centers.

Nvidia’s latest AI chip, Grace Blackwell, incorporates Arm-based CPUs it says can run generative AI models on 25 times less power than the previous generation.

“Saving every last bit of power is going to be a fundamentally different design than when you’re trying to maximize the performance,” Vachani said.

This strategy of reducing power use by improving compute efficiency, often referred to as “more work per watt,” is one answer to the AI ​​energy crisis. But it’s not nearly enough.

One ChatGPT query uses nearly 10 times as much energy as a typical Google search, according to a report by Goldman Sachs. Generating an AI image can use as much power as charging your smartphone. 

This problem isn’t new. Estimates in 2019 found training one large language model produced as much CO2 as the entire lifetime of five gas-powered cars. 

And in Kansas City, where Meta is building an AI-focused data center, power needs are so high that plans to close a coal-fired power plant are being put on hold.

Chasing Power

Hardening The Grid

Cooling Servers Down

______

NJFX Future Ready Infrastructure

NJFX has strategically developed an infrastructure capable of supporting AI customers and high-density applications, positioning itself as a leader in the industry. With a 5MW data center designed specifically for high-density AI compute requirements, NJFX offers advanced liquid cooling solutions that can be implemented upon request, optimizing both power efficiency and environmental sustainability. This robust infrastructure not only meets the demands of AI workloads but also ensures seamless connectivity, creating an unrivaled hub for edge AI inference.

In line with this commitment to AI innovation, NJFX is hosting a significant AI event in October, in collaboration with Bulk Infrastructure, EXA Infrastructure, and Supermicro. This event will showcase solutions tailored for AI applications in the enterprise and financial markets, further solidifying NJFX’s role as a critical player in the AI ecosystem. Through these efforts, NJFX continues to demonstrate its dedication to providing the infrastructure and expertise needed to drive the future of AI.

More specifically, the trained neural network is put to work out in the digital world using what it has learned — to recognize images, spoken words, a blood disease, predict the next word or phrase in a sentence, or suggest the shoes someone is likely to buy next, you name it — in the streamlined form of an application. This speedier and more efficient version of a neural network infers things about new data it’s presented with based on its training. In the AI lexicon this is known as “inference.”

So let’s break down the progression from AI training to AI inference, and how they both function.

Training Deep Neural Network

While the goal is the same – knowledge — the educational process, or training, of a neural network is (thankfully) not quite like our own. Neural networks are loosely modeled on the biology of our brains — all those interconnections between the neurons. Unlike our brains, where any neuron can connect to any other neuron within a certain physical distance, artificial neural networks have separate layers, connections, and directions of data propagation.

When training a neural network, training data is put into the first layer of the network, and individual neurons assign a weighting to the input — how correct or incorrect it is — based on the task being performed.

To learn more, check out NVIDIA’s AI inference solutions for the data center, self-driving cars, video analytics and more.

In an image recognition network, the first layer might look for edges. The next might look for how these edges form shapes — rectangles or circles. The third might look for particular features — such as shiny eyes and button noses. Each layer passes the image to the next, until the final layer and the final output determined by the total of all those weightings is produced.

But here’s where the training differs from our own. Let’s say the task was to identify images of cats. The neural network gets all these training images, does its weightings and comes to a conclusion of cat or not. What it gets in response from the training algorithm is only “right” or “wrong.”

Deep Learning Training Is Compute Intensive

And if the algorithm informs the neural network that it was wrong, it doesn’t get informed what the right answer is. The error is propagated back through the network’s layers and it has to guess at something else. In each attempt it must consider other attributes — in our example attributes of “catness” — and weigh the attributes examined at each layer higher or lower. Then it guesses again. And again. And again. Until it has the correct weightings and gets the correct answer practically every time. It’s a cat.

Now you have a data structure and all the weights in there have been balanced based on what it has learned as you sent the training data through. It’s a finely tuned thing of beauty. The problem is, it’s also a monster when it comes to consuming compute. For example, GPT-3 with 175 billion parameters requires roughly 300 zettaflops, which is 300,000 billion billion math operations across the entire training cycle. Try getting that to run on a smartphone.

That’s where inference comes in.

Congratulations! Your Neural Network Is Trained and Ready for Inference

What you had to put in place to get your properly weighted neural network to learn — in our education analogy all those pencils, books, teacher’s dirty looks — is now way more than you need to get any specific task accomplished.

If anyone is going to make use of all that training in the real world, and that’s the whole point, what you need is a speedy application that can retain the learning and apply it quickly to data it’s never seen. That’s inference: taking smaller batches of real-world data and quickly coming back with the same correct answer (really a prediction that something is correct).

There are two main approaches to taking that hulking neural network and modifying it for speed and improved latency in applications that run across other networks.

How AI Inferencing Works

The first approach looks at parts of the neural network that don’t get activated after it’s trained. These sections just aren’t needed and can be “pruned” away. The second approach looks for ways to fuse multiple layers of the neural network into a single computational step.

It’s akin to the compression that happens to a digital image. Designers might work on these huge, beautiful, million pixel-wide and tall images, but when they go to put it online, they’ll turn into a jpeg. It’ll be almost exactly the same, indistinguishable to the human eye, but at a smaller resolution. Similarly with inference you’ll get almost the same accuracy of the prediction, but simplified, compressed and optimized for runtime performance.

What that means is we all use inference all the time. Your smartphone’s voice-activated assistant uses inference, as do image search and spam filtering applications. Facebook’s image recognition and Amazon’s and Netflix’s recommendation engines all rely on inference.

GPUs, thanks to their parallel computing capabilities — or ability to do many things at once — are good at both training and inference.

Systems trained with GPUs allow computers to identify patterns and objects as well as — or in some cases, better than — humans (see “Accelerating AI with GPUs: A New Computing Model”).

After training is completed, the networks are deployed into the field for “inference” — classifying data to “infer” a result. Here too, GPUs — and their parallel computing capabilities — offer benefits, where they run billions of computations based on the trained network to identify known patterns or objects.

The parallel computing of GPUs also provides multi-factor speedups in traditional machine learning, using algorithms like gradient-boosted decision trees, for both training and inference.

You can see how these models and applications will just get smarter, faster and more accurate. Inference will bring new applications to every aspect of our lives. It seems the same admonition applies to AI as it does to our youth — don’t be a fool, stay in school. Inference awaits.

Countless analysts and businesses are talking about and implementing edge computing, which traces its origins to the 1990s, when content delivery networks were created to serve web and video content from edge servers deployed close to users.

Today, almost every business has job functions that can benefit from the adoption of edge AI. In fact, edge applications are driving the next wave of AI computing in ways that improve our lives at home, at work, in school and in transit.

Learn more about what edge AI is, its benefits and how it works, examples of edge AI use cases, and the relationship between edge computing and cloud computing.

What Is Edge AI? 

Edge AI is the deployment of AI applications in devices throughout the physical world. It’s called “edge AI” because the AI computation is done near the user at the edge of the network, close to where the data is located, rather than centrally in a cloud computing facility or private data center.

Since the internet has global reach, the edge of the network can connote any location. It can be a retail store, factory, hospital or devices all around us, like traffic lights, autonomous machines and phones.

Edge AI: Why Now? 

Organizations from every industry are looking to increase automation to improve processes, efficiency and safety.

To help them, computer programs need to recognize patterns and execute tasks repeatedly and safely. But the world is unstructured and the range of tasks that humans perform covers infinite circumstances that are impossible to fully describe in programs and rules.

Advances in edge AI have opened opportunities for machines and devices, wherever they may be, to operate with the “intelligence” of human cognition. AI-enabled smart applications learn to perform similar tasks under different circumstances, much like real life.

The efficacy of deploying AI models at the edge arises from three recent innovations.

1. Maturation of neural networks: Neural networks and related AI infrastructure have finally developed to the point of allowing for generalized machine learning. Organizations are learning how to successfully train AI models and deploy them in production at the edge.
2. Advances in compute infrastructure: Powerful distributed computational power is required to run AI at the edge. Recent advances in highly parallel GPUs have been adapted to execute neural networks.
3. Adoption of IoT devices: The widespread adoption of the Internet of Things has fueled the explosion of big data. With the sudden ability to collect data in every aspect of a business — from industrial sensors, smart cameras, robots and more — we now have the data and devices necessary to deploy AI models at the edge. Moreover, 5G is providing IoT a boost with faster, more stable and secure connectivity.

Why Deploy AI at the Edge? What Are the Benefits of Edge AI? 

Since AI algorithms are capable of understanding language, sights, sounds, smells, temperature, faces and other analog forms of unstructured information, they’re particularly useful in places occupied by end users with real-world problems. These AI applications would be impractical or even impossible to deploy in a centralized cloud or enterprise data center due to issues related to latency, bandwidth and privacy.

The benefits of edge AI include:

• Intelligence: AI applications are more powerful and flexible than conventional applications that can respond only to inputs that the programmer had anticipated. In contrast, an AI neural network is not trained how to answer a specific question, but rather how to answer a particular type of question, even if the question itself is new. Without AI, applications couldn’t possibly process infinitely diverse inputs like texts, spoken words or video.
• Real-time insights: Since edge technology analyzes data locally rather than in a faraway cloud delayed by long-distance communications, it responds to users’ needs in real time.
• Reduced cost: By bringing processing power closer to the edge, applications need less internet bandwidth, greatly reducing networking costs.
• Increased privacy: AI can analyze real-world information without ever exposing it to a human being, greatly increasing privacy for anyone whose appearance, voice, medical image or any other personal information needs to be analyzed. Edge AI further enhances privacy by containing that data locally, uploading only the analysis and insights to the cloud. Even if some of the data is uploaded for training purposes, it can be anonymized to protect user identities. By preserving privacy, edge AI simplifies the challenges associated with data regulatory compliance.
• High availability: Decentralization and offline capabilities make edge AI more robust since internet access is not required for processing data. This results in higher availability and reliability for mission-critical, production-grade AI applications.
• Persistent improvement: AI models grow increasingly accurate as they train on more data. When an edge AI application confronts data that it cannot accurately or confidently process, it typically uploads it so that the AI can retrain and learn from it. So the longer a model is in production at the edge, the more accurate the model will be.

How Does Edge AI Technology Work?

For machines to see, perform object detection, drive cars, understand speech, speak, walk or otherwise emulate human skills, they need to functionally replicate human intelligence.

AI employs a data structure called a deep neural network to replicate human cognition. These DNNs are trained to answer specific types of questions by being shown many examples of that type of question along with correct answers.

This training process, known as “deep learning,” often runs in a data center or the cloud due to the vast amount of data required to train an accurate model, and the need for data scientists to collaborate on configuring the model. After training, the model graduates to become an “inference engine” that can answer real-world questions.

In edge AI deployments, the inference engine runs on some kind of computer or device in far-flung locations such as factories, hospitals, cars, satellites and homes. When the AI stumbles on a problem, the troublesome data is commonly uploaded to the cloud for further training of the original AI model, which at some point replaces the inference engine at the edge. This feedback loop plays a significant role in boosting model performance; once edge AI models are deployed, they only get smarter and smarter.

What Are Examples of Edge AI Use Cases? 

AI is the most powerful technology force of our time. We’re now at a time where AI is revolutionizing the world’s largest industries.

Across manufacturing, healthcare, financial services, transportation, energy and more, edge AI is driving new business outcomes in every sector, including:

• Intelligent forecasting in energy: For critical industries such as energy, in which discontinuous supply can threaten the health and welfare of the general population, intelligent forecasting is key. Edge AI models help to combine historical data, weather patterns, grid health and other information to create complex simulations that inform more efficient generation, distribution and management of energy resources to customers.
• Predictive maintenance in manufacturing: Sensor data can be used to detect anomalies early and predict when a machine will fail. Sensors on equipment scan for flaws and alert management if a machine needs a repair so the issue can be addressed early, avoiding costly downtime.
• AI-powered instruments in healthcare: Modern medical instruments at the edge are becoming AI-enabled with devices that use ultra-low-latency streaming of surgical video to allow for minimally invasive surgeries and insights on demand.
• Smart virtual assistants in retail: Retailers are looking to improve the digital customer experience by introducing voice ordering to replace text-based searches with voice commands. With voice ordering, shoppers can easily search for items, ask for product information and place online orders using smart speakers or other intelligent mobile devices.

What Role Does Cloud Computing Play in Edge Computing? 

AI applications can run in a data center like those in public clouds, or out in the field at the network’s edge, near the user. Cloud computing and edge computing each offer benefits that can be combined when deploying edge AI.

The cloud offers benefits related to infrastructure cost, scalability, high utilization, resilience from server failure, and collaboration. Edge computing offers faster response times, lower bandwidth costs and resilience from network failure.

There are several ways in which cloud computing can support an edge AI deployment:

• The cloud can run the model during its training period.
• The cloud continues to run the model as it is retrained with data that comes from the edge.
• The cloud can run AI inference engines that supplement the models in the field when high compute power is more important than response time. For example, a voice assistant might respond to its name, but send complex requests back to the cloud for parsing.
• The cloud serves up the latest versions of the AI model and application.
• The same edge AI often runs across a fleet of devices in the field with software in the cloud

Learn more about the best practices for hybrid edge architectures.

The Future of Edge AI 

Thanks to the commercial maturation of neural networks, proliferation of IoT devices, advances in parallel computation and 5G, there is now robust infrastructure for generalized machine learning. This is allowing enterprises to capitalize on the colossal opportunity to bring AI into their places of business and act upon real-time insights, all while decreasing costs and increasing privacy.

We are only in the early innings of edge AI, and still the possible applications seem endless.

New York has yet to see the AI-driven explosion of data center development that has emerged in other top industry hubs, due in large part to the New York market’s high energy prices. AI will be a significant growth catalyst for data centers in New York, industry leaders said at Bisnow’s DICE: Northeast, July 18 at the Astor Ballroom in Manhattan, but that growth is likely to manifest differently than in any other primary market. 

Rather than massive cloud and AI campuses that account for the bulk of the industry’s recent growth, they say New York will see increased demand for colocation facilities and data centers specializing in access to fiber networks that connect the market to AI infrastructure largely being built elsewhere.  

“This market is doing really well for a lot of reasons that have nothing to do with power,” said Bob DeSantis, CEO of colocation provider 365 Data Centers. “New York just has so much volume. It’s expensive, but there’s already such a desire to have proximity that you add a little AI to that demand, and it overcomes any issues on the power side.”

The New York data center market, which includes New Jersey and Connecticut, is the seventh-largest data center hub in the U.S. As of the beginning of the year, the region had more than 700 megawatts of total data center inventory, the majority of it in New Jersey, and around 100 megawatts under construction, according to JLL. New York data centers have seen robust demand growth over the past two years, led by the financial services sector, with a growing share of leasing from major cloud providers. 

While the fundamentals of the area’s AI landscape are strong, New York hasn’t had the kind of unprecedented inventory growth and development pipeline seen in other primary markets. 

In Northern Virginia, Atlanta and Hillsboro, Oregon, data center inventory grew by 107%, 118% and 334%, respectively, between 2020 and the end of 2023, according to CBRE. It’s a record pace of growth that was first driven by surging demand for cloud services but has accelerated further as tech firms — led by Amazon, Microsoft, Google and Meta — engage in an AI arms race that is expected to surpass $1T in total infrastructure spending. Yet in the same time period, data center inventory in the New York Tri-State area grew by just 29%, the slowest pace among primary markets. 

The reason the AI data center bump has seemingly skipped New York comes down to the price of power, executives said at DICE: Northeast. Energy costs have always been a top siting consideration in a sector where facility size is measured in megawatts rather than square feet, but AI has dramatically increased the amount of power used by the largest data centers and has subsequently made power pricing paramount for hyperscale developers.

Companies like Amazon and Microsoft are building their largest AI data centers anywhere they can find the cheapest power. Markets attracting major hyperscale investment like Atlanta, Dallas and Chicago had average power rates last year of less than 7 cents per kilowatt hour, according to JLL. New York, by contrast, was more than twice as expensive, with an average rate of 16 cents. New Jersey, which has the cheapest power in the Tri-State market, is still relatively expensive at 11 cents per kilowatt hour.  

 “The large deployments of AI are largely in areas where power costs are down and power is more readily available from utilities,” said Phillip Koblence, co-founder and chief operating officer of colocation firm NYI. “But the New York market, this is where a lot of the data that is being manipulated by AI is created because this is where the eyeballs are and this is where the internet is most evolved.” 

Indeed, New York and the broader Northeast region is the country’s densest population center and therefore has the largest volume of consumers watching Netflix, interacting with ChatGPT and generating real-time data through phones, Apple Watches and other smart devices. Perhaps more importantly, the disproportionate number of financial institutions, major corporations and other large organizations that are based in New York represent an enormous amount of data that is only going to increase — along with its performance requirements — with AI adoption.  

Much of this growing flood of data will be processed in cheaper power markets, but first it has to get there. This means more demand for carrier hotels and connectivity-focused colocation facilities, many of which have proprietary private networks that allow faster data transfer speed, known as latency, from New York to other major data center hubs.

Connectivity has always been a big part of New York’s digital infrastructure ecosystem, experts say, but it is primed for significant growth along with AI adoption. 

“The AI boom is going to inherently benefit our market, but the driver of market growth here is going to be entirely based on connectivity to enable AI,” Koblence said. “All this AI and digital infrastructure growth is enabled by data being created in your pockets and on your rings and your watches and being transported to these large AI farms in places where the power is cheaper.”

Not all AI deployments will be located outside the New York market. Industry leaders expect AI adoption will boost demand for colocation facilities in the Tri-state area beyond what is expected elsewhere. 

This is largely due to the outsized presence of financial services firms in the New York data center ecosystem, along with health care organizations like hospital systems and pharmaceutical companies and major educational and research institutions, said 365’s DeSantis. While many companies utilize public cloud from companies like Amazon Web Services for their AI infrastructure, these sectors have huge amounts of proprietary or private data for which the public cloud presents a security or compliance risk, pushing them toward colocation providers.  

“There’s a lot of proprietary applications that those type of industries run, and there’s a lot of personal information,” DeSantis said. “Those aren’t cloud-first strategy types of data sets. Those are colocation types of data sets.”

Many of these colocation AI deployments for New York-based enterprises are going to New Jersey due to the lower power cost and other pricing advantages, and DICE panelists indicated they expect this trend to accelerate.

Digital Realty, Equinix, CoreSite and Iron Mountain plan to add a combined 145 megawatts in New Jersey by 2027, according to JLL. Other providers are building facilities in New York outside the city, such as DataBank’s development in Rockland County. 

This enterprise demand for colocation capacity exists in the more expensive New York market largely due to the financial services sector, said Jeffrey Moerdler, a longtime data center and telecom attorney and a member at Mintz.

Financial firms are executing latency-sensitive trades and other transactions where hundredths of a second make a difference. Achieving this kind of low latency performance requires having the company’s computing infrastructure as close as possible. 

“So much of the financial services industry, the brokerage industry and trading are in New York, and much of that data can’t be pushed out of the region and sent to Iowa,” Moerdler said. “It has to stay here and be processed regionally because of the latency problem.”