Researchers Push Boundaries To Lead Industry in Sustainable Directions
Some things are treacherous when mixed, like water and electricity. So when advanced-computing
experts at the National Renewable Energy Laboratory (NREL) made the move over a decade
ago to use water to cool central processing units in its data center, it is understandable
why some folks instinctively thought it was a bad idea.
“Most people balked at the idea of having water anywhere near a computer,” said Aaron
Andersen, NREL’s group manager for advanced computing operations. He understood why
the computing industry had doubts. But because NREL is the U.S. Department of Energy’s
(DOE’s) laboratory dedicated to energy-efficiency research, NREL was uniquely positioned
to challenge conventional energy-intensive mechanisms for cooling, like air conditioning.
It was a calculated risk to install NREL’s first water-based cooling system inside
its Energy Systems Integration Facility (ESIF) data center in 2012. NREL’s collaboration
with Hewlett-Packard (HP) on innovative liquid-cooled supercomputing was recognized
with a 2014 R&D 100 award and a 2014 R&D 100 Editor’s Choice Award for Sustainability,
recognizing the best of the 100 winners. “Water is much better than air at conducting
heat, and getting the liquid cooling close to the hot computer chips was essential
to the high efficiency of the cooling approach,” said Steve Hammond, NREL’s computational
science center director from 2002 to 2019.
That risk paid off, and not just in the $4.05 million operational-cost savings to
date.
By not relying on air conditioning, the data center has avoided expelling 22,829 metric
tons of carbon dioxide over the past 10 years. That is the equivalent of sequestering
carbon dioxide emissions from over 25.2 million pounds of coal.
But NREL researchers know that water is a precious resource too. Since introducing
the innovative and energy-efficient thermosyphon cooling system to the ESIF data center
in 2018, NREL has saved over 6.5 million gallons of water.
These massive efficiency achievements are what have kept the NREL data center’s power
usage effectiveness (PUE) as low as 1.028. PUE measures energy efficiency by dividing
the total amount of power entering a data center by the power needed to run the IT
equipment within it. The computing industry’s average PUE has ranged from 1.57 to
1.67 since 2013.
As a result of these efforts and others, the ESIF data center has been recognized
with numerous awards, including the “Oscars” of the data center industry—the Data
Center Dynamics Data Center Eco-Sustainability Award—which recognizes innovative and
pioneering approaches to sustainable data center design. The award established NREL’s
place as a recognized leader in the field, and the ESIF regularly hosts many people
in the computing industry to share the laboratory’s knowledge.
“We benefitted from a team of talented, world-class researchers matched with strong
industry partners,” said Ray Grout, NREL’s center director for computational science.
“We are always searching for opportunities to share what we’ve learned with others.
Collaboration is one way we find computing’s next big questions—and the answers.”
How To Raise the Supercomputing Bar
A paradigm-shift mentality has kept NREL’s researchers pushing the boundaries of computing.
When DOE’s Office of Energy Efficiency and Renewable Energy installed the Eagle supercomputer
at NREL in 2012, the advanced computing team had four major objectives involving the
responsible stewardship of energy, water, and waste products. They also knew they
needed to stay nimble as they pressed forward. “The team left room to adjust course
to accommodate discoveries made along the way,” Andersen said.
That mentality left the door open for achievements like thermosyphon cooling to break
through. “The efficient use of water was always part of the plan,” explained Andersen,
“but the thermosyphon approach came after four years of operations.”
Thermosyphon cooling is a form of evaporative cooling, which is an effective technique
for managing the massive amounts of heat that supercomputers produce. It utilizes
dry sensible cooling by flowing the return (hot) water from the data center through
a flooded shell and tube evaporator that is filled with a refrigerant. Automatic controls
vary the fan speed to control the evaporation cycle and reject much of the sensible
heat to the atmosphere, further cooling the return water and reducing or eliminating
the evaporation of water depending on outside atmospheric conditions.
NREL, with partners Johnson Controls and Sandia National Laboratories, initially deployed
this innovative thermosyphon cooler as a test bed on the roof of NREL’s ESIF. A key
ingredient to its success was the addition of the Johnson Controls BlueStream Hybrid
Cooling System—an advanced dry cooler that uses refrigerant in a passive cycle to
dissipate heat.
In its first two years of operation, the thermosyphon saved the equivalent of more
than three Olympic-size pools of water. This accomplishment earned NREL and its partners
a 2018 Federal Energy and Water Management Award and the Data Center Dynamics 2018
Eco-Sustainability Award.
The efficient use of energy and water were the first two objectives NREL researchers
tackled simultaneously. A typical data center needs to dedicate 70% of its energy
consumption to the task of keeping its equipment cool. Within 10 years, NREL managed
to whittle that percentage down to just 3% for not only its data center but also its
entire ESIF—a Leadership in Energy and Environmental Design (LEED) Platinum-certified
facility where 14 labs, a data visualization center, a control room, and a supercomputer
occupy 182,500 square feet.
“The third objective focused on recycling or reusing waste products,” Andersen said,
“and the ESIF’s ability to heat the NREL campus with its waste heat is an illustration
of that.”
NREL collaborated with HP for the HP Apollo 8000 System. This innovative system uses
component-level warm-water cooling to dissipate heat generated by the supercomputer,
thus eliminating the need for expensive and inefficient chillers in the data center.
Water circulates through heat exchangers in the high-performance computing (HPC) systems
to efficiently capture waste heat. The water is heated to around 100°F by the HPC
systems and is used as a source of heating for laboratory and office spaces.
R&D Magazine recognized NREL and HP’s achievement with a 2014 Laboratory of the Year award.
But researchers will not rest on yesterday’s advanced computing achievements; they are using them as stepping stones on the path to their next big breakthrough.
Prepared for Future Partnerships
Andersen credits NREL’s success to the unwavering dedication of his team members,
who are “empowered by NREL’s mission, which allows us to take the bold steps forward
that many organizations with large data centers can’t take based on the risk.” Experts
from across disciplines are drawn to NREL’s living laboratory ecosystem, where taking
risks is an important part of scientific discovery.
NREL’s pursuit of more energy-efficient computing is not over. In March 2022, NREL
partnered with the Joint Institute for Strategic Energy Analysis to launch its Green Computing Catalyzer to explore algorithmic energy consumption improvements and measurements, data center
efficiency, and the waste cycle of computers and materials.
Advancing the science of computing is complemented by NREL’s dedication to provide
world-class HPC capabilities to its researchers and partner organizations. Advanced-computing
researchers are searching for ways to reduce the energy consumption of powerful research
applications, like the DOE’s ExaWind code used to create wind turbine models and simulations,
or the Vienna Ab Initio Simulation Package (VASP) code used for atomic-scale materials
modelling. As NREL continues to lead on the frontiers of artificial intelligence and machine learning, more efficient algorithms are needed to optimize GPU usage and ultimately yield
faster time-to-solution.
As NREL’s digital-twin capability grows, it offers partners the chance to simulate
and evaluate tweaks to a data center configuration before investing millions of dollars
in upgrades. (A typical data center upgrade investment can cost $70 to $80 million.)
In support of the Advanced Research Projects Agency-Energy’s COOLERCHIPS program,
NREL is creating a digital twin to develop testing protocols that evaluate energy-efficient
cooling technologies for data center operations.
The thermal-energy potential of the NREL data center is now being explored to provide
residential hot water heating at higher thermal temperatures. The Kestrel supercomputer—built by HPE—is now installed in the ESIF data center. It will amplify NREL’s data
center advancements and support NREL’s journey to net-zero-lab status.
NREL is where bold ideas and collaborative partnerships bring the paradigm-shifting
changes our future demands. To learn how NREL’s advanced computing capabilities can support your mission, contact Steve Gorin.