Datacentres in the United States are getting a bad rap this year. Data Center Watch counted 75 projects, worth at least $130bn, blocked or delayed by communities since it started keeping records. In Cape Town, the Housing Assembly, a social movement, and UK non-profit Foxglove, which aims “to make tech fair for everyone”, filed a formal complaint in May with city planners opposing plans by US datacentre company Equinix to build two new facilities in the city. Equinix has bought 327 000sqm of land close to the airport, on which it plans to further its datacentre capacity by 160MW. Equinix told Reuters in June it had not yet submitted any planning applications to the city, and that it was committed to be fully transparent about the build. It declined to comment on the objections.
The civil society organisations say there is not enough information about the project, and Equinix had not provided any details about water use, emissions, electricity demand and noise and air pollution. Cape Town residents have particular sensitivities around water use, and the metro came perilously close to running out of water altogether in mid-2018. Equinix already has a 4MW facility close to the OR Tambo International Airport, of which it says it has 100% renewable energy coverage.
Saadiyah Kwada, an attorney at the non-profit, Legal Resources Centre in Cape Town, was quoted by Reuters as saying that there seems to be “this rush to develop datacentres without people properly thinking through what the impact will be”.
You can invest millions into IT infrastructure, but don’t go cheap on your power and your cooling.
Steve Santini, Schneider-Electric
Other than this local nascent protest, opposition is concentrated in the US, particularly in Northern Virginia, which now has over 600 datacentres in operation. There have also been protests in the EU, and there is opposition to a proposed hyperscale facility in an old brewery in Brick Lane in London. Bryce Allan, head of sustainability at Teraco Data Environments, says some of the reporting around the Cape Town project has been “sensationalist” in claiming the datacentres will use inordinate amounts of power and water. He believes one of the drivers for local protesters is that they're looking at the community action in the US, and then applying the same lens to South Africa.
“When people look at the United States, and see these 500MW, [or] 1GW datacentres being deployed in communities, hoovering up the power, and depleting water resources using open[-circuit] water tower coolers, that anxiety filters across here.
“Our South African guys have been a lot more responsible in how they’ve built out their infrastructure, and it’s also demand-led. We’re not going to see those 500MW, 1GW installations here,” he says, adding these will only be built in the US, the Middle East and China.
He says Teraco, which has a closed-loop water system, uses 98% or 99% less water than open water tower installations, which expel heat by cooling water through evaporation.
Cooling now accounts for between 30% and 40% of a datacentre’s power use. Almost all the rest of the power is consumed by the IT equipment running the compute and storage, but some power is also lost to heat through transformers and UPS units.
Our South African guys have been a lot more responsible in how they’ve built out their infrastructure, and it’s also demand-led.
Bryce Allan, Teraco
There are a number of ways of cooling the IT equipment in datacentres. The traditional way is though air handlers that circulate air under the floor, which then flows through perforated floor tiles. There is also aisle containment, so that hot and cold air don’t mix.
More recently, as compute workloads become more demanding, and thus draw more power and produce more heat, direct liquid cooling is being deployed. This will see a cold plate with coolant attached directly onto the CPU and GPU. The fluid is then cycled to cooling distribution unit, and returned to the plate. Another solution is reardoor heat exchanges which, as the name says, is attached to back of the cabinet to carry away the hot air. For ultra-high power density equipment, immersion cooling can be used, which will see the server submerged in a non-conductive dielectric fluid.
Allan says its next datacentre, JB7, to be built at its Isando campus will be the company’s first dedicated liquid cooling facility to support power-dense installations, designed, as he says, from the ground up.
He says it can cool high-density compute up to about 55Kw in a normal raised floor environment. These cabinets will also need rear-door cooling, which he likens to a large radiator attached to the back of the cabinet. Cloud customers, on the other hand, have racks between 30Kw to 50Kw, and these can be looked after in a normal air-cooled environment without the bolt-on radiators.
Allan says he’s heard rumours of a megawatt cabinet, but that still seems some way off. In the near term, he’s also heard talk of cabinets drawing between 100kW and 200kW, which, without a doubt, will need liquid-to-liquid cooling.
He says it already has some clients running AI workloads, but that they’re still limited to 55kW cabinets, with rear-door cooling. “But the newer stuff that our clients are talking about will definitely require liquid-to-liquid cooling. We see that coming in the near term.”
The reality is, if you had to take the compute equipment sitting in our facilities, and push them back into disparate, legacy on-prem datacentres, you’d do double, maybe triple the electricity usage.
Bryce Allan, Teraco
He adds that it has heat maps of all its datacentres. “Blue is too cold, green is just right, and [it will register] yellow if it’s getting hot. Orange is too hot, and with red, you need to act immediately. If you’re cooling to a single set point, you’d measure the different temperatures along an aisle,” he says. “You could find a measurement of 22°, and then walk down the aisle, and you could find that another portion of the aisle is at 12° or 16° because it’s getting too much cooling, and you don’t have enough servers down there. Or you could have a hot pocket that’s not being monitored.”
He says it has sensors every metre to a metre-and- a-half down the rows of servers, which feed back into its Vigilant system, an AI cooling management system. This will either speed up, or slow down the fans under the raised floor to make sure the cooling is delivered to where it’s needed. He says best practise is getting the cooling to where it’s needed, at the right temperature. Anything below that temperature and you’re overcooling and wasting energy; anything above, you’re at risk of breaching SLAs and damaging equipment. He says most clients’ equipment will run consistent IT loads throughout the day and at night, but there can be an uptick in usage around a particular event, such as a month-end payment run, a big retail sale or streaming event, to which the cooling system will need to adapt.
Allan says there isn’t any immersion cooling at any of its facilities yet, and with the increased compute density, the floor will need to be specially designed to carry the extra weight of the equipment.
Tracing the evolution of compute, Allan says racks used to be in the 1kW to 3kW range for carriers and ISPs, and then this increased to around 5kW as enterprises and content providers rented space. This then jumped to between 5kW and 15kW as the cloud customers arrived, and “all these [numbers] are moving northwards”. Its first datacentres were 500kW total IT load, and then it moved to 1MW and then to 2.5MW. Now, individual halls are at 5MW.
“Our designs adapt as our clients’ compute density increases,” he says, “and we need to adapt to pathways to support that in our campuses. Schneider Electric is a major player in datacentre projects, and Steve Santini, the company’s vice president, secure power, for Sub-Saharan Africa, says it does “everything from the grid to the chip”.
This means it supplies the racks, the UPS power and power distribution, cable management, the heating, ventilation, and air-conditioning systems, as well as grid and power management. The customer provides the servers, storage and networking.
It also does cooling, and supplies the cooling distribution units, and the chip cooling itself. “We even do some submersion. We do the whole bag. “You can invest millions into IT infrastructure, but don’t go cheap on your power and your cooling, because it will all mean nothing when that goes down. If that fails, you’re dead in the water.”
He says the datacentre market is expanding rapidly in Africa, and that the continent has a massive appetite for datacentres. Other than South Africa, he says it’s seeing builds in Nigeria, Kenya, and little pockets in Rwanda, and Ethiopia. “The friction for deploying datacentres in Africa is very low, compared to Europe and the West. The continent is saying, ‘Don’t worry about the process; the answer is yes, and we’ll figure it out later’.” Around 70% of its solutions are sold through the channel, and in South Africa it works with, among others, DCC and Tarsus. But for a big AI deployment or large datacentre, it will deal direct because the channel doesn’t have the capacity.
He says the company has a relationship with Nvidia, with whom it works closely on many of its cooling designs.
Santini says there are some AI workloads running in local colo environments, particularly from fintechs.
Allan, from Teraco, admits the datacentre industry could probably do a better job of communicating how the industry operates.
The company owns the Great Westerford building in Newlands, Cape Town, and its neighbour is the Westerford High School. “I was in the principal’s office last Thursday, apologising for the noise we were making for our DC5 expansion”, he says.
And all pupils and the principal are reading the media stories about datacentres.
“The reality is, if you had to take the compute equipment sitting in our facilities, and push them back into disparate, legacy on-prem datacentres, you’d do double, maybe triple the electricity usage,” he says, adding that he thinks it’s time he delivered a talk at the school.
* Article first published on www.itweb.co.za
Share
