Every six months, Earth’s biggest supercomputers have a giant race to see which can lay claim to being the world’s fastest high-performance computing cluster.
In the latest Top 500 Supercomputer Sites list unveiled Monday morning, a newly assembled cluster built with IBM hardware at the Department of Energy’s Lawrence Livermore National Laboratory (LLNL) takes the top prize. Its speed? A whopping 16.32 petaflops, or 16 thousand trillion calculations per second. With 96 racks, 98,304 compute nodes, 1.6 million cores, and 1.6 petabytes of memory across 4,500 square feet, the IBM Blue Gene/Q system installed at LLNL overtakes the 10-petaflop, 705,000-core “K computer” in Japan's RIKEN Advanced Institute for Computational Science.
The Japanese computer had been world’s fastest twice in a row. Before that, the top spot was held by a Chinese system. The DOE computer, named “Sequoia,” was delivered to LLNL between January and April. It's the first US system to be ranked #1 since November 2009.
To get to 16 petaflops, Sequoia ran the Linpack benchmark for 23 hours without a single core failing, LLNL division leader Kim Cupps told Ars Friday in advance of the list’s release. The system is capable of hitting more than 20 petaflops—during the tests it ran at 81 percent efficiency.
“For a machine with 1.6 million cores to run for over 23 hours six weeks after the last rack arrived on our floor is nothing short of amazing,” she said.
The cluster is extremely efficient for one so large, with 7,890 kilowatts of power, compared to 12,659 kilowatts for the second-best K Computer. It’s primarily cooled by water running through tiny copper pipes encircling the node cards. Each card holds 32 chips, with each chip having 16 cores.
The entire cluster is Linux-based. Compute Node Linux is run on nearly 98,000 nodes, and Red Hat Enterprise Linux runs on 768 I/O nodes which connect to the file system, Cupps said.
To start, the cluster is on a relatively open network, allowing many scientists to use it. But after IBM’s debugging process is over around February 2013, the cluster will be moved to a classified network that isn’t open to academics or outside organizations. At that point, it will be devoted almost exclusively to simulations aimed at extending the lifespan of nuclear weapons.
“The kind of science we need to do is lifetime extension programs for nuclear weapons,” Cupps said. “That requires suites of codes running. What we’re able to do on this machine is to run large numbers of calculations simultaneously on the machine. You can turn many knobs in a short amount of time.”
Blue Gene/Q uses a PowerPC architecture that includes hardware support for transactional memory, allowing more extensive real-world testing of technology.
In November 2011's Top 500 list, three of the top five clusters used NVIDIA GPUs (graphics processing units) in combination with CPUs to achieve very high speeds. This time around, only one of the top five integrates GPUs, although the overall number in the Top 500 integrating GPUs or similar accelerators rose from 39 to 58.
The use of GPUs in supercomputing tends to be experimental so far, said Dave Turek, IBM vice president of high performance computing. “The objective of this is to do real science,” he said. GPUs are a bit more difficult to program for, he said.
While the majority of Top 500 computers use Ethernet or Infiniband as their primary interconnects, Sequoia uses IBM’s proprietary 5D Torus. It's an optical network that provides 40 Gbps throughput to IBM’s Blue Gene/Q clusters. I/O nodes are connected to the file system via Infiniband and the management network uses Ethernet, Cupps said.
IBM leads the Top 500 list with 213 systems, ahead of HP’s 138. Nearly 80 percent—372 of the 500 systems—use Intel processors, followed by 63 using AMD Operton and 58 using IBM Power.
Three DOE systems are in the top 10. The rest hail from Japan, Germany, China, Italy, and France. All 10 have performance of at least at least 1.27 petaflops.
Petascale computers have become relatively commonplace since the IBM Roadrunner system at Los Alamos National Laboratory was the first to hit a petaflop in 2008. In fact, each of the top 20 systems on the new list hit at least a petaflop. Exascale, which would be 1,000 times faster, is the next big breakthrough for the IBMs, HPs, and Crays of the world to aspire to.
But a big advance in price-performance is necessary. Today’s technology could scale up a lot higher—it just wouldn’t be practical. Supercomputers are naturally expensive (even more expensive than the new MacBook Pro). The K Computer in Japan, for example, cost more than $1 billion to build and $10 million to operate each year. Livermore told us it spent roughly $250 million on Sequoia.
“We could get another order of magnitude with this technology if someone would write a check,” Turek said. “But no one would want to write that check.”
reader comments
95