Edit: At the time of this writing we are waiting to begin the POWER10 Journey
Originally posted December 5, 2017 on AIXchange
One of the great benefits of being an IBM Champion is the ability to attend IBM briefings about unannounced products. For instance, recently, IBM gave us some details about the new Linux-only POWER9™ server. The AC922 (Machine Type 8335-GTG) for high-performance computing (HPC) is being officially announced today, and will be generally available Dec. 22. Learn more about the AC922 here.
The AC922 will be the first GA system to run a POWER9 processor. This machine takes advantage of the new faster speeds we’ll see from PCIe Gen 4–which is twice as fast as PCIe Gen 3–along with the improved speeds of CAPI 2.0 and next-generation NVIDIA NVLink, where POWER remains the only processor for NVLink from the processor to the GPU accelerators.
It contains two POWER9 sockets with up to 40 cores, and up to four NVIDIA Volta-based Tesla V100 GPUs. It will max out at 1TB of memory if you use the 16 x 64 GB DIMMs (eight per socket). It has four PCIe Gen 4 slots, and can have up to 7.7 TB of storage and 3.2TB NVMe adapters. It’s not designed for virtualization; it’s intended to be configured as a bare metal “single server.”
There are two processor modules: 16-core and 20-core. Initially, the available memory options are 16, 32, or 64GB industry standard DIMMs. There are two hard drive slots per machine. You can choose from 1TB and 2TB HDD options and 960GB, 1.92TB and 3.84TB SSDs. You have your choice of RAID0, RAID1 and RAID10.
I’ll share some information I received during the call. These notes come from presentations created by IBM experts:
- The AI era is going to be a journey. Clients are faced with challenges of commodity hardware combined with open source software. IBM has built the best systems in the marketplace to crush the data challenges of the AI era. These are enabled with advanced I/O interfaces, new shared memory structures and co-optimized hardware and software.
- There are three key points with the AC922, which make it the best server for AI. First, it is designed from the ground up for AI workloads; this starts with the acceleration superhighway. In AC922, IBM has introduced second generation NVLink between the CPU and GPU, this is 5.6x faster than PCIe Gen 3 architectures. Second, IBM did not only focus on the NVLink and the GPU, but designed a balanced system, one that is designed for the AI era with industry-leading memory bandwidth, PCIe Gen 4 buses for the best network connectivity with Infiniband and high performance storage adapters. Lastly, IBM took the open source deep learning frameworks and optimized them around this advanced design. This results in the best server and solution for enterprise AI. Additionally, this server design will find use in applications such as HPC and accelerated databases…so do not think it is just for AI.
- At the center of Power Systems’s differentiation is the processor. Everything starts from here and it is designed for the cognitive era. Power has always had a stronger core with up to 4x the threads over x86. The architecture also enables advantaged memory bandwidth for a balanced system design, enabling ease of data movement within the system. One of the core differentiators Power delivers is the advanced I/O interfaces. Last fall IBM introduced POWER8 with NVLink. This was the first processor with NVLink between the CPU and the GPU. With POWER9 IBM introduced more advanced interfaces such as next generation NVLink, PCIe Gen 4 and OpenCAPI.
- This remains the only processor in the industry to leverage NVLink between CPU-GPU.
- When IBM talks about AC922, they discuss CORAL. CORAL is the collaboration between Oak Ridge, Argonne and Lawrence Livermore research labs for the Department Of Energy. It all starts with the POWER9 processor and the NVIDIA Tesla V100. IBM is combining these on a motherboard, which is differentiated with the connectivity between them. All nodes are contained in a standard rack mount chassis. It is the repeatable building block used for this super computer.
Resources
On Twitter, the OpenPOWER Foundation shared photos of this system that were shown at November’s SC17 conference. Check out picture 10.
This article describes the nodes that make up the Summit Supercomputer. It gives you an idea about potential real-world uses for these nodes:
Oak Ridge National Laboratory’s new Summit supercomputer, projected to be the fastest in the world, should rocket the U.S. back into the lead over China on the top 500 list of fastest supercomputers. At SuperComputing 2017, IBM demoed its Power Systems AC922 server nodes that serve as the backbone of the Summit supercomputer. …
Summit promises to deliver 5-10x more performance than its predecessor, Titan, but it crams much more power into a smaller footprint. Titan featured 18,688 nodes, but Summit will overpower it with “only” ~4,600 nodes. That capability stems from increased node performance; Summit will offer more than 40 TeraFLOPS per node, whereas each Titan node weighed in at 1.4 TeraFLOPS. Packing all that power into a single node begins with IBM’s water-cooled Power Systems AC922 node. Each node is equipped with two IBM POWER9 processors and six Nvidia Volta GV100 GPUs. The nodes also feature an aggregate of 512GB of coherent DDR4 and HBM2 (High Bandwidth Memory) along with 1,600GB of non-volatile RAM. …
Supercomputers are all about parallel computation and moving data between the CPUs, GPUs, memory, and networking, so Summit provides numerous layers of extreme bandwidth. The system features 96 lanes of PCIe 4.0 that comes in handy for the dual-port Mellanox EDR InfiniBand adapter, which has a theoretical maximum throughput of 400Gb/s. IBM has measured throughput at 392Gb/s, which is twice the bandwidth of a PCIe 3.0 adapter. The Volta GV100’s connect via PCIe 3.0 and NVLink 2.0. The NVLink interface provides 100GB/s of throughput for CPU-to-GPU and GPU-to-GPU traffic. The GPUs are arranged in a dual-mesh design. Interestingly, IBM also produces a model with four GPUs that will power CORAL’s Sierra supercomputer. The four-GPU model (the last picture in the album above) touts 150GBps for inter-GPU/CPU communication. Due to the reduced number of GPUs, IBM can provision more links (“bricks” in NVLink parlance) to the CPUs and GPUs, which increases throughput. …
The POWER9 processors have eight memory channels, for a total of 16 channels per server that provide 340GB/s of aggregate bandwidth. Each Summit node will wield a maximum of 2TB of DDR4-2666 memory.
In this video, “Scott Soutter, IBM; Steve Fields, IBM Power Systems; and Dylan Boday, IBM Power Systems discuss Power AI, deep learning frameworks, continued partnership with Nvidia for POWER9, and Open CAPI, from SC17 in Denver, Colorado.”
Although there’s nothing AIX-specific in today’s announcement, more announcements that cover the AIX and IBM i ecosystem will be made in the future.
IBM has issued a statement of direction for the POWER9 Enterprise hardware. I’ve also seen timelines for AIX and IBM i that, I assure you, extend years into the future.
Obviously, there’s much more ahead with POWER9, but this machine is the first step on that journey.