Edit: The link points to POWER8 servers at the time of this writing, but the principle is still the same.
Originally posted March 22, 2011 on AIXchange
When ordering a Power server, the number of sockets you pick and the dual inline memory module (DIMM) size you use matter. Consider the 8233-E8B server, commonly called the Model 750. This would be the same model machine that was selected to build the Watson cluster.
With a new machine you have a number of choices to make. A particularly important one is the amount of memory you want. You can choose from memory kits of different sizes, which will allow for different memory densities on the machine.
According to the facts and features guide, the maximum available memory for a 750 is 512 GB. The guide also notes that the machine supports from one to up to four sockets.
The number of sockets you choose will tell you how much memory you can order. If you have one socket, you’ll have eight memory slots. The ratio stays the same moving up: with two sockets, you’ll have 16 memory slots, with three sockets, 24 memory slots and with four sockets, 32 slots. If you’re looking to max out the memory on the machine, you’ll want to max out the number of sockets. While other choices can be made here — e.g., CPU clock speed and the number of cores per socket (either six or eight for the 750) — to reach 512GB on the system, you must choose the largest memory size, 32 GB, which is packaged as 2x16GB memory DIMMs.
Of course, other memory sizes are available for your machines: 8 GB (2x4GB) or 16 GB (2x8GB) DIMMs. But once you make your memory size selection, you need to stick with it, because, if you upgrade the machine in the future, 16 GB and 32 GB features won’t mix. They must be the same feature code. Since, in this scenario, you’re trying to max out the machine, the 32GB (2x16GB) memory option is the choice. And since you have 32 slots for memory in your 4-socket machine, you can see how 16GBx32 slots gives you 512 GB.
So how do I know that you cannot mix DIMM sizes? Here’s a little story, as told to me by someone who was there.
Once upon a time, a Power server was getting a memory upgrade. This system (not a 750) was being boosted from 16 GB to 32 GB total memory. There were 4x2GB DIMMs attached to each processor (eight total), and eight new memory DIMMs were ordered. Since the memory was going to be doubled, it seemed logical to just plug in the eight new DIMMs and power the machine back on. So, the new memory was installed and the machine was rebooted. And then? Error codes started flashing across the LED.
Perhaps the memory was not seated properly? After pulling and reseating all of the memory, the same error came up. At this moment, someone finally thought to check the boxes that the new memory came in. Sure enough, the new memory DIMMs were of a higher density — 8x4GB rather than 8x2GB. Because you can’t mix memory sizes, the machine issued errors. Once the 8x2GB memory was pulled, the machine came right back up with 32 GB.
Here’s something to think about when ordering machines: Do you expect to add more memory in the near future? Sure, this can be difficult to predict, but if you think you’ll eventually upgrade the memory, try to leave yourself some open slots. If you max out your memory with a smaller DIMM size, your only option down the road may be to pull out the smaller DIMMS and replace them with larger DIMMs.
If your machine supports it, think about using Capacity on Demand. That way you’ll have a machine with max memory physically installed, but you’ll only use (and pay for) the activated memory you need now. Should you eventually elect to upgrade, additional physical memory can be activated later. It can help take the guesswork out of future upgrades.
Whatever choices you make, be sure you know what’s installed on your machine — and what you’re adding to your machine — before opening it up.