Introduction -- Platforms -- Measuring Power -- Applications -- Reducing Power During Idle Cycles -- Tuning CPU Power During Application Run-Time -- Network Bandwidth Tuning During Application Run-Time -- Energy Delay Product -- Conclusions.

Recognition of the importance of power and energy in the field of high performance computing (HPC) has never been greater. Research has been conducted in a number of areas related to power and energy, but little existing research has focused on large-scale HPC. Part of the reason is the lack of measurement capability currently available on small or large platforms. Typically, research is conducted using coarse methods of measurement such as inserting a power meter between the power source and the platform, or fine grained measurements using custom instrumented boards (with obvious limitations in scale). To analyze real scientific computing applications at large scale, an in situ measurement capability is necessary that scales to the size of the platform. In response to this challenge, the unique power measurement capabilities of the Cray XT architecture were exploited to gain an understanding of power and energy use and the effects of tuning both CPU and network bandwidth. Modifications were made at the operating system level to deterministically halt cores when idle. Additionally, capabilities were added to alter operating P-state. At the application level, an understanding of the power requirements of a range of important DOE/NNSA production scientific computing applications running at large scale (thousands of nodes) is gained by simultaneously collecting current and voltage measurements on the hosting nodes. The effects of both CPU and network bandwidth tuning are examined and energy savings opportunities of up to 39% with little or no impact on run-time performance is demonstrated. Capturing scale effects was key. This research provides strong evidence that next generation large-scale platforms should not only approach CPU frequency scaling differently, as we will demonstrate, but could also benefit from the capability to tune other platform components, such as the network, to achieve more energy efficient performance.