A first law in building large systems to High Performance Computing is more processors are better, for many years the development of processors was dominated by the x86 family, beating including the PowerPC (more in cost than in performance in my opinion) and new families emerge like the x86_64 and the multicore chips.
Beside the CPU technology advances, other part of the computer had acquired a new task, the graphic card, in early stages of computer evolution, its function was only connect and control the visual output for the system and the monitor. Later computer games demand more intensive usage for the GPU, principally by 3D environments, and new technologies were created to satisfy the demand. Actually a high performance graphic card is like a small but very efficient version of a complete system, including one or more special GPUs, own memory and can share information with the host system or other graphics card very fast (SLI).
How do include this computing power? Because apart from video games there are another problems which require graphical processing or similar calculations (vectorial maths), the GPU is a specific processor to manage this problems. Nvidia has release a specific C extension named CUDA, now you can write or prepare your code to use the power of the GPU and build systems really impressive like this GPU2 Farm or buy a Tesla.
The new paradigm makes us to think in how to code to use the potential of this, even for no-3D.
I need to practice my C-fu.