There are 3 big problems when you want a supercomputer (besides the cost, of course), the first one is the space, a cheap and common approximation is to use a cluster design which you wire, as many as you can, independent computers. So you need to reserve an adequate space for the equipment, if you're not using servers racks, the space is a problem.
The second problem is the calorific power generated by the computers, a single desktop computer can be really hot in intensive use, so imagine when you put many machines in a same room together. I remember the first cluster of the bioinformatic lab, the provider installed the hardware one day before they installed the software, but when the guy turned-on the systems, all the RAMs were burned. The provider changed all the RAMs for a more heat resistant brand. After some time and after some expansions in the cluster, the cooling systems also was required to be changed for an industrial equipment, in summer it feels like an oven.
The third problem is the electricity cost, because many energy is loose in form of calorific transformation, the system is not efficient.
Last day reading the Icaza's blog, I noted a company named SiCortex, who is selling a green supercomputer solution, based in multi-CPU towers, they have from desktop size with 72 processors, a mid-size configurable with 324 to 1,458 processors and up to 1.9 terabytes of memory, and the "I-want-of-this"-size: the SC5832 with up to 8.2 TFlops of compute power in a single, self-cooling cabinet and configurations range from 1,944 to 5,832 processors and up to 8 terabytes of memory. I did not see prices, except for the desktop version which cost ~$24k usd (a special offer can reduce the price to $16k), I cannot imagine how much cost the others.
The secret is the tower design and the type of processors used, the system use very few space, few energy and don't need A/C. Of course it runs GNU/Linux (RedHat) and support common tools (GCC, Fortran, MPI, ...). For more technical details check the Architecture Tour.