Initially, when we first connected 13 GPUs, the system refused to boot. After discussing this with ASUS, we concluded that the problem was that the GPUs required a larger block of physical address space than the BIOS was able to provide. The 32 bit BIOS can only map the PCI devices (including PCI-E) below the 4GB boundary, so this meant there was at most roughly 3GB of address space available for the devices. Because each GPU requires a block of 16MB, a block of 32MB and a block of 256MB, only 8 or 9 GPUs worked, depending on how many on-board devices we disabled in the BIOS setup. Adding more cards than that caused a boot failure.
ASUS was extremely helpful with solving this, and they provided a custom BIOS for our motherboard that skipped the address space allocation of the GTX295 cards entirely. This is also the reason we have a single GTX275 card in the FASTRA II: it is the one card that is fully initialized by the BIOS and can provide graphics output to the monitor.
With this custom BIOS, the system booted successfully, but without working GTX295 cards since those were not initialized yet. To enable these cards, we modified a Linux 2.6.29.1 kernel (the latest at the time) to allocate physical address space to the GPUs manually. Since the kernel is 64-bit, we could map the large 256MB resource blocks above the 4GB boundary, thereby ensuring there was plenty of room for them. The smaller 16MB and 32MB blocks easily fit below 4GB, where the GPU required them.
The remaining problem was unexpected: each GPU requires a block of 4KB of I/O port space, for which only 64KB is reserved in total. Together with low-level system devices and devices like network and USB controllers also taking up I/O space this was a very tight fit. We needed to re-map inefficiently allocated system devices and disable as many devices as possible entirely, such as the RAID controller and the second network controller. From later experiments we suspect it might actually only be necessary to allocate this 4KB block of I/O ports for the primary VGA controller, but we haven’t verified that.
