[Sysadmin] Serious, egregious problem with Servermatrix Debianboxes

[Seb Potter]

John Handelaar wrote:
> Hm.
> 
> As William points out, we've actually lucked out some,
> since hdparm insists that our box is the only one out
> of the 6 identical Servermatrix boxes I have access to
> where it's *not* buggered.
> 
> The BIOS thing came from the mouths of the rather
> slackwitted engineers I was dealing with all day
> yesterday.  I'm going to go figure out whether our good
> fortune is our own doing or not.

I'm a part-time kernel hacker, and I've been checking with a couple of 
kernel devs who I know working on Debian. Here's the skinny:

Debian stable considers parallel ATA support to be unstable in kernel 
2.4, because it breaks with some old VIA mobos.

However, it's kinda recognised as a bit of a problem, as nearly every 
IDE driver manufactured in the last 5 years supports ATA-100, and 
therefore DMA makes an *enormous* performance difference for ATA disks.

How big a difference? Well, on modern 7200rpm disks with 4MB of cache, 
somewhere between a factor of 10 and 100.

I've also been told that you'd have to be out of your goddamn mind to 
build a production server with debian stable and not rebuild your kernel 
first, or at least choose "USE DMA BY DEFAULT" as an option in the old 
installer. (Apparently the new installer doesn't have this issue.)

So, I downloaded Debian and tried it. Lo and behold, hdparm reports DMA 
disabled on all my ATA-100 disks.

hdparm -t -T /dev/hda (with DMA)

Timing buffered disk reads:  162 MB in  3.02 seconds =  53.63 MB/sec

hdparm -t -T /dev/hda (without DMA)

Timing buffered disk reads:  14 MB in  3.13 seconds =  4.47 MB/sec


I haven't done write performance test, but when you get into a swap 
situation where the IO scheduler needs your disk to be able to read and 
write asynchronously at several meg/second, you really don't want to be 
running with DMA disabled, as your processing speed can be reduced by up 
to 5 orders of magnitude whilst your processes are in the iowait state.


Bottom (non techy) line:

Servermatrix have seriously fucked up the deployment of many of their 
debian servers and are in need of an administrative kicking to sort it 
the hell out. More importantly for an ISP, their support guys don't have 
a clue about setting up servers, and don't seem able to fix the problem. 
They should be making good on this problem for *all* of their customers, 
and doing it out of their own pocket.

- seb