Full Stack Linux Tuning

Here is a list of things I tune on RHEL6 linux servers with 10g ethernet and dell PERC raid cards running a RAID6.

/usr/local/sbin/10gtxqueuelen.p1p1.sh
/usr/local/sbin/set_irq_affinity.p1p1.sh
echo noop > /sys/block/sdb/queue/scheduler
/usr/local/sbin/10gtxqueuelen.p1p2.sh
/usr/local/sbin/set_irq_affinity.p1p2.sh
/usr/local/sbin/10gtxqueuelen.bond0.sh

$ cat /usr/local/sbin/10gtxqueuelen.p1p1.sh
ifconfig p1p1 txqueuelen 10000

$ cat /usr/local/sbin/set_irq_affinity.p1p2.sh
#!/bin/bash
/usr/local/sbin/set_irq_affinity.sh p1p2


set_irq_affinity.sh

https://gist.github.com/syuu1228/4352382


Megaraid tuning for dell perc6 servers:
 http://mycusthelp.info/LSI/_cs/AnswerPreview.aspx?sSessionID=6624973237SOJMBBDMCFLZRNJDJQAXSSBVDAYBNA&inc=8273

$ uname -r
2.6.32-431.11.2.el6.x86_64

So our kernel is new enough, however by default our kernel does not use 6.12 or newer:

$ modinfo megaraid_sas
filename:       /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko
description:    LSI MegaRAID SAS Driver
author:         megaraidlinux@lsi.com
version:        06.700.06.00-rh1
license:        GPL
srcversion:     9426C5CDA69586AABE13C02
alias:          pci:v00001000d0000005Fsv*sd*bc*sc*i*
alias:          pci:v00001000d0000005Dsv*sd*bc*sc*i*
alias:          pci:v00001000d0000005Bsv*sd*bc*sc*i*
alias:          pci:v00001028d00000015sv*sd*bc*sc*i*
alias:          pci:v00001000d00000413sv*sd*bc*sc*i*
alias:          pci:v00001000d00000071sv*sd*bc*sc*i*
alias:          pci:v00001000d00000073sv*sd*bc*sc*i*
alias:          pci:v00001000d00000079sv*sd*bc*sc*i*
alias:          pci:v00001000d00000078sv*sd*bc*sc*i*
alias:          pci:v00001000d0000007Csv*sd*bc*sc*i*
alias:          pci:v00001000d00000060sv*sd*bc*sc*i*
alias:          pci:v00001000d00000411sv*sd*bc*sc*i*
depends:
vermagic:       2.6.32-431.11.2.el6.x86_64 SMP mod_unload modversions
parm:           max_sectors:Maximum number of sectors per IO command (int)
parm:           msix_disable:Disable MSI-X interrupt handling. Default: 0 (int)
parm:           msix_vectors:MSI-X max vector count. Default: Set by FW (int)
parm:           throttlequeuedepth:Adapter queue depth when throttled due to I/O timeout. Default: 16 (int)
parm:           resetwaittime:Wait time in seconds after I/O timeout before resetting adapter. Default: 180 (int)


So some how you have to download the newer driver in order to spread the interrupts from your raid card across all CPUS...