Daperture

Many of us get a SAN to improve the performance of IO intensive applications such as databases. Typically, these SANs come with huge 2-4-8 or more GB caches. This means that the SAN is able to queue and handle a massive number of IO requests while it flushes the requests to disk at its leisure.

The defaults scheduler on Linux is typically set to CFQ which stands for Completely Fair Queuing. It's designed to give all processes a fair amount of time for disk access by dividing the IO requests by priority and allocating the appropriate timeslices for the given process. Simply, each process gets a timeslice for disk access.

Typically, the default CFQ setting offers good all around performance on a linux box, but when using a SAN, it's best to allow the optimization happen on the SAN level rather than having the kernel optimize the IO requests before sending them out to disk. In a typical DAS system, CFQ invests in prepping IO requests before sending them to disk in anticipation of a greater return when the IO requests complete. The bottleneck introduced is the extra time waiting for synchronous requests to queue up in order to intelligently reorder the requests for minimal disk head seeking.

Under the CFQ disk scheduler, when one runs "iostat 1" under a heavy disk load, one will see the disks take 3-6-10k requests followed by little or no requests another second, followed by another 3-6-10k requests. The hiccuping pattern of IO requests emerges and shows that the disk isn't being fully utilized. Not a good way to get your moneys worth, especially when one drops a $100K+ on a SAN with fibre channel disks.

By changing the disk scheduler to NOOP, it places IO requests in a FIFO queue. This assumes that the requests will be optimized elsewhere. This takes up far less CPU time and does not hold up the IO requests as long as CFQ would. Since the kernel does not know the disk layout of a SAN LUN (typically raid 5) it also doesn't make sense to do CFQ. Instead we allow the SAN with its big caches and intelligent controllers to handle all the heavy work for us.

On my mail server, changing from CFQ to NOOP brought down load averages from an easy 50-60-100 down to a manageable .67 even with about 5000 writes per second.

To figure out what scheduler you are using:

cat /sys/block//queue/scheduler

To change the scheduler to noop during runtime:

echo noop > /sys/block//queue/scheduler

To change the scheduler to cfq during runtime:

echo cfq > /sys/block//queue/scheduler