RRD stress-test on kif

I got a new pair of OpenNMS servers, HP DL360 G5s with one dual-core Xeon Woodcrest CPU each. The daemons node, kif, was specced to optimize disk throughput. Here's the physical layout:

• OS and apps on a RAID1 -- a single two-disk mirror
• JRB files on a RAID10 -- a single stripe of two two-disk mirrors

I laid out all but the /boot and swap volumes on LVM volumes, for future expandability. The observant reader will realize that this setup maxes out the disk bays in a G5 DL360, but I can add up to two more physical disks by upgrading to a DL380.

The JRB disks are 72GB 15KRPM SAS disks. The RAID is done in hardware by an HP P400i controller with the 512MB battery-backed write cache upgrade. I am trying to determine what filesystem and options offers the best performance for an OpenNMS installation on this physical setup.

I ran the RRD stresser with the following invocation against three different filesystem setups on the same physical / logical layout defined above:

java -jar -Xmx1024m  -Dorg.opennms.rrd.queuing.writethreads=10 -Dstresstest.filecount=30000 \
-Dstresstest.modulus=1000 -Dstresstest.threadcount=75 -Dstresstest.maxupdates=100000 \
-Dstresstest.file=/rrd/stress/ opennms-rrd-stresser-1.3.3-SNAPSHOT-jar-with-dependencies.jar

The raw results are available here

I post-processed these results with an ugly shell+Perl one-liner that averages the significantOpsPending and produces a histogram of these values:

gehlbachj@gehlbachj-laptop:~/projects/rrd-stress$ for file in * ; do echo $file ; \
 grep 'significantOpsPending' $file | perl -e 'my %counts; my $samp = 0; my $tot = 0; while (<STDIN>) { \
 if ($_ =~ /significantOpsPending=(\d+),/) { $counts{$1}++; $samp++; $tot += $1; } } print "Average: " \
 . ($tot / $samp) . "\nHistogram:\n"; foreach my $count (sort { $a <=> $b} keys %counts) { print \
 "\t$count: $counts{$count}\n"; }' ; done

Results:

rrd-stressresults_ext3+data=writeback,noexec,nosuid,nodev,noatime.txt
Average: 2.96
Histogram:
        0: 15
        1: 26
        2: 9
        3: 14
        4: 8
        5: 9
        6: 6
        7: 11
        8: 1
        12: 1
rrd-stressresults_ext3+noexec,nosuid,nodev,noatime.txt
Average: 1.68
Histogram:
        0: 39
        1: 17
        2: 17
        3: 9
        4: 8
        5: 4
        6: 4
        7: 2
rrd-stressresults_xfs+noexec,nosuid,nodev,noatime.txt
Average: 2.16
Histogram:
        0: 28
        1: 20
        2: 11
        3: 16
        4: 9
        5: 8
        6: 7
        8: 1

MIB Studies

• MIB Study IP Unity
• MIB Study Asterisk
  • MIB Revamp Proposal Asterisk