8-STABLE performance issues on Supermicro Core i7

[Bryce Edwards]

Hello,

I've got a new Supermicro X58 system with an Intel Core i7 930 with 6
GB ram that is not performing nearly as fast as it should in many ways
(compiling, network transfers).  To give an example, it has been
building the gcc44 port for about 10 hours now and at the same time
rsync'ing from a Linux box on the same Gigabit network is only getting
throughput of between 10-25 MB/sec.  When I did a buildkernel for
8-STABLE, it took 17 hours!

In the BIOS, I have played with a few settings and some actually made
it worse.  What I have done now is disabled Hyperthreading and
Speedstep.  I have attached the dmesg details, and here's some system
info and then some system stats:


bryce@tahiti[~]>uname -a
FreeBSD tahiti.bryce.net 8.0-STABLE FreeBSD 8.0-STABLE #0: Wed Apr 28
10:53:37 CDT 2010
[hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64



bryce@tahiti[~]>cat /boot/loader.conf
ahci_load="YES"
ichsmb_load="YES"
smb_load="YES"
coretemp_load="YES"

zfs_load="YES"
vfs.root.mountfrom="zfs:system"

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1


bryce@tahiti[~]>cat /etc/sysctl.conf
kern.timecounter.hardware=HPET


bryce@tahiti[~]>vmstat 1
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad0 ad1   in   sy
cs us sy id
 5 0 0   1068M  3478M   572   1   1   0   862   0   0   0 9370 16514
16157 71 22  7
 5 0 0   1068M  3478M     2   0   0   0     0   0   0   0 8008 14504
11716 81 17  2
 5 0 0   1068M  3478M     0   0   0   0     0   0   0   0 12429 22323
18125 77 23  0
 5 0 0   1068M  3478M     0   0   0   0     0   0   0   0 12348 22125
17988 73 27  0


bryce@tahiti[~]>vmstat -i
interrupt                          total       rate
irq1: atkbd0                        9291          0
irq17: fwohci0                         1          0
cpu0: timer                     75416246       2000
irq256: em0                    137590284       3649
irq257: em0                    206367605       5473
irq260: em0                            1          0
irq266: ahci0                    9892384        262
cpu2: timer                     75415653       2000
cpu3: timer                     75415702       2000
cpu1: timer                     75415561       2000
Total                          655522728      17385


bryce@tahiti[~]>netstat -I em0 -h 1
            input          (em0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      7.7K     0     0        11M       7.2K     0       475K     0
      8.1K     0     0        12M       7.4K     0       491K     0
      7.8K     0     0        11M       7.2K     0       476K     0

bryce@tahiti[/usr/adm]>iostat 1
       tty            ada0             ada1             ada2             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0   108 22.35   3  0.07  20.61   3  0.07  58.60   0  0.00  71  0  4 17  7
   0   222 64.00   1  0.06  128.00   1  0.12   0.00   0  0.00  87  0  2 11  0

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

[George Neville-Neil]

On May 1, 2010, at 11:10 , Bryce Edwards wrote:

> Hello,
>
> I've got a new Supermicro X58 system with an Intel Core i7 930 with 6
> GB ram that is not performing nearly as fast as it should in many ways
> (compiling, network transfers).  To give an example, it has been
> building the gcc44 port for about 10 hours now and at the same time
> rsync'ing from a Linux box on the same Gigabit network is only getting
> throughput of between 10-25 MB/sec.  When I did a buildkernel for
> 8-STABLE, it took 17 hours!
>
> In the BIOS, I have played with a few settings and some actually made
> it worse.  What I have done now is disabled Hyperthreading and
> Speedstep.  I have attached the dmesg details, and here's some system
> info and then some system stats:
>
>
> bryce@tahiti[~]>uname -a
> FreeBSD tahiti.bryce.net 8.0-STABLE FreeBSD 8.0-STABLE #0: Wed Apr 28
> 10:53:37 CDT 2010
> [hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64
>
>
>
> bryce@tahiti[~]>cat /boot/loader.conf
> ahci_load="YES"
> ichsmb_load="YES"
> smb_load="YES"
> coretemp_load="YES"
>
> zfs_load="YES"
> vfs.root.mountfrom="zfs:system"
>
> hint.p4tcc.0.disabled=1
> hint.acpi_throttle.0.disabled=1
>
>
> bryce@tahiti[~]>cat /etc/sysctl.conf
> kern.timecounter.hardware=HPET
>
>
> bryce@tahiti[~]>vmstat 1
> procs      memory      page                    disks     faults         cpu
> r b w     avm    fre   flt  re  pi  po    fr  sr ad0 ad1   in   sy
> cs us sy id
> 5 0 0   1068M  3478M   572   1   1   0   862   0   0   0 9370 16514
> 16157 71 22  7
> 5 0 0   1068M  3478M     2   0   0   0     0   0   0   0 8008 14504
> 11716 81 17  2
> 5 0 0   1068M  3478M     0   0   0   0     0   0   0   0 12429 22323
> 18125 77 23  0
> 5 0 0   1068M  3478M     0   0   0   0     0   0   0   0 12348 22125
> 17988 73 27  0
>
>
> bryce@tahiti[~]>vmstat -i
> interrupt                          total       rate
> irq1: atkbd0                        9291          0
> irq17: fwohci0                         1          0
> cpu0: timer                     75416246       2000
> irq256: em0                    137590284       3649
> irq257: em0                    206367605       5473
> irq260: em0                            1          0
> irq266: ahci0                    9892384        262
> cpu2: timer                     75415653       2000
> cpu3: timer                     75415702       2000
> cpu1: timer                     75415561       2000
> Total                          655522728      17385
>
>
> bryce@tahiti[~]>netstat -I em0 -h 1
>            input          (em0)           output
>   packets  errs idrops      bytes    packets  errs      bytes colls
>      7.7K     0     0        11M       7.2K     0       475K     0
>      8.1K     0     0        12M       7.4K     0       491K     0
>      7.8K     0     0        11M       7.2K     0       476K     0
>
> bryce@tahiti[/usr/adm]>iostat 1
>       tty            ada0             ada1             ada2             cpu
> tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
>   0   108 22.35   3  0.07  20.61   3  0.07  58.60   0  0.00  71  0  4 17  7
>   0   222 64.00   1  0.06  128.00   1  0.12   0.00   0  0.00  87  0  2 11  0

First thing to try.  Turn off Legacy USB in the BIOS.

Is this a 2 CPU machine with only one CPU installed?  Make sure your memory is all
in the correct slots, on modern (non Front Side Bus) machines like Nehalem
only the memory in the slots that are adjacent to a CPU are seen by the CPU.

I gather you have a kernel with INVARIANTS, WITNESS and the like turned off, correct?

Best,
George

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

[Case van Rij]

I was a little surprised to read this, I've been running 7.1 based
code on some supermicro nehalem machines (5500, Xeon, X8DT6/X8DTE)
which perform reasonably well, and was hoping to test some X8SAX based
systems with core i7 to compare performance.

The configuration Ive been using so far:
2x E5520 Xeon, SMT disabled ie. 2 x 4 cores
24GB (3x 4 GB per CPU socket)
LSILOGIC SASX36 A1 7015, SEAGATE ST9300603SS 0006

This made me curious enough to put 8.0-RELEASE on one of these to find
out how it would do on buildkernel, and it seems to perform 'as
expected',

nhm005# /usr/bin/time -h make buildkernel
        18m20.31s real          13m13.84s user          5m1.47s sys

nhm005# /usr/bin/time -h make -j4 buildkernel
        8m53.96s real           13m22.34s user          6m54.47s sys

A misbehaving mpt controller/driver seemed to slow this down a bit, it
was logging full queue events during the initial install and continued
to do so during buildkernel.

mpt0: QUEUE FULL EVENT: Bus 0x00 Target 0x00 Depth 127
mpt0: QUEUE FULL EVENT: Bus 0x00 Target 0x00 Depth 128
mpt0: QUEUE FULL EVENT: Bus 0x00 Target 0x00 Depth 128

I think it's safe to say the x8sax tests just moved up on my TODO list
a little, unfortunately that hardware is tied up at the moment.

I'd be interested to see lmbench results if you have any you want to
share. (/usr/ports/benchmarks/lmbench)
I'll attach dmesg & lmbench results for the X8DT6/E5520 configuration.

Case

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

[Bryce Edwards]

Bingo, looks it was the Legacy USB setting in the BIOS.  I had it on
"Auto" (default) and the docs say it will disable unless there is
legacy USB devices.

I have now put all the other BIOS options to default (except enabling
AHCI) and here's what buildworld now looks like:

        2h39m29.20s real                11h35m35.36s user
 3h28m19.15s sys

Now I need to go research more about this to understand more about the
Legacy USB behavior.

::Bryce::


On Mon, May 3, 2010 at 1:34 PM, George Neville-Neil
<[hidden email]> wrote:

>
> On May 1, 2010, at 11:10 , Bryce Edwards wrote:
>
>> Hello,
>>
>> I've got a new Supermicro X58 system with an Intel Core i7 930 with 6
>> GB ram that is not performing nearly as fast as it should in many ways
>> (compiling, network transfers).  To give an example, it has been
>> building the gcc44 port for about 10 hours now and at the same time
>> rsync'ing from a Linux box on the same Gigabit network is only getting
>> throughput of between 10-25 MB/sec.  When I did a buildkernel for
>> 8-STABLE, it took 17 hours!
>>
>> In the BIOS, I have played with a few settings and some actually made
>> it worse.  What I have done now is disabled Hyperthreading and
>> Speedstep.  I have attached the dmesg details, and here's some system
>> info and then some system stats:
>>
>>
>> bryce@tahiti[~]>uname -a
>> FreeBSD tahiti.bryce.net 8.0-STABLE FreeBSD 8.0-STABLE #0: Wed Apr 28
>> 10:53:37 CDT 2010
>> [hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64
>>
>>
>>
>> bryce@tahiti[~]>cat /boot/loader.conf
>> ahci_load="YES"
>> ichsmb_load="YES"
>> smb_load="YES"
>> coretemp_load="YES"
>>
>> zfs_load="YES"
>> vfs.root.mountfrom="zfs:system"
>>
>> hint.p4tcc.0.disabled=1
>> hint.acpi_throttle.0.disabled=1
>>
>>
>> bryce@tahiti[~]>cat /etc/sysctl.conf
>> kern.timecounter.hardware=HPET
>>
>>
>> bryce@tahiti[~]>vmstat 1
>> procs      memory      page                    disks     faults         cpu
>> r b w     avm    fre   flt  re  pi  po    fr  sr ad0 ad1   in   sy
>> cs us sy id
>> 5 0 0   1068M  3478M   572   1   1   0   862   0   0   0 9370 16514
>> 16157 71 22  7
>> 5 0 0   1068M  3478M     2   0   0   0     0   0   0   0 8008 14504
>> 11716 81 17  2
>> 5 0 0   1068M  3478M     0   0   0   0     0   0   0   0 12429 22323
>> 18125 77 23  0
>> 5 0 0   1068M  3478M     0   0   0   0     0   0   0   0 12348 22125
>> 17988 73 27  0
>>
>>
>> bryce@tahiti[~]>vmstat -i
>> interrupt                          total       rate
>> irq1: atkbd0                        9291          0
>> irq17: fwohci0                         1          0
>> cpu0: timer                     75416246       2000
>> irq256: em0                    137590284       3649
>> irq257: em0                    206367605       5473
>> irq260: em0                            1          0
>> irq266: ahci0                    9892384        262
>> cpu2: timer                     75415653       2000
>> cpu3: timer                     75415702       2000
>> cpu1: timer                     75415561       2000
>> Total                          655522728      17385
>>
>>
>> bryce@tahiti[~]>netstat -I em0 -h 1
>>            input          (em0)           output
>>   packets  errs idrops      bytes    packets  errs      bytes colls
>>      7.7K     0     0        11M       7.2K     0       475K     0
>>      8.1K     0     0        12M       7.4K     0       491K     0
>>      7.8K     0     0        11M       7.2K     0       476K     0
>>
>> bryce@tahiti[/usr/adm]>iostat 1
>>       tty            ada0             ada1             ada2             cpu
>> tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
>>   0   108 22.35   3  0.07  20.61   3  0.07  58.60   0  0.00  71  0  4 17  7
>>   0   222 64.00   1  0.06  128.00   1  0.12   0.00   0  0.00  87  0  2 11  0
>
> First thing to try.  Turn off Legacy USB in the BIOS.
>
> Is this a 2 CPU machine with only one CPU installed?  Make sure your memory is all
> in the correct slots, on modern (non Front Side Bus) machines like Nehalem
> only the memory in the slots that are adjacent to a CPU are seen by the CPU.
>
> I gather you have a kernel with INVARIANTS, WITNESS and the like turned off, correct?
>
> Best,
> George
>
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"