Strange CPU distributionat very high level bandwidth

[fabrizio]

Hi all

i am going on with some performance tests on a 10gbe network card with FreeBSD.

I am doing this test: I send UDP traffic to be forwarded to the other port of the card on both the card ports.
Using 1492-long packets i am uppering the number  of packets per second i sent In order to see wich is the maximum bandwidth (or pps) the system can support without losses.

The limit seems to be about 1890Mbps per port (3870 Mbps total).
Looking more in deep the CPU behaviour i see this :
  - uppering the sent pps results in uppering the intterrupt time (about 90%)
  - when i am very strict to the limit, interrupt time falls to about 10% and CPU is always (85%) in system (rx/tx driver procedure)

Questions:
- Is not the AIM intended to contrast this behaviour to limit interrupts sent to CPU? (nothing changes if i disable it)
- Why does the system start loosing pkts in that condition?
- Why does the system seem to perform better when it is managing more context switches?



These are my system details:

- HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core installed.

- Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit Ethernet PCI Express Server Adapter Intel(r) based (chip 82598EB).

- FreeBSD 7.2-RELEASE (64 bit)

Driver ixgbe-1.8.6

hw.intr_storm_threshold:2000000

dev.ix.0.low_latency: 128
dev.ix.0.ave_latency: 400
dev.ix.0.bulk_latency: 1200
dev.ix.1.low_latency: 128
dev.ix.1.ave_latency: 400
dev.ix.1.bulk_latency: 1200

------------------------------------------------------------------
Telecom Italia
Fabrizio INVERNIZZI
Technology - TILAB
Accesso Fisso e Trasporto
Via Reiss Romoli, 274 10148 Torino
Tel.  [phone]
Mob. [phone]
Fax [phone]

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks.

[cid:[hidden email]]Rispetta l'ambiente. Non stampare questa mail se non ? necessario.


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

[Gergely CZUCZY]

Hello,

Just a question. May I ask how many pps is this traffic (packet per
second). Forward performance actually depends on the pps rate and not
on the bandwidth usage as far as my experience goes. As I've calculated
by your given data, it might be around 166Kpps, but i might be wrong
there.

On Wed, 19 Aug 2009 12:13:37 +0200
Invernizzi Fabrizio <[hidden email]> wrote:

> Hi all
>
> i am going on with some performance tests on a 10gbe network card
> with FreeBSD.
>
> I am doing this test: I send UDP traffic to be forwarded to the other
> port of the card on both the card ports. Using 1492-long packets i am
> uppering the number  of packets per second i sent In order to see
> wich is the maximum bandwidth (or pps) the system can support without
> losses.
>
> The limit seems to be about 1890Mbps per port (3870 Mbps total).
> Looking more in deep the CPU behaviour i see this :
>   - uppering the sent pps results in uppering the intterrupt time
> (about 90%)
>   - when i am very strict to the limit, interrupt time falls to about
> 10% and CPU is always (85%) in system (rx/tx driver procedure)
>
> Questions:
> - Is not the AIM intended to contrast this behaviour to limit
> interrupts sent to CPU? (nothing changes if i disable it)
> - Why does the system start loosing pkts in that condition?
> - Why does the system seem to perform better when it is managing more
> context switches?
>
>
>
> These are my system details:
>
> - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2
> cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core
> installed.
>
> - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit
> Ethernet PCI Express Server Adapter Intel(r) based (chip 82598EB).
>
> - FreeBSD 7.2-RELEASE (64 bit)
>
> Driver ixgbe-1.8.6
>
> hw.intr_storm_threshold:2000000
>
> dev.ix.0.low_latency: 128
> dev.ix.0.ave_latency: 400
> dev.ix.0.bulk_latency: 1200
> dev.ix.1.low_latency: 128
> dev.ix.1.ave_latency: 400
> dev.ix.1.bulk_latency: 1200
>
> ------------------------------------------------------------------
> Telecom Italia
> Fabrizio INVERNIZZI
> Technology - TILAB
> Accesso Fisso e Trasporto
> Via Reiss Romoli, 274 10148 Torino
> Tel.  [phone]
> Mob. [phone]
> Fax [phone]
>
> Questo messaggio e i suoi allegati sono indirizzati esclusivamente
> alle persone indicate. La diffusione, copia o qualsiasi altra azione
> derivante dalla conoscenza di queste informazioni sono rigorosamente
> vietate. Qualora abbiate ricevuto questo documento per errore siete
> cortesemente pregati di darne immediata comunicazione al mittente e
> di provvedere alla sua distruzione, Grazie.
>
> This e-mail and any attachments is confidential and may contain
> privileged information intended for the addressee(s) only.
> Dissemination, copying, printing or use by anybody else is
> unauthorised. If you are not the intended recipient, please delete
> this message and any attachments and advise the sender by return
> e-mail, Thanks.
>
> [cid:[hidden email]]Rispetta
> l'ambiente. Non stampare questa mail se non ? necessario.
>
>

--
Sincerely,
Gergely CZUCZY
Harmless Digital Bt

[phone]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

[fabrizio]

There is an error in my previous email: the upper limit I see is 5722 Mbps (total).

I am sending 239700 pps  per interface, 1492-bytes long.

Fabrizio

> -----Original Message-----
> From: Gergely CZUCZY [mailto:[hidden email]]
> Sent: mercoledì 19 agosto 2009 12.57
> To: Invernizzi Fabrizio
> Cc: [hidden email]
> Subject: Re: Strange CPU distributionat very high level bandwidth
>
> Hello,
>
> Just a question. May I ask how many pps is this traffic
> (packet per second). Forward performance actually depends on
> the pps rate and not on the bandwidth usage as far as my
> experience goes. As I've calculated by your given data, it
> might be around 166Kpps, but i might be wrong there.
>
> On Wed, 19 Aug 2009 12:13:37 +0200
> Invernizzi Fabrizio <[hidden email]> wrote:
>
> > Hi all
> >
> > i am going on with some performance tests on a 10gbe
> network card with
> > FreeBSD.
> >
> > I am doing this test: I send UDP traffic to be forwarded to
> the other
> > port of the card on both the card ports. Using 1492-long
> packets i am
> > uppering the number  of packets per second i sent In order
> to see wich
> > is the maximum bandwidth (or pps) the system can support without
> > losses.
> >
> > The limit seems to be about 1890Mbps per port (3870 Mbps total).
> > Looking more in deep the CPU behaviour i see this :
> >   - uppering the sent pps results in uppering the intterrupt time
> > (about 90%)
> >   - when i am very strict to the limit, interrupt time
> falls to about
> > 10% and CPU is always (85%) in system (rx/tx driver procedure)
> >
> > Questions:
> > - Is not the AIM intended to contrast this behaviour to limit
> > interrupts sent to CPU? (nothing changes if i disable it)
> > - Why does the system start loosing pkts in that condition?
> > - Why does the system seem to perform better when it is
> managing more
> > context switches?
> >
> >
> >
> > These are my system details:
> >
> > - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed:
> 1333 MHz, L2
> > cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core
> > installed.
> >
> > - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR)
> 10 Gigabit
> > Ethernet PCI Express Server Adapter Intel(r) based (chip 82598EB).
> >
> > - FreeBSD 7.2-RELEASE (64 bit)
> >
> > Driver ixgbe-1.8.6
> >
> > hw.intr_storm_threshold:2000000
> >
> > dev.ix.0.low_latency: 128
> > dev.ix.0.ave_latency: 400
> > dev.ix.0.bulk_latency: 1200
> > dev.ix.1.low_latency: 128
> > dev.ix.1.ave_latency: 400
> > dev.ix.1.bulk_latency: 1200
> >
> > ------------------------------------------------------------------
> > Telecom Italia
> > Fabrizio INVERNIZZI
> > Technology - TILAB
> > Accesso Fisso e Trasporto
> > Via Reiss Romoli, 274 10148 Torino
> > Tel.  [phone]
> > Mob. [phone]
> > Fax [phone]
> >
> > Questo messaggio e i suoi allegati sono indirizzati esclusivamente
> > alle persone indicate. La diffusione, copia o qualsiasi
> altra azione
> > derivante dalla conoscenza di queste informazioni sono
> rigorosamente
> > vietate. Qualora abbiate ricevuto questo documento per errore siete
> > cortesemente pregati di darne immediata comunicazione al
> mittente e di
> > provvedere alla sua distruzione, Grazie.
> >
> > This e-mail and any attachments is confidential and may contain
> > privileged information intended for the addressee(s) only.
> > Dissemination, copying, printing or use by anybody else is
> > unauthorised. If you are not the intended recipient, please delete
> > this message and any attachments and advise the sender by return
> > e-mail, Thanks.
> >
> > [cid:[hidden email]]Rispetta
> > l'ambiente. Non stampare questa mail se non ? necessario.
> >
> >
>
>
>
> --
> Sincerely,
> Gergely CZUCZY
> Harmless Digital Bt
>
> [phone]
>

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

[Ivan Voras-7]

Invernizzi Fabrizio wrote:

> Hi all
>
> i am going on with some performance tests on a 10gbe network card with FreeBSD.
>
> I am doing this test: I send UDP traffic to be forwarded to the other port of the card on both the card ports.
> Using 1492-long packets i am uppering the number  of packets per second i sent In order to see wich is the maximum bandwidth (or pps) the system can support without losses.
>
> The limit seems to be about 1890Mbps per port (3870 Mbps total).
> Looking more in deep the CPU behaviour i see this :
>   - uppering the sent pps results in uppering the intterrupt time (about 90%)
>   - when i am very strict to the limit, interrupt time falls to about 10% and CPU is always (85%) in system (rx/tx driver procedure)
>
> Questions:
> - Is not the AIM intended to contrast this behaviour to limit interrupts sent to CPU? (nothing changes if i disable it)
> - Why does the system start loosing pkts in that condition?
> - Why does the system seem to perform better when it is managing more context switches?
>

> - FreeBSD 7.2-RELEASE (64 bit)

One idea for you, not directly tied to forwarding as is but to the
recent development of multithreaded packet acceptance code, is to use
8.x (currently in BETA so usual precautions about debugging being
enabled apply) and then play with netisr and worker thread settings.

See the source here:

http://svn.freebsd.org/viewvc/base/head/sys/net/netisr.c?view=markup&pathrev=195078

and the comments starting at "Three direct dispatch policies are supported".

The code is experimental and thus disabled in 8.0 unless a combination
of the following loader tunables are set:

net.isr.direct_force
net.isr.direct
net.isr.maxthreads
net.isr.bindthreads

I think you can start simply by turning off net.isr.direct_force and
then start increasing net.isr.maxthreads until the benefits (if any) go
away. Since it is experimental code, your benchmarks would be nice to have.

[fabrizio]

Thanks for your suggestion.

I hope to have time to do some tests on 8.0  and send some result on the ML next week.



------------------------------------------------------------------
Telecom Italia
Fabrizio INVERNIZZI
Technology - TILAB
Accesso Fisso e Trasporto
Via Reiss Romoli, 274 10148 Torino
Tel.  [phone]
Mob. [phone]
Fax [phone]

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Ivan Voras
> Sent: venerdì 21 agosto 2009 1.14
> To: [hidden email]
> Subject: Re: Strange CPU distributionat very high level bandwidth
>
> Invernizzi Fabrizio wrote:
> > Hi all
> >
> > i am going on with some performance tests on a 10gbe
> network card with FreeBSD.
> >
> > I am doing this test: I send UDP traffic to be forwarded to
> the other port of the card on both the card ports.
> > Using 1492-long packets i am uppering the number  of
> packets per second i sent In order to see wich is the maximum
> bandwidth (or pps) the system can support without losses.
> >
> > The limit seems to be about 1890Mbps per port (3870 Mbps total).
> > Looking more in deep the CPU behaviour i see this :
> >   - uppering the sent pps results in uppering the
> intterrupt time (about 90%)
> >   - when i am very strict to the limit, interrupt time
> falls to about
> > 10% and CPU is always (85%) in system (rx/tx driver procedure)
> >
> > Questions:
> > - Is not the AIM intended to contrast this behaviour to limit
> > interrupts sent to CPU? (nothing changes if i disable it)
> > - Why does the system start loosing pkts in that condition?
> > - Why does the system seem to perform better when it is
> managing more context switches?
> >
>
> > - FreeBSD 7.2-RELEASE (64 bit)
>
> One idea for you, not directly tied to forwarding as is but
> to the recent development of multithreaded packet acceptance
> code, is to use 8.x (currently in BETA so usual precautions
> about debugging being enabled apply) and then play with
> netisr and worker thread settings.
>
> See the source here:
>
> http://svn.freebsd.org/viewvc/base/head/sys/net/netisr.c?view=

markup&pathrev=195078

>
> and the comments starting at "Three direct dispatch policies
> are supported".
>
> The code is experimental and thus disabled in 8.0 unless a
> combination of the following loader tunables are set:
>
> net.isr.direct_force
> net.isr.direct
> net.isr.maxthreads
> net.isr.bindthreads
>
> I think you can start simply by turning off
> net.isr.direct_force and then start increasing
> net.isr.maxthreads until the benefits (if any) go away. Since
> it is experimental code, your benchmarks would be nice to have.
>
>
>
>

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"