igb(4) Raising IGB_MAX_TXD ??

[Sean Bruno-5]

We're running a service with a 82576 configured with 4 queues and a
maxed rxd/txd configuration:

http://people.freebsd.org/~sbruno/igb_stats.txt

We still see, under higher load spikes, a tendency to drop packets (I
suspect an application issue at this point, but want to attempt to
alleviate some congestion).

I note that IGB_MAX_RXD is set to 4k.  Reviewing the Intel spec shell on
the 82576 I see that the maximum value for the Descriptor Ring Length
(8.10.8) is 32k descriptors.

Does that mean I should be able to go higher that 4k?  I suspect that
even if I can, that would merely make the traffic fill more buffer
space, but maybe its enough to make it work a bit better.

Sean



_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Luigi Rizzo-5]

On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote:
> We're running a service with a 82576 configured with 4 queues and a
> maxed rxd/txd configuration:
>
> http://people.freebsd.org/~sbruno/igb_stats.txt

these stats show that over half of your incoming traffic is
made of small packets (65..127 bytes) but especially, that
the "missed packets" count is very small (18k out of 40G packets)
none of them is reported as "no_desc_avail", and only 76 are
"recv_no_buffer".

Are you dropping packets in the ip interrupt handler by chance ?
what are your settings there ?

BTW it seems that there is only one global setting for the dispatch
policy, but for instance there are two netisr_dispatch() calls
in the incoming path, one for layer2 and one for layer3.
The former has relatively little work to do and so it might
make sense to have direct dispatch, the other can be expensive
so i wonder if it wouldn't be better to use deferred dispatch.
If not, perhaps you might try to reduce the rx_processing_limit
to bring down the load on the intr thread.

        > sysctl net | grep intr
        net.inet.ip.intr_queue_maxlen: 256
        net.inet.ip.intr_queue_drops: 253
        > sysctl net.isr
        net.isr.numthreads: 1
        net.isr.maxprot: 16
        net.isr.defaultqlimit: 256
        net.isr.maxqlimit: 10240
        net.isr.bindthreads: 0
        net.isr.maxthreads: 1
        net.isr.direct: 0
        net.isr.direct_force: 0
        net.isr.dispatch: direct

> We still see, under higher load spikes, a tendency to drop packets (I
> suspect an application issue at this point, but want to attempt to
> alleviate some congestion).
>
> I note that IGB_MAX_RXD is set to 4k.  Reviewing the Intel spec shell on
> the 82576 I see that the maximum value for the Descriptor Ring Length
> (8.10.8) is 32k descriptors.
>
> Does that mean I should be able to go higher that 4k?  I suspect that
> even if I can, that would merely make the traffic fill more buffer
> space, but maybe its enough to make it work a bit better.

With your numbers i doubt that raising the queue size helps.

cheers
luigi

> Sean
>
>
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Sean Bruno-5]

On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote:

> On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote:
> > We're running a service with a 82576 configured with 4 queues and a
> > maxed rxd/txd configuration:
> >
> > http://people.freebsd.org/~sbruno/igb_stats.txt
>
> these stats show that over half of your incoming traffic is
> made of small packets (65..127 bytes) but especially, that
> the "missed packets" count is very small (18k out of 40G packets)
> none of them is reported as "no_desc_avail", and only 76 are
> "recv_no_buffer".
>
> Are you dropping packets in the ip interrupt handler by chance ?
> what are your settings there ?
>

nope, doesn't look like it.  

http://people.freebsd.org/~sbruno/igb_ip_stats.txt

> BTW it seems that there is only one global setting for the dispatch
> policy, but for instance there are two netisr_dispatch() calls
> in the incoming path, one for layer2 and one for layer3.
> The former has relatively little work to do and so it might
> make sense to have direct dispatch, the other can be expensive
> so i wonder if it wouldn't be better to use deferred dispatch.
> If not, perhaps you might try to reduce the rx_processing_limit
> to bring down the load on the intr thread.

I don't really see any issue with horsepower on this host at the moment
with 4 queues.  I mean it looks a little something like this under high
load:

http://people.freebsd.org/~sbruno/igb_top.txt

I guess my question still stands though, since the ethernet controller
is reporting that it doesn't have any more descriptors available is the
hardcoded 4k max descriptors a limit that an be raised?

> With your numbers i doubt that raising the queue size helps.

Indeed, you're probably right and this is more than likely an
application problem that will have to be resolved.  However, I'm still
curious if the MAX_RXD/TXD is really 4k or if the documentation is
correct and we can raise it to 32k for testing?

Sean

[Jack Vogel]

The MAX value is something I set, not a hardware thing, it was based on
reports
I had from the various driver engineers in our org. If you increase the
ring size
you might run into other performance issues, however there's nothing
stopping
you from trying. Just be aware that its not something that's been tested.

Let me know how it goes please :)

Jack


On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno <[hidden email]> wrote:

> On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote:
> > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote:
> > > We're running a service with a 82576 configured with 4 queues and a
> > > maxed rxd/txd configuration:
> > >
> > > http://people.freebsd.org/~sbruno/igb_stats.txt
> >
> > these stats show that over half of your incoming traffic is
> > made of small packets (65..127 bytes) but especially, that
> > the "missed packets" count is very small (18k out of 40G packets)
> > none of them is reported as "no_desc_avail", and only 76 are
> > "recv_no_buffer".
> >
> > Are you dropping packets in the ip interrupt handler by chance ?
> > what are your settings there ?
> >
> nope, doesn't look like it.
>
> http://people.freebsd.org/~sbruno/igb_ip_stats.txt
>
> > BTW it seems that there is only one global setting for the dispatch
> > policy, but for instance there are two netisr_dispatch() calls
> > in the incoming path, one for layer2 and one for layer3.
> > The former has relatively little work to do and so it might
> > make sense to have direct dispatch, the other can be expensive
> > so i wonder if it wouldn't be better to use deferred dispatch.
> > If not, perhaps you might try to reduce the rx_processing_limit
> > to bring down the load on the intr thread.
>
> I don't really see any issue with horsepower on this host at the moment
> with 4 queues.  I mean it looks a little something like this under high
> load:
>
> http://people.freebsd.org/~sbruno/igb_top.txt
>
> I guess my question still stands though, since the ethernet controller
> is reporting that it doesn't have any more descriptors available is the
> hardcoded 4k max descriptors a limit that an be raised?
>
> > With your numbers i doubt that raising the queue size helps.
>
> Indeed, you're probably right and this is more than likely an
> application problem that will have to be resolved.  However, I'm still
> curious if the MAX_RXD/TXD is really 4k or if the documentation is
> correct and we can raise it to 32k for testing?
>
> Sean
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Sean Bruno-5]

ok, good.  that at least confirms that I correctly translated between
the driver code and documented specification.  

I will try 8k as a test for now and see how that runs.  

sean

On Wed, 2012-04-18 at 09:46 -0700, Jack Vogel wrote:

> The MAX value is something I set, not a hardware thing, it was based
> on reports
> I had from the various driver engineers in our org. If you increase
> the ring size
> you might run into other performance issues, however there's nothing
> stopping
> you from trying. Just be aware that its not something that's been
> tested.
>
> Let me know how it goes please :)
>
> Jack
>
>
> On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno <[hidden email]>
> wrote:
>         On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote:
>         > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote:
>         > > We're running a service with a 82576 configured with 4
>         queues and a
>         > > maxed rxd/txd configuration:
>         > >
>         > > http://people.freebsd.org/~sbruno/igb_stats.txt
>         >
>         > these stats show that over half of your incoming traffic is
>         > made of small packets (65..127 bytes) but especially, that
>         > the "missed packets" count is very small (18k out of 40G
>         packets)
>         > none of them is reported as "no_desc_avail", and only 76 are
>         > "recv_no_buffer".
>         >
>         > Are you dropping packets in the ip interrupt handler by
>         chance ?
>         > what are your settings there ?
>         >
>        
>         nope, doesn't look like it.
>        
>         http://people.freebsd.org/~sbruno/igb_ip_stats.txt
>        
>         > BTW it seems that there is only one global setting for the
>         dispatch
>         > policy, but for instance there are two netisr_dispatch()
>         calls
>         > in the incoming path, one for layer2 and one for layer3.
>         > The former has relatively little work to do and so it might
>         > make sense to have direct dispatch, the other can be
>         expensive
>         > so i wonder if it wouldn't be better to use deferred
>         dispatch.
>         > If not, perhaps you might try to reduce the
>         rx_processing_limit
>         > to bring down the load on the intr thread.
>        
>        
>         I don't really see any issue with horsepower on this host at
>         the moment
>         with 4 queues.  I mean it looks a little something like this
>         under high
>         load:
>        
>         http://people.freebsd.org/~sbruno/igb_top.txt
>        
>         I guess my question still stands though, since the ethernet
>         controller
>         is reporting that it doesn't have any more descriptors
>         available is the
>         hardcoded 4k max descriptors a limit that an be raised?
>        
>         > With your numbers i doubt that raising the queue size helps.
>        
>        
>         Indeed, you're probably right and this is more than likely an
>         application problem that will have to be resolved.  However,
>         I'm still
>         curious if the MAX_RXD/TXD is really 4k or if the
>         documentation is
>         correct and we can raise it to 32k for testing?
>        
>         Sean
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Sean Bruno-5]

On Wed, 2012-04-18 at 09:49 -0700, Sean Bruno wrote:
> ok, good.  that at least confirms that I correctly translated between
> the driver code and documented specification.  
>
> I will try 8k as a test for now and see how that runs.  
>
> sean

For now, I've patched one front end server with:
/usr/src/sys/dev/e1000/if_igb.h:#define IGB_MAX_RXD             4096 * 4

And adjusted hw.igb.rxd: 8192

So far so good, been running in production for a couple of hours so the
"smoke test" for this setting seems to be happy.

We'll continue to adjust and test tomorrow during higher load
conditions.

Sean

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On Wednesday, April 18, 2012 7:40:17 pm Sean Bruno wrote:

>
> On Wed, 2012-04-18 at 09:49 -0700, Sean Bruno wrote:
> > ok, good.  that at least confirms that I correctly translated between
> > the driver code and documented specification.  
> >
> > I will try 8k as a test for now and see how that runs.  
> >
> > sean
>
> For now, I've patched one front end server with:
> /usr/src/sys/dev/e1000/if_igb.h:#define IGB_MAX_RXD             4096 * 4
>
> And adjusted hw.igb.rxd: 8192
>
> So far so good, been running in production for a couple of hours so the
> "smoke test" for this setting seems to be happy.
>
> We'll continue to adjust and test tomorrow during higher load
> conditions.

FWIW, at my current employer we run with both rxd and txd cranked up to 32k
(we had to patch the driver as you suggested) and have not had any problems
doing that for a couple of years now.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Jack Vogel]

OH, well that's interesting to know, thanks John.

Jack


On Thu, Apr 19, 2012 at 5:22 AM, John Baldwin <[hidden email]> wrote:

> On Wednesday, April 18, 2012 7:40:17 pm Sean Bruno wrote:
> >
> > On Wed, 2012-04-18 at 09:49 -0700, Sean Bruno wrote:
> > > ok, good.  that at least confirms that I correctly translated between
> > > the driver code and documented specification.
> > >
> > > I will try 8k as a test for now and see how that runs.
> > >
> > > sean
> >
> > For now, I've patched one front end server with:
> > /usr/src/sys/dev/e1000/if_igb.h:#define IGB_MAX_RXD             4096 * 4
> >
> > And adjusted hw.igb.rxd: 8192
> >
> > So far so good, been running in production for a couple of hours so the
> > "smoke test" for this setting seems to be happy.
> >
> > We'll continue to adjust and test tomorrow during higher load
> > conditions.
>
> FWIW, at my current employer we run with both rxd and txd cranked up to 32k
> (we had to patch the driver as you suggested) and have not had any problems
> doing that for a couple of years now.
>
> --
> John Baldwin
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Sean Bruno-5]

On Thu, 2012-04-19 at 07:09 -0700, Jack Vogel wrote:
> OH, well that's interesting to know, thanks John.
>
> Jack
>


Front end box looks pretty happy today at 8k descriptors.

http://people.freebsd.org/~sbruno/igb_8k_stats.txt

Under peak, we're approaching 20MBytes/sec in and out of the
interface.  :-)  Nifty.

-bash-4.2$ netstat 1
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
     59542     0     0   18189602      59131     0   19884085     0
     58941     0     0   18036651      58673     0   19702671     0
     58790     0     0   18069235      58422     0   19897858     0
     58226     0     0   17948175      57969     0   19648810     0
     58689     0     0   18167855      58479     0   19909843     0
     58633     0     0   17952951      58437     0   19760197     0
     61019     0     0   18779030      60592     0   20394481     0
     56696     0     0   17647407      56552     0   19261155     0
     58853     0     0   18186019      58530     0   19886197     0
     58739     0     0   18314790      58768     0   20165654     0
     58748     0     0   18267243      58539     0   20016668     0
     58672     0     0   17914657      58378     0   19558833     0
     59885     0     0   18332641      59780     0   20239241     0


We're going to crank one server up to 8 igb queues and
hw.igb.max_rxd/txd to 32k and see what blows up.

Sean


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Jack Vogel]

On Thu, Apr 19, 2012 at 12:26 PM, Sean Bruno <[hidden email]> wrote:

> On Thu, 2012-04-19 at 07:09 -0700, Jack Vogel wrote:
> > OH, well that's interesting to know, thanks John.
> >
> > Jack
> >
>
>
> Front end box looks pretty happy today at 8k descriptors.
>
> http://people.freebsd.org/~sbruno/igb_8k_stats.txt
>
> Under peak, we're approaching 20MBytes/sec in and out of the
> interface.  :-)  Nifty.
>
> -bash-4.2$ netstat 1
>            input        (Total)           output
>   packets  errs idrops      bytes    packets  errs      bytes colls
>     59542     0     0   18189602      59131     0   19884085     0
>     58941     0     0   18036651      58673     0   19702671     0
>     58790     0     0   18069235      58422     0   19897858     0
>     58226     0     0   17948175      57969     0   19648810     0
>     58689     0     0   18167855      58479     0   19909843     0
>     58633     0     0   17952951      58437     0   19760197     0
>     61019     0     0   18779030      60592     0   20394481     0
>     56696     0     0   17647407      56552     0   19261155     0
>     58853     0     0   18186019      58530     0   19886197     0
>     58739     0     0   18314790      58768     0   20165654     0
>     58748     0     0   18267243      58539     0   20016668     0
>     58672     0     0   17914657      58378     0   19558833     0
>     59885     0     0   18332641      59780     0   20239241     0
>
>
> We're going to crank one server up to 8 igb queues and
> hw.igb.max_rxd/txd to 32k and see what blows up.
>
> Sean
>
>
>

Great, look forward to the results. Thanks Sean.

Jack
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[vijay singh-5]

>> FWIW, at my current employer we run with both rxd and txd cranked up to 32k
>> (we had to patch the driver as you suggested) and have not had any problems
>> doing that for a couple of years now.

John et al, is this limit specific to igb, or is it applicable to the
ixgbe as well. From the 8.x driver code I see:

100/*
101 * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the
102 * number of transmit descriptors allocated by the driver. Increasing this
103 * value allows the driver to queue more transmits. Each descriptor is 16
104 * bytes. Performance tests have show the 2K value to be optimal for top
105 * performance.
106 */
107#define DEFAULT_TXD 1024
108#define PERFORM_TXD 2048
109#define MAX_TXD 4096
110#define MIN_TXD 64

Can the igxbe be extended beyond 4k transmit descriptors well?

-vijay
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Jack Vogel]

Yes, I'm not sure what the actual hard limit is, will check that on Monday,
but
you can go over 4K, it was just a limit I created.

Jack


On Sat, Apr 21, 2012 at 10:51 PM, Vijay Singh <[hidden email]> wrote:

> >> FWIW, at my current employer we run with both rxd and txd cranked up to
> 32k
> >> (we had to patch the driver as you suggested) and have not had any
> problems
> >> doing that for a couple of years now.
>
> John et al, is this limit specific to igb, or is it applicable to the
> ixgbe as well. From the 8.x driver code I see:
>
> 100/*
> 101 * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is
> the
> 102 * number of transmit descriptors allocated by the driver. Increasing
> this
> 103 * value allows the driver to queue more transmits. Each descriptor is
> 16
> 104 * bytes. Performance tests have show the 2K value to be optimal for top
> 105 * performance.
> 106 */
> 107#define DEFAULT_TXD  1024
> 108#define PERFORM_TXD  2048
> 109#define MAX_TXD              4096
> 110#define MIN_TXD              64
>
> Can the igxbe be extended beyond 4k transmit descriptors well?
>
> -vijay
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"