ipfw meets netmap (6.5 Mpps in userspace)

[Luigi Rizzo-5]

I just finished a netmap-enabled version of ipfw/dummynet, which
runs in userspace and is able to process over 6 million packets per
second (Mpps) with simple rulesets, and over 2.2 Mpps through
dummynet pipes (tested on an i7-3400 connected to VALE ports;
VALE is a software switch part of netmap).
You can find it at

        http://info.iet.unipi.it/~luigi/dummynet/#8696

It works on FreeBSD and Linux.

It even run on OSX, but there is no netmap support there;
any interest, Apple ? In any case, it should be simple to
adapt the network backend to talk to other devices.

To run this you might want to use the most recent version of netmap,
also recently updated so it works with recent kernels

        http://info.iet.unipi.it/~luigi/netmap/

(the code is already in FreeBSD HEAD).

On passing, I have also updated the dummynet distributions
for Linux and Windows, which should now compile for most
recent version of Linux, and for Windows 32 and 64 bit:

        http://info.iet.unipi.it/~luigi/dummynet/

cheers
luigi
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Olivier Cochard-Labbé-2]

On Mon, Aug 13, 2012 at 1:17 PM, Luigi Rizzo <[hidden email]> wrote:
> I just finished a netmap-enabled version of ipfw/dummynet, which
> runs in userspace and is able to process over 6 million packets per
> second (Mpps) with simple rulesets, and over 2.2 Mpps through
> dummynet pipes (tested on an i7-3400 connected to VALE ports;
> VALE is a software switch part of netmap).

Hi,

Reading the README file: "Real packet I/O is possible using netmap",
Can we use it for high-speed firewalling among real NICs now?

Can you confirm that we just need:
1. An up-to-date FreeBSD -current (build from source synced the
2012-08-03 mininum) with netmap module loaded;
2. netmap compliant NICs (ixgbe, e1000 or re);
3. compile, configure and start ipfw-user.

Can ipfw-user be directly connected to two netmap-enabled NICs in
place of vale switches->netmap bridge->NIC ?

Thanks,

Olivier
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Luigi Rizzo-5]

On Mon, Aug 13, 2012 at 02:42:43PM +0200, Olivier Cochard-Labb? wrote:

> On Mon, Aug 13, 2012 at 1:17 PM, Luigi Rizzo <[hidden email]> wrote:
> > I just finished a netmap-enabled version of ipfw/dummynet, which
> > runs in userspace and is able to process over 6 million packets per
> > second (Mpps) with simple rulesets, and over 2.2 Mpps through
> > dummynet pipes (tested on an i7-3400 connected to VALE ports;
> > VALE is a software switch part of netmap).
>
> Hi,
>
> Reading the README file: "Real packet I/O is possible using netmap",
> Can we use it for high-speed firewalling among real NICs now?
>
> Can you confirm that we just need:
> 1. An up-to-date FreeBSD -current (build from source synced the
> 2012-08-03 mininum) with netmap module loaded;
> 2. netmap compliant NICs (ixgbe, e1000 or re);
> 3. compile, configure and start ipfw-user.
>
> Can ipfw-user be directly connected to two netmap-enabled NICs in
> place of vale switches->netmap bridge->NIC ?

yes to all three (though i have not tried yet as i do not have
access to 10G hardware now, vale ports behave exactly the same
as a real card).
Whoever feels like trying, performance numbers are welcome.
I'll prepare a picobsd image with all the tools shortly.

cheers
luigi

>
> Olivier
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Özkan KIRIK]

does igb driver supported by netmap ? if yes, how multiqueue support works?

On Mon, Aug 13, 2012 at 4:08 PM, Luigi Rizzo <[hidden email]> wrote:

> On Mon, Aug 13, 2012 at 02:42:43PM +0200, Olivier Cochard-Labb? wrote:
>> On Mon, Aug 13, 2012 at 1:17 PM, Luigi Rizzo <[hidden email]> wrote:
>> > I just finished a netmap-enabled version of ipfw/dummynet, which
>> > runs in userspace and is able to process over 6 million packets per
>> > second (Mpps) with simple rulesets, and over 2.2 Mpps through
>> > dummynet pipes (tested on an i7-3400 connected to VALE ports;
>> > VALE is a software switch part of netmap).
>>
>> Hi,
>>
>> Reading the README file: "Real packet I/O is possible using netmap",
>> Can we use it for high-speed firewalling among real NICs now?
>>
>> Can you confirm that we just need:
>> 1. An up-to-date FreeBSD -current (build from source synced the
>> 2012-08-03 mininum) with netmap module loaded;
>> 2. netmap compliant NICs (ixgbe, e1000 or re);
>> 3. compile, configure and start ipfw-user.
>>
>> Can ipfw-user be directly connected to two netmap-enabled NICs in
>> place of vale switches->netmap bridge->NIC ?
>
> yes to all three (though i have not tried yet as i do not have
> access to 10G hardware now, vale ports behave exactly the same
> as a real card).
> Whoever feels like trying, performance numbers are welcome.
> I'll prepare a picobsd image with all the tools shortly.
>
> cheers
> luigi
>
>>
>> Olivier
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Luigi Rizzo-5]

On Mon, Aug 13, 2012 at 03:54:41PM +0300, ?zkan KIRIK wrote:
> does igb driver supported by netmap ? if yes, how multiqueue support works?

yes igb is supported, and multiqueue is supported (ipfw-user
just reads from all queues).
Note that igb is a 1Gbit/card peaking at 1.488 Mpps, which is
much slower than what ipfw-user can deal with, so you do not
really need multiqueue.

You might find some benefits at 10G if you run multiple ipfw-user
instances, one per queue, bound to separate cores
(you need some small changes in the netmap I/O routines to
bind to individual queues).

cheers
luigi

> On Mon, Aug 13, 2012 at 4:08 PM, Luigi Rizzo <[hidden email]> wrote:
> > On Mon, Aug 13, 2012 at 02:42:43PM +0200, Olivier Cochard-Labb? wrote:
> >> On Mon, Aug 13, 2012 at 1:17 PM, Luigi Rizzo <[hidden email]> wrote:
> >> > I just finished a netmap-enabled version of ipfw/dummynet, which
> >> > runs in userspace and is able to process over 6 million packets per
> >> > second (Mpps) with simple rulesets, and over 2.2 Mpps through
> >> > dummynet pipes (tested on an i7-3400 connected to VALE ports;
> >> > VALE is a software switch part of netmap).
> >>
> >> Hi,
> >>
> >> Reading the README file: "Real packet I/O is possible using netmap",
> >> Can we use it for high-speed firewalling among real NICs now?
> >>
> >> Can you confirm that we just need:
> >> 1. An up-to-date FreeBSD -current (build from source synced the
> >> 2012-08-03 mininum) with netmap module loaded;
> >> 2. netmap compliant NICs (ixgbe, e1000 or re);
> >> 3. compile, configure and start ipfw-user.
> >>
> >> Can ipfw-user be directly connected to two netmap-enabled NICs in
> >> place of vale switches->netmap bridge->NIC ?
> >
> > yes to all three (though i have not tried yet as i do not have
> > access to 10G hardware now, vale ports behave exactly the same
> > as a real card).
> > Whoever feels like trying, performance numbers are welcome.
> > I'll prepare a picobsd image with all the tools shortly.
> >
> > cheers
> > luigi
> >
> >>
> >> Olivier
> > _______________________________________________
> > [hidden email] mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Nikolay Denev]

On Aug 13, 2012, at 2:17 PM, Luigi Rizzo <[hidden email]> wrote:

> I just finished a netmap-enabled version of ipfw/dummynet, which
> runs in userspace and is able to process over 6 million packets per
> second (Mpps) with simple rulesets, and over 2.2 Mpps through
> dummynet pipes (tested on an i7-3400 connected to VALE ports;
> VALE is a software switch part of netmap).
> You can find it at
>
> http://info.iet.unipi.it/~luigi/dummynet/#8696
>
> It works on FreeBSD and Linux.
>
> It even run on OSX, but there is no netmap support there;
> any interest, Apple ? In any case, it should be simple to
> adapt the network backend to talk to other devices.
>
> To run this you might want to use the most recent version of netmap,
> also recently updated so it works with recent kernels
>
> http://info.iet.unipi.it/~luigi/netmap/
>
> (the code is already in FreeBSD HEAD).
>
> On passing, I have also updated the dummynet distributions
> for Linux and Windows, which should now compile for most
> recent version of Linux, and for Windows 32 and 64 bit:
>
> http://info.iet.unipi.it/~luigi/dummynet/
>
> cheers
> luigi
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"

Hi Luigi,

I've tried copying netmap sources from HEAD to the latest RELENG_9 branch, but running
kipfw with two VALE interfaces produced immediate kernel panic (transcribed by hand, possible typos):

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x28
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff808274da
stack pointer = 0x28:0xffffff88df1a5530
frame pointer = 0x28:0xffffff88df1a5550
code segment = base 0x0, limit 0xfffff, type 0x1b
                                = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 2349 (kipfw)
[ thread pid 2349 tid 100612 ]
Stopped at ifunit_ref+0x2a:
db> bt
Tracing pid 2349 tid 100612 td 0xfffffe0094fd1000
ifunit_ref() at ifunit_ref+0x2a
ifioctl() at ifioctl+0x1cd
netmap_ioctl() at netmap_ioctl+0x121
devfs_ioctl_f() at devfs_ioctl_f+0x7b
kern_ioctl() at kern_ioctl+0x115
sys_ioctl() at sys_ioctl+0xfd
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800b5eaec, rsp = 0x7fffffffd
888, rbp = 0xc0206911 ---



Anyways, many thanks for you great work on NETMAP and not the super cool user-space IPFW!

Cheers,
Nikolay_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Adrian Chadd-2]

Hi!

This stuff is very cool.

So why is it that we can do such ridiculous packet processing rates
via netmap and ipfw in userland but not in the traditional driver path
in-kernel?

(I think I know the answer, I'm just tossing it out there for discussion.)

THanks,



Adrian


On 13 August 2012 04:17, Luigi Rizzo <[hidden email]> wrote:

> I just finished a netmap-enabled version of ipfw/dummynet, which
> runs in userspace and is able to process over 6 million packets per
> second (Mpps) with simple rulesets, and over 2.2 Mpps through
> dummynet pipes (tested on an i7-3400 connected to VALE ports;
> VALE is a software switch part of netmap).
> You can find it at
>
>         http://info.iet.unipi.it/~luigi/dummynet/#8696
>
> It works on FreeBSD and Linux.
>
> It even run on OSX, but there is no netmap support there;
> any interest, Apple ? In any case, it should be simple to
> adapt the network backend to talk to other devices.
>
> To run this you might want to use the most recent version of netmap,
> also recently updated so it works with recent kernels
>
>         http://info.iet.unipi.it/~luigi/netmap/
>
> (the code is already in FreeBSD HEAD).
>
> On passing, I have also updated the dummynet distributions
> for Linux and Windows, which should now compile for most
> recent version of Linux, and for Windows 32 and 64 bit:
>
>         http://info.iet.unipi.it/~luigi/dummynet/
>
> cheers
> luigi
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Luigi Rizzo-5]

On Tue, Aug 14, 2012 at 01:22:54PM -0700, Adrian Chadd wrote:
> Hi!
>
> This stuff is very cool.
>
> So why is it that we can do such ridiculous packet processing rates
> via netmap and ipfw in userland but not in the traditional driver path
> in-kernel?
>
> (I think I know the answer, I'm just tossing it out there for discussion.)

I'd point you to the slides of my usenix talk

        http://info.iet.unipi.it/~luigi/netmap/talk-atc12.html

or even the video of the talk itself

        https://www.usenix.org/conference/usenixfederatedconferencesweek/netmap-novel-framework-fast-packet-io

(on passing, i think the Usenix policy of making material publicly
available is really commendable.)

        cheers
        luigi

>
> On 13 August 2012 04:17, Luigi Rizzo <[hidden email]> wrote:
> > I just finished a netmap-enabled version of ipfw/dummynet, which
> > runs in userspace and is able to process over 6 million packets per
> > second (Mpps) with simple rulesets, and over 2.2 Mpps through
> > dummynet pipes (tested on an i7-3400 connected to VALE ports;
> > VALE is a software switch part of netmap).
> > You can find it at
> >
> >         http://info.iet.unipi.it/~luigi/dummynet/#8696
> >
> > It works on FreeBSD and Linux.
> >
> > It even run on OSX, but there is no netmap support there;
> > any interest, Apple ? In any case, it should be simple to
> > adapt the network backend to talk to other devices.
> >
> > To run this you might want to use the most recent version of netmap,
> > also recently updated so it works with recent kernels
> >
> >         http://info.iet.unipi.it/~luigi/netmap/
> >
> > (the code is already in FreeBSD HEAD).
> >
> > On passing, I have also updated the dummynet distributions
> > for Linux and Windows, which should now compile for most
> > recent version of Linux, and for Windows 32 and 64 bit:
> >
> >         http://info.iet.unipi.it/~luigi/dummynet/
> >
> > cheers
> > luigi
> > _______________________________________________
> > [hidden email] mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "[hidden email]"
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"

[Luigi Rizzo-5]

On Tue, Aug 14, 2012 at 10:53:38PM +0300, Nikolay Denev wrote:
> On Aug 13, 2012, at 2:17 PM, Luigi Rizzo <[hidden email]> wrote:
...
> Hi Luigi,
>
> I've tried copying netmap sources from HEAD to the latest RELENG_9 branch, but running
> kipfw with two VALE interfaces produced immediate kernel panic (transcribed by hand, possible typos):

i suspect kernel and userland headers out of sync, causing
some ioctl() to be called with invalid parameters
(and perhaps a missing check in the kernel does the rest,
otherwise of course the kernel should not crash).

Please make sure you also update net/netmap*.h

FWIW, i tested the code in RELENG_9 on my workstation.

cheers
luigi

> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 01
> fault virtual address = 0x28
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff808274da
> stack pointer = 0x28:0xffffff88df1a5530
> frame pointer = 0x28:0xffffff88df1a5550
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 2349 (kipfw)
> [ thread pid 2349 tid 100612 ]
> Stopped at ifunit_ref+0x2a:
> db> bt
> Tracing pid 2349 tid 100612 td 0xfffffe0094fd1000
> ifunit_ref() at ifunit_ref+0x2a
> ifioctl() at ifioctl+0x1cd
> netmap_ioctl() at netmap_ioctl+0x121
> devfs_ioctl_f() at devfs_ioctl_f+0x7b
> kern_ioctl() at kern_ioctl+0x115
> sys_ioctl() at sys_ioctl+0xfd
> amd64_syscall() at amd64_syscall+0x5ea
> Xfast_syscall() at Xfast_syscall+0xf7
> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800b5eaec, rsp = 0x7fffffffd
> 888, rbp = 0xc0206911 ---
>
>
>
> Anyways, many thanks for you great work on NETMAP and not the super cool user-space IPFW!
>
> Cheers,
> Nikolay

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[hidden email]"