[PATCH] Add locking to twe(4) so it no longer uses Giant

[John Baldwin]

We only have a few storage drivers left that use Giant and twe(4) is one of
them.  I don't have any hardware to test with, however.  I have verified the
patch compiles, but I'd appreciate it if some folks with twe(4) hardware could
test it.

The patch is at http://www.FreeBSD.org/~jhb/patches/twe_locking.patch

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Kevin Oberman-3]

I'll try to find time to try it later today, but I'm in the middle of
budget wrangling and I can't make any promises.

Before I try, will these patches apply to the twe driver in
9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
I really don't have time to install current right now.
--
R. Kevin Oberman, Network Engineer
E-mail: [hidden email]

On Fri, Aug 3, 2012 at 11:18 AM, John Baldwin <[hidden email]> wrote:
> We only have a few storage drivers left that use Giant and twe(4) is one of
> them.  I don't have any hardware to test with, however.  I have verified the
> patch compiles, but I'd appreciate it if some folks with twe(4) hardware could
> test it.
>
> The patch is at http://www.FreeBSD.org/~jhb/patches/twe_locking.patch
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote:
> I'll try to find time to try it later today, but I'm in the middle of
> budget wrangling and I can't make any promises.
>
> Before I try, will these patches apply to the twe driver in
> 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
> I really don't have time to install current right now.

I believe the patches should apply fine on 8 and 9.  If you use it on 8 or 9
please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for
initial testing.  Thanks!

> --
> R. Kevin Oberman, Network Engineer
> E-mail: [hidden email]
>
> On Fri, Aug 3, 2012 at 11:18 AM, John Baldwin <[hidden email]> wrote:
> > We only have a few storage drivers left that use Giant and twe(4) is one of
> > them.  I don't have any hardware to test with, however.  I have verified the
> > patch compiles, but I'd appreciate it if some folks with twe(4) hardware could
> > test it.
> >
> > The patch is at http://www.FreeBSD.org/~jhb/patches/twe_locking.patch
>

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/3/2012 3:39 PM, John Baldwin wrote:

> On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote:
>> I'll try to find time to try it later today, but I'm in the middle of
>> budget wrangling and I can't make any promises.
>>
>> Before I try, will these patches apply to the twe driver in
>> 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
>> I really don't have time to install current right now.
>
> I believe the patches should apply fine on 8 and 9.  If you use it on 8 or 9
> please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for
> initial testing.  Thanks!

Seems to apply to RELENG_9 just fine.  Are there any stress tests you
suggest I run that might expose some bugs ? The machine is not
production yet, so its ok to crash it.


{offsite2}# patch -p9 < twe_locking.patch
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe.c       2009-12-25
17:35:14.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe.c 2012-08-03
04:00:49.000000000 0000
--------------------------
Patching file twe.c using Plan A...
Hunk #1 succeeded at 151.
Hunk #2 succeeded at 167.
Hunk #3 succeeded at 189.
Hunk #4 succeeded at 208.
Hunk #5 succeeded at 226.
Hunk #6 succeeded at 234.
Hunk #7 succeeded at 271.
Hunk #8 succeeded at 288.
Hunk #9 succeeded at 310.
Hunk #10 succeeded at 337.
Hunk #11 succeeded at 349.
Hunk #12 succeeded at 405.
Hunk #13 succeeded at 521.
Hunk #14 succeeded at 557.
Hunk #15 succeeded at 580.
Hunk #16 succeeded at 595.
Hunk #17 succeeded at 608.
Hunk #18 succeeded at 646.
Hunk #19 succeeded at 769.
Hunk #20 succeeded at 863.
Hunk #21 succeeded at 921.
Hunk #22 succeeded at 952.
Hunk #23 succeeded at 1038.
Hunk #24 succeeded at 1051.
Hunk #25 succeeded at 1082.
Hunk #26 succeeded at 1104.
Hunk #27 succeeded at 1140.
Hunk #28 succeeded at 1151.
Hunk #29 succeeded at 1177.
Hunk #30 succeeded at 1206.
Hunk #31 succeeded at 1309.
Hunk #32 succeeded at 1447.
Hunk #33 succeeded at 1469.
Hunk #34 succeeded at 1499.
Hunk #35 succeeded at 1513.
Hunk #36 succeeded at 1531.
Hunk #37 succeeded at 1554.
Hunk #38 succeeded at 1589.
Hunk #39 succeeded at 1618.
Hunk #40 succeeded at 1644.
Hunk #41 succeeded at 1696.
Hunk #42 succeeded at 1778.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_compat.h
2005-05-29 04:45:51.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_compat.h  2012-08-03
03:58:12.000000000 0000
--------------------------
Patching file twe_compat.h using Plan A...
Hunk #1 succeeded at 43.
Hunk #2 succeeded at 68.
Hunk #3 succeeded at 82.
Hunk #4 succeeded at 92.
Hunk #5 succeeded at 108.
Hunk #6 succeeded at 128.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_freebsd.c
2012-03-12 08:05:24.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
18:10:04.000000000 0000
--------------------------
Patching file twe_freebsd.c using Plan A...
Hunk #1 succeeded at 69.
Hunk #2 succeeded at 83.
Hunk #3 succeeded at 101.
Hunk #4 succeeded at 180.
Hunk #5 succeeded at 188.
Hunk #6 succeeded at 211.
Hunk #7 succeeded at 241.
Hunk #8 succeeded at 292.
Hunk #9 succeeded at 414.
Hunk #10 succeeded at 425.
Hunk #11 succeeded at 461.
Hunk #12 succeeded at 496.
Hunk #13 succeeded at 518.
Hunk #14 succeeded at 533.
Hunk #15 succeeded at 566.
Hunk #16 succeeded at 585.
Hunk #17 succeeded at 604.
Hunk #18 succeeded at 680.
Hunk #19 succeeded at 729.
Hunk #20 succeeded at 834.
Hunk #21 succeeded at 857.
Hunk #22 succeeded at 876.
Hunk #23 succeeded at 1038.
Hunk #24 succeeded at 1097.
Hunk #25 succeeded at 1151.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twevar.h    2009-12-25
17:35:14.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twevar.h      2012-08-03
04:00:49.000000000 0000
--------------------------
Patching file twevar.h using Plan A...
Hunk #1 succeeded at 134.
Hunk #2 succeeded at 210.
Hunk #3 succeeded at 255.
done
0{offsite2}#



--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Garrett Cooper]

On Aug 3, 2012, at 1:42 PM, Mike Tancsa wrote:

> On 8/3/2012 3:39 PM, John Baldwin wrote:
>> On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote:
>>> I'll try to find time to try it later today, but I'm in the middle of
>>> budget wrangling and I can't make any promises.
>>>
>>> Before I try, will these patches apply to the twe driver in
>>> 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
>>> I really don't have time to install current right now.
>>
>> I believe the patches should apply fine on 8 and 9.  If you use it on 8 or 9
>> please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for
>> initial testing.  Thanks!
>
> Seems to apply to RELENG_9 just fine.  Are there any stress tests you
> suggest I run that might expose some bugs ? The machine is not
> production yet, so its ok to crash it.

        Looking really quickly at the patch, it looks like most of the locking changes are made around management pieces, and not data handling pieces (but I might have missed something).
        If there's a tool for poking at the drives/controller, I would use that, plus camcontrol. Of course you want a data intensive workload (iometer/iozone/xdd with async and sync mode, random reads and sequential reads, etc), and maybe resort to manual testing like pulling drives (power, data) if you don't mind creating failures. If you have some failed/failing drives kicking around, that would be a good test as well (see that all/some of the failure paths are properly stimulated).
        Any variance or combining of these tests will help build confidence in the change being proposed.
HTH,
-Garrett_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On Friday, August 03, 2012 4:42:59 pm Mike Tancsa wrote:

> On 8/3/2012 3:39 PM, John Baldwin wrote:
> > On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote:
> >> I'll try to find time to try it later today, but I'm in the middle of
> >> budget wrangling and I can't make any promises.
> >>
> >> Before I try, will these patches apply to the twe driver in
> >> 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
> >> I really don't have time to install current right now.
> >
> > I believe the patches should apply fine on 8 and 9.  If you use it on 8 or

9
> > please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for
> > initial testing.  Thanks!
>
> Seems to apply to RELENG_9 just fine.  Are there any stress tests you
> suggest I run that might expose some bugs ? The machine is not
> production yet, so its ok to crash it.

Probably pho's stress2 stuff.  Thinks like dbench might be a good start as
well for initial testing.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On Friday, August 03, 2012 5:17:09 pm Garrett Cooper wrote:

> On Aug 3, 2012, at 1:42 PM, Mike Tancsa wrote:
>
> > On 8/3/2012 3:39 PM, John Baldwin wrote:
> >> On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote:
> >>> I'll try to find time to try it later today, but I'm in the middle of
> >>> budget wrangling and I can't make any promises.
> >>>
> >>> Before I try, will these patches apply to the twe driver in
> >>> 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
> >>> I really don't have time to install current right now.
> >>
> >> I believe the patches should apply fine on 8 and 9.  If you use it on 8 or 9
> >> please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for
> >> initial testing.  Thanks!
> >
> > Seems to apply to RELENG_9 just fine.  Are there any stress tests you
> > suggest I run that might expose some bugs ? The machine is not
> > production yet, so its ok to crash it.
>
> Looking really quickly at the patch, it looks like most of the locking changes are made around management pieces, and not data handling pieces

(but I might have missed something).

No, it changes those paths, too.  There just aren't many of them.  All the
data enters through twed_strategy() and leaves via twed_intr() (invoked from
twe_intr()).

> If there's a tool for poking at the drives/controller, I would use that, plus camcontrol. Of course you want a data intensive workload
(iometer/iozone/xdd with async and sync mode, random reads and sequential reads, etc), and maybe resort to manual testing like pulling drives
(power, data) if you don't mind creating failures. If you have some failed/failing drives kicking around, that would be a good test as well (see
that all/some of the failure paths are properly stimulated).

3dm2 testing would be good for the ioctl handling, but the most critical
tests are basic I/O.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/3/2012 5:26 PM, John Baldwin wrote:
>> If there's a tool for poking at the drives/controller, I would use that, plus camcontrol. Of course you want a data intensive workload
> (iometer/iozone/xdd with async and sync mode, random reads and sequential reads, etc), and maybe resort to manual testing like pulling drives
> (power, data) if you don't mind creating failures. If you have some failed/failing drives kicking around, that would be a good test as well (see
> that all/some of the failure paths are properly stimulated).
>
> 3dm2 testing would be good for the ioctl handling, but the most critical
> tests are basic I/O.
>


Looks like it breaks 3dm2 and the tw_cli.  With the patch, I am not able
to see the 8006 controller I added.

tw_cli without the patch

0{offsite2}# tw_cli
//offsite2> /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache
AVrfy
------------------------------------------------------------------------------
u0    RAID-0    OK             -       -       64K     931.521   ON
-

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     465.76 GB   976773168     WD-WCAYUEY18298

p1     OK               u0     465.76 GB   976773168     WD-WMAYUL256317


//offsite2>
//offsite2> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    8006-2LP     2         2        1       0       3       -      -

c1    9650SE-2LP   2         2        1       0       1       1      -


The boot array is c1 (the off twa controller)

with the patch (invariants and invariants_support in the kernel)

0{offsite2}# tw_cli
//offsite2> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    9650SE-2LP   2         2        1       0       1       1      -


//offsite2>



        ---Mike



--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On Tuesday, August 07, 2012 10:11:01 AM Mike Tancsa wrote:

> On 8/3/2012 5:26 PM, John Baldwin wrote:
> >> If there's a tool for poking at the drives/controller, I would use
> >> that, plus camcontrol. Of course you want a data intensive workload
> >
> > (iometer/iozone/xdd with async and sync mode, random reads and sequential
> > reads, etc), and maybe resort to manual testing like pulling drives
> > (power, data) if you don't mind creating failures. If you have some
> > failed/failing drives kicking around, that would be a good test as well
> > (see that all/some of the failure paths are properly stimulated).
> >
> > 3dm2 testing would be good for the ioctl handling, but the most critical
> > tests are basic I/O.
>
> Looks like it breaks 3dm2 and the tw_cli.  With the patch, I am not able
> to see the 8006 controller I added.

Ugh, ok.  A few questions:

1) Does the driver see any attached drives/volumes?

2) If it does, does basic I/O to the drives work?

3) Can you add some debugging printfs to twe_ioctl() to see what, if anything,
   fails in that routine when tw_cli makes a request?

Thanks.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/8/2012 7:27 AM, John Baldwin wrote:
>> Looks like it breaks 3dm2 and the tw_cli.  With the patch, I am not able
>> to see the 8006 controller I added.
>
> Ugh, ok.  A few questions:
>
> 1) Does the driver see any attached drives/volumes?

Yes

>
> 2) If it does, does basic I/O to the drives work?

yes

>
> 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything,
>    fails in that routine when tw_cli makes a request?

Yes, for sure. Let me know what you would like me to add.

        ---Mike


--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/8/2012 2:39 PM, Mike Tancsa wrote:

> On 8/8/2012 7:27 AM, John Baldwin wrote:
>>> Looks like it breaks 3dm2 and the tw_cli.  With the patch, I am not able
>>> to see the 8006 controller I added.
>>
>> Ugh, ok.  A few questions:
>>
>> 1) Does the driver see any attached drives/volumes?
>
> Yes
>
>>
>> 2) If it does, does basic I/O to the drives work?
>
> yes
>
>>
>> 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything,
>>    fails in that routine when tw_cli makes a request?
>
> Yes, for sure. Let me know what you would like me to add.

One more data point, /dev/twe0 does not exist with the patch and I think
thats why the utils fail outright.

        ---Mike

--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On 8/9/12 8:22 AM, Mike Tancsa wrote:

> On 8/8/2012 2:39 PM, Mike Tancsa wrote:
>> On 8/8/2012 7:27 AM, John Baldwin wrote:
>>>> Looks like it breaks 3dm2 and the tw_cli.  With the patch, I am not able
>>>> to see the 8006 controller I added.
>>>
>>> Ugh, ok.  A few questions:
>>>
>>> 1) Does the driver see any attached drives/volumes?
>>
>> Yes
>>
>>>
>>> 2) If it does, does basic I/O to the drives work?
>>
>> yes

Ok, so that's good.  I didn't break anything fundamental with driver
commands.

>>> 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything,
>>>    fails in that routine when tw_cli makes a request?
>>
>> Yes, for sure. Let me know what you would like me to add.
>
> One more data point, /dev/twe0 does not exist with the patch and I think
> thats why the utils fail outright.

Oh, hmm.  That's odd.  Do you get any error messages on the console
when twe0 attaches?  Also, you have INVARIANTS enabled, yes?
(make_dev() panics when it fails if INVARIANTS is enabled).

Maybe try something like this (relative to the patched driver):

--- //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
18:10:04.000000000 0000
+++ /Users/jhb/work/p4/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
18:10:04.000000000 0000
@@ -342,9 +342,12 @@
     /*
      * Create the control device.
      */
+    device_printf(sc->twe_dev, "Calling make_dev()\n");
     sc->twe_dev_t = make_dev(&twe_cdevsw, device_get_unit(sc->twe_dev),
UID_ROOT, GID_OPERATOR,
      S_IRUSR | S_IWUSR, "twe%d", device_get_unit(sc->twe_dev));
     sc->twe_dev_t->si_drv1 = sc;
+    device_printf(sc->twe_dev, "make_dev() returned %p (%s)\n",
sc->twe_dev_t,
+ sc->twe_dev_t->si_name);
     /*
      * Schedule ourselves to bring the controller up once interrupts
are available.
      * This isn't strictly necessary, since we disable interrupts while
probing the


--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/9/2012 9:16 AM, John Baldwin wrote:
>> One more data point, /dev/twe0 does not exist with the patch and I think
>> thats why the utils fail outright.
>
> Oh, hmm.  That's odd.  Do you get any error messages on the console
> when twe0 attaches?  Also, you have INVARIANTS enabled, yes?
> (make_dev() panics when it fails if INVARIANTS is enabled).

I have INVARIANTS in the kernel, sorry, do I need to do something to
make it active ?

0{offsite2}# sysctl -a | grep -i invar
kern.features.invariant_support: 1
options INVARIANT_SUPPORT
options INVARIANTS
kern.timecounter.invariant_tsc: 1
  TSC: P-state invariant, performance statistics
  TSC: P-state invariant, performance statistics
  TSC: P-state invariant, performance statistics
  TSC: P-state invariant, performance statistics
0{offsite2}#

Nothing odd in dmesg

ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0


pci6: <ACPI PCI bus> on pcib6
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
0x2000-0x200f mem 0xe2810000-0xe281000f,0xe2000000-0xe
27fffff at device 0.0 on pci6
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
isab0: <PCI-ISA bridge> at device 31.0 on pci0


Patch below is causing a panic now on boot. Just going to add debugging
to the serial console to see what it is.... and

0{offsite2}# kldload twe
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
0x2000-0x200f mem 0xe2810000-0xe281000f,0xe2000000-0xe27fffff at device
0.0 on pci6
twe0: [GIANT-LOCKED]


Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address   = 0x3
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff81813eb3
stack pointer           = 0x28:0xffffff84529d33f0
frame pointer           = 0x28:0xffffff84529d3430
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1324 (kldload)
[ thread pid 1324 tid 100146 ]
Stopped at      twe_setup+0x153:        movzbl  0x3(%rdx),%eax
db>

>
> Maybe try something like this (relative to the patched driver):
>
> --- //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
> 18:10:04.000000000 0000
> +++ /Users/jhb/work/p4/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
> 18:10:04.000000000 0000
> @@ -342,9 +342,12 @@
>      /*
>       * Create the control device.
>       */
> +    device_printf(sc->twe_dev, "Calling make_dev()\n");
>      sc->twe_dev_t = make_dev(&twe_cdevsw, device_get_unit(sc->twe_dev),
> UID_ROOT, GID_OPERATOR,
>       S_IRUSR | S_IWUSR, "twe%d", device_get_unit(sc->twe_dev));
>      sc->twe_dev_t->si_drv1 = sc;
> +    device_printf(sc->twe_dev, "make_dev() returned %p (%s)\n",
> sc->twe_dev_t,
> + sc->twe_dev_t->si_name);
>      /*
>       * Schedule ourselves to bring the controller up once interrupts
> are available.
>       * This isn't strictly necessary, since we disable interrupts while
> probing the
>
>

--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On 8/9/12 10:31 AM, Mike Tancsa wrote:

> On 8/9/2012 9:16 AM, John Baldwin wrote:
>>> One more data point, /dev/twe0 does not exist with the patch and I think
>>> thats why the utils fail outright.
>>
>> Oh, hmm.  That's odd.  Do you get any error messages on the console
>> when twe0 attaches?  Also, you have INVARIANTS enabled, yes?
>> (make_dev() panics when it fails if INVARIANTS is enabled).
>
> I have INVARIANTS in the kernel, sorry, do I need to do something to
> make it active ?
>
> 0{offsite2}# sysctl -a | grep -i invar
> kern.features.invariant_support: 1
> options INVARIANT_SUPPORT
> options INVARIANTS
> kern.timecounter.invariant_tsc: 1
>   TSC: P-state invariant, performance statistics
>   TSC: P-state invariant, performance statistics
>   TSC: P-state invariant, performance statistics
>   TSC: P-state invariant, performance statistics
> 0{offsite2}#
>
> Nothing odd in dmesg
>
> ZFS filesystem version 5
> ZFS storage pool version 28
> Timecounters tick every 1.000 msec
> twed0 on twe0
> twed0: 953878MB (1953542144 sectors)
> usbus0: 480Mbps High Speed USB v2.0
> usbus1: 480Mbps High Speed USB v2.0
>
>
> pci6: <ACPI PCI bus> on pcib6
> twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
> 0x2000-0x200f mem 0xe2810000-0xe281000f,0xe2000000-0xe
> 27fffff at device 0.0 on pci6
> twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
>
>
> Patch below is causing a panic now on boot. Just going to add debugging
> to the serial console to see what it is.... and
>
> 0{offsite2}# kldload twe
> twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
> 0x2000-0x200f mem 0xe2810000-0xe281000f,0xe2000000-0xe27fffff at device
> 0.0 on pci6
> twe0: [GIANT-LOCKED]

Odd, the patched driver shouldn't be triggering this message.

> Fatal trap 12: page fault while in kernel mode
> cpuid = 4; apic id = 04
> fault virtual address   = 0x3
> fault code              = supervisor read data, page not present
> instruction pointer     = 0x20:0xffffffff81813eb3
> stack pointer           = 0x28:0xffffff84529d33f0
> frame pointer           = 0x28:0xffffff84529d3430
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 1324 (kldload)
> [ thread pid 1324 tid 100146 ]
> Stopped at      twe_setup+0x153:        movzbl  0x3(%rdx),%eax
> db>

Humm, I wonder if this module has a mix of old and new .o files somehow?
Perhaps try rebuilding twe.ko from scratch after doing a 'make clean'?

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/9/2012 1:21 PM, John Baldwin wrote:
>
> Humm, I wonder if this module has a mix of old and new .o files somehow?
> Perhaps try rebuilding twe.ko from scratch after doing a 'make clean'?

I think you are right. I nuked all the kernel files and recompiled
again, and it no longer panics and I see the entry in /dev now!?




0{offsite2}# kldload twe
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
0x2000-0x200f mem 0xe2810000-0xe281000f,0xe2000000-0xe27fffff at device
0.0 on pci6
twe0: [GIANT-LOCKED]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
twe0: Calling make_dev()
twe0: make_dev() returned 0xfffffe0122105200 (twe0)
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
0{offsite2}#

        ---Mike

--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/9/2012 3:28 PM, Mike Tancsa wrote:

> On 8/9/2012 1:21 PM, John Baldwin wrote:
>>
>> Humm, I wonder if this module has a mix of old and new .o files somehow?
>> Perhaps try rebuilding twe.ko from scratch after doing a 'make clean'?
>
> I think you are right. I nuked all the kernel files and recompiled
> again, and it no longer panics and I see the entry in /dev now!?
>
> 0{offsite2}# kldload twe
> twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
> 0x2000-0x200f mem 0xe2810000-0xe281000f,0xe2000000-0xe27fffff at device
> 0.0 on pci6
> twe0: [GIANT-LOCKED]
> twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
> twe0: Calling make_dev()
> twe0: make_dev() returned 0xfffffe0122105200 (twe0)
> twed0 on twe0
> twed0: 953878MB (1953542144 sectors)
> 0{offsite2}#

OK, some more mystery.


If I load it as a kld,

I see the device. (Note, I added MDT into the version string)

0{offsite2}# twe0: <3ware Storage Controller. Driver version
1.50.01.002MDT> port 0x2000-0x200f mem
0xe2810000-0xe281000f,0xe2000000-0xe27fffff at device 0.0 on pci6
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
twe0: Calling make_dev()
twe0: make_dev() returned 0xfffffe0026581a00 (twe0)
twed0 on twe0
twed0: 953878MB (1953542144 sectors)

0{offsite2}# ls -l /dev/tw*
crw-------  1 root  operator  -   0,  37 Aug  9 16:58 /dev/twa0
crw-------  1 root  operator  -   0, 173 Aug  9 17:00 /dev/twe0
crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0



Start up the tw_cli client

0{offsite2}# tw_cli
//offsite2> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    9650SE-2LP   2         2        1       0       1       1      -


//offsite2> exit
0{offsite2}# ls -l /dev/tw*
crw-------  1 root  operator  -   0,  37 Aug  9 16:58 /dev/twa0
crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0
0{offsite2}#


It then disappears ?!

        ---Mike



--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On 8/9/12 5:04 PM, Mike Tancsa wrote:

> Start up the tw_cli client
>
> 0{offsite2}# tw_cli
> //offsite2> show
>
> Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
> ------------------------------------------------------------------------
> c0    9650SE-2LP   2         2        1       0       1       1      -
>
>
> //offsite2> exit
> 0{offsite2}# ls -l /dev/tw*
> crw-------  1 root  operator  -   0,  37 Aug  9 16:58 /dev/twa0
> crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0
> 0{offsite2}#
>
>
> It then disappears ?!

Bizarre, it seems to disappear while tw_cli is running?  I'm curious if
'rm -W /dev/twe0' brings it back?

If so, it might be interesting to see a ktrace of tw_cli.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/10/2012 9:31 AM, John Baldwin wrote:

> On 8/9/12 5:04 PM, Mike Tancsa wrote:
>> Start up the tw_cli client
>>
>> 0{offsite2}# tw_cli
>> //offsite2> show
>>
>> Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
>> ------------------------------------------------------------------------
>> c0    9650SE-2LP   2         2        1       0       1       1      -
>>
>>
>> //offsite2> exit
>> 0{offsite2}# ls -l /dev/tw*
>> crw-------  1 root  operator  -   0,  37 Aug  9 16:58 /dev/twa0
>> crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0
>> 0{offsite2}#
>>
>>
>> It then disappears ?!
>
> Bizarre, it seems to disappear while tw_cli is running?  I'm curious if
> 'rm -W /dev/twe0' brings it back?

nada
0{offsite2}# rm -W /dev/twe0
rm: /dev/twe0: No such file or directory
1{offsite2}# rm -W /dev/twe0
rm: /dev/twe0: No such file or directory
1{offsite2}# ls -l /dev/twe*
crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0
0{offsite2}#

>
> If so, it might be interesting to see a ktrace of tw_cli.
>

I just ran ktrace tw_cli
File is available at

http://www.tancsa.com/misc/ktrace.out
and
http://www.tancsa.com/misc/ktrace.txt

        ---Mike

--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[John Baldwin]

On 8/10/12 9:52 AM, Mike Tancsa wrote:

> On 8/10/2012 9:31 AM, John Baldwin wrote:
>> On 8/9/12 5:04 PM, Mike Tancsa wrote:
>>> Start up the tw_cli client
>>>
>>> 0{offsite2}# tw_cli
>>> //offsite2> show
>>>
>>> Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
>>> ------------------------------------------------------------------------
>>> c0    9650SE-2LP   2         2        1       0       1       1      -
>>>
>>>
>>> //offsite2> exit
>>> 0{offsite2}# ls -l /dev/tw*
>>> crw-------  1 root  operator  -   0,  37 Aug  9 16:58 /dev/twa0
>>> crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0
>>> 0{offsite2}#
>>>
>>>
>>> It then disappears ?!
>>
>> Bizarre, it seems to disappear while tw_cli is running?  I'm curious if
>> 'rm -W /dev/twe0' brings it back?
>
> nada
> 0{offsite2}# rm -W /dev/twe0
> rm: /dev/twe0: No such file or directory
> 1{offsite2}# rm -W /dev/twe0
> rm: /dev/twe0: No such file or directory
> 1{offsite2}# ls -l /dev/twe*
> crw-r-----  1 root  operator  -   0, 174 Aug  9 17:00 /dev/twed0
> 0{offsite2}#
>
>>
>> If so, it might be interesting to see a ktrace of tw_cli.
>>
>
> I just ran ktrace tw_cli
> File is available at
>
> http://www.tancsa.com/misc/ktrace.out
> and
> http://www.tancsa.com/misc/ktrace.txt

  5972 tw_cli   CALL
__sysctl(0x7fffffffd798,0x2,0x7fffffffd7a0,0x7fffffffd790,0x7fffffffd960,0x16)
  5972 tw_cli   SCTL  "sysctl.name2oid"
  5972 tw_cli   RET   __sysctl -1 errno 2 No such file or directory
  5972 tw_cli   CALL  unlink(0x7fffffffd9d0)
  5972 tw_cli   NAMI  "/dev/twe0"
  5972 tw_cli   RET   unlink 0
...

Oh!  I moved the sysctl's for twe out of hw and under the device node.
That seem to have broken tw_cli.  I will revert that and generate an
updated patch.

Try http://www.FreeBSD.org/~jhb/patches/twe_locking2.patch

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[mdtancsa]

On 8/10/2012 10:05 AM, John Baldwin wrote:

>
>   5972 tw_cli   CALL
> __sysctl(0x7fffffffd798,0x2,0x7fffffffd7a0,0x7fffffffd790,0x7fffffffd960,0x16)
>   5972 tw_cli   SCTL  "sysctl.name2oid"
>   5972 tw_cli   RET   __sysctl -1 errno 2 No such file or directory
>   5972 tw_cli   CALL  unlink(0x7fffffffd9d0)
>   5972 tw_cli   NAMI  "/dev/twe0"
>   5972 tw_cli   RET   unlink 0
> ...
>
> Oh!  I moved the sysctl's for twe out of hw and under the device node.
> That seem to have broken tw_cli.  I will revert that and generate an
> updated patch.
>
> Try http://www.FreeBSD.org/~jhb/patches/twe_locking2.patch
>

Looks good! Both the tw_cli and 3dm2 web interface work now.

Doing some simple disk tests show everything to be pretty well the same.

new driver
0{offsite2}# dd if=/dev/zero of=/mnt/test bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 14.677410 secs (71441488 bytes/sec)
0{offsite2}# dd if=/mnt/test of=/dev/null bs=1024
1024000+0 records in
1024000+0 records out
1048576000 bytes transferred in 10.325622 secs (101550879 bytes/sec)
0{offsite2}# umount /mnt ; mount /dev/twed0 /mnt
0{offsite2}# dd if=/mnt/test of=/dev/null bs=2048
512000+0 records in
512000+0 records out
1048576000 bytes transferred in 10.347670 secs (101334503 bytes/sec)
0{offsite2}#

old driver

0{offsite2}# dd if=/dev/zero of=/mnt/test bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 14.695589 secs (71353111 bytes/sec)
0{offsite2}# umount /mnt ; mount /dev/twed0 /mnt
0{offsite2}#  dd if=/mnt/test of=/dev/null bs=2048
512000+0 records in
512000+0 records out
1048576000 bytes transferred in 10.305835 secs (101745856 bytes/sec)
0{offsite2}#

I will do some more stress tests through the day





0{offsite2}# patch -p9 < twe_locking2.patch
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe.c       2009-12-25
17:35:14.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe.c 2012-08-03
04:00:49.000000000 0000
--------------------------
Patching file twe.c using Plan A...
Hunk #1 succeeded at 151.
Hunk #2 succeeded at 167.
Hunk #3 succeeded at 189.
Hunk #4 succeeded at 208.
Hunk #5 succeeded at 226.
Hunk #6 succeeded at 234.
Hunk #7 succeeded at 271.
Hunk #8 succeeded at 288.
Hunk #9 succeeded at 310.
Hunk #10 succeeded at 337.
Hunk #11 succeeded at 349.
Hunk #12 succeeded at 405.
Hunk #13 succeeded at 521.
Hunk #14 succeeded at 557.
Hunk #15 succeeded at 580.
Hunk #16 succeeded at 595.
Hunk #17 succeeded at 608.
Hunk #18 succeeded at 646.
Hunk #19 succeeded at 769.
Hunk #20 succeeded at 863.
Hunk #21 succeeded at 921.
Hunk #22 succeeded at 952.
Hunk #23 succeeded at 1038.
Hunk #24 succeeded at 1051.
Hunk #25 succeeded at 1082.
Hunk #26 succeeded at 1104.
Hunk #27 succeeded at 1140.
Hunk #28 succeeded at 1151.
Hunk #29 succeeded at 1177.
Hunk #30 succeeded at 1206.
Hunk #31 succeeded at 1309.
Hunk #32 succeeded at 1447.
Hunk #33 succeeded at 1469.
Hunk #34 succeeded at 1499.
Hunk #35 succeeded at 1513.
Hunk #36 succeeded at 1531.
Hunk #37 succeeded at 1554.
Hunk #38 succeeded at 1589.
Hunk #39 succeeded at 1618.
Hunk #40 succeeded at 1644.
Hunk #41 succeeded at 1696.
Hunk #42 succeeded at 1778.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_compat.h
2005-05-29 04:45:51.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_compat.h  2012-08-03
03:58:12.000000000 0000
--------------------------
Patching file twe_compat.h using Plan A...
Hunk #1 succeeded at 43.
Hunk #2 succeeded at 68.
Hunk #3 succeeded at 82.
Hunk #4 succeeded at 92.
Hunk #5 succeeded at 108.
Hunk #6 succeeded at 128.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_freebsd.c
2012-03-12 08:05:24.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-10
14:04:10.000000000 0000
--------------------------
Patching file twe_freebsd.c using Plan A...
Hunk #1 succeeded at 69.
Hunk #2 succeeded at 83.
Hunk #3 succeeded at 101.
Hunk #4 succeeded at 179.
Hunk #5 succeeded at 189.
Hunk #6 succeeded at 218.
Hunk #7 succeeded at 248.
Hunk #8 succeeded at 299.
Hunk #9 succeeded at 421.
Hunk #10 succeeded at 432.
Hunk #11 succeeded at 468.
Hunk #12 succeeded at 503.
Hunk #13 succeeded at 525.
Hunk #14 succeeded at 540.
Hunk #15 succeeded at 573.
Hunk #16 succeeded at 592.
Hunk #17 succeeded at 611.
Hunk #18 succeeded at 687.
Hunk #19 succeeded at 736.
Hunk #20 succeeded at 841.
Hunk #21 succeeded at 864.
Hunk #22 succeeded at 883.
Hunk #23 succeeded at 1045.
Hunk #24 succeeded at 1104.
Hunk #25 succeeded at 1158.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- //depot/vendor/freebsd/src/sys/dev/twe/twevar.h    2009-12-25
17:35:14.000000000 0000
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twevar.h      2012-08-03
04:00:49.000000000 0000
--------------------------
Patching file twevar.h using Plan A...
Hunk #1 succeeded at 134.
Hunk #2 succeeded at 210.
Hunk #3 succeeded at 255.
done
0{offsite2}#


Looks good!

//offsite2> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    8006-2LP     2         2        1       0       3       -      -

c1    9650SE-2LP   2         2        1       0       1       1      -


//offsite2> /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache
AVrfy
------------------------------------------------------------------------------
u0    RAID-0    OK             -       -       64K     931.521   ON
-

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     465.76 GB   976773168     WD-WCAYUEY18298

p1     OK               u0     465.76 GB   976773168     WD-WMAYUL256317


//offsite2> /c1 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache
AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       465.651   RiW
ON

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   465.76 GB SATA  0   -            WDC
WD5002AALX-00J3
p1    OK             u0   465.76 GB SATA  1   -            WDC
WD5002AALX-00J3

//offsite2>



--
-------------------
Mike Tancsa, tel [phone]
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"