FreeBSD › freebsd-current

geom mirror now rebuilding on every reboot?

Classic

List

Threaded

24 messages Options

Michael Butler

geom mirror now rebuilding on every reboot?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Something in -current and recently MFC'd to -stable is causing all of my
gmirror drives to rebuild on reboot :-(

Being remote and these being production machines, I suspect SVN r237929
and r237930 in -current and SVN r238500 to -stable but haven't yet been
able to prove it.

Is anyone else seeing this?

imb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAlAdUqwACgkQQv9rrgRC1JIVSwCfctGE6tASTXyhW1ejHsWLTDRs
MTsAoMXqcQ3dwlurELdqm2ZBqTCCgRaR
=J/No
-----END PGP SIGNATURE-----
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Maksim Yevmenkin-2

Re: geom mirror now rebuilding on every reboot?

Michael,

> Something in -current and recently MFC'd to -stable is causing all of my
> gmirror drives to rebuild on reboot :-(
>
> Being remote and these being production machines, I suspect SVN r237929
> and r237930 in -current and SVN r238500 to -stable but haven't yet been
> able to prove it.
>
> Is anyone else seeing this?

yes, i've seem something similar only much, much worse. one of our
production systems completely kept loosing its gmirror volumes on
every reboot. it looked like gmirror metadata were completely
corrupted. rebuilding mirrors and reverting back to previous kernel
seemed to work. someone else is tracking it down.

thanks,
max
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Garrett Cooper

Re: geom mirror now rebuilding on every reboot?

On Mon, Aug 6, 2012 at 8:34 AM, Maksim Yevmenkin
<[hidden email]> wrote:

> Michael,
>
>> Something in -current and recently MFC'd to -stable is causing all of my
>> gmirror drives to rebuild on reboot :-(
>>
>> Being remote and these being production machines, I suspect SVN r237929
>> and r237930 in -current and SVN r238500 to -stable but haven't yet been
>> able to prove it.
>>
>> Is anyone else seeing this?
>
> yes, i've seem something similar only much, much worse. one of our
> production systems completely kept loosing its gmirror volumes on
> every reboot. it looked like gmirror metadata were completely
> corrupted. rebuilding mirrors and reverting back to previous kernel
> seemed to work. someone else is tracking it down.

Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
label is...)? If so, it seems like this would be a ship blocker.
Thanks!
-Garrett
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Maksim Yevmenkin-2

Re: geom mirror now rebuilding on every reboot?

On Mon, Aug 6, 2012 at 9:23 AM, Garrett Cooper <[hidden email]> wrote:

> On Mon, Aug 6, 2012 at 8:34 AM, Maksim Yevmenkin
> <[hidden email]> wrote:
>> Michael,
>>
>>> Something in -current and recently MFC'd to -stable is causing all of my
>>> gmirror drives to rebuild on reboot :-(
>>>
>>> Being remote and these being production machines, I suspect SVN r237929
>>> and r237930 in -current and SVN r238500 to -stable but haven't yet been
>>> able to prove it.
>>>
>>> Is anyone else seeing this?
>>
>> yes, i've seem something similar only much, much worse. one of our
>> production systems completely kept loosing its gmirror volumes on
>> every reboot. it looked like gmirror metadata were completely
>> corrupted. rebuilding mirrors and reverting back to previous kernel
>> seemed to work. someone else is tracking it down.
>
> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
> label is...)? If so, it seems like this would be a ship blocker.

Kevin Oberman-3

Re: geom mirror now rebuilding on every reboot?

On Mon, Aug 6, 2012 at 9:26 AM, Maksim Yevmenkin
<[hidden email]> wrote:

> On Mon, Aug 6, 2012 at 9:23 AM, Garrett Cooper <[hidden email]> wrote:
>> On Mon, Aug 6, 2012 at 8:34 AM, Maksim Yevmenkin
>> <[hidden email]> wrote:
>>> Michael,
>>>
>>>> Something in -current and recently MFC'd to -stable is causing all of my
>>>> gmirror drives to rebuild on reboot :-(
>>>>
>>>> Being remote and these being production machines, I suspect SVN r237929
>>>> and r237930 in -current and SVN r238500 to -stable but haven't yet been
>>>> able to prove it.
>>>>
>>>> Is anyone else seeing this?
>>>
>>> yes, i've seem something similar only much, much worse. one of our
>>> production systems completely kept loosing its gmirror volumes on
>>> every reboot. it looked like gmirror metadata were completely
>>> corrupted. rebuilding mirrors and reverting back to previous kernel
>>> seemed to work. someone else is tracking it down.
>>
>> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
>> label is...)? If so, it seems like this would be a ship blocker.
>
> sorry. its releng_9/9-stable. gmirrors are on two ssds. we use gpt and
> gmirror individual partitions, not entire disks.

I may well be confused, but I don't understand how you can use GPT for
a single partition. Looks to me like a disk is GPT or legacy.
Declaring which is the first command when setting up a new disk.
--
R. Kevin Oberman, Network Engineer
E-mail: [hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Maksim Yevmenkin-2

Re: geom mirror now rebuilding on every reboot?

On Mon, Aug 6, 2012 at 9:33 AM, Kevin Oberman <[hidden email]> wrote:

> On Mon, Aug 6, 2012 at 9:26 AM, Maksim Yevmenkin
> <[hidden email]> wrote:
>> On Mon, Aug 6, 2012 at 9:23 AM, Garrett Cooper <[hidden email]> wrote:
>>> On Mon, Aug 6, 2012 at 8:34 AM, Maksim Yevmenkin
>>> <[hidden email]> wrote:
>>>> Michael,
>>>>
>>>>> Something in -current and recently MFC'd to -stable is causing all of my
>>>>> gmirror drives to rebuild on reboot :-(
>>>>>
>>>>> Being remote and these being production machines, I suspect SVN r237929
>>>>> and r237930 in -current and SVN r238500 to -stable but haven't yet been
>>>>> able to prove it.
>>>>>
>>>>> Is anyone else seeing this?
>>>>
>>>> yes, i've seem something similar only much, much worse. one of our
>>>> production systems completely kept loosing its gmirror volumes on
>>>> every reboot. it looked like gmirror metadata were completely
>>>> corrupted. rebuilding mirrors and reverting back to previous kernel
>>>> seemed to work. someone else is tracking it down.
>>>
>>> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
>>> label is...)? If so, it seems like this would be a ship blocker.
>>
>> sorry. its releng_9/9-stable. gmirrors are on two ssds. we use gpt and
>> gmirror individual partitions, not entire disks.
>
> I may well be confused, but I don't understand how you can use GPT for
> a single partition. Looks to me like a disk is GPT or legacy.
> Declaring which is the first command when setting up a new disk.

i'm sorry, to make it (hopefully) clear, we are using gpt partitioning
scheme (as opposed to mbr partitioning scheme), and then use gmirror
on individual partitions, i.e. we gimirror /dev/ada0p1 and /dev/ada1p1
and not /dev/ada0 and /dev/ada1.

thanks
max
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Garrett Cooper

Re: geom mirror now rebuilding on every reboot?

On Aug 6, 2012, at 9:40 AM, Maksim Yevmenkin <[hidden email]> wrote:

> On Mon, Aug 6, 2012 at 9:33 AM, Kevin Oberman <[hidden email]> wrote:
>> On Mon, Aug 6, 2012 at 9:26 AM, Maksim Yevmenkin
>> <[hidden email]> wrote:
>>> On Mon, Aug 6, 2012 at 9:23 AM, Garrett Cooper <[hidden email]> wrote:
>>>> On Mon, Aug 6, 2012 at 8:34 AM, Maksim Yevmenkin
>>>> <[hidden email]> wrote:
>>>>> Michael,
>>>>>
>>>>>> Something in -current and recently MFC'd to -stable is causing all of my
>>>>>> gmirror drives to rebuild on reboot :-(
>>>>>>
>>>>>> Being remote and these being production machines, I suspect SVN r237929
>>>>>> and r237930 in -current and SVN r238500 to -stable but haven't yet been
>>>>>> able to prove it.
>>>>>>
>>>>>> Is anyone else seeing this?
>>>>>
>>>>> yes, i've seem something similar only much, much worse. one of our
>>>>> production systems completely kept loosing its gmirror volumes on
>>>>> every reboot. it looked like gmirror metadata were completely
>>>>> corrupted. rebuilding mirrors and reverting back to previous kernel
>>>>> seemed to work. someone else is tracking it down.
>>>>
>>>> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
>>>> label is...)? If so, it seems like this would be a ship blocker.
>>>
>>> sorry. its releng_9/9-stable. gmirrors are on two ssds. we use gpt and
>>> gmirror individual partitions, not entire disks.
>>
>> I may well be confused, but I don't understand how you can use GPT for
>> a single partition. Looks to me like a disk is GPT or legacy.
>> Declaring which is the first command when setting up a new disk.
>
> i'm sorry, to make it (hopefully) clear, we are using gpt partitioning
> scheme (as opposed to mbr partitioning scheme), and then use gmirror
> on individual partitions, i.e. we gimirror /dev/ada0p1 and /dev/ada1p1
> and not /dev/ada0 and /dev/ada1.

Can you provide more helpful info like svn revisions?_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Gleb Smirnoff

Re: geom mirror now rebuilding on every reboot?

In reply to this post by Michael Butler

Michael,

On Sat, Aug 04, 2012 at 12:49:49PM -0400, Michael Butler wrote:
M> Something in -current and recently MFC'd to -stable is causing all of my
M> gmirror drives to rebuild on reboot :-(
M>
M> Being remote and these being production machines, I suspect SVN r237929
M> and r237930 in -current and SVN r238500 to -stable but haven't yet been
M> able to prove it.

I'd appreciate if you test that and either confirm or disclaim that
r238500 introduces such regression. Thanks!

--
Totus tuus, Glebius.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Michael Butler

RE: geom mirror now rebuilding on every reboot?

In reply to this post by Michael Butler

Ding, ding - I use the old partitioning scheme but also mirror individual partitions rather than whole disks,

imb

-----Original Message-----
From: Maksim Yevmenkin <[hidden email]>
Sent: Monday, 6 August 2012 12:26
To: Garrett Cooper <[hidden email]>
Cc: [hidden email]; Michael Butler <[hidden email]>; [hidden email]
Subject: Re: geom mirror now rebuilding on every reboot?

On Mon, Aug 6, 2012 at 9:23 AM, Garrett Cooper <[hidden email]> wrote:

sorry. its releng_9/9-stable. gmirrors are on two ssds. we use gpt and
gmirror individual partitions, not entire disks.

thanks,
max
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Michael Butler

RE: geom mirror now rebuilding on every reboot?

In reply to this post by Michael Butler

Sorry but I'm not going to be able to get to this until next w/e as I'm ~1500 miles away from them at present :-(

-----Original Message-----
From: Gleb Smirnoff <[hidden email]>
Sent: Monday, 6 August 2012 14:06
To: Michael Butler <[hidden email]>
Cc: [hidden email]; [hidden email]
Subject: Re: geom mirror now rebuilding on every reboot?

Michael,

On Sat, Aug 04, 2012 at 12:49:49PM -0400, Michael Butler wrote:
M> Something in -current and recently MFC'd to -stable is causing all of my
M> gmirror drives to rebuild on reboot :-(
M>
M> Being remote and these being production machines, I suspect SVN r237929
M> and r237930 in -current and SVN r238500 to -stable but haven't yet been
M> able to prove it.

I'd appreciate if you test that and either confirm or disclaim that
r238500 introduces such regression. Thanks!

--
Totus tuus, Glebius.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Tom Uffner

Re: geom mirror now rebuilding on every reboot?

not quite the same issue, but i tried to update a system with gmirror from

FreeBSD eris.uffner.com 10.0-CURRENT FreeBSD 10.0-CURRENT #210: Thu Mar 15
16:41:18 EDT 2012 [hidden email]:/usr/obj/usr/src/sys/ERIS i386

to -current from Aug 4th, and booting now fails into mountroot> with

GEOM_RAID: Promise: Array Promise Created
GEOM_RAID: Promise: Disk ad4 state changed from NONE to SPARE
GEOM_MIRROR: Cannot open consumer ad4 (error=1)
GEOM_MIRROR: Cannot add disk ad4 to gm0 (error=1)
GEOM_MIRROR: Device gm0 destroyed
GEOM_RAID: Promise: Array Promise Created
GEOM_RAID: Promise: Disk ad6 state changed from NONE to SPARE
GEOM_MIRROR: Cannot open consumer ad6 (error=1)
GEOM_MIRROR: Cannot add disk ad6 to gm0 (error=1)
GEOM_MIRROR: Device gm0 destroyed

nor will it boot from the underlying partitions (ad4s1a, ad6s1a), even if i
unplug one of the drives. the partitions & metadata are not corrupt and work
just fine if i revert to the previous kernel.

i haven't had time yet to track down the commit where this breaks. i was just
wondering if anyone knew offhand what is going on & how to fix it.

tom
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Scott Long-2

Re: geom mirror now rebuilding on every reboot?

In reply to this post by Gleb Smirnoff

On Aug 6, 2012, at 12:06 PM, Gleb Smirnoff <[hidden email]> wrote:

> Michael,
>
> On Sat, Aug 04, 2012 at 12:49:49PM -0400, Michael Butler wrote:
> M> Something in -current and recently MFC'd to -stable is causing all of my
> M> gmirror drives to rebuild on reboot :-(
> M>
> M> Being remote and these being production machines, I suspect SVN r237929
> M> and r237930 in -current and SVN r238500 to -stable but haven't yet been
> M> able to prove it.
>
> I'd appreciate if you test that and either confirm or disclaim that
> r238500 introduces such regression. Thanks!
>

I'm not sure how r238500 could affect what Max, myself, and presumably the original poster are seeing. There's one other change in 9-stable, r235599, but it looks like a benign change as well.

Scott

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Frank Reppin-3

Re: geom mirror now rebuilding on every reboot?

by any chance...

... is this maybe related to my PR:

http://www.freebsd.org/cgi/query-pr.cgi?pr=170038

cheers,
Frank Reppin

--
43rd Law of Computing:
Anything that can go wr
fortune: Segmentation violation -- Core dumped
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Andrey V. Elsukov

Re: geom mirror now rebuilding on every reboot?

In reply to this post by Tom Uffner

On 07.08.2012 04:03, Tom Uffner wrote:
> GEOM_RAID: Promise: Array Promise Created
> GEOM_RAID: Promise: Disk ad4 state changed from NONE to SPARE
> GEOM_MIRROR: Cannot open consumer ad4 (error=1)
> GEOM_MIRROR: Cannot add disk ad4 to gm0 (error=1)
> wondering if anyone knew offhand what is going on & how to fix it.

You can remove "options GEOM_RAID" from your kernel config.

--
WBR, Andrey V. Elsukov
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Ian FREISLICH-3

Speaking of ship blockers for 9....

In reply to this post by Garrett Cooper

Garrett Cooper
> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
> label is...)? If so, it seems like this would be a ship blocker.

I have a problem that's been getting progressively worse as the
source progresses. So much so that it's had me searching all the
way from 8.0-RELEASE to 10-CURRENT without luck on both amd64 and
i386.

pf(4) erroneously mismatches state and then blocks an active flow.
It seems that 8.X does so silently and 9 to -CURRENT do so verbosely.
Whether silent or loud, the effect on traffic makes it impracticle
to use FreeBSD+PF for a firewall in any setting (my use is home,
small office, large office and moderately large datacenter core
router). It appears that this has actually been a forever problem
that just being tickled more now.

Here's from my home firewall:
Status: Enabled for 7 days 02:57:58 Debug: Urgent

State Table Total Rate
current entries 1653
searches 45792251 74.4/s
inserts 428375 0.7/s
removals 426722 0.7/s
...
state-mismatch 1586 0.0/s

Here's from a moderately busy firewall:
Status: Enabled for 0 days 21:40:44 Debug: Urgent

State Table Total Rate
current entries 122395
searches 4428641685 56745.4/s
inserts 202644593 2596.5/s
removals 202522198 2595.0/s
...
state-mismatch 277767 3.6/s

That's 277767 flows terminated in the last almost 22 hours due to
this pf bug. (!!!)

9.1-PRERELEASE logs (as does -CURRENT):
Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60985, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:52995, a1: 41.154.2.100:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60095, a1: 206.223.136.200:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:50463, a1: 206.223.136.200:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:56748, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60793, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.

Ian

--
Ian Freislich
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Garrett Cooper

Re: Speaking of ship blockers for 9....

On Aug 7, 2012, at 11:17 AM, Ian FREISLICH <[hidden email]> wrote:

> Garrett Cooper
>> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
>> label is...)? If so, it seems like this would be a ship blocker.
>
> I have a problem that's been getting progressively worse as the
> source progresses. So much so that it's had me searching all the
> way from 8.0-RELEASE to 10-CURRENT without luck on both amd64 and
> i386.
>
> pf(4) erroneously mismatches state and then blocks an active flow.
> It seems that 8.X does so silently and 9 to -CURRENT do so verbosely.
> Whether silent or loud, the effect on traffic makes it impracticle
> to use FreeBSD+PF for a firewall in any setting (my use is home,
> small office, large office and moderately large datacenter core
> router). It appears that this has actually been a forever problem
> that just being tickled more now.
>
> Here's from my home firewall:
> Status: Enabled for 7 days 02:57:58 Debug: Urgent
>
> State Table Total Rate
> current entries 1653
> searches 45792251 74.4/s
> inserts 428375 0.7/s
> removals 426722 0.7/s
> ...
> state-mismatch 1586 0.0/s
>
>
> Here's from a moderately busy firewall:
> Status: Enabled for 0 days 21:40:44 Debug: Urgent
>
> State Table Total Rate
> current entries 122395
> searches 4428641685 56745.4/s
> inserts 202644593 2596.5/s
> removals 202522198 2595.0/s
> ...
> state-mismatch 277767 3.6/s
>
> That's 277767 flows terminated in the last almost 22 hours due to
> this pf bug. (!!!)
>
> 9.1-PRERELEASE logs (as does -CURRENT):
> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60985, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:52995, a1: 41.154.2.100:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60095, a1: 206.223.136.200:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:50463, a1: 206.223.136.200:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:56748, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60793, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.

Filed a PR yet with packet captures?
Thanks,
-Garrett_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Alexander Motin-3

Re: geom mirror now rebuilding on every reboot?

In reply to this post by Andrey V. Elsukov

On 07.08.2012 17:09, Andrey V. Elsukov wrote:
> On 07.08.2012 04:03, Tom Uffner wrote:
>> GEOM_RAID: Promise: Array Promise Created
>> GEOM_RAID: Promise: Disk ad4 state changed from NONE to SPARE
>> GEOM_MIRROR: Cannot open consumer ad4 (error=1)
>> GEOM_MIRROR: Cannot add disk ad4 to gm0 (error=1)
>> wondering if anyone knew offhand what is going on & how to fix it.
>
> You can remove "options GEOM_RAID" from your kernel config.

I think the right answer would be to remove stale Promise format RAID
metadata from the disk. You should be able to do it by booting from any
source and doing `graid list -a` to see all RAID GEOMs (even broken) and
then remove them with `graid remove Promise ad4`.

--
Alexander Motin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Matt-332

Re: Speaking of ship blockers for 9....

In reply to this post by Garrett Cooper

On 08/07/12 11:43, Garrett Cooper wrote:

> On Aug 7, 2012, at 11:17 AM, Ian FREISLICH <[hidden email]> wrote:
>
>> Garrett Cooper
>>> Is this is in 9.1 -PRERELEASE, -RELEASE (or whatever the official
>>> label is...)? If so, it seems like this would be a ship blocker.
>> I have a problem that's been getting progressively worse as the
>> source progresses. So much so that it's had me searching all the
>> way from 8.0-RELEASE to 10-CURRENT without luck on both amd64 and
>> i386.
>>
>> pf(4) erroneously mismatches state and then blocks an active flow.
>> It seems that 8.X does so silently and 9 to -CURRENT do so verbosely.
>> Whether silent or loud, the effect on traffic makes it impracticle
>> to use FreeBSD+PF for a firewall in any setting (my use is home,
>> small office, large office and moderately large datacenter core
>> router). It appears that this has actually been a forever problem
>> that just being tickled more now.
>>
>> Here's from my home firewall:
>> Status: Enabled for 7 days 02:57:58 Debug: Urgent
>>
>> State Table Total Rate
>> current entries 1653
>> searches 45792251 74.4/s
>> inserts 428375 0.7/s
>> removals 426722 0.7/s
>> ...
>> state-mismatch 1586 0.0/s
>>
>>
>> Here's from a moderately busy firewall:
>> Status: Enabled for 0 days 21:40:44 Debug: Urgent
>>
>> State Table Total Rate
>> current entries 122395
>> searches 4428641685 56745.4/s
>> inserts 202644593 2596.5/s
>> removals 202522198 2595.0/s
>> ...
>> state-mismatch 277767 3.6/s
>>
>> That's 277767 flows terminated in the last almost 22 hours due to
>> this pf bug. (!!!)
>>
>> 9.1-PRERELEASE logs (as does -CURRENT):
>> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60985, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
>> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:52995, a1: 41.154.2.100:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
>> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60095, a1: 206.223.136.200:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
>> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:50463, a1: 206.223.136.200:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
>> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:56748, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
>> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60793, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.
> Filed a PR yet with packet captures?
> Thanks,
> -Garrett_______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"
>

I was having this problem on one machine but not another (different
pf.confs). Are you using synproxy state or modulate state? Feel OK
posting a basic pf.conf that experiences the issue?

I feel like there was something with either scrub or synproxy I had to
remove to make the hurting stop.
Obviously that means something is probably borked, and I will share in
the no-pr shame.

Matt
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Tom Uffner

Re: geom mirror now rebuilding on every reboot?

In reply to this post by Alexander Motin-3

Alexander Motin wrote:
> I think the right answer would be to remove stale Promise format RAID metadata
> from the disk. You should be able to do it by booting from any source and
> doing `graid list -a` to see all RAID GEOMs (even broken) and then remove them
> with `graid remove Promise ad4`.

thanks, that fixed it.

tom
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

Gleb Smirnoff

Re: Speaking of ship blockers for 9....

In reply to this post by Ian FREISLICH-3

Ian,

On Tue, Aug 07, 2012 at 08:17:56PM +0200, Ian FREISLICH wrote:
I> I have a problem that's been getting progressively worse as the
I> source progresses. So much so that it's had me searching all the
I> way from 8.0-RELEASE to 10-CURRENT without luck on both amd64 and
I> i386.
I>
I> pf(4) erroneously mismatches state and then blocks an active flow.
I> It seems that 8.X does so silently and 9 to -CURRENT do so verbosely.
I> Whether silent or loud, the effect on traffic makes it impracticle
I> to use FreeBSD+PF for a firewall in any setting (my use is home,
I> small office, large office and moderately large datacenter core
I> router). It appears that this has actually been a forever problem
I> that just being tickled more now.
...
I> ...
I> state-mismatch 277767 3.6/s
I>
I> That's 277767 flows terminated in the last almost 22 hours due to
I> this pf bug. (!!!)
I>
I> 9.1-PRERELEASE logs (as does -CURRENT):
I> Jul 22 08:54:25 brane kernel: pf: state key linking mismatch! dir=OUT, if=tun0, stored af=2, a0: 10.0.2.220:60985, a1: 192.41.162.30:53, proto=17, found af=2, a0: 41.154.2.53:1701, a1: 41.133.165.161:59051, proto=17.

Let me give you link to my branch of pf:

http://lists.freebsd.org/pipermail/freebsd-pf/2012-June/006643.html
http://lists.freebsd.org/pipermail/freebsd-pf/2012-June/006662.html

In that branch the code that puts the "reverse" pointer on state keys,
as well as the m_addr_changed() function and the pf_compare_state_keys()
had been cut away.

So, this exact bug definitely can't be reproduced there. However, others
may hide in :)

Let me encourage you to try and test my branch (instructions in URLs
above).

P.S. I plan to merge it to head at the and of August.

--
Totus tuus, Glebius.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"