[zfs] need help with a bug (PR 156781)

[Patrick Proniewski]

Hello,

I'm running various FreeBSD 8 amd64 servers, and some of them use ZFS extensively. Unfortunately, I'm hit by the bug "PR 156781" (<http://www.FreeBSD.org/cgi/query-pr.cgi?pr=156781&cat=kern>).

After an undetermined uptime, the affected server will lose the ability to list some of its .zfs/.snappshot:

ls -l .zfs
ls: snapshot: Bad file descriptor

It doesn't look critical at this point, because snapshots still work (creation/deletion), but you won't be able to access them via cd .zfs/snapshot/...

In fact, it is critical. If you try to unmount the affected ZFS volume, your shell will freeze, and you have to reconnect in a new one. And more importantly, the whole OS will freeze when you issue a reboot, meaning it will never reboot until someone pushes the power button of the box.
It's a very nasty bug when your box is 400 km away in a data center, and you need to ask your hosting provider to push that button.

I would be happy if the PR 156781 could be modified so that it says its Severity is critical, and it affects both FreeBSD 8.1-RELEASE-p5 amd64, FreeBSD 8.2-RELEASE-p3 amd64, and FreeBSD 8.2-STABLE sparc64 (ie. not just FreeBSD 8.2-STABLE sparc64).
And I would be very very happy, if someone could fix this bug, it's unfortunately way out of my reach :/


        FreeBSD si 8.2-RELEASE-p3 FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011    
        [hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64

        $ grep ZFS /var/run/dmesg.boot
        ZFS filesystem version 4
        ZFS storage pool version 15

        $ zpool upgrade -a
        This system is currently running ZFS pool version 15.

        All pools are formatted using this version.

        $ zfs upgrade -a
        0 filesystems upgraded
        265 filesystems already at this version

and

        FreeBSD rack.patpro.net 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #0: Tue Sep 27 16:49:00 UTC 2011    
        [hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64

        $ grep ZFS /var/run/dmesg.boot
        ZFS filesystem version 3
        ZFS storage pool version 14

        $ zpool upgrade -a
        This system is currently running ZFS pool version 14.

        All pools are formatted using this version.

        $ zfs upgrade -ra
        0 filesystems upgraded
        19 filesystems already at this version



regards,
patpro
(I'm subscribed to the digest, fell free to Cc. me)_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Mark Felder-4]

FreeBSD 8.1 and 8.2 are actually quite old these days. I don't think
this bug is reproducible on 8.3 or 9.0. Can you upgrade and report back?


Thanks

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Matthew Seaman-5]

On 11/06/2012 00:36, Mark Felder wrote:
> FreeBSD 8.1 and 8.2 are actually quite old these days. I don't think
> this bug is reproducible on 8.3 or 9.0. Can you upgrade and report back?

I'm seeing exactly this on stable/9.  Seems to be triggered by doing
certain IO patterns -- so csup of the ports cvs repo and subsequent
checkout will cause the effect, but direct csup of the ports tree doesn't.

This is using a zpool created under 8.0-RELEASE and subsequently
updated.  I wanted to try splitting the mirror using zfs {send,receive}
to copy the filesystem into a brand-new zpool to see if that made any
difference, but this is my main server and I haven't had the chance to
do that yet.

        Cheers,

        Matthew

--
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey

[Andriy Gapon]

I can observer a similar problem where I can not list .zfs directory under
filesystems which do not have any snapshots.  I can not tell if the problem
happens when all snapshots are removed or if it only happens if a filesystem
doesn't have any snapshots when ZFS is initialized (a system is booted).

I am not sure what's going on but a quick check with DTrace points to negative
caching as a culprit.  I see that cache_lookup() returns ENOENT for .zfs.


--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Patrick Proniewski]

Hi,

On 11 juin 2012, at 12:30, Andriy Gapon wrote:

> I am not sure what's going on but a quick check with DTrace points to negative
> caching as a culprit.  I see that cache_lookup() returns ENOENT for .zfs.

What is the FreeBSD release? And do you experience similar freezes, or just the listing problem?

regards,
patpro_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Patrick Proniewski]

On 11 juin 2012, at 06:29, Matthew Seaman wrote:

> On 11/06/2012 00:36, Mark Felder wrote:
>> FreeBSD 8.1 and 8.2 are actually quite old these days. I don't think
>> this bug is reproducible on 8.3 or 9.0. Can you upgrade and report back?
>
> I'm seeing exactly this on stable/9.  Seems to be triggered by doing
> certain IO patterns -- so csup of the ports cvs repo and subsequent
> checkout will cause the effect, but direct csup of the ports tree doesn't.

Ok, I'm not exactly relieved the bug hits also 9.0, but it's good news I won't have to plan a remote upgrade of FreeBSD and ZFS over ssh. I master the process, having done FreeBSD upgrades over ssh since 5.x (either sources and binary way), but the less I do it, the better I feel.

> This is using a zpool created under 8.0-RELEASE and subsequently
> updated.

mine have been created under 8.1 and 8.2 respectively, as far as I remember (btw, is there any way to list the change history of a system, other than keeping manual trace of every updates/upgrades?).

regards,
patpro

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Ronald Klop-2]

On Mon, 11 Jun 2012 15:52:03 +0200, Patrick Proniewski <[hidden email]>  
wrote:

> On 11 juin 2012, at 06:29, Matthew Seaman wrote:
>
>> On 11/06/2012 00:36, Mark Felder wrote:
>>> FreeBSD 8.1 and 8.2 are actually quite old these days. I don't think
>>> this bug is reproducible on 8.3 or 9.0. Can you upgrade and report  
>>> back?
>>
>> I'm seeing exactly this on stable/9.  Seems to be triggered by doing
>> certain IO patterns -- so csup of the ports cvs repo and subsequent
>> checkout will cause the effect, but direct csup of the ports tree  
>> doesn't.
>
> Ok, I'm not exactly relieved the bug hits also 9.0, but it's good news I  
> won't have to plan a remote upgrade of FreeBSD and ZFS over ssh. I  
> master the process, having done FreeBSD upgrades over ssh since 5.x  
> (either sources and binary way), but the less I do it, the better I feel.
>
>> This is using a zpool created under 8.0-RELEASE and subsequently
>> updated.
>
> mine have been created under 8.1 and 8.2 respectively, as far as I  
> remember (btw, is there any way to list the change history of a system,  
> other than keeping manual trace of every updates/upgrades?).

zpool history


> regards,
> patpro
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Andreas Nilsson-8]

On Mon, Jun 11, 2012 at 4:17 PM, Ronald Klop <[hidden email]>wrote:

> On Mon, 11 Jun 2012 15:52:03 +0200, Patrick Proniewski <[hidden email]>
> wrote:
>
>  On 11 juin 2012, at 06:29, Matthew Seaman wrote:
>>
>>  On 11/06/2012 00:36, Mark Felder wrote:
>>>
>>>> FreeBSD 8.1 and 8.2 are actually quite old these days. I don't think
>>>> this bug is reproducible on 8.3 or 9.0. Can you upgrade and report back?
>>>>
>>>
>>> I'm seeing exactly this on stable/9.  Seems to be triggered by doing
>>> certain IO patterns -- so csup of the ports cvs repo and subsequent
>>> checkout will cause the effect, but direct csup of the ports tree
>>> doesn't.
>>>
>>
>> Ok, I'm not exactly relieved the bug hits also 9.0, but it's good news I
>> won't have to plan a remote upgrade of FreeBSD and ZFS over ssh. I master
>> the process, having done FreeBSD upgrades over ssh since 5.x (either
>> sources and binary way), but the less I do it, the better I feel.
>>
>>  This is using a zpool created under 8.0-RELEASE and subsequently
>>> updated.
>>>
>>
>> mine have been created under 8.1 and 8.2 respectively, as far as I
>> remember (btw, is there any way to list the change history of a system,
>> other than keeping manual trace of every updates/upgrades?).
>>
>
> zpool history
>
> Just out of interest: did you upgrade pool as part of 8->9 update, ie

zpool upgrade?

/Andreas

>
>
>  regards,
>> patpro
>>
> ______________________________**_________________
> [hidden email] mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-fs<http://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org<[hidden email]>
> "
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"

[Matthew Seaman-5]

On 11/06/2012 14:52, Patrick Proniewski wrote:
> mine have been created under 8.1 and 8.2 respectively, as far as I
> remember (btw, is there any way to list the change history of a
> system, other than keeping manual trace of every updates/upgrades?).

# zpool history zroot
History for 'zroot':
2010-05-14.20:31:21 zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk2
2010-05-14.20:31:43 zpool set bootfs=zroot zroot
[...etc...]

        Cheers,

        Matthew

--
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey

[Patrick Proniewski]

On 11 juin 2012, at 17:15, Matthew Seaman wrote:

> On 11/06/2012 14:52, Patrick Proniewski wrote:
>> mine have been created under 8.1 and 8.2 respectively, as far as I
>> remember (btw, is there any way to list the change history of a
>> system, other than keeping manual trace of every updates/upgrades?).
>
> # zpool history zroot
> History for 'zroot':
> 2010-05-14.20:31:21 zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk2
> 2010-05-14.20:31:43 zpool set bootfs=zroot zroot
> [...etc...]

I know zpool history, I was refering to something more "system-wide", keeping track of updates and upgrades of FreeBSD. I use freebsd-update for 1 or 2 years now, but can't find a proper log file. The purpose would be to know for sure what release was installed when I created my zfs pools.

patpro

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"