buildworld fails on FreeBSD 7.x for HEAD from 19.04.2012

[Jan Sieka]

Hello!

I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48 UTC) on
a machine running FreeBSD 7.3. I know it's EoLed, but I have found information
in getline(3) manual that this function first appeared in FreeBSD 8, so I
presume that the problem is present also for FreeBSD 7.4 (which is still before
EoL). I have confirmed that problem doesn't exists for FreeBSD 9.0.

Here's excerpt from build log:

===> lib/libmagic (obj,build-tools)
cc -O2 -pipe -DMAGIC='"/usr/share/misc/magic"' -DHAVE_CONFIG_H
-I/usr/src/current/lib/libmagic
-I/usr/src/current/lib/libmagic/../../contrib/file
-std=gnu99
-I/usr/obj/usr/src/current/tmp/legacy/usr/include
-DCOMPILE_ONLY
-L/usr/obj/usr/src/current/tmp/legacy/usr/lib
-o mkmagic
/usr/src/current/lib/libmagic/../../contrib/file/apprentice.c
/usr/src/current/lib/libmagic/../../contrib/file/funcs.c
/usr/src/current/lib/libmagic/../../contrib/file/magic.c
/usr/src/current/lib/libmagic/../../contrib/file/print.c
/usr/src/current/lib/libmagic/../../contrib/file/apprentice.c:
In function 'load_1':
/usr/src/current/lib/libmagic/../../contrib/file/apprentice.c:701:
warning: implicit declaration of function 'getline'
/var/tmp//cc7NqjVu.o(.text+0x699): In function `load_1':
: undefined reference to `getline'
*** Error code 1

Stop in /usr/src/current/lib/libmagic.
*** Error code 1

Stop in /usr/src/current.
*** Error code 1

Stop in /usr/src/current.
*** Error code 1

Stop in /usr/src/current.


Regards,

Jan Sieka
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[dougb]

On 4/20/2012 5:16 AM, Jan Sieka wrote:
> I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48
> UTC) on a machine running FreeBSD 7.3.

That's not a supported configuration. We don't promise support for
$VERSION on anything less than the most recent version of $VERSION - 1.

--

    This .signature sanitized for your protection
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Adrian Chadd-2]

Not even a full crossbuild?



adrian
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Garrett Cooper]

On Apr 20, 2012, at 10:26 AM, Doug Barton wrote:

> On 4/20/2012 5:16 AM, Jan Sieka wrote:
>> I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48
>> UTC) on a machine running FreeBSD 7.3.
>
> That's not a supported configuration. We don't promise support for
> $VERSION on anything less than the most recent version of $VERSION - 1.

        I'm sorry, but given the error below shown by Jan I don't buy this argument. It seems like file's objects are is trying to link against an ABI incompatible libc and failing. The build system should handle this properly if you invoke the top-level targets (buildworld, buildkernel, distribution).
Thanks,
-Garrett_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Dimitry Andric-4]

On 2012-04-22 16:51, Garrett Cooper wrote:
> On Apr 20, 2012, at 10:26 AM, Doug Barton wrote:
>> On 4/20/2012 5:16 AM, Jan Sieka wrote:
>>> I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48
>>> UTC) on a machine running FreeBSD 7.3.
>>
>> That's not a supported configuration. We don't promise support for
>> $VERSION on anything less than the most recent version of $VERSION - 1.
>
> I'm sorry, but given the error below shown by Jan I don't buy this argument. It seems like file's objects are is trying to link against an ABI incompatible libc and failing. The build system should handle this properly if you invoke the top-level targets (buildworld, buildkernel, distribution).

This is trickier than you think: the error occurs during the build-tools
stage, where everything that builds still uses the host's toolchain and
libraries (e.g. the 7.x versions).  So if you try to build a program
that uses getline(), it will not be able to link.  This is what happens
with the 'mkmagic' tool.

However, this will have started failing just very recently, since obrien
updated to a new file(1) version, and regenerated the config.h file:

  http://svnweb.freebsd.org/base/head/lib/libmagic/config.h?r1=208341&r2=234449

Maybe some hackery could be inserted in lib/libmagic/Makefile, so during
the early stages a 'toned down' config.h file is used.  The libmagic
source has its own getline implementation in contrib/file/getline.c, so
maybe that could be used instead.

On the other hand, going from 7.x to HEAD via the 'official' route would go like:

- Update 7.x to the latests 7-STABLE
- Update 7-STABLE to 8-STABLE
- Update 8-STABLE to 9-STABLE
- Update 9-STABLE to 10-CURRENT

It is *way* easier to just grab a recent 10-CURRENT snapshot ISO and
just reinstall. :)
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Garrett Cooper]

On Apr 22, 2012, at 8:51 AM, Dimitry Andric wrote:

> On 2012-04-22 16:51, Garrett Cooper wrote:
>> On Apr 20, 2012, at 10:26 AM, Doug Barton wrote:
>>> On 4/20/2012 5:16 AM, Jan Sieka wrote:
>>>> I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48
>>>> UTC) on a machine running FreeBSD 7.3.
>>>
>>> That's not a supported configuration. We don't promise support for
>>> $VERSION on anything less than the most recent version of $VERSION - 1.
>>
>> I'm sorry, but given the error below shown by Jan I don't buy this argument. It seems like file's objects are is trying to link against an ABI incompatible libc and failing. The build system should handle this properly if you invoke the top-level targets (buildworld, buildkernel, distribution).
>
> This is trickier than you think: the error occurs during the build-tools
> stage, where everything that builds still uses the host's toolchain and
> libraries (e.g. the 7.x versions).  So if you try to build a program
> that uses getline(), it will not be able to link.  This is what happens
> with the 'mkmagic' tool.
>
> However, this will have started failing just very recently, since obrien
> updated to a new file(1) version, and regenerated the config.h file:
>
>  http://svnweb.freebsd.org/base/head/lib/libmagic/config.h?r1=208341&r2=234449

And this is why NetBSD runs autotools on their tree sources in their build instead of dealing with stale config.h files -- which also brings up the bug of what happens if support is available on one architecture, but not the other? Doesn't happen often and I hope it wouldn't happen in sources imported into FreeBSD, but I've seen it happen in the past with various open source projects.

> Maybe some hackery could be inserted in lib/libmagic/Makefile, so during
> the early stages a 'toned down' config.h file is used.  The libmagic
> source has its own getline implementation in contrib/file/getline.c, so
> maybe that could be used instead.

That seems like a good idea for the time being.

On the other hand (and this is probably a dumb question), but why is libmagic required in the build-tools stage?

> On the other hand, going from 7.x to HEAD via the 'official' route would go like:
>
> - Update 7.x to the latests 7-STABLE
> - Update 7-STABLE to 8-STABLE
> - Update 8-STABLE to 9-STABLE
> - Update 9-STABLE to 10-CURRENT
>
> It is *way* easier to just grab a recent 10-CURRENT snapshot ISO and
> just reinstall. :)

Ugh. The usecase (that's now broken) is that Jan from Semihalf might have been running CURRENT builds on an older (stable) build machine. I have done this a couple times in the past (but not regularly because I usually follow the above prescribed method or just hop 1-2 major versions).

Other groups I've worked with use the same major version or higher than the running build because of chroot hackery involved in the build process.

Thanks,
-Garrett_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Dimitry Andric-4]

On 2012-04-22 18:06, Garrett Cooper wrote:
> On Apr 22, 2012, at 8:51 AM, Dimitry Andric wrote:
...
>> However, this will have started failing just very recently, since obrien
>> updated to a new file(1) version, and regenerated the config.h file:
>>
>>  http://svnweb.freebsd.org/base/head/lib/libmagic/config.h?r1=208341&r2=234449
>
> And this is why NetBSD runs autotools on their tree sources in their build instead of dealing with stale config.h files

Well, I wouldn't want to run autoconf during build, firstly because it
is horribly slow, and second because the results will be less
predictable.  Maybe during the bootstrap stage, it would be acceptable.

But even then, one of the configure scripts could fail due to too-old
system components, and you would be SOL.


> which also brings up the bug of what happens if support is available on one architecture, but not the other? Doesn't happen often and I hope it wouldn't happen in sources imported into FreeBSD, but I've seen it happen in the past with various open source projects.

Usually, if something is arch-dependent in a config.h file, we simply
surround it with #ifdefs.


>> Maybe some hackery could be inserted in lib/libmagic/Makefile, so during
>> the early stages a 'toned down' config.h file is used.  The libmagic
>> source has its own getline implementation in contrib/file/getline.c, so
>> maybe that could be used instead.
>
> That seems like a good idea for the time being.
>
> On the other hand (and this is probably a dumb question), but why is libmagic required in the build-tools stage?

Apparently the file(1) build needs a 'mkmagic' tool, which generates
.mgc files (the 'compiled' version of magic files).  This requirement
was originally added in r81845, more than 10 years ago.

>> On the other hand, going from 7.x to HEAD via the 'official' route would go like:
>>
>> - Update 7.x to the latests 7-STABLE
>> - Update 7-STABLE to 8-STABLE
>> - Update 8-STABLE to 9-STABLE
>> - Update 9-STABLE to 10-CURRENT
>>
>> It is *way* easier to just grab a recent 10-CURRENT snapshot ISO and
>> just reinstall. :)
>
> Ugh. The usecase (that's now broken) is that Jan from Semihalf might have been running CURRENT builds on an older (stable) build machine. I have done this a couple times in the past (but not regularly because I usually follow the above prescribed method or just hop 1-2 major versions).

Yes, it might work, but there is no guarantee.  I'm not sure if there is
enough incentive to change this policy.  It would potentially require a
lot effort to make it always work.


> Other groups I've worked with use the same major version or higher than the running build because of chroot hackery involved in the build process.

I wasn't aware of any chroot hackery?
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Garrett Cooper]

On Apr 22, 2012, at 10:34 AM, Dimitry Andric wrote:

> Well, I wouldn't want to run autoconf during build, firstly because it
> is horribly slow, and second because the results will be less
> predictable.  Maybe during the bootstrap stage, it would be acceptable.

Sure -- that seems reasonable.

> But even then, one of the configure scripts could fail due to too-old
> system components, and you would be SOL.

… but it would be a step forward from where things are currently at. I'm not sure how well tested "source upgrade" paths are, but being able to upgrade from the lowest supported version to the latest supported version, then upgrading to CURRENT (at the very least) would be nice.

> Usually, if something is arch-dependent in a config.h file, we simply
> surround it with #ifdefs.

Makes sense (assumption being that it can be controlled via the config.h/configure.{ac,in} file). However, jemalloc recently disproved this >_<.

> Apparently the file(1) build needs a 'mkmagic' tool, which generates
> .mgc files (the 'compiled' version of magic files).  This requirement
> was originally added in r81845, more than 10 years ago.

I tested out removing libmagic from Makefile.inc1 and see that there's some dependency magic going on there where building the library failed.

> Yes, it might work, but there is no guarantee.  I'm not sure if there is
> enough incentive to change this policy.  It would potentially require a
> lot effort to make it always work.

Understood and I guess the ownness is upon the stakeholders to fix this, but there are a lot of companies that depend on things like this working (at least to reduce pain when doing source upgrades). This would probably be less of an issue for developers that use freebsd-update or for companies that roll their own freebsd-update (and servers). I have yet to run into a company that does this though (not saying there aren't groups that could or do do this, but it's not the standard path).

> I wasn't aware of any chroot hackery?

A publicly available example is available in FreeNAS ( http://freenas.svn.sourceforge.net/viewvc/freenas?view=revision&revision=8193 ); the hangup is building packages for a target system that doesn't match the build host.

Cheers!
-Garrett_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[David O'Brien]

On Sun, Apr 22, 2012 at 09:06:18AM -0700, Garrett Cooper wrote:
> >>> On 4/20/2012 5:16 AM, Jan Sieka wrote:
> >>>> I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48
> >>>> UTC) on a machine running FreeBSD 7.3.
...
> Ugh. The usecase (that's now broken) is that Jan from Semihalf might
> have been running CURRENT builds on an older (stable) build machine.

Lets not guess.  If you've found that any version of 10-CURRENT cannot
build HEAD post r234449 please let me know.

I've verified that I can build HEAD on 8.3-PRERELEASE (r231882).

--
-- David  ([hidden email])
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Vladimir Sharun-2]

===> usr.bin/file (all)
/usr/bin/clang -O2 -pipe  -DMAGIC='"/usr/share/misc/magic"'
-DHAVE_CONFIG_H -I/usr/src/usr.bin/file/../../lib/libmagic -std=gnu99
-Qunused-arguments -fstack-protector -Wsystem-headers -Wall
-Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
-Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual
-Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align
-Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls
-Wold-style-definition -Wno-pointer-sign -Wno-empty-body
-Wno-string-plus-int  -o file file.o -lmagic -lz
file.o: In function `main':
/usr/src/usr.bin/file/../../contrib/file/file.c:(.text+0x717): undefined
reference to `magic_getpath'
/usr/src/usr.bin/file/../../contrib/file/file.c:(.text+0x7df): undefined
reference to `magic_list'
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
*** [file] Error code 1

r234657

clang -v:
FreeBSD clang version 3.1 (trunk 154661) 20120413
Target: x86_64-unknown-freebsd10.0
Thread model: posix

FreeBSD 10.0-CURRENT #6: Thu Apr 12 08:56:05 EEST 2012 amd64

make buildworld without j's

On Sun, Apr 22, 2012 at 09:06:18AM -0700, Garrett Cooper wrote:
> >>> On 4/20/2012 5:16 AM, Jan Sieka wrote:
> >>>> I can't build world from recent sources (HEAD as of 2012.04.19 11:06:48
> >>>> UTC) on a machine running FreeBSD 7.3.
...
> Ugh. The usecase (that's now broken) is that Jan from Semihalf might
> have been running CURRENT builds on an older (stable) build machine.

Lets not guess.  If you've found that any version of 10-CURRENT cannot
build HEAD post r234449 please let me know.

I've verified that I can build HEAD on 8.3-PRERELEASE (r231882).

--
-- David  ([hidden email])
[hidden email] mailing listhttp://lists.freebsd.org/mailman/listinfo/freebsd-currentTo unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[Jan Sieka]

On 2012.04.22 21:07, Garrett Cooper wrote:

> On Apr 22, 2012, at 10:34 AM, Dimitry Andric wrote:
>
>> Well, I wouldn't want to run autoconf during build, firstly because it
>> is horribly slow, and second because the results will be less
>> predictable.  Maybe during the bootstrap stage, it would be acceptable.
>
> Sure -- that seems reasonable.
>
>> But even then, one of the configure scripts could fail due to too-old
>> system components, and you would be SOL.
>
> … but it would be a step forward from where things are currently at. I'm not sure how well tested "source upgrade" paths are, but being able to upgrade from the lowest supported version to the latest supported version, then upgrading to CURRENT (at the very least) would be nice.
>
>> Usually, if something is arch-dependent in a config.h file, we simply
>> surround it with #ifdefs.
>
> Makes sense (assumption being that it can be controlled via the config.h/configure.{ac,in} file). However, jemalloc recently disproved this>_<.
>
>> Apparently the file(1) build needs a 'mkmagic' tool, which generates
>> .mgc files (the 'compiled' version of magic files).  This requirement
>> was originally added in r81845, more than 10 years ago.
>
> I tested out removing libmagic from Makefile.inc1 and see that there's some dependency magic going on there where building the library failed.
>
>> Yes, it might work, but there is no guarantee.  I'm not sure if there is
>> enough incentive to change this policy.  It would potentially require a
>> lot effort to make it always work.
>
> Understood and I guess the ownness is upon the stakeholders to fix this, but there are a lot of companies that depend on things like this working (at least to reduce pain when doing source upgrades). This would probably be less of an issue for developers that use freebsd-update or for companies that roll their own freebsd-update (and servers). I have yet to run into a company that does this though (not saying there aren't groups that could or do do this, but it's not the standard path).
>
>> I wasn't aware of any chroot hackery?
>
> A publicly available example is available in FreeNAS ( http://freenas.svn.sourceforge.net/viewvc/freenas?view=revision&revision=8193 ); the hangup is building packages for a target system that doesn't match the build host.
>
> Cheers!
> -Garrett

Hello!

I have prepared workaround/fix for this issue. It has been tested on FreeBSD 7.3
and 9.0 (both RELEASE) with good results: world build finishes OK. I attach the
patch in case someone else is hit by this issue. Feel free to use it.

Thanks for all suggestions and comments.

Regards,

Jan Sieka


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[David O'Brien]

On Fri, Apr 27, 2012 at 02:05:37PM +0200, Jan Sieka wrote:
> --- a/lib/libmagic/Makefile
> +++ b/lib/libmagic/Makefile
> @@ -10,9 +10,16 @@ DPADD= ${LIBZ}
>  LDADD= -lz
>  MAN= libmagic.3 magic.5
>  
> +HOSTOSRELDATE!= echo ${VERSION} | cut -d " " -f 4

    $ cd lib/libmagic
    $ make -V VERSION
    {empty}

Where is VERSION set?


> +.if ${HOSTOSRELDATE} < 800000
> +GETLINE_C= getline.c
> +.else
> +GETLINE_C=
> +.endif
>  SRCS= apprentice.c apptype.c ascmagic.c cdf.c cdf_time.c compress.c \
>   encoding.c fsmagic.c funcs.c \
> - is_tar.c magic.c print.c readcdf.c readelf.c softmagic.c
> + is_tar.c magic.c print.c readcdf.c readelf.c softmagic.c ${GETLINE_C}

Just
    .if ${HOSTOSRELDATE} < 800000
    GETLINE_C= getline.c
    .endif
is shorter.

Also r189136 (which added getline(3)) bumped __FreeBSD_version from
800066 to 800067.  So your patch is broken for some systems.  Getting
this right is one reason I'm loath to carry around this type of hack
in -CURRENT.


> --- a/lib/libmagic/config.h
> +++ b/lib/libmagic/config.h
>  /* Define to 1 if you have the `getline' function. */
> -#define HAVE_GETLINE 1
> +#if (__FreeBSD_version >= 800000)

Where is "__FreeBSD_version" getting set?

> +#else
> +#define HAVE_GETLINE 0
> +#endif

You've basically defined HAVE_GETLINE always as "0".

--
-- David  ([hidden email])
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"

[David O'Brien]

On Tue, Apr 24, 2012 at 09:34:58PM +0300, Vladimir Sharun wrote:
> ===> usr.bin/file (all)
...
> file.o: In function `main':
> /usr/src/usr.bin/file/../../contrib/file/file.c:(.text+0x717): undefined
> reference to `magic_getpath'
> /usr/src/usr.bin/file/../../contrib/file/file.c:(.text+0x7df): undefined
> reference to `magic_list'
> clang: error: linker command failed with exit code 1 (use -v to see
> invocation)
> *** [file] Error code 1

How are you building this?  Did libmagic get [re]built first?


> r234657

$ svn info -r234657 svn info -r234657 svn://svn.freebsd.org/base/head
Last Changed Date: 2012-04-24 10:51:36 -0700 (Tue, 24 Apr 2012)

I do not believe that a 10-CURRENT system from 2012-04-24 has trouble
building the new file(1).

The issue that started this thread is due to a commit from
2009-02-27 -- which pre-dates 10-CURRENT.

--
-- David  ([hidden email])
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"