Query about the usage of FreeBSD mirror sites for other projects

[tabthorpe]

Greetings mirror maintainers and others.

Recently, a question came up within our community, should we be using
other project sites for distfile fetching?  Sites like Debian and Gentoo
mirror many of the tarballs we use for our ports.

I engaged the infrastructure people and the mirror maintainers at the
Debian and Gentoo projects, and the answer was simply yes, feel free to
use them for your purposes, basically use but don't abuse it, standard
rules of the sandbox :)

Which now rather begs to ask the question, what is the opinion of our
mirror maintainers?  How do you feel about non-FreeBSD projects using
our resources?  Our community supports and sponsors some of these sites,
should others be allowed to use them?

Please feel free to reply publicly, and private emails directly to me
will be held in the strictist of confidence.

I am curious to see how our people feel on the matter.

Thanks for your time and attention to this matter.


Thomas

--
Thomas Abthorpe | FreeBSD Committer
[hidden email] | http://people.freebsd.org/~tabthorpe

[Paul Mezzanini]

Personally, I mirror all of the mentioned projects and more.  I don't mind
if we "cross the streams" but it would mess up when I run stats on my log
files.   A very minor issue for me but it may mess up other mirror setups.

The real issue is what if FreeBSD relies on a package that Gentoo has in
the mirror structure and then Gentoo prunes it out.  (Examples only, feel
free to change the project names to any that fit your mental model)

I wonder how useful a setup would be that is just a giant bucket for all
project to pull from.  It has the potential to be great but also horrible.
 If hardlinking was used across the whole tree then each project could
have its own directory hardlinked across each other.  Each project could
then add and delete from their own tree without worry about destroying a
dependency somewhere else.


I'm going to run a hardlink.py dry run on my whole mirror to see what the
potential savings will be.  I'm betting it will actually be fairly low.

-paul
(aka mirrors.rit.edu)

On 2/20/12 9:39 PM, "Thomas Abthorpe" <[hidden email]> wrote:

>Greetings mirror maintainers and others.
>
>Recently, a question came up within our community, should we be using
>other project sites for distfile fetching?  Sites like Debian and Gentoo
>mirror many of the tarballs we use for our ports.
>
>I engaged the infrastructure people and the mirror maintainers at the
>Debian and Gentoo projects, and the answer was simply yes, feel free to
>use them for your purposes, basically use but don't abuse it, standard
>rules of the sandbox :)
>
>Which now rather begs to ask the question, what is the opinion of our
>mirror maintainers?  How do you feel about non-FreeBSD projects using
>our resources?  Our community supports and sponsors some of these sites,
>should others be allowed to use them?
>
>Please feel free to reply publicly, and private emails directly to me
>will be held in the strictist of confidence.
>
>I am curious to see how our people feel on the matter.
>
>Thanks for your time and attention to this matter.
>
>
>Thomas
>
>--
>Thomas Abthorpe | FreeBSD Committer
>[hidden email] | http://people.freebsd.org/~tabthorpe

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"

[Fabian Wenk]

Hello Thomas

On 21.02.2012 03:39, Thomas Abthorpe wrote:
> Recently, a question came up within our community, should we be using
> other project sites for distfile fetching?  Sites like Debian and Gentoo
> mirror many of the tarballs we use for our ports.

For my local FreeBSD and Gentoo servers I use a shared distfiles
folder over NFS. All systems are allowed to save new downloads,
and if the file already exists, they are using it instead of
downloading.

As far as I see it, in the FreeBSD distfiles there are also sub
directories, where Gentoo does not use sub directories.

I did a quick check with sysutils/fdupes and there are a few
files which are saved at two different locations. On my systems
the following projects are affected from this:
Perl, Ruby, Apache and Gnome

The files with root:wheel have been downloaded from FreeBSD
Ports, and the files with portage:portage from Gentoo (sorry for
the line wraps):

# cd /usr/ports/distfiles

# find . -name perl-5.12.4.tar.bz2 -ls
329842 24160 -rw-r--r-- 1 root    wheel   12350353 Jun 20  2011
./perl/perl-5.12.4.tar.bz2
237981 24160 -rw-rw-r-- 1 portage portage 12350353 Nov 10  2010
./perl-5.12.4.tar.bz2

# find . -name ruby-1.8.7-p352.tar.bz2 -ls
237747  8256 -rw-r--r-- 1 root    wheel    4207576 Jul  2  2011
./ruby/ruby-1.8.7-p352.tar.bz2
237991  8256 -rw-rw-r-- 1 portage portage  4207576 Jul  2  2011
./ruby-1.8.7-p352.tar.bz2

# find . -name httpd-2.2.21.tar.bz2 -ls
282742 10464 -rw-r--r-- 1 root    wheel    5324905 Sep 12 19:02
./apache22/httpd-2.2.21.tar.bz2
237936 10464 -rw-rw-r-- 1 portage portage  5324905 Sep 15 11:50
./httpd-2.2.21.tar.bz2

# find . -name glib-2.28.8.tar.xz -ls
259229 10240 -rw-r--r-- 1 root    wheel    5223564 Jun  6  2011
./gnome2/glib-2.28.8.tar.xz
237741 10240 -rw-rw-r-- 1 portage portage  5223564 Jun  6  2011
./glib-2.28.8.tar.xz

As far as I see it, it is the same structure on the mirror
servers too. So I guess to really be able to use each others
mirrors, some adjustments are needed either in Ports (FreeBSD) or
in Portage (Gentoo). Else some of the downloads will fail or use
the fall back download source.


bye
Fabian
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"

[Oliver Fromme]

Paul Mezzanini wrote:
 > The real issue is what if FreeBSD relies on a package that Gentoo has in
 > the mirror structure and then Gentoo prunes it out.  (Examples only, feel
 > free to change the project names to any that fit your mental model)
 >
 > I wonder how useful a setup would be that is just a giant bucket for all
 > project to pull from.  It has the potential to be great but also horrible.

Ultimately, it would be cool to have a more flexible and
dynamic system for distfiles.

Clients wouldn't download directly from certain FTP mirrors
anymore, but they would contact an "index server".  The index
servers maintain a list of locations where each distfile can
be found, similar to a very specialized search engine.  These
locations can be FreeBSD mirrors or mirrors of other projects
(Debian, Gentoo, whatever) or just random sites anywhere on
the internet.  The "MASTER_SITES" information currently
contained in the ports' Makefiles will be moved to the index
servers.

Either the index server returns a list of locations back to
the client (just like a search engine), so the client can
choose to download from one of them.  Or, alternatively,
the index server also acts as a proxy, so it fetches the
distfile from one of the known locations and hands it to the
client.  In this case it might also make sense to cache the
distfiles on the index server for a limited time (depending
on how often this file is requested, or its size).

I'm aware that such a system would require quite some work.
It's just an idea that came to my mind, but maybe it's worth
further investigation.  Maybe it should get an entry on
FreeBSD's "project ideas" page.

Best regards
   Oliver

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"The ITU has offered the IETF formal alignment with its
corresponding technology, Penguins, but that won't fly."
        -- RFC 2549
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"

[Fabian Wenk]

Hello Oliver

On 21.02.2012 16:29, Oliver Fromme wrote:

> Ultimately, it would be cool to have a more flexible and
> dynamic system for distfiles.
>
> Clients wouldn't download directly from certain FTP mirrors
> anymore, but they would contact an "index server".  The index
> servers maintain a list of locations where each distfile can
> be found, similar to a very specialized search engine.  These
> locations can be FreeBSD mirrors or mirrors of other projects
> (Debian, Gentoo, whatever) or just random sites anywhere on
> the internet.  The "MASTER_SITES" information currently

I really like it, that I can download from a mirror which is near
on the network, as it is much faster then from anywhere else on
the internet. So random distribution will just generate more
global network traffic, which is often more costly (for the ISPs)
then local traffic with free peering between regional ISPs.

> I'm aware that such a system would require quite some work.
> It's just an idea that came to my mind, but maybe it's worth
> further investigation.  Maybe it should get an entry on
> FreeBSD's "project ideas" page.

Eventually doing something with regional reverse caching proxy,
which is able to get missing files in the background from the
master. Subsequent request will then be served from the local cache.


bye
Fabian
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"

[Oliver Fromme]

Fabian Wenk wrote:
 > On 21.02.2012 16:29, Oliver Fromme wrote:
 > > Ultimately, it would be cool to have a more flexible and
 > > dynamic system for distfiles.
 > >
 > > Clients wouldn't download directly from certain FTP mirrors
 > > anymore, but they would contact an "index server".  The index
 > > servers maintain a list of locations where each distfile can
 > > be found, similar to a very specialized search engine.  These
 > > locations can be FreeBSD mirrors or mirrors of other projects
 > > (Debian, Gentoo, whatever) or just random sites anywhere on
 > > the internet.  The "MASTER_SITES" information currently
 >
 > I really like it, that I can download from a mirror which is near
 > on the network, as it is much faster then from anywhere else on
 > the internet.

The index server / proxy system that I suggested would also
allow that, of course.

Best regards
   Oliver

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

$ dd if=/dev/urandom of=test.pl count=1
$ file test.pl
test.pl: perl script text executable
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"

[Kevin Day-3]

On Feb 20, 2012, at 8:39 PM, Thomas Abthorpe wrote:

> Which now rather begs to ask the question, what is the opinion of our
> mirror maintainers?  How do you feel about non-FreeBSD projects using
> our resources?  Our community supports and sponsors some of these sites,
> should others be allowed to use them?
>
> Please feel free to reply publicly, and private emails directly to me
> will be held in the strictist of confidence.
>
> I am curious to see how our people feel on the matter.
>
> Thanks for your time and attention to this matter.
>

We run ftp3.us.freebsd.org, cvsup6.us.freebsd.org, etc which is also home to several other projects available by ftp/http/rsync/cvsup/etc.

We really don't care who uses the data we've mirrored, if we're putting it up to the public, anyone is free to grab/link to it.

-- Kevin

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hubs
To unsubscribe, send any mail to "[hidden email]"

[tabthorpe]

On Tue, Feb 21, 2012 at 12:10:38PM -0600, Kevin Day wrote:

>
> On Feb 20, 2012, at 8:39 PM, Thomas Abthorpe wrote:
> > Which now rather begs to ask the question, what is the opinion of our
> > mirror maintainers?  How do you feel about non-FreeBSD projects using
> > our resources?  Our community supports and sponsors some of these sites,
> > should others be allowed to use them?
> >
> > Please feel free to reply publicly, and private emails directly to me
> > will be held in the strictist of confidence.
> >
> > I am curious to see how our people feel on the matter.
> >
> > Thanks for your time and attention to this matter.
> >
>
>
> We run ftp3.us.freebsd.org, cvsup6.us.freebsd.org, etc which is also home to several other projects available by ftp/http/rsync/cvsup/etc.
>
> We really don't care who uses the data we've mirrored, if we're putting it up to the public, anyone is free to grab/link to it.
>
> -- Kevin
>

Thanks Kevin.

The general sentiment I have been receiving from mirror maintainers who
have replied is just that, it is a public site, feel free to use it,
just don't abuse it.

I suspect that the sense of sharing comes from our choice to work in
opensource projects.


Thomas

--
Thomas Abthorpe | FreeBSD Committer
[hidden email] | http://people.freebsd.org/~tabthorpe