Switching /etc/*.db from bdb to tinycdb
Locked
 
|
Hi,
I plan to work on switching all the /etc/*.db files from bdb 1.85 to cdb (using the tinycdb implementation) Reason for doing that: 1/ we use the *.db files as a constant database on recreating them using *_mkdb(8) commands 2/ bdb 1.85 database are endianness dependant which can cause problem if I want pkgng to be able to create users when cross installing packages (like adding packages in a mips eb chroot environment on an amd64 host) 3/ the tinycdb API is clean and easy to use (I find it easier and cleaner to deal with it) 4/ tinycdb is public domain so not problem for us to use it. On the side effects, I'm not sure this is relevant but tinycdb is also faster to read then bdb 1.85. How do I plan to do it. I plan to import the read part or tinycdb into libc and the write part into libutil (to avoid cluttering libc). Why importing to libc? to allow all the get*ent to parse the /etc/*.db files. How to implment it? slowly with backward compatibility: for each kind of files: try to open with cdb, if fail fall back to bdb. I plan to rewrite all the *_mkdb for make them use cdb. Link for tinycdb: http://www.corpit.ru/mjt/tinycdb.html Any opinion, ideas, advices on this? regards, Bapt |
attachment0 (203 bytes)
|
|
|
On Wed, May 02, 2012 at 02:26:36PM +0200, Luigi Rizzo wrote:
> On Wed, May 2, 2012 at 1:41 PM, Baptiste Daroussin <[hidden email]> wrote: > > > Hi, > > > > I plan to work on switching all the /etc/*.db files from bdb 1.85 to cdb > > (using > > the tinycdb implementation) > > > > Reason for doing that: > > 1/ we use the *.db files as a constant database on recreating them using > > *_mkdb(8) commands > > 2/ bdb 1.85 database are endianness dependant which can cause problem if I > > want > > pkgng to be able to create users when cross installing packages (like > > adding > > packages in a mips eb chroot environment on an amd64 host) > > 3/ the tinycdb API is clean and easy to use (I find it easier and cleaner > > to > > deal with it) > > 4/ tinycdb is public domain so not problem for us to use it. > > > > On the side effects, I'm not sure this is relevant but tinycdb is also > > faster to > > read then bdb 1.85. > > > > How do I plan to do it. > > > > I plan to import the read part or tinycdb into libc and the write part into > > libutil (to avoid cluttering libc). > > > > Why importing to libc? to allow all the get*ent to parse the /etc/*.db > > files > > > > just curious, where do we have the bdb 1.85 routines now ? > Are they all in libc, or split between libc and libutil ? regards, Bapt |
|
|
|
On Wed, May 02, 2012 at 02:57:07PM +0200, Dag-Erling Smørgrav wrote:
> Baptiste Daroussin <[hidden email]> writes: > > I plan to import the read part or tinycdb into libc and the write part into > > libutil (to avoid cluttering libc). > > I would prefer having everything in one place. > Ok I'll do that way. Bapt |
|
|
|
|
On Wed, May 02, 2012 at 04:02:35PM +0200, Luigi Rizzo wrote:
> On Wed, May 02, 2012 at 02:31:49PM +0200, Baptiste Daroussin wrote: > ... > > > > Why importing to libc? to allow all the get*ent to parse the /etc/*.db > > > > files > > > > > > > > > > just curious, where do we have the bdb 1.85 routines now ? > > > Are they all in libc, or split between libc and libutil ? > > > > All in libc > > ok then it makes sense to preserve the structure and have > everything in libc as des suggested. Just a curiosity again, > any estimate of code size ? > 545 cdb.c 76 cdb_find.c 81 cdb_findnext.c 19 cdb_hash.c 113 cdb_init.c 184 cdb_make.c 50 cdb_make_add.c 204 cdb_make_put.c 100 cdb_seek.c 29 cdb_seq.c 18 cdb_unpack.c 126 cdb.h 41 cdb_int.h 1586 total regards, Bapt |
|
|
|
|
On Wed, May 02, 2012 at 04:13:49PM +0200, Dag-Erling Smørgrav wrote:
> Baptiste Daroussin <[hidden email]> writes: > > That is pretty small: > > 545 cdb.c > > 76 cdb_find.c > > 81 cdb_findnext.c > > 19 cdb_hash.c > > 113 cdb_init.c > > 184 cdb_make.c > > 50 cdb_make_add.c > > 204 cdb_make_put.c > > 100 cdb_seek.c > > 29 cdb_seq.c > > 18 cdb_unpack.c > > 126 cdb.h > > 41 cdb_int.h > > 1586 total > > kB or LOC? |
|
In reply to this post by Baptiste Daroussin-2
On Wed, May 02, 2012 at 03:45:47PM +0200, Baptiste Daroussin wrote:
> On Wed, May 02, 2012 at 04:02:35PM +0200, Luigi Rizzo wrote: > > On Wed, May 02, 2012 at 02:31:49PM +0200, Baptiste Daroussin wrote: > > ... > > > > > Why importing to libc? to allow all the get*ent to parse the /etc/*.db > > > > > files > > > > > > > > > > > > > just curious, where do we have the bdb 1.85 routines now ? > > > > Are they all in libc, or split between libc and libutil ? > > > > > > All in libc > > > > ok then it makes sense to preserve the structure and have > > everything in libc as des suggested. Just a curiosity again, > > any estimate of code size ? > > > > That is pretty small: > 545 cdb.c > 76 cdb_find.c > 81 cdb_findnext.c > 19 cdb_hash.c > 113 cdb_init.c > 184 cdb_make.c > 50 cdb_make_add.c > 204 cdb_make_put.c > 100 cdb_seek.c > 29 cdb_seq.c > 18 cdb_unpack.c > 126 cdb.h > 41 cdb_int.h > 1586 total > regards, > Bapt What I do care a lot there, is the namespace pollution. I would strongly object against exposing cdb symbols from libc, even in the private namespace. Having the symbols hidden in libc is fine. The reasoning is that we do not want our libc unneccessary interpose symbols from third-party libs, and do not want to make a surprise for somebody who wants to use the never version of the same library, or use a symbol not documented in SUSv4 etc while linking to libc/libpthread only. |
|
|
On Wed, May 02, 2012 at 05:47:06PM +0300, Konstantin Belousov wrote:
> On Wed, May 02, 2012 at 03:45:47PM +0200, Baptiste Daroussin wrote: > > On Wed, May 02, 2012 at 04:02:35PM +0200, Luigi Rizzo wrote: > > > On Wed, May 02, 2012 at 02:31:49PM +0200, Baptiste Daroussin wrote: > > > ... > > > > > > Why importing to libc? to allow all the get*ent to parse the /etc/*.db > > > > > > files > > > > > > > > > > > > > > > > just curious, where do we have the bdb 1.85 routines now ? > > > > > Are they all in libc, or split between libc and libutil ? > > > > > > > > All in libc > > > > > > ok then it makes sense to preserve the structure and have > > > everything in libc as des suggested. Just a curiosity again, > > > any estimate of code size ? > > > > > > > That is pretty small: > > 545 cdb.c > > 76 cdb_find.c > > 81 cdb_findnext.c > > 19 cdb_hash.c > > 113 cdb_init.c > > 184 cdb_make.c > > 50 cdb_make_add.c > > 204 cdb_make_put.c > > 100 cdb_seek.c > > 29 cdb_seq.c > > 18 cdb_unpack.c > > 126 cdb.h > > 41 cdb_int.h > > 1586 total > > regards, > > Bapt > I do not think that the size matters at all, as far as it fits into tens > of KB of compiled code. > > What I do care a lot there, is the namespace pollution. I would strongly > object against exposing cdb symbols from libc, even in the private > namespace. Having the symbols hidden in libc is fine. > > The reasoning is that we do not want our libc unneccessary interpose > symbols from third-party libs, and do not want to make a surprise for > somebody who wants to use the never version of the same library, or use > a symbol not documented in SUSv4 etc while linking to libc/libpthread > only. regards, Bapt |
|
|
|
|
On Wed, May 02, 2012 at 06:13:45PM +0200, Dag-Erling Sm??rgrav wrote:
> Baptiste Daroussin <[hidden email]> writes: > > Konstantin Belousov <[hidden email]> writes: > > > The reasoning is that we do not want our libc unneccessary interpose > > > symbols from third-party libs, and do not want to make a surprise for > > > somebody who wants to use the never version of the same library, or use > > > a symbol not documented in SUSv4 etc while linking to libc/libpthread > > > only. > > I do understand that, I know need to learn how we can do that cleaning > > If we go that route, I would suggest having the complete library as > libcdb, and a copy of the read part hidden inside libc for getpw*() etc. > > To hide the cdb code inside libc, you need to add > > #define cdb_foo _cdb_foo > > to src/lib/libc/include/namespace.h and > > #undef cdb_foo > > to src/lib/libc/include/un-namespace.h, for appropriate values of "foo" > ("init", "find" etc.), then wrap the #include directives in the cdb > source with #include <namespace.h> /* ... */ #include <un-namespace.h>. > Within libc, you must call _cdb_foo() instead of cdb_foo(); outside of > libc, you call cdb_foo() as usual, and link with -lcdb. > > To avoid duplicating the cdb source code, you will need to create > src/lib/libcdb/{un-,}namespace.h as empty files and add -I${SRCDIR} to > CFLAGS in src/lib/libcdb/Makefile. symbols, to allow both interposing and libc usage of the right symbol internally. I do not think we need this at all for cdb, which symbols we should not export from libc. Just do not expose symbols, i.e. do not mention them in the version script for libc, is enough. The static linking is the lost case anyway. |
|
|
|
|
|
On Wed, May 02, 2012 at 05:51:46PM -0700, Adrian Chadd wrote:
> Hi, > > Which .db files are you proposing modifying in a cross-build > environment, rather than having a local tool generate them? > > (I do cross-builds all the time. I re-generate the .db files in /etc > in startup scripts on my devices..) > > All have a local tool to generate, but a local tool I can call from and amd64 pkgng trying to install arm eb packages in a arm eb chroot user for example, in preinstallation some users can be created that would be needed during the extraction phase for example, etc. regards, Bapt |
|
In reply to this post by Adrian Chadd-2
On Wed, May 02, 2012 at 05:51:46PM -0700, Adrian Chadd wrote:
> Hi, > > Which .db files are you proposing modifying in a cross-build > environment, rather than having a local tool generate them? > > (I do cross-builds all the time. I re-generate the .db files in /etc > in startup scripts on my devices..) While that's often a useful model, it seems to me that we'd prefer to be able to generate .db files for use on readonly root file systems. Sure we've go tricks to work around that, but if we can avoid them that's all to the good. -- Brooks |
|
|
|
On Mon, May 07, 2012 at 01:23:25PM -0700, Stanislav Sedov wrote:
> On Wed, 2 May 2012 13:41:15 +0200 > Baptiste Daroussin <[hidden email]> mentioned: > > > > > Any opinion, ideas, advices on this? > > Did you look at leveldb [1], btw? It seems to be very fast, better supported, and > support some advanced features that can be useful for other base applications > (like transactions, custom sorting, snapshots). > > It's distributed under BSD license. > > [1] http://code.google.com/p/leveldb/ > 1/ leveldb is c++ while for all our get*ent functions are in libc and needs to acees the db files. 2/ leveldb is really huge compare to tinycdb. Btw: an unexpected gain of cdb is that for services for example the size of the db is greatly reduces from 2M to 800K here, doesn't matter much but it is cool :) regards, Bapt |
Loading...
| Powered by Nabble | Edit this page |