miriup.de

...because open source matters

  • Increase font size
  • Default font size
  • Decrease font size

gentoo-alt-converter

DATE_FORMAT_LC2 Dirk Tilger
E-Mail schreiben Print
There are no translations available.

What this script does and where it comes from 

Gentoo-prefix has been quite practical for me: I migrated from a Gentoo Linux machine to an Apple Intel Laptop.

Since prefix is still very develop-mental and a large number of packages are missing. However I found for many the packages from the main-tree with the automated prefix modifications (eprefixify) applied do just fine. Also the Gentoo prefix developers will not accept any diffs of ebuilds that have not been eprefixified. It's handy to have a eprefixified main tree around!

The following article is about a script I'm using to deal with the main tree and some simple actions I perform on maintree ebuilds. It does these things:

  • Using the portage main tree for some packages, synchronizing with it and automatically converting related packages
  • Addind my platform keyword into ebuilds conveniently
Lately EAPI="prefix" has been dropped from the ebuilds, so I feel we're close to not needing this script anymore.

Configuration of the script

The script has a view configuration variables in its beginning that need to be set.
  • NONPREFIX_PORTDIR_ORIGINAL sets where the copy of the main tree will reside
  • NONPREFIX_PORTDIR_CONVERTED sets where the converted ebuilds go
  • PREFIX_PORTDIR sets where the prefix tree sits
  • NONPREFIX_SYNC sets where we get the main tree from
  • MY_KEYWORD sets the keyword this architecture runs on
NONPREFIX_SYNC can be overridden by the SYNC variable in the environment of the script, if necessary.

Synchronizing with the main tree

Synchronizing with the main tree has the same syntax as its correspondent `emerge` command:

gentoo-alt-converter --sync

Automatically converting the ebuilds

For automatically converting the tree, there are two options: 

  1. Converting the tree one ebuild one by one
  2. Converting the tree on machines with more than one CPU. This method is fast, but it has a bug I never managed to find
What this script does for each ebuild directory:
  1. Check whether the unconverted ebuild checksums are correct.
  2. All ebuilds are copied into the 'converted' directory.
  3. All of the ebuilds in the converted ebuild directory are eprefixified.
  4. Using `ebuild manifest` the checksum of the ebuilds are updated. 
Since the ebuilds needs to be modified to run under gentoo-prefix, their checksums have to be recalculated. The first step prevents a malicious ebuild from slipping in. In the last step, I'm using `ebuild ... manifest`, because it's supposedly not updating the checksums of files outside the portage tree, such as packages that need download.
 
On successive runs, the converter checks whether the files in the source directory have changed since their last conversion and will reconvert only if ebuilds have been added or changed in the main tree. If an ebuild for the same directory exists in the prefix tree, the ebuild from the main tree will not be converted. If this would not be done, these converted ebuilds get in the way with broken patches and compilations. Also, if that directory exists from a previous run, it will be deleted.
 
For some reason though for some packages `ebuild manifest` will still attempt to download an ebuild, so I'm setting the variable FETCHCOMMAND to an empty value to prevent that from happening.

The command I'm typcially using to convert the ebuilds is:

FETCHCOMMAND= gentoo-alt-converter --convert

For parallel conversion, the command line is a little bit longer. The script has been split in a master part and a slave part for that purpose. The idea is to start one master part with:

gentoo-alt-converter -m --convert

The master part will output names of package directories that need conversion. The slave part takes these package directory as input and will conduct the conversion. Thus, the command for the normal one CPU conversion is thus:

gentoo-alt-converter -m --convert | bin/gentoo-alt-converter -s

To have use more CPU, you could simply add more slaves, as the following example for two CPU illustrates:

gentoo-alt-converter -m --convert | ( gentoo-alt-converter -s & gentoo-alt-converter -s )

However the above doesn't always work for me. So what I do instead is I give the pipe above a name and create a FIFO somewhere and then in the first terminal window I do:

mkfifo /tmp/pipe 
gentoo-alt-converter -m --convert > /tmp/pipe &
gentoo-alt-converter -s < /tmp/pipe

In the second Terminal I just repeat the last line. That also has the advantage that I can see the outputs of the two conversion processes without intermix.

For the more technical among here: the above assumes that no package name is longer than the operating system buffer, because reading one line from the pipe should be atomic. I have no clue whether this is the case or not. As mentioned earlier, the multiprocessor approach does not work as good as the single processor approach. It skips sometimes packages and the problem of atomicity might be the reason.

Attachments:
Download this file (check-ebuilds.py.txt)check-ebuilds.py[Helper script to check whether the ebuilds match their stored checksums]1 Kb04.05.2009 14:10
Download this file (gentoo-alt-converter.txt)gentoo-alt-converter[Converter to convert the main tree to be a gentoo-prefix compatible overlay]6 Kb04.05.2009 14:08
< Prev   Weiter >
LAST_UPDATED2