Simple recipe for custom Ubuntu/Debian repositories with apt-ftparchive

Debian-based Linux distributions, which use the .deb package format and the apt package management system, are relatively free from dependency hell and thus are generally a joy to use and maintain. However, in order to properly manage a local archive of packages you need to build a repository in which to keep them. This is the simplest recipe I know for putting together a HTTP repository.

In its simplest form, a repository is a directory full of files with a "Packages" file. This file describes the contents. When you do an apt-get update, Releases files (which can also be compressed with gzip or bzip2 and named appropriately) are downloaded and their contents processed by apt, then copied into your lists. If you want to have your own repo available to multiple computers, it needs to be network-accessible, and it needs to have a Packages file. You can make this file with the apt-ftparchive command.

If you install the apt-utils package (on Ubuntu Gutsy, at least) you will get this command, among others. I could go into detail on its various options if I knew much about them, but I don't; the following is the script I run to update my repository index files.

#!/bin/sh
#
# a script to updates the archive files
#
COMPRESS=bzip2 # compressor (gzip, bzip2, ?)
COMPRESSOPTS=-c # whichever opts cause stdin->compress->stdout (usually -c)
COMPRESSEXT=bz2 # whichever extension matches your compressor (gz,bz2,?)

( apt-ftparchive packages . > Packages && \
    "${COMPRESS}" "${COMPRESSOPTS}" < Packages > "Packages.${COMPRESSEXT}" \
    && \
    apt-ftparchive contents . > Contents && \
    "${COMPRESS}" "${COMPRESSOPTS}" < Contents > "Contents.${COMPRESSEXT}" ) \
    && \
    apt-ftparchive release . > Release && \ 
    "${COMPRESS}" "${COMPRESSOPTS}" < Release > "Release.${COMPRESSEXT}"

A bit unnecessary, but I was tired when I wrote it and I have a tendency towards unnecessary variables in that state, and it's convenient for you now if you don't want to use bzip2.

You can do away with the Contents and Release files; Package is the most important. Contents provides the information used by apt-file, which can tell you which archive a package is in even if you don't have the archive yet (let alone have it installed.)

You will need to make this directory web-accessible. For information on that, try reading in my guide Getting a handle on LTSP in Ubuntu Gutsy or for further exposition (and information on using the teensy tiny bozohttpd instead of apache2) you could see How to use ntop with mapper.pl on Ubuntu Feisty.

Now, add the following to every computer you want to use this repository, preferably as the first non-comment (and non-blank) line in the appropriate file (/etc/apt/sources.list):

deb http://address/directory /

Where address is the IP address or resolvable host name of the system hosting the files, and directory is the name of the directory or accessible symbolic link in that host's web directory (for apache2 on Ubuntu, the default is /var/www.) Mine says deb http://192.168.1.42/debs / but the actual directory is in a user's homedir; the directory has appropriate permissions and is symlinked to /var/www/debs. You will need to make sure you have Options FollowSymlinks in the apache configuration which covers /var/www/ in order for a symbolic link to work. You could just move the directory into /var/www.

Updating the directory from another system via rsync is easy, with a command like this one:

rsync -av -e ssh /var/cache/apt/archives/*.deb user@address:/home/user/debs

or of course

rsync -av -e ssh /var/cache/apt/archives/*.deb user@address:/var/www/debs

This is what I do to update my package cache at home after doing updates on the road (right now I have dialup access at home. Yuck.)

Troubleshooting

404 errors

I was getting 404 errors from my repository resulting in partial updates. It turned out that I had somehow ended up with filenames with undecoded URLs in them. I wanted a single-line fix for this and ended up using php simply because I remembered the name of the function to decode a URL... urldecode().

cd $HOME/debs
$ for i in *%* ; do mv $i "`echo $i | php -R 'echo urldecode($argn)."\n";'`"; done

I used the package php5-cli for this purpose. Read the manpage for php (down near the end) to see other, similar examples of using php to mangle text on the command line.

Comments

I had been having my apt-get -f being failed again and again unless i tried your script/technique and it all worked like a charm, thanks. Sincerely,Jeff Marson from iphone application development

Add new comment