How to create a local Ubuntu repository, update/upgrade distros ‘locally’ and thereby save bandwidth

We use only Linux at work and all our workstations are powered by Ubuntu and the network has more than 20 Ubuntu machines in it. Being admin’s, we are always keen to keep everything  updated and stable. Because of this reason, a lot of bandwidth will be eaten on every 6 months, yeah “the distro upgrade” process :)  To save the bandwidth and maintain a stable network traffic, we decided to create a local repository for Ubuntu and each and every steps to the target has been clearly explained in this post.

The whole process has two stages:

1) Setting up the local repository server- ‘reposerver’.

2) Setting up other machines/clients to use our server as the repository source.

STAGE 1:

1) Setting up the local repository server: reposerver.

We have to keep a machine dedicated for the local repository , lets call this machine as ‘reposerver The main requirement for reposerver is the ‘disk space’, 100GB  is recommended.

Here, I have created a separate partition for /var (and mounted it under /media) with 100 GB in size and assigned this space for storing the repo packages. You can also use an external storage device for this purpose. Once the disk space is ready, we need to install two packages on reposerver

apt-mirror , apache2

apt-mirror:

‘apt-mirror’ can easily create a mirror of repository from the Ubuntu server on our local machine(reposerver). It is a perl-based utility for downloading and mirroring the entire packages of a public Ubuntu repository. To install apt-mirror,

sudo apt-get install apt-mirror

Open the configuration file for apt-mirror

sudo gedit /etc/apt/mirror.list

You will find the below sample configuration file.

[plain]

############# config ##################
#
# set base_path /var/spool/apt-mirror
#
# if you change the base path you must create the directories below with write privileges
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
# set postmirror_script $var_path/postmirror.sh
set run_postmirror 0
set nthreads 20
set _tilde 0
#
############# end config ##############
deb http://archive.ubuntu.com/ubuntu karmic main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu karmic-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu karmic-updates main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu karmic-proposed main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu karmic-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu karmic main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu karmic-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu karmic-updates main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu karmic-proposed main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu karmic-backports main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu

[/plain]

All our machines are now installed with Ubuntu karmic Koala 9.10. We are now going to mirror the repository for the latest release Lucid Lynx so that we can use this to update/upgrade all machines. We need to replace ‘karmic’ with ‘lucid’ on mirror.list. You can use the ‘replace all’ option on gedit. The modified mirror.list file will looks like below.

[plain]

############# config ##################
#
set base_path /var/spool/apt-mirror
#
#set mirror_path $base_path/mirror
#set skel_path $base_path/skel
#set var_path $base_path/var
#set cleanscript $var_path/clean.sh
#set defaultarch <running host architecture>
#set postmirror_script $var_path/postmirror.sh
set run_postmirror 0
set nthreads 20
set _tilde 0
#
############# end config ##############
deb http://archive.ubuntu.com/ubuntu lucid main restricted
deb http://archive.ubuntu.com/ubuntu lucid-updates main restricted
deb http://archive.ubuntu.com/ubuntu lucid universe
deb http://archive.ubuntu.com/ubuntu lucid-updates universe
deb http://archive.ubuntu.com/ubuntu lucid multiverse
deb http://archive.ubuntu.com/ubuntu lucid-updates multiverse
deb http://archive.ubuntu.com/ubuntu lucid-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-backports main restricted universe multiverse
deb http://archive.canonical.com/ubuntu lucid partner
deb-src http://archive.canonical.com/ubuntu lucid partner
deb http://security.ubuntu.com/ubuntu lucid-security main restricted
deb http://security.ubuntu.com/ubuntu lucid-security universe
deb http://security.ubuntu.com/ubuntu lucid-security multiverse
clean http://archive.ubuntu.com/ubuntu

[/plain]

Once you are done with the configuration, start mirroring by issuing the below command

[bash]
apt-mirror
[/bash]

Mirroring will now begin:

[plain]Downloading 146 index files using 20 threads…
Begin time: Sat May 15 16:08:15 2010
[20]… [19]… [18]… [17]… [16]… [15]… [14]… [13]… [12]… [11]… [10]… [9]… [8]… [7]… [6]… [5]… [4]… [3]… [2]… [1]… [0]…
End time: Sat May 15 16:08:22 2010

Proceed indexes: [SSPPPPPPPPPPP]…..[/plain]

It will show the total size of packages that is going to be downloaded.

Check whether the process is running properly

adminsage@adminsage-desktop:~$ ps aux | grep wget
root      3267  0.0  0.0   5392  1704 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.1 -i /media/var/spool/apt-mirror/var/archive-urls.1
root      3271  0.0  0.0   5392  1724 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.5 -i /media/var/spool/apt-mirror/var/archive-urls.5
root      3272  0.0  0.0   5392  1720 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.6 -i /media/var/spool/apt-mirror/var/archive-urls.6
root      3273  0.0  0.0   5392  1724 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.7 -i /media/var/spool/apt-mirror/var/archive-urls.7
root      3277  0.0  0.0   5392  1700 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.11 -i /media/var/spool/apt-mirror/var/archive-urls.11
root      3278  0.0  0.0   5392  1704 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.12 -i /media/var/spool/apt-mirror/var/archive-urls.12
root      3279  0.0  0.0   5392  1724 pts/2    S+   15:19   0:00 wget --limit-rate=100m -t 0 -r -N -l inf -o /media/var/spool/apt-mirror/var/archive-log.13 -i /media/var/spool/apt-mirror/var/archive-urls.13

We can see several threads of wget running simultaneously. Once the download is complete, you will see a message like below.

[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...

End time: Sat May 16 18:01:03 2010
20MB in 50092 files and 384 directories can be freed.
Run /media/var/spool/apt-mirror/var/clean.sh for this purpose.

We are now done with the ‘mirroring’ of repository and all packages are now available on your machine :) Now we need to make this repo available to all other machines on the network via http. Install apache for this purpose.

[bash]sudo apt-get install apache2[/bash]

Create Symbolic links

sudo ln -s /media/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu/ /var/www/ubuntu
sudo ln -s /media/var/spool/apt-mirror/mirror/archive.canonical.com/ /var/www/canonical

Try accessing the repo locally from http://localhost/ubuntu and http://localhost/canonical

We have now successfully created the ‘reposerver’ and completed the stage1 of the whole process.

STAGE 2:

2) Setting up other machines/clients to use our server(reposerver) as the source repository .

NB': The following steps must be done on the machine that we wish to update/upgrade using the ‘reposerver’, our local repository server.

Backup your current  /etc/apt/sources.list

cp -av /etc/apt/sources.list /etc/apt/sources.list_backup

Modify sources.list to use our ‘reposerver’ as the repository. Consider the IP of ‘reposerver’ as 192.168.1.52

sudo gedit /etc/apt/sources.list

Replace the URLs http://archive.ubuntu.com/ubuntu/dists/lucid/Release with http://ip_of_reposerver/ubuntu/dists/lucid/Release and here it is 192.168.1.52

Use replace all option on gedit and replace like what is mentioned below.

Replace http://in.archive.ubuntu.com with http://192.168.1.52

http://archive.canonical.com with http://192.168.1.52

http://security.ubuntu.com with http://192.168.1.52.

The sample sources.list will be like below.

[plain]
# deb cdrom:[Ubuntu 9.10 _lucid Koala_ - Release i386 (20091028.5)]/ lucid main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://192.168.1.52/ubuntu lucid main restricted
deb-src http://192.168.1.52/ubuntu lucid main restricted
## Major bug fix updates produced after the final release of the
## distribution.
deb http://192.168.1.52/ubuntu lucid-updates main restricted
deb-src http://192.168.1.52/ubuntu lucid-updates main restricted
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://192.168.1.52/ubuntu lucid universe
deb-src http://192.168.1.52/ubuntu lucid universe
deb http://192.168.1.52/ubuntu lucid-updates universe
deb-src http://192.168.1.52/ubuntu lucid-updates universe
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://192.168.1.52/ubuntu lucid multiverse
deb-src http://192.168.1.52/ubuntu lucid multiverse
deb http://192.168.1.52/ubuntu lucid-updates multiverse
deb-src http://192.168.1.52/ubuntu lucid-updates multiverse
## Uncomment the following two lines to add software from the ‘backports’
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://192.168.1.52/ubuntu lucid-backports main restricted universe multiverse
deb-src http://192.168.1.52/ubuntu lucid-backports main restricted universe multiverse
## Uncomment the following two lines to add software from Canonical’s
## ‘partner’ repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
deb http://192.168.1.52/canonical/ubuntu lucid partner
deb-src http://192.168.1.52/canonical/ubuntu lucid partner
deb http://192.168.1.52/ubuntu lucid-security main restricted
deb-src http://192.168.1.52/ubuntu lucid-security main restricted
deb http://192.168.1.52/ubuntu lucid-security universe
deb-src http://192.168.1.52/ubuntu lucid-security universe
deb http://192.168.1.52/ubuntu lucid-security multiverse
deb http://192.168.1.52/ubuntu lucid main universe restricted multiverse
deb-src http://192.168.1.52/ubuntu lucid-security multiverse
[/plain]

Update the repo by issuing the command

sudo apt-get update

You will not be able to install and update all packages from our ‘reposerver’.

But this isn’t enough if you wish to do a complete dist upgrade.The dist upgrade reads certain files and we need to modify these files to have an error free upgrade.

Dist-Upgradation:

Go back to our server (reposerver) and download the following packages under /var/www/

wget -c http://changelogs.ubuntu.com/meta-release
wget -c http://changelogs.ubuntu.com/meta-release-lts

Create the directory:

sudo mkdir -p /media/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current

change directory to /media/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current

cd /media/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current

And download the below packages under on the current directory, ‘/media/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current’

wget -c http://archive.ubuntu.com/ubuntu/dists/lucid/Release
wget -c http://archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current/ReleaseAnnouncement
wget -c http://archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current/ReleaseAnnouncement
wget -c http://archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current/lucid.tar.gz
wget -c http://archive.ubuntu.com/ubuntu/dists/lucid/main/dist-upgrader-all/current/lucid.tar.gz.gpg

You are done with the server side modifications. Now go back to your client machine for which the distribution has to be upgraded. Modify the below files to point the URLs to our ‘reposerver’

[bash]
sudo gedit /etc/update-manager/meta-release
sudo gedit /usr/share/update-manager/mirrors.cfg
[/bash]

Sample files

[plain]
/usr/share/update-manager/mirrors.cfg

#ubuntu

http://192.168.1.52/ubuntu/

http://192.168.1.52/ubuntu/

ftp://192.168.1.52/ubuntu/
ftp://192.168.1.52/ubuntu/
mirror://launchpad.net/ubuntu/+countrymirrors-archive

http://ports.ubuntu.com/

ftp://ports.ubuntu.com/

http://ports.ubuntu.com/ubuntu-ports/

ftp://ports.ubuntu.com/ubuntu-ports/

http://old-releases.ubuntu.com/

ftp://old-releases.ubuntu.com/[/plain]

modify the URL at /etc/update-manager/meta-release

[plain]
# default location for the meta-release file

[METARELEASE]
URI = http://192.168.1.52/meta-release
URI_LTS = http://192.168.1.52/meta-release-lts
URI_UNSTABLE_POSTFIX = -development
URI_PROPOSED_POSTFIX = -proposed[/plain]

Update  the repository

[bash]sudo apt-get update[/bash]

To upgrade the distribution, issue the below command.

[bash]
sudo do-release-upgrade
[/bash]

The machine will now automatically detects the new release and will ask confirmation for an upgrade. Voila! Everything is now local and we have saved huge bandwidth and ‘TIME’ :)

NB: Run apt-mirror at regular intervals using cron so that the packages will always be update and hence synced with the Ubuntu server.

Post to Twitter Tweet This Post

One Response
Tweets that mention How to create a local Ubuntu repository, update, upgrade distros locally and there by saving bandwidth | SupportSages — Topsy.com

[...] This post was mentioned on Twitter by Corny Waysmith. Corny Waysmith said: How to create a local Ubuntu repository, update, upgrade distros … http://cli.gs/yBArR [...]

Comment Form

About this blog

This blog, acts as a knowledge repository for the world and is unofficial! Anything we find interesting in the cyber world will go here. Most cases, this blog will reflect the happiness of our staff in reaching successful solution to an issue (s)he worked on. A reference for other fellow SAGEs who come across similar issues later