For many years, Sun Microsystems have run the Sun Software, Information and Technology Exchange (Sun SITE) program. The goals of the SunSITE program are:
You can find more details at Sun’s Sun SITE program web site.
Since the early 1990s, the Department of Computing at Imperial College London has hosted SunSITE Northern Europe, focussing on mirroring a large amount of freely available software for everyone. Over the years, Sun have provided us with several succeeding generations of Sun servers and associated storage. In November 2003, we put the finishing touches to the latest incarnation of Sun SITE Northern Europe and now pleased to announce that it’s ready for use!
In 2002, Sun Microsystems provided us with two SunFire E280R servers (each with dual 750Mhz Ultrasparc III processors, 8GB of RAM and 72GB of local disk), two smaller Sun Netras, and 4 T3 hardware RAID arrays with a total capacity of 2.2TB. In addition, Sun put us in touch with Veritas Software Corporation who have kindly provided us licenses (and considerable help!) for their latest top-of-the-range Veritas Cluster Server software, and Brocade who donated two Brocade Silkworm Fibre Channel switches to join the T3s to the servers forming a Storage Area Network. Together with a dedicated network link from Thus PLC/Demon Internet this setup forms the first clustered Sun SITE Northern Europe.
Having hit single server limits several times before, our design concept this time round was to build a scalable system out of standard components, which we could scale up further (more storage, more servers etc) when needed.
For those who are interested in a little more detail, both servers can access all four T3 RAID5 volumes via the SAN, and the Veritas Cluster software allows both servers to see a logical merged view of a single 2.2TB pool of storage, which can then easily be sliced up into resizable volumes. Each “slice” can then have a clustered Veritas filesystem layered on top. The upshot is that the servers can then both locally mount these filesystems and simultaneously read and update the data on the filesystems. In addition to this back end clustering, the Veritas Cluster software also allows each front-facing service (web, rsync, ftp and readonly NFS) to be placed flexibly on either server (using a Virtual IP address for each service). In addition, each service may be moved from one server to the other at a moment’s notice - this happens automatically if one server goes down, or can be forced to happen when necessary (for example, if we want to perform planned maintenance on one server).
Add two Unix system administrators, several ice-cream meetings, lots of top-quality open-source software (eg. rsync, mirror, fmirror, fget, apache, mod_perl and vsftpd), our own home-brewed perl mirroring and indexing system, and some simple mod_perl presentation scripts, stir vigorously, and voila… you end up with this service.
At present, we have a 2.1TB usable public mirror volume, about 90% full at present - we hope to be able to expand this further in future. Using a mixture of rsync and ftp mirroring techniques, we have mirrored a large amount of open source/free software and also some commercial ftp sites with downloads and technical support information. By specific request, we are also mirroring several Games-review sites. If you would like to suggest any sites to mirror, or report any problems with the current mirrored packages, please email to us on firstname.lastname@example.org.
Notationally, we split the material we mirror up into packages where a package is any collection of material from a single site that hangs together in a (mostly) coherent fashion. It might be (most obviously) a single software package (like xv, say) or a complete Unix/Linux distribution (like Debian Linux or NetBSD) or a collection (like CPAN or Adobe’s support ftp site). Note that this does not correspond very well to a Linux or Solaris package - a .rpm, .deb, or .pkg file.
We store these packages physically based on the URL of the site we mirrored them from - for example, CPAN happens to be stored within the public area as /sites/rsync.nic.funet.fi/CPAN/, and we make this physical view available via:
the web URL: http://www.sunsite.org.uk/sites/rsync.nic.funet.fi/CPAN/ the ftp URL: ftp://ftp.sunsite.org.uk/sites/rsync.nic.funet.fi/CPAN/ and the rsync “pseudo-URL”: rsync://rsync.sunsite.org.uk/sites/rsync.nic.funet.fi/CPAN/
In addition, we also store some meta-data about each package, and use this to index the packages and present a logical package view. This is frankly inspired by the Janet mirror service mirror.ac.uk, which is extremely nicely designed. This logical view is dynamically generated by a mod_perl module and can only be presented on the web via:
the URL: http://www.sunsite.org.uk/package/cpan/ (now no longer served)
Of course, this fancy package navigation is only available via the web interface; when accessing the data over ftp, either use the /sites/ method described above, or alternatively you’ll find a tree of symlinks under /package/ - so, for example, you can also get to CPAN via:
the URL: ftp://ftp.sunsite.org.uk/package/cpan/
Like the man says, there’s more than one way to do it!
Note that this is not exported via rsync at present, at rsync really dislikes symlinks.
In the near future, we will also provide searching facilities and a by-category index.
This service would not have been possible without the help of the following organisations:
Sun Microsystems Sun Microsystems - provided the hardware (servers, Fibre-channel cards, and storage), the Solaris operating system, and support and maintenance.
thus Thus PLC/Demon Internet - provided the network link.
Veritas Veritas Software Corporation - provided the Clustering software and software development in London protocols.
Brocade Brocade - provided the SAN fabric.
Imperial College, London The Department of Computing at Imperial College London, one of the top tier of scientific, engineering and medical Universities in the world, is very pleased to continue to host Sun SITE Northern Europe.
With the above organisations, several individuals are especially involving in setting up and running this service:
Duncan White is the lead developer setting up and running the new Sun SITE service. In real life (!) he is the Deputy Manager of the Department’s Computer Support Group. He has particularly enjoyed the opportunity of getting to grips with Veritas Clustering software and finally using mod_perl in anger.
Matt Johnson is the second developer working on the new Sun SITE service, and has done a tremendous amount of work on setting up the mirroring, building software, debugging systems etc.
Stuart McRobert is the Manager of the Department’s Computer Support Group, and has run several previous generations of Sun SITE Northern Europe. He was responsible for getting the new generation of Sun SITE Northern Europe organised, and getting Veritas and Brocade and Thus on board.
Greg Cooper is a support engineer working at Veritas who graciously donated an enormous amount of his time, energy and enthusiasm to the project - bootstrapping our knowledge of Veritas Clustering up from approximately zero to, umm, the giddy heights of wizardliness where we now reside. (Editor’s note - we think it was the ice cream that made Greg such a regular visitor:-))