Although its reputation suggests the contrary Freenet does have many mirroring uses.
Freenet has generated a lot of hype for being the “son of Napster” or some equally degrading report. For Freenet to survive, it needs to get away from being just another way to get illegal MP3s. In legal terms, Freenet needs “massive non-infringing use”. Sorry, but getting the latest copy of Quake doesn't count as non-infringing.
This article describes a use for Freenet that few will find a problem with (unless you use an RPM-based distro, to which I say “Get a real package manager!”). It also once again illustrates the power of small, flexible tools that do one thing and do it well.
As a Debian user who often uses my distribution of choice in production environments, I am confronted with problems in choosing mirror FTP sites from which to download new packages. Sometimes a mirror site is already overloaded with users, sometimes it isn't holding the packages I need and still other times it fails completely. I could resort to using the official ftp.debian.org, but that's not very polite. It would be nice if I didn't have to think about any of this.
Fortunately, the creators of Freenet have given me an answer and didn't even realize it at first. While Freenet has much to do with anonymity, it also provides a highly efficient mirroring system that no centralized system could hope to achieve.
The system described in this article requires no new code for either Freenet or Apt-get. It employs a standard user interface add-in for Freenet called FProxy, which originally was intended for the user to access Freenet through a web browser. Since it communicates with the browser via HTTP, and Apt-get already understands how to talk HTTP, this process only requires running a few simple commands and modifying the /etc/apt/sources.list.
Before proceeding, be sure to download the latest version of Freenet (0.3.9.1 as of this writing, although it will probably change by the time you read this) from www.freenetproject.org. Set up a node on your local machine and use some test files to insert and request. Open your favorite browser and point it to http://localhost:8081. If it brings up a page for requesting and inserting files, everything should be set up correctly. Be warned that it may take awhile for FProxy to come up, depending on the speed of your machine and which Java Virtual Machine you're running.
On a side note, you can get a Debian package that installs Freenet for you. It is available either at the web site above or in Debian unstable. Unfortunately, due to a buggy implementation of the big number classes, Freenet cannot work on the current version of Kaffe (1.0.6), and thus it can only run on a non-Free virtual machine; you'll find it in “contrib”. If you have a problem with this, you either can use a patched version of Kaffe, the latest CVS version of Kaffe or wait until the next stable version arrives. Current plans are to make future releases of Freenet compliable under GCJ, which brings Java code into native machine code, bypassing the problem of a non-Free VM completely.
In case you were wondering, Freenet makes rather heavy use of strong cryptography, so you'll find it in the non-US area, too. Edit your sources.list as needed.
FProxy was meant to be the default user interface for Freenet. Although it is not a full proxy server, it communicates to the browser through HTTP (this is what was shown earlier when you opened up your browser to http://localhost:8081). After realizing that Apt-get could already understand HTTP, it was seen that getting packages from Freenet would be quite simple.
Before going any further, make a backup of your current /etc/apt/sources.list. I didn't do this the first time, and it took a few annoying hours to get that system to download packages correctly again.
Now, add this line to the real sources.list:
deb http://localhost:8081/debian-test/dists stable
For more information, see the sources.list(5) man page. You'll need to modify http://localhost:8081 to reflect the port on which FProxy is running. This port is set in the configuration file .freenetrc, located in the top-level directory in which the Freenet tarball was decompressed. If you used the Debian package, it will be in /etc/freenet/freenetrc.
You also need to add application/x-debian-package to FProxy's passthroughMimeTypes. This will be in fproxyrc (prior to 0.3.8.1) or you can find it in freenetrc under services.fproxy.passthroughMimeTypes.
Next, go to a Debian mirror and download (from the directory /debian/dists/stable/main/binary-<your-arch>/, replacing <your-arch> as necessary) Packages, Packages.gz and Release. Insert all of these (see below). Now download the “hello” package from the devel directory on the Debian mirror and insert that too, being sure to place it under exactly the same name. Now you should be able to run apt-get update (which will download the Packages file off your node) and then apt-get install hello, which will download and install the GNU Hello program. If it worked, congratulations! You can now get packages off Freenet.
To insert these packages, first you will need to create an SVK key pair. This allows you to insert into a subspace that only someone with the private key can access. To do this, enter
freenet_insert -makeKeypair &> keypair.svk
Now, open up keypair.svk in your favorite text editor and delete everything except the value of the private key, and save the result into prv.svk. You now can insert with:
freenet_insert -serverAddress 127.0.0.1:<port> SSK@`cat prv.svk`/<dir>/<filename> <filename>replacing <port> with the port number your Freenet node is running on, <dir> with the “directory” (Freenet doesn't really have directories, but we can pretend it does) that you want to insert under and <filename> with the name of the file. You can retrieve the document by going back into keypair.svk, deleting everything except the value of the public key and saving it under pub.svk. Then type
freenet_insert -serverAddress 127.0.0.1:<port> SSK@`cat pub.svk`/<dir><filename> <filename>replacing things as described above. You can allow others to access these files by sending them the public key.
This method avoids a problem that arises with a simpler, guessable key type called KSK, which is actually a subspace where the private key is publicly known. The problem is, there is no way of knowing if a KSK actually has the data it's supposed to. Thus, it can be replaced by an attacker inserting a different document at a hops-to-live value of 1 and cutting the link before the transmission to the next node is done. Doing this enough times will replace the document completely for the majority of Freenet. This means that it would be easy to replace, say, the libc6 package with a virus. Think about how packages generally have to be installed from root, and you see what kind of damage this could do. Even with a subspace, it is strongly suggested that you check the MD5 sums of the packages you're downloading.
The Everything Over Freenet (EOF) Project keeps a list of currently maintained distributions that are under subspaces, which cannot be taken over without knowing the private key associated. Subspaces are slightly more complicated to use but infinitely safer. Please see eof.sourceforge.net/APT for instructions on this.
So you can get Debian packages off Freenet. Big deal. You could have done this with the FTP or HTTP servers you had been using all along. Not only that, but now you have to worry about packages dropping out of the store just because they aren't popular enough.
Well, it is a big deal. The huge list of mirrors that are a pain to keep maintained are now largely unnecessary, replaced by automatic updating within Freenet. This relieves a lot of bandwidth congestion.
Still, how do we keep packages from falling out? On a healthy Freenet, a file would have to be very unpopular to ever fall out. Needless to say, the current Freenet isn't so healthy (although having a few thousand über-stable Debian boxen might help). However, we need a way to ensure that little-known but oh-so-needed packages will remain available.
One suggestion is to create a wrapper script around Apt-get. This will first invoke apt-get install <package> through a Freenet request. If the package is unavailable, it will automatically download it from an FTP or HTTP mirror, e-mail the Freenet maintainer for your particular Debian distribution (provided you have mail set up properly on your machine) and warn them the package has dropped out of Freenet. Then it actually will install the package it downloaded.
But that won't work. You would have to know the private key of the subspace for it to be of any use to anybody except you. On the other hand, documents in a subspace are almost always going to be redirects to a CHK (a key where the name is a hash of the data, so the same data will result in the same CHK name). If the redirect still exists but the data under the CHK has fallen out, you could re-insert the CHK. However, the best we can hope for if the redirect document fell out is to e-mail the maintainer to ask them to re-insert.
Moving along to other thoughts, we come to the problem of the unstable distribution, which provides a unique challenge to Freenet. Current versions provide no real means to update content. This does not work well with the fast-moving Debian unstable. A hack of using date- and version-based redirects actually was put into Freenet in response to the problem of keeping Debian unstable the same in Freenet and FTP servers. Even so, this still takes a lot of bandwidth to accomplish.
The concept of date-based updating (putting the date in the name of the key and re-inserting every day) was brought up, specifically due to the question of how to update Debian packages. My personal feelings are that date-based updating is fine for relatively small web sites, but it's too much for the number of files being put into an entire Debian distro. Others may disagree.
In any case, those FTP or HTTP mirrors still will be needed to some extent. However, most of the mirrors are now unnecessary as most of the bandwidth can be distributed efficiently by Freenet.
Also note that many new means to make Freenet more efficient are being placed in the upcoming 0.4 version of Freenet. The 0.3 version was broken in some fundamental ways, making only the most popular files easily available. With any luck, we should see more files survive in version 0.4 of the network, which may very well be available by the time you read this.
One great way to help the Freenet and Debian projects is to help us mirror the Debian packages in Freenet. Scripts are available to aid in this; we just need people with a lot of bandwidth to help.
Basic requirements for helping the mirroring effort are that you have a Freenet node running and are willing to use a lot of bandwidth all at once. These are preferable so your ISP won't come knocking because you blew up a T3 trying to insert all this stuff. This isn't something to do casually; the current Debian 2.2r2 main archive for i386 weighs in around 2GB. The non-US version is quite a bit smaller, so if you don't live in the US and don't have much bandwidth, this might be a good option for you.
Complete information on helping mirror Debian packages can be found on eof.sourceforge.net/APT. There you will find the script for mirroring and a list of what distributions need to be mirrored.
Timm Murray is a student who spends his free time trying to come up with a useful contribution for Freenet.
