[IMC-Tech] Re: new pushtoeye.pl script installed

[Michael deBeer]

> > Micah wrote:
> > 
> > > i also hacked the C code for the copydir (aka mirrordir) program that
> > > is called by pushtoeye.pl to do all the heavy lifting. i think i have
> > > patched it so that it won't make an ftp connection unless there are
> > > actually files to consider mirroring. this should cut down the number
> > > of ftp connections to loudeye by (total rough guess) about 90%, and
> > > make troy happy. :)
> >  
> > Have you looked at rsync for mirroring? rsync.samba.org 

On Tue, 2 Jan 2001, Matthew Arnison wrote:
> unfortunately we have only got ftp access to loudeye.
> one of the first things we did was ask if we could
> use rsync, but unfortunately, only ftp is possible.

Ah, OK.

On loudeye -- is the concern that they don't want to give you a unix shell
account there?  If so, they may be willing to use the new 'rsync server'
mode of rsync.  They could setup an rsync server that only allows
transfers into a specific directory.  No shell account is used.  This can
use a variety of authorizations (by host, by .htaccess-style username,
etc..)  The rsync server can either run as a service out of inetd, or can
be run as a constantly run daemon.  rsync is very stable and I bet more
secure than their ftp setup.

> i have outlined the mirroring issues below:
> 
> http://active.org.au/doc/mirror.html
> 
> any comments very welcome.

If loudeye just isn't willing to use rsync, I might want to start to
explore testing another server.  Perhaps get a commercial account like
this one:
  http://www.altaway.com/unixsuper.html 
It allows 10 GB/month traffic and has realaudio.  You can do rsync over
ssh (push or pull) to it.  I've used altaway, and their tech support is
the best I've ever used.  You could also get a 'unix unlimited' account to
unlimited bandwidth for images (but no realaudio server).

Comments on the current scheme: As a way of dealing with thousands of
files per directory, perhaps do an MD5 hash of the filename, and use that
as the directory.  This would only have to be computed once, and the
directory name stored in the database.  This would allow practically
infinite number of files, instead of just 100x the current limit. There
are two perl modules to generate MD5 hashes.

Do you have a script similar to the one described in:
 'Improving the intelligence of the mirroring'
If not, I do have a ksh script that does a lot of the
sql/rcp/check/backout stuff that you describe.

good luck,
Michael de Beer