Automating the FTP process

Running ftp by hand is tedious, boring, and unproductive. It's a waste of everyone's time.

I've written a Perl script intended to do everything without human intervention. It FTP's to the data repository; looks in the appropriate directories for new files; grabs any new data files and installs them in a local directory; and runs a user-specified command on each of those data files.

All this is available simply by typing:

 % mirror-ifeds5 -sleep_until 4:45
 Password for lzdata@150.144.211.77: xxxxx
 %

(The -sleep_until option tells the script to wake up every morning at 4:45. There is an upcoming "-d" option, to tell the script to run as a daemon and not hog your tty, but don't hold your breath).

With mirror running in the background, you can devote your time to more productive tasks.

Please note that the remote data account is password-protected, and I'm the kind of neurotic programmer who will not put a password in a plaintext file, such as a script. Thus, mirror cannot be run from cron, since it must ask for a password each time it is run. If your system uptimes are anything like mine, though -- measured in months -- this should not be a problem.

I cannot prevent you from coding the password into the script, since you will have your own copy of the source. However, I strongly discourage you from doing so. Please be careful. The net is no longer the nice place it was fifteen years ago.

Installing

1. If you don't have the Net::FTP Perl module, grab it from CPAN

2. Download the mirror module and calling script.

3. Put mirror-ifeds5 somewhere in your PATH. Fix its first line so it runs your copy of perl.

4. Edit the -localdir line in the script, to reflect your local setup. You also want to edit the "foreach" loop, e.g., change "mena" to your experiment of choice.

5. If you want, download the "mirror-frio" script (the UDF source repository). Although I haven't written a command-line replacement for the "Install UDF Software" clicky in UDFAdmin, it's still a lot easier to use mirror for your FTPs.

Yes, this requires some small UNIX ability. If you can't do it, ask your sysadmin.

Disclaimer

This is not the best code I've ever written. I would like to find time, some day, to rewrite the whole thing. Still, it gets the job done for now.

Operated by the University of California for the National Nuclear Security Administration, of the US Department of Energy.     Copyright © 2002 UC | Disclaimer/Privacy