UDF File Layout

This document describes the data flow and layout for UDF files. It is a sort of "distilled wisdom" of what I've learned in the process of writing my UDF.pm package.

My goal is to describe how and where we get data, and what happens to it in the installation process. I deal solely with directories and files. If you want to learn how to write UDF code, you'll have to wait for me to write that document.

Data Directory Tree

UDF files are unpacked into the directory $UDF_DATA, which currently (21 Oct 1999) points to /packages/lib/udf/data/. It is possible, and indeed likely, that I will change this to /n/projects/mena/data/ in the near future.

The UDF directory tree for the IMAGE project looks like:

$UDF_DATA
 `-- IMAGE                 project
     `-- IMAGE1            mission
         |-- CIDP
         |   `-- CIDP
         |-- Database
         |-- EUV
         |   |-- EUVHSKP
         |   `-- EUVSCI
         |-- FUV
         |   |-- FUVHSKP
         |   `-- FUVSCI
         |-- HENA
         |   |-- HENAHSKP
         |   `-- HENASCI
         |-- LENA
         |   |-- LENAHSKP
         |   `-- LENASCI
         |-- MENA          experiment
         |   |-- MENAHSKP  instrument
         |   `-- MENASCI   instrument
         |-- OA
         |   `-- ORBIT
         |-- PIDFS
         |-- RPI
         |   |-- RPIHSKP
         |   `-- RPISCI
         |-- SAT
         |   `-- SAT
         `-- VIDFS

(highlights indicate directories we will talk about later).

The basic UDF hierarchy is: project, mission, experiment, instrument. For our porpoises, project will always be IMAGE, and mission will always be IMAGE1.

Adding Data

The above (empty) set of directories is what gets created when you click on "CREATE UDF DATA TREE" in the GUI. To use UDF, you must populate that with infrastructure files (PIDF and VIDF) and experiment data.

All data files are obtained from NASA Goddard, at:

• ftp://lzdata@ifeds5.gsfc.nasa.gov/mission/image/sci/data/day/

Files are of the form

 project.mission.UDFexperiment.timestamp.tgz

e.g.,

 IMAGE.IMAGE1.UDFv.199928213.tgz
 IMAGE.IMAGE1.UDFM.199919012.tgz

There are two special classes of files: PIDF and VIDF. These are indicated as such in the experiment part of the filename via the use of lower-case "p" or "v" respectively. The rest use the upper-case first letter of the experiment name ("M" for MENA, "R" for RPI, usw). PIDF and VIDF files must be installed before you can even think about installing or doing anything with UDF data.

Infrastructure: PIDF and VIDF

You can install PIDF and VIDF files from the GUI, or using my udfadm script. I'm not going to go into details here on how to do that.

Processing PIDF Files

PIDF is easy; the tar file simply contains a bunch of files that go into the $UDF_DATA/IMAGE/IMAGE1/PIDFS directory, and that's that.

Processing VIDF files

VIDF is much more complicated. Like PIDF, the tar file has a bunch of files that go into $UDF_DATA/IMAGE/IMAGE1/VIDFS. However, those files must then be converted into binary, and indexed into some form of database.

Here is an abbreviated look into the VIDFS directory:

$UDF_DATA/IMAGE/IMAGE1/VIDFS/
|-- AA
|-- CALS
|-- IMCBDDMP19980010000V
....
|-- IMMNHKA19980010000V
|-- IMMNHKB19980010000V
|-- IMMNHKC19980010000V
...
|-- IMRTTD19980010000V
`-- VIDFLIST.A

There are over 100 files in there. Apart from AA, CALS, and VIDFLIST.A, the rest are all of the form xxxxxtimestampV. These must be converted to a binary file with an "I" at the end (instead of the "V"), then moved to the appropriate instrument directory.

To do this, the installation script reads the file VIDFLIST.A (possibly called just VIDFLIST). This file contains a series of lines of the form

IMCBDDMP19980010000 CIDP CIDP
...
IMMNHKA19980010000 MENA MENAHSKP
IMMNHKB19980010000 MENA MENAHSKP
IMMNHKC19980010000 MENA MENAHSKP
...
IMRTTD19980010000 RPI RPISCI

For every line in the file, the script converts the indicated file from "V" to "I" (calling $UDF_HOME/bin/mk_idf to do the dirty work), then moves the "I" file to the directory indicated by experiment and instrument (e.g., MENA/MENAHSKP) on the same line. At the end of this process, the instrument directory will look like:

$UDF_DATA/IMAGE/IMAGE1/MENA/MENAHSKP/
|-- IMMLHSKP19980010000I
|-- IMMMEMD19980010000I
|-- IMMNHKA19980010000I
|-- IMMNHKB19980010000I
|-- IMMNHKC19980010000I
|-- IMMRHSKP19980010000I
`-- IMMSHSKP19980010000I

But we're still not done. Those "I" files must now be indexed into a database in the Database directory.

There is one IDF database for each instrument, even if there are multiple "I" files in the instrument directory. Note that when I say "one database", I mean "two database files": there is an index file (.NDX) and a data file (.DBF). For MENAHSKP, those will be:

$UDF_DATA/IMAGE/IMAGE1/Database/
|-- MENAHSKP.I.DBF
`-- MENAHSKP.I.NDX

Don't worry your pretty little head about how the databases get created. It's kind of unpleasant, but I plan to rewrite the entire code that does it. Maybe then I'll document it.

IMPORTANT NOTE: Since the filename of the database includes only the instrument name, and not the experiment, there must not be any two instruments with identical names on the same mission. That would be stupid, anyway.

Data

The data tar files unpack into a flat directory. There will be a bunch of files ending in "D" or "H", plus one file called UDFXFER.

The installation script reads the UDFXFER file, which contains a bunch of lines of the form:

  XFER
  IMMSHSKP19991901354D MENA/MENAHSKP
  IMMSHSKP19991901354H MENA/MENAHSKP
  ...
  IMMSTATS19991911703H MENA/MENASCI

The first line must be "XFER"; subsequent lines list a filename and its associated experiment/instrument directory. The script duly moves each file into its directory, then (surprise!) adds it to another database.

This database is called "HD" instead of "I". The resulting files look like:

$UDF_DATA/IMAGE/IMAGE1/Database/
|-- MENAHSKP.HD.DBF
`-- MENAHSKP.HD.NDX

Summary

In short, the fundamental layout of a UDF tree consists of:

• A directory tree with branches for each instrument

• A directory containing "PIDF" files.

• A directory containing "VIDF" files, plus:
  • one "I" file for each "V" file, spread across all the instruments (there may be multiple "I" files in each instrument directory).
  • a database of all the "I" files for each instrument.

• A bunch of "H" and "D" files in each instrument subdirectory, plus a database.

A snapshot of the IMAGE UDF tree after installing PIDF, VIDF, and one MENA data tar file, can be viewed here.

Operated by the University of California for the National Nuclear Security Administration, of the US Department of Energy.     Copyright © 2002 UC | Disclaimer/Privacy