How To Make Archive Tapes of Stokes Polarimeter Data

Mees to Manoa

Presently, the Haleakala Stokes Polarimeter data is FTP'ed from Mees Solar Observatory to Manoa in the early morning the day after the data was taken, at approximately 4 AM. Data from each day is stored in a separate directory where the name is given by the day number of the year. The magnetograms, calibration files, and preliminary reductions are stored in the directory. For a "normal" day of observing, with three to four magnetograms, the amount of data for a single day is 5-15 Mb of data. This of course strongly depends on the number of active regions present and the occurence of special campaign observations. Even after the data is transferred to Manoa, copies are kept on-line at Mees. These on-line files are retained until the data is sucessfully copied onto exabyte in Manoa. In addition, the Stokes data is also stored on system backup tapes of the computers at Mees. Therefore, there should not be any single point failure allowing data to be lost.

Archiving the Data in Manoa

Once the amount of data transferred to Mees reaches a critical mass, it can be tarred off to exabyte and stored. This should generally be done when the amount of data reaches 200-300 Mb, or when Tom needs disk space, whichever comes first. It generally takes about two months for this much data to accumulate.

The data is stored on milo on /solarm/FTP/pub/stokes in individual directories as detailed above. The data is archived using the tar command to write all the files/directories into a single compressed file on an exabyte tape. However, milo does not have its own exabyte drive, so the tar file must be transferred over the network to a machine that does have an exabyte. The basic command sequence for tarring the data and transferring it to the exabyte drive on akala is as follows (these commands are explained in the UNIX man pages for the tar command):

akala>102% allocate st0
akala>103% rlogin milo
milo>101% cd /solarm/FTP/pub/stokes
milo>102% tar -cvfb - 20 * | rsh akala dd of=/dev/nrst0 obs=20b
This will take an hour or so. After the tar is complete, always rewind the tape and check it again with tar to make sure it can be read properly using the following commands:
akala>104% mt -f /dev/nrst0 rew
akala>105% tar -tvf /dev/nrst0 > /solar/Stokes/tape_logs/stokes.tar.list.DATE
where DATE is the current date in the DDmmmYY format (e.g. 06apr95).
For the truly paranoid, this check can be done on another exabyte drive on a separate machine in order to guarantee it is generally readable. The output from this tar listing is saved into a new file. This file is kept on-line in /solar/Stokes/tape_logs (e.g. /solar/Stokes/tape_logs/stokes.tar.list.11oct95) in order to allow quick searches of the tape contents, see what data is available, etc.

In order to also assure that all the data in the directories was actually stored on the tape, one can compare the out from the "tar -t" command above with a general directory listing, produced with the following command:

milo>103% ls -lR > /solar/Stokes/tape_logs/

The numbers of files, their names, and their sizes can be compared between the two files to assure that no data was lost. Comparing the entire listing can be quite time consuming, so generally a spot check of several directories is all that is done. The above command will also store the output of the "ls -lR" command in the Stokes tape log directory (e.g. /solar/Stokes/tape_logs/ This is simply done for the sake of completeness and to allow one to check in the future the actual contents of the directory with what was archived.

When the integrity of the tar file on exabyte has been checked, a listing of all the archived directories is made and sent to Elaine Kiernan (kiernan@koa), using a command such as:

milo>104% ls -l * > ~/stokes.dir.list
This lets Elaine knows that the data has been sucessfully transferred to exabyte and can also be deleted from the machine there.

After, the data has been archived, it can be deleted from /solarm. I actually like to leave the data for the past week or so on /solarm so that there is some recent data there to look at, should the need arise. To delete the data from /solarm, the following command can be used:

milo>105% \rm -r NN*
where NN is the beginning two numbers of the directories you want to delete (e.g. 21*).

The exabyte tape is then labelled and put upstairs in the data archive. The command used to archive the data ("tar -xvf ... | dd ...") and what machine was used should be written on the card in the tape case. In addition, the days of the year included on that tape, and the range of dates that covers, should also be written on the tape.


There is a automatic email message from polsyn@koa every time data is transferred from Mees to Manoa sent to the current Stokes archiver and the observers. I have been keeping a file with all these messages (/solar/Stokes/tape_logs/ since they may be useful in the future if there is some discrepancy between the taken and archived data (i.e. where did the link break down?). But, as you can tell, I just like to be overly cautious.

If there is a problem in the transfer, or if /solarm fills up, then polsyn@koa will automatically send a message detailing any discrepencies between the directories for the data on koa and the directories on milo. If you clear space on /solarm, the missing files will automatically be re-transferred the following night.

In the past, a listing of the data that was archived off to tape was stored in a master file that showed the contents of each tape. This was useful and appropriate when the data was stored on nine-track tape (which only held two or three days) and the notification and backup procedures were more complex. However, the files in /solarm/Stokes/tape_logs (described above) take over most of this functionality. I mostly kept the list up to date, but I don't think it is necessary anymore. The file, which lists tapes back to May, 1991, is in /solar/daily/Mees/tape_logs/stokes.tapes.logs.

And that's all there is to it. One thing to watch out for is that new data may come in between the time you made the tar archive and when you go to delete the archived data. This could happen if you start the tar job in the evening, go home, and then delete the archived files the next day. In this case, be careful not to delete files that haven't been archived yet.

This page written by Kevin Reardon on 13 November, 1995
Current Stokes Data Guru is Alex Pevtsov
Last Updated - 13 November, 1995