Mees Distributed Data Archive Update 01-Nov-2001 0. INTRODUCTION --------------- The description of the Mees Distributed Data Archive circa 1996 is http://www.solar.ifa.hawaii.edu/Reference/dataplan_960416.txt The design philosophy of the distributed archive remains the same: The data from an instrument are kept by the individual most actively conducting science with the data. This maximizes their efficiency and in turn maximizes the scientific productivity of the Mees data. In return, that person is responsible for maintaining the archive in good order and for distributing the archival data to requestors. 1. INTERIM MODIFICATIONS ------------------------ Since 1996 several modifications to the data flow and archiving have occurred, mostly driven by hardware failures and modifications. A. HALEAKALA STOKES POLARIMETER The data from the Stokes is transfered via the network daily to Kukui, and is preserved in the backup tapes. This is (I believe) copied on to Montana State as well in near realtime. B. MEES WHITE LIGHT TELESCOPE The data from the MWLT is transfered via the network in realtime (with nightly verification and retransmission) to Kukui. It is written onto TAR tapes at intervals. This was driven by the failure of the tape drive and the ease of net transfer for the small data volume. C. IMAGING VECTOR MAGNETOGRAPH In 1998 the tapes drives on the IVM were upgraded to Mammoths. The copying of tapes for shipment to Lockheed was stopped at that time. With the regular observations in movie mode, the datarate is up at a few terabytes per year. D. MEES CCD IMAGING SPECTROGRAPH The MCCD archive site remains Montana. Beginning in August 1999 the tapes were transshipped via Manoa so that a copy onto Mammoth format could be made. That copying process has run irregularly; about 9 months have been finished but about 15 months have not. 2. IMMINENT CHANGES ------------------- A. MCCD upgrade The new MCCD datasystem will no longer use the old Exabyte tapes but DvDs. The datarate will jump up closer to that of the IVM. C. IVM Conversion to DvD The media cost for data storage is approximately: Mammoth: $1.50 / GB DvD: $5.00 / GB Network Attached RAID: $12.00 / GB This is what makes tapes look cheap. However, the manpower cost of waiting for the data to read off the tape is much higher than for the disk storage. At even a few $ / GB per usage, the media cost becomes misleading as an estimate of the total cost. For this reason, I plan to transition the IVM and MCCD to use DvD as the archival medium (the network to Mees will never be fast enough to pipe the data directly out to mass storage elsewhere). The DvD is easily read and this should solve many problems. D. Present and future Hawaii scientists Jing Li is beginning to work with Mees data to a greater extent. Future hire(s) will use Mees data once they are in place. 3. REVISED PLAN --------------- A. K-LINE and POI 1. Present database K: April 1988 to December 1998: Approx 2000 Exabytes in Manoa POI: May 1991 to April 1998: Approx 100 Exabytes in Manoa 2. Present media production and storage Both instruments were removed from operation in 1998 3. Changes to media storage Tape archive will move to JHUAPL 4. Future data access BJL will be the keeper of the JHUAPL archive He will respond to requests with a copy or original, best way B. MCCD 1. Present database Permanent archive still at Montana Approx 6 months overlap in 1996 at Manoa Approx 15 months of 1999 to 2001 data at Manoa for copying 2. Present media production and storage Exabytes written at Mees, mailed to Manoa 3. Changes to media production/distribution Redirect original tapes back to Montana effective 18-Jul-2001 Future transition to DvD TBD. 4. Changes to media storage BJL will take the original tapes at Manoa to JHUAPL, finish copying and forward to Montana 5. Future data access RCC will be the keeper of the Montana archive He will respond to requests with a copy or original, best way C. MWLT 1. Present database Nov-92 - Aug-93 integrated with MCCD image monitor data Aug 93 - Feb-96 Approx 115 tapes stored at LMSAL Nov 1995 - present Approx 30 tapes stored at Manoa 2. Present media production and storage Image files transmitted to Manoa via network in realtime Images written to tape at intervals 3. Changes to media production/distribution Images to be copied to JHUAPL for tape archiving 4. Changes to media storage Tape archive will move to JHUAPL 5. Future data access RCC for the 1992 - 1993 data TRM for the 1993 - 1996 data BJL for the 1995 - 2001 data Manoa for 2001 - data D. IVM 1. Present database May 1992 - present: Approx 1200 tapes in Manoa 2. Present media production and storage Tapes written at Mees and mailed to Manoa 3. Changes to media production/distribution Transition to DvD began 23-Oct-2001, finished 24-Apr-2002. 4. Changes to media storage BJL will take original tapes to JHUAPL 5. Future data access BJL will be the keeper of the tape archive. He will respond to requests with a copy or original, best way Manoa will be the keeper of the DvD archive. E. HSP and H-ALPHA CORONAGRAPH No changes to these are planned. 4. SUMMARY COMMENTS ------------------- Over the next year or two the situation in Hawaii will progress to a more active level. More of the data archive would then be captured locally, and probably put online. -------------------------------------------------------------------------