SAFEGUARDING AUSTRALIA'S ARCHIVE OF SATELLITE REMOTELY SENSED DATA
Paul J. Wise
Centre for Remote Sensing
Australian Surveying and Land Information Group,
Department of Administrative Services
The changing face of Australia and its oceans, as captured by sensors onboard earth orbiting satellites and transmitted to the Australian Centre for Remote Sensing (ACRES), is archived on high density digital magnetic tape at the ACRES Data Processing Facility in Canberra. After 13 years the archive now consists of some 10 000 high density digital tapes that are not duplicated and pose a storage and stability problem. This paper describes the reasoning behind ACRES decision to transcribe the archive to optical tape, the optical tape recording system and its impact on maintaining and managing the ACRES archive to ensure that this increasingly important national resource is safeguarded.
SAFEGUARDING AUSTRALIA'S ARCHIVE OF SATELLITE REMOTELY SENSED DATA
How much data is in a terabyte (1012 bytes)? How much space do you need to hold an archive of 10 000 High Density Digital Tapes (HDDTs) which is increasing by an average of four HDDTs daily? The answers to these questions have been sought by staff of the Australian Centre for Remote Sensing (ACRES) over the last five years.
ACRES, a business unit of the Australian Surveying and Land Information Group (AUSLIG) of the Department of Administrative Services, is mandated to receive, process, archive and distribute earth resources data gathered by satellite.
Satellite data gathered, are recorded by satellite track allowing any region in a track to be extracted by its geographical coordinates. To assist cataloguing and ordering, a pass is segmented into nominal scenes. The length of each scene being the same as the swath width. An extracted scene can be reconstituted to form an image for display and used for manual or computer assisted analysis. Data for computer assisted analysis are supplied by ACRES on open reel, cartridge or floppy disk. Photographic images, not unlike aerial photographs are available for manual interpretation.
ACRES commenced operations in 1979 acquiring data via the American LANDSAT satellite. In 1990 data from the French SPOT and in 1991 data from the European ERS satellites were added to the daily acquisition schedule. Over thirteen years later the archive comprises some 10 000 HDDTs and is expanding by an average of four HDDTs each day.
To cope with the existing archive and its future growth ACRES has commenced a program of transcribing the data on HDDT to optical tape. This paper discusses this program that has the long term aim of managing and safeguarding the Australian archive of satellite remotely sensed data.
High Density Digital Magnetic Tape
High density digital magnetic tape was the original media adopted by ground stations for the acquisition of earth resources data transmitted by orbiting satellites. This media is able to be transported at the speeds necessary to record the data at the high rates of transmittal from the satellite and played back at lower speeds for ingest by the processing system. Because this media also stored the digits efficiently, once recorded, each HDDT became the archive media for the data. As such the HDDTs are held in an archival environment.
Diagram showing the ACRES HDDT acquisition/archiving process.
In 1980 only two HDDTs were used each day. Currently between four and six HDDTs are used each day to record data and when aggregated over the full year the archive grows at a rate of some 2 000 HDDTs per year.
The cost of the HDDTs currently precludes the backing up of the data so protection of the archive is paramount but even with the sophisticated environmental controls it was found that early HDDTs had some degradation of the polyester binder and were unable to be read. The problem manifested itself as a residue on the tape recorder heads and guides. The data was not lost, however, as ACRES adopted a method of baking the tapes in an industrial fan-forced oven. After cooling the tapes were readable again and so allowed their data to be transcribed.
Nevertheless the lack of back-up, the storage and stability problem needed to be addressed through the use of appropriate technology.
In 1988, a series of studies were started to look at alternatives to the use of high density digital tape for the long term storage and retrieval of ACRES satellite remotely sensed data.
The aspects considered included:
- cost of the medium;
- cost of the overhead to transfer all the existing archive onto the new medium (with the realisation that this would have to be done periodically);
- cost of the archive vault (real estate, environmental control, fire protection especially the cost of replacing the halon system);
- cost of maintaining the integrity of the archived data;
- the tape recorders;
- optical disc;
- optical tape.
The existing machines used to record the satellite data are of the type called Instrumentation Recorders. These recorders use inch wide magnetic tape on open reels, with the data recorded on longitudinal tracks along the length of the tape. They are able to record data at rates up to 105 Megabits per second).
Other recording techniques have evolved, especially through the television recording field, where helical scan methods are used to increase the data packing density. When combined with the improvement in magnetic tape formulations several machines met the ACRES criteria for replacement recorders.
Indeed, several ground stations have decided to retain their archive on magnetic tape by using helical scan recorders combined with "juke-box stackers" to give easy access to large volumes of data in a cost effective way.
Another medium on the market is optical disk. These disks have held promise for many years as a possible medium for the storage of large volumes of data. However, when the calculations were made many thousands of optical disks were still required to hold the ACRES archive. Even with large "juke-boxes" the task was just too large to consider practical.
This left the option of optical tape.
Optical Tape Recording
The need for an alternative storage media for large volumes of data was already being addressed by the Canadian Centre for Remote Sensing in collaboration with Canadian industry. The result of this collaboration was optical tape and associated recorder. This development had brought together successfully the use of high power, solid state laser technology and sophisticated error correcting methods to enable data to be fully recovered error free over areas of the optical medium having spot sizes 1 micron in diameter obscured by dust, scratches, etc.
CREO Optical Tape Recorder schematic.
The potential offered by optical tape recording was almost unbelievable and led to the first optical tape recorder, and only the fifth in the world, being installed at ACRES in July 1991.
CREO, a company based in Burnaby, B.C. Canada, set out in 1984 to develop an Optical Tape Recorder (OTR). The essential components of CREO's OTR that produced a practical machine are:-
CREO developed an air bearing linear scanner which transverses the optical tape. The scanner writes or reads data by scanning laser beams across the tape while the tape is held stationary.
A solid state laser with power output in the 10 milliWatt range is focussed to produce a 1 micron spot size. An array of such diode lasers lay down 32 data tracks simultaneously, with a spot spacing of 1.5 microns.
The Transport System
CREO has developed a precision tape transport system. The tape transport never allows the active side of the tape to touch any of the mechanical structure. The system is an open loop predictive-servo capable of moving the tape in small steps to an accuracy of 2 microns, and moving an 860 metre length of tape from one end to the other in less than 60 seconds.
Optical Tape Recorder linear scanning system.
Physical records are laid across the tape in swaths containing 32 data tracks and a control track. As the linear scanner crosses the tape, it writes an 80 Kilobyte record made up of 16 Kilobytes of error correction and 64 kilobytes of user data.
The CREO recorder uses a Reed-Solomon error correction code, heavily interleaved in order to avoid burst or cluster errors.
All data is encoded during the write operation and corrected during the read. It is very difficult to lose data as scratches, dust particles or even small holes in the tape will not result in any data loss. The system reconstructs the missing data with an accuracy of 1 in 1012. This is an essential measure for archival integrity in that every time any portion of a tape is read, the recorder reports the amount of data reconstruction needed. This report provides an excellent record of the error growth allowing the archive manager to decide to recopy the data in question long before data reconstruction fails.
An essential component to an optical tape recording system is the optical tape itself. To date, the only company manufacturing and delivering optical tape is ICI of UK. A second source is expected to come onto the market from Southwall Technologies of Palo Alto, California. Southwall and Dow joined to develop Dow's patented alloy system into the manufacture of optical tape in 1991. To date only test samples have been delivered by Southwall.
The optical tape used on the CREO OTR is up to 860 metres in length and 35 mm wide wound onto a precision open reel. This length of tape will hold one Terabyte of data.
Diagram showing the construction of the optical tape.
Optical tape is made up of layered coatings on a polyester substrate. Of these coatings only a single recording layer of dye polymer is responsible for the actual reflectance change representing a bit of data.
Cost and Performance Features of the CREO OTR
The cost and performance features of ICI 1012 Optical Tape on the CREO 1003 OTR are:
· the media cost is less than 1 cent per Megabyte and will fall with time the recording data rate is up to 3 Megabyte per second
· the volume of data on an optical tape is ten times that of advanced helical magnetic tape
· unlimited reading (greater than 40 000 rewind cycles)
· long media life (greater than 30 years in the presence of corrosive gases typical of the standard office environment)
· tape rewind periods are well in excess of magnetic media requirements, thus allowing reduced archive maintenance costs
· archival space savings (one 860m optical tape holds the data equivalent to 850 Multispectral Scanner HDDTs or 150 Thematic Mapper HDDTs).
With such credentials two more CREO OTRs were installed at ACRES during November 1992.
The ACRES Transcription Program
The transcription process is complex in that it must preserve the integrity of the data as it is copied onto the new media.
As with any new technology there were some abnormal "teething" problems which had to be worked through with the Canadian suppliers. In addition, ACRES had to develop the interfacing methodology and software. Part of the methodology was to reduce, as much as possible, the reliance on the HDDT media.
The optical tape and recorder could not be operated at the speeds necessary to directly record the data transmitted by the satellites so that the HDDT capability had to be retained for this process. However, the data on the HDDT could be transcribed onto optical tape and the optical tape recorder could be interfaced with the processing system. This meant that once the data was transcribed onto optical tape it could be read from the optical tape into the system for generating customer products. HDDTs would only be needed for the recording process and could be recycled after the data was transcribed. Further, there was a market for excess used HDDTs.
During the transcription process the optical tape is broken up into a number of partitions with each partition holding one pass or the equivalent of one HDDT. This makes finding a scene similar to the approach used now.
As part of the transcription process error detection has been built into the data transfer path that monitors the error detected in the data and brings it to the notice of the transcription operator.
Should the error be correctable (the usual source of error is contamination on the magnetic tape recorder read head), the operator may choose to re-run the transcription. Should the error be absolute, then this can be noted in the catalogue data base, or it can be chosen not to transcribe that magnetic tape and thus delete the entry from the refreshed data base.
The Catalogue Update
To make the ACRES digital data archive "visible" to its customers ACRES generates a data base describing each scene, the data quality and estimated cloud cover, as well as a microimage catalogue on microfiche. The micro-image catalogue contains each scene subsampled to give a pictorial view of coverage and cloud cover.
Theoretically every scene, held in the ACRES archive, is listed in the data base and pictorially held on microfiche.
Over the years, however, a small but unknown percentage of the archive has been lost, due to accident, magnetic tape deterioration, removal of unsatisfactory or unreadable data, etc. As the archive is refreshed onto optical tape, the ACRES catalogue has the opportunity to be updated to really reflect what is stored in the ACRES archive.
The transcription process therefore includes the extraction of original information from the data base and its replacement with the data being transcribed onto the optical tape. This update also includes information that tells the Operator which optical tape and where on that tape the data is held for processing a customer order.
Diagram showing the ACRES transcription procedure for archiving on optical tape.
When the first optical tape was generated a visual check of data transcription quality was undertaken by producing several photographic products. No evidence of data corruption was evident thus giving ACRES every confidence in this process.
To date, ACRES have filled three Optical Tapes with Multispectral Scanner data, and four with Thematic Mapper data. This has freed up about 2 500 High Density Digital Tapes that are being recycled thereby making a significant savings in magnetic tape purchases, and freeing valuable shelf space in the archive vault.
Pictorial indication of the HDDT archive being reduced by writing to Optical Tape.
The use of optical tape technology has solved a significant problem at ACRES that of safeguarding the Australian archive of satellite remotely sensed data.
When completed, in about two years time, the existing archive will be contained-on some thirty reels of optical tape and leave only the duplication of these tapes to achieve a dual archive and insure almost total data security for the first time.
The assistance of the staff, at the Australian Centre for Remote Sensing, in editing this paper, is acknowledged.