Jeudi, 27 janvier 2022
Managing tape drives and libraries with the Unix/Linux CLI
Summary
1. tools
2. identifying devices
3. handling tape cartridges in a library
4. handling tape in a drive
Drive information
Density code
Reading from the cartridge's RFID tag
mt commands
5. Identifying an unknown tape
6. Basic use of gnu tar
7. Using LTFS
Appendix: other useful commands
1. tools
Tools used in this article are: mt (from mt-st package), mtx, tar and ltfs. We’ll optionnally need lsscsi, dd and sg3-utils.
mt is used to control the tape drive (tape movements, ejection, and drive settings). mtx is used to control tape libraries (moving cartridges from slots to drives, etc). We’ll use mostly tar to read and write, and dd to identify unknown tapes
During the initial configuration, we’ll use lsscsi to identify tape devices.
2.identifying devices
Let’s identify our devices using lsscsi.
~# lsscsi [6:0:0:0] disk AMCC 9650SE-16M DISK 3.08 /dev/sda [7:0:0:0] tape HP Ultrium 5-SCSI Z21U /dev/st0 [7:0:0:1] mediumx TANDBERG StorageLoader 0346 /dev/sch0
Our tape drives appears as /dev/st0. Unfortunately, by default the device listed for the tape library is useless, we’ll have to resort to lsscsi- g to get the corresponding generic SCSI device:
~# lsscsi -g [6:0:0:0] disk AMCC 9650SE-16M DISK 3.08 /dev/sda /dev/sg0 [7:0:0:0] tape HP Ultrium 5-SCSI Z21U /dev/st0 /dev/sg1 [7:0:0:1] mediumx TANDBERG StorageLoader 0346 /dev/sch0 /dev/sg2
mt can use either /dev/stXX or /dev/nstXX devices. the difference is as follows: /dev/stXX rewind the tape before any operation. /dev/nstXX doesn’t move the tape. Therefore we’ll generally use /dev/nstXX. tar (or in some cases dd) also uses the tape device /dev/nstXX (or /dev/stXX). mtx on the other hand, works with the generic SCSI device /dev/sgXX. Know the difference.
3. handling tape cartridges in a library
First, let’s check what’s in our tape library:
~# mtx -f /dev/sg2 status
Storage Changer /dev/sg2:1 Drives, 8 Slots ( 0 Import/Export )
Data Transfer Element 0:Empty
Storage Element 1:Full :VolumeTag=AAA013L4
Storage Element 2:Empty
Storage Element 3:Empty
Storage Element 4:Empty
Storage Element 5:Empty
Storage Element 6:Empty
Storage Element 7:Empty
Storage Element 8:Empty
The “Data Transfer Element” lines represent tape drives (only one present in this example). “Storage Element” lines represent the cartridge slots. Depending upon your hardware, you may also have “Input Output Element” lines, which are “mail slots” that allow moving tapes in and out the library without interrupting normal operations.
If the cartridges carry barcode labels and your library supports these, mtx also display that label value, as shown in the example above.
We can see that our “Data Transfer Element 0” is empty; let’s load it with a tape:
~# mtx -f /dev/sg2 load 1 0 ~#
As a proper Unix program, mtx is quiet if nothing’s wrong. Let’s check the status again; we’ll see that mtx remembers from which slot comes the tape now in our drive:
~# mtx -f /dev/sg2 status
Storage Changer /dev/sg2:1 Drives, 8 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag=AAA013L4
Storage Element 1:Empty
Storage Element 2:Empty
Storage Element 3:Empty
Storage Element 4:Empty
Storage Element 5:Empty
Storage Element 6:Empty
Storage Element 7:Empty
Storage Element 8:Empty
Using the command mtx -f /dev/sg2 unload without source nor destination will unload the tape from the first drive into its original location. Of course you could alternatively unload the tape to any empty slot instead.
~# mtx -f /dev/sg2 unload 4 0 Unloading Data Transfer Element into Storage Element 4...done
Notice that mtx emit a message in that case. So it’s not that conforming to the Unix way after all :)
Nota Bene: with some libraries or drives (but generally not LTO drives), it may be necessary to first ask for the drive to eject the tape before asking for the library to move out the tape from the drive. So you may want to run mt offline with the appropriate options before any mtx unload command, just to be sure.
Last, mtx can also move tapes from slot to slot, which is particularly useful when using a IO slot (or mail slot):
~# mtx -f /dev/sg2 transfer 4 2
4. handling tape in a drive
mt can move the tape ( forward, rewinf, eject), erase, enable or disable compression, define the block size to use, etc.
First let’s check drive status:
~# mt -f /dev/nst0 status drive type = Generic SCSI-2 tape drive status = 1476395008 sense key error = 0 residue count = 0 file number = 0 block number = 0 Tape block size 0 bytes. Density code 0x58 (unknown). Soft error count since last status=0 General status bits on (41010000): BOT ONLINE IM_REP_EN
Drive informations
Drive status is read line by line :
| statut | signification |
|---|---|
| drive type | attachment type (normally always SCSI) |
| drive status | some code I don’t know much about… |
| sense key error | communication error status |
| residue count | ? |
| file number | record number (file) on which the tape is currently positioned |
| block number | block number on which the tape is currently positioned |
| Tape block size X bytes | current hardware block size, it’s generally advised to let it at zero (soft block size) |
| Density code | Tape type and density, see table below |
| Soft error count | number of errors in the current session |
Status Codes
The two last lines of output display “status bits”, listed below:
| Status Bit | Description |
|---|---|
| BOT | Beginning Of tape (Beginning of First File) |
| EOT | End Of Tape (either physical End of Tape, or End of Data area) |
| EOF | At the End Of some File |
| WR_PROT | Write Protected. Either the drive or tape is in write-protected mode; or the current drive only supports this tape type in read-only mode. |
| ONLINE | Drive loaded and ready to work. |
| DR_OPEN | Drive Open. The drive is empty, or the tape has been ejected and hangs at the door (in that case, you can use *mt load* to load the tape). |
| IM_REP_EN | Immediate Response Mode. the drive buffers data and acknowledge read and write operations before data has hit the tape. |
| SM | Tape is on a Marker (Set Mark). May work only on DDS drives(?) |
| EOD | End Of Data. The tape is at the end of recorded area. Depending upon drives, may be the same thing as EOT |
density codes
Here is a list of density codes compiled along the years from various sources (Internet, our own drives, etc):
| code | description |
|---|---|
| 0x00 | default |
| 0x01 | NRZI (800 bpi) |
| 0x02 | PE (1600 bpi) |
| 0x03 | GCR (6250 bpi) |
| 0x04 | QIC-11 |
| 0x05 | QIC-45/60 (GCR, 8000 bpi) |
| 0x06 | PE (3200 bpi) |
| 0x07 | IMFM (6400 bpi) |
| 0x08 | GCR (8000 bpi) |
| 0x09 | GCR (37871 bpi) |
| 0x0a | MFM (6667 bpi) |
| 0x0b | PE (1600 bpi) |
| 0x0c | GCR (12960 bpi) |
| 0x0d | GCR (25380 bpi) |
| 0x0f | QIC-120 (GCR 10000 bpi) |
| 0x10 | QIC-150/250 (GCR 10000 bpi) |
| 0x11 | QIC-320/525 (GCR 16000 bpi) |
| 0x12 | QIC-1350 (RLL 51667 bpi) |
| 0x13 | DDS (61000 bpi) |
| 0x14 | EXB-8200 (RLL 43245 bpi) |
| 0x15 | EXB-8500 or QIC-1000 |
| 0x16 | MFM 10000 bpi |
| 0x17 | MFM 42500 bpi |
| 0x18 | TZ86 |
| 0x19 | DLT 10GB |
| 0x1a | DLT 20GB |
| 0x1b | DLT 35GB |
| 0x1c | QIC-385M |
| 0x1d | QIC-410M |
| 0x1e | QIC-1000C |
| 0x1f | QIC-2100C |
| 0x20 | QIC-6GB |
| 0x21 | QIC-20GB |
| 0x22 | QIC-2GB |
| 0x23 | QIC-875 |
| 0x24 | DDS-2 |
| 0x25 | DDS-3 |
| 0x26 | DDS-4 or QIC-4GB |
| 0x27 | Exabyte Mammoth |
| 0x28 | Exabyte Mammoth-2 |
| 0x29 | QIC-3080MC |
| 0x30 | AIT-1 or MLR3 |
| 0x31 | AIT-2 |
| 0x32 | AIT-3 / SLR7 |
| 0x33 | SLR6 |
| 0x34 | SLR100 |
| 0x40 | DLT1 40 GB, or Ultrium |
| 0x41 | DLT 40GB, or Ultrium2 |
| 0x42 | LTO-2 |
| 0x44 | LTO-3 |
| 0x45 | QIC-3095-MC (TR-4) |
| 0x46 | LTO-4 |
| 0x47 | TR-5 / DDS-5 |
| 0x48 | Quantum SDLT220 |
| 0x49 | Quantum SDLT320 |
| 0x51 | IBM 3592 J1A |
| 0x52 | IBM 3592 E05 |
| 0x58 | LTO-5 |
| 0x5a | LTO-6 |
| 0x5c | LTO-7 |
| 0x5d | LTO-M8 |
| 0x5e | LTO-8 |
| 0x80 | DLT 15GB uncomp. or Ecrix / VXA-1 |
| 0x81 | DLT 15GB compressed / VXA-2 |
| 0x82 | DLT 20GB uncompressed / VXA-3 / VXA-320 |
| 0x83 | DLT 20GB compressed |
| 0x84 | DLT 35GB uncompressed |
| 0x85 | DLT 35GB compressed |
| 0x86 | DLT1 40 GB uncompressed |
| 0x87 | DLT1 40 GB compressed |
| 0x88 | DLT 40GB uncompressed |
| 0x89 | DLT 40GB compressed |
| 0x8c | EXB-8505 compressed |
| 0x90 | SDLT110 uncompr/EXB-8205 compr |
| 0x91 | SDLT110 compressed |
| 0x92 | SDLT160 uncompressed |
| 0x93 | SDLT160 compressed |
Reading from the cartridge’s RFID tag
LTO cartridges carry an RFID chip that stores all sorts of information. To read this chip using the tape drive, you’ll need the sg_logs command from sg3-utils package. The first line of output is drive information, and everything else is about the tape:
~# sg_logs -p 0x17 /dev/sg2
TANDBERG LTO-5 HH Z629
Volume statistics page (ssc-4), subpage=0
Page valid: 1
Thread count: 40
Total data sets written: 1429926
Total write retries: 2
Total unrecovered write errors: 0
Total suspended writes: 43
Total fatal suspended writes: 0
Total data sets read: 4186
Total read retries: 1
Total unrecovered read errors: 0
Last mount unrecovered write errors: 0
Last mount unrecovered read errors: 0
Last mount megabytes written: 0
Last mount megabytes read: 0
Lifetime megabytes written: 3534834
Lifetime megabytes read: 10347
Last load write compression ratio: 0
Last load read compression ratio: 8391
Medium mount time: 73309203
Medium ready time: 73309203
Total native capacity [MB]: 1529930
Total used native capacity [MB]: 7250
Volume serial number: MG1NVN224A
Tape lot identifier: G5ABP25R
Volume barcode: ST1008
Volume manufacturer: FUJIFILM
Volume license code: U107
Volume personality: Ultrium-5
Write protect: 0
WORM: 0
Maximum recommended tape path temperature exceeded: 0
Beginning of medium passes: 267
Middle of medium passes: 203
Logical position of first encrypted logical object:
partition number: 0, partition record data counter: 0xffffffffffff
Logical position of first unencrypted logical object after first
encrypted logical object:
partition number: 0, partition record data counter: 0xffffffffffff
Native capacity partition(s) [MB]:
partition number: 0, partition record data counter: 1529930
Used native capacity partition(s) [MB]:
partition number: 0, partition record data counter: 7250
Vendor specific parameter code (0xf000), payload in hex
00 00 01 ..
mt Command
Most common options of mt are listed below:
| commande | description |
|---|---|
| Information | |
| status | displays tape and drive information |
| Drive parameters | |
| setdensity X | Set Density Code. Used to enable or disable hardware compression. |
| setblk X | Define hardware block size. Normally always at 0 (“dynamic” or “soft blocks”) |
| drvbuffer X | enable (1) or disable (0)drive buffering. Other values may apply to some types of drives. |
| stoptions X | Set options for the *st* tape driver. Usual values are: 1 to enable write buffering, 2 to enable asynchronous writes, and 8 for debugging. |
| Movement | |
| rewind | rewind tape. |
| retension | rewind tape, then fast forward to the end of tape, then rewind again. |
| offline | rewind then eject tape. |
| fsf X | Forward X files. tape is positioned on the first block of the next file. |
| fsfm X | Forward X files. tape is positioned on the last block of the previous file. Same effect as fsf X + bsf 1 |
| bsf X | Backward X files. Tape is positioned on the last block of the previous file |
| bsfm X | Backward X files. Tape is positioned on the first block of the next file, same effect as bsf X + fsf 1 |
| asf X | Position to the Xth file on tape: rewind then forward X files ( like fsf ). |
| eod,seod | Forward to the end of data. Obsolete and equivalent : eom. |
| fsr X | Forward X records. |
| bsr X | Backward X records |
| fss X | Forward X markers (setmarks). |
| bss X | Backward X markers |
| seek X | Move tape to block X. |
| tell | Display the current block |
| Writing to tape | |
| eof, weof X | Write X “End of file” markers at current position. |
| wset X | Write X “Set mark” markers at current position. |
| erase | Erase the tape from the beginning. Caution: the drive will remain in use until the whole tape has been zeroed out, typically for several hours |
Each writing operaion on tape with a tool like tar, cpio, etc creates a “file”, identified with an “EOF” (End of file) marker. After writing 3 files on tape, you may move from one file to the other using mt fsf/bsf.
5. Identifying an unknown tape
To identify an unknown tape, we’ll first try to determine the block size used to write it. We’ll set up our drive to zero block size (dynamic block size):
mt -f /dev/nst0 setblk 0
Then we’ll read a full block using dd. We’ll suppose that the bloc probably won’t be bigger than 512KiB and write the output into a file /tmp/TESTTAPE :
~# dd if=/dev/nst0 bs=512k of=/tmp/TESTTAPE count=1 1+0 records in 1+0 records out 32768 bytes (33 kB, 32 KiB) copied, 3,76924 s, 8,7 kB/s
In our example we see that our block is 32KiB. Now what sort of file is this? Let’s use the file command :
~# file /tmp/TESTTAPE /tmp/TESTTAPE2: ASCII cpio archive (SVR4 with no CRC)
If file can’t name our data, we can try strings :
:~# file /tmp/TESTTAPE /tmp/TESTTAPE: data ~# strings /tmp/TESTTAPE -**#( NetVault Header )#**- rcserveur 22 sept. 04:29-1 rcpartages
6. Basic use of gnu tar
We’ll list the most common parameters for tar :
-c write to the target device
-t list content of target device
-x restore from target device
-f target device ( for example */dev/nst0* )
-v verbose mode (list current operation)
-b block size to use in units of 512 bytes. Common values: 64, 128; 256 (default value is a paltry 20)
Writing to tape:
tar -c -v -f /dev/nst0 -b 128 /un/dossier/à/archiver/
Write to /dev/nst0 in 64 KiB ( 128 * 512 bytes) blocks. Overwrite whatever was at this position and to the end of tape.
7. Using LTFS
LTFS (Long Term File System) is a multi-platform industry standard that provides wide compatibility of LTO-5 (and higher) cartridges between operating systems and software solutions. An LTFS-formatted LTO tape appears as a file system similar to a USB disk drive. Free utilities are available from tape drive providers (IBM, Quantum, HP, etc) for Windows, MacOS and Linux.
As LTFS for Debian is generally unavailable, here is a version I’ve built here (Debian 10 or 11). Dependancies aren’t defined, so you may need to manually install fuse tools.
The Linux LTFS tools are quite simple. The main one is the ltfs command that mounts (using fuse) the tape for reading and writing. Caution : just like mtx and other tools, ltfs uses the generic SCSI device /dev/sgXX instead of the tape device /dev/stXX.
Using ltfs is very easy. The program is very verbose, so I’ll omit some of its output in the example below:
~# ltfs -o devname=/dev/sg2 /mnt/ltfs
6127 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
6127 LTFS14058I LTFS Format Specification version 2.4.0.
6127 LTFS14104I Launched by "ltfs -o devname=/dev/sg2 /mnt/ltfs".
6127 LTFS14105I This binary is built for Linux (x86_64).
6127 LTFS14106I GCC version is 8.3.0.
6127 LTFS17087I Kernel version: Linux version 4.19.0-16-amd64
(debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6))
#1 SMP Debian 4.19.181-1 (2021-03-19) i386.
6127 LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 10 (buster)".
[...]
6127 LTFS14111I Initial setup completed successfully.
6127 LTFS14112I Invoke 'mount' command to check the result of final setup.
6127 LTFS14113I Specified mount point is listed if succeeded.
Following the advice from the last line of output, you should use mount to check that the tape is mounted:
~# mount -t ltfs ltfs:/dev/sg2 on /mnt/ltfs type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
You can then read and write to the tape like on any filesystem, but with very high latency. Tape positioning can take several minutes. Simply unmount the tape with umount when done (it can be pretty long, too).
Appendix: other useful commands
Sometimes your tape library may be locked, for instance because a software using it crashed, or it was unplugged during operations. You can unlock it with the following command (replace /dev/sg2 with the appropriate device):
sg_prevent -av /dev/sg2
posté à: 17:33 permalink