storiq logo

La traînée de poudre

Le blog StorIQ


Jeudi, 27 janvier 2022

Managing tape drives and libraries with the Unix/Linux CLI


    1. tools
    2. identifying devices
    3. handling tape cartridges in a library
    4. handling tape in a drive
        Drive information
        Density code
        Reading from the cartridge's RFID tag
        mt commands
    5. Identifying an unknown tape
    6. Basic use of gnu tar
    7. Using LTFS
    Appendix: other useful commands


1. tools

Tools used in this article are: mt (from mt-st package), mtx, tar and ltfs. We’ll optionnally need lsscsi, dd and sg3-utils.

mt is used to control the tape drive (tape movements, ejection, and drive settings). mtx is used to control tape libraries (moving cartridges from slots to drives, etc). We’ll use mostly tar to read and write, and dd to identify unknown tapes

During the initial configuration, we’ll use lsscsi to identify tape devices.

2.identifying devices

Let’s identify our devices using lsscsi.

~# lsscsi
[6:0:0:0]    disk    AMCC     9650SE-16M DISK  3.08  /dev/sda
[7:0:0:0]    tape    HP       Ultrium 5-SCSI   Z21U  /dev/st0
[7:0:0:1]    mediumx TANDBERG StorageLoader    0346  /dev/sch0

Our tape drives appears as /dev/st0. Unfortunately, by default the device listed for the tape library is useless, we’ll have to resort to lsscsi- g to get the corresponding generic SCSI device:

~# lsscsi -g
[6:0:0:0]    disk    AMCC     9650SE-16M DISK  3.08  /dev/sda  /dev/sg0
[7:0:0:0]    tape    HP       Ultrium 5-SCSI   Z21U  /dev/st0  /dev/sg1
[7:0:0:1]    mediumx TANDBERG StorageLoader    0346  /dev/sch0  /dev/sg2

mt can use either /dev/stXX or /dev/nstXX devices. the difference is as follows: /dev/stXX rewind the tape before any operation. /dev/nstXX doesn’t move the tape. Therefore we’ll generally use /dev/nstXX. tar (or in some cases dd) also uses the tape device /dev/nstXX (or /dev/stXX). mtx on the other hand, works with the generic SCSI device /dev/sgXX. Know the difference.

3. handling tape cartridges in a library

First, let’s check what’s in our tape library:

~# mtx -f /dev/sg2 status
  Storage Changer /dev/sg2:1 Drives, 8 Slots ( 0 Import/Export )
Data Transfer Element 0:Empty
      Storage Element 1:Full :VolumeTag=AAA013L4                        
      Storage Element 2:Empty
      Storage Element 3:Empty
      Storage Element 4:Empty
      Storage Element 5:Empty
      Storage Element 6:Empty
      Storage Element 7:Empty
      Storage Element 8:Empty

The “Data Transfer Element” lines represent tape drives (only one present in this example). “Storage Element” lines represent the cartridge slots. Depending upon your hardware, you may also have “Input Output Element” lines, which are “mail slots” that allow moving tapes in and out the library without interrupting normal operations.

If the cartridges carry barcode labels and your library supports these, mtx also display that label value, as shown in the example above.

We can see that our “Data Transfer Element 0” is empty; let’s load it with a tape:

~# mtx -f /dev/sg2 load 1 0

As a proper Unix program, mtx is quiet if nothing’s wrong. Let’s check the status again; we’ll see that mtx remembers from which slot comes the tape now in our drive:

~# mtx -f /dev/sg2 status
  Storage Changer /dev/sg2:1 Drives, 8 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag=AAA013L4
      Storage Element 1:Empty
      Storage Element 2:Empty
      Storage Element 3:Empty
      Storage Element 4:Empty
      Storage Element 5:Empty
      Storage Element 6:Empty
      Storage Element 7:Empty
      Storage Element 8:Empty

Using the command mtx -f /dev/sg2 unload without source nor destination will unload the tape from the first drive into its original location. Of course you could alternatively unload the tape to any empty slot instead.

~# mtx -f /dev/sg2 unload 4 0 
Unloading Data Transfer Element into Storage Element 4...done

Notice that mtx emit a message in that case. So it’s not that conforming to the Unix way after all :)

Nota Bene: with some libraries or drives (but generally not LTO drives), it may be necessary to first ask for the drive to eject the tape before asking for the library to move out the tape from the drive. So you may want to run mt offline with the appropriate options before any mtx unload command, just to be sure.

Last, mtx can also move tapes from slot to slot, which is particularly useful when using a IO slot (or mail slot):

~# mtx -f /dev/sg2 transfer 4 2

4. handling tape in a drive

mt can move the tape ( forward, rewinf, eject), erase, enable or disable compression, define the block size to use, etc.

First let’s check drive status:

 ~# mt -f /dev/nst0 status
drive type = Generic SCSI-2 tape
drive status = 1476395008
sense key error = 0
residue count = 0
file number = 0
block number = 0
Tape block size 0 bytes. Density code 0x58 (unknown).
Soft error count since last status=0
General status bits on (41010000):

Drive informations

Drive status is read line by line :

statut signification
drive type attachment type (normally always SCSI)
drive status some code I don’t know much about…
sense key error communication error status
residue count ?
file number record number (file) on which the tape is currently positioned
block number block number on which the tape is currently positioned
Tape block size X bytes current hardware block size, it’s generally advised to let it at zero (soft block size)
Density code Tape type and density, see table below
Soft error count number of errors in the current session

Status Codes

The two last lines of output display “status bits”, listed below:

Status Bit Description
BOT Beginning Of tape (Beginning of First File)
EOT End Of Tape (either physical End of Tape, or End of Data area)
EOF At the End Of some File
WR_PROT Write Protected. Either the drive or tape is in write-protected mode; or the current drive only supports this tape type in read-only mode.
ONLINE Drive loaded and ready to work.
DR_OPEN Drive Open. The drive is empty, or the tape has been ejected and hangs at the door (in that case, you can use *mt load* to load the tape).
IM_REP_EN Immediate Response Mode. the drive buffers data and acknowledge read and write operations before data has hit the tape.
SM Tape is on a Marker (Set Mark). May work only on DDS drives(?)
EOD End Of Data. The tape is at the end of recorded area. Depending upon drives, may be the same thing as EOT

density codes

Here is a list of density codes compiled along the years from various sources (Internet, our own drives, etc):

code description
0x00 default
0x01 NRZI (800 bpi)
0x02 PE (1600 bpi)
0x03 GCR (6250 bpi)
0x04 QIC-11
0x05 QIC-45/60 (GCR, 8000 bpi)
0x06 PE (3200 bpi)
0x07 IMFM (6400 bpi)
0x08 GCR (8000 bpi)
0x09 GCR (37871 bpi)
0x0a MFM (6667 bpi)
0x0b PE (1600 bpi)
0x0c GCR (12960 bpi)
0x0d GCR (25380 bpi)
0x0f QIC-120 (GCR 10000 bpi)
0x10 QIC-150/250 (GCR 10000 bpi)
0x11 QIC-320/525 (GCR 16000 bpi)
0x12 QIC-1350 (RLL 51667 bpi)
0x13 DDS (61000 bpi)
0x14 EXB-8200 (RLL 43245 bpi)
0x15 EXB-8500 or QIC-1000
0x16 MFM 10000 bpi
0x17 MFM 42500 bpi
0x18 TZ86
0x19 DLT 10GB
0x1a DLT 20GB
0x1b DLT 35GB
0x1c QIC-385M
0x1d QIC-410M
0x1e QIC-1000C
0x1f QIC-2100C
0x20 QIC-6GB
0x21 QIC-20GB
0x22 QIC-2GB
0x23 QIC-875
0x24 DDS-2
0x25 DDS-3
0x26 DDS-4 or QIC-4GB
0x27 Exabyte Mammoth
0x28 Exabyte Mammoth-2
0x29 QIC-3080MC
0x30 AIT-1 or MLR3
0x31 AIT-2
0x32 AIT-3 / SLR7
0x33 SLR6
0x34 SLR100
0x40 DLT1 40 GB, or Ultrium
0x41 DLT 40GB, or Ultrium2
0x42 LTO-2
0x44 LTO-3
0x45 QIC-3095-MC (TR-4)
0x46 LTO-4
0x47 TR-5 / DDS-5
0x48 Quantum SDLT220
0x49 Quantum SDLT320
0x51 IBM 3592 J1A
0x52 IBM 3592 E05
0x58 LTO-5
0x5a LTO-6
0x5c LTO-7
0x5d LTO-M8
0x5e LTO-8
0x80 DLT 15GB uncomp. or Ecrix / VXA-1
0x81 DLT 15GB compressed / VXA-2
0x82 DLT 20GB uncompressed / VXA-3 / VXA-320
0x83 DLT 20GB compressed
0x84 DLT 35GB uncompressed
0x85 DLT 35GB compressed
0x86 DLT1 40 GB uncompressed
0x87 DLT1 40 GB compressed
0x88 DLT 40GB uncompressed
0x89 DLT 40GB compressed
0x8c EXB-8505 compressed
0x90 SDLT110 uncompr/EXB-8205 compr
0x91 SDLT110 compressed
0x92 SDLT160 uncompressed
0x93 SDLT160 compressed

Reading from the cartridge’s RFID tag

LTO cartridges carry an RFID chip that stores all sorts of information. To read this chip using the tape drive, you’ll need the sg_logs command from sg3-utils package. The first line of output is drive information, and everything else is about the tape:

 ~# sg_logs -p 0x17 /dev/sg2 
    TANDBERG  LTO-5 HH          Z629
Volume statistics page (ssc-4), subpage=0
  Page valid: 1
  Thread count: 40
  Total data sets written: 1429926
  Total write retries: 2
  Total unrecovered write errors: 0
  Total suspended writes: 43
  Total fatal suspended writes: 0
  Total data sets read: 4186
  Total read retries: 1
  Total unrecovered read errors: 0
  Last mount unrecovered write errors: 0
  Last mount unrecovered read errors: 0
  Last mount megabytes written: 0
  Last mount megabytes read: 0
  Lifetime megabytes written: 3534834
  Lifetime megabytes read: 10347
  Last load write compression ratio: 0
  Last load read compression ratio: 8391
  Medium mount time: 73309203
  Medium ready time: 73309203
  Total native capacity [MB]: 1529930
  Total used native capacity [MB]: 7250
  Volume serial number: MG1NVN224A
  Tape lot identifier: G5ABP25R
  Volume barcode: ST1008                          
  Volume manufacturer: FUJIFILM
  Volume license code: U107
  Volume personality: Ultrium-5
  Write protect: 0
  WORM: 0
  Maximum recommended tape path temperature exceeded: 0
  Beginning of medium passes: 267
  Middle of medium passes: 203
  Logical position of first encrypted logical object:
    partition number: 0, partition record data counter: 0xffffffffffff
  Logical position of first unencrypted logical object after first
  encrypted logical object:
    partition number: 0, partition record data counter: 0xffffffffffff
  Native capacity partition(s) [MB]:
    partition number: 0, partition record data counter: 1529930
  Used native capacity partition(s) [MB]:
    partition number: 0, partition record data counter: 7250
  Vendor specific parameter code (0xf000), payload in hex
 00     00 01                                               ..

mt Command

Most common options of mt are listed below:

commande description
status displays tape and drive information
Drive parameters
setdensity X Set Density Code. Used to enable or disable hardware compression.
setblk X Define hardware block size. Normally always at 0 (“dynamic” or “soft blocks”)
drvbuffer X enable (1) or disable (0)drive buffering. Other values may apply to some types of drives.
stoptions X Set options for the *st* tape driver. Usual values are: 1 to enable write buffering, 2 to enable asynchronous writes, and 8 for debugging.
rewind rewind tape.
retension rewind tape, then fast forward to the end of tape, then rewind again.
offline rewind then eject tape.
fsf X Forward X files. tape is positioned on the first block of the next file.
fsfm X Forward X files. tape is positioned on the last block of the previous file. Same effect as fsf X + bsf 1
bsf X Backward X files. Tape is positioned on the last block of the previous file
bsfm X Backward X files. Tape is positioned on the first block of the next file, same effect as bsf X + fsf 1
asf X Position to the Xth file on tape: rewind then forward X files ( like fsf ).
eod,seod Forward to the end of data. Obsolete and equivalent : eom.
fsr X Forward X records.
bsr X Backward X records
fss X Forward X markers (setmarks).
bss X Backward X markers
seek X Move tape to block X.
tell Display the current block
Writing to tape
eof, weof X Write X “End of file” markers at current position.
wset X Write X “Set mark” markers at current position.
erase Erase the tape from the beginning. Caution: the drive will remain in use until the whole tape has been zeroed out, typically for several hours

Each writing operaion on tape with a tool like tar, cpio, etc creates a “file”, identified with an “EOF” (End of file) marker. After writing 3 files on tape, you may move from one file to the other using mt fsf/bsf.

5. Identifying an unknown tape

To identify an unknown tape, we’ll first try to determine the block size used to write it. We’ll set up our drive to zero block size (dynamic block size):

mt -f /dev/nst0 setblk 0

Then we’ll read a full block using dd. We’ll suppose that the bloc probably won’t be bigger than 512KiB and write the output into a file /tmp/TESTTAPE :

~# dd if=/dev/nst0 bs=512k of=/tmp/TESTTAPE count=1
1+0 records in
1+0 records out
32768 bytes (33 kB, 32 KiB) copied, 3,76924 s, 8,7 kB/s

In our example we see that our block is 32KiB. Now what sort of file is this? Let’s use the file command :

 ~# file /tmp/TESTTAPE 
/tmp/TESTTAPE2: ASCII cpio archive (SVR4 with no CRC)

If file can’t name our data, we can try strings :

:~# file /tmp/TESTTAPE
/tmp/TESTTAPE: data
~# strings /tmp/TESTTAPE 
-**#( NetVault Header )#**-
rcserveur 22 sept. 04:29-1

6. Basic use of gnu tar

We’ll list the most common parameters for tar :

-c  write to the target device
-t  list content of target device
-x  restore from target device
-f  target device ( for example */dev/nst0* )
-v  verbose mode (list current operation)
-b  block size to use in units of 512 bytes. Common values: 64, 128; 256 (default value is a paltry 20)

Writing to tape:

tar -c -v -f /dev/nst0 -b 128 /un/dossier/à/archiver/

Write to /dev/nst0 in 64 KiB ( 128 * 512 bytes) blocks. Overwrite whatever was at this position and to the end of tape.

7. Using LTFS

LTFS (Long Term File System) is a multi-platform industry standard that provides wide compatibility of LTO-5 (and higher) cartridges between operating systems and software solutions. An LTFS-formatted LTO tape appears as a file system similar to a USB disk drive. Free utilities are available from tape drive providers (IBM, Quantum, HP, etc) for Windows, MacOS and Linux.

As LTFS for Debian is generally unavailable, here is a version I’ve built here (Debian 10 or 11). Dependancies aren’t defined, so you may need to manually install fuse tools.

The Linux LTFS tools are quite simple. The main one is the ltfs command that mounts (using fuse) the tape for reading and writing. Caution : just like mtx and other tools, ltfs uses the generic SCSI device /dev/sgXX instead of the tape device /dev/stXX.

Using ltfs is very easy. The program is very verbose, so I’ll omit some of its output in the example below:

 ~# ltfs -o devname=/dev/sg2 /mnt/ltfs

6127 LTFS14000I LTFS starting, LTFS version (Prelim), log level 2.                  
6127 LTFS14058I LTFS Format Specification version 2.4.0.                        
6127 LTFS14104I Launched by "ltfs -o devname=/dev/sg2 /mnt/ltfs".                     
6127 LTFS14105I This binary is built for Linux (x86_64).
6127 LTFS14106I GCC version is 8.3.0.
6127 LTFS17087I Kernel version: Linux version 4.19.0-16-amd64 
    ( (gcc version 8.3.0 (Debian 8.3.0-6)) 
    #1 SMP Debian 4.19.181-1 (2021-03-19) i386.
6127 LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 10 (buster)".


6127 LTFS14111I Initial setup completed successfully.
6127 LTFS14112I Invoke 'mount' command to check the result of final setup.
6127 LTFS14113I Specified mount point is listed if succeeded.

Following the advice from the last line of output, you should use mount to check that the tape is mounted:

 ~# mount -t ltfs
 ltfs:/dev/sg2 on /mnt/ltfs type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

You can then read and write to the tape like on any filesystem, but with very high latency. Tape positioning can take several minutes. Simply unmount the tape with umount when done (it can be pretty long, too).

Appendix: other useful commands

Sometimes your tape library may be locked, for instance because a software using it crashed, or it was unplugged during operations. You can unlock it with the following command (replace /dev/sg2 with the appropriate device):

sg_prevent -av /dev/sg2

posté à: 17:33 permalink