LTFS tapes won't mount again

[smartinventure]

Describe the bug
We have BDT tape library with an IBM LTO-9 drive and LTO-9 tapes.
I had formatted tapes 1, 2, and 3 with LTFS.
I have written so many files to tapes 1 + 2 that they were full. I dismounted tape 1 correctly and moved the media into another slot.
In tape 2, I admittedly made the error that I shutdown the server while the tape was idle, but LTFS not unmounted.
Both cannot be mounted again.
I then formatted a new tape (3) with LTFS, wrote 2 files onto it, unmounted, unloaded, loaded again and successfully mounted.
Below I added the result of the mount command (same for tapes 1 + 2, whereas 3 is successful) and some log.

At least, with basic commands like:
sudo mt -f /dev/st0 fsf 1
sudo dd if=/dev/st0 of=/mnt/synology/DigitalSlides001/recovery/recovered.tar bs=1M

I can recover data from tapes 1+2.

The tapes are super new, the drive is super new, almost unused.

To Reproduce
Only in our setting

Expected behavior
Tapes 1+2 should mount as LTFS

Additional context
Here I run the mount command unsuccessfully and some more info:

root@tapeserver:/mnt/tape# ltfs -o recoverindex -o devname=/dev/sg0 /mnt/tape
1450 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
1450 LTFS14058I LTFS Format Specification version 2.4.0.
1450 LTFS14104I Launched by "ltfs -o recoverindex -o devname=/dev/sg0 /mnt/tape".
1450 LTFS14105I This binary is built for Linux (x86_64).
1450 LTFS14106I GCC version is 13.2.0.
1450 LTFS17087I Kernel version: Linux version 6.8.0-60-generic (buildd@lcy02-amd64-054) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 i386.
1450 LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
1450 LTFS17089I Distribution: PRETTY_NAME="Ubuntu 24.04 LTS".
1450 LTFS14063I Sync type is "time", Sync time is 300 sec.
1450 LTFS17085I Plugin: Loading "sg" tape backend.
1450 LTFS17085I Plugin: Loading "unified" iosched backend.
1450 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
1450 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
1450 LTFS30250I Opened the SCSI tape device 0.0.0.0 (/dev/sg0).
1450 LTFS30207I Vendor ID is IBM     .
1450 LTFS30208I Product ID is ULTRIUM-HH9     .
1450 LTFS30214I Firmware revision is Q9E1.
1450 LTFS30215I Drive serial is 10WT027598.
1450 LTFS30285I The reserved buffer size of /dev/sg0 is 1048576.
1450 LTFS30294I Setting up timeout values from RSOC.
1450 LTFS17160I Maximum device block size is 1048576.
1450 LTFS11330I Loading cartridge.
1450 LTFS30252I Logical block protection is disabled.
1450 LTFS11332I Load successful.
1450 LTFS17157I Changing the drive setting to write-anywhere mode.
1450 LTFS11005I Mounting the volume from device.
1450 LTFS30252I Logical block protection is disabled.
1450 LTFS30216W Length mismatch is detected. (Act = 4096, resid = 0, resid_sense = -520192).
1450 LTFS12049E Cannot read: backend call failed (-21716).
1450 LTFS11174E Cannot read ANSI label: read failed (-21716).
1450 LTFS11171E Failed to read label (-21716) from partition 1.
1450 LTFS11009E Cannot read volume: failed to read partition labels.
1450 LTFS14013E Cannot mount the volume from device.
1450 LTFS30252I Logical block protection is disabled.

root@tapeserver:/mnt/tape# ltfsck -z /dev/sg0
LTFS16000I Starting ltfsck, LTFS version 2.5.0.0 (Prelim), log level 2.
LTFS16088I Launched by "ltfsck -z /dev/sg0".
LTFS16089I This binary is built for Linux (x86_64).
LTFS16090I GCC version is 13.2.0.
LTFS17087I Kernel version: Linux version 6.8.0-60-generic (buildd@lcy02-amd64-054) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 i386.
LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
LTFS17089I Distribution: PRETTY_NAME="Ubuntu 24.04 LTS".
LTFS17085I Plugin: Loading "sg" tape backend.
LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
LTFS30250I Opened the SCSI tape device 0.0.0.0 (/dev/sg0).
LTFS30207I Vendor ID is IBM     .
LTFS30208I Product ID is ULTRIUM-HH9     .
LTFS30214I Firmware revision is Q9E1.
LTFS30215I Drive serial is 10WT027598.
LTFS30285I The reserved buffer size of /dev/sg0 is 1048576.
LTFS30294I Setting up timeout values from RSOC.
LTFS17160I Maximum device block size is 1048576.
LTFS11330I Loading cartridge.
LTFS30252I Logical block protection is disabled.
LTFS11332I Load successful.
LTFS17157I Changing the drive setting to write-anywhere mode.
LTFS16014I Checking LTFS file system on '/dev/sg0'.
LTFS30252I Logical block protection is disabled.
LTFS30216W Length mismatch is detected. (Act = 4096, resid = 0, resid_sense = -520192).
LTFS12049E Cannot read: backend call failed (-21716).
LTFS11174E Cannot read ANSI label: read failed (-21716).
LTFS11171E Failed to read label (-21716) from partition 1.
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS16080E Cannot check volume (8).
LTFS30252I Logical block protection is disabled.

[piste-jp]

It looks your HBA have a funny behavior. Try the --enable-buggy-ifs option in the configure script. -> https://github.com/LinearTapeFileSystem/ltfs/wiki/HBA-info#note-for-buggy-hbas

Also you can refer the issues below.

#336
#295
#157
#74
#51

[smartinventure]

Thank you very much for your swift reply. BTW, I have this SAS PCI Card:
https://www.amazon.de/SAS-Controller-Broadcoms-3008-Compatible/dp/B07VV976RB/ref=sr_1_2?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=ANJUW7ZNS9K9&dib=eyJ2IjoiMSJ9.fp_wgkunDrQP0LcxmjaQtqwF4iWlwU1DAuPUKv1oBilJ8dHSgmBdV2r5VXSRvktehNpc-IYTqF70x_gwU6od-penna16oalRwH3qYRnWUvvQueRRzsAcCML1UWs2i1I6WAJYzKO_ZJ_IZ9WB8gv23IlHJjmb-XGjmoViR0b_Qt4azt5bU9klMRjUaVo0Z9mz0LHQgY2jRHMh7Qo6qmaklStiIdgJDz34NZdLwFRtRDM._IsgLwtkdoz2iO7b6jYLYIZmla6O_eu8oZUxzR8jgLU&dib_tag=se&keywords=Broadcom%2BSAS%2B9300-8e%2BSAS1200%2BController&qid=1721382103&refinements=p_85%3A20943776031&rnid=20943775031&rps=1&sprefix=broadcom%2Bsas%2B9300-8e%2Bsas1200%2Bcontroller%2Caps%2C149&sr=8-2&th=1

Is there a Linux/Ubuntu version for x64 pre-compiled with the --enable-buggy-ifs somewhere for download?
Martin

[smartinventure]

Now I built the LTFS anew with the option you mentioned. When I try to mount, it says the tape would not be partitioned. This is really weird, because I remember that tapes 1 + 2 were formatted with LTFS and I copied (fortunately TAR-) files onto them and from time to time did a ls -l and saw the content of yet copied files.

tapeserver@tapeserver:/tmp/tape/ltfs$ sudo ltfs -o recoverindex -o devname=/dev/sg0 /mnt/tape
60dc LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
60dc LTFS14058I LTFS Format Specification version 2.4.0.
60dc LTFS14104I Launched by "ltfs -o recoverindex -o devname=/dev/sg0 /mnt/tape".
60dc LTFS14105I This binary is built for Linux (x86_64).
60dc LTFS14106I GCC version is 13.3.0.
60dc LTFS17087I Kernel version: Linux version 6.8.0-60-generic (buildd@lcy02-amd64-054) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 i386.
60dc LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
60dc LTFS17089I Distribution: PRETTY_NAME="Ubuntu 24.04 LTS".
60dc LTFS14063I Sync type is "time", Sync time is 300 sec.
60dc LTFS17085I Plugin: Loading "sg" tape backend.
60dc LTFS17085I Plugin: Loading "unified" iosched backend.
60dc LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
60dc LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
60dc LTFS30250I Opened the SCSI tape device 0.0.0.0 (/dev/sg0).
60dc LTFS30207I Vendor ID is IBM     .
60dc LTFS30208I Product ID is ULTRIUM-HH9     .
60dc LTFS30214I Firmware revision is Q9E1.
60dc LTFS30215I Drive serial is 10WT027598.
60dc LTFS30285I The reserved buffer size of /dev/sg0 is 1048576.
60dc LTFS30205I RSOC (0xa3) returns -20601.
60dc LTFS30263I RSOC returns Not Ready to Ready Transition, Medium May Have Changed (-20601) /dev/sg0.
60dc LTFS30262I Forcing drive dump.
60dc LTFS39802W Unknown SCSI OP code 0x1d, use default timeout.
60dc LTFS30253I Saving drive dump to /tmp/ltfs_10WT027598_2025_0609_193655_f.dmp.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS39802W Unknown SCSI OP code 0x3c, use default timeout.
60dc LTFS30294I Setting up timeout values from vendor and device.
60dc LTFS17160I Maximum device block size is 1048576.
60dc LTFS11330I Loading cartridge.
60dc LTFS30252I Logical block protection is disabled.
60dc LTFS11332I Load successful.
60dc LTFS17157I Changing the drive setting to write-anywhere mode.
60dc LTFS11005I Mounting the volume from device.
60dc LTFS30252I Logical block protection is disabled.
60dc LTFS17168E Cannot read volume: medium is not partitioned.
60dc LTFS14013E Cannot mount the volume from device.
60dc LTFS30252I Logical block protection is disabled.

tapeserver@tapeserver:/tmp/tape/ltfs$ sudo ltfs -o recoverindex -o devname=/dev/sg0 -o trace /mnt/tape
60e1 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 3.
60e1 LTFS14058I LTFS Format Specification version 2.4.0.
60e1 LTFS14104I Launched by "ltfs -o recoverindex -o devname=/dev/sg0 -o trace /mnt/tape".
60e1 LTFS14105I This binary is built for Linux (x86_64).
60e1 LTFS14106I GCC version is 13.3.0.
60e1 LTFS17087I Kernel version: Linux version 6.8.0-60-generic (buildd@lcy02-amd64-054) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 i386.
60e1 LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
60e1 LTFS17089I Distribution: PRETTY_NAME="Ubuntu 24.04 LTS".
60e1 LTFS14063I Sync type is "time", Sync time is 300 sec.
60e1 LTFS17085I Plugin: Loading "sg" tape backend.
60e1 LTFS17085I Plugin: Loading "unified" iosched backend.
60e1 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
60e1 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
60e1 LTFS30250I Opened the SCSI tape device 0.0.0.0 (/dev/sg0).
60e1 LTFS30207I Vendor ID is IBM     .
60e1 LTFS30208I Product ID is ULTRIUM-HH9     .
60e1 LTFS30214I Firmware revision is Q9E1.
60e1 LTFS30215I Drive serial is 10WT027598.
60e1 LTFS30285I The reserved buffer size of /dev/sg0 is 1048576.
60e1 LTFS30294I Setting up timeout values from RSOC.
60e1 LTFS12023D Reserving device.
60e1 LTFS30392D Backend reserve (PRO) 10WT027598.
60e1 LTFS12028D Unlocking medium.
60e1 LTFS30392D Backend allow medium removal 10WT027598.
60e1 LTFS30392D Backend read block limits 10WT027598.
60e1 LTFS17160I Maximum device block size is 1048576.
60e1 LTFS11330I Loading cartridge.
60e1 LTFS30392D Backend load 10WT027598.
60e1 LTFS30398D Backend readpos: (0, 0) FM = 0 10WT027598.
60e1 LTFS12026D Locking medium in the drive.
60e1 LTFS30392D Backend prevent medium removal 10WT027598.
60e1 LTFS30398D Backend readpos: (0, 0) FM = 0 10WT027598.
60e1 LTFS30392D Backend sg_set_default Resetting LBP.
60e1 LTFS30393D Backend LBP Enable: 0 .
60e1 LTFS30393D Backend LBP Method: 2 .
60e1 LTFS30252I Logical block protection is disabled.
60e1 LTFS30392D Backend read block limits 10WT027598.
60e1 LTFS11332I Load successful.
60e1 LTFS17157I Changing the drive setting to write-anywhere mode.
60e1 LTFS11005I Mounting the volume from device.
60e1 LTFS11012D Loading the tape.
60e1 LTFS30392D Backend load 10WT027598.
60e1 LTFS30398D Backend readpos: (0, 0) FM = 0 10WT027598.
60e1 LTFS30398D Backend readpos: (0, 0) FM = 0 10WT027598.
60e1 LTFS30392D Backend sg_set_default Resetting LBP.
60e1 LTFS30393D Backend LBP Enable: 0 .
60e1 LTFS30393D Backend LBP Method: 2 .
60e1 LTFS30252I Logical block protection is disabled.
60e1 LTFS30392D Backend read block limits 10WT027598.
60e1 LTFS30397D Backend locate: (0, 0) 10WT027598.
60e1 LTFS30398D Backend readpos: (0, 0) FM = 0 10WT027598.
60e1 LTFS11007D Tape is loaded.
60e1 LTFS17168E Cannot read volume: medium is not partitioned.
60e1 LTFS14013E Cannot mount the volume from device.
60e1 LTFS12028D Unlocking medium.
60e1 LTFS30392D Backend allow medium removal 10WT027598.
60e1 LTFS12025D Releasing device.
60e1 LTFS30392D Backend release (PRO) 10WT027598.
60e1 LTFS30393D Backend LBP Enable: 0 .
60e1 LTFS30393D Backend LBP Method: 2 .
60e1 LTFS30252I Logical block protection is disabled.

[smartinventure]

On another thought:
After I had mounted tape 1 and/or tape 2, I used the TAR command to TAR folders (packed) onto my /mnt/tape, where I had the LTFS mounted.
After I had copied everything I could ls -l /mnt/tape successfully.
Is it maybe possible that the TAR command is not allowed in conjunction with LTFS?

[piste-jp]

On another thought: After I had mounted tape 1 and/or tape 2, I used the TAR command to TAR folders (packed) onto my /mnt/tape, where I had the LTFS mounted. After I had copied everything I could ls -l /mnt/tape successfully. Is it maybe possible that the TAR command is not allowed in conjunction with LTFS?

I believe you created a tar file like tar -cvf /mnt/ltfs/foo.tar .... If so, LTFS treats the tar archive as a file. LTFS never recognize any file types of any file on a LTFS tape at all.

[piste-jp]

I read through your bug description carefully. But your description confuses me a lot.

Can I make some questions?

Describe the bug We have BDT tape library with an IBM LTO-9 drive and LTO-9 tapes. I had formatted tapes 1, 2, and 3 with LTFS. I have written so many files to tapes 1 + 2 that they were full. I dismounted tape 1 correctly and moved the media into another slot. In tape 2, I admittedly made the error that I shutdown the server while the tape was idle, but LTFS not unmounted. Both cannot be mounted again.

OK. I believe you have only experience that the tape 1 and 2 are successfully mounted. It is just after format. In other words your sequence is as below. Is it correct?

1. Format tape 1 by mkltfs command
2. Mount tape 1 by ltfs command (Success)
3. Write so many files
4. Unmount tape 1 by umount or fusermount command
5. Mount tape 1 by ltfs command (Failed)
6. Format tape 2 by mkltfs command
7. Mount tape 2 by ltfs command (Success)
8. Write so many files
9. Shutdown the machine without unmount tape 2 by umount or fusermount command
10. Mount tape 2 by ltfs command (Failed)

I then formatted a new tape (3) with LTFS, wrote 2 files onto it, unmounted, unloaded, loaded again and successfully mounted. Below I added the result of the mount command (same for tapes 1 + 2, whereas 3 is successful) and some log.

I cannot understand this description. You said you experienced following sequence.

1. Format tape 1 by mkltfs command
2. Mount tape 1 by ltfs command (Success)
3. Write 2 files
4. Unmount tape 1 by umount or fusermount command
5. Mount tape 1 by ltfs command (Success)

But it is different result from tape 1 and 2....

Q1. What is your point here?
Q2. What is the "the result of the mount command" and the "some log" ? I cannot find them at all.

At least, with basic commands like: sudo mt -f /dev/st0 fsf 1 sudo dd if=/dev/st0 of=/mnt/synology/DigitalSlides001/recovery/recovered.tar bs=1M

I can recover data from tapes 1+2.

I cannot understand the descriptions above at all.

Q1. Why you are using mt command and dd command what is it?
Q2. You said I can recover data from tapes 1+2.. What is this mean?

[smartinventure]

Thank you for your answer:
you are absolutely right in what I have done:
I formatted 2 different tapes with LTFS. I stored files and I only unmounted one of the tapes incorrectly.
HOWEVER, I cannot remount neither of the 2 tapes at all.

Because I am very desparate (not being able to retrieve 16 TB of data, which I dont have anywhere else), I tried a dd command to read files.

Curiously, the first file on the tape is appr. 1 MB and a seconf file 9 MB.

The first bytes of the 1 MB file shows in the hex editor:

56 4f 4c 31 20 20 20 20 20 20 4c 20 20 20 20 20
20 20 20 20 20 20 20 20 4c 54 46 53 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 34

Which translates to "VOL1 L LTFS 4" (with many spaces in between).
The second file is an XML with all the data of what is inside.

Is there a way to "restore" the tape to being mountable again?

[smartinventure]

In the XML, I can see this:

IBM LTFS 2.5.0.0 (Prelim) - Linux - ltfs - Periodic Sync Periodic Sync - 2025-06-05T05:22:57.000529186Z 299530b5-e4d1-4f4a-bc55-55b233a0feb4 211 2025-06-05T05:22:57.002782628Z b 65107 b 10380591 true 6862 unlocked false 2025-06-04T08:03:08.858931403Z 2025-06-05T05:22:52.900498542Z 2025-06-05T05:22:52.900498542Z 2025-06-04T11:28:05.466822647Z 2025-06-04T08:03:08.858931403Z 1 HR24.6013 heOL.dcm.tar 2399191040 false 2025-06-04T11:29:14.599944755Z 2025-06-04T11:29:55.578957045Z 2025-06-04T11:29:55.578957045Z 2025-06-04T11:29:14.599944755Z 2025-06-04T11:29:14.599944755Z 3 0 b 675 0 2399191040 HR24.6110 b he_HR.dcm.tar 1866670080 false 2025-06-04T11:29:55.595803136Z 2025-06-04T11:30:42.317375639Z 2025-06-04T11:30:42.317375639Z 2025-06-04T11:29:55.595803136Z 2025-06-04T11:29:55.595803136Z 4 0 b

....

[Moredread]

Just in case, please be careful, you could easily damage the filesystem further. I personally would at least engage the write protect tab on the cartridge until I checked a few things!

I’m simply another user exploring the low-level aspects—use this advice cautiously and at your own risk; I won’t be responsible for any unintended outcomes.

Do NOT try to overwrite something inplace on the tape or write a file mark manually!!! This will likely make everything after it permanently unreadable.

Usually I think the next step would be to use ltfsck to check the filesystem and restore the latest index.

I'm not sure you haven't damaged the index partition though, so I would first try to get it reading read only first.

What you saw reading manually looks like the data on the index partition.

A LTFS filesystem uses two partitions on the tape, typically on the first the current "normal" index is written when you unmount the device, on the second is the data, and a second copy of the indices.

When you read from /dev/nst0 or another tape device you will see only the index partition without switching it manually. You should see at least three files (I think) containing data, with an empty one between some of them. The first is a ANSI VOL1 record, that is the 1MB file. The second should be the LTFS Label, a xml file describing the general structure of the filesystem. Then an empty file should come, then the index.

It could be that you read the first two files as one in the 1MB file. The larger 9MB file likely is your index, but maybe not the latest one.

The details are in https://www.snia.org/sites/default/files/technical-work/ltfs/release/SNIA-LTFS-Format-2.4.0-TechPosition.pdf btw.

Does the 9MB file look like the example on page 70?

Can you please post what you have up to ''<directory>''?

With the tape inserted where you got that 9MB file from (and the read-only slider engaged), can you post what you get when you run the ltfs mount command with "-o rollback_mount=", where is the path to the 9MB file. Theoretically it should mount the filesystem with the given index file, but I'm not sure it disables all other consistency checks, so that could still fail.

[piste-jp]

Thank you for your answer:
you are absolutely right in what I have done:
I formatted 2 different tapes with LTFS. I stored files and I only unmounted one of the tapes incorrectly.
HOWEVER, I cannot remount neither of the 2 tapes at all.

Hmm, it is really funny to me.

If you made the operations below, first tape unmounted at step4 shall be able to be mounted again. Because you can mount it at step 2. But the log said the tape is unpartitioned. If this really happens, LTFS is not a component to break the tape. Because LTFS only issues the command to make the tape partitioned to unpartitioned by mkltfs --wipe.

1. Format tape 1 by mkltfs command
2. Mount tape 1 by ltfs command (Success)
3. Write so many files
4. Unmount tape 1 by umount or fusermount command
5. Mount tape 1 by ltfs command (Failed)
6. Format tape 2 by mkltfs command
7. Mount tape 2 by ltfs command (Success)
8. Write so many files
9. Shutdown the machine without unmount tape 2 by umount or fusermount command
10. Mount tape 2 by ltfs command (Failed)

From the previous description and latest one, in my sense, something bad thing happens in HBA when LTFS issues READATTR or LOGSENSE command to detect partition. What is your HBA?

I strongly recommend that you pick a HBA IBM says 'supported'.