Compressing a Timeshift image, why is it failing

By hell this must be possible somehow!!!

Mint 21.1 Vera, MATE 64 bit

I've been trying to compress a Timeshift snapshot on an external drive so I can free up more space on that drive. I haven't been having much success. I'll explain below what I tried, but it doesn't seem to have worked and I can't figure out why.

I took the existing image, the Timeshift directory on an ext4 partition of the drive and did

sudo dd if=/dev/sdb1 of=/home/username/timeshift_img/Backup.img status=progress

This copied without error. And I was able to mount it, which had it looking just like the original partition when I did so. But I note that on the original partition there must have been some sort of permissions which meant some folders had padlock symbols shown on them in the caja GUI, whereas when browsing the img file once mounted all the folders showed normally, though as dd makes an exact copy it surely must have preserved permissions in practice.

I then compressed that img file using 7zip

7z a BackupCompressed.7z Backup.img -mx=3

which worked.

So now it was time to test. I went to a second computer and on it created a VM (Oracle Virtual box) as I had no spare actual machine to test on, and installed default Mint 21.1 Mate 64 bit in to it, I ensured the VM had the same sort of partition structure and the same partition names as the system from which the Timshift snapshot was taken.

I put the 7z file on a USB, brought it across.

On this PC I uncompressed the 7z to make the img file again.

I put shared the img file in a read and write capable way with the guest of the VM, specifically I put it on another external HDD and then gave the VM USB port access via which to access this external HDD.

Within the VM I mounted the img file so it showed as a dev/loop0 drive inside the VM, looking just like the original partition had, but with no padlock logo superimposed on the folder logo when browsing.

In the guest I opened Timeshift, and it did detect this snapshot, showing correctly the time and date of the snapshot and the associated comment.

So I tried to restore from it, it appeared to work, but then it got to the final stage and clicking next then, rather than starting a long copying process, skipped straight to the very last stage in which it said it had completed, despite not having done anything, even though at an earlier stage it had corectly listed all the files it would be deleting/creating/modifying.

Also, in Timeshift while the snapshot was found and shown to be on the /dev/loop0 drive, I could not in the Settings pop up window get Timeshift to list the /dev/loop0 device, it listed only the virtual partitions of the VM and the existence of the plugged in external HDD. This view of Timeshift's settings didn't show the /dev/loop0 mounted img file, yet Timeshift's main window found it well enough.

What went wrong?

How come the permissions padlock wasn't shown in the dd 'd copy?
Why did Timeshift see the Timeshift image within the mounted img file, and ist the changes to make when restoring from it, but then take zero time to complete and in the process not restore to the Timeshift snapshot at all?

My end goal is to be able to get Timeshift snapshots saved iside of 7z (or zip or other compressed older file types) which can be saved on ntfs drives, rather than needing to have a whole ext4 partition to live on. I don't mind if when I want to use the snapshots I have to spend a while decompresing them via a series of copies, I just want to make sure they work however the snapshot has been stored in the meantime.

Thanks

Edited by rp88, 24 September 2023 - 08:31 PM.

@rp88

 

I was under the impression that Timeshift for Linux worked something like System Restore for Windows.

 

That is, it takes a "snapshot" of the existing system, and creates a restore point for that system.

 

So, if you change something and the system stops working, then you can revert to that point in time.

 

I don't think that it includes the content of /home directories.  I could be wrong.

 

That being said, transferring a Timeshift file to another system will not work, because it needs the original system HDD it was made on.

 

Timeshift will be looking to restore files from a file system table, that does not match the different, (new), HDD.

 

Hence, when you tried to restore, Timeshift had nothing to do.

 

Cheers!

 

Naught

 

@ rp88 :-

 

I agree with Naught's assessment of the situation. By attempting to "restore" to a different disk, you're losing the very thing that Timeshift needs to be able to do its job.......the UUID of the original disk that it initially backed-up.

 

I WILL also add this; there comes a point when further compression of already-compressed files actually begins to lose data. Not because the compression algorithms aren't working, but because the fine-grained detail of the 'metadata' - the information that tells the system how to organise the actual data itself on the disk - becomes lost in the compression process.

 

You can see this process at work when re-sizing images. Resize an image of, say, 1000x1000 pixels down to 100x100. The small image looks fine, so you save it. But now try to re-size the new, smaller image back up to the size it was originally.....and it looks really bad. Blocky, pixelated, all the detail is lost.....rather like a MineCraft image. And you can never recover it.

 

NOT recommended.

 

 

Mike.

Timeshift definitely does work to clone a Linux system, I've done it before. You can manually tell it to back up files from the home folder where needed, or just specific subfolders and hidden folders within it. After restoring it on a different mahine you have to fiddle around with fstab a bit to edit UUIDs and make it boot properly, but with that done it really does appear to work for system cloning.

 

The trouble here is that in this particular circumstance it does not work, despite supposedly evervything being the same, except that the image is now stored in an img rather than on a real drive partition. Either something about dd ing the original partition to the img file broke the file permissions or linking of the ext4 formatted timeshift snapshot, or when trying to restore Timeshift somehow can see a /dev/loop0 drive for the purposes of listing it as an available snapshot, but can't access the files on it the way it can for /dev/sdXY partitions.

 

As for compresson, 7z is lossless, fully reversible, it can compressbinary files of all kins just fine, and disk images are the type of files which shrink pretty substantially under lossless compression, much more so than image or video files do. Compressing a disk image of an ext4 partition in to a 7z as a shrunken sized storage location is fundamentally not the same as compressing a 2000x2000 pixel image in to a 1000x1000 area.

 

It is because I have got Timeshift as a system cloning tool to work before, and work well, that I'm confused as to why it won't work when I try to use dd to make img file copies of a timeshift partition rather than have to have inconvenient ext4 partitions all over the place. A dd copied img file copied from a phsycial disc partition should be the same in every way? That's what dd is designed to do.

Edited by rp88, 27 September 2023 - 04:36 PM.

Personally, I would have been satisfied with Old School tools like dd and gz, and not bother with tools I don't understand like timeshift backup.

 

Perhaps these new tools changed somehow?  Perhaps just a new bug?  I would recommend that you do a test run and compare the backed-up data to the original data to make sure that the procedure is still relevant.

Can I check, dd does indeed make an exact copy of a partition, errors and all, so ext4 specific linking and permissions should always be the same after dd cloning?

 

Does this work for dd cloning of a partition alone (/dev/sdb1 for example) or does one have to do a dd of the whole disk (/dev/sdb)?

 

Does one have to clone the whole disk sdb, to get ext4 specific permissions and linking to copy over properly, or should it work when cloning just the partition sdb1?

 

Also, with dd, if you have a drive, say sdc, where there are two partitions sdc1, sdc2 and maybe some free space afterwards... if you dd that whole drive in such a manner that it is trying to copy on to another drive (sdd) which is big enough for sdc1, but not big enough for sdc1+sdc2+(the free space)... would the cloned drive still have a valid copy of sdc1 on it, but a corrupted sdc2 and no free space? If only cloning sdc1 mattered, and you didn't care about sdc2 or the free space, would it work? Or would this not work at all?

 

Thanks

rp88, on 28 Sept 2023 - 5:25 PM, said:

Can I check, dd does indeed make an exact copy of a partition, errors and all, so ext4 specific linking and permissions should always be the same after dd cloning?

I would say YES!

 

rp88, on 28 Sept 2023 - 5:25 PM, said:
Does this work for dd cloning of a partition alone (/dev/sdb1 for example) or does one have to do a dd of the whole disk (/dev/sdb)?

dd requires a device, at least my understanding using UNIX. Everything within that device is duplicated verbatim.

 

rp88, on 28 Sept 2023 - 5:25 PM, said:
Does one have to clone the whole disk sdb, to get ext4 specific permissions and linking to copy over properly, or should it work when cloning just the partition sdb1?

Just use the device that you need.

 

rp88, on 28 Sept 2023 - 5:25 PM, said:
Also, with dd, if you have a drive, say sdc, where there are two partitions sdc1, sdc2 and maybe some free space afterwards... if you dd that whole drive in such a manner that it is trying to copy on to another drive (sdd) which is big enough for sdc1, but not big enough for sdc1+sdc2+(the free space)... would the cloned drive still have a valid copy of sdc1 on it, but a corrupted sdc2 and no free space? If only cloning sdc1 mattered, and you didn't care about sdc2 or the free space, would it work? Or would this not work at all?
 
Thanks

dd will fail because free space is still content to be duplicated.
 
 

I found a way to get it to complete without error, my earlier problems were from:

a ) dd'ing a clone of only the timeshift partition, not the whole timeshift USB
and
b ) trying to restore it inside a VM, this time I restored on to a real HDD on a physically real computer instead

hence

NOW IT WORKS!!!

Guide included below so anyone interested can do it to.

Here's how to make the compressed image:
1. Create a timeshift snapshot of the system, with the various settings to copy home and other directories it normally wouldn't, but which you'll want to copy if you're to make the image contain programs (like exe files under wine) which live in a home directory. You'll probably want to copy hidden files (like the bash history) as well as all files.
2. Make this snapshot on the smallest USB which is big enough to fit it
3. I'd advise some restarting of the system and such at this point, in my use case it was months between making the timeshift USB and learning how to do the next bit. The next bit could also be done on a totally different system to the one on which the original timeshift USB was made.
4. Run
sudo dd if=/dev/sdX of=/home/username/timeshift_img/NameBackupSDB.img status=progress
with sdX being the name the USB gets given when plugged in (warning, it might have a different name depending what order you plug it in with compared to other devices connected at the time). It is very important to clone it as /dev/sdb, NOT as /dev/sdb1, you want the whole drive, not just the partition. Ensure you have spare HDD space big enough the whole USB's size before you begin.
5. Run, from inside the timeshift_img folder,
7z a SDBNameBackupCompressed.7z NameBackupSDB.img -p -mhe=on -mx=3
, mx=3 is useful here as it quickens the compressing by not trying to optimise too hard for file size
, mhe=on is optional, if you want to encrypt the backup, a wise move if your original timeshift snapshot had password files on the system which you copied and therefore now stored within it. Before running, ensure you have spare HDD space for a new 7z file sized to be a good proportion of the size of the img file.
6. That 7z, albeit potentially a rather big file, is now fully portable. You can save it on any drive, anywhere, with any formatting for the drive so long as it can cope with BIG files (30GB or so for the 7z in my use case) (so NTFS formatted filesystems, but not FAT32 or exFAT).

Here's how to use it:
7. Start by installing your Linux OS fresh on a new system, ensure you install Timeshift within it if Timeshift isn't supplied by default (it is there by default in Mint 21.1, where I tested this). When installing make sure to match partition names, sizes and order to that of the original system you are cloning.
8. Once the fresh install is done and rebooted, so you don't have the live USB for it plugged in any more, copy the 7z file on to this system, uncompress it with 7z either via terminal or via the GUI archive extraction file program so as to create an extracted copy of NameBackupSDB.img on the new system.
9. Get an extra USB, larger than the one originally used for storing the snapshot, and use dd to write the img file to it. It is vital this be bigger than the original, it absolutely must not be smaller. If the original USB was 62GB*, even if only 30GB was used, you need a USB >62GB here even though only the first 30GB of the img file is used. I hear rumours that many drives keep a partition table at the very back end of the drive, so writing an overly large img file overwrites this and makes the drive useless until you've used gparted to wipe it and write a new partition table. I definitely found this the case, I had a 61555605504 byte original USB containing a 57GB timeshift partition plus free space, I initially tried to dd the img to a drive which had a real capacity of 61,530,439,680 bytes, dd gave no errors, but the USB created could not be read even though the only partition which mattered in the drive I had cloned from was only 57GB. I had to use Gparted to make a new partition table, which wiped what I had just dd'd on to it. I retried with a drive which read as 64GB actual capacity, all was fine this time.
sudo dd if=ExtractedNameBackupSDB.img of=/dev/sdc status=progress
sdc here is for the new USB ( >(original size) )you are writing to. If your original was, for example, 62GB and you can't find a truly 64GB USB, use a 128GB here.

*here you notice how important the diferences of advertised capacity and actual can be, many 64GB USBs are 62GB or 58GB... and you also note the irritaing differences of GB vs GiB (Giga vs Gibi, 1000 based vs 1024 based) and how you're never really sure which of Gb and GiB any given program displays figures in and a USB is advertised as

10. Safely eject the USB and restart the freshly installed system
11. When you have restarted, start timeshift on the new system and set it NOT to take automated (daily, or other timeframe) snapshots
11.5. Turn OFF screensaver settings, tell the PC not to go to sleep, or lock the screen
12. Now plug in the USB which you dd'd the img file to, in Timeshift's settings tell Timeshift to look to that USB's partition as the place for snapshots
13. Leaving timeshift's settings alone, proceed to restore the snapshot from that USB
13.5. At the "select target device" stage pick the paths for "/" and "/home" to be the relevant sdaY and sdaZ partitions of the internal HDD, leave "/boot" and "/boot/efi" as "Keep on root device". In bootloder options select the defaults (do "(Re)install Grub2 on:" and pick your SDA hard drive as the device, do not pick a partition, do not update initramfs, do "update grub menu").
14. Timeshift will do its stuff, and then put the PC in to a state where the whole screen is taken up by a listing of the files Timeshift is restoring
15. Then it will try to restart, and then it will crash
16. Power down the PC, and boot from he live USB again
17. With the live USB booted, mount the root partition of the internal HDD, and run
sudo caja
to create an elevated privilege file browser.
18. Browse to /etc/fstab on the internal HDD's root partition and open it in the elevated privilege text editor
19. Open the Live USB's "Disks" program
20. Edit /etc/fstab so each UUID it presently shows is replaced by the appropriate UUID seen in the Disks window, copy and paste from the Disks window to the /etc/fstab text file each UUID, or the EFI partition, root partition, home partition and any others. Note, if there were a way to work the other way round, telling the live system what UUIDs you want for each partition at an earlier point in time, I'd be interested in understanding that option too if anyone can tell me of ways to choose the UUID during installation of the OS to the HDD while booted from a Live USB.
21. Shut down the PC, remove the Live USB, now boot normally, log in, and open Timeshift again. Note that during pre-login the desktop background will match any custom one you have set, but after login it will still look like the default used on the fresh install.
22. At this point some parts of the snapshot have been restored, but some haven't, now we'll get them all restored
23. When you open Timeshift now it will have all the correct folder filters as were used when your original snapshot was made. Plug in your timeshift snapshot USB, select it as the device to restore from under "Location". When restoring from it use the same "select target device" settings as you did earlier on.
24. Allow it to restore, and go to the screen of scrolling file names again, and shut itself down
25. This time when it restarts it will crash again, you'll need to use the live USB again to make the /etc/fstab changes
26. With those changes made, the same way as before, shut down
27. Then take out the live USB, power up, and you'll have a new system as a clone of the original, having been transferred via a fully portable 7z file
28. You can then wipe the USB used as the timeshift snapshot (wipe just the larger later one, or both of them) and use for normal file storage. You have a 7z file which can be used as a means of generating a timeshift partition USB when one is needed, but when it isn't needed you don't need to use up USB space with the timeshift snapshot, you can store it as just a file on a large enough ntfs partition within any data storage device you have to hand.

Hope this helps you.

Edited by rp88, 11 October 2023 - 08:52 PM.

@rp88

 

I found a way . . . the whole timeshift USB . . . the whole timeshift USB . . . to a real HDD on a physically real computer . . .NOW IT WORKS!!!

 

Congratulations! 

 

You should repost the solution as a "How To" in the Tutorial Section. 

 

Cheers!

 

Naught

Edited by Naught McNoone, 12 October 2023 - 02:10 PM.