Rsync on ghostbsd seems defective

mikethe1wheelnut

I would put this in support & help requests if I planned on trying to fix the problem. I tried to back-up an external ssd onto an external hhd using ghost and rsync. I assumed the job had been completed correctly, it worked on it for like 2 days. Close to 1T. Now.. looking at the hdd, I see that there are tons of folders with nothing in them. I'll have to repeat the process, or do it some other way. One of you might want to check rsync for any obvious bugs there might be in the implementation.. (I -did- try to rsync a second time, from a different ssd, so maybe the second operation messed-up the first?) I'm not an expert rsync user, clearly.

ericbsd

mikethe1wheelnut What is the command you ran?

mikethe1wheelnut

ericbsd I used this "tutorial": [https://wiki.freeirc.org/?n=Openrsync.Usage](https://)

something like this:

openrsync -av --rsync-path=openrsync /path/to/folder username@example.com:/path/to/destination/

but with the correct source and destination, and some extra v's when I got frustrated that it wasn't telling me anything about what it was doing. I did the operation several times, using Ctrl-c to kill the process when several files seemed to give it trouble. Mostly on the second time around, when it was computing hashes of giant files and seemed to get stuck.

brianthehughes

Your situation sounds a lot more like a basic cp to me, and I went searching and found this thread talking about the best way to do a large directory tree locally.

https://serverfault.com/questions/43014/copying-a-large-directory-tree-locally-cp-or-rsync

Doing a large copy and having confidence that the information was correctly/fully copied leans toward rsync.

But why use openrsync?

It's also possible you have both hard drives on the same USB channel, which may escalate at least double the length of time.

Copying a 1 TB file using a USB 3.0 port could take anywhere from 1 to 2 hours, depending on the actual transfer speed, which can vary between 85 MB/s and 625 MB/s.

I only managed 300MB/s from one nvme drive to a folder on another nvme drive on my desktop.

Did you get a complete statement?

Transfer complete: 356 B sent, 54.097 GB read, 54.090 GB file size

mikethe1wheelnut

brianthehughes All I remember is that the process itself stopped, it brought me back to the prompt, so I assumed that the process had completed successfully. If required to place bet, I'd put it on -not- having gotten such a statement. I used openrsync because it was there in the repositories. I don't remember if any more ordinary rsync was also there. Assuming it would work, I didn't look into it further. As for recursively copying, or even using something like dd, the best I can give you is that I asked a similar question on reddit, and somebody suggested using rsync. I used it then, and it seemed to work, so why not keep using it. That was on ubuntu. I'll check out your link, it definitely looks a-propos. As for "the same usb channel".. this is new vocabulary for me. ..I had them both mounted on /media, using two different sub-folders.. My laptops all have 3 usb-ports, one used for usb mice, so I don't have too many options. ..always lots more research to do..

mikethe1wheelnut

brianthehughes If I ever wanted a definition of overwhelming, I've found it. That's not a tutorial on the best way to do it, it's a whole discussion naming 16 different options. I -was- going to just start all over again doing a basic recursive cp. Now I'm wondering. I'm cp'ing a terabite of data. [Lots of tiny files, lots of large files, lots of directories, lots of all kinds of files.. Just about any kind of file that can be obtained from the internet, it might be in there. Because at the time, I wouldn't have had any reason to -not- download and store a given type of file. ..Obviously you want to avoid mallware, if you can..] Apparently if it fails I have to start all over? Not encouraging. As for openrsync.. it doesn't mention openrsync at all, only rsync. Do they work the same way? They should, surely? ..well.. I -did- ask for this. Half my objective in switch operating systems is to learn a new operating system. Given that I don't know any, not in any detail, this is in no way redundant.

cp .....

`
-R If source_file designates a directory, cp copies the directory
and the entire subtree connected at that point. If the
source_file ends in a /, the contents of the directory are copied
rather than the directory itself. This option also causes sym-
bolic links to be copied, rather than indirected through, and for
cp to create special files rather than copying them as normal
files. Created directories have the same mode as the correspond-
ing source directory, unmodified by the process' umask.

     Note  that	**cp** copies hard linked files as separate	files.	If you
     need to preserve hard links, consider using [_tar_(1)](https://man.freebsd.org/cgi/man.cgi?query=tar&sektion=1&apropos=0&manpath=FreeBSD+14.2-RELEASE),  [_cpio_(1)](https://man.freebsd.org/cgi/man.cgi?query=cpio&sektion=1&apropos=0&manpath=FreeBSD+14.2-RELEASE),  or
     [_pax_(1)](https://man.freebsd.org/cgi/man.cgi?query=pax&sektion=1&apropos=0&manpath=FreeBSD+14.2-RELEASE) instead.`

Ok, so cp -R does -not- do what I want. What I want is -supposed- to be really simple. I want to take the data from one device and copy it to another device. Everything. Because I don't know what is in there. There might be sim-links, there might not. I'm not even entirely sure what those are. Surely there must be a simple way of doing that? These aren't -really- questions I asking for answers to. This is more of a vent than anything. I need to check back in your repositories to see if rsync is available. I think that if it was, I would have used it. So I think it can't have been.

mikethe1wheelnut

brianthehughes Why use openrsync? This is actually a very good question. Looking at the packages available, it is one of many. I think 13, if I count correctly. grsync, libr, open, ps5 (x3).. down to rubygem..

I went for openrsync because I have become enamoured with OpenBSD generally. So I assumed that their implementation would be "superior". Obviously now I have reason for looking deeper. One link.. [https://www.libhunt.com/compare-openrsync-vs-rsync](https://) ..another link.. [https://news.ycombinator.com/item?id=43605003](https://)

..and.. I have now confirmed the "We are still working on it... so please wait." claim in the above. [https://www.openrsync.org/](https://)

..so that pretty-much is the answer to this whole mess. ..re-start, using regular normal rsync. ..that said.. why the hell is it in the repositories if it is so publicly not ready for prime-time? seriously!?

..let this be part of my learning curve. I read in the OpenBSD documentation (somewhere) that package/port maintainers put most of the efforts into cleaning the packages, not the ports, so I had assumed that this also applied to other distributions. ..I wonder if openrsync is available as a package for openbsd? It should not be. ..right?

michaelcarlos

It almost seems like something went wrong with your rsync backup process such that most of the folders were left empty. This might have been due to a number of factors—such as an interrupted or incomplete transfer, or perhaps something happened with the second attempt at rsync that corrupted the first one. Possibly, too, rsync was having some difficulty syncing large volumes of data between external drives. You could attempt a new backup, double-check the rsync commands, or even use a different tool to check the integrity of the files. If you are still worried about rsync itself, it may be worth searching for any known bugs or inconsistencies in the version you are running.

mikethe1wheelnut

michaelcarlos I agree with your list of possibilities. I'm going to check out brian's link, cause it sounds like somebody has a fairly comprehensive answer written out. At this point, if I can find the right tool for the job, the motivation for figuring out why the wrong tool doesn't work isn't there.

mikethe1wheelnut

mikethe1wheelnut Having reflected on this a bit more, it seems quite clear to me that -GhostBSD-, at least, should not have software in it's repositories, easy to install, that is not ready for prime-time. That is what GhostBSD -is-, after all, right? The operating system for people curious about the bsd's in general, but not ready yet for the full dive? The last thing you want is for people trying it out to trip up on software that is not yet "fully polished". This is a marketing thing. GhostBSD is a marketing thing. Or am I wrong? Either way, -particularly if you want users staying around, ie, not switching over to freebsd or something else-, you really want to only be giving them easy access to programs that really are known to be solidly checked out and ready. What you could do is have a dedicated repository for these codes. accessed through a different command. so that people who don't want any head-aches can use that as a check of what they should and shouldn't be considering. ..my two-cents worth :-)

brianthehughes

Lots to review in your efforts and follow up.

Any software in the ports tree is available for GhostBSD, like FreeBSD - so it’s up to a user to ensure they are running appropriate software for their needs. GhostBSD can’t reasonably carry the effort of validating the 36000+ softwares.

Think about the USB connectivity on your laptop as a street with a few lanes (USB ports). All your usb traffic in both directions is likely travelling on the same street. At best, ports on opposite sides of your laptop are likely two separate USB streets.

When working with medium-large datasets such as you describe, I generally grab a subset, perhaps a directory that represents 5-10% of the overall and test transfer of that subset first to

validate commands
measure data transfer / time

Sometimes, I simply plug in both devices, open each in a file manager window and make my way through copying directories, one at a time. This provides a visual progression and if things fail, I can nuke the last directory and restart it.

What are the two, (three?) file systems in play here?

source drive
destination drive
system

mikethe1wheelnut

brianthehughes

Any software in the ports tree is available for GhostBSD, like FreeBSD - so it’s up to a user to ensure they are running appropriate software for their needs. GhostBSD can’t reasonably carry the effort of validating the 36000+ softwares.

This is probably clearly laid out in the documentation I read.. months and months, possibly years, ago. And since then I've built my own idea of what GhostBSD should be, or is supposed to be, divorced from reality :-)

mikethe1wheelnut

brianthehughes What are the two, (three?) file systems in play here?
source drive
destination drive
system

ok, source: exfat, destination: ext2fs, system: ..whatever ghost uses. ..I'd try looking that up, but.. somebody said the wiki was out of date, and I haven't actually tried tracking down the "real official" documentation yet.

(as determined by fstyp -l)

mikethe1wheelnut

brianthehughes Think about the USB connectivity on your laptop as a street with a few lanes (USB ports). All your usb traffic in both directions is likely travelling on the same street. At best, ports on opposite sides of your laptop are likely two separate USB streets.

This is actually really really neat. I didn't expect this exploration to teach me much about -hardware-..

mikethe1wheelnut

mikethe1wheelnut ..is this it? the official documentation? [https://ghostbsd-documentation-portal.readthedocs.io/en/latest/](https://) ..I mean, it looks like it..

ericbsd

Oh, it's openrsync. I've never used it. I always use rsync.

ericbsd

You might want to use the rsync that everyone uses.

mikethe1wheelnut

ericbsd Yes, that is the current plan :-)