FreeNAS (FreeBSD) Shell rsync & cp Commands
How do you copy files (and subordinate files/folders) from one folder/volume in FreeNAS to another folder/volume using the rsync command from shell?
rsync Command - Copy files and directories.
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
One FreeNAS Server - Copy files and folders to a new location in FreeNAS
Run the following command from shell inside FreeNAS:
rsync [options] [source] [destination]
rsync -Whrtv --progress --stats /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUMENAME2/FOLDER3/FOLLDER4/
OR if you need/want to copy file permissions use:
rsync -Whav --progress --stats /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUMENAME2/FOLDER3/FOLLDER4/
This will copy the contents of "folder2" into "folder4". Note that "folder4" must exist prior to running the command. If your folder name contains a space include the entire path between apostrophes, e.g. 'folder 2'.
-v, --verbose
This option increases the amount of information the daemon logs
during its startup phase. After the client connects, the dae-
mon's verbosity level will be controlled by the options that the
client used and the "max verbosity" setting in the module's con-
fig section.
--partial
By default, rsync will delete any partially transferred file if
the transfer is interrupted. In some circumstances it is more
desirable to keep partially transferred files. Using the --par-
tial option tells rsync to keep the partial file which should
make a subsequent transfer of the rest of the file much faster.
--progress
This option tells rsync to print information showing the
progress of the transfer. This gives a bored user something to
watch.
This option is normally combined with -v. Using this option
without the -v option will produce weird results on your dis-
play.
--stats
This tells rsync to print a verbose set of statistics on the
file transfer, allowing you to tell how effective the rsync
algorithm is for your data.
-e, --rsh=COMMAND
This option allows you to choose an alternative remote shell
program to use for communication between the local and remote
copies of rsync. By default, rsync will use rsh, but you may
like to instead use ssh because of its high security.
You can also choose the remote shell program using the RSYNC_RSH
environment variable.
See also the --blocking-io option which is affected by this
option.
----------------------------------------------------------------------
Two FreeNAS Servers - Copy files between two different FreeNAS servers
Turn on SSH service of destination FreeNAS server. Destination dataset must be set to UNIX share type. Run the following command from shell inside FreeNAS:
rsync -Whav --progress --stats /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ [email protected]:/mnt/VOLUMENAME2/FOLDER3/FOLDER4/
"x.x.x.x" is the IP of the destination FreeNAS server. This will copy the contents of "folder2" into "folder4". Note that "FOLDER4" must exist prior to running the command. You will get approximately 50 MB/s transfer speed.
----------------------------------------------------------------------
cp Command - Copy files and directories.
In the first synopsis form, the cp utility copies the contents of the source_file to the target_file. In the second synopsis form, the contents of each named source_file is copied to the destination target_directory. The names of the files themselves are not changed. If cp detects an attempt to copy a file to itself, the copy will fail.
One FreeNAS Server - Copy files and folders to a new location in FreeNAS
Run the following command from shell inside FreeNAS:
cp [options] [source] [destination]
cp -iav /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUME NAME2/FOLDER3/FOLLDER4/
OR if you do not want to copy file permissions (will not copy times):
cp -iRv /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUME NAME2/FOLDER3/FOLLDER4/
Useful cp commands:
-i Interactive. Prompt you if overwriting any files.
-a Archive mode. Same as -RpP.
-R Recursive. Will copy sub-directories.
-p Preserve file attributes like date, time, etc.
-P If the -R option is specified, no symbolic links are followed. This is the default.
-v Verbose. Show you each file as it is copied in the command line console.
This will copy the contents of "folder2" into "folder4". Note that "folder4" must exist prior to running the command.
Long explanations of cp commands:
-i Cause cp to write a prompt to the standard error output before
copying a file that would overwrite an existing file. If the
response from the standard input begins with the character `y' or
`Y', the file copy is attempted. (The -i option overrides any pre-
vious -f or -n options.)
-p Cause cp to preserve the following attributes of each source file
in the copy: modification time, access time, file flags, file mode,
ACL, user ID, and group ID, as allowed by permissions.
If the user ID and group ID cannot be preserved, no error message
is displayed and the exit value is not altered.
-R If source_file designates a directory, cp copies the directory and
the entire subtree connected at that point. If the source_file
ends in a /, the contents of the directory are copied rather than
the directory itself. This option also causes symbolic links to be
copied, rather than indirected through, and for cp to create spe-
cial files rather than copying them as normal files. Created
directories have the same mode as the corresponding source direc-
tory, unmodified by the process' umask.
-v Cause cp to be verbose, showing files as they are copied.
--------------------------------------
Transfer rates for each that I experienced:
1st - cp one server - .51 GB/s
2nd - rsync one server - 0.18 GB/s
3rd - rsync two server - 0.06 GB/s
rsync Command - Copy files and directories.
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
One FreeNAS Server - Copy files and folders to a new location in FreeNAS
Run the following command from shell inside FreeNAS:
rsync [options] [source] [destination]
rsync -Whrtv --progress --stats /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUMENAME2/FOLDER3/FOLLDER4/
OR if you need/want to copy file permissions use:
rsync -Whav --progress --stats /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUMENAME2/FOLDER3/FOLLDER4/
This will copy the contents of "folder2" into "folder4". Note that "folder4" must exist prior to running the command. If your folder name contains a space include the entire path between apostrophes, e.g. 'folder 2'.
Useful rsync commands:
-W, --whole-file copy files whole (w/o delta-xfer algorithm)
-h, --human-readable output numbers in a human-readable format
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
--devices preserve device files (super-user only)
--specials preserve special files
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
--devices preserve device files (super-user only)
--specials preserve special files
-v, --verbose increase verbosity
-P same as --partial --progress
--partial keep partially transferred files
--progress show progress during transfer
--stats give some file transfer stats
-e, --rsh=COMMAND specify rsh replacement
-z, --compress compress file data during the transfer
-n, --dry-run perform a trial run with no changes made
--delete delete extraneous files from dest dirs
--partial keep partially transferred files
--progress show progress during transfer
--stats give some file transfer stats
-e, --rsh=COMMAND specify rsh replacement
-z, --compress compress file data during the transfer
-n, --dry-run perform a trial run with no changes made
--delete delete extraneous files from dest dirs
Long explanations of commands:
-W, --whole-file
With this option rsync's delta-transfer algorithm is not used
and the whole file is sent as-is instead. The transfer may be
faster if this option is used when the bandwidth between the
source and destination machines is higher than the bandwidth to
disk (especially when the "disk" is actually a networked
filesystem). This is the default when both the source and des-
tination are specified as local paths.
-h, --human-readable
Output numbers in a more human-readable format. This makes big
numbers output using larger units, with a K, M, or G suffix. If
this option was specified once, these units are K (1000), M
(1000*1000), and G (1000*1000*1000); if the option is repeated,
the units are powers of 1024 instead of 1000.
-a, --archive
This is equivalent to -rlptgoD. It is a quick way of saying you
want recursion and want to preserve almost everything (with -H
being a notable omission). The only exception to the above
equivalence is when --files-from is specified, in which case -r
is not implied.
Note that -a does not preserve hardlinks, because finding multi-
ply-linked files is expensive. You must separately specify -H.
-v, --verbose
This option increases the amount of information the daemon logs
during its startup phase. After the client connects, the dae-
mon's verbosity level will be controlled by the options that the
client used and the "max verbosity" setting in the module's con-
fig section.
--partial
By default, rsync will delete any partially transferred file if
the transfer is interrupted. In some circumstances it is more
desirable to keep partially transferred files. Using the --par-
tial option tells rsync to keep the partial file which should
make a subsequent transfer of the rest of the file much faster.
--progress
This option tells rsync to print information showing the
progress of the transfer. This gives a bored user something to
watch.
This option is normally combined with -v. Using this option
without the -v option will produce weird results on your dis-
play.
--stats
This tells rsync to print a verbose set of statistics on the
file transfer, allowing you to tell how effective the rsync
algorithm is for your data.
-e, --rsh=COMMAND
This option allows you to choose an alternative remote shell
program to use for communication between the local and remote
copies of rsync. By default, rsync will use rsh, but you may
like to instead use ssh because of its high security.
You can also choose the remote shell program using the RSYNC_RSH
environment variable.
See also the --blocking-io option which is affected by this
option.
----------------------------------------------------------------------
Turn on SSH service of destination FreeNAS server. Destination dataset must be set to UNIX share type. Run the following command from shell inside FreeNAS:
rsync -Whav --progress --stats /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ [email protected]:/mnt/VOLUMENAME2/FOLDER3/FOLDER4/
"x.x.x.x" is the IP of the destination FreeNAS server. This will copy the contents of "folder2" into "folder4". Note that "FOLDER4" must exist prior to running the command. You will get approximately 50 MB/s transfer speed.
----------------------------------------------------------------------
In the first synopsis form, the cp utility copies the contents of the source_file to the target_file. In the second synopsis form, the contents of each named source_file is copied to the destination target_directory. The names of the files themselves are not changed. If cp detects an attempt to copy a file to itself, the copy will fail.
One FreeNAS Server - Copy files and folders to a new location in FreeNAS
Run the following command from shell inside FreeNAS:
cp [options] [source] [destination]
cp -iav /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUME NAME2/FOLDER3/FOLLDER4/
OR if you do not want to copy file permissions (will not copy times):
cp -iRv /mnt/VOLUMENAME1/FOLDER1/FOLDER2/ /mnt/VOLUME NAME2/FOLDER3/FOLLDER4/
Useful cp commands:
-i Interactive. Prompt you if overwriting any files.
-a Archive mode. Same as -RpP.
-R Recursive. Will copy sub-directories.
-p Preserve file attributes like date, time, etc.
-P If the -R option is specified, no symbolic links are followed. This is the default.
-v Verbose. Show you each file as it is copied in the command line console.
This will copy the contents of "folder2" into "folder4". Note that "folder4" must exist prior to running the command.
Long explanations of cp commands:
-i Cause cp to write a prompt to the standard error output before
copying a file that would overwrite an existing file. If the
response from the standard input begins with the character `y' or
`Y', the file copy is attempted. (The -i option overrides any pre-
vious -f or -n options.)
-p Cause cp to preserve the following attributes of each source file
in the copy: modification time, access time, file flags, file mode,
ACL, user ID, and group ID, as allowed by permissions.
If the user ID and group ID cannot be preserved, no error message
is displayed and the exit value is not altered.
-R If source_file designates a directory, cp copies the directory and
the entire subtree connected at that point. If the source_file
ends in a /, the contents of the directory are copied rather than
the directory itself. This option also causes symbolic links to be
copied, rather than indirected through, and for cp to create spe-
cial files rather than copying them as normal files. Created
directories have the same mode as the corresponding source direc-
tory, unmodified by the process' umask.
-v Cause cp to be verbose, showing files as they are copied.
--------------------------------------
Transfer rates for each that I experienced:
1st - cp one server - .51 GB/s
2nd - rsync one server - 0.18 GB/s
3rd - rsync two server - 0.06 GB/s
