How to optimize data syn­chro­niza­tion with rsync

rsync is a versatile tool that sim­pli­fies file transfer over network con­nec­tions and speeds up the syn­chro­niza­tion of local di­rec­to­ries. The high flex­i­bil­i­ty makes the syn­chro­niza­tion tool an excellent option for a variety of file-level op­er­a­tions.

What is rsync?

rsync, short for “remote syn­chro­niza­tion”, is a flexible and network-com­pat­i­ble syn­chro­niza­tion tool under Linux. The open-source program can be used to syn­chro­nize files and di­rec­to­ries between local systems or across networks. The tool uses a dif­fer­en­tial data transfer technique, whereby only those sections of data that have actually been changed are trans­ferred. This minimizes the amount of data exchange and con­sid­er­ably speeds up the syn­chro­niza­tion process. Thanks to a variety of options, rsync allows precise control of syn­chro­niza­tion behavior. The flexible syntax makes both simple local copies and complex network syn­chro­niza­tions possible.

What is the syntax for rsync?

The command syntax of rsync has a simple structure and is similar to that of SSH, SCP and CP. The basic structure is as follows:

rsync [OPTION] source destination

The source path that the data should be syn­chro­nized from is entered in source, while the des­ti­na­tion path is specified as destination. rsync offers a variety of options which users can use to adapt the syn­chro­niza­tion process to their re­quire­ments. The most fre­quent­ly used options are:

• -a (archives): Preserves recursive file per­mis­sions, time­stamps, groups, owners and special file prop­er­ties
• -v (verbose): Displays detailed in­for­ma­tion about the syn­chro­niza­tion process
• -r (recursive): Syn­chro­nizes di­rec­to­ries and their contents re­cur­sive­ly
• -u (update): Only transfers files that are newer than those already in the target directory
• -z (compress): Reduces data traffic over the network
• -n –itemize-changes: Displays a list of the changes to be made
• --delete: Deletes files in the target directory that no longer exist in the source
• --exclude: Excludes certain files or di­rec­to­ries from syn­chro­niza­tion
• --dry-run: Simulates the syn­chro­niza­tion process without actually trans­fer­ring files
• --progress: Shows the progress of the file transfer
• --partial: Files that have been partially trans­ferred remain in the target directory if the transfer is in­ter­rupt­ed. When the transfer is resumed, the file is continued from its last state

Examples of rsync syntax

The following examples of rsync syntax should make it easier to un­der­stand how the command is used. The following code example creates the directory dir1 including 100 empty test files and a second empty directory dir2:

$ cd ~
$ mkdir dir1
$ mkdir dir2
$ touch dir1/file{1..100}

The contents of dir1 can be syn­chro­nized on the same system with dir2 using the -r option:

$ rsync -r dir1/ dir2

Al­ter­na­tive­ly, the -a option can be used, which syn­chro­nizes re­cur­sive­ly and contains symbolic links, special device files, mod­i­fi­ca­tion times, groups, owners and au­tho­riza­tions:

$ rsync -a dir1/ dir2

Note: The slash (/) at the end of the source directory in an rsync command is important because it indicates that the contents of the directory should be syn­chro­nized, not the directory itself.

$ rsync -a dir1/ dir2

Here’s an example of the output:

sending incremental file list
./
file1
file10
file100
file11
file12
file13
file14
file15
file16
file17
file18
. . .

If the source directory doesn’t have a trailing slash, the source directory will be copied to the target directory:

$ rsync -a dir1 dir2

Here’s the output:

sending incremental file list
dir1/
dir1/file1
dir1/file10
dir1/file100
dir1/file11
dir1/file12
dir1/file13
dir1/file14
dir1/file15
dir1/file16
dir1/file17
dir1/file18
. . .

Using the slash at the end of the source directory ensures that the syn­chro­niza­tion process runs as expected and that the contents of the source directory end up in the correct target directory.

How to syn­chro­nize rsync with a remote system

Syn­chro­niz­ing a remote system with rsync is usually not difficult, provided you have SSH access to the remote computer and have the necessary au­then­ti­ca­tion in­for­ma­tion. Rsync often uses SSH (Secure Shell) for secure com­mu­ni­ca­tion with remote systems. To use this tool, it has to be installed on both sides.

If SSH access between the two computers is verified, the dir1 folder can be syn­chro­nized on a remote computer. In this case, the actual directory needs to be trans­ferred, which is why the trailing slash has been omitted in the following command:

$ rsync -a ~/dir1 username@remote_host:destination_directory

If a directory is moved from a local system to a remote system, this is referred to as a push operation. In contrast, when a remote directory is syn­chro­nized with a local system, this is referred to as a pull operation. The syntax for this is as follows:

$ rsync -a username@remote_host:/home/username/dir1 place_to_sync_on_local_machine

What other options are there in rsync?

The standard behavior of rsync can be further adapted using the options below.

Trans­fer­ring non-com­pressed files with rsync

The network load when trans­fer­ring non-com­pressed files can be reduced using the -z option:

$ rsync -az source destination

Dis­play­ing progress and resuming in­ter­rupt­ed trans­mis­sions

With -P you can combine the options --progress and --partial. This gives you an overview of the progress of trans­mis­sions and also allows you to resume in­ter­rupt­ed trans­mis­sions at the same time:

$ rsync -azP source destination

Here’s the output:

sending incremental file list
./
file1
    0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=99/101)
file10
    0 100% 0.00kB/s 0:00:00 (xfer#2, to-check=98/101)
file100
    0 100% 0.00kB/s 0:00:00 (xfer#3, to-check=97/101)
file11
    0 100% 0.00kB/s 0:00:00 (xfer#4, to-check=96/101)
. . .

Execute the command again to obtain a shorter output. This allows rsync to determine whether changes have been made based on change times.

$ rsync -azP source destination

Here’s the output:

sending incremental file list
sent 818 bytes received 12 bytes 1660.00 bytes/sec
total size is 0 speedup is 0.00

Keep di­rec­to­ries syn­chro­nized with rsync

To ensure that two di­rec­to­ries are actually kept in sync, it’s necessary to delete files that have been removed from the source directory in the target directory. But rsync doesn’t remove files from the target directory au­to­mat­i­cal­ly. This can be modified with the --delete option. However, it’s important to use this option with caution since it deletes files in the target directory that no longer exist in the source.

Before using this option, you should use the --dry-run option. This will allow you to perform a sim­u­la­tion of the syn­chro­niza­tion process without deleting any actual files. That way you can ensure that only the desired changes are made without ac­ci­den­tal­ly losing important data:

$ rsync -a --delete source destination

Exclude files and di­rec­to­ries from syn­chro­niza­tion

In rsync, you can use the --exclude option to exclude certain files and di­rec­to­ries from syn­chro­niza­tion. This is useful if, for example, you don’t want to syn­chro­nize temporary files, log files or other content.

$ rsync -a --exclude=pattern_to_exclude source destination

If you’ve specified a pattern for excluding files, you can use the --include= option to overwrite this exclusion for certain files that match a different pattern.

$ rsync -a --exclude=pattern_to_exclude --include=pattern_to_include source destination

Save backups with rsync

The --backup option allows you to save backups of important files. It can be used in con­junc­tion with the --backup-dir option to specify the directory where the backup files should be saved:

$ rsync -a --delete --backup --backup-dir=/path/to/backups /path/to/source destination

You can find a detailed overview of the various backup scenarios in our article about server backups with rsync.