Using Rsync for Filespace backup (macOS)

Workflow Guide

Workflow Details

Prerequisites: An understanding of LucidLink Filespaces and macOS Terminal
Time required: 30 minutes

Introduction

Maintaining a copy of your critical data in a secure location is a key aspect of a robust disaster recovery strategy. While LucidLink Filespaces offer excellent redundancy and availability, backing up your data to an additional storage system ensures that you have multiple layers of protection.

Rsync is a fast, versatile tool for synchronizing files and directories between mount points. Using efficient delta transfers. Rsync preserves file attributes, supports incremental backups, and can resume interrupted transfers. Commonly used for backups and data migration, it operates locally or over secure remote connections.

The architecture requires the following components:

LucidLink Filespace
macOS system with internet connection and the LucidLink Client installed
Network Attached Storage (NAS) Device connected to macOS System.

Example usage

In the following example, a LucidLink Filespace serves as the source, while a mounted NAS volume acts as the destination. The rsync command will be used to copy data from the LucidLink Filespace to the NAS as part of a data backup plan. The recommended best practice is to mount the Filespace using a user account that has read-only permissions to the root level of the Filespace.

Here's the macOS Rsync command we'll be discussing:

rsync -aEvP --ignore-existing --stats --log-file=<logfile> <source>/ <destination>/

❗️The version of rsync that is included with recent macOS versions (as of Sonoma: version 2.6.9, protocol version 29) is different than the rsync version included with most Linux distributions. Some command options such as -E, --extended-attributes on the macOS version are different from the Linux version. When using rsync on macOS make sure you are using the correct command options and syntax for macOS. For a list of all macOS rsync options, consult man rsync.

What This Command Does

The command performs a synchronization operation between a source directory and a destination directory. On initial run it ensures that the contents of the destination directory mirror those of the source directory. The command also includes provisions for handling potential interruptions during the transfer process and logs the activity for monitoring purposes. Subsequent command runs will skip copying identical files that already exist on the destination and only copy incremental updates.

Explanation of Each Component

rsync: Initiates the rsync utility.
<source>/: The source directory where the files are copied from.
- Note: A trailing slash on the source avoids creating an additional directory level at the destination.
<destination>/: The destination directory where the files are copied to. If the directory does not exist on the destination, rsync will create it.

Switches and Options

Switch	Effect
-a	Archive mode, which preserves permissions, ownership, timestamps, and recursively copies subdirectories. The `-a` option is equivalent to `-rlptgoD` (recursive, links, permissions, times, group, owner, devices).
-E	Short form of `--extended-attributes`. Preserves extended file metadata.
-v	A single `-v` will give you information about what files are being transferred and a brief summary at the end. Two `-v` flags will give you information on what files are being skipped and slightly more information at the end.
-P	Same as `--partial` and `--progress options`. Shows progress during transfer and keeps partially transferred files.
--ignore-existing	Skips files that already exist in the destination directory, effectively excluding older files. Provides for incremental backups after an initial full backup.
--delete	(Optional) Deletes extraneous files from destination directories that do not exist on source. Useful for periodically purging files on the backup destination that have been deleted or moved from the source, especially as storage capacity on the destination becomes constrained.
--stats	Provides a summary of the transfer, including statistics on the number of files transferred, total bytes sent, and transfer speed.
--log-file=/path	Redirects log output to a specified file, allowing for monitoring and analysis of the backup process. Replace `/path` with a valid file path on the destination.

Rsync Workflow Considerations

No, you can use the same rsync command on subsequent days to back up new or modified data. Rsync is designed for incremental backups, meaning it only transfers the changes (new files or modifications) since the last synchronization.
Yes, but if the source is LucidLink, snapshots can help you recover.
A partially uploaded file will be copied, but the remaining data will be copied in the next backup run.
If a file or folder is renamed on the source, rsync will treat it as a new file or folder. This means that the renamed file or folder will be copied to the destination as a new item, and the original file or folder on the destination will remain unchanged unless explicitly deleted.
Unless the --delete option is used, the file will remain on the destination.
No, backup data should only be accessed if the primary storage is unavailable. Backup data should be tested during scheduled test periods without modifying it during backup runs.
Yes. A simple rsync configuration would be a host system with the LucidLink client installed and both the LucidLink Filespace and NAS volumes mounted for data transfers.
No, the provided rsync command does not include an option to delete files on the destination if they don't exist on the source. The command is set to only copy files and directories from the source to the destination, while ignoring files that already exist in the destination (--ignore-existing). It does not perform any deletions on the destination based on the contents of the source. If you want rsync to delete files on the destination that don't exist on the source, you would need to include the --delete option. However, use this option with caution as it will remove any files in the destination directory that are not present in the source directory.
Check the rsync log file (see next section) and compare the folder properties (see the following section).
The rsync command itself can handle a wide range of characters, including those that may not be compatible with Windows file systems. However, the compatibility with characters depends on the file systems and operating systems of both the source and destination. Here are some points to consider: Character Encoding: rsync supports various character encodings. Ensure both the source and destination systems use compatible encodings (e.g., UTF-8) to handle a wide range of characters properly.

File System Limitations:

Linux/macOS: These operating systems and their native file systems (e.g., ext4, HFS+, APFS) generally support a wide range of characters in filenames.

Windows: The Windows file system (e.g., NTFS, FAT32) has some restrictions on characters allowed in filenames. Characters like \ / : * ? " < > | are not allowed in Windows filenames.
Egress (data download) charges are included for LucidLink Advanced and Basic FIlespaces however a Custom Filespace may generate data transfer charges from your chosen cloud storage provider (e.g. AWS).

Checking the backup process status

Rsync Stats Log

The following is an example of the log-file output from a small backup job:

sending incremental file list
production/
production/01_Proxy.mp4
    16,419,442 100%  186.04MB/s    0:00:00 (xfr#1, to-chk=9/11)
production/BlueDesktop.png
     3,505,803 100%   12.11MB/s    0:00:00 (xfr#2, to-chk=8/11)
production/GreenDesktopClean.jpg
     2,712,381 100%    7.72MB/s    0:00:00 (xfr#3, to-chk=7/11)
production/Screenshot.png
    10,662,277 100%   25.17MB/s    0:00:00 (xfr#4, to-chk=6/11)
production/StarshipGroove.mov
   211,958,391 100%   75.57MB/s    0:00:02 (xfr#5, to-chk=5/11)
production/StarshipGroove2.mov
   211,958,391 100%   59.51MB/s    0:00:03 (xfr#6, to-chk=4/11)
production/Titanic with a Cat.mp4
    27,594,274 100%   32.73MB/s    0:00:00 (xfr#7, to-chk=3/11)

Number of files: 8 (reg: 7, dir: 1)
Number of created files: 8 (reg: 7, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 10
Total file size: 556,352,559 bytes
Total transferred file size: 556,352,559 bytes
Literal data: 556,352,559 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 556,489,243
Total bytes received: 210

sent 556,489,243 bytes  received 210 bytes  65,469,347.41 bytes/sec
total size is 556,352,559  speedup is 1.00

Comparing Folder Properties

You can right-click a folder and select Get Info to display the folder properties.

Size: The total amount of data contained within all the files in the folder, displayed in units such as bytes, kilobytes, megabytes, or gigabytes and the number of items.
Created: The timestamp indicating when the folder was originally created, showing when the folder first came into existence.
Modified: The timestamp indicating when the folder was last modified.

After the first backup process runs, comparing the source and backup properties can help you check if the backup was successful.

Size: Data size and number of items should match.

Comparing "Size" and is a quick and easy method to verify that a backup has captured all files and folders from the original location. "Size" ensures that the total data volume matches. This approach can quickly highlight major discrepancies, such as missing files or folders, or significantly different data sizes.

After additional backups, the properties might be different if any of the following happens:

A file is renamed on the source. This will trigger a copy of a new file to the destination (without deleting the original name file from the backup)
A file is deleted from the source (the file will not be deleted from the back up).

For more reliable verification, additional methods can be employed:

Checksum/Hash Comparisons: Generate and compare checksums (like MD5, SHA-256) for each file in both the original and backup locations. Identical checksums ensure that the files are exactly the same byte-for-byte.
File Comparison Tools: Use specialized software (e.g., WinMerge, Beyond Compare) to compare files and folders in detail. These tools can compare not only file content but also timestamps and attributes.
Automated Scripts: Use scripts or programs to automate the comparison process, ensuring consistency and thoroughness in large data sets.

These methods provide a higher degree of certainty that the backup is an exact replica of the original, ensuring data integrity and reliability in the event of a restore.