Purpose of this article is to setup an Rclone transfer process on a Linux compute node that transfers data between a Filespace and a cloud storage provider continuously to a predetermined scheduled interval.
Rclone supports many of the most common cloud storage providers. Utilizing this process you can configure a virtual or physical machine, on-premises or within the cloud to perform the transfer.
Please note that depending on where your Filespace is hosted, where you locate your compute node, along with which destination cloud provider selected to maintain the backup, you many incur egress and/or Internet service provider transfer or network fees.
Employing `rclone sync` will allow you to purge the destination and keep your backup in lockstep with your Filespace. Your cloud provider might provide a versioning option (typically object storage) which enables you to maintain multiple versions of your data, in-line with your transfer schedule.
This process will complement our Filespace snapshots with granular data points in time within the backup, retaining data separately to the live Filespace content and snapshots.
Follow the following steps to prepare your compute node and configure your systemd services, transfer script, and systemd timer to trigger the backup process.
Install Lucid and Rclone
apt-get update; wget https://www.lucidlink.com/download/latest/lin64/stable/ -O lucidinstaller.deb; apt-get install ./lucidinstaller.deb -y; rm ./lucidinstaller.deb
sudo -v ; curl https://rclone.org/install.sh | sudo bash
Configure Rclone and setup your storage provide as preferred
rclone config
Create lucidlink.service systemd
nano /etc/systemd/system/lucidlink.service
[Unit]
Description=LucidLink Filespace Daemon
After=network-online.target
[Service]
Type=simple
#User=<user>
#Group=<group>
#WorkingDirectory=/home/<user>
ExecStart=/usr/bin/lucid daemon --fs <llfilespace> --user <lluser> --password <llpassword> --mount-point <llmount> --fuse-allow-other
ExecStop=/usr/bin/lucid exit
Restart=on-abort
[Install]
WantedBy=multi-user.target
Edit fuse.conf and uncomment (remove #)
sudo nano /etc/fuse.conf
`user_allow_other`
Reload systemd daemon
systemctl daemon-reload
Enable LucidLink daemon systemd service
systemctl enable lucidlink.service
Start LucidLink daemon systemd service
systemctl start lucidlink.service
Check LucidLink systemd service
systemctl status lucidlink.service
Rclone transfer script to `sync` (or `copy`) from source to destination.
nano /usr/local/bin/rclone-script.sh
#!/bin/bash
if [[ "`pidof -x $(basename $0) -o %PPID`" ]]; then
echo "rclone-script.sh is already running"
exit;
fi
rclone sync -v --fast-list --checkers 30 --transfers 30 <source> <destination>
Make script executable
chmod +x /usr/local/bin/rclone-script.sh
Create a systemd service
nano /etc/systemd/system/rclone.service
[Unit]
Description=Rclone Transfer Service
[Service]
Type=simple
ExecStart=/usr/local/bin/rclone-script.sh
Create Rclone systemd timer https://www.freedesktop.org/software/systemd/man/systemd.timer.html#
nano /etc/systemd/system/rclone.timer
[Unit]
Description=Rclone Transfer Timer
[Timer]
Unit=rclone.service
# Run 15 minutes after boot, since the timer must run at least once
# before OnUnitInactiveSec will trigger
OnBootSec=15m
# Run 15 minutes after rclone.service last finished
OnUnitInactiveSec=15m
# Run once when the timer is first started
OnActiveSec=1s
[Install]
WantedBy=timers.target
Reload systemd daemon
systemctl daemon-reload
Enable Rclone timer
systemctl enable rclone.timer
Start Rclone timer
systemctl start rclone.timer
Manually trigger Rclone transfer
systemctl start rclone.service
Reach out to LucidLink support should you require any assistance.
Kudos to https://codingnotions.com/fully-automated-backup-rclone/