Backup cloud storage to a Filespace

  • Updated

Should you backup the cloud? I certainly would! In fact, I already maintain a copy of all my online resources within a media platform I control.

While I appreciate the redundancy efforts and availability promises of each and every provider, I remain resolute that "best practice" dictates I must satisfy my own disaster recovery requirements, with a secure copy of my data on a 3rd party storage technology.

Equally, many Managed Services Provider's (MSP) confront an abundance of cloud technologies that they need to implement solutions against. Designing solutions for each and every cloud storage becomes cumbersome.

Is there a tool that can connect to many disparate cloud storage vendors? there sure is... Rclone.

Rclone supports various cloud storage providers, simplifies data access and aspires to provide a unified interface to enable typical file-level operations in a familiar Linux style command-line interface (CLI). 

Sample storage providers and protocols:

From an MSPs perspective a customer's environment is often a complex mix of cloud and storage technologies. Our partners and customers leverage a LucidLink Filespace for many use cases and workflows from collaboration, file server replacement, active archive to backup and recovery. 

In this article, we will cover off a familiar use case of consolidating data out of a common cloud provider into our distributed filesystem which can be accessed virtually anywhere, by any application.

Our Filespace will become a convenient mount-point for ensuring the data hosted within any given provider is universally available for peace of mind.

To round out the concept further: a Filespace could be hosted within Wasabi, to backup a Dropbox account to low-cost object storage, as a 3rd party provider. 

Should for any reason, your Dropbox became inaccessible, you have instant access to your data, off your Filespace which given our performance benefits, through unique data streaming vs. sync and share, you may choose to leave it there!

Migration is simplified too. Say you wish to move from any of the typical cloud providers to a Filespace, you can seamlessly develop a process to enact those aspirations. 

Let's step you through our real-world example:

Download Rclone for Windows, macOS or Linux 

Get familiar with Rclone commands Usage Documentation

  rclone [flags]
  rclone [command]

Available Commands:
  about           Get quota information from the remote.
  authorize       Remote authorization.
  cachestats      Print cache stats for a remote
  cat             Concatenates any files and sends them to stdout.
  check           Checks the files in the source and destination match.
  cleanup         Clean up the remote if possible
  config          Enter an interactive configuration session.
  copy            Copy files from source to dest, skipping already copied
  copyto          Copy files from source to dest, skipping already copied
  copyurl         Copy url content to dest.
  cryptcheck      Cryptcheck checks the integrity of a crypted remote.
  cryptdecode     Cryptdecode returns unencrypted file names.
  dbhashsum       Produces a Dropbox hash file for all the objects in the path.
  dedupe          Interactively find duplicate files and delete/rename them.
  delete          Remove the contents of path.
  deletefile      Remove a single file from remote.
  genautocomplete Output completion script for a given shell.
  gendocs         Output markdown docs for rclone to the directory supplied.
  hashsum         Produces an hashsum file for all the objects in the path.
  help            Show help for rclone commands, flags and backends.
  link            Generate public link to file/folder.
  listremotes     List all the remotes in the config file.
  ls              List the objects in the path with size and path.
  lsd             List all directories/containers/buckets in the path.
  lsf             List directories and objects in remote:path formatted for parsing
  lsjson          List directories and objects in the path in JSON format.
  lsl             List the objects in path with modification time, size and path.
  md5sum          Produces an md5sum file for all the objects in the path.
  mkdir           Make the path if it doesn't already exist.
  mount           Mount the remote as file system on a mountpoint.
  move            Move files from source to dest.
  moveto          Move file or directory from source to dest.
  ncdu            Explore a remote with a text based user interface.
  obscure         Obscure password for use in the rclone.conf
  purge           Remove the path and all of its contents.
  rc              Run a command against a running rclone.
  rcat            Copies standard input to file on remote.
  rcd             Run rclone listening to remote control commands only.
  rmdir           Remove the path if empty.
  rmdirs          Remove empty directories under the path.
  serve           Serve a remote over a protocol.
  settier         Changes storage class/tier of objects in remote.
  sha1sum         Produces an sha1sum file for all the objects in the path.
  size            Prints the total size and number of objects in remote:path.
  sync            Make source and dest identical, modifying destination only.
  touch           Create new file or change file modification time.
  tree            List the contents of the remote in a tree like fashion.
  version         Show the version number.

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

Rclone includes a user interface (UI) to configure and access your remote storage providers. 

Launch UI

rclone rcd --rc-web-gui

Let's use the UI to configure a GoogleDrive storage provider profile. Within the Dashboard, select "Configs"


Choose "New Config"


Provide a profile name and choose your storage provider


Leave the next step default with no "Advanced Options"


Select "Create Config"

Provide and authorize your storage account. select "Allow"


You'll receive a "Success!" notification. 


Return to the Dashboard and you'll have your storage account within your configs


That's it, you can now access your GoogleDrive and apply Rclone usage commands against your remote storage provider. 


Let's now perform a CLI based configuration of a OneDrive profile, followed by a typical backup (synchronization) to a LucidLink Filespace mounted eg. L: 

Connect to your Filespace and configure your mount-point according to your preference or Install as a Service.

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\Administrator>lucid unmount
Filespace umounted.

C:\Users\Administrator>lucid mount L:
Filespace mounted on L:


The initial setup for OneDrive involves getting a token from Microsoft which you need to do in your browser  rclone config walks you through it.

Configure Rclone

L:\>rclone config
2020/01/27 22:43:28 NOTICE: Config file "C:\\Users\\Administrator\\.config\\rclone\\rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Provide a remote profile name

name> OneDrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").

Choose your remote storage provider: select 22 for OneDrive

Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Dropbox
   \ "dropbox"
10 / Encrypt/Decrypt a remote
   \ "crypt"
11 / FTP Connection
   \ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
13 / Google Drive
   \ "drive"
14 / Google Photos
   \ "google photos"
15 / Hubic
   \ "hubic"
16 / JottaCloud
   \ "jottacloud"
17 / Koofr
   \ "koofr"
18 / Local Disk
   \ "local"
19 / Cloud
   \ "mailru"
20 / Mega
   \ "mega"
21 / Microsoft Azure Blob Storage
   \ "azureblob"
22 / Microsoft OneDrive
   \ "onedrive"
23 / OpenDrive
   \ "opendrive"
24 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
25 / Pcloud
   \ "pcloud"
26 /
   \ "putio"
27 / QingCloud Object Storage
   \ "qingstor"
28 / SSH/SFTP Connection
   \ "sftp"
29 / Transparently chunk/split large files
   \ "chunker"
30 / Union merges the contents of several remotes
   \ "union"
31 / Webdav
   \ "webdav"
32 / Yandex Disk
   \ "yandex"
33 / http Connection
   \ "http"
34 /
   \ "premiumizeme"
Storage> 22
** See help for onedrive backend at: **

Leave the next options default ("") and ignore advanced config

Microsoft App Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
Microsoft App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

Auto configure remote profile

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y

Rclone should automatically launch your browser with your authorization code. login and provide Rclone authorization

If your browser doesn't open automatically go to the following link:
Log in and authorize rclone for access
Waiting for code...
Got code


Select your remote provider storage type

Choose a number from below, or type in an existing value
 1 / OneDrive Personal or Business
   \ "onedrive"
 2 / Root Sharepoint site
   \ "sharepoint"
 3 / Type in driveID
   \ "driveid"
 4 / Type in SiteID
   \ "siteid"
 5 / Search a Sharepoint site
   \ "search"
Your choice> 1

Select your storage account

Found 1 drives, please select the one you want to use:
0:  (personal) id=Edited+12345myuniqueidnumber
Chose drive to use:> 0
Found drive 'root' of type 'personal', URL:
Is that okay?
y) Yes
n) No
y/n> y

Confirm your Rclone remote profile is correct 

type = onedrive
token = {"access_token":"Edited+12345myaccesstokenuniqueid","expiry":"2020-01-27T23:44:15.9925604-08:00"}
drive_id = Edited+12345myuniqueidnumber
drive_type = personal
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

You've now configured your OneDrive Rclone remote profile. 

Current remotes:

Name                 Type
====                 ====
OneDrive             onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Synchronize your OneDrive with your Filespace

rclone sync - synchronizes the source to the destination, changing the destination only. Ignores unchanged files, by size and modification time or MD5SUM. The destination is updated to match the source, including deleting files if necessary to achieve a mirror clone.

Paths are specified as remote:path
Paths may be as deep as required, eg remote:directory/subdirectory
Paths are based on "remote" = profile "name" specified during configuration
L:\>md OneDrive
L:\>rclone sync OneDrive: l:\OneDrive -P

In our example: we will sync OneDrive root path along with all recursive content with our Filespace, under our newly created OneDrive folderBC10.png


Each storage provider or protocol offers different performance results. 

Backing up cloud storage providers, or providing enterprise migration services is easily achieved with a combination of LucidLink and Rclone. 

You can simply interface with many disparate cloud storage vendors and secure your disaster recovery strategies in the 3rd party object storage providers of your choice.

A Filespace secures your data through our end-to-end encryption, therefore only you have access. 

And streams data efficiently from object storage providing immediate access to interact with your data, across geographically separate operating environments as if the data were local. 


Was this article helpful?

0 out of 0 found this helpful