Should you backup the cloud? I certainly would! In fact, I already maintain a copy of all my online resources within a media platform I control.
While I appreciate the redundancy efforts and availability promises of each and every provider, I remain resolute that "best practice" dictates I must satisfy my own disaster recovery requirements, with a secure copy of my data on a 3rd party storage technology.
Equally, many Managed Services Provider's (MSP) confront an abundance of cloud technologies that they need to implement solutions against. Designing solutions for each and every cloud storage becomes cumbersome.
Is there a tool that can connect to many disparate cloud storage vendors? there sure is... Rclone.
Rclone supports various cloud storage providers, simplifies data access and aspires to provide a unified interface to enable typical file-level operations in a familiar Linux style command-line interface (CLI).
Sample storage providers and protocols:
- Amazon Drive
- Amazon S3 (and compatible)
- Backblaze B2
- Box
- Citrix ShareFile
- DigitalOcean Spaces
- Dropbox
- FTP
- Google Cloud Storage
- Google Drive
- Google Photos
- HTTP
- Microsoft Azure Blob Storage
- Microsoft OneDrive
- Openstack Swift / Rackspace Cloudfiles / Memset Memstore
- SFTP
- The local filesystem
From an MSPs perspective a customer's environment is often a complex mix of cloud and storage technologies. Our partners and customers leverage a LucidLink Filespace for many use cases and workflows from collaboration, file server replacement, active archive to backup and recovery.
In this article, we will cover off a familiar use case of consolidating data out of a common cloud provider into our distributed filesystem which can be accessed virtually anywhere, by any application.
Our Filespace will become a convenient mount-point for ensuring the data hosted within any given provider is universally available for peace of mind.
To round out the concept further: a Filespace could be hosted within Wasabi, to backup a Dropbox account to low-cost object storage, as a 3rd party provider.
Should for any reason, your Dropbox became inaccessible, you have instant access to your data, off your Filespace which given our performance benefits, through unique data streaming vs. sync and share, you may choose to leave it there!
Migration is simplified too. Say you wish to move from any of the typical cloud providers to a Filespace, you can seamlessly develop a process to enact those aspirations.
Let's step you through our real-world example:
Download Rclone for Windows, macOS or Linux https://rclone.org/downloads/
Get familiar with Rclone commands Usage Documentation
Usage: rclone [flags] rclone [command] Available Commands: about Get quota information from the remote. authorize Remote authorization. cachestats Print cache stats for a remote cat Concatenates any files and sends them to stdout. check Checks the files in the source and destination match. cleanup Clean up the remote if possible config Enter an interactive configuration session. copy Copy files from source to dest, skipping already copied copyto Copy files from source to dest, skipping already copied copyurl Copy url content to dest. cryptcheck Cryptcheck checks the integrity of a crypted remote. cryptdecode Cryptdecode returns unencrypted file names. dbhashsum Produces a Dropbox hash file for all the objects in the path. dedupe Interactively find duplicate files and delete/rename them. delete Remove the contents of path. deletefile Remove a single file from remote. genautocomplete Output completion script for a given shell. gendocs Output markdown docs for rclone to the directory supplied. hashsum Produces an hashsum file for all the objects in the path. help Show help for rclone commands, flags and backends. link Generate public link to file/folder. listremotes List all the remotes in the config file. ls List the objects in the path with size and path. lsd List all directories/containers/buckets in the path. lsf List directories and objects in remote:path formatted for parsing lsjson List directories and objects in the path in JSON format. lsl List the objects in path with modification time, size and path. md5sum Produces an md5sum file for all the objects in the path. mkdir Make the path if it doesn't already exist. mount Mount the remote as file system on a mountpoint. move Move files from source to dest. moveto Move file or directory from source to dest. ncdu Explore a remote with a text based user interface. obscure Obscure password for use in the rclone.conf purge Remove the path and all of its contents. rc Run a command against a running rclone. rcat Copies standard input to file on remote. rcd Run rclone listening to remote control commands only. rmdir Remove the path if empty. rmdirs Remove empty directories under the path. serve Serve a remote over a protocol. settier Changes storage class/tier of objects in remote. sha1sum Produces an sha1sum file for all the objects in the path. size Prints the total size and number of objects in remote:path. sync Make source and dest identical, modifying destination only. touch Create new file or change file modification time. tree List the contents of the remote in a tree like fashion. version Show the version number. Use "rclone [command] --help" for more information about a command. Use "rclone help flags" for to see the global flags. Use "rclone help backends" for a list of supported services.
Rclone includes a user interface (UI) to configure and access your remote storage providers.
Launch UI
rclone rcd --rc-web-gui
Let's use the UI to configure a GoogleDrive storage provider profile. Within the Dashboard, select "Configs"
Choose "New Config"
Provide a profile name and choose your storage provider
Leave the next step default with no "Advanced Options"
Select "Create Config"
Provide and authorize your storage account. select "Allow"
You'll receive a "Success!" notification.
Return to the Dashboard and you'll have your storage account within your configs
That's it, you can now access your GoogleDrive and apply Rclone usage commands against your remote storage provider.
...
Let's now perform a CLI based configuration of a OneDrive profile, followed by a typical backup (synchronization) to a LucidLink Filespace mounted eg. L:
Connect to your Filespace and configure your mount-point according to your preference or Install as a Service.
Microsoft Windows [Version 10.0.14393] (c) 2016 Microsoft Corporation. All rights reserved. C:\Users\Administrator>lucid unmount Filespace umounted. C:\Users\Administrator>lucid mount L: Filespace mounted on L: C:\Users\Administrator>L:
The initial setup for OneDrive involves getting a token from Microsoft which you need to do in your browser rclone config walks you through it.
Configure Rclone
L:\>rclone config 2020/01/27 22:43:28 NOTICE: Config file "C:\\Users\\Administrator\\.config\\rclone\\rclone.conf" not found - using defaults No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n
Provide a remote profile name
name> OneDrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose your remote storage provider: select 22 for OneDrive
Choose a number from below, or type in your own value 1 / 1Fichier \ "fichier" 2 / Alias for an existing remote \ "alias" 3 / Amazon Drive \ "amazon cloud drive" 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) \ "s3" 5 / Backblaze B2 \ "b2" 6 / Box \ "box" 7 / Cache a remote \ "cache" 8 / Citrix Sharefile \ "sharefile" 9 / Dropbox \ "dropbox" 10 / Encrypt/Decrypt a remote \ "crypt" 11 / FTP Connection \ "ftp" 12 / Google Cloud Storage (this is not Google Drive) \ "google cloud storage" 13 / Google Drive \ "drive" 14 / Google Photos \ "google photos" 15 / Hubic \ "hubic" 16 / JottaCloud \ "jottacloud" 17 / Koofr \ "koofr" 18 / Local Disk \ "local" 19 / Mail.ru Cloud \ "mailru" 20 / Mega \ "mega" 21 / Microsoft Azure Blob Storage \ "azureblob" 22 / Microsoft OneDrive \ "onedrive" 23 / OpenDrive \ "opendrive" 24 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) \ "swift" 25 / Pcloud \ "pcloud" 26 / Put.io \ "putio" 27 / QingCloud Object Storage \ "qingstor" 28 / SSH/SFTP Connection \ "sftp" 29 / Transparently chunk/split large files \ "chunker" 30 / Union merges the contents of several remotes \ "union" 31 / Webdav \ "webdav" 32 / Yandex Disk \ "yandex" 33 / http Connection \ "http" 34 / premiumize.me \ "premiumizeme" Storage> 22
** See help for onedrive backend at: https://rclone.org/onedrive/ **
Leave the next options default ("") and ignore advanced config
Microsoft App Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Microsoft App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Auto configure remote profile
Remote config Use auto config? * Say Y if not sure * Say N if you are working on a remote or headless machine y) Yes n) No y/n> y
Rclone should automatically launch your browser with your authorization code. login and provide Rclone authorization
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=H_We5S4DKvQyHxabcdefg Log in and authorize rclone for access Waiting for code... Got code
Select your remote provider storage type
Choose a number from below, or type in an existing value 1 / OneDrive Personal or Business \ "onedrive" 2 / Root Sharepoint site \ "sharepoint" 3 / Type in driveID \ "driveid" 4 / Type in SiteID \ "siteid" 5 / Search a Sharepoint site \ "search" Your choice> 1
Select your storage account
Found 1 drives, please select the one you want to use: 0: (personal) id=Edited+12345myuniqueidnumber Chose drive to use:> 0 Found drive 'root' of type 'personal', URL: https://onedrive.live.com/?cid=Edited+12345myuniqueidnumber Is that okay? y) Yes n) No y/n> y
Confirm your Rclone remote profile is correct
--------------------
[OneDrive]
type = onedrive
token = {"access_token":"Edited+12345myaccesstokenuniqueid","expiry":"2020-01-27T23:44:15.9925604-08:00"}
drive_id = Edited+12345myuniqueidnumber
drive_type = personal
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
You've now configured your OneDrive Rclone remote profile.
Current remotes: Name Type ==== ==== OneDrive onedrive e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q
Synchronize your OneDrive with your Filespace
rclone sync - synchronizes the source to the destination, changing the destination only. Ignores unchanged files, by size and modification time or MD5SUM. The destination is updated to match the source, including deleting files if necessary to achieve a mirror clone.
Paths are specified as remote:path
Paths may be as deep as required, eg remote:directory/subdirectory
Paths are based on "remote" = profile "name" specified during configuration
L:\>md OneDrive L:\>rclone sync OneDrive: l:\OneDrive -P
In our example: we will sync OneDrive root path along with all recursive content with our Filespace, under our newly created OneDrive folder
Completed.
Each storage provider or protocol offers different performance results.
Backing up cloud storage providers, or providing enterprise migration services is easily achieved with a combination of LucidLink and Rclone.
You can simply interface with many disparate cloud storage vendors and secure your disaster recovery strategies in the 3rd party object storage providers of your choice.
A Filespace secures your data through our end-to-end encryption, therefore only you have access.
And streams data efficiently from object storage providing immediate access to interact with your data, across geographically separate operating environments as if the data were local.