Migrating data to your Filespace needs to be achieved through the LucidLink client application. All data must traverse through our client to reap the unique benefits of our data streaming. You have many options to perform the transfer however the goal is to select the most efficient method. 

Each operating system has appropriate tools built-in to perform efficient, resumable file copies. In this article, we will provide some example concepts for Windows, Linux and macOS operating environments. 

We will leverage Robocopy and Rsync in our examples and provide additional guidance for segmenting your data to enable you to prioritize the transfer of your most important, active data. 

Estimate your transfer speed based using a bandwidth calculator. You'll require a speed test to determine your upload Mbps and the size of your data.

Please note data transfers may differ depending on data types. LucidLink compresses data for upload therefore some data types like documents compress reasonably well and consume less upload bandwidth. Transfer calculation estimates don't take into account the benefits of our compression.

It is important to size your Filespace file index and cache (root-path) before beginning any migration. Ensure that your cache is ample to act as an appropriate buffer to complement your upload rate:

  1. The default cache is 5GiB and can be increased to 1TiB
  2. Relocate your file index (metadata) and cache to a suitable location

We refer to <source> logically as the data to be migrated, and <destination> the Filespace mount-point. Default LucidLink mount-points appear within the user profile or home directories, located within a Lucid subdirectory. You can change your mount-point to a different location to simplify migration.

Windows:

Robcopy copy all folders, including empty, with resume support, multiple (8 default) threads, no retry, no wait and outputting to log with no percentage progress

robocopy <source> <destination> /e /z /mt /r:0 /w:0 /log:<file.log> /np

A very similar Robocopy command excluding files older than 90 days. The concept here is if you have limited time and need to copy the last 3 months of active data, you can copy that data 1st 

robocopy <source> <destination> /e /z /mt /r:0 /w:0 /log:<file.log> /np /maxage:90

Following up after the initial 90-day exclusion copy, excluding older files to transfer the remainder of the data

robocopy <source> <destination> /e /z /mt /r:0 /w:0 /log:<file.log> /np /xo

Linux and macOS:

Rsync can perform all the same functionality with a slight nuance and by incorporating a couple of additional tools. 

A general Rsync of files between source and destination 

rsync -aAXvP <source> <destination> --log-file=<file.log>

Rsync of files 90-days old or less

rsync -aAXvP --files-from=<(find <source> -mtime -90 -type f -exec basename {} \;) <source> <destination> --log-file=<file.log>

Update destination files

rsync -aAXvP --update <source> <destination> --log-file=<file.log>

Ignore existing files

rsync -aAXvP --ignore-existing <source> <destination> --log-file=<file.log>
Should you be considering a migration over time and continuing to use the source, you might consider with caution --delete (Rsync) and or /mir (Robocopy) to ensure your destination maintains a true reflection of your source.

Cloud:

If you have data hosted within a cloud vendor that would benefit from the performance characteristics of a LucidLink Filespace you can easily migrate these via Rclone using a familiar Linux style command-line interface.

Online cloud storage vendors are limited to how they integrate into workflows and operating environments whereas a Filespace presents universally across operating environments as a convenient extension of your operating system.

Our migrating cloud storage providers KB provides guidance for data hosted within Google Drive, Dropbox, OneDrive etc. or object storage accounts of S3 compatible or blob

Equally, data within a LucidLink mount-point can also be accessed via S3 APIs for applications that require native API List, Put, Get, Delete through an article we put together in conjunction with MinIO which can also be used in building your own object storage cluster as a backend for our Filespaces. 

We assume in this article you have LucidLink client installed however if you have an operating environment that inhibits your ability to install software, and you can leverage the local area network: you may like to consider configuring an NFS export or SMB server share to facilitate your migration.