LucidLink is a log-structured distributed file system for object storage. LucidLink writes to object storage using a unique data layout which allows random read access and data streaming. Our data layout is not a traditional 1:1 file-to-object relationship, objects are laid out to match a specific block size. One file could consist of multiple objects, depending on the file size and block size (default 256KiB). In some cases you may want to choose a different block size depending on your use case.
A Filespace leverages metadata coordination and synchronization to make requests on individual blocks within the object store. This dramatically improves performance as we don't need to retrieve data not yet required. The result is, when data is read, we don't always need to read the whole file and all objects. We can stream only the data required, when the application or user requires it.
Equally. When it comes to writing, we create new objects for fresh data, and garbage collect expired objects. Objects within the object store are encrypted and compressed at the client-side, therefore cannot be accessed from the object store directly.
If the default 256KiB block size doesn't suit your data type, you can initialize a Filespace with a preferred block size. Filespace block size is determined at initialization and cannot be changed. To initialize your Filespace, please follow the steps below, and do not hesitate to reach out to LucidLink Support for assistance.
- Create a Filespace
- Define Filespace name
- Choose Your or LucidLink storage
- Select Your Cloud Provider and Region
- Specify a HTTP or HTTPS endpoint URL (including PORT if required)
- Optional: specify Region if required
- Optional: specify Bucket name if required
- Review and confirm, finally Create Filespace
At this point that the Filespace is being set up - this will take a minute or two, please be patient. Once setup is complete and the portal tile is waiting on Initialize Next you will proceed to initialization of your Filespace with your object storage credentials from the command line with the --block-size option.
Ensure that the LucidLink OS client is downloaded and installed on the machine performing the initialization.
At the point that OS client has been installed, please ensure it is open and is prompting to "connect", please leave this window open because in the background the LucidLink OS client daemon/service is successfully running.
Alternatively, if you are running an OS client without GUI (Linux) you can launch a LucidLink OS client daemon from your command line with "lucid daemon". Please ensure lucid daemon remains running in the background.
Open command line, terminal, depending on what OS client you are using and enter the following command, making sure to supply the full Filespace and Domain, Shared secret, Endpoint:Port, Credentials and Block size in required, followed by a Provider "text" as a simple 1 word vendor identity ie. AWS
lucid init-<s3/azure> --fs <filespace.domain> --password <rootpassword> --endpoint <ipadddress/url:port> --access-key <access-key> --secret-key <secret-key> --https --region <region> --block-size <KiB> --Provider <vendor>
Supported unit suffixes: K(iB), M(iB). Min 1KiB. Max 32MiB.
Please consult "lucid help init-s3" or "lucid help init-azure" for additional initialization options such as --bucket/container-name. Should you want to see the full list of initialization parameters to troubleshoot please reach out via our LucidLink Support.
You should receive "Daemon init request sent." for a successful initialization although should your initial attempt not complete successfully, additional information for guidance might be provided in the error output, please update your combination of parameters and try again.
You can try multiple times, with parameters such as --http instead of --https for example, and in certain circumstances parameters such as region are case sensitive. Until your initialization is successful your Filespace will be waiting patiently in the portal for this step to complete.
Note that if you receive a "connection refused" error please open the LucidLink Client user interface or launch a command line or terminal daemon with 'lucid daemon'. Also note that the endpoint URL does not require http:// or https:// like it did through the portal as we supply this manually through our command line options of --https or --http.