LucidLink is a log-structured distributed file system for object storage. LucidLink writes to object storage using a unique data layout which allows random read access and data streaming. Our data layout is not a traditional 1:1 file-to-object relationship, objects are laid out to match a specific block size. One file could consist of multiple objects, depending on the file size and block size (default 256KiB). In some cases you may want to choose a different block size depending on your use case.
A Filespace leverages metadata coordination and synchronization to make requests on individual blocks within the object store. This dramatically improves performance as we don't need to retrieve data not yet required. The result is, when data is read, we don't always need to read the whole file and all objects. We can stream only the data required, when the application or user requires it.
Equally. When it comes to writing, we create new objects for fresh data, and garbage collect expired objects. Objects within the object store are encrypted and compressed at the client-side, therefore cannot be accessed from the object store directly.
If the default 256KiB block size doesn't suit your data type, you can initialize a Filespace with a preferred block size. Filespace block size is determined at initialization and cannot be changed. To initialize your Filespace, please follow the steps below, and do not hesitate to reach out to LucidLink Support for assistance.
Steps:
1. Create a new filespace.
2. Choose your plan. Use provided object storage your own object storage.
3. Give your filespace a name.
4. Choose a provider if using your own object storage
5. Choose a region if required.
If you wish to change the block size upon creation via the web portal:
6. Configure advanced settings.
7. Choose a bucket name. LucidLink will create one for you if you leave this empty.
8. Choose the block size you wish to have for your filespace.
Make sure that you are running the latest LucidLink Client version.
9. Create your filespace.
10. Ensure the LucidLink client is installed. At this point you can 'Initialize' after your filespace has been created, by clicking the link in the portal which will refer you to the LucidLink OS client. This may take a few minutes to become available.
If you wish to use the command line:
6. Create your filespace. Ensure the filespace is ready to initialize.
7. Ensure the LucidLink client is installed. At this point you can follow manual initialization steps, from the command line with the --block-size option.
8. Ensure the LucidLink client is running and prompting to connect. Please leave this open as this ensures that the LucidLink OS client daemon/service is running in the background.
If you are running an OS client without GUI you can launch a LucidLink OS client daemon from your command line with lucid daemon. Please ensure this remains running in the background.
9. Open command line or terminal, depending on the OS client you are using and enter the following command, making sure you supply the full Filespace.Domain name, Root password, Endpoint:Port, Credentials, and Block Size, followed by a Provider "text" value as a simple one word value to describe the vendor identity, ie. AWS.
lucid init-<s3/azure> --fs <filespace.domain> --password <rootpassword> --endpoint <ipadddress/url:port> --access-key <access-key> --secret-key <secret-key> --https --region <region> --block-size <KiB> --Provider <vendor>
Supported unit suffixes: K(iB), M(iB). Minimum 1KiB. Maximum 32MiB. Remember that the block size choice can really affect performance.
Please consult lucid help init-s3 or lucid help init-azure for additional initialization options such as --bucket or container-name. Should you want to see the full list of initialization parameters to troubleshoot please reach out via our LucidLink Support.
You should receive "daemon init request sent" for a successful initialization, although should the first attempt not complete successfully, additional information for guidance may be provided by the error output. Please update your combination of parameters and try again.
You can try multiple times, with parameters such as --http instead of --https for example, and in certain circumstances parameters such as region are case sensitive. Until your initialization is successful your Filespace will be waiting patiently in the portal for this step to complete.
Note that if you receive a "connection refused" error please open the LucidLink Client user interface or launch a command line or terminal daemon with lucid daemon. Also note that the endpoint URL does not require "http://" or "https://" like it did through the portal as we supply this manually through our command line options of --http or --https