Overview
In this article, we will walk you through best practices for the Lucid cache and the importance of selecting the right disk technology and working with particular filesystem formats of each operating system (OS) as per our system requirements article.
We strongly recommend Solid-state drive (SSD) or Non-Volatile Memory Express (NVMe) as ideal storage types for your cache location. Although hard drives (HDD) can be utilized they often do not yield the performance required for high transactional workloads.
Lucid requires high-performant storage for its cache operations. The Filespace data is streamed in a block-level manner, with data written and cached locally needing to be processed in a granular manner, comprised of lots of little disk operations.
It is important to choose a cache location that meets your application expectations and workflow requirements, it is for this reason we recommend, given the ubiquity of SSD/NVMe, that you utilize these disk types at a minimum.
Equally as important is the OS drive format. Each filesystem behaves differently. Some filesystems provide compatibility but compromise on performance. Selecting the correct filesystem format to complement your disk technology is imperative.
Most external drives (portable media) used in professional and consumer-grade electronics are pre-formatted as exFAT because of their multiplatform & multi-OS capabilities.
Configuration
Currently, the only option for changing the cache size and location is from the command line.
To change the cache size, please follow this article:
Change cache size from the CLI
You can read more about the cache settings here:
Adjust local cache settings in the command-line interface
Best practices
Use New Technology File System (NTFS) under Windows
NTFS has special handling for very small files storing them directly in the Master File Table.
NTFS drive is less likely to get corrupted as it can recover errors if found. For best performance please format your external SSD drive to NTFS before starting to use it for the Lucid cache.
Use Apple File System (APFS) under macOS
Apple File System is designed to avoid metadata corruption caused by system crashes. Instead of overwriting existing metadata records in place, it writes entirely new records, points to the new ones, and then releases the old ones, an approach known as redirect-on-write. This avoids corrupted records containing partially old and partially new data caused by crashes that may occur during an update.
Use Ext3/4 and above with Linux distributions
Ext3 was designed to include a journal, online file system growth, and HTree indexing for larger directories. Every newer Ext Filesystem is based on a journaling system that should be secure enough for the purpose of the Lucid cache.
Why exFAT is not a viable option
Due to the large allocation unit (cluster size), exFAT is inefficient if you produce lots of small files. The purpose of the Lucid cache is to split your data into small blocks which get compressed, encrypted, and streamed to your object store. The same happens when you read data from the endpoint. Your Lucid cache is constantly responding to application data requests and evicting older ones that require sufficient read-write operations.
exFAT is not a journaled system. A journaling file system is a file system that keeps track of changes not yet committed to the file system's main part by recording the goal of such changes in a data structure known as a "journal". In the event of a system crash or power failure, such file systems can be brought back online more quickly with a lower likelihood of becoming corrupted.
The difference in quality and durability between Consumer grade SSD vs Enterprise grade storage devices.
Choose the best connection interface Internal/External connection interfaces for your Lucid app cache
Please format your internal or external device with the proper disk format before you put data on it or start using it for the Lucid cache.
To change where the cache and metadata is stored, link to your filespace with the --root-path option:
lucid3 daemon --fs --user --password --mount-point --root-path