Summary

LucidLink’s Audit Trail feature creates Newline Delimited JSON (NDJSON) formatted log files that record filesystem operations on each client system. Although admins can inspect these logs directly, a better user experience is achieved by importing the log data into a searchable database accessed through a web UI.

This article demonstrates a containerized setup of integrated services that:

1. Collect and format audit trail log data.
2. Index the log data in a database.
3. Provide a web UI service for searching and interacting with the indexed data.

We will use the following tools:

• fluent-bit for log shipping.
• OpenSearch for the database index.
• OpenSearch Dashboards for the web UI.

This example illustrates a basic local host installation with minimal security. Advanced security features for public network production deployments are beyond the scope of this article.

 Refer to the OpenSearch project website for more information about security topics and production deployment recommendations.

Docker and Docker Compose will simplify the configuration and deployment of these services on a local host system.

Prerequisites

• LucidLink client installed and filespace connected with administrator user
  • Administrator user account needed for access to /.lucid_audit directory
  • Consult this KB article on enabling the Audit Trail feature.
• Docker and Docker Compose installed
  • Docker Desktop provides easy installation of required binaries
• Docker-compose YAML and configuration files (see .zip attachment below):

.env
docker-compose.yml
Dockerfile-fluent-bit
fs-audit-trail.yaml
json-parser.conf

Important host system settings when running Docker: It is recommended to set vm.max_map_count to at least 262144 when using OpenSearch with Docker. Please refer to this guide with instructions for making this system setting change on the OS of your host system.

Instructions

Once you have Docker Desktop installed and the Docker runtime is running, download the collection of deployment files and unzip into a directory on your system. For this example, we will collect the files in a directory called fs-audit-trail.

├── fs-audit-trail
│   ├── Dockerfile-fluent-bit
│   ├── docker-compose.yml
│   ├── fs-audit-trail.yaml
│   └── json-parser.conf

The .env file is used for setting two environment variables for your deployment: the OpenSearch admin password and the local mount point of the LucidLink filespace.

Using a text editor replace the example values with your admin password and the mount-point of your filespace.

If you are unsure of the filespace mount-point, refer to the LucidApp or run the CLI command lucid status.

The password must pass the following complexity requirements:

• Minimum 8 characters
• Must contain at least one uppercase letter [A–Z]
• One lowercase letter [a–z]
• One digit [0–9]
• One special character

OPENSEARCH_INITIAL_ADMIN_PASSWORD=<custom-admin-password>

FSMOUNTPOINT=<filespace-mount-point>

Once these values are set in the .env file, you are ready to launch your deployment. From your deployment directory, run the following:

docker compose -f docker-compose.yml up -d

The container deployment will take a minute or so to complete on the first launch as it downloads and runs the various container images.

 ✔ Network fs-audit-trail_opensearch-net     Created                                                                                                              
 ✔ Volume "fs-audit-trail_opensearch-data1"  Created                                                                                                              
 ✔ Volume "fs-audit-trail_opensearch-data2"  Created                                                                                                              
 ✔ Volume "fs-audit-trail_fluent-bit-db"     Created                                                                                                              
 ✔ Container opensearch-node1                Healthy                                                                                                             
 ✔ Container opensearch-node2                Healthy                                                                                                             
 ✔ Container opensearch-dashboards           Healthy                                                                                                             
 ✔ Container fluent-bit                      Started  

Once the Docker Compose output displays that all services are started and healthy, you are ready to connect to OpenSearch Dashboards.

In a web browser, navigate to the URL address http://localhost:5601. Once the login page loads enter ‘admin’ for the username and the OPENSEARCH_INITIAL_ADMIN_PASSWORD value you defined in the .env file.

After the Home page loads, click the main menu in the upper left and select Discover from the pull down menu.

From the Discover page, click the ‘Create index pattern’ button.

In the ‘Index pattern name’ field, type ‘audit-trail*’.

In Step 2, select ‘@timestamp’ from the pulldown and then click ‘Create index pattern’.

You now have an index for the filespace audit trail data and can select Discover again from the main menu in order to search and query audit trail data. Select the appropriate time range in order to see the relevant data for your search.

Admins executing search queries on log data will often need to filter the dataset by specific user operations. The following list comprises all of the possible values for operation:action:

FileCreate
FileDelete
FileRead
FileWritten
DirectoryCreate
DirectoryDelete
Move
SymlinkCreate
ExtendedAttributeSet
ExtendedAttributeDelete
Pin
Unpin

• fs-audit-trail.zip6 KB