S3Proxy https://github.com/gaul/s3proxy provides a convenient way to translate object storage providers brokering S3 APIs between Azure, Backblaze, Google Cloud Storage and/or a filesystem mount-point.

In this KB we will concentrate on providing S3 Compatible API access into your Filespace. "You may run multiple S3Proxy compute instances to the same Filespace to provide increased performance as aggregate distributed access".

A Filespace utilizes the object store in a unique manner that enables random read/write streaming of your data on demand. This unique data layout breaks the data up into blocks. Each LucidLink client liaises with local metadata and draws only the blocks required, as requested. 

There are certain situations where applications cannot natively access operating system filesystems, as they require S3 APIs to access their data. These applications may need to collaborate with data within a Filespace therefore in order to achieve this we must overlay S3 APIs over the Filespace mount-point via a proxy or gateway. 

Download the latest S3Proxy release https://github.com/gaul/s3proxy/releases 

S3Proxy is Java https://www.java.com/en/download/manual.jsp based (and requires Java 11 or newer) to run across Linux, Windows, macOS likewise is our LucidLink client and can be setup in unison across platforms.

Link and mount your Filespace as a mount-point accessible within your filesystem. In this KB we will provide Windows and Linux example S3Proxy configurations.

Our environments we will have our Filespace mounted within paths that allow S3 APIs to treat the folder structure in a manner that emulates buckets.

Windows

lucid mount d:\s3proxy\filespace

Linux

lucid mount /media/s3proxy/filespace

If you do not mount in this fashion, you will see folders as prefixes however root path files will not be visible as they will remain outside the S3 bucket conventions. 

S3Proxy requires a configuration file s3proxy.conf. Our Linux file we will specify no S3 credentials whereas our Windows we will provide an access-key and secret-key authorization method.

We will utilize the loopback address on port 8080 although you can choose another IP and configure port forwarding to provide external access.  

Linux

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/media/s3proxy

Windows

s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=12345abcde
s3proxy.credential=uvxyz67890
jclouds.provider=filesystem
jclouds.filesystem.basedir=d:

Launch S3Proxy against your configuration.

Linux

chmod +x s3proxy
s3proxy --properties s3proxy.conf

If you do not have Java installed apt install openjdk-18-jre-headless

Windows

java -jar d:\s3proxy\s3proxy --properties d:\s3proxy\s3proxy.conf

Once S3Proxy is up and running you now have S3 compatible API access into your Filespace! Issue your conventional APIs as you would against a traditional S3 compatible object storage endpoint. 

We will connect with AWS CLI https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Our Windows AWS CLI credentials file will require our access-key and secret-key

[default]
aws_access_key_id = 12345abcde
aws_secret_access_key = uvxyz67890

Linux based on our configuration file /filespace will be the bucket.

Windows based on our mounted and running location our bucket will be /s3proxy and our Filespace a /filespace prefix.

Basic AWS CLI S3 operations copy cp file to and from the Filespace, delete rm a file from the Filespace.

Applications that are natively written to liaise directly with object storage in Puts, Gets and S3 Compatible API based Deletes, can now co-exist with their filesystem based counterparts.

Interacting with Filespace datasets, via our highly scalable, high performant distributed architecture in conjunction with load balanced proxy nodes provides seamless cross platform access to your data.

Feel free to ask our support team for assistance via a ticket.