MinIO distributed 4 node 12 disk erasure coding, bitrot protected cluster

  • Updated

MinIO distributed mode lets you pool multiple servers and drives into a clustered object store. Data is distributed across several nodes, can withstand node, multiple drive failures and provide data protection with aggregate performance.

Distributed mode creates a highly-available object storage system cluster. Our environment comprises of 4x nodes each with 4x drives which should be provisioned behind a load balancer for optimum performance and availability.

example, you can have 2 nodes with 4 drives each, 4 nodes with 4 drives each, 8 nodes with 2 drives each, 32 servers with 64 drives each and so on.

Configure name resolutions across hosts and ensure each host can resolve all other hosts within the cluster:

sudo nano /etc/hostname

sudo nano /etc/hosts
<ipaddress> node<x>

Prepare available drives on all hosts:

sudo mkfs -t ext4 /dev/sd<x>
sudo mkdir /data<x>
sudo mount /dev/sd<x> /data<x>

Download MinIO Server and Client:

sudo wget -O /usr/local/bin/minio
sudo chmod +x /usr/local/bin/minio

sudo wget -O /usr/local/bin/mc
sudo chmod +x /usr/local/bin/mc

Launch your cluster by running MinIO on each host referencing all hosts and disk mount resources:

sudo su
export MINIO_ROOT_USER=abc123
export MINIO_ROOT_PASSWORD=secret456xyz
minio server http://node1/data1 http://node1/data2 http://node1/data3 http://node1/data4 http://node2/data1 http://node2/data2 http://node2/data3 http://node2/data4 http://node3/data1 http://node3/data2 http://node3/data3 http://node3/data4 http://node4/data1 http://node4/data2 http://node4/data3 http://node4/data4

Quick launch your cluster:

minio server http://node{1...4}/data{1...4}
NOTE: Always use ellipses syntax {1...n} (3 dots!) for optimal erasure-code distribution

Implement systemd MinIO service to survive reboots with MinIO environment variables provided within the service via a configuration file to facilitate access-key and secret-key on all cluster nodes:

ensure your EnvironmentFile= and ExecStart= resources match your binary, path and environment file locations along with appropriate User= permissions for adequate system security
sudo nano /etc/minioenvironment

sudo nano /etc/systemd/system/MinIO.service

ExecStartPre=/bin/sleep 60
ExecStart=/usr/local/bin/minio server http://node{1...3}/data{1...4}


Setup MinIO service within systemd:

sudo systemctl enable MinIO
sudo systemctl start MinIO
sudo systemctl status MinIO

Your object storage cluster is now operational. You can connect to your cluster through any node with your environment variable access-key and secret-key's via its URL http://node<x>:9000.

It's recommended you provision a load balancer in-front of your deployment, pointing your load balancer to each node within your cluster you would connect to your cluster with http://loadbalancerIP:9000. See

Please follow our Getting Started Guide to create a Filespace selecting "Other Cloud" and provide your load balancer or any node within the cluster directly based URL http://endpoint:port

Distributed MinIO has a per cluster limit of minimum of 2 and maximum of 32 servers. As your requirements grow you can expand by federating multiple clusters

Administration and monitoring of your MinIO distributed cluster comes courtesy of MinIO Client. Use the admin sub-command to perform administrative tasks on your cluster.

Common commands are listed below with their correct syntax against our cluster example. Further documentation can be sourced from MinIO's Admin Complete Guide.

Configure MinIO Client:

mc config host add node<x> http://node<x>:9000 abc123 secret456xyz

Show server/cluster console logs:

mc admin console node<x>

Display server/cluster information:

mc admin info server node<x>

Server http connection trace:

mc admin trace node<x>

Heal disks, buckets, objects

mc admin heal -r node<x>
mc admin heal -r node<x>/bucket

Show cluster locks

$ mc admin top locks node<x>

MinIO is an extremely scalable and flexible object store, light enough to run in the most simplest environment, yet architected to satisfy enterprise applications. 

Checkout the MinIO Overview Quickstart Guide for more information.


Was this article helpful?

0 out of 0 found this helpful