MinIO distributed mode lets you pool multiple servers and drives into a clustered object store. Data is distributed across several nodes, can withstand node, multiple drive failures and provide data protection with aggregate performance.
Distributed mode creates a highly-available object storage system cluster. Our environment comprises of 4x nodes each with 4x drives which should be provisioned behind a load balancer for optimum performance and availability.
example, you can have 2 nodes with 4 drives each, 4 nodes with 4 drives each, 8 nodes with 2 drives each, 32 servers with 64 drives each and so on.
Configure name resolutions across hosts and ensure each host can resolve all other hosts within the cluster:
sudo nano /etc/hostname node<x> sudo nano /etc/hosts <ipaddress> node<x>
Prepare available drives on all hosts:
sudo mkfs -t ext4 /dev/sd<x> sudo mkdir /data<x> sudo mount /dev/sd<x> /data<x>
Download MinIO Server and Client:
sudo wget -O /usr/local/bin/minio https://dl.min.io/server/minio/release/linux-amd64/minio sudo chmod +x /usr/local/bin/minio sudo wget -O /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc sudo chmod +x /usr/local/bin/mc
Launch your cluster by running MinIO on each host referencing all hosts and disk mount resources:
sudo su export MINIO_ROOT_USER=abc123 export MINIO_ROOT_PASSWORD=secret456xyz minio server http://node1/data1 http://node1/data2 http://node1/data3 http://node1/data4 http://node2/data1 http://node2/data2 http://node2/data3 http://node2/data4 http://node3/data1 http://node3/data2 http://node3/data3 http://node3/data4 http://node4/data1 http://node4/data2 http://node4/data3 http://node4/data4
Quick launch your cluster:
minio server http://node{1...4}/data{1...4}
NOTE: Always use ellipses syntax {1...n} (3 dots!) for optimal erasure-code distribution
Implement systemd MinIO service to survive reboots with MinIO environment variables provided within the service via a configuration file to facilitate access-key and secret-key on all cluster nodes:
ensure your EnvironmentFile= and ExecStart= resources match your binary, path and environment file locations along with appropriate User= permissions for adequate system security
sudo nano /etc/minioenvironment MINIO_ROOT_USER=abc123 MINIO_ROOT_PASSWORD=secret456xyz sudo nano /etc/systemd/system/MinIO.service [Unit] Description=MinIO Requires=network-online.target After=network-online.target [Service] Type=simple ExecStartPre=/bin/sleep 60 RemainAfterExit=yes EnvironmentFile=/etc/minioenvironment ExecStart=/usr/local/bin/minio server http://node{1...3}/data{1...4} Restart=on-abort [Install] WantedBy=multi-user.target
Setup MinIO service within systemd:
sudo systemctl enable MinIO sudo systemctl start MinIO sudo systemctl status MinIO
Your object storage cluster is now operational. You can connect to your cluster through any node with your environment variable access-key and secret-key's via its URL http://node<x>:9000.
It's recommended you provision a load balancer in-front of your deployment, pointing your load balancer to each node within your cluster you would connect to your cluster with http://loadbalancerIP:9000. See https://www.nginx.com/blog/enterprise-grade-cloud-storage-nginx-plus-minio/
Please follow our Getting Started Guide to create a Filespace selecting "Other Cloud" and provide your load balancer or any node within the cluster directly based URL http://endpoint:port
Distributed MinIO has a per cluster limit of minimum of 2 and maximum of 32 servers. As your requirements grow you can expand by federating multiple clusters https://docs.minio.io/docs/minio-federation-quickstart-guide.html
Administration and monitoring of your MinIO distributed cluster comes courtesy of MinIO Client. Use the admin sub-command to perform administrative tasks on your cluster.
Common commands are listed below with their correct syntax against our cluster example. Further documentation can be sourced from MinIO's Admin Complete Guide.
Configure MinIO Client:
mc config host add node<x> http://node<x>:9000 abc123 secret456xyz
Show server/cluster console logs:
mc admin console node<x>
Display server/cluster information:
mc admin info server node<x>
Server http connection trace:
mc admin trace node<x>
Heal disks, buckets, objects
mc admin heal -r node<x> mc admin heal -r node<x>/bucket
Show cluster locks
$ mc admin top locks node<x>
MinIO is an extremely scalable and flexible object store, light enough to run in the most simplest environment, yet architected to satisfy enterprise applications.
Checkout the MinIO Overview https://min.io/product/overview Quickstart Guide https://docs.min.io/docs/minio-quickstart-guide.html for more information.