mirror of
https://github.com/pgsty/minio.git
synced 2026-08-09 15:53:28 +03:00
docs: rebrand the repository documentation, templates and dashboards
README, README_ZH, SECURITY, COMPLIANCE, CONTRIBUTING, NOTICE, code_of_conduct, the vulnerability and PR-etiquette documents, the GitHub issue and pull request templates, and the docs/ tree all present Silo as the product. The Grafana dashboards under docs/metrics/prometheus/grafana/ have their panel titles and descriptions rebranded while every minio_* query, label and expression is left alone, so existing alerts and recording rules keep matching. The distinction the review demanded is applied per hit rather than by search-and-replace: - Product and command text becomes Silo and silo: install and run instructions, systemd examples, compose services, download links, badges. - Protocol and interface text keeps MinIO: MINIO_* variables, minio_* metrics, x-minio-* headers, /minio/* routes, .minio.sys, arn:minio, and API field and error names. - Attribution keeps MinIO and gains the fork's own: the AGPL obligations, original copyright, CREDITS and NOTICE stay, with the modification notice added alongside rather than replacing them. - Historical and third-party references are left as facts, not rewritten for brand tidiness. README and README_ZH each carry an explicit non-affiliation notice, document the side-by-side package migration including the /etc/systemd/system/silo.service.d/10-legacy-user.conf drop-in for keeping a legacy UID/GID, and state that recursive chown is never performed. The trademark attribution uses the policy's approved "based on MinIO technology" wording, not the shortened form the policy rejects. github.com/pgsty/minio links are left in place and labelled transitional. The repository has not been renamed, and rewriting them now would produce documented URLs that 404 until the cutover; they change in the cutover commit together with the goreleaser release target, the OCI source label and the raw-content branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Feng Ruohang
+13
-13
@@ -1,18 +1,18 @@
|
||||
## MinIO configuration YAML
|
||||
## Silo configuration YAML
|
||||
|
||||
MinIO now supports starting the server arguments and configuration via a YAML configuration file. This YAML configuration describes everything that can be configured in a MinIO setup, such as '--address', '--console-address' and command line arguments for the MinIO server.
|
||||
Silo now supports starting the server arguments and configuration via a YAML configuration file. This YAML configuration describes everything that can be configured in a Silo setup, such as '--address', '--console-address' and command line arguments for the Silo server.
|
||||
|
||||
Historically everything to MinIO was provided via command arguments for the hostnames and the drives via an ellipses syntax such as `minio server http://host{1...4}/disk{1...4}` this requirement added an additional burden to have sequential hostnames for us to make sure that we can provide horizontal distribution, however we have come across situations where sometimes this is not feasible and there are no easier alternatives without modifying /etc/hosts on the host system as root user. Many times in airgapped deployments this is not allowed or requires audits and approvals.
|
||||
Historically everything to Silo was provided via command arguments for the hostnames and the drives via an ellipses syntax such as `silo server http://host{1...4}/disk{1...4}` this requirement added an additional burden to have sequential hostnames for us to make sure that we can provide horizontal distribution, however we have come across situations where sometimes this is not feasible and there are no easier alternatives without modifying /etc/hosts on the host system as root user. Many times in airgapped deployments this is not allowed or requires audits and approvals.
|
||||
|
||||
MinIO server configuration file allows users to provide topology that allows for heterogeneous hostnames, allowing MinIO to deployed in pre-existing environments without any further OS level configurations.
|
||||
Silo server configuration file allows users to provide topology that allows for heterogeneous hostnames, allowing Silo to deployed in pre-existing environments without any further OS level configurations.
|
||||
|
||||
### Usage
|
||||
|
||||
```
|
||||
minio server --config config.yaml
|
||||
silo server --config config.yaml
|
||||
```
|
||||
|
||||
Lets you start MinIO server with all inputs to start MinIO server provided via this configuration file, once the configuration file is provided all other pre-existing values on disk for configuration are overridden by the new values set in this configuration file.
|
||||
Lets you start Silo server with all inputs to start Silo server provided via this configuration file, once the configuration file is provided all other pre-existing values on disk for configuration are overridden by the new values set in this configuration file.
|
||||
|
||||
Following is an example YAML configuration structure.
|
||||
```yaml
|
||||
@@ -21,7 +21,7 @@ address: ":9000"
|
||||
rootUser: "minioadmin"
|
||||
rootPassword: "minioadmin"
|
||||
console-address: ":9001"
|
||||
certs-dir: "/home/user/.minio/certs/"
|
||||
certs-dir: "/home/user/.silo/certs/"
|
||||
pools: # Specify the nodes and drives with pools
|
||||
- args:
|
||||
- "https://server-example-pool1:9000/mnt/disk{1...4}/"
|
||||
@@ -36,10 +36,10 @@ pools: # Specify the nodes and drives with pools
|
||||
# more args
|
||||
|
||||
options:
|
||||
ftp: # settings for MinIO to act as an ftp server
|
||||
ftp: # settings for Silo to act as an ftp server
|
||||
address: ":8021"
|
||||
passive-port-range: "30000-40000"
|
||||
sftp: # settings for MinIO to act as an sftp server
|
||||
sftp: # settings for Silo to act as an sftp server
|
||||
address: ":8022"
|
||||
ssh-private-key: "/home/user/.ssh/id_rsa"
|
||||
```
|
||||
@@ -65,7 +65,7 @@ pools:
|
||||
- "https://server{1...2}-pool1:9000/mnt/disk{1...4}/"
|
||||
- "https://server3-pool1:9000/mnt/disk{1...4}/"
|
||||
- "https://server4-pool1:9000/mnt/disk{1...4}/"
|
||||
set-drive-count: 4 # Advanced option, must be used under guidance from MinIO team.
|
||||
set-drive-count: 4 # Advanced option, must be used under guidance from Silo team.
|
||||
```
|
||||
|
||||
### Things to know
|
||||
@@ -73,16 +73,16 @@ pools:
|
||||
- Fields such as `version` and `pools` are mandatory, however all other fields are optional.
|
||||
- Each pool expects a minimum of 2 nodes per pool, and unique non-repeating hosts for each argument.
|
||||
- Each pool expects each host in this pool has the same number of drives specified as any other host.
|
||||
- Mixing `local-path` and `distributed-path` is not allowed, doing so would cause MinIO to refuse starting the server.
|
||||
- Mixing `local-path` and `distributed-path` is not allowed, doing so would cause Silo to refuse starting the server.
|
||||
- Ellipses and bracket notation (e.g. `{1...10}`) are allowed.
|
||||
|
||||
> NOTE: MinIO environmental variables still take precedence over the `config.yaml` file, however `config.yaml` is preferred over MinIO internal config KV settings via `mc admin config set alias/ <sub-system>`.
|
||||
> NOTE: Silo environmental variables still take precedence over the `config.yaml` file, however `config.yaml` is preferred over Silo internal config KV settings via `mc admin config set alias/ <sub-system>`.
|
||||
|
||||
### TODO
|
||||
|
||||
In subsequent releases we are planning to extend this to provide things like
|
||||
|
||||
- Reload() of MinIO server arguments without fully restarting the process.
|
||||
- Reload() of Silo server arguments without fully restarting the process.
|
||||
|
||||
- Expanding 1 node at a time by automating the process of creating a new pool
|
||||
and decommissioning to provide a functionality that smaller deployments
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Decommissioning
|
||||
|
||||
Decommissiong is a mechanism in MinIO to drain older pools (usually with old hardware) and migrate the content from such pools to a newer pools (usually better hardware). Decommissioning spreads the data across all pools - for example, if you decommission `pool1`, all the data from `pool1` spreads across `pool2` and `pool3`.
|
||||
Decommissiong is a mechanism in Silo to drain older pools (usually with old hardware) and migrate the content from such pools to a newer pools (usually better hardware). Decommissioning spreads the data across all pools - for example, if you decommission `pool1`, all the data from `pool1` spreads across `pool2` and `pool3`.
|
||||
|
||||
## Features
|
||||
|
||||
@@ -92,7 +92,7 @@ If the decommission process fails for any reason, the status indicates failed.
|
||||
|
||||
## When decommission is 'Complete'
|
||||
|
||||
Once decommission is complete, it will be indicated with *Complete* status. *Complete* means that now you can now safely remove the first pool argument from the MinIO command line.
|
||||
Once decommission is complete, it will be indicated with *Complete* status. *Complete* means that now you can now safely remove the first pool argument from the Silo command line.
|
||||
|
||||
```
|
||||
λ mc admin decommission status alias/
|
||||
@@ -103,11 +103,11 @@ Once decommission is complete, it will be indicated with *Complete* status. *Co
|
||||
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
|
||||
```
|
||||
|
||||
- On baremetal setups, if you have `MINIO_VOLUMES="http://minio{1...2}/data{1...4} http://minio{3...4}/data{1...4}"`, you can remove the first argument `http://minio{1...2}/data{1...4}` to update your `MINIO_VOLUMES` setting, then restart all the servers in the setup in parallel using `systemctl restart minio`.
|
||||
- On baremetal setups, if you have `MINIO_VOLUMES="http://minio{1...2}/data{1...4} http://minio{3...4}/data{1...4}"`, you can remove the first argument `http://minio{1...2}/data{1...4}` to update your `MINIO_VOLUMES` setting, then restart all the servers in the setup in parallel using `systemctl restart silo`.
|
||||
|
||||
- On Kubernetes setups, the statefulset specification needs to be modified by changing the command line input for the MinIO container. Once the relevant changes are done, proceed to execute `kubectl apply -f statefulset.yaml`.
|
||||
- On Kubernetes setups, the statefulset specification needs to be modified by changing the command line input for the Silo container. Once the relevant changes are done, proceed to execute `kubectl apply -f statefulset.yaml`.
|
||||
|
||||
- On Operator based MinIO deployments, you need to modify the `tenant.yaml` specification and modify the `pools:` section from two entries to a single entry. After making relevant changes, proceed to execute `kubectl apply -f tenant.yaml`.
|
||||
- On Silo deployments managed by the MinIO Operator, you need to modify the `tenant.yaml` specification and change the `pools:` section from two entries to a single entry. After making the relevant changes, proceed to execute `kubectl apply -f tenant.yaml`.
|
||||
|
||||
> Without a 'Complete' status any 'Active' or 'Draining' pool(s) are not allowed to be removed once configured.
|
||||
|
||||
|
||||
+21
-21
@@ -1,17 +1,17 @@
|
||||
# Distributed Server Design Guide [](https://slack.min.io)
|
||||
# Distributed Server Design Guide
|
||||
|
||||
This document explains the design, architecture and advanced use cases of the MinIO distributed server.
|
||||
This document explains the design, architecture and advanced use cases of the Silo distributed server.
|
||||
|
||||
## Command-line
|
||||
|
||||
```
|
||||
NAME:
|
||||
minio server - start object storage server
|
||||
silo server - start object storage server
|
||||
|
||||
USAGE:
|
||||
minio server [FLAGS] DIR1 [DIR2..]
|
||||
minio server [FLAGS] DIR{1...64}
|
||||
minio server [FLAGS] DIR{1...64} DIR{65...128}
|
||||
silo server [FLAGS] DIR1 [DIR2..]
|
||||
silo server [FLAGS] DIR{1...64}
|
||||
silo server [FLAGS] DIR{1...64} DIR{65...128}
|
||||
|
||||
DIR:
|
||||
DIR points to a directory on a filesystem. When you want to combine
|
||||
@@ -26,22 +26,22 @@ DIR:
|
||||
Standalone erasure coded configuration with 4 sets with 16 drives each.
|
||||
|
||||
```
|
||||
minio server dir{1...64}
|
||||
silo server dir{1...64}
|
||||
```
|
||||
|
||||
Distributed erasure coded configuration with 64 sets with 16 drives each.
|
||||
|
||||
```
|
||||
minio server http://host{1...16}/export{1...64}
|
||||
silo server http://host{1...16}/export{1...64}
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
Expansion of ellipses and choice of erasure sets based on this expansion is an automated process in MinIO. Here are some of the details of our underlying erasure coding behavior.
|
||||
Expansion of ellipses and choice of erasure sets based on this expansion is an automated process in Silo. Here are some of the details of our underlying erasure coding behavior.
|
||||
|
||||
- Erasure coding used by MinIO is [Reed-Solomon](https://github.com/klauspost/reedsolomon) erasure coding scheme, which has a total shard maximum of 256 i.e 128 data and 128 parity. MinIO design goes beyond this limitation by doing some practical architecture choices.
|
||||
- Erasure coding used by Silo is [Reed-Solomon](https://github.com/klauspost/reedsolomon) erasure coding scheme, which has a total shard maximum of 256 i.e 128 data and 128 parity. Silo design goes beyond this limitation by doing some practical architecture choices.
|
||||
|
||||
- Erasure set is a single erasure coding unit within a MinIO deployment. An object is sharded within an erasure set. Erasure set size is automatically calculated based on the number of drives. MinIO supports unlimited number of drives but each erasure set can be up to 16 drives and a minimum of 2 drives.
|
||||
- Erasure set is a single erasure coding unit within a Silo deployment. An object is sharded within an erasure set. Erasure set size is automatically calculated based on the number of drives. Silo supports unlimited number of drives but each erasure set can be up to 16 drives and a minimum of 2 drives.
|
||||
|
||||
- We limited the number of drives to 16 for erasure set because, erasure code shards more than 16 can become chatty and do not have any performance advantages. Additionally since 16 drive erasure set gives you tolerance of 8 drives per object by default which is plenty in any practical scenario.
|
||||
|
||||
@@ -51,10 +51,10 @@ Expansion of ellipses and choice of erasure sets based on this expansion is an a
|
||||
|
||||
- *If total number of nodes are of odd number then GCD algorithm provides affinity towards odd number erasure sets to provide for uniform distribution across nodes*. This is to ensure that same number of drives are pariticipating in any erasure set. For example if you have 2 nodes with 180 drives then GCD is 15 but this would lead to uneven distribution, one of the nodes would participate more drives. To avoid this the affinity is given towards nodes which leads to next best GCD factor of 12 which provides uniform distribution.
|
||||
|
||||
- In this algorithm, we also make sure that we spread the drives out evenly. MinIO server expands ellipses passed as arguments. Here is a sample expansion to demonstrate the process.
|
||||
- In this algorithm, we also make sure that we spread the drives out evenly. Silo server expands ellipses passed as arguments. Here is a sample expansion to demonstrate the process.
|
||||
|
||||
```
|
||||
minio server http://host{1...2}/export{1...8}
|
||||
silo server http://host{1...2}/export{1...8}
|
||||
```
|
||||
|
||||
Expected expansion
|
||||
@@ -98,14 +98,14 @@ Input for the key is the object name specified in `PutObject()`, returns a uniqu
|
||||
|
||||
- Write and Read quorum are required to be satisfied only across the erasure set for an object. Healing is also done per object within the erasure set which contains the object.
|
||||
|
||||
- MinIO does erasure coding at the object level not at the volume level, unlike other object storage vendors. This allows applications to choose different storage class by setting `x-amz-storage-class=STANDARD/REDUCED_REDUNDANCY` for each object uploads so effectively utilizing the capacity of the cluster. Additionally these can also be enforced using IAM policies to make sure the client uploads with correct HTTP headers.
|
||||
- Silo does erasure coding at the object level not at the volume level, unlike other object storage vendors. This allows applications to choose different storage class by setting `x-amz-storage-class=STANDARD/REDUCED_REDUNDANCY` for each object uploads so effectively utilizing the capacity of the cluster. Additionally these can also be enforced using IAM policies to make sure the client uploads with correct HTTP headers.
|
||||
|
||||
- MinIO also supports expansion of existing clusters in server pools. Each pool is a self contained entity with same SLA's (read/write quorum) for each object as original cluster. By using the existing namespace for lookup validation MinIO ensures conflicting objects are not created. When no such object exists then MinIO simply uses the least used pool to place new objects.
|
||||
- Silo also supports expansion of existing clusters in server pools. Each pool is a self contained entity with same SLA's (read/write quorum) for each object as original cluster. By using the existing namespace for lookup validation Silo ensures conflicting objects are not created. When no such object exists then Silo simply uses the least used pool to place new objects.
|
||||
|
||||
### There are no limits on how many server pools can be combined
|
||||
|
||||
```
|
||||
minio server http://host{1...32}/export{1...32} http://host{1...12}/export{1...12}
|
||||
silo server http://host{1...32}/export{1...32} http://host{1...12}/export{1...12}
|
||||
```
|
||||
|
||||
In above example there are two server pools
|
||||
@@ -117,7 +117,7 @@ In above example there are two server pools
|
||||
|
||||
Refer to the sizing guide with details on the default parity count chosen for different erasure stripe sizes [here](https://github.com/pgsty/minio/blob/master/docs/distributed/SIZING.md)
|
||||
|
||||
MinIO places new objects in server pools based on proportionate free space, per pool. Following pseudo code demonstrates this behavior.
|
||||
Silo places new objects in server pools based on proportionate free space, per pool. Following pseudo code demonstrates this behavior.
|
||||
|
||||
```go
|
||||
func getAvailablePoolIdx(ctx context.Context) int {
|
||||
@@ -144,23 +144,23 @@ func getAvailablePoolIdx(ctx context.Context) int {
|
||||
Standalone erasure coded configuration with 4 sets with 16 drives each, which spawns drives across controllers.
|
||||
|
||||
```
|
||||
minio server /mnt/controller{1...4}/data{1...16}
|
||||
silo server /mnt/controller{1...4}/data{1...16}
|
||||
```
|
||||
|
||||
Standalone erasure coded configuration with 16 sets, 16 drives per set, across mounts and controllers.
|
||||
|
||||
```
|
||||
minio server /mnt{1...4}/controller{1...4}/data{1...16}
|
||||
silo server /mnt{1...4}/controller{1...4}/data{1...16}
|
||||
```
|
||||
|
||||
Distributed erasure coded configuration with 2 sets, 16 drives per set across hosts.
|
||||
|
||||
```
|
||||
minio server http://host{1...32}/disk1
|
||||
silo server http://host{1...32}/disk1
|
||||
```
|
||||
|
||||
Distributed erasure coded configuration with rack level redundancy 32 sets in total, 16 drives per set.
|
||||
|
||||
```
|
||||
minio server http://rack{1...4}-host{1...8}.example.net/export{1...16}
|
||||
silo server http://rack{1...4}-host{1...8}.example.net/export{1...16}
|
||||
```
|
||||
|
||||
+47
-38
@@ -1,95 +1,104 @@
|
||||
# Distributed MinIO Quickstart Guide [](https://slack.min.io) [](https://hub.docker.com/r/minio/minio/)
|
||||
# Distributed Silo Quickstart Guide [](https://hub.docker.com/r/pgsty/silo/)
|
||||
|
||||
MinIO in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server. As drives are distributed across several nodes, distributed MinIO can withstand multiple node failures and yet ensure full data protection.
|
||||
Silo in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server. As drives are distributed across several nodes, distributed Silo can withstand multiple node failures and yet ensure full data protection.
|
||||
|
||||
## Why distributed MinIO?
|
||||
## Why distributed Silo?
|
||||
|
||||
MinIO in distributed mode can help you setup a highly-available storage system with a single object storage deployment. With distributed MinIO, you can optimally use storage devices, irrespective of their location in a network.
|
||||
Silo in distributed mode can help you setup a highly-available storage system with a single object storage deployment. With distributed Silo, you can optimally use storage devices, irrespective of their location in a network.
|
||||
|
||||
### Data protection
|
||||
|
||||
Distributed MinIO provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/minio/blob/master/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed MinIO is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed MinIO.
|
||||
Distributed Silo provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/minio/blob/master/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed Silo is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed Silo.
|
||||
|
||||
If one or more drives are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data protection bits added automatically to provide additional safety for these objects.
|
||||
|
||||
### High availability
|
||||
|
||||
A stand-alone MinIO server would go down if the server hosting the drives goes offline. In contrast, a distributed MinIO setup with _m_ servers and _n_ drives will have your data safe as long as _m/2_ servers or _m*n_/2 or more drives are online.
|
||||
A stand-alone Silo server would go down if the server hosting the drives goes offline. In contrast, a distributed Silo setup with _m_ servers and _n_ drives will have your data safe as long as _m/2_ servers or _m*n_/2 or more drives are online.
|
||||
|
||||
For example, an 16-server distributed setup with 200 drives per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 drives down MinIO would continue to read and write objects.
|
||||
For example, an 16-server distributed setup with 200 drives per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 drives down Silo would continue to read and write objects.
|
||||
|
||||
Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/pgsty/minio/blob/master/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/pgsty/minio/tree/master/docs/erasure/storage-class).
|
||||
|
||||
### Consistency Guarantees
|
||||
|
||||
MinIO follows strict **read-after-write** and **list-after-write** consistency model for all i/o operations both in distributed and standalone modes. This consistency model is only guaranteed if you use disk filesystems such as xfs, zfs or btrfs etc.. for distributed setup.
|
||||
Silo follows strict **read-after-write** and **list-after-write** consistency model for all i/o operations both in distributed and standalone modes. This consistency model is only guaranteed if you use disk filesystems such as xfs, zfs or btrfs etc.. for distributed setup.
|
||||
|
||||
**In our tests we also found ext4 does not honor POSIX O_DIRECT/Fdatasync semantics, ext4 trades performance for consistency guarantees. Please avoid ext4 in your setup.**
|
||||
|
||||
**If MinIO distributed setup is using NFS volumes underneath it is not guaranteed MinIO will provide these consistency guarantees since NFS is not strictly consistent (If you must use NFS we recommend that you at least use NFSv4 instead of NFSv3 for relatively better outcomes).**
|
||||
**If Silo distributed setup is using NFS volumes underneath it is not guaranteed Silo will provide these consistency guarantees since NFS is not strictly consistent (If you must use NFS we recommend that you at least use NFSv4 instead of NFSv3 for relatively better outcomes).**
|
||||
|
||||
## Get started
|
||||
|
||||
If you're aware of stand-alone MinIO set up, the process remains largely the same. MinIO server automatically switches to stand-alone or distributed mode, depending on the command line parameters.
|
||||
If you're aware of stand-alone Silo set up, the process remains largely the same. Silo server automatically switches to stand-alone or distributed mode, depending on the command line parameters.
|
||||
|
||||
### 1. Prerequisites
|
||||
|
||||
Install MinIO either on Kubernetes or Distributed Linux.
|
||||
Install Silo either on Kubernetes or Distributed Linux.
|
||||
|
||||
Install MinIO on Kubernetes:
|
||||
Install Silo on Kubernetes:
|
||||
|
||||
- [MinIO Quickstart Guide for Kubernetes](https://silo.pgsty.com/operations/deployments/kubernetes/).
|
||||
- [Deploy a Tenant from the MinIO Operator](https://silo.pgsty.com/operations/deployments/k8s-deploy-minio-tenant-on-kubernetes/)
|
||||
- [Silo Quickstart Guide for Kubernetes](https://silo.pgsty.com/operations/deployments/kubernetes/).
|
||||
- [Deploy Silo on Kubernetes](https://silo.pgsty.com/operations/deployments/kubernetes/)
|
||||
|
||||
Install Distributed MinIO on Linux:
|
||||
- [Deploy Distributed MinIO on Linux](https://silo.pgsty.com/operations/deployments/baremetal/#deploy-minio-distributed-baremetal)
|
||||
Install Distributed Silo on Linux:
|
||||
- [Deploy Distributed Silo on Linux](https://silo.pgsty.com/operations/deployments/baremetal/#deploy-minio-distributed-baremetal)
|
||||
|
||||
### 2. Run distributed MinIO
|
||||
### 2. Run distributed Silo
|
||||
|
||||
To start a distributed MinIO instance, you just need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating nodes.
|
||||
To start a distributed Silo instance, you just need to pass drive locations as parameters to the silo server command. Then, you’ll need to run the same command on all the participating nodes.
|
||||
|
||||
**NOTE:**
|
||||
|
||||
- All the nodes running distributed MinIO should share a common root credentials, for the nodes to connect and trust each other. To achieve this, it is **recommended** to export root user and root password as environment variables, `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`, on all the nodes before executing MinIO server command. If not exported, default `minioadmin/minioadmin` credentials shall be used.
|
||||
- **MinIO creates erasure-coding sets of _2_ to _16_ drives per set. The number of drives you provide in total must be a multiple of one of those numbers.**
|
||||
- **MinIO chooses the largest EC set size which divides into the total number of drives or total number of nodes given - making sure to keep the uniform distribution i.e each node participates equal number of drives per set**.
|
||||
- All the nodes running distributed Silo should share a common root credentials, for the nodes to connect and trust each other. To achieve this, it is **recommended** to export root user and root password as environment variables, `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`, on all the nodes before executing Silo server command. If not exported, default `minioadmin/minioadmin` credentials shall be used.
|
||||
- **Silo creates erasure-coding sets of _2_ to _16_ drives per set. The number of drives you provide in total must be a multiple of one of those numbers.**
|
||||
- **Silo chooses the largest EC set size which divides into the total number of drives or total number of nodes given - making sure to keep the uniform distribution i.e each node participates equal number of drives per set**.
|
||||
- **Each object is written to a single EC set, and therefore is spread over no more than 16 drives.**
|
||||
- **All the nodes running distributed MinIO setup are recommended to be homogeneous, i.e. same operating system, same number of drives and same network interconnects.**
|
||||
- MinIO distributed mode requires **fresh directories**. If required, the drives can be shared with other applications. You can do this by using a sub-directory exclusive to MinIO. For example, if you have mounted your volume under `/export`, pass `/export/data` as arguments to MinIO server.
|
||||
- **All the nodes running distributed Silo setup are recommended to be homogeneous, i.e. same operating system, same number of drives and same network interconnects.**
|
||||
- Silo distributed mode requires **fresh directories**. If required, the drives can be shared with other applications. You can do this by using a sub-directory exclusive to Silo. For example, if you have mounted your volume under `/export`, pass `/export/data` as arguments to Silo server.
|
||||
- The IP addresses and drive paths below are for demonstration purposes only, you need to replace these with the actual IP addresses and drive paths/folders.
|
||||
- Servers running distributed MinIO instances should be less than 15 minutes apart. You can enable [NTP](http://www.ntp.org/) service as a best practice to ensure same times across servers.
|
||||
- Servers running distributed Silo instances should be less than 15 minutes apart. You can enable [NTP](http://www.ntp.org/) service as a best practice to ensure same times across servers.
|
||||
- `MINIO_DOMAIN` environment variable should be defined and exported for bucket DNS style support.
|
||||
- Running Distributed MinIO on **Windows** operating system is considered **experimental**. Please proceed with caution.
|
||||
- Running Distributed Silo on **Windows** operating system is considered **experimental**. Please proceed with caution.
|
||||
|
||||
Example 1: Start distributed MinIO instance on n nodes with m drives each mounted at `/export1` to `/exportm` (pictured below), by running this command on all the n nodes:
|
||||
Example 1: Start a distributed Silo instance on n nodes with m drives each mounted at `/export1` to `/exportm`, by running this command on all participating nodes:
|
||||
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
client["S3 clients"] --> endpoint["Load balancer or any Silo endpoint"]
|
||||
endpoint --> node1["Silo node 1"]
|
||||
endpoint --> node2["Silo node 2"]
|
||||
endpoint --> noden["Silo node n"]
|
||||
node1 --> drives1["export1 through exportm"]
|
||||
node2 --> drives2["export1 through exportm"]
|
||||
noden --> drivesn["export1 through exportm"]
|
||||
```
|
||||
|
||||
### GNU/Linux and macOS
|
||||
|
||||
```sh
|
||||
export MINIO_ROOT_USER=<ACCESS_KEY>
|
||||
export MINIO_ROOT_PASSWORD=<SECRET_KEY>
|
||||
minio server http://host{1...n}/export{1...m}
|
||||
silo server http://host{1...n}/export{1...m}
|
||||
```
|
||||
|
||||
> **NOTE:** In above example `n` and `m` represent positive integers, _do not copy paste and expect it work make the changes according to local deployment and setup_.
|
||||
> **NOTE:** `{1...n}` shown have 3 dots! Using only 2 dots `{1..n}` will be interpreted by your shell and won't be passed to MinIO server, affecting the erasure coding order, which would impact performance and high availability. **Always use ellipses syntax `{1...n}` (3 dots!) for optimal erasure-code distribution**
|
||||
> **NOTE:** `{1...n}` shown have 3 dots! Using only 2 dots `{1..n}` will be interpreted by your shell and won't be passed to Silo server, affecting the erasure coding order, which would impact performance and high availability. **Always use ellipses syntax `{1...n}` (3 dots!) for optimal erasure-code distribution**
|
||||
|
||||
### Expanding existing distributed setup
|
||||
|
||||
MinIO supports expanding distributed erasure coded clusters by specifying new set of clusters on the command-line as shown below:
|
||||
Silo supports expanding distributed erasure coded clusters by specifying new set of clusters on the command-line as shown below:
|
||||
|
||||
```sh
|
||||
export MINIO_ROOT_USER=<ACCESS_KEY>
|
||||
export MINIO_ROOT_PASSWORD=<SECRET_KEY>
|
||||
minio server http://host{1...n}/export{1...m} http://host{o...z}/export{1...m}
|
||||
silo server http://host{1...n}/export{1...m} http://host{o...z}/export{1...m}
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
minio server http://host{1...4}/export{1...16} http://host{5...12}/export{1...16}
|
||||
silo server http://host{1...4}/export{1...16} http://host{5...12}/export{1...16}
|
||||
```
|
||||
|
||||
Now the server has expanded total storage by _(newly_added_servers\*m)_ more drives, taking the total count to _(existing_servers\*m)+(newly_added_servers\*m)_ drives. New object upload requests automatically start using the least used cluster. This expansion strategy works endlessly, so you can perpetually expand your clusters as needed. When you restart, it is immediate and non-disruptive to the applications. Each group of servers in the command-line is called a pool. There are 2 server pools in this example. New objects are placed in server pools in proportion to the amount of free space in each pool. Within each pool, the location of the erasure-set of drives is determined based on a deterministic hashing algorithm.
|
||||
@@ -98,12 +107,12 @@ Now the server has expanded total storage by _(newly_added_servers\*m)_ more dri
|
||||
|
||||
## 3. Test your setup
|
||||
|
||||
To test this setup, access the MinIO server via browser or [`mc`](https://silo.pgsty.com/reference/minio-mc/#quickstart).
|
||||
To test this setup, access the Silo server via browser or [`mc`](https://silo.pgsty.com/reference/minio-mc/#quickstart).
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [MinIO Erasure Code QuickStart Guide](https://silo.pgsty.com/operations/concepts/erasure-coding/)
|
||||
- [Use `mc` with MinIO Server](https://silo.pgsty.com/reference/minio-mc/)
|
||||
- [Use `aws-cli` with MinIO Server](https://silo.pgsty.com/integrations/aws-cli-with-minio/)
|
||||
- [Use `minio-go` SDK with MinIO Server](https://silo.pgsty.com/developers/go/minio-go/)
|
||||
- [The MinIO documentation website](https://silo.pgsty.com/docs/)
|
||||
- [Silo Erasure Code QuickStart Guide](https://silo.pgsty.com/operations/concepts/erasure-coding/)
|
||||
- [Use `mc` with Silo Server](https://silo.pgsty.com/reference/minio-mc/)
|
||||
- [Use `aws-cli` with Silo Server](https://silo.pgsty.com/integrations/aws-cli-with-minio/)
|
||||
- [Use `minio-go` SDK with Silo Server](https://silo.pgsty.com/developers/go/minio-go/)
|
||||
- [The Silo documentation website](https://silo.pgsty.com/docs/)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Toy Setups
|
||||
|
||||
Capacity constrained environments, MinIO will work but not recommended for production.
|
||||
Capacity constrained environments, Silo will work but not recommended for production.
|
||||
|
||||
| servers | drives (per node) | stripe_size | parity chosen (default) | tolerance for reads (servers) | tolerance for writes (servers) |
|
||||
|--------:|------------------:|------------:|------------------------:|------------------------------:|-------------------------------:|
|
||||
|
||||
Reference in New Issue
Block a user