mirror of
https://github.com/pgsty/minio.git
synced 2026-08-09 07:43:29 +03:00
chore: cut over to pgsty/silo and main
The transitional references land in one commit, because they are only correct together: the repository is pgsty/silo, its default branch is main, and nothing in the tree should point a user at the old names. Changed: - Workflow branch filters. go.yml and vulncheck.yml gated on `branches: master` for both push and pull_request, so renaming the default branch would have taken automatic CI offline with no error and no signal - the workflows would simply never trigger again. They now name main. - Release target. goreleaser's `release.github.name` becomes silo, which is what actually decides where a tagged build publishes. sign-release-rpms.sh's GH_REPO default follows. - The OCI `image.source` label, the Helm chart `sources` entry, the security advisory link in the issue-template config, and the go.mod comment citing the LDAP TLS fix. - 115 occurrences across README, README_ZH, SECURITY, CONTRIBUTING and 30 docs pages, including 72 links that also carried the master branch in their path. Those matter most: GitHub redirects clone, fetch, push and web URLs after a rename, but raw.githubusercontent.com does not, and neither follows a branch rename - every one of those links would 404 twice over. - Three error strings in cmd/erasure-sets.go, cmd/storage-errors.go and internal/config/errors.go that print an issue URL to operators. These are Go string literals inside rebrand-guard's brand allowlist, so the baseline is regenerated. The regeneration removes exactly those three entries and adds none; all twelve other protected sets, including the 9014 exported symbols, are byte-identical. - The transitional-naming disclaimers in README, README_ZH, SECURITY and CONTRIBUTING are dropped, since they no longer describe anything. Deliberately unchanged, all three because they exist to reject or freeze the old name rather than to point at it: - buildscripts/minio-upgrade.sh pins pgsty/minio@sha256:b6bfe72... - the frozen pre-rebrand image is the control group for the MinIO-to-Silo upgrade test. - helm-migration-guard rejects any rendered container still pulling pgsty/minio. - verify-rebrand.sh rejects the same in the delivery surfaces. Also unchanged: docs/config/README.md links to pgsty/mc/blob/master, and that repository's default branch really is still master. It moves when mc does. verify-rebrand.sh gains three assertions so this cannot silently regress: no source reference may name pgsty/minio outside the three allowlisted guards, no link may target pgsty/silo's master branch, and go.yml and vulncheck.yml must filter on main. Both new rejections were negative-tested - reintroducing a master branch filter and adding a pgsty/minio URL each fail the gate with the specific message. This commit assumes the rename actually happens. Until the GitHub branch and repository renames are executed, the links it introduces do not resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Feng Ruohang
+12
-12
@@ -2,7 +2,7 @@
|
||||
|
||||
## **1. Cloud-native Architecture**
|
||||
|
||||
|
||||
|
||||
|
||||
Kubernetes manages stateless Spark and Hive containers elastically on the compute nodes. Spark has native scheduler integration with Kubernetes. Hive, for legacy reasons, uses YARN scheduler on top of Kubernetes.
|
||||
|
||||
@@ -23,17 +23,17 @@ Silo also supports multi-cluster, multi-site federation similar to AWS regions a
|
||||
|
||||
After successful installation navigate to the Ambari UI `http://<ambari-server>:8080/` and login using the default credentials: [**_username: admin, password: admin_**]
|
||||
|
||||
|
||||
|
||||
|
||||
### **3.1 Configure Hadoop**
|
||||
|
||||
Navigate to **Services** -> **HDFS** -> **CONFIGS** -> **ADVANCED** as shown below
|
||||
|
||||
|
||||
|
||||
|
||||
Navigate to **Custom core-site** to configure Silo parameters for `_s3a_` connector
|
||||
|
||||
|
||||
|
||||
|
||||
```
|
||||
sudo pip install yq
|
||||
@@ -100,17 +100,17 @@ The rest of the other optimization options are discussed in the links below
|
||||
|
||||
Once the config changes are applied, proceed to restart **Hadoop** services.
|
||||
|
||||
|
||||
|
||||
|
||||
### **3.2 Configure Spark2**
|
||||
|
||||
Navigate to **Services** -> **Spark2** -> **CONFIGS** as shown below
|
||||
|
||||
|
||||
|
||||
|
||||
Navigate to “**Custom spark-defaults**” to configure Silo parameters for `_s3a_` connector
|
||||
|
||||
|
||||
|
||||
|
||||
Add the following optimal entries for _spark-defaults.conf_ to configure Spark with **Silo**.
|
||||
|
||||
@@ -146,17 +146,17 @@ spark.hadoop.fs.s3a.threads.max 2048 # maximum number of threads for S3A
|
||||
|
||||
Once the config changes are applied, proceed to restart **Spark** services.
|
||||
|
||||
|
||||
|
||||
|
||||
### **3.3 Configure Hive**
|
||||
|
||||
Navigate to **Services** -> **Hive** -> **CONFIGS**-> **ADVANCED** as shown below
|
||||
|
||||
|
||||
|
||||
|
||||
Navigate to “**Custom hive-site**” to configure Silo parameters for `_s3a_` connector
|
||||
|
||||
|
||||
|
||||
|
||||
Add the following optimal entries for `hive-site.xml` to configure Hive with **Silo**.
|
||||
|
||||
@@ -171,11 +171,11 @@ mapreduce.input.fileinputformat.list-status.num-threads=50
|
||||
|
||||
For more information about these options please visit [https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html](https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html)
|
||||
|
||||
|
||||
|
||||
|
||||
Once the config changes are applied, proceed to restart all Hive services.
|
||||
|
||||
|
||||
|
||||
|
||||
## **4. Run Sample Applications**
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# ILM Tiering Design [](https://hub.docker.com/r/pgsty/silo/)
|
||||
|
||||
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/pgsty/minio/blob/master/docs/bucket/lifecycle/README.md) allows tiering of content from Silo object store to public clouds or other Silo clusters.
|
||||
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/pgsty/silo/blob/main/docs/bucket/lifecycle/README.md) allows tiering of content from Silo object store to public clouds or other Silo clusters.
|
||||
|
||||
Transition tiers can be added to Silo using `mc admin tier add` command to associate a `gcs`, `s3` or `azure` bucket or prefix path on a bucket to the tier name.
|
||||
Lifecycle transition rules can be applied to buckets (both versioned and un-versioned) by specifying the tier name defined above as the transition storage class for the lifecycle rule.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Bucket Quota Configuration Quickstart Guide [](https://hub.docker.com/r/pgsty/silo/)
|
||||
|
||||
|
||||
|
||||
|
||||
Buckets can be configured to have `Hard` quota - it disallows writes to the bucket after configured quota limit is reached.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Bucket Replication Design [](https://hub.docker.com/r/pgsty/silo/)
|
||||
|
||||
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/README.md) first.
|
||||
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/pgsty/silo/blob/main/docs/bucket/replication/README.md) first.
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -59,7 +59,7 @@ If 3 or more targets are participating in active-active replication, the replica
|
||||
|
||||
### Internal metadata for replication
|
||||
|
||||
`xl.meta` that is in use for [versioning](https://github.com/pgsty/minio/blob/master/docs/bucket/versioning/DESIGN.md) has additional metadata for replication of objects,delete markers and versioned deletes.
|
||||
`xl.meta` that is in use for [versioning](https://github.com/pgsty/silo/blob/main/docs/bucket/versioning/DESIGN.md) has additional metadata for replication of objects,delete markers and versioned deletes.
|
||||
|
||||
### Metadata for object replication - on source
|
||||
|
||||
|
||||
@@ -94,7 +94,7 @@ The access key provided for the replication *target* cluster should have these m
|
||||
}
|
||||
```
|
||||
|
||||
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/setup_replication.sh)
|
||||
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/pgsty/silo/blob/main/docs/bucket/replication/setup_replication.sh)
|
||||
|
||||
To set up replication from `srcbucket` on the `mysilo` cluster to `destbucket`
|
||||
on a target Silo cluster at `https://replica-endpoint:9000`, use:
|
||||
@@ -162,9 +162,9 @@ Replication status can be seen in the metadata on the source and destination obj
|
||||
|
||||
To perform bi-directional replication, repeat the above process on the target site - this time setting the source bucket as the replication target. It is recommended that replication be run in a system with at least two CPU's available to the process, so that replication can run in its own thread.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Replica Modification sync
|
||||
|
||||
@@ -211,11 +211,11 @@ Also note that `mc` version `RELEASE.2021-09-02T09-21-27Z` or older supports onl
|
||||
|
||||
Status of delete marker replication can be viewed by doing a GET/HEAD on the object version - it will return a `X-Minio-Replication-DeleteMarker-Status` header and http response code of `405`. In the case of permanent deletes, if the delete replication is pending or failed to propagate to the target cluster, GET/HEAD will return additional `X-Minio-Replication-Delete-Status` header and a http response code of `405`.
|
||||
|
||||
|
||||
|
||||
|
||||
The status of replication can be monitored by configuring event notifications on the source and target buckets using `mc event add`.On the source side, the `s3:PutObject`, `s3:Replication:OperationCompletedReplication` and `s3:Replication:OperationFailedReplication` events show the status of replication in the `X-Amz-Replication-Status` metadata.
|
||||
|
||||
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see <https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/list.md> for more details.
|
||||
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see <https://github.com/pgsty/silo/blob/main/docs/metrics/prometheus/list.md> for more details.
|
||||
|
||||
### Sync/Async Replication
|
||||
|
||||
@@ -277,6 +277,6 @@ Silo does not support SSE-C encrypted objects on replicated buckets, any applica
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [Silo Bucket Replication Design](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/DESIGN.md)
|
||||
- [Silo Bucket Replication Design](https://github.com/pgsty/silo/blob/main/docs/bucket/replication/DESIGN.md)
|
||||
- [Silo Bucket Versioning Implementation](https://silo.pgsty.com/administration/object-management/object-retention/)
|
||||
- [Silo Client Quickstart Guide](https://silo.pgsty.com/reference/minio-mc/#quickstart)
|
||||
|
||||
@@ -42,7 +42,7 @@ Therefore, the metadata is wrapped as a binary array for easy skipping.
|
||||
- LegacyObjectType (preserves existing deployments and older xl.json format)
|
||||
- DeleteMarker (a versionId to capture the DELETE sequences implemented primarily for AWS spec compatibility)
|
||||
|
||||
A sample msgpack-JSON `xl.meta`, you can debug the content inside `xl.meta` using [xl-meta.go](https://github.com/pgsty/minio/tree/master/docs/debugging#decoding-metadata) program.
|
||||
A sample msgpack-JSON `xl.meta`, you can debug the content inside `xl.meta` using [xl-meta.go](https://github.com/pgsty/silo/tree/main/docs/debugging#decoding-metadata) program.
|
||||
|
||||
```json
|
||||
{
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Silo versioning is designed to keep multiple versions of an object in one bucket. For example, you could store `spark.csv` (version `ede336f2`) and `spark.csv` (version `fae684da`) in a single bucket. Versioning protects you from unintended overwrites, deletions, protect objects with retention policies.
|
||||
|
||||
To control data retention and storage usage, use object versioning with [object lifecycle management](https://github.com/pgsty/minio/blob/master/docs/bucket/lifecycle/README.md). If you have an object expiration lifecycle policy in your non-versioned bucket and you want to maintain the same permanent delete behavior when on versioning-enabled bucket, you must add a noncurrent expiration policy. The noncurrent expiration lifecycle policy will manage the deletes of the noncurrent object versions in the versioning-enabled bucket. (A version-enabled bucket maintains one current and zero or more noncurrent object versions.)
|
||||
To control data retention and storage usage, use object versioning with [object lifecycle management](https://github.com/pgsty/silo/blob/main/docs/bucket/lifecycle/README.md). If you have an object expiration lifecycle policy in your non-versioned bucket and you want to maintain the same permanent delete behavior when on versioning-enabled bucket, you must add a noncurrent expiration policy. The noncurrent expiration lifecycle policy will manage the deletes of the noncurrent object versions in the versioning-enabled bucket. (A version-enabled bucket maintains one current and zero or more noncurrent object versions.)
|
||||
|
||||
Versioning must be explicitly enabled on a bucket, versioning is not enabled by default. Object locking enabled buckets have versioning enabled automatically. Enabling and suspending versioning is done at the bucket level.
|
||||
|
||||
@@ -10,25 +10,25 @@ Only Silo generates version IDs, and they can't be edited. Version IDs are simpl
|
||||
|
||||
When you PUT an object in a versioning-enabled bucket, the noncurrent version is not overwritten. The following figure shows that when a new version of `spark.csv` is PUT into a bucket that already contains an object with the same name, the original object (ID = `ede336f2`) remains in the bucket, Silo generates a new version (ID = `fae684da`), and adds the newer version to the bucket.
|
||||
|
||||
|
||||
|
||||
|
||||
This protects against accidental overwrites or deletes of objects, allows previous versions to be retrieved.
|
||||
|
||||
When you DELETE an object, all versions remain in the bucket and Silo adds a delete marker, as shown below:
|
||||
|
||||
|
||||
|
||||
|
||||
Now the delete marker becomes the current version of the object. GET requests by default always retrieve the latest stored version. So performing a simple GET object request when the current version is a delete marker would return `404` `The specified key does not exist` as shown below:
|
||||
|
||||
|
||||
|
||||
|
||||
GET requests by specifying a version ID as shown below, you can retrieve the specific object version `fae684da`.
|
||||
|
||||
|
||||
|
||||
|
||||
To permanently delete an object you need to specify the version you want to delete, only the user with appropriate permissions can permanently delete a version. As shown below DELETE request called with a specific version id permanently deletes an object from a bucket. Delete marker is not added for DELETE requests with version id.
|
||||
|
||||
|
||||
|
||||
|
||||
## Concepts
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Configuration Directory
|
||||
|
||||
Silo stores all its config as part of the server deployment, config is erasure coded on Silo. On a fresh deployment Silo automatically generates a new `config` and this config is available to be configured via `mc admin config` command. Silo also encrypts all the config, IAM and policies content if KMS is configured. Please refer to how to encrypt your config and IAM credentials [here](https://github.com/pgsty/minio/blob/master/docs/kms/IAM.md).
|
||||
Silo stores all its config as part of the server deployment, config is erasure coded on Silo. On a fresh deployment Silo automatically generates a new `config` and this config is available to be configured via `mc admin config` command. Silo also encrypts all the config, IAM and policies content if KMS is configured. Please refer to how to encrypt your config and IAM credentials [here](https://github.com/pgsty/silo/blob/main/docs/kms/IAM.md).
|
||||
|
||||
### Certificate Directory
|
||||
|
||||
@@ -65,7 +65,7 @@ silo server /data
|
||||
|
||||
### Storage Class
|
||||
|
||||
By default, parity for objects with standard storage class is set to `N/2`, and parity for objects with reduced redundancy storage class objects is set to `2`. Read more about storage class support in Silo server [here](https://github.com/pgsty/minio/blob/master/docs/erasure/storage-class/README.md).
|
||||
By default, parity for objects with standard storage class is set to `N/2`, and parity for objects with reduced redundancy storage class objects is set to `2`. Read more about storage class support in Silo server [here](https://github.com/pgsty/silo/blob/main/docs/erasure/storage-class/README.md).
|
||||
|
||||
```
|
||||
KEY:
|
||||
@@ -91,7 +91,7 @@ MINIO_STORAGE_CLASS_COMMENT (sentence) optionally add a comment to this setti
|
||||
|
||||
#### Etcd
|
||||
|
||||
Silo supports storing encrypted IAM assets in etcd, if KMS is configured. Please refer to how to encrypt your config and IAM credentials [here](https://github.com/pgsty/minio/blob/master/docs/kms/IAM.md).
|
||||
Silo supports storing encrypted IAM assets in etcd, if KMS is configured. Please refer to how to encrypt your config and IAM credentials [here](https://github.com/pgsty/silo/blob/main/docs/kms/IAM.md).
|
||||
|
||||
> NOTE: if *path_prefix* is set then Silo will not federate your buckets, namespaced IAM assets are assumed as isolated tenants, only buckets are considered globally unique but performing a lookup with a *bucket* which belongs to a different tenant will fail unlike federated setups where Silo would port-forward and route the request to relevant cluster accordingly. This is a special feature, federated deployments should not need to set *path_prefix*.
|
||||
|
||||
@@ -125,7 +125,7 @@ MINIO_ETCD_COMMENT (sentence) optionally add a comment to this setting
|
||||
|
||||
### API
|
||||
|
||||
By default, there is no limitation on the number of concurrent requests that a server/cluster processes at the same time. However, it is possible to impose such limitation using the API subsystem. Read more about throttling limitation in Silo server [here](https://github.com/pgsty/minio/blob/master/docs/throttle/README.md).
|
||||
By default, there is no limitation on the number of concurrent requests that a server/cluster processes at the same time. However, it is possible to impose such limitation using the API subsystem. Read more about throttling limitation in Silo server [here](https://github.com/pgsty/silo/blob/main/docs/throttle/README.md).
|
||||
|
||||
```
|
||||
KEY:
|
||||
|
||||
@@ -115,7 +115,7 @@ In above example there are two server pools
|
||||
|
||||
> Notice the requirement of common SLA here original cluster had 1024 drives with 16 drives per erasure set with default parity of '4', second pool is expected to have a minimum of 8 drives per erasure set to match the original cluster SLA (parity count) of '4'. '12' drives stripe per erasure set in the second pool satisfies the original pool's parity count.
|
||||
|
||||
Refer to the sizing guide with details on the default parity count chosen for different erasure stripe sizes [here](https://github.com/pgsty/minio/blob/master/docs/distributed/SIZING.md)
|
||||
Refer to the sizing guide with details on the default parity count chosen for different erasure stripe sizes [here](https://github.com/pgsty/silo/blob/main/docs/distributed/SIZING.md)
|
||||
|
||||
Silo places new objects in server pools based on proportionate free space, per pool. Following pseudo code demonstrates this behavior.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Silo in distributed mode can help you setup a highly-available storage system wi
|
||||
|
||||
### Data protection
|
||||
|
||||
Distributed Silo provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/minio/blob/master/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed Silo is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed Silo.
|
||||
Distributed Silo provides protection against multiple node/drive failures and [bit rot](https://github.com/pgsty/silo/blob/main/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://silo.pgsty.com/operations/concepts/erasure-coding/). As the minimum drives required for distributed Silo is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed Silo.
|
||||
|
||||
If one or more drives are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data protection bits added automatically to provide additional safety for these objects.
|
||||
|
||||
@@ -18,7 +18,7 @@ A stand-alone Silo server would go down if the server hosting the drives goes of
|
||||
|
||||
For example, an 16-server distributed setup with 200 drives per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 drives down Silo would continue to read and write objects.
|
||||
|
||||
Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/pgsty/minio/blob/master/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/pgsty/minio/tree/master/docs/erasure/storage-class).
|
||||
Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/pgsty/silo/blob/main/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class).
|
||||
|
||||
### Consistency Guarantees
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
See the [Silo documentation](https://silo.pgsty.com/docs/) for production
|
||||
deployment and operations guidance.
|
||||
|
||||
For images built from this `pgsty/minio` fork, the container also bundles `mcli` and a compatibility `mc` symlink from `pgsty/mc`.
|
||||
For images built from this `pgsty/silo` fork, the container also bundles `mcli` and a compatibility `mc` symlink from `pgsty/mc`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ Silo protects data against hardware failures and silent data corruption using er
|
||||
|
||||
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Silo uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.
|
||||
|
||||
By default, Silo shards the objects across N/2 data and N/2 parity drives. Though, you can use [storage classes](https://github.com/pgsty/minio/tree/master/docs/erasure/storage-class) to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.
|
||||
By default, Silo shards the objects across N/2 data and N/2 parity drives. Though, you can use [storage classes](https://github.com/pgsty/silo/tree/main/docs/erasure/storage-class) to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.
|
||||
|
||||
In 12 drive example above, with Silo server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.
|
||||
|
||||
@@ -14,7 +14,7 @@ In 12 drive example above, with Silo server running in the default configuration
|
||||
|
||||
Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in Silo erasure code you can lose as many as half of drives and still the data remains safe. Further, Silo's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As Silo encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. Silo's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.
|
||||
|
||||
|
||||
|
||||
|
||||
## What is Bit Rot protection?
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ export MINIO_STORAGE_CLASS_STANDARD=EC:3
|
||||
export MINIO_STORAGE_CLASS_RRS=EC:2
|
||||
```
|
||||
|
||||
Storage class can also be set via `mc admin config` get/set commands to update the configuration. Refer [storage class](https://github.com/pgsty/minio/tree/master/docs/config#storage-class) for
|
||||
Storage class can also be set via `mc admin config` get/set commands to update the configuration. Refer [storage class](https://github.com/pgsty/silo/tree/main/docs/config#storage-class) for
|
||||
more details.
|
||||
|
||||
#### Note
|
||||
|
||||
@@ -23,9 +23,9 @@ All properties except the file size are tied to the zip file. This means that mo
|
||||
|
||||
## Code Examples
|
||||
|
||||
[Using minio-go library](https://github.com/pgsty/minio/blob/master/docs/extensions/s3zip/examples/minio-go/main.go)
|
||||
[Using AWS JS SDK v2](https://github.com/pgsty/minio/blob/master/docs/extensions/s3zip/examples/aws-js/main.js)
|
||||
[Using boto3](https://github.com/pgsty/minio/blob/master/docs/extensions/s3zip/examples/boto3/main.py)
|
||||
[Using minio-go library](https://github.com/pgsty/silo/blob/main/docs/extensions/s3zip/examples/minio-go/main.go)
|
||||
[Using AWS JS SDK v2](https://github.com/pgsty/silo/blob/main/docs/extensions/s3zip/examples/aws-js/main.js)
|
||||
[Using boto3](https://github.com/pgsty/silo/blob/main/docs/extensions/s3zip/examples/boto3/main.py)
|
||||
|
||||
## Requirements and limits
|
||||
|
||||
|
||||
@@ -34,7 +34,7 @@ Follow the steps from the Veeam documentation for adding Silo as an object store
|
||||
|
||||
For Veeam Backup with Immutability, choose the amount of days you want to make backups immutable for
|
||||
|
||||
|
||||
|
||||
|
||||
### Creating the Scale-out Backup Repository
|
||||
|
||||
@@ -58,7 +58,7 @@ For Veeam Backup with Immutability, choose the amount of days you want to make b
|
||||
|
||||
- For Veeam Backup with Immutability, you can choose a number of restore points or days to make backups immutable.
|
||||
|
||||
|
||||
|
||||
|
||||
#### Backup Office 365 with VBO
|
||||
|
||||
@@ -70,7 +70,7 @@ mc mb -l mysilo/vbo
|
||||
|
||||
- Under Backup Infrastructure, right click on Object Storage Repositories and choose "Add object storage"
|
||||
|
||||
|
||||
|
||||
|
||||
- Follow through the wizard as above for Veeam Backup and Replication as the steps are the same between both products
|
||||
|
||||
@@ -78,7 +78,7 @@ mc mb -l mysilo/vbo
|
||||
|
||||
- Follow the wizard. Under the "Object Storage Backup Repository" section, choose the Silo object storage you created above
|
||||
|
||||
|
||||
|
||||
|
||||
- When you create your backup job, choose the backup repository you created above.
|
||||
|
||||
|
||||
+2
-2
@@ -7,7 +7,7 @@ Silo supports encrypting config, IAM assets with KMS provided keys. If the KMS i
|
||||
Silo supports two ways of encrypting IAM and configuration data.
|
||||
You can either use KES - together with an external KMS - or, much simpler,
|
||||
set the env. variable `MINIO_KMS_SECRET_KEY` and start/restart the Silo server. For more details about KES and how
|
||||
to set it up refer to our [KMS Guide](https://github.com/pgsty/minio/blob/master/docs/kms/README.md).
|
||||
to set it up refer to our [KMS Guide](https://github.com/pgsty/silo/blob/main/docs/kms/README.md).
|
||||
|
||||
Instead of configuring an external KMS you can start with a single key by
|
||||
setting the env. variable `MINIO_KMS_SECRET_KEY`. It expects the following
|
||||
@@ -45,7 +45,7 @@ kes key create my-minio-key OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
|
||||
|
||||
- For instructions on setting up KES, see the [KES Getting Started guide](https://github.com/minio/kes/wiki/Getting-Started)
|
||||
|
||||
- For instructions on using KES for encrypting the Silo backend, follow the [KMS Quick Start](https://github.com/pgsty/minio/tree/master/docs/kms). The SSE-S3 configuration setup also supports Silo KMS backend encryption.
|
||||
- For instructions on using KES for encrypting the Silo backend, follow the [KMS Quick Start](https://github.com/pgsty/silo/tree/main/docs/kms). The SSE-S3 configuration setup also supports Silo KMS backend encryption.
|
||||
|
||||
## FAQ
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ Silo server has two healthcheck related un-authenticated endpoints, a liveness p
|
||||
- Liveness probe available at `/minio/health/live`
|
||||
- Cluster probe available at `/minio/health/cluster`
|
||||
|
||||
Read more on how to use these endpoints in [Silo healthcheck guide](https://github.com/pgsty/minio/blob/master/docs/metrics/healthcheck/README.md).
|
||||
Read more on how to use these endpoints in [Silo healthcheck guide](https://github.com/pgsty/silo/blob/main/docs/metrics/healthcheck/README.md).
|
||||
|
||||
## Prometheus Probe
|
||||
|
||||
@@ -25,7 +25,7 @@ The additional bucket specific metrics which include additional go metrics or pr
|
||||
The additional resource specific metrics which include additional go metrics or process metrics are exposed at
|
||||
`<Address for Silo Node>/minio/v2/metrics/resource`.
|
||||
|
||||
To use this endpoint, setup Prometheus to scrape data from this endpoint. Read more on how to configure and use Prometheus to monitor Silo server in [How to monitor Silo server with Prometheus](https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/README.md).
|
||||
To use this endpoint, setup Prometheus to scrape data from this endpoint. Read more on how to configure and use Prometheus to monitor Silo server in [How to monitor Silo server with Prometheus](https://github.com/pgsty/silo/blob/main/docs/metrics/prometheus/README.md).
|
||||
|
||||
### **Deprecated metrics monitoring**
|
||||
|
||||
|
||||
@@ -171,7 +171,7 @@ Prometheus sets the `Host` header to `domain:port` as part of HTTP operations ag
|
||||
|
||||
### 6. Configure Grafana
|
||||
|
||||
After Prometheus is configured, you can use Grafana to visualize Silo metrics. Refer the [document here to setup Grafana with Silo prometheus metrics](https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/grafana/README.md).
|
||||
After Prometheus is configured, you can use Grafana to visualize Silo metrics. Refer the [document here to setup Grafana with Silo prometheus metrics](https://github.com/pgsty/silo/blob/main/docs/metrics/prometheus/grafana/README.md).
|
||||
|
||||
## List of metrics exposed by Silo
|
||||
|
||||
@@ -189,8 +189,8 @@ curl http://127.0.0.1:9000/minio/v2/metrics/cluster
|
||||
|
||||
### List of metrics reported Cluster and Bucket level
|
||||
|
||||
[The list of metrics reported can be here](https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/list.md)
|
||||
[The list of metrics reported can be here](https://github.com/pgsty/silo/blob/main/docs/metrics/prometheus/list.md)
|
||||
|
||||
### Configure Alerts for Prometheus
|
||||
|
||||
[The Prometheus AlertManager and alerts can be configured following this](https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/alerts.md)
|
||||
[The Prometheus AlertManager and alerts can be configured following this](https://github.com/pgsty/silo/blob/main/docs/metrics/prometheus/alerts.md)
|
||||
|
||||
@@ -117,4 +117,4 @@ To verify the above sample alert follow below steps
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Prometheus and Silo configured as explained in [document here](https://github.com/pgsty/minio/blob/master/docs/metrics/prometheus/README.md).
|
||||
- Prometheus and Silo configured as explained in [document here](https://github.com/pgsty/silo/blob/main/docs/metrics/prometheus/README.md).
|
||||
- Grafana installed as explained [here](https://grafana.com/grafana/download).
|
||||
|
||||
## Silo Grafana dashboards
|
||||
|
||||
@@ -41,7 +41,7 @@ For optimal production setup Silo recommends Linux kernel version 4.x and later.
|
||||
|
||||
## List of Amazon S3 APIs not supported on Silo
|
||||
|
||||
We found the following APIs to be redundant or less useful outside of AWS S3. If you have a different view on any of the APIs we missed, please consider opening a [GitHub issue](https://github.com/pgsty/minio/issues) with relevant details on why Silo must implement them.
|
||||
We found the following APIs to be redundant or less useful outside of AWS S3. If you have a different view on any of the APIs we missed, please consider opening a [GitHub issue](https://github.com/pgsty/silo/issues) with relevant details on why Silo must implement them.
|
||||
|
||||
### List of Amazon S3 Bucket APIs not supported on Silo
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ In this document we will explain in detail on how to configure multiple users.
|
||||
|
||||
- Install mc - [Silo Client Quickstart Guide](https://silo.pgsty.com/reference/minio-mc/#quickstart)
|
||||
- Install Silo - [Silo Quickstart Guide](https://silo.pgsty.com/operations/deployments/baremetal-deploy-minio-on-redhat-linux/)
|
||||
- Configure etcd - [Etcd V3 Quickstart Guide](https://github.com/pgsty/minio/blob/master/docs/sts/etcd.md)
|
||||
- Configure etcd - [Etcd V3 Quickstart Guide](https://github.com/pgsty/silo/blob/main/docs/sts/etcd.md)
|
||||
|
||||
### 2. Create a new user with canned policy
|
||||
|
||||
|
||||
+8
-8
@@ -15,17 +15,17 @@ Following are advantages for using temporary credentials:
|
||||
|
||||
| AuthN | Description |
|
||||
| :---------------------- | ------------------------------------------ |
|
||||
| [**WebIdentity**](https://github.com/pgsty/minio/blob/master/docs/sts/web-identity.md) | Let users request temporary credentials using any OpenID(OIDC) compatible web identity providers such as KeyCloak, Dex, Facebook, Google etc. |
|
||||
| [**AD/LDAP**](https://github.com/pgsty/minio/blob/master/docs/sts/ldap.md) | Let AD/LDAP users request temporary credentials using AD/LDAP username and password. |
|
||||
| [**AssumeRole**](https://github.com/pgsty/minio/blob/master/docs/sts/assume-role.md) | Let Silo users request temporary credentials using user access and secret keys. |
|
||||
| [**WebIdentity**](https://github.com/pgsty/silo/blob/main/docs/sts/web-identity.md) | Let users request temporary credentials using any OpenID(OIDC) compatible web identity providers such as KeyCloak, Dex, Facebook, Google etc. |
|
||||
| [**AD/LDAP**](https://github.com/pgsty/silo/blob/main/docs/sts/ldap.md) | Let AD/LDAP users request temporary credentials using AD/LDAP username and password. |
|
||||
| [**AssumeRole**](https://github.com/pgsty/silo/blob/main/docs/sts/assume-role.md) | Let Silo users request temporary credentials using user access and secret keys. |
|
||||
|
||||
### Understanding JWT Claims
|
||||
|
||||
> NOTE: JWT claims are only meant for WebIdentity and ClientGrants.
|
||||
> AssumeRole or LDAP users can skip the entire portion and directly visit one of the links below.
|
||||
>
|
||||
> - [**AssumeRole**](https://github.com/pgsty/minio/blob/master/docs/sts/assume-role.md)
|
||||
> - [**AD/LDAP**](https://github.com/pgsty/minio/blob/master/docs/sts/ldap.md)
|
||||
> - [**AssumeRole**](https://github.com/pgsty/silo/blob/main/docs/sts/assume-role.md)
|
||||
> - [**AD/LDAP**](https://github.com/pgsty/silo/blob/main/docs/sts/ldap.md)
|
||||
|
||||
The id_token received is a signed JSON Web Token (JWT). Use a JWT decoder to decode the id_token to access the payload of the token that includes following JWT claims, `policy` claim is mandatory and should be present as part of your JWT claim. Without this claim the generated credentials will not have access to any resources on the server, using these credentials application would receive 'Access Denied' errors.
|
||||
|
||||
@@ -37,12 +37,12 @@ The id_token received is a signed JSON Web Token (JWT). Use a JWT decoder to dec
|
||||
|
||||
In this document we will explain in detail on how to configure all the prerequisites.
|
||||
|
||||
> NOTE: If you are interested in AssumeRole API only, skip to [here](https://github.com/pgsty/minio/blob/master/docs/sts/assume-role.md)
|
||||
> NOTE: If you are interested in AssumeRole API only, skip to [here](https://github.com/pgsty/silo/blob/main/docs/sts/assume-role.md)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- [Configuring keycloak](https://github.com/pgsty/minio/blob/master/docs/sts/keycloak.md) or [Configuring Casdoor](https://github.com/pgsty/minio/blob/master/docs/sts/casdoor.md)
|
||||
- [Configuring etcd](https://github.com/pgsty/minio/blob/master/docs/sts/etcd.md)
|
||||
- [Configuring keycloak](https://github.com/pgsty/silo/blob/main/docs/sts/keycloak.md) or [Configuring Casdoor](https://github.com/pgsty/silo/blob/main/docs/sts/casdoor.md)
|
||||
- [Configuring etcd](https://github.com/pgsty/silo/blob/main/docs/sts/etcd.md)
|
||||
|
||||
### Setup Silo with Identity Provider
|
||||
|
||||
|
||||
@@ -97,7 +97,7 @@ silo server /mnt/export
|
||||
```
|
||||
|
||||
Testing with an example
|
||||
> Obtaining client ID and secrets follow [Keycloak configuring documentation](https://github.com/pgsty/minio/blob/master/docs/sts/keycloak.md)
|
||||
> Obtaining client ID and secrets follow [Keycloak configuring documentation](https://github.com/pgsty/silo/blob/main/docs/sts/keycloak.md)
|
||||
|
||||
```
|
||||
$ go run client-grants.go -cid PoEgXP6uVO45IsENRngDXj5Au5Ya -csec eKsw6z8CtOJVBtrOWvhRWL4TUCga
|
||||
|
||||
+2
-2
@@ -50,7 +50,7 @@ NOTE: If `etcd` is configured with `Client-to-server authentication with HTTPS c
|
||||
|
||||
Once etcd is configured, **any STS configuration** will work including Client Grants, Web Identity or AD/LDAP.
|
||||
|
||||
For example, you can configure STS with Client Grants (KeyCloak) using the guides at [Silo STS Quickstart Guide](https://silo.pgsty.com/developers/security-token-service/) and [KeyCloak Configuration Guide](https://github.com/pgsty/minio/blob/master/docs/sts/keycloak.md). Once this is done, STS credentials can be generated:
|
||||
For example, you can configure STS with Client Grants (KeyCloak) using the guides at [Silo STS Quickstart Guide](https://silo.pgsty.com/developers/security-token-service/) and [KeyCloak Configuration Guide](https://github.com/pgsty/silo/blob/main/docs/sts/keycloak.md). Once this is done, STS credentials can be generated:
|
||||
|
||||
```
|
||||
go run client-grants.go -cid PoEgXP6uVO45IsENRngDXj5Au5Ya -csec eKsw6z8CtOJVBtrOWvhRWL4TUCga
|
||||
@@ -64,7 +64,7 @@ go run client-grants.go -cid PoEgXP6uVO45IsENRngDXj5Au5Ya -csec eKsw6z8CtOJVBtrO
|
||||
}
|
||||
```
|
||||
|
||||
These credentials can now be used to perform Silo API operations, these credentials automatically expire in 1hr. To understand more about credential expiry duration and client grants STS API read further [here](https://github.com/pgsty/minio/blob/master/docs/sts/client-grants.md).
|
||||
These credentials can now be used to perform Silo API operations, these credentials automatically expire in 1hr. To understand more about credential expiry duration and client grants STS API read further [here](https://github.com/pgsty/silo/blob/main/docs/sts/client-grants.md).
|
||||
|
||||
## Explore Further
|
||||
|
||||
|
||||
+1
-1
@@ -351,7 +351,7 @@ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=groupOfNames)(mem
|
||||
silo server ~/test
|
||||
```
|
||||
|
||||
You can make sure it works appropriately using our [example program](https://raw.githubusercontent.com/pgsty/minio/master/docs/sts/ldap.go):
|
||||
You can make sure it works appropriately using our [example program](https://raw.githubusercontent.com/pgsty/silo/main/docs/sts/ldap.go):
|
||||
|
||||
```
|
||||
$ go run ldap.go -u foouser -p foopassword
|
||||
|
||||
+1
-1
@@ -101,7 +101,7 @@ go run client-grants.go -cid PoEgXP6uVO45IsENRngDXj5Au5Ya -csec eKsw6z8CtOJVBtrO
|
||||
}
|
||||
```
|
||||
|
||||
These credentials can now be used to perform Silo API operations, these credentials automatically expire in 1hr. To understand more about credential expiry duration and client grants STS API read further [here](https://github.com/pgsty/minio/blob/master/docs/sts/client-grants.md).
|
||||
These credentials can now be used to perform Silo API operations, these credentials automatically expire in 1hr. To understand more about credential expiry duration and client grants STS API read further [here](https://github.com/pgsty/silo/blob/main/docs/sts/client-grants.md).
|
||||
|
||||
## Explore Further
|
||||
|
||||
|
||||
+1
-1
@@ -237,7 +237,7 @@ Silo can connect to other servers, including Silo nodes or other server types su
|
||||
|
||||
## Explore Further
|
||||
|
||||
* [TLS Configuration for Silo server on Kubernetes](https://github.com/pgsty/minio/tree/master/docs/tls/kubernetes)
|
||||
* [TLS Configuration for Silo server on Kubernetes](https://github.com/pgsty/silo/tree/main/docs/tls/kubernetes)
|
||||
* [Silo Client Complete Guide](https://silo.pgsty.com/reference/minio-mc/)
|
||||
* [Silo Network Encryption Overview](https://silo.pgsty.com/operations/network-encryption/)
|
||||
* [Generate Let's Encrypt Certificate](https://silo.pgsty.com/integrations/generate-lets-encrypt-certificate-using-certbot-for-minio/)
|
||||
|
||||
@@ -12,7 +12,7 @@ This document explains how to configure Silo server with TLS certificates on Kub
|
||||
|
||||
For a [distributed Silo setup](https://silo.pgsty.com/operations/deployments/kubernetes/), where there are multiple pods with different domain names expected to run, you will either need wildcard certificates valid for all the domains or have specific certificates for each domain. If you are going to use specific certificates, make sure to create Kubernetes secrets accordingly.
|
||||
|
||||
For testing purposes, here is [how to create self-signed certificates](https://github.com/pgsty/minio/tree/master/docs/tls#3-generate-self-signed-certificates).
|
||||
For testing purposes, here is [how to create self-signed certificates](https://github.com/pgsty/silo/tree/main/docs/tls#3-generate-self-signed-certificates).
|
||||
|
||||
## 2. Create Kubernetes secret
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Please make sure the following packages are already installed via `dnf` or `apt`
|
||||
|
||||
#### Step 1 - download `tuned.conf` from the referenced link
|
||||
```
|
||||
wget https://raw.githubusercontent.com/pgsty/minio/master/docs/tuning/tuned.conf
|
||||
wget https://raw.githubusercontent.com/pgsty/silo/main/docs/tuning/tuned.conf
|
||||
```
|
||||
|
||||
#### Step 2 - install tuned.conf as supported performance profile on all nodes
|
||||
|
||||
Reference in New Issue
Block a user