Feng Ruohang
c46b16ec62
The transitional references land in one commit, because they are only correct together: the repository is pgsty/silo, its default branch is main, and nothing in the tree should point a user at the old names. Changed: - Workflow branch filters. go.yml and vulncheck.yml gated on `branches: master` for both push and pull_request, so renaming the default branch would have taken automatic CI offline with no error and no signal - the workflows would simply never trigger again. They now name main. - Release target. goreleaser's `release.github.name` becomes silo, which is what actually decides where a tagged build publishes. sign-release-rpms.sh's GH_REPO default follows. - The OCI `image.source` label, the Helm chart `sources` entry, the security advisory link in the issue-template config, and the go.mod comment citing the LDAP TLS fix. - 115 occurrences across README, README_ZH, SECURITY, CONTRIBUTING and 30 docs pages, including 72 links that also carried the master branch in their path. Those matter most: GitHub redirects clone, fetch, push and web URLs after a rename, but raw.githubusercontent.com does not, and neither follows a branch rename - every one of those links would 404 twice over. - Three error strings in cmd/erasure-sets.go, cmd/storage-errors.go and internal/config/errors.go that print an issue URL to operators. These are Go string literals inside rebrand-guard's brand allowlist, so the baseline is regenerated. The regeneration removes exactly those three entries and adds none; all twelve other protected sets, including the 9014 exported symbols, are byte-identical. - The transitional-naming disclaimers in README, README_ZH, SECURITY and CONTRIBUTING are dropped, since they no longer describe anything. Deliberately unchanged, all three because they exist to reject or freeze the old name rather than to point at it: - buildscripts/minio-upgrade.sh pins pgsty/minio@sha256:b6bfe72... - the frozen pre-rebrand image is the control group for the MinIO-to-Silo upgrade test. - helm-migration-guard rejects any rendered container still pulling pgsty/minio. - verify-rebrand.sh rejects the same in the delivery surfaces. Also unchanged: docs/config/README.md links to pgsty/mc/blob/master, and that repository's default branch really is still master. It moves when mc does. verify-rebrand.sh gains three assertions so this cannot silently regress: no source reference may name pgsty/minio outside the three allowlisted guards, no link may target pgsty/silo's master branch, and go.yml and vulncheck.yml must filter on main. Both new rejections were negative-tested - reintroducing a master branch filter and adding a pgsty/minio URL each fail the gate with the specific message. This commit assumes the rename actually happens. Until the GitHub branch and repository renames are executed, the links it introduces do not resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.0 KiB
Silo Erasure Code Quickstart Guide
Silo protects data against hardware failures and silent data corruption using erasure code and checksums. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data.
What is Erasure Code?
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Silo uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.
By default, Silo shards the objects across N/2 data and N/2 parity drives. Though, you can use storage classes to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.
In 12 drive example above, with Silo server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.
Why is Erasure Code useful?
Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in Silo erasure code you can lose as many as half of drives and still the data remains safe. Further, Silo's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As Silo encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. Silo's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.
What is Bit Rot protection?
Bit Rot, also known as data rot or silent data corruption is a data loss issue faced by disk drives today. Data on the drive may silently get corrupted without signaling an error has occurred, making bit rot more dangerous than a permanent hard drive failure.
Silo's erasure coded backend uses high speed HighwayHash checksums to protect against Bit Rot.
How are drives used for Erasure Code?
Silo divides the drives you provide into erasure-coding sets of 2 to 16 drives. Therefore, the number of drives you present must be a multiple of one of these numbers. Each object is written to a single erasure-coding set.
Silo uses the largest possible EC set size which divides into the number of drives given. For example, 18 drives are configured as 2 sets of 9 drives, and 24 drives are configured as 2 sets of 12 drives. This is true for scenarios when running Silo as a standalone erasure coded deployment. In distributed setup however node (affinity) based erasure stripe sizes are chosen.
The drives should all be of approximately the same size.
Get Started with Silo in Erasure Code
1. Prerequisites
Install Silo - Silo Quickstart Guide
2. Run Silo Server with Erasure Code
Example: Start Silo server in a 12 drives setup, using Silo binary.
silo server /data{1...12}
Example: Start Silo server in a 8 drives setup, using Silo Docker image.
podman run \
-p 9000:9000 \
-p 9001:9001 \
--name silo \
-v /mnt/data1:/data1 \
-v /mnt/data2:/data2 \
-v /mnt/data3:/data3 \
-v /mnt/data4:/data4 \
-v /mnt/data5:/data5 \
-v /mnt/data6:/data6 \
-v /mnt/data7:/data7 \
-v /mnt/data8:/data8 \
docker.io/pgsty/silo server /data{1...8} --console-address ":9001"
3. Test your setup
You may unplug drives randomly and continue to perform I/O on the system.