mirror of
https://github.com/pgsty/minio.git
synced 2026-08-09 15:53:28 +03:00
Feng Ruohang
c46b16ec62
The transitional references land in one commit, because they are only correct together: the repository is pgsty/silo, its default branch is main, and nothing in the tree should point a user at the old names. Changed: - Workflow branch filters. go.yml and vulncheck.yml gated on `branches: master` for both push and pull_request, so renaming the default branch would have taken automatic CI offline with no error and no signal - the workflows would simply never trigger again. They now name main. - Release target. goreleaser's `release.github.name` becomes silo, which is what actually decides where a tagged build publishes. sign-release-rpms.sh's GH_REPO default follows. - The OCI `image.source` label, the Helm chart `sources` entry, the security advisory link in the issue-template config, and the go.mod comment citing the LDAP TLS fix. - 115 occurrences across README, README_ZH, SECURITY, CONTRIBUTING and 30 docs pages, including 72 links that also carried the master branch in their path. Those matter most: GitHub redirects clone, fetch, push and web URLs after a rename, but raw.githubusercontent.com does not, and neither follows a branch rename - every one of those links would 404 twice over. - Three error strings in cmd/erasure-sets.go, cmd/storage-errors.go and internal/config/errors.go that print an issue URL to operators. These are Go string literals inside rebrand-guard's brand allowlist, so the baseline is regenerated. The regeneration removes exactly those three entries and adds none; all twelve other protected sets, including the 9014 exported symbols, are byte-identical. - The transitional-naming disclaimers in README, README_ZH, SECURITY and CONTRIBUTING are dropped, since they no longer describe anything. Deliberately unchanged, all three because they exist to reject or freeze the old name rather than to point at it: - buildscripts/minio-upgrade.sh pins pgsty/minio@sha256:b6bfe72... - the frozen pre-rebrand image is the control group for the MinIO-to-Silo upgrade test. - helm-migration-guard rejects any rendered container still pulling pgsty/minio. - verify-rebrand.sh rejects the same in the delivery surfaces. Also unchanged: docs/config/README.md links to pgsty/mc/blob/master, and that repository's default branch really is still master. It moves when mc does. verify-rebrand.sh gains three assertions so this cannot silently regress: no source reference may name pgsty/minio outside the three allowlisted guards, no link may target pgsty/silo's master branch, and go.yml and vulncheck.yml must filter on main. Both new rejections were negative-tested - reintroducing a master branch filter and adding a pgsty/minio URL each fail the gate with the specific message. This commit assumes the rename actually happens. Until the GitHub branch and repository renames are executed, the links it introduces do not resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
139 lines
5.7 KiB
Markdown
139 lines
5.7 KiB
Markdown
# Bucket Versioning Design Guide [](https://hub.docker.com/r/pgsty/silo/)
|
|
|
|
## Description of `xl.meta`
|
|
|
|
`xl.meta` is a new self describing backend format used by Silo to support AWS S3 compatible versioning.
|
|
This file is the source of truth for each `version` at rest. `xl.meta` is a msgpack file serialized from a
|
|
well defined data structure. To understand `xl.meta` here are the few things to start with
|
|
|
|
`xl.meta` carries first 8 bytes an XL header which describes the current format and the format version,
|
|
allowing the unmarshaller's to automatically use the right data structures to parse the subsequent content in the stream.
|
|
|
|
### v1.0
|
|
|
|
| Entry | Encoding | Content
|
|
| ----------|-------------|----------------------------------------
|
|
| xlHeader | [4]byte | `'X', 'L', '2', ' '`
|
|
| xlVersion | [4]byte | `'1', ' ', ' ', ' '`
|
|
| xlMetaV2 | msgp object | All versions as single messagepack object
|
|
| [EOF] | |
|
|
|
|
### v1.1+
|
|
|
|
Version 1.1 added inline data, which will be placed after the metadata.
|
|
|
|
Therefore, the metadata is wrapped as a binary array for easy skipping.
|
|
|
|
| Entry | Encoding | Content
|
|
| ---------------|----------------|----------------------------------------
|
|
| xlHeader | [4]byte | `'X', 'L', '2', ' '`
|
|
| xlVersionMajor | uint16 | Major xl-meta version.
|
|
| xlVersionMinor | uint16 | Minor xl-meta version.
|
|
| xlMetaV2 | msgp bin array | Bin array with serialized metadata
|
|
| crc | msgp uint | Lower 32 bits of 64 bit xxhash of previous array contents (v1.2+ only)
|
|
| inline data | binary | Inline data if any, see Inline Data section for encoding.
|
|
| [EOF] | |
|
|
|
|
## v1.0-v1.2 Versions
|
|
|
|
`xl.meta` carries three types of object entries which designate the type of version object stored.
|
|
|
|
- ObjectType (default)
|
|
- LegacyObjectType (preserves existing deployments and older xl.json format)
|
|
- DeleteMarker (a versionId to capture the DELETE sequences implemented primarily for AWS spec compatibility)
|
|
|
|
A sample msgpack-JSON `xl.meta`, you can debug the content inside `xl.meta` using [xl-meta.go](https://github.com/pgsty/silo/tree/main/docs/debugging#decoding-metadata) program.
|
|
|
|
```json
|
|
{
|
|
"Versions": [
|
|
{
|
|
"Type": 1,
|
|
"V2Obj": {
|
|
"ID": "KWUs8S+8RZq4Vp5TWy6KFg==",
|
|
"DDir": "X3pDAFu8Rjyft7QD6t7W5g==",
|
|
"EcAlgo": 1,
|
|
"EcM": 2,
|
|
"EcN": 2,
|
|
"EcBSize": 10485760,
|
|
"EcIndex": 3,
|
|
"EcDist": [3, 4, 1, 2],
|
|
"CSumAlgo": 1,
|
|
"PartNums": [1],
|
|
"PartETags": [""],
|
|
"PartSizes": [314],
|
|
"PartASizes": [282],
|
|
"Size": 314,
|
|
"MTime": 1591820730,
|
|
"MetaSys": {
|
|
"X-Minio-Internal-Server-Side-Encryption-S3-Kms-Key-Id": "bXktbWluaW8ta2V5",
|
|
"X-Minio-Internal-Server-Side-Encryption-S3-Kms-Sealed-Key": "ZXlKaFpXRmtJam9pUVVWVExUSTFOaTFIUTAwdFNFMUJReTFUU0VFdE1qVTJJaXdpYVhZaU9pSkJMMVZzZFVnelZYVjZSR2N6UkhGWUwycEViRmRCUFQwaUxDSnViMjVqWlNJNklpdE9lbkJXVWtseFlWSlNVa2t2UVhNaUxDSmllWFJsY3lJNklrNDBabVZsZG5WU1NWVnRLMFoyUWpBMVlYTk9aMU41YVhoU1RrNUpkMDlhTkdKa2RuaGpLMjFuVDNnMFFYbFJhbE15V0hkU1pEZzNRMk54ZUN0SFFuSWlmUT09",
|
|
"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "REFSRXYyLUhNQUMtU0hBMjU2",
|
|
"X-Minio-Internal-Server-Side-Encryption-Iv": "bW5YRDhRUGczMVhkc2pJT1V1UVlnbWJBcndIQVhpTUN1dnVBS0QwNUVpaz0=",
|
|
"X-Minio-Internal-Server-Side-Encryption-S3-Sealed-Key": "SUFBZkFPeUo5ZHVVSEkxYXFLU0NSRkJTTnM0QkVJNk9JWU1QcFVTSXFhK2dHVThXeE9oSHJCZWwwdnRvTldUNE8zS1BtcWluR0cydmlNNFRWa0N0Mmc9PQ=="
|
|
},
|
|
"MetaUsr": {
|
|
"content-type": "application/octet-stream",
|
|
"etag": "20000f00f58c508b40720270929bd90e9f07b9bd78fb605e5432a67635fc34722e4fc53b1d5fab9ff8400eb9ded4fba2"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### v1.3+ versions
|
|
|
|
Version 1.3 introduces changes to help with [faster metadata reads and updates](https://blog.min.io/minio-versioning-metadata-deep-dive/)
|
|
|
|
| Entry | Encoding | Content
|
|
| ----------------|-----------------------------|----------------------------------------
|
|
| xlHeaderVersion | msgp uint | header version identifier
|
|
| xlMetaVersion | msgp uint | metadata version identifier
|
|
| versions | msgp int | Number of versions following
|
|
| header_1 | msgp bin array | Header of version 1
|
|
| metadata_1 | msgp bin array | Metadata of version 1
|
|
| ...header_n | msgp bin array | Header of last version
|
|
| ...metadata_n | msgp bin array | Metadata of last version
|
|
|
|
Each header contains a mspg array (tuple) encoded object:
|
|
|
|
xlHeaderVersion version == 1:
|
|
|
|
```
|
|
//msgp:tuple xlMetaV2VersionHeader
|
|
type xlMetaV2VersionHeader struct {
|
|
VersionID [16]byte // Version UUID, raw.
|
|
ModTime int64 // Unix nanoseconds.
|
|
Signature [4]byte // Signature of metadata.
|
|
Type uint8 // Type if the version
|
|
Flags uint8
|
|
}
|
|
```
|
|
|
|
The following flags are defined:
|
|
|
|
```
|
|
const (
|
|
FreeVersion = 1 << 0
|
|
UsesDataDir = 1 << 1
|
|
InlineData = 1 << 2
|
|
)
|
|
```
|
|
|
|
The "Metadata" section contains a single version, encoded in similar fashion as each version in the `Versions` array
|
|
of the previous version.
|
|
|
|
## Inline Data
|
|
|
|
Inline data is optional. If no inline data is present, it is encoded as 0 bytes.
|
|
|
|
| Entry | Encoding | Content
|
|
| --------------------|-----------------------------|----------------------------------------
|
|
| xlMetaInlineDataVer | byte | version identifier
|
|
| id -> data | msgp `map[string][]byte` | Map of string id -> byte content
|
|
|
|
Currently only xlMetaInlineDataVer == 1 exists.
|
|
|
|
The ID is the string encoded Version ID of which the data corresponds.
|