Two coupled changes that must land together, because the same files carry both:
the server now identifies itself as Silo, and every path that would have called
home to a MinIO-operated service is closed.
Product identity
- build-constants.go: store name, UA name and startup banner become Silo. The
Go identifiers (MinioStoreName, MinioBannerName, ...) keep their names on
purpose - renaming exported symbols would churn the compatibility surface for
a cosmetic gain, and the rebrand guard freezes that surface.
- main.go, server-startup-msg.go, ftp-server.go and the user-visible log, help
and error strings across cmd/ and internal/ switch to Silo. Original MinIO
copyright, LICENSE, NOTICE and CREDITS are untouched; --version now prints
the upstream copyright, the pgsty modification notice, and the trademark
policy's approved "based on MinIO technology" attribution.
- api-headers.go: the HTTP Server header becomes "Silo". This is the one
externally observable identity change, so TestCommonHeadersUseSiloProductName
pins it - probes that sniff for "MinIO" must move to capability detection.
- Prometheus metric HELP strings keep their MinIO wording. They are part of the
metrics contract the guard protects, not product copy.
Configuration directory
- config-dir.go: new installs use ~/.silo. If only ~/.minio exists it is still
read, with a one-time notice and no files moved. If both exist ~/.silo wins
and an ambiguity warning is emitted; an explicit --config-dir always wins.
Covered by TestSelectDefaultConfigDir. The internal .minio.sys layout is
never renamed - this rule applies to the user config directory only.
Upstream service lockdown
- globalInplaceUpdateDisabled is now true at initialization rather than being
set from MINIO_UPDATE. common-main.go still parses MINIO_UPDATE so upgrading
nodes do not fail on an unknown key, but warns that the value is ignored;
there is no way to re-enable the updater. TestInplaceUpdateCannotBeEnabled
guards that. Without this, an admin with mc could have overwritten
/usr/bin/silo with an upstream MinIO binary.
- verifyBinary and commitBinary refuse early; the ServerUpdate v1/v2 admin
routes and the peer-rest update endpoints stay registered and keep returning
the existing programmatic error, so clients see a stable failure rather than
a 404.
- MinioReleaseBaseURL and defaultMinisignPubkey are emptied: no dl.min.io
download root, and upstream's minisign key is no longer a trust root for
anything this fork ships.
- cmd/callhome.go is deleted and internal/config/subnet/ is reduced to parsing
its old keys and reporting that the integration is disabled. config-current.go
warns instead of failing when callhome or SUBNET settings are present, so an
upgraded node with those keys still starts.
- internal/config/errors.go replaces the MinIO Slack and support entry points
with Silo documentation and issue links. Error codes and programmatic fields
are unchanged.
Verified: the compatibility baseline is unchanged except for the deliberate
removal of the /api/health/upload SUBNET route; go build, go vet and the full
cmd/ and internal/ unit suites pass; a locally built binary starts, serves
S3/Admin/metrics on the unchanged /minio/* routes, answers with Server: Silo,
and falls back to a pre-existing ~/.minio with the expected notice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
Optionally allows customers to enable
- Enable an external cache to catch GET/HEAD responses
- Enable skipping disks that are slow to respond in GET/HEAD
when we have already achieved a quorum
This PR is a continuation of the previous change instead
of returning an error, instead trigger a spot heal on the
'xl.meta' and return only after the healing is complete.
This allows for future GETs on the same resource to be
consistent for any version of the object.
Main motivation is move towards a common backend format
for all different types of modes in MinIO, allowing for
a simpler code and predictable behavior across all features.
This PR also brings features such as versioning, replication,
transitioning to single drive setups.
it would seem like using `bufio.Scan()` is very
slow for heavy concurrent I/O, ie. when r.Body
is slow , instead use a proper
binary exchange format, to marshal and unmarshal
the LockArgs datastructure in a cleaner way.
this PR increases performance of the locking
sub-system for tiny repeated read lock requests
on same object.
```
BenchmarkLockArgs
BenchmarkLockArgs-4 6417609 185.7 ns/op 56 B/op 2 allocs/op
BenchmarkLockArgsOld
BenchmarkLockArgsOld-4 1187368 1015 ns/op 4096 B/op 1 allocs/op
```
we will allow situations such as
```
a/b/1.txt
a/b
```
and
```
a/b
a/b/1.txt
```
we are going to document that this usecase is
not supported and we will never support it, if
any application does this users have to delete
the top level parent to make sure namespace is
accessible at lower level.
rest of the situations where the prefixes get
created across sets are supported as is.
reduce the page-cache pressure completely by moving
the entire read-phase of our operations to O_DIRECT,
primarily this is going to be very useful for chatty
metadata operations such as listing, scanner, ilm, healing
like operations to avoid filling up the page-cache upon
repeated runs.
currently crawler waits for an entire readdir call to
return until it processes usage, lifecycle, replication
and healing - instead we should pass the applicator all
the way down to avoid building any special stack for all
the contents in a single directory.
This allows for
- no need to remember the entire list of entries per directory
before applying the required functions
- no need to wait for entire readdir() call to finish before
applying the required functions
Rewrite parentIsObject() function. Currently if a client uploads
a/b/c/d, we always check if c, b, a are actual objects or not.
The new code will check with the reverse order and quickly quit if
the segment doesn't exist.
So if a, b, c in 'a/b/c' does not exist in the first place, then returns
false quickly.
reference format should be source of truth
for inconsistent drives which reconnect,
add them back to their original position
remove automatic fix for existing offline
disk uuids
add a hint on the disk to allow for tracking fresh disk
being healed, to allow for restartable heals, and also
use this as a way to track and remove disks.
There are more pending changes where we should move
all the disk formatting logic to backend drives, this
PR doesn't deal with this refactor instead makes it
easier to track healing in the future.
- Implement a new xl.json 2.0.0 format to support,
this moves the entire marshaling logic to POSIX
layer, top layer always consumes a common FileInfo
construct which simplifies the metadata reads.
- Implement list object versions
- Migrate to siphash from crchash for new deployments
for object placements.
Fixes#2111
instead perform a liveness check call to
verify if server is online and print relevant
errors.
Also introduce a StorageErr string error type
instead of errors.New() deprecate usage of
VerifyFileError, DeleteFileError for gob,
change in datastructure also requires bump in
storage REST version to v13.
Fixes#8811
- Heal if the part.1 is truncated from its original size
- Heal if the part.1 fails while being verified in between
- Heal if the part.1 fails while being at a certain offset
Other cleanups include make sure to flush the HTTP responses
properly from storage-rest-server, avoid using 'defer' to
improve call latency. 'defer' incurs latency avoid them
in our hot-paths such as storage-rest handlers.
Fixes#8319
posix.VerifyFile() doesn't know how to check if a file
is corrupted if that file is empty. We do have the part
size in xl.json so we pass it to VerifyFile to return
an error so healing empty parts can work properly.
In scenario 1
```
- bucket/object-prefix
- bucket/object-prefix/object
```
Server responds with `XMinioParentIsObject`
In scenario 2
```
- bucket/object-prefix/object
- bucket/object-prefix
```
Server responds with `XMinioObjectExistsAsDirectory`
Fixes#6566
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.
This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.
Some design details and restrictions:
- Objects are distributed using consistent ordering
to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
requirement, you can start with multiple
such sets statically.
- Static sets set of disks and cannot be
changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
slower since List happens on all servers,
and is merged at this sets layer.
Fixes#5465Fixes#5464Fixes#5461Fixes#5460Fixes#5459Fixes#5458Fixes#5460Fixes#5488Fixes#5489Fixes#5497Fixes#5496
Implement an offline mode for remote storage to cache the
offline status of a node in order to prevent network calls
that are bound to fail. After a time interval an attempt
will be made to restore the connection and mark the node
as online if successful.
Fixes#4183
This is an enhancement to the XL/distributed-XL mode. FS mode is
unaffected.
The ReadFileWithVerify storage-layer call is similar to ReadFile with
the additional functionality of performing bit-rot checking. It
accepts additional parameters for a hashing algorithm to use and the
expected hex-encoded hash string.
This patch provides significant performance improvement because:
1. combines the step of reading the file (during
erasure-decoding/reconstruction) with bit-rot verification;
2. limits the number of file-reads; and
3. avoids transferring the file over the network for bit-rot
verification.
ReadFile API is implemented as ReadFileWithVerify with empty hashing
arguments.
Credits to AB and Harsha for the algorithmic improvement.
Fixes#4236.
Disks when are offline for a long period of time, we should
ignore the disk after trying Login upto 5 times.
This is to reduce the network chattiness, this also reduces
the overall time spent on `net.Dial`.
Fixes#3286
These messages based on our prep stage during XL
and prints more informative message regarding
drive information.
This change also does a much needed refactoring.
Feng Ruohang