mirror of
https://github.com/pgsty/minio.git
synced 2026-09-17 15:58:26 +03:00
revert: remove access-frequency ILM tiering (#60)
Reverse the first-parent diff of a3df317ae0,
including the feature branch compatibility and mover follow-up fixes.
Retain the independent multi-pool correctness fixes from #178 and migrate
their shared test fixture away from access-tier code.
Tolerate retired ILM keys and XML, read old v9 statistics while writing v8,
and document migration without moving objects or rewriting their metadata.
Include regression coverage using a historical scanner/writer v9 fixture.
Signed-off-by: Feng Ruohang <rh@vonng.com>
This commit is contained in:
Feng Ruohang
@@ -226,149 +226,11 @@ aws s3api restore-object --bucket srcbucket \
|
||||
|
||||
Note that transition event notification is a Silo extension.
|
||||
|
||||
## 5. Access-based tiering between server pools
|
||||
|
||||
Silo can move frequently read objects to a faster server pool and return them
|
||||
to a slower pool after they become idle. This is different from remote ILM
|
||||
transition: the object remains a native local object and all versions move
|
||||
together.
|
||||
|
||||
Access tiering requires at least two server pools. Pool indices follow the
|
||||
order on the server command line:
|
||||
|
||||
~~~sh
|
||||
silo server /srv/nvme{1...4} /srv/hdd{1...8}
|
||||
# pool 0 (fast) pool 1 (slow)
|
||||
~~~
|
||||
|
||||
Server pools are erasure-coding expansion units, not individual drives. Each
|
||||
pool should consist of internally homogeneous media.
|
||||
|
||||
The feature is disabled by default. Configure the topology and safety limits
|
||||
with `mc admin config set`; ILM is a dynamic subsystem, so this applies without
|
||||
a restart. Environment variables (`MINIO_ILM_ACCESS_TIERING`,
|
||||
`MINIO_ILM_ACCESS_POOLS`, `MINIO_ILM_ACCESS_MAX_SIZE`,
|
||||
`MINIO_ILM_ACCESS_PROMOTE_WATERMARK`, `MINIO_ILM_ACCESS_BIN_WIDTH`,
|
||||
`MINIO_ILM_ACCESS_BINS`, `MINIO_ILM_ACCESS_FLUSH`,
|
||||
`MINIO_ILM_ACCESS_MIN_RESIDENCY`, `MINIO_ILM_ACCESS_WORKERS`,
|
||||
`MINIO_ILM_ACCESS_MAX_TRACKED`) are read at process start and override the
|
||||
stored config.
|
||||
|
||||
~~~sh
|
||||
mc admin config set local ilm \
|
||||
access_tiering=on \
|
||||
access_pools="0,1" \
|
||||
access_max_size="2TiB" \
|
||||
access_promote_watermark=85 \
|
||||
access_bin_width=1m \
|
||||
access_bins=12 \
|
||||
access_flush=1m \
|
||||
access_min_residency=24h \
|
||||
access_workers=10 \
|
||||
access_max_tracked=1000000
|
||||
~~~
|
||||
|
||||
The pool list is ordered hottest to coldest. With three or more pools,
|
||||
promotion always targets the first index and demotion always targets the last;
|
||||
intermediate pools are not hop targets. An access_max_size value of zero means
|
||||
no cluster-wide logical-byte cap. Promotion also stops when the hottest pool
|
||||
reaches access_promote_watermark.
|
||||
|
||||
New PUTs still land via the usual free-space pool picker; they are not steered
|
||||
onto the cold pool. Size the capacity pool larger than the hot pool so new
|
||||
objects tend to land there.
|
||||
|
||||
Add an AccessTransition to the bucket lifecycle XML:
|
||||
|
||||
~~~xml
|
||||
<LifecycleConfiguration>
|
||||
<AccessTierQuota>500GiB</AccessTierQuota>
|
||||
<Rule>
|
||||
<ID>hot-logs</ID>
|
||||
<Status>Enabled</Status>
|
||||
<Filter>
|
||||
<And>
|
||||
<Prefix>logs/</Prefix>
|
||||
<ObjectSizeGreaterThan>65536</ObjectSizeGreaterThan>
|
||||
</And>
|
||||
</Filter>
|
||||
<AccessTransition>
|
||||
<Window>10m</Window>
|
||||
<PromoteAfterAccesses>100</PromoteAfterAccesses>
|
||||
<DemoteAfterAccesses>5</DemoteAfterAccesses>
|
||||
<DemoteAfterIdle>24h</DemoteAfterIdle>
|
||||
</AccessTransition>
|
||||
</Rule>
|
||||
</LifecycleConfiguration>
|
||||
~~~
|
||||
|
||||
This promotes a matching object after 100 successful GETs in 10 minutes. An
|
||||
object becomes eligible to return to the coldest configured pool only after
|
||||
access tiering has already moved it (the `x-minio-internal-ilm-atier` stamp),
|
||||
it has stayed put for the server-wide minimum residency, it has been idle at
|
||||
least 24 hours, and it has no more than 5 GETs in the window. Objects that
|
||||
landed on the hot pool via a normal PUT never demote. Prefix, tag, and
|
||||
object-size lifecycle filters are honored.
|
||||
|
||||
Access-based moves are a parallel path: they are not lifecycle `Eval` actions
|
||||
and do not appear in S3 prediction headers. If the same object is also due
|
||||
for age-based remote `Transition` or expiry, that scanner action wins and
|
||||
demotion discovery is skipped for that pass; promotions still run from the
|
||||
GET tracker. Site replication copies expiry rules only, same as remote
|
||||
Transition, so AccessTransition stays local to the cluster.
|
||||
|
||||
AccessTierQuota is an optional bucket-wide cap. Promotion checks, in order:
|
||||
|
||||
1. hot-pool used percentage;
|
||||
2. cluster-wide access_max_size;
|
||||
3. bucket AccessTierQuota.
|
||||
|
||||
Demotion is not blocked by these caps and is processed before promotion.
|
||||
Access moves pause during rebalance or decommission, never target a suspended
|
||||
pool, skip remotely transitioned objects and objects with excessive version
|
||||
counts, and recheck eligibility while holding the object namespace lock.
|
||||
Moves also lock the source and destination write locations. All lock servers
|
||||
for those locations must be reachable; otherwise the background move is
|
||||
deferred. Ordinary S3 requests keep their existing quorum requirements.
|
||||
|
||||
Each move preserves version IDs, delete markers, ETags, checksums, encryption
|
||||
and user metadata. A retry copies only missing versions and retains versions
|
||||
already at the destination. If the same version ID has conflicting metadata
|
||||
in the two pools, the move is skipped and both copies are left intact. Source
|
||||
data is removed only after the complete version stack exists at the destination.
|
||||
|
||||
The hit counter is intentionally best effort. Only successfully served GET
|
||||
requests count; HEAD requests do not. Counters are merged across nodes and
|
||||
bounded by access_max_tracked. A rule window longer than
|
||||
access_bin_width multiplied by access_bins is clamped to retained history.
|
||||
|
||||
AccessTransition and AccessTierQuota are Silo lifecycle extensions. A stock
|
||||
AWS SDK that reads and rewrites the lifecycle configuration may discard
|
||||
unknown fields. Use a raw signed S3 PUT lifecycle request, such as
|
||||
[setup_ilm_access_tiering.sh](setup_ilm_access_tiering.sh), when installing
|
||||
the rule. Save the XML above as `rule.xml`, then run:
|
||||
|
||||
~~~sh
|
||||
AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \
|
||||
./setup_ilm_access_tiering.sh http://127.0.0.1:9000 testbucket us-east-1 rule.xml
|
||||
~~~
|
||||
|
||||
Access-tier activity is exposed under /minio/metrics/v3/ilm, including move
|
||||
counts, moved bytes, queue depth, hot bytes per bucket, failed moves, dropped
|
||||
GET samples, and separate skip counters for each capacity limit.
|
||||
|
||||
To stop scheduling moves, set `access_tiering=off` (and remove any environment
|
||||
override). Objects already moved remain in their current pools and stay
|
||||
accessible; disabling the feature does not move them back. Before downgrading
|
||||
to a release without access tiering, disable it and let active moves finish.
|
||||
The object storage format is unchanged. The data-usage cache advances from
|
||||
v8 to v9: this release reads both, but an older binary discards v9 caches and
|
||||
rebuilds usage statistics through the scanner. Usage and quota statistics can
|
||||
therefore take time to repopulate after a downgrade. Keep a copy of lifecycle
|
||||
XML containing Silo extensions, since an older binary may omit those fields
|
||||
when rewriting a lifecycle rule.
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [MinIO Go client API reference (S3-compatible SDK)](https://pkg.go.dev/github.com/minio/minio-go/v7)
|
||||
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
|
||||
|
||||
## Access-frequency tiering removal
|
||||
|
||||
The opt-in cross-pool access-frequency extension has been removed. For upgrades from a main/snapshot build that included it, see [the migration notes](access-tiering-removal.md). Ordinary lifecycle expiration and remote-tier transitions remain supported.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Removing access-frequency pool tiering
|
||||
|
||||
PR #60 introduced an opt-in scheduler that moved objects between local server pools according to GET frequency. It and its feature-specific fixes have been removed. This does not remove ordinary lifecycle expiration, transitions to remote tiers, rebalance, decommission, or the general multi-pool correctness fixes from PR #178.
|
||||
|
||||
The published Server 20260903 predates this feature. These instructions concern main/snapshot deployments that included #60; upgrading from the published version does not require access-tier configuration cleanup.
|
||||
|
||||
## Before upgrading a build with access tiering
|
||||
|
||||
1. Save a copy of the server ILM configuration and each affected bucket's lifecycle XML. Use an API client that preserves the nonstandard XML; do not rely on a client model that silently omits unknown elements.
|
||||
2. On the old server, set `ilm access_tiering=off` and remove or disable any access-tier environment overrides. Allow in-progress moves to finish before replacing nodes. This reduces movement intermediate states; the removal itself changes no storage RPC protocol.
|
||||
3. Remove top-level `AccessTierQuota` and rule-level `AccessTransition` elements. **Delete rules whose only action was `AccessTransition`**. For a mixed rule, retain its filter, status, ID and ordinary expiration/transition actions. An access-only rule loads harmlessly after upgrade, but becomes an actionless rule and fails validation on the next lifecycle edit. If no rules remain, delete the lifecycle configuration through the S3 API.
|
||||
4. Use a coordinated maintenance window: stop the deployment, install the same new binary on every node, then restart all nodes. The existing bootstrap check compares binary checksums; in a four-node test the first new node could not finish starting among three old nodes. Do not assume that an unchanged RPC protocol permits replacing one node at a time and waiting for it to become ready. This removal does not relax that check. Apply the same environment changes on every node: bootstrap also compares server environment settings, so removing an old override on only some nodes can block startup even with matching binaries. The check runs only during startup and is not a safety guarantee for nodes already running different binaries.
|
||||
5. After restarting, verify object reads, bucket listing, ILM worker settings and a lifecycle edit. Complete distributed upgrade acceptance for the exact binaries before production rollout.
|
||||
|
||||
## What happens to stored state
|
||||
|
||||
| State | Behavior after removal |
|
||||
| --- | --- |
|
||||
| Ten old ILM keys | `access_tiering`, `access_pools`, `access_max_size`, `access_promote_watermark`, `access_bin_width`, `access_bins`, `access_flush`, `access_min_residency`, `access_workers`, `access_max_tracked` are accepted but ignored. Existing transition/expiration worker settings are preserved. |
|
||||
| Admin configuration | Deprecated keys may still appear in `mcli admin config get ilm`; setting them may succeed but has no effect, even with `access_tiering=on`. Remove obsolete environment settings from deployment manifests. |
|
||||
| Lifecycle XML | `AccessTierQuota` and `AccessTransition` are ignored when read and omitted when re-encoded. The same parser handles new PUT requests, so these extensions are also silently discarded there; access-only rules still fail action validation. |
|
||||
| Data-usage cache | Both v8 and v9 caches are read, preserving ordinary counts, sizes, histograms and remote-tier statistics. The retired hot-tier byte count is discarded; subsequent writes use v8. No feature-driven full statistics rebuild is required. |
|
||||
| Objects already moved | Remain in their current pools with the same versions and timestamps. There is no bulk move-back or object metadata rewrite. |
|
||||
| Internal leftovers | `x-minio-internal-ilm-atier` and `.minio.sys/config/ilm/access/` counter objects may remain unused. They do not require a cleanup service or an object scan. |
|
||||
|
||||
Interrupted rebalance/decommission can leave the same version in more than one pool independently of access tiering. Removing the scheduler does not remove such existing copies. General Object Lock, conditional-delete, metadata reconciliation and shared remote-tier reference protections remain in place.
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Install a lifecycle XML document without an SDK normalizing away Silo's
|
||||
# AccessTransition and AccessTierQuota extension elements.
|
||||
set -eu
|
||||
|
||||
if [ "$#" -ne 4 ]; then
|
||||
echo "usage: AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... $0 ENDPOINT BUCKET REGION LIFECYCLE_XML" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
endpoint=$1
|
||||
bucket=$2
|
||||
region=$3
|
||||
lifecycle_file=$4
|
||||
|
||||
: "$AWS_ACCESS_KEY_ID"
|
||||
: "$AWS_SECRET_ACCESS_KEY"
|
||||
|
||||
if [ ! -r "$lifecycle_file" ]; then
|
||||
echo "cannot read lifecycle document: $lifecycle_file" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
content_md5=$(openssl dgst -md5 -binary "$lifecycle_file" | openssl base64)
|
||||
endpoint=$(printf '%s' "$endpoint" | sed 's:/*$::')
|
||||
|
||||
curl --fail-with-body --silent --show-error \
|
||||
--request PUT \
|
||||
--aws-sigv4 "aws:amz:$region:s3" \
|
||||
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
|
||||
--header "Content-MD5: $content_md5" \
|
||||
--header "Content-Type: application/xml" \
|
||||
--data-binary "@$lifecycle_file" \
|
||||
"$endpoint/$bucket?lifecycle"
|
||||
|
||||
echo "installed lifecycle configuration on $bucket"
|
||||
@@ -227,17 +227,6 @@ For deployments with [bucket](https://silo.pgsty.com/administration/bucket-repli
|
||||
| `minio_node_ilm_transition_active_tasks` | Number of active ILM transition tasks. |
|
||||
| `minio_node_ilm_transition_pending_tasks` | Number of pending ILM transition tasks in the queue. |
|
||||
| `minio_node_ilm_transition_missed_immediate_tasks` | Number of missed immediate ILM transition tasks. |
|
||||
| `minio_node_ilm_access_tier_active_tasks` | Number of active access-tier pool moves. |
|
||||
| `minio_node_ilm_access_tier_pending_tasks` | Number of pending access-tier pool moves. |
|
||||
| `minio_node_ilm_access_tier_promotions_total` | Total objects promoted by access-tier ILM. |
|
||||
| `minio_node_ilm_access_tier_demotions_total` | Total objects demoted by access-tier ILM. |
|
||||
| `minio_node_ilm_access_tier_bytes_moved_total` | Total logical bytes moved by access-tier ILM. |
|
||||
| `minio_node_ilm_access_tier_failures_total` | Total failed access-tier ILM moves. |
|
||||
| `minio_node_ilm_access_tier_skipped_watermark_total` | Promotions skipped because the hot pool reached its watermark. |
|
||||
| `minio_node_ilm_access_tier_skipped_max_size_total` | Promotions skipped because the cluster hot-tier size cap was reached. |
|
||||
| `minio_node_ilm_access_tier_skipped_bucket_quota_total` | Promotions skipped because the bucket hot-tier quota was reached. |
|
||||
| `minio_node_ilm_access_tier_hot_bytes` | Logical bytes currently accounted to the hot tier, labeled by bucket. |
|
||||
| `minio_node_ilm_access_tier_samples_dropped_total` | GET samples dropped because the access tracker queue was full. |
|
||||
| `minio_node_ilm_versions_scanned` | Total number of object versions checked for ilm actions since server start. |
|
||||
| `minio_node_ilm_action_count_delete_action` | Total action outcome of lifecycle checks since server start for deleting object |
|
||||
| `minio_node_ilm_action_count_delete_version_action` | Total action outcome of lifecycle checks since server start for deleting a version |
|
||||
|
||||
Reference in New Issue
Block a user