minio/docs/site-replication/iam-revocations.md

Durable IAM revocations in site replication

SILO retains the version of a deleted IAM record so an offline site cannot restore an older identity or grant when it reconnects. This covers built-in users, service accounts, groups, policy documents, and policy mappings in their actual user/STS-parent/group namespaces. It also revokes the deleted built-in user's older service accounts, STS credentials and group grants across deliberate same-name recreation.

Ordering and persistence

Deletion records occupy the original IAM configuration paths. They contain the originating timestamp, Deleted, and, where required, RevokedBefore; identity deletion records contain no secret key or session token. Normal IAM listings and authorization hide deleted records. Object storage and etcd both serialize each path's version comparison and write with a distributed lock. The source timestamp is persisted without replacing it with the receiving node's clock.

Older events cannot overwrite a newer revision. At an identical timestamp, a deletion wins over a live record. A local deliberate recreation receives a version newer than the stored deletion. An older user/group deletion arriving after recreation retains its revocation boundary while preserving the newer live record. Receiving an already-applied tombstone does not rewrite or advance it. These rules also apply after cold loading persistent IAM state.

The user or group revision is the commit point of deletion. Cleanup of mappings and children follows that commit; cleanup failure cannot undo it. The API returns the cleanup error and still notifies sibling nodes to reload the committed state. Such an error does not mean the identity is still active. Retrying the operation is safe. When the API returns a committed-cleanup error, the admin handler does not send its immediate cross-site hook; cross-site propagation relies on the normal deletion-healing retry. Sibling deletion notifications reload current shared state, so a delayed notification cannot delete a subsequently recreated identity. Etcd siblings also receive persistent changes through watches. Failed notifications/watches remain eventual propagation, not a distributed instantaneous revocation transaction.

Keep node clocks synchronized and monitor offsets. Ordering uses source wall clock timestamps, with monotonic advancement for local writes to the same path. It does not establish causal order between concurrent writers at different sites or resolve conflicting live updates with identical timestamps deterministically.

Users, child credentials and groups

A recreated user retains RevokedBefore. New service accounts and built-in STS issuances carry a signed siloParentRevocation claim identifying the parent boundary known at issuance. Editing or replaying an old child does not update this claim. Old children remain invalid even when their own update timestamp is newer than the parent deletion; children issued for the recreated parent remain valid. Claims are read from verified tokens on credential load/write.

A newer replicated service-account snapshot can replace an older service with the same access key, including a deliberate change of owner or secret. Local duplicate creates remain rejected. Snapshots preserve disabled status and the service's own revocation boundary, so earlier mappings cannot attach to the new service. Equal-version retries reload the committed identity without rewriting it. Periodic live healing compares source versions even when the public status summary is unchanged, and includes disabled identities as healing sources. Service snapshots also preserve their absolute expiration. The receiving site does not reapply the local minimum lifetime for a newly issued credential. A newer already-expired snapshot still supersedes the old key, is denied by authentication, and is collected into a durable service tombstone by normal loading. Cache/claims loading failures are returned for retry, not acknowledged; after a committed replacement the stale cached secret is evicted immediately. Collisions with an existing built-in or cached STS identity report an error and require an explicit administrative resolution; replication cannot change its credential kind. Concurrent conflicting service updates with exactly the same timestamp can retain different winners at different sites; the status summary does not resolve that case.

Each group member has its own MemberGrants timestamp. Changing another member or the group's enabled status does not reissue everyone else's grants. Effective membership requires the grant to be newer than both the user's and the group's retained boundaries. Listings and policy evaluation use the same effective membership. Peer snapshots preserve grant times, including unknown legacy grant times; they cannot treat a recent snapshot time as a fresh grant to a revoked identity. A new explicit administrative group grant can restore access.

This does not implement a general conflict-resolution protocol for all group membership edits. In particular, the inherited live-group snapshot merge adds members and does not reconcile a missed ordinary member removal. Removing a member from a live group during a site outage is a separate known limitation; do not infer that this change resolves it. User/group deletion boundaries and same-name recreation are covered here.

Retention and expiration

Permanent identities, groups, policy documents and mappings have no automatic tombstone TTL. A disconnected peer or an old backup may return arbitrarily late. Successful replay acknowledgements are an optimization, not permission to garbage-collect this history.

Natural expiration of an immutable STS token physically removes its token-key record and any legacy token-key mapping, without generating a permanent tombstone. An early STS revocation is retained until that token's expiration plus the existing clock-skew allowance. Replaying the same revoked token with a later event timestamp cannot recreate it. Etcd uses an expiration lease; object storage collects expired STS tombstones during its existing credential loading/purge. A record with unknown expiration is retained conservatively. Cleanup writes are best effort and use a short lock budget. If one fails, that load stops optional reclamation, reports the error and still loads healthy users; expired credentials stay denied and retain their existing durable version. The next load retries. Healthy cleanup has no fixed record quota. The reusable STS parent policy mapping is not assigned the token's TTL by deletion cleanup. External-IDP disablement is an early revocation, not natural token expiration, and its cached STS and service accounts are included in cleanup. Expiring service accounts retain a durable revision because their access keys are reusable and an older version might have no expiration.

Direct per-token RevokeTokens delivery between sites is not a new guarantee of this change. The guarantee for built-in parent deletion follows from the durable parent boundary, including children not currently present in the deleting node's cache.

Healing, failures and operational cost

Each process starts one healing loop. Losing its distributed leadership lease pauses work until leadership is reacquired; it does not permanently terminate healing. Configuration reloads do not create extra loops. The 30-second interval starts after leadership is acquired and after each completed pass. Initial lock retries and endpoint recovery can add further delay; it is not a convergence SLA.

The normal IAM loaders maintain an in-memory index of deletion records and retained boundaries, without secrets. Healing uses this index; it does not add a second full walk of config/iam/ every cycle. Existing full IAM loading still scans persistent records, including tombstones, at startup and on refresh.

Each peer receives batches of at most 128 records. The sender remembers which path/version each peer acknowledged. Unrelated new changes at either site do not reset that progress. A failed batch remains pending while later independent batches can progress; a lost response may cause safe idempotent replay. A pass has a bounded duration, and its successful acknowledgements survive that timeout. The protocol reports each node name and process instance. Switching between known node instances behind a load balancer preserves acknowledgements. A new node instance conservatively invalidates prior acknowledgements once; repeated switches among those known instances do not reset progress. Restore persistent state only with the affected processes stopped, so a restore cannot reuse an old process acknowledgement.

Steady-state healing still traverses/sorts the retained in-memory set and checks the peer's protocol status. It suppresses repeated deletion PUTs once acknowledged. Memory use scales with retained paths and peers; startup storage reads scale with history. This release does not provide general history compaction. After upgrading, older live records whose receiving sites originally assigned different timestamps can require an initial reconciliation wave. Allow for its storage writes and sibling notifications when planning the maintenance window.

The cluster IAM metrics include revocation_records, revocation_heal_failures, revocation_heal_duration_millis, and revocation_heal_last_success_timestamp_seconds. Errors are also logged. A nominal 30-second scheduler interval is not a convergence deadline: outages, large backlogs, lock contention and failed requests can require more passes.

Cached credential lookup checks the in-memory parent revision index and performs no additional storage read. STS issuance and cold credential loading still consult the persistent parent revision. Revision I/O and distributed lock waits release the IAM cache lock while a separate local writer mutex preserves write order. These operations have bounded contexts, including etcd lock and lease cleanup.

Protocol and supported upgrade

The server-owned versioned route is /minio/admin/v3/site-replication/peer/iam-revisions. It carries source versions, member grant times and distinct user/group revocation items without changing the admin client SDK or S3 API. Older servers reject this route. The sender reports the failure and does not fall back to a route that would discard the metadata. The existing legacy IAM route remains readable for best-effort compatibility; this does not confer the new guarantees on an older peer.

All participating servers must be upgraded for the guarantee in this document. Mixed old/new nodes sharing an IAM backend and rolling downgrade are unsupported: older binaries do not interpret tombstones or signed parent boundaries correctly. Use a maintenance window for coordinated upgrade:

1. Pause IAM changes and isolate any offline site or backup whose state is unknown.
2. Back up each site's complete IAM storage and required encryption material. A live IAM admin export omits deletion history and is not an adequate backup.
3. Stop all nodes sharing each site's IAM backend, replace their binaries, and restart them on the upgraded version. Complete this for every participating site before relying on the new revocation semantics.
4. Check IAM loading, site-replication errors and revocation convergence. Verify representative old credentials are denied and deliberately reissued ones work.
5. Resolve pre-upgrade revocations explicitly. Absence cannot reconstruct an already-lost deletion version: remove surviving old records on the sites that still have them, and rebuild stale offline peers from approved state before admitting them. Do not reconnect an unknown old snapshot just to discover its deleted credentials.

Credentials issued by an older server for a recreated parent lack the required signed boundary and must be reissued by an upgraded server. Parents without any retained revocation history preserve existing credential behavior.

A pristine built-in policy remains protected from local deletion. If an administrator explicitly overrides that policy and later deletes the override, the durable deletion now suppresses automatic recreation of the built-in policy on reload. This prevents reload from undoing the deletion. Restore the policy by an explicit policy-create operation if desired. Local deletion of a nonexistent policy remains idempotent and does not create a new tombstone; replicated unknown deletions retain their version.

For rollback, stop and isolate the affected sites and assess changes since the backup before restoring compatible state. Restoring an older backup can itself lose later revocations and requires reconciliation/rekeying before access is reopened. Do not delete tombstones online or convert only live IAM records to make an older binary start. Server, client, Console, package and deployment acceptance remain separate delivery gates of the maintained PGSTY stack.

Regression and performance checks

Focused coverage is in iam-revocation_test.go, iam-revision_test.go, iam-revision-lock_test.go, iam-revision-boundary_test.go, iam-replication-protocol_test.go, iam-credential-retention_test.go, iam-peer-reload_test.go, and iam-replay_test.go. Set SILO_TEST_IAM_REVOCATION_ETCD to a disposable etcd endpoint to include backend lifecycle/locking/boundary tests; they use isolated key namespaces.

BenchmarkIAMCachedCredential and BenchmarkIAMSetTempUser can be run against the pre-change source for a comparable local baseline. BenchmarkIAMRevisionConvergedHealing covers 1,000 and 10,000 retained records; it measures steady-state index/network work and asserts zero repeated PUTs. Its fake peer does not measure durable catch-up throughput. BenchmarkIAMColdLoadExpiredServices measures loading and cleanup with 100 or 1,000 expired reusable credentials; run it with -benchtime=1x. Use actual multi-site signed S3/STS tests and deployment-specific latency/scale measurements in addition to these component tests.