mirror of
https://github.com/pgsty/minio.git
synced 2026-11-11 10:05:34 +03:00
Compare commits
39 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 8587e061bb | |||
| 293ab78b4c | |||
| 67a90dfef4 | |||
| 62b8784428 | |||
| 5dc4bd6a1d | |||
| cf486df661 | |||
| 1d714834c5 | |||
| 14ecb222e7 | |||
| e51bff35a3 | |||
| 20b3bfb6e7 | |||
| 322cd8d2c1 | |||
| c3340ec7f5 | |||
| 7c91dc17b2 | |||
| 2f93e0c6b3 | |||
| 65acda0114 | |||
| c6949a38c2 | |||
| 80dd3e3ce1 | |||
| 89028732ab | |||
| 55bd906d30 | |||
| 296c9710b1 | |||
| f3b42b0eb8 | |||
| da8d8a54b1 | |||
| 2b62df876a | |||
| 518915d241 | |||
| ce5d191077 | |||
| b404f78764 | |||
| d138996983 | |||
| d52ba0290a | |||
| 8f3657dc52 | |||
| 7efdedd5ba | |||
| 63650a31a1 | |||
| 6a65b43076 | |||
| 3269d2214e | |||
| b3e69bca2f | |||
| 2c06b28dbb | |||
| a2ff3c2ecd | |||
| 67bc32f401 | |||
| 3c4a6296f9 | |||
| afa35d16e9 |
+34
-10
@@ -15,10 +15,11 @@
|
|||||||
In the Console, clearing the policy editor of an access key no longer removes
|
In the Console, clearing the policy editor of an access key no longer removes
|
||||||
the policy; enter `{}` to remove it. The fix does not bring back a policy
|
the policy; enter `{}` to remove it. The fix does not bring back a policy
|
||||||
that was removed, and a site of a replicated deployment that does not run the
|
that was removed, and a site of a replicated deployment that does not run the
|
||||||
fix still removes the policy when such an edit reaches it, so sites can
|
fix still removes the policy on such an edit that it takes itself or receives
|
||||||
differ. Upgrade every site, then review on each site the accounts that
|
from another site that does not run it, so sites can differ. Upgrade every
|
||||||
`svcacct info` reports with `Policy: implied`, and set the policy again on
|
site, then review on each site the accounts that `svcacct info` reports with
|
||||||
those that were meant to be restricted.
|
`Policy: implied`, and set the policy again on those that were meant to be
|
||||||
|
restricted.
|
||||||
|
|
||||||
- S3 `DeleteObjects` on a single pool stops at an entry whose name fails the
|
- S3 `DeleteObjects` on a single pool stops at an entry whose name fails the
|
||||||
object name check: the entry returns `XMinioInvalidObjectName` and is not
|
object name check: the entry returns `XMinioInvalidObjectName` and is not
|
||||||
@@ -31,13 +32,35 @@
|
|||||||
such as a bucket's configuration. The other entries of a request are deleted
|
such as a bucket's configuration. The other entries of a request are deleted
|
||||||
as before.
|
as before.
|
||||||
|
|
||||||
|
- Site replication sends a service account or access key as the record its
|
||||||
|
site has stored, whole: for a create, for every edit, at the first sync of a
|
||||||
|
new site and when healing. IAM export writes that record too. Until now an
|
||||||
|
edit was sent as the fields it changed. A peer that an earlier edit had not
|
||||||
|
reached, or reached late, kept the old secret key, status or inline policy
|
||||||
|
under the revision of the later edit, and healing took the sites for equal:
|
||||||
|
a key that was rotated, disabled or restricted on one site stayed as it was
|
||||||
|
on the other. Sync, heal and export took the claims and the inline policy, and
|
||||||
|
heal the secret key and status as well, from the cache of the server that ran
|
||||||
|
them, which can predate an edit made through another server of the site, and
|
||||||
|
sent them under the revision of the stored record. A service account created
|
||||||
|
for a user that had been deleted and created again did not reach the peers
|
||||||
|
before healing. Peers on 20261109 and 20261026 apply the new items as they
|
||||||
|
apply a healed account, and an edit that names no policy no longer removes the
|
||||||
|
inline policy on such a peer; a site that is not upgraded still sends its own
|
||||||
|
edits the old way. An edit now gives every site the whole account of the site
|
||||||
|
that took it, so accounts that already differ between sites are not repaired
|
||||||
|
by the upgrade, but by the next edit: after upgrading every site, review such
|
||||||
|
accounts and edit each one at a site that has it right.
|
||||||
|
|
||||||
## RELEASE.2026-11-09T00-00-00Z
|
## RELEASE.2026-11-09T00-00-00Z
|
||||||
|
|
||||||
Package version: `20261109000000.0.0`. Publication status and artifacts are
|
Package version: `20261109000000.0.0`; published from source `d998292387cc`.
|
||||||
listed in [GitHub Releases](https://github.com/pgsty/silo/releases). This
|
Artifacts are listed in the [GitHub release](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z).
|
||||||
section covers the changes after the published 20261026 source `c3ad836aed0d`.
|
The coordinated [English release notes](https://silo.pgsty.com/blog/release/silo-20261109/)
|
||||||
Most fixes come from reports and patches by
|
and [Chinese release notes](https://silo.pgsty.com/zh/blog/release/silo-20261109/)
|
||||||
[@botdeniska](https://github.com/botdeniska); see [CONTRIBUTORS.md](CONTRIBUTORS.md).
|
cover all components, upgrade guidance, and validation scope since the
|
||||||
|
published 20261026 source `c3ad836aed0d`. Most fixes come from reports and
|
||||||
|
patches by [@botdeniska](https://github.com/botdeniska); see [CONTRIBUTORS.md](CONTRIBUTORS.md).
|
||||||
|
|
||||||
Lifecycle, multi-pool and tiering:
|
Lifecycle, multi-pool and tiering:
|
||||||
|
|
||||||
@@ -52,7 +75,8 @@ Lifecycle, multi-pool and tiering:
|
|||||||
`retainVersions: 0`) on the expired key instead of the keys below it; a key
|
`retainVersions: 0`) on the expired key instead of the keys below it; a key
|
||||||
with a tiered or Object Lock protected version is refused
|
with a tiered or Object Lock protected version is refused
|
||||||
([#297](https://github.com/pgsty/silo/issues/297)).
|
([#297](https://github.com/pgsty/silo/issues/297)).
|
||||||
- Stop taking the free version of a deleted tiered version for a version
|
- Exclude the internal free-version marker of a deleted tiered version from
|
||||||
|
visible version history and lifecycle decisions
|
||||||
([#263](https://github.com/pgsty/silo/issues/263)), apply rules whose only
|
([#263](https://github.com/pgsty/silo/issues/263)), apply rules whose only
|
||||||
action is `DelMarkerExpiration` ([#268](https://github.com/pgsty/silo/issues/268)),
|
action is `DelMarkerExpiration` ([#268](https://github.com/pgsty/silo/issues/268)),
|
||||||
and report and account only the removals that happened
|
and report and account only the removals that happened
|
||||||
|
|||||||
+5
-2
@@ -165,8 +165,11 @@ license grant to SILO or upstream MinIO maintainers is required.
|
|||||||
|
|
||||||
### How does Silo manage dependencies?
|
### How does Silo manage dependencies?
|
||||||
|
|
||||||
Silo uses Go modules. Preserve the compatibility module and import paths in
|
Silo uses Go modules. Shared packages in `github.com/pgsty/silo-pkg/v3` are
|
||||||
`go.mod`; downstream forks are selected with explicit `replace` directives.
|
required and imported directly under their own module path, upstream
|
||||||
|
`github.com/minio/minio-go/v7` is pinned to a verified commit, and Console
|
||||||
|
(`github.com/minio/console`) and MC (`github.com/minio/mc`) are selected with
|
||||||
|
explicit `replace` directives in `go.mod`.
|
||||||
|
|
||||||
- Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
|
- Run `go get foo/bar` in the source folder to add the dependency to `go.mod` file.
|
||||||
|
|
||||||
|
|||||||
+35
-23
File diff suppressed because one or more lines are too long
@@ -42,7 +42,8 @@ PRs must flow smoothly through review to reach production. Large PRs should be s
|
|||||||
- Document testing steps in the PR summary if manual testing was performed.
|
- Document testing steps in the PR summary if manual testing was performed.
|
||||||
|
|
||||||
4. **Before Submitting**:
|
4. **Before Submitting**:
|
||||||
- Run `make verify` to check formatting, linting, and tests.
|
- Sign off every commit under the Developer Certificate of Origin (`git commit -s`).
|
||||||
|
- Run `make verifiers` to check formatting, linting, code generation, and `rebrand-guard`.
|
||||||
- Reference related issues (e.g., “Closes #1234”).
|
- Reference related issues (e.g., “Closes #1234”).
|
||||||
- Notify team members via GitHub `@mentions` if urgent or complex.
|
- Notify team members via GitHub `@mentions` if urgent or complex.
|
||||||
|
|
||||||
|
|||||||
@@ -36,7 +36,7 @@
|
|||||||
|
|
||||||
PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
|
PGSTY SILO keeps one maintained release line of the open-source MinIO server alive after upstream ended community distribution: builds, packages, multi-arch images, security fixes, and the full web console. Pigsty runs it in production as its PostgreSQL backup repository.
|
||||||
|
|
||||||
It follows one rule — **the product and its delivery surfaces are renamed; the protocol and your data are not.** Everything else lives on [silo.pgsty.com](https://silo.pgsty.com/).
|
It follows one rule: **the product and its delivery surfaces are renamed; the protocol and your data are not.** Full documentation is published at [silo.pgsty.com](https://silo.pgsty.com/).
|
||||||
|
|
||||||
**Related:** [`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
|
**Related:** [`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
|
||||||
|
|
||||||
@@ -44,10 +44,10 @@ It follows one rule — **the product and its delivery surfaces are renamed; the
|
|||||||
<img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo Console">
|
<img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo Console">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
#### SILO 20261109
|
### SILO 20261109
|
||||||
|
|
||||||
> [!WARNING]
|
> [!WARNING]
|
||||||
> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) fixes bugs and security vulnerabilities. We strongly recommend scheduling an upgrade to reduce security risks.
|
> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) (`20261109000000.0.0`, source `d998292387cc`) fixes storage, lifecycle, native restore, Object Lock, batch authorization, and IAM defects, embedding Console `v2.7.0` with `mcli 20261109`, `silo-pkg v3.16.0`, and Go `1.27.3`. Review the [release notes and upgrade requirements](https://silo.pgsty.com/blog/release/silo-20261109/) before upgrading.
|
||||||
|
|
||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
> On 2026-08-06, `pgsty/minio` became `pgsty/silo`, and the default branch changed from `master` to `main`. The last MinIO-branded release is [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z) on the [`minio`](https://github.com/pgsty/silo/tree/minio) branch.
|
> On 2026-08-06, `pgsty/minio` became `pgsty/silo`, and the default branch changed from `master` to `main`. The last MinIO-branded release is [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z) on the [`minio`](https://github.com/pgsty/silo/tree/minio) branch.
|
||||||
@@ -86,7 +86,7 @@ docker exec silo mcli mb local/demo && docker exec silo mcli ls local
|
|||||||
| Kubernetes | Helm chart, see [Download & Install](https://silo.pgsty.com/download/) |
|
| Kubernetes | Helm chart, see [Download & Install](https://silo.pgsty.com/download/) |
|
||||||
| Source | `go build -o silo . && ./silo --version` |
|
| Source | `go build -o silo . && ./silo --version` |
|
||||||
|
|
||||||
Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/); migrating from upstream MinIO — taking over an existing `minio.service` and its `/etc/default/minio`, and keeping data ownership stable with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in — is covered by the [migration guide](https://silo.pgsty.com/compatibility/migration/) and the [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
|
Every release ships checksums, SPDX SBOMs, Sigstore-signed manifests, and GitHub build attestations. Installation methods and verification commands are documented at [Download & Install](https://silo.pgsty.com/download/). To migrate from upstream MinIO, take over `minio.service` and `/etc/default/minio`, and preserve data ownership with a `/etc/systemd/system/silo.service.d/10-legacy-user.conf` drop-in. See the [migration guide](https://silo.pgsty.com/compatibility/migration/) and [binary & service notes](https://silo.pgsty.com/compatibility/binary/).
|
||||||
|
|
||||||
## Compatibility
|
## Compatibility
|
||||||
|
|
||||||
@@ -98,7 +98,7 @@ Every divergence from upstream is listed in the code-verified [compatibility aud
|
|||||||
|
|
||||||
### TLS and Go upgrades
|
### TLS and Go upgrades
|
||||||
|
|
||||||
Server 20260916 restored Go's default key exchange; Server 20261018 retains it with Go 1.27.2.
|
Server 20260916 restored Go's default key exchange; Server 20261109 retains it with Go 1.27.3.
|
||||||
TLS key exchange follows Go's defaults across the S3 listener, node links,
|
TLS key exchange follows Go's defaults across the S3 listener, node links,
|
||||||
replication, identity providers, etcd, and external HTTP services. If an endpoint
|
replication, identity providers, etcd, and external HTTP services. If an endpoint
|
||||||
cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges
|
cannot accept ML-KEM, `GODEBUG=tlsmlkem=0` disables the default hybrid exchanges
|
||||||
|
|||||||
+6
-6
@@ -36,7 +36,7 @@
|
|||||||
|
|
||||||
上游停止社区发行后,Silo 为开源 MinIO 服务端维护一条持续可用的版本线:构建、软件包、多架构镜像、安全修复与完整 Web 控制台。Pigsty 在生产环境中用它承载 PostgreSQL 备份存储。
|
上游停止社区发行后,Silo 为开源 MinIO 服务端维护一条持续可用的版本线:构建、软件包、多架构镜像、安全修复与完整 Web 控制台。Pigsty 在生产环境中用它承载 PostgreSQL 备份存储。
|
||||||
|
|
||||||
它只遵循一条原则:**改名的是产品与交付物,不是协议与你的数据。** 其余内容都在 [silo.pgsty.com](https://silo.pgsty.com/zh/)。
|
它只遵循一条原则:**改名的是产品与交付物,不是协议与你的数据。** 完整文档发布在 [silo.pgsty.com](https://silo.pgsty.com/zh/)。
|
||||||
|
|
||||||
**相关项目:**[`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
|
**相关项目:**[`pgsty/mc`](https://github.com/pgsty/mc) · [`pgsty/silo-console`](https://github.com/pgsty/silo-console) · [`pgsty/silo-pkg`](https://github.com/pgsty/silo-pkg) · [`pgsty/kes`](https://github.com/pgsty/kes) · [`pgsty/pigsty`](https://github.com/pgsty/pigsty)
|
||||||
|
|
||||||
@@ -44,10 +44,10 @@
|
|||||||
<img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo 控制台">
|
<img src="https://silo.pgsty.com/images/silo-console/console-metrics-simple.webp" alt="Silo 控制台">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
#### SILO 20261109
|
### SILO 20261109
|
||||||
|
|
||||||
> [!WARNING]
|
> [!WARNING]
|
||||||
> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z) 修复了缺陷与安全漏洞,强烈建议尽快安排升级,以降低安全风险。
|
> [SILO 20261109](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-11-09T00-00-00Z)(包版本 `20261109000000.0.0`,源码 `d998292387cc`)修复了多池生命周期、原生恢复、Object Lock、批处理授权与 IAM 等缺陷,内嵌 Console `v2.7.0` 并配套 `mcli 20261109`、`silo-pkg v3.16.0` 与 Go `1.27.3`。升级前请阅读[发布说明与升级要求](https://silo.pgsty.com/zh/blog/release/silo-20261109/)。
|
||||||
|
|
||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
> 2026-08-06,`pgsty/minio` 更名为 `pgsty/silo`,默认分支由 `master` 改为 `main`。最后一个 MinIO 品牌发布是 [`minio`](https://github.com/pgsty/silo/tree/minio) 分支上的 [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z)。
|
> 2026-08-06,`pgsty/minio` 更名为 `pgsty/silo`,默认分支由 `master` 改为 `main`。最后一个 MinIO 品牌发布是 [`minio`](https://github.com/pgsty/silo/tree/minio) 分支上的 [`20260804`](https://github.com/pgsty/silo/releases/tag/RELEASE.2026-08-04T00-00-00Z)。
|
||||||
@@ -98,9 +98,9 @@ Silo 保留 S3 与存储格式兼容性,包括既有 `MINIO_*` 环境变量、
|
|||||||
|
|
||||||
### TLS 与 Go 升级
|
### TLS 与 Go 升级
|
||||||
|
|
||||||
Server 20260916 已恢复 Go 默认密钥交换策略,20261018 在 Go 1.27.2 下保留该行为。
|
Server 20260916 恢复了 Go 默认密钥交换策略,Server 20261109 在 Go 1.27.3 下保留该行为。S3 监听端口、节点间连接、站点与桶复制、身份提供方、etcd 及外部 HTTP 服务均使用 Go 默认 TLS 密钥交换。若对端无法接受 ML-KEM,设置 `GODEBUG=tlsmlkem=0` 可在当前进程禁用默认混合密钥交换(证书校验保持开启);该选项不会禁用 ML-DSA 签名,也不能解决所有 TLS 连接重置问题,建议在升级不兼容对端后移除该临时配置。若仅新增的 SecP 混合组引发问题,可设置 `GODEBUG=tlssecpmlkem=0` 仅禁用这些组并保留 `X25519MLKEM768`。
|
||||||
`GODEBUG=tlsmlkem=0`、`tlssecpmlkem=0` 的适用范围、macOS 根证书来源变化,
|
|
||||||
以及 OIDC discovery 的诊断方法见 [Go 1.27 TLS 与 OIDC 指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
|
在 Go 1.27 构建中,macOS 上设置 `SSL_CERT_FILE` 或 `SSL_CERT_DIR` 会以磁盘根证书和 Go 验证器替代系统钥匙串信任;陈旧或不完整的 CA 路径可能导致原先受信任的连接失败,清除继承的环境变量即可恢复钥匙串信任。配置的 `CAs` 目录中的显式证书仍会叠加加入所选根证书池。Go 1.27 二进制要求 macOS 13 或更高版本。详见 [Go 1.27 发行说明](https://go.dev/doc/go1.27)与 [Go 1.27 TLS 与 OIDC 发现指南](https://silo.pgsty.com/zh/blog/design/go127-tls-oidc-discovery/)。
|
||||||
|
|
||||||
## 文档归属
|
## 文档归属
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -38,7 +38,7 @@ For vulnerabilities in this fork:
|
|||||||
|
|
||||||
1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
|
1. Follow the fork-specific expectations in [VULNERABILITY_REPORT.md](VULNERABILITY_REPORT.md).
|
||||||
2. Prefer this repository's [private GitHub security advisory](https://github.com/pgsty/silo/security/advisories/new) workflow.
|
2. Prefer this repository's [private GitHub security advisory](https://github.com/pgsty/silo/security/advisories/new) workflow.
|
||||||
3. If you cannot use GitHub private reporting, email [ron@pgsty.com](mailto:ron@pgsty.com) or rh@vonng.com
|
3. If you cannot use GitHub private reporting, email [ron@pgsty.com](mailto:ron@pgsty.com) or [rh@vonng.com](mailto:rh@vonng.com)
|
||||||
directly. Do not publish vulnerability details or exploit code in public
|
directly. Do not publish vulnerability details or exploit code in public
|
||||||
issues or pull requests.
|
issues or pull requests.
|
||||||
4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.
|
4. If you confirm the issue also affects upstream `minio/minio`, report it upstream separately.
|
||||||
|
|||||||
@@ -169,8 +169,9 @@ func TestIAMAdminEmptyPolicyDelegation(t *testing.T) {
|
|||||||
{"{}", http.StatusOK, true},
|
{"{}", http.StatusOK, true},
|
||||||
{`{"Statement":null}`, http.StatusOK, true},
|
{`{"Statement":null}`, http.StatusOK, true},
|
||||||
{limitedPolicy, http.StatusOK, false},
|
{limitedPolicy, http.StatusOK, false},
|
||||||
{`{"Version":"2012-10-17"}`, http.StatusForbidden, false},
|
{`{"Version":"2012-10-17"}`, http.StatusOK, true},
|
||||||
{`{"Statement":[]}`, http.StatusForbidden, false},
|
{`{"Statement":[]}`, http.StatusOK, true},
|
||||||
|
{`{"Version":"2012-10-17","Statement":[]}`, http.StatusOK, true},
|
||||||
} {
|
} {
|
||||||
expires := UTCNow().Add(time.Hour)
|
expires := UTCNow().Add(time.Hour)
|
||||||
req := madmin.AddServiceAccountReq{Expiration: &expires}
|
req := madmin.AddServiceAccountReq{Expiration: &expires}
|
||||||
|
|||||||
+19
-41
@@ -914,20 +914,12 @@ func (a adminAPIHandlers) UpdateServiceAccount(w http.ResponseWriter, r *http.Re
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Call site replication hook - non-root user accounts are replicated.
|
// Call site replication hook - non-root user accounts are replicated.
|
||||||
|
// The hook sends the account as it is stored after this edit, not the
|
||||||
|
// fields the request changed (see replicationItem).
|
||||||
if svcAccount.ParentUser != globalActiveCred.AccessKey {
|
if svcAccount.ParentUser != globalActiveCred.AccessKey {
|
||||||
replLogIf(ctx, globalSiteReplicationSys.IAMChangeHook(ctx, madmin.SRIAMItem{
|
replLogIf(ctx, globalSiteReplicationSys.IAMChangeHook(ctx, madmin.SRIAMItem{
|
||||||
Type: madmin.SRIAMItemSvcAcc,
|
Type: madmin.SRIAMItemSvcAcc,
|
||||||
SvcAccChange: &madmin.SRSvcAccChange{
|
SvcAccChange: &madmin.SRSvcAccChange{Update: &madmin.SRSvcAccUpdate{AccessKey: accessKey}},
|
||||||
Update: &madmin.SRSvcAccUpdate{
|
|
||||||
AccessKey: accessKey,
|
|
||||||
SecretKey: opts.secretKey,
|
|
||||||
Status: opts.status,
|
|
||||||
Name: opts.name,
|
|
||||||
Description: opts.description,
|
|
||||||
SessionPolicy: madmin.SRSessionPolicy(updateReq.NewPolicy),
|
|
||||||
Expiration: updateReq.NewExpiration,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
UpdatedAt: updatedAt,
|
UpdatedAt: updatedAt,
|
||||||
}))
|
}))
|
||||||
}
|
}
|
||||||
@@ -2399,37 +2391,13 @@ func (a adminAPIHandlers) ExportIAM(w http.ResponseWriter, r *http.Request) {
|
|||||||
// skip site-replication service account.
|
// skip site-replication service account.
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
claims, err := globalIAMSys.GetClaimsForSvcAcc(ctx, acc.Credentials.AccessKey)
|
// The stored record only: this server's cache may not have
|
||||||
|
// heard of an edit made through another server.
|
||||||
|
svcAccts[user], err = serviceAccountSnapshot(acc)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeErrorResponse(ctx, w, exportError(ctx, err, iamFile, ""), r.URL)
|
writeErrorResponse(ctx, w, exportError(ctx, err, iamFile, ""), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
sa, policy, err := globalIAMSys.GetServiceAccount(ctx, acc.Credentials.AccessKey)
|
|
||||||
if err != nil {
|
|
||||||
writeErrorResponse(ctx, w, exportError(ctx, err, iamFile, ""), r.URL)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
var policyJSON []byte
|
|
||||||
if policy != nil {
|
|
||||||
policyJSON, err = json.Marshal(policy)
|
|
||||||
if err != nil {
|
|
||||||
writeErrorResponse(ctx, w, exportError(ctx, err, iamFile, ""), r.URL)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
svcAccts[user] = madmin.SRSvcAccCreate{
|
|
||||||
Parent: acc.Credentials.ParentUser,
|
|
||||||
AccessKey: user,
|
|
||||||
SecretKey: acc.Credentials.SecretKey,
|
|
||||||
Groups: acc.Credentials.Groups,
|
|
||||||
Claims: claims,
|
|
||||||
SessionPolicy: policyJSON,
|
|
||||||
Status: acc.Credentials.Status,
|
|
||||||
Name: sa.Name,
|
|
||||||
Description: sa.Description,
|
|
||||||
Expiration: &sa.Expiration,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
svcAccData, err := json.Marshal(svcAccts)
|
svcAccData, err := json.Marshal(svcAccts)
|
||||||
@@ -2650,8 +2618,17 @@ func (a adminAPIHandlers) importIAM(w http.ResponseWriter, r *http.Request, apiV
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if _, gerr := globalIAMSys.AddUsersToGroup(ctx, group, grpInfo.Members); gerr != nil {
|
_, gerr := globalIAMSys.AddUsersToGroup(ctx, group, grpInfo.Members)
|
||||||
failed.Groups = append(failed.Groups, madmin.IAMErrEntity{Name: group, Error: err})
|
if gerr == nil {
|
||||||
|
switch grpInfo.Status {
|
||||||
|
case statusDisabled:
|
||||||
|
_, gerr = globalIAMSys.SetGroupStatus(ctx, group, false)
|
||||||
|
case statusEnabled:
|
||||||
|
_, gerr = globalIAMSys.SetGroupStatus(ctx, group, true)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if gerr != nil {
|
||||||
|
failed.Groups = append(failed.Groups, madmin.IAMErrEntity{Name: group, Error: gerr})
|
||||||
} else {
|
} else {
|
||||||
added.Groups = append(added.Groups, group)
|
added.Groups = append(added.Groups, group)
|
||||||
}
|
}
|
||||||
@@ -2731,6 +2708,7 @@ func (a adminAPIHandlers) importIAM(w http.ResponseWriter, r *http.Request, apiV
|
|||||||
opts := newServiceAccountOpts{
|
opts := newServiceAccountOpts{
|
||||||
accessKey: user,
|
accessKey: user,
|
||||||
secretKey: svcAcctReq.SecretKey,
|
secretKey: svcAcctReq.SecretKey,
|
||||||
|
status: svcAcctReq.Status,
|
||||||
sessionPolicy: sp,
|
sessionPolicy: sp,
|
||||||
claims: svcAcctReq.Claims,
|
claims: svcAcctReq.Claims,
|
||||||
name: svcAcctReq.Name,
|
name: svcAcctReq.Name,
|
||||||
@@ -3087,7 +3065,7 @@ func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.
|
|||||||
}
|
}
|
||||||
// Empty service policies mean inheritance, including for expiring keys.
|
// Empty service policies mean inheritance, including for expiring keys.
|
||||||
// Normalize the replication request as well as the local credential.
|
// Normalize the replication request as well as the local credential.
|
||||||
if sp.Version == "" && sp.Statements == nil && sp.ID == "" {
|
if sp.IsEmpty() {
|
||||||
sp = nil
|
sp = nil
|
||||||
createReq.Policy = nil
|
createReq.Policy = nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,387 @@
|
|||||||
|
// Copyright (c) 2026 Denis Babchuk
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful,
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/minio/minio-go/v7/pkg/set"
|
||||||
|
"github.com/minio/minio-go/v7/pkg/tags"
|
||||||
|
"github.com/minio/minio/internal/arn"
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
"github.com/minio/minio/internal/logger"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/policy"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/policy/condition"
|
||||||
|
"github.com/puzpuzpuz/xsync/v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
// authorizeRequestWithUnreadTags decides whether to disclose readErr for an
|
||||||
|
// object whose stored tags could not be read. It must not grant object access.
|
||||||
|
//
|
||||||
|
// Without the tags, an Allow statement that tests s3:ExistingObjectTag cannot
|
||||||
|
// match, so a caller allowed only by such a statement would get AccessDenied,
|
||||||
|
// which clients do not retry, for a transient server error such as lost read
|
||||||
|
// quorum. For a server error, try a bounded set of tags from the caller's
|
||||||
|
// policies and return the storage error if normal authorization accepts one.
|
||||||
|
// If no candidate passes, keep AccessDenied. This is not an exhaustive policy
|
||||||
|
// solver. For any other error, NoSuchKey among them, the tags stay empty.
|
||||||
|
func authorizeRequestWithUnreadTags(ctx context.Context, r *http.Request, action policy.Action, readErr error) APIErrorCode {
|
||||||
|
s3Err := authorizeRequest(ctx, r, action)
|
||||||
|
if s3Err == ErrNone || toAPIError(ctx, readErr).HTTPStatusCode < http.StatusInternalServerError {
|
||||||
|
return s3Err
|
||||||
|
}
|
||||||
|
return authorizeRequestWithTagCandidates(ctx, r, action, s3Err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// authorizeRequestWithPotentialTags performs the authorization needed before
|
||||||
|
// a remote replica can be read. An empty tag set is enough to reject a caller
|
||||||
|
// with no possible access, but a caller whose Allow depends on existing tags
|
||||||
|
// must be admitted so the remote copy can supply the authoritative tags.
|
||||||
|
// The returned decision is only a gate; the actual tags must be authorized
|
||||||
|
// again after they are read.
|
||||||
|
func authorizeRequestWithPotentialTags(ctx context.Context, r *http.Request, action policy.Action) APIErrorCode {
|
||||||
|
s3Err := authorizeRequest(ctx, r, action)
|
||||||
|
if s3Err == ErrNone {
|
||||||
|
return ErrNone
|
||||||
|
}
|
||||||
|
return authorizeRequestWithTagCandidates(ctx, r, action, s3Err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func authorizeRequestWithTagCandidates(ctx context.Context, r *http.Request, action policy.Action, s3Err APIErrorCode) APIErrorCode {
|
||||||
|
for _, existingTags := range policyExistingObjectTags(ctx, r, action) {
|
||||||
|
if authorizeRequestWithExistingTags(ctx, r, action, existingTags) == ErrNone {
|
||||||
|
return ErrNone
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return s3Err
|
||||||
|
}
|
||||||
|
|
||||||
|
// maxExistingTagCandidates bounds the tag sets tried for one request; each
|
||||||
|
// one costs an authorization.
|
||||||
|
const maxExistingTagCandidates = 64
|
||||||
|
|
||||||
|
// policyExistingObjectTags returns tag sets under which the policies that
|
||||||
|
// authorize the request may let action through on its object: the bucket
|
||||||
|
// policy for an anonymous request, else the cached IAM policies and, as a
|
||||||
|
// second policy set, the session policy of the credential. A tag set passes
|
||||||
|
// the s3:ExistingObjectTag tests of one Allow statement of each policy set and,
|
||||||
|
// as far as the same tags allow, keeps the Deny statements that test tags from
|
||||||
|
// matching. The tag sets are only candidates; the regular authorization
|
||||||
|
// decides what they allow.
|
||||||
|
func policyExistingObjectTags(ctx context.Context, r *http.Request, action policy.Action) []string {
|
||||||
|
reqInfo := logger.GetReqInfo(ctx)
|
||||||
|
if reqInfo == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
bucket, object := reqInfo.BucketName, reqInfo.ObjectName
|
||||||
|
var (
|
||||||
|
allows [][]map[string]string // per policy set, the tags its Allow statements ask for
|
||||||
|
denies [][]map[string]string // per Deny statement, alternative tags that keep it from matching
|
||||||
|
)
|
||||||
|
// add collects the tags of a statement that applies to action on the
|
||||||
|
// object, whatever its conditions.
|
||||||
|
add := func(effect policy.Effect, conditions condition.Functions, values map[string][]string, fromSet *[]map[string]string) {
|
||||||
|
candidates := existingObjectTagsCandidatesFor(effect, conditions, values)
|
||||||
|
if effect == policy.Allow {
|
||||||
|
*fromSet = append(*fromSet, candidates...)
|
||||||
|
} else if len(candidates) > 0 {
|
||||||
|
denies = append(denies, candidates)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case reqInfo.Cred.AccessKey == "":
|
||||||
|
p, err := globalPolicySys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
values := getConditionValues(r, reqInfo.Region, auth.AnonymousCredentials)
|
||||||
|
var fromSet []map[string]string
|
||||||
|
for _, st := range p.Statements {
|
||||||
|
match := st
|
||||||
|
match.Effect, match.Conditions = policy.Allow, nil
|
||||||
|
if match.IsAllowed(policy.BucketPolicyArgs{Action: action, BucketName: bucket, ObjectName: object, ConditionValues: values}) {
|
||||||
|
add(st.Effect, st.Conditions, values, &fromSet)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
allows = append(allows, fromSet)
|
||||||
|
case newGlobalAuthZPluginFn() == nil && globalIAMSys.Initialized():
|
||||||
|
values := getConditionValues(r, "", reqInfo.Cred)
|
||||||
|
policySets := globalIAMSys.unreadTagPolicySets(reqInfo.Cred)
|
||||||
|
for _, policies := range policySets {
|
||||||
|
var fromSet []map[string]string
|
||||||
|
for _, p := range policies {
|
||||||
|
for _, st := range p.Statements {
|
||||||
|
match := st
|
||||||
|
match.Effect, match.Conditions = policy.Allow, nil
|
||||||
|
if match.IsAllowed(policy.Args{Action: action, BucketName: bucket, ObjectName: object, ConditionValues: values}) {
|
||||||
|
add(st.Effect, st.Conditions, values, &fromSet)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
allows = append(allows, fromSet)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return combineExistingObjectTags(append(allows, denies...))
|
||||||
|
}
|
||||||
|
|
||||||
|
// unreadTagMappings reuses the normal user/group mapping lookup without lazy
|
||||||
|
// storage reads. Missing cache entries contribute no candidates; the regular
|
||||||
|
// authorization still decides whether any candidate is allowed.
|
||||||
|
type unreadTagMappings struct {
|
||||||
|
*IAMStoreSys
|
||||||
|
}
|
||||||
|
|
||||||
|
func (unreadTagMappings) loadMappedPolicy(context.Context, string, IAMUserType, bool, *xsync.MapOf[string, MappedPolicy]) error {
|
||||||
|
return errNoSuchPolicy
|
||||||
|
}
|
||||||
|
|
||||||
|
func (unreadTagMappings) loadGroup(context.Context, string, map[string]GroupInfo) error {
|
||||||
|
return errNoSuchGroup
|
||||||
|
}
|
||||||
|
|
||||||
|
// unreadTagPolicySets selects only this credential's cached policies. Base
|
||||||
|
// and session policies remain separate because both must allow the request.
|
||||||
|
// This only supplies candidates, so identity, claims and policy validity are
|
||||||
|
// left to the normal authorization on each candidate.
|
||||||
|
func (sys *IAMSys) unreadTagPolicySets(cred auth.Credentials) [][]policy.Policy {
|
||||||
|
derived := cred.IsTemp() || cred.IsServiceAccount()
|
||||||
|
name := cred.AccessKey
|
||||||
|
if derived {
|
||||||
|
name = cred.ParentUser
|
||||||
|
}
|
||||||
|
cache := sys.store.rlock()
|
||||||
|
defer sys.store.runlock()
|
||||||
|
var names []string
|
||||||
|
roleARN := (policy.Args{Claims: cred.Claims}).GetRoleArn()
|
||||||
|
switch {
|
||||||
|
case name == globalActiveCred.AccessKey:
|
||||||
|
case derived && roleARN != "":
|
||||||
|
parsed, err := arn.Parse(roleARN)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
names = newMappedPolicy(sys.rolesMap[parsed]).toSlice()
|
||||||
|
default:
|
||||||
|
loader := unreadTagMappings{sys.store}
|
||||||
|
var err error
|
||||||
|
names, _, err = cache.policyDBGet(loader, name, false, false)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
groups, err := cache.policyDBGetGroups(loader, len(names) > 0, cred.Groups...)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
names = append(names, groups...)
|
||||||
|
if derived && len(names) == 0 {
|
||||||
|
claims, _ := policy.GetPoliciesFromClaims(cred.Claims, iamPolicyClaimNameOpenID())
|
||||||
|
names = claims.ToSlice()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
sets := [][]policy.Policy{nil}
|
||||||
|
slices.Sort(names)
|
||||||
|
for _, name := range slices.Compact(names) {
|
||||||
|
if doc, ok := cache.iamPolicyDocsMap[name]; ok {
|
||||||
|
sets[0] = append(sets[0], doc.Policy)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if derived && (cred.IsTemp() || cred.Claims[iamPolicyClaimNameSA()] != inheritedPolicyType) {
|
||||||
|
if sp, ok := cred.Claims[sessionPolicyNameExtracted].(string); ok {
|
||||||
|
if p, err := policy.ParseConfig(strings.NewReader(sp)); err == nil {
|
||||||
|
sets = append(sets, []policy.Policy{*p})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return sets
|
||||||
|
}
|
||||||
|
|
||||||
|
// combineExistingObjectTags combines alternatives from each base/session
|
||||||
|
// Allow set and each tag-dependent Deny. A set may also contribute no tags:
|
||||||
|
// another statement or another set may already supply what it needs. First
|
||||||
|
// values win conflicts; the normal authorization decides whether they work.
|
||||||
|
func combineExistingObjectTags(sets [][]map[string]string) []string {
|
||||||
|
combos := []map[string]string{{}}
|
||||||
|
for _, fromSet := range sets {
|
||||||
|
fromSet = append(uniqueExistingObjectTagMaps(fromSet), nil)
|
||||||
|
var next []map[string]string
|
||||||
|
seen := set.NewStringSet()
|
||||||
|
combinations:
|
||||||
|
for _, combo := range combos {
|
||||||
|
for _, addition := range fromSet {
|
||||||
|
merged := maps.Clone(combo)
|
||||||
|
for k, v := range addition {
|
||||||
|
if _, ok := merged[k]; !ok {
|
||||||
|
merged[k] = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t, err := tags.MapToObjectTags(merged)
|
||||||
|
if err != nil || seen.Contains(t.String()) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen.Add(t.String())
|
||||||
|
next = append(next, merged)
|
||||||
|
if len(next) == maxExistingTagCandidates {
|
||||||
|
break combinations
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
combos = next
|
||||||
|
}
|
||||||
|
var candidates []string
|
||||||
|
for _, combo := range combos {
|
||||||
|
if len(combo) != 0 { // empty tags were already authorized
|
||||||
|
t, _ := tags.MapToObjectTags(combo)
|
||||||
|
candidates = append(candidates, t.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return candidates
|
||||||
|
}
|
||||||
|
|
||||||
|
// existingObjectTagsCandidatesFor returns tag sets under which the
|
||||||
|
// s3:ExistingObjectTag tests of conditions all pass, for an Allow statement,
|
||||||
|
// or one of them fails, for a Deny statement; a tag left out is absent. For an
|
||||||
|
// Allow, values for different tag keys are combined, while a Deny needs only
|
||||||
|
// one failing key. It tries the values named in the conditions, with policy
|
||||||
|
// variables resolved, wildcard fillers, and absence, then lets the policy
|
||||||
|
// engine decide whether each candidate really satisfies the conditions.
|
||||||
|
func existingObjectTagsCandidatesFor(effect policy.Effect, conditions condition.Functions, values map[string][]string) []map[string]string {
|
||||||
|
// Each function tests one key, so the tests of one tag pass or fail
|
||||||
|
// whatever the other tags are.
|
||||||
|
byKey := make(map[string]condition.Functions)
|
||||||
|
for _, f := range conditions {
|
||||||
|
for key := range condition.NewFunctions(f).Keys() {
|
||||||
|
if key.Is(condition.ExistingObjectTag) {
|
||||||
|
byKey[key.Name()] = append(byKey[key.Name()], f)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(byKey) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
keys := slices.Sorted(maps.Keys(byKey))
|
||||||
|
if effect == policy.Deny {
|
||||||
|
var candidates []map[string]string
|
||||||
|
for _, key := range keys {
|
||||||
|
candidates = append(candidates, existingObjectTagValues(key, strings.TrimPrefix(key, condition.ExistingObjectTag.Name()+"/"), byKey[key], values, false)...)
|
||||||
|
}
|
||||||
|
return uniqueExistingObjectTagMaps(candidates)
|
||||||
|
}
|
||||||
|
|
||||||
|
candidates := []map[string]string{{}}
|
||||||
|
for _, key := range keys {
|
||||||
|
keyCandidates := existingObjectTagValues(key, strings.TrimPrefix(key, condition.ExistingObjectTag.Name()+"/"), byKey[key], values, true)
|
||||||
|
if len(keyCandidates) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
next := make([]map[string]string, 0, min(maxExistingTagCandidates, len(candidates)*len(keyCandidates)))
|
||||||
|
for _, base := range candidates {
|
||||||
|
for _, keyCandidate := range keyCandidates {
|
||||||
|
merged := maps.Clone(base)
|
||||||
|
maps.Copy(merged, keyCandidate)
|
||||||
|
next = append(next, merged)
|
||||||
|
if len(next) == maxExistingTagCandidates {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(next) == maxExistingTagCandidates {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
candidates = next
|
||||||
|
}
|
||||||
|
return uniqueExistingObjectTagMaps(candidates)
|
||||||
|
}
|
||||||
|
|
||||||
|
func uniqueExistingObjectTagMaps(candidates []map[string]string) []map[string]string {
|
||||||
|
seen := set.NewStringSet()
|
||||||
|
unique := make([]map[string]string, 0, len(candidates))
|
||||||
|
for _, candidate := range candidates {
|
||||||
|
t, err := tags.MapToObjectTags(candidate)
|
||||||
|
if err != nil || seen.Contains(t.String()) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen.Add(t.String())
|
||||||
|
unique = append(unique, candidate)
|
||||||
|
}
|
||||||
|
return unique
|
||||||
|
}
|
||||||
|
|
||||||
|
// existingObjectTagValues looks for values of tag tagKey, or its absence,
|
||||||
|
// under which the functions that test the condition key key evaluate to want.
|
||||||
|
func existingObjectTagValues(key, tagKey string, tests condition.Functions, values map[string][]string, want bool) []map[string]string {
|
||||||
|
defer delete(values, key)
|
||||||
|
delete(values, key)
|
||||||
|
var candidates []map[string]string
|
||||||
|
if tests.Evaluate(values) == want {
|
||||||
|
candidates = append(candidates, map[string]string{})
|
||||||
|
}
|
||||||
|
// The functions keep their values unexported; JSON exposes them.
|
||||||
|
buf, err := json.Marshal(tests)
|
||||||
|
if err != nil {
|
||||||
|
return candidates
|
||||||
|
}
|
||||||
|
var byName map[string]map[string]condition.ValueSet
|
||||||
|
if err = json.Unmarshal(buf, &byName); err != nil {
|
||||||
|
return candidates
|
||||||
|
}
|
||||||
|
var tried []string
|
||||||
|
for name, byKey := range byName {
|
||||||
|
if name == "Null" {
|
||||||
|
continue // its operand says whether the tag is there, not a value
|
||||||
|
}
|
||||||
|
for _, valueSet := range byKey {
|
||||||
|
for _, v := range valueSet.ToSlice() {
|
||||||
|
s := substituteConditionVariables(v.String(), values)
|
||||||
|
tried = append(tried, s, strings.NewReplacer("*", "", "?", "_").Replace(s))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
slices.Sort(tried)
|
||||||
|
for _, v := range append(slices.Compact(tried), "", "_") {
|
||||||
|
if _, err := tags.MapToObjectTags(map[string]string{tagKey: v}); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
values[key] = []string{v}
|
||||||
|
if tests.Evaluate(values) == want {
|
||||||
|
candidates = append(candidates, map[string]string{tagKey: v})
|
||||||
|
if len(candidates) == maxExistingTagCandidates {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return uniqueExistingObjectTagMaps(candidates)
|
||||||
|
}
|
||||||
|
|
||||||
|
// substituteConditionVariables resolves the policy variables in v from the
|
||||||
|
// condition values as the policy engine does for the values of a condition;
|
||||||
|
// silo-pkg keeps its own function for this unexported.
|
||||||
|
func substituteConditionVariables(v string, values map[string][]string) string {
|
||||||
|
for _, key := range condition.CommonKeys {
|
||||||
|
if rvalues, ok := values[key.Name()]; ok && len(rvalues) > 0 && rvalues[0] != "" {
|
||||||
|
v = strings.ReplaceAll(v, key.VarName(), rvalues[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
@@ -0,0 +1,443 @@
|
|||||||
|
// Copyright (c) 2026 Denis Babchuk
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful,
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"encoding/xml"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/minio-go/v7/pkg/tags"
|
||||||
|
"github.com/minio/minio/internal/arn"
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/policy"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/policy/condition"
|
||||||
|
)
|
||||||
|
|
||||||
|
// objectReadFault takes the first n drives of the set that holds object
|
||||||
|
// offline for the metadata reads of that object only, so that IAM and bucket
|
||||||
|
// metadata stay readable. It returns a function that restores the drives.
|
||||||
|
func objectReadFault(z *erasureServerPools, bucket, object string, n int) func() {
|
||||||
|
pool := z.serverPools[0]
|
||||||
|
setIndex := pool.getHashedSetIndex(object)
|
||||||
|
pool.erasureDisksMu.Lock()
|
||||||
|
defer pool.erasureDisksMu.Unlock()
|
||||||
|
previous := pool.erasureDisks[setIndex]
|
||||||
|
disks := append([]StorageAPI(nil), previous...)
|
||||||
|
for i := range n {
|
||||||
|
disks[i] = consistencyReadFaultDisk{StorageAPI: disks[i], bucket: bucket, object: object}
|
||||||
|
}
|
||||||
|
pool.erasureDisks[setIndex] = disks
|
||||||
|
return func() {
|
||||||
|
pool.erasureDisksMu.Lock()
|
||||||
|
pool.erasureDisks[setIndex] = previous
|
||||||
|
pool.erasureDisksMu.Unlock()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A caller allowed to read an object only through s3:ExistingObjectTag must
|
||||||
|
// get the storage error, which clients retry, and not AccessDenied, when the
|
||||||
|
// object's tags cannot be read because read quorum is lost. A caller that no
|
||||||
|
// tags would let through must still get AccessDenied, and nothing changes
|
||||||
|
// while the tags can be read.
|
||||||
|
func TestExistingObjectTagAuthReadQuorumLost(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
defer cancel()
|
||||||
|
obj, dirs, err := prepareErasure16(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
z := obj.(*erasureServerPools)
|
||||||
|
previous := newObjectLayerFn()
|
||||||
|
setObjectLayer(z)
|
||||||
|
defer func() {
|
||||||
|
z.Shutdown(context.Background())
|
||||||
|
removeRoots(dirs)
|
||||||
|
setObjectLayer(previous)
|
||||||
|
}()
|
||||||
|
bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
statement := func(effect, resource, conditions string) string {
|
||||||
|
st := fmt.Sprintf(`{"Effect": %q, "Action": ["s3:GetObject", "s3:GetObjectTagging", "s3:PutObjectTagging", "s3:DeleteObjectTagging"], "Resource": ["arn:aws:s3:::%s/%s"]`,
|
||||||
|
effect, bucket, resource)
|
||||||
|
if conditions != "" {
|
||||||
|
st += `, "Condition": ` + conditions
|
||||||
|
}
|
||||||
|
return st + "}"
|
||||||
|
}
|
||||||
|
tagStatement := func(effect, resource, value string) string {
|
||||||
|
return statement(effect, resource, fmt.Sprintf(`{"StringEquals": {"s3:ExistingObjectTag/k": %q}}`, value))
|
||||||
|
}
|
||||||
|
newUser := func(statements ...string) auth.Credentials {
|
||||||
|
return newDeleteAuthzPolicyUser(t, "", bucket, "["+strings.Join(statements, ",")+"]")
|
||||||
|
}
|
||||||
|
newServiceAccount := func(parent auth.Credentials, statements string) auth.Credentials {
|
||||||
|
sessionPolicy, err := policy.ParseConfig(strings.NewReader(`{"Version": "2012-10-17", "Statement": [` + statements + `]}`))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cred, _, err := globalIAMSys.NewServiceAccount(ctx, parent.AccessKey, nil, newServiceAccountOpts{sessionPolicy: sessionPolicy})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return cred
|
||||||
|
}
|
||||||
|
newTempUser := func(parent auth.Credentials, statements string) auth.Credentials {
|
||||||
|
session := `{"Version": "2012-10-17", "Statement": [` + statements + `]}`
|
||||||
|
secret, err := getTokenSigningKey()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cred, err := auth.GetNewCredentialsWithMetadata(map[string]any{
|
||||||
|
parentClaim: parent.AccessKey, "exp": UTCNow().Add(time.Hour).Unix(),
|
||||||
|
policy.SessionPolicyName: base64.StdEncoding.EncodeToString([]byte(session)),
|
||||||
|
}, secret)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cred.ParentUser = parent.AccessKey
|
||||||
|
if _, err := globalIAMSys.SetTempUser(ctx, cred.AccessKey, cred, ""); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return cred
|
||||||
|
}
|
||||||
|
// An unrelated cached policy must not consume the candidate limit. The
|
||||||
|
// target user's policy is created below and sorts after this policy.
|
||||||
|
fillerStatements := make([]string, 0, maxExistingTagCandidates)
|
||||||
|
for i := range maxExistingTagCandidates {
|
||||||
|
fillerStatements = append(fillerStatements, statement("Allow", "*",
|
||||||
|
fmt.Sprintf(`{"StringEquals": {"s3:ExistingObjectTag/fill%02d": "v"}}`, i)))
|
||||||
|
}
|
||||||
|
fillerPolicy, err := policy.ParseConfig(strings.NewReader(`{"Version": "2012-10-17", "Statement": [` + strings.Join(fillerStatements, ",") + `]}`))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fillerPolicyName := "000-unread-tags-filler-" + mustGetUUID()
|
||||||
|
if _, err = globalIAMSys.SetPolicy(ctx, fillerPolicyName, *fillerPolicy); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() {
|
||||||
|
if err := globalIAMSys.DeletePolicy(ctx, fillerPolicyName, false); err != nil {
|
||||||
|
t.Logf("delete filler policy: %v", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
type user struct {
|
||||||
|
name string
|
||||||
|
creds auth.Credentials
|
||||||
|
// The status with the object's tags readable and with read quorum
|
||||||
|
// lost: 200 and 503 for a caller that some tags let through, 403
|
||||||
|
// and 403 for a caller that none do.
|
||||||
|
healthy, quorumLost int
|
||||||
|
}
|
||||||
|
allowedByTags := func(name string, creds auth.Credentials) user {
|
||||||
|
return user{name, creds, http.StatusOK, http.StatusServiceUnavailable}
|
||||||
|
}
|
||||||
|
deniedByAnyTags := func(name string, creds auth.Credentials) user {
|
||||||
|
return user{name, creds, http.StatusForbidden, http.StatusForbidden}
|
||||||
|
}
|
||||||
|
// Each case below that tests a tag key of its own uses a key no other
|
||||||
|
// policy asks for.
|
||||||
|
ownerCondition := `{"StringEquals": {"s3:ExistingObjectTag/owner": "${aws:username}"}}`
|
||||||
|
owner := newUser(statement("Allow", "*", ownerCondition))
|
||||||
|
users := []user{
|
||||||
|
// Allowed only while the object carries k=v.
|
||||||
|
allowedByTags("tagged", newUser(tagStatement("Allow", "*", "v"))),
|
||||||
|
// Allowed while the object carries a tag np, whatever its value.
|
||||||
|
allowedByTags("nullFalse", newUser(statement("Allow", "*", `{"Null": {"s3:ExistingObjectTag/np": "false"}}`))),
|
||||||
|
// Allowed while a tag ne is there and is not w.
|
||||||
|
allowedByTags("notEquals", newUser(statement("Allow", "*",
|
||||||
|
`{"Null": {"s3:ExistingObjectTag/ne": "false"}, "StringNotEquals": {"s3:ExistingObjectTag/ne": "w"}}`))),
|
||||||
|
// Allowed while every value of tag fa differs from w; that needs a tag fa.
|
||||||
|
allowedByTags("forAllNotEquals", newUser(statement("Allow", "*", `{"ForAllValues:StringNotEquals": {"s3:ExistingObjectTag/fa": "w"}}`))),
|
||||||
|
// Allowed while the object is tagged with the caller's name.
|
||||||
|
allowedByTags("owner", owner),
|
||||||
|
// Allowed, but denied unless the object carries du=1.
|
||||||
|
allowedByTags("denyUnlessTagged", newUser(statement("Allow", "*", ""),
|
||||||
|
statement("Deny", "*", `{"StringNotEquals": {"s3:ExistingObjectTag/du": "1"}}`))),
|
||||||
|
// A service account whose parent asks for k=v and whose own policy
|
||||||
|
// asks for j=w: only an object carrying both lets it through.
|
||||||
|
allowedByTags("serviceAccount", newServiceAccount(newUser(tagStatement("Allow", "*", "v")),
|
||||||
|
statement("Allow", "*", `{"StringEquals": {"s3:ExistingObjectTag/j": "w"}}`))),
|
||||||
|
allowedByTags("temporary", newTempUser(newUser(tagStatement("Allow", "*", "v")),
|
||||||
|
statement("Allow", "*", `{"StringEquals": {"s3:ExistingObjectTag/j": "w"}}`))),
|
||||||
|
allowedByTags("rootTemporary", newTempUser(globalActiveCred, tagStatement("Allow", "*", "v"))),
|
||||||
|
allowedByTags("rootServiceAccount", newServiceAccount(globalActiveCred, tagStatement("Allow", "*", "v"))),
|
||||||
|
// The effective policy must be selected before applying the candidate
|
||||||
|
// limit; the unrelated cached policy above has 64 other tag statements.
|
||||||
|
allowedByTags("unrelatedPolicies", newUser(statement("Allow", "*",
|
||||||
|
`{"StringEquals": {"s3:ExistingObjectTag/target": "ok"}}`))),
|
||||||
|
// An alternative allowed value must survive a Deny that excludes the
|
||||||
|
// first value. With unread tags, k=v is the only valid candidate.
|
||||||
|
allowedByTags("alternativeAllowedValue", newUser(
|
||||||
|
statement("Allow", "*", `{"StringEquals": {"s3:ExistingObjectTag/k": ["a", "v"]}}`),
|
||||||
|
tagStatement("Deny", "*", "a"))),
|
||||||
|
allowedByTags("alternativeValuesAcrossKeys", newUser(
|
||||||
|
statement("Allow", "*", `{"StringEquals": {"s3:ExistingObjectTag/k": ["a", "v"], "s3:ExistingObjectTag/j": ["b", "w"]}}`),
|
||||||
|
tagStatement("Deny", "*", "a"),
|
||||||
|
statement("Deny", "*", `{"StringEquals": {"s3:ExistingObjectTag/j": "b"}}`))),
|
||||||
|
// Both Deny statements must be avoided. The first candidate (k="")
|
||||||
|
// avoids the Null Deny but still matches the second Deny.
|
||||||
|
allowedByTags("multipleDenies", newUser(statement("Allow", "*", ""),
|
||||||
|
statement("Deny", "*", `{"Null": {"s3:ExistingObjectTag/k": "true"}}`),
|
||||||
|
tagStatement("Deny", "*", ""))),
|
||||||
|
// The Deny can be avoided by k=v OR j=w. Only j=w is compatible
|
||||||
|
// with this Allow, so stopping at the first Deny key loses it.
|
||||||
|
{
|
||||||
|
"alternativeDenyKey", newUser(tagStatement("Allow", "*", "a"),
|
||||||
|
statement("Deny", "*", `{"StringNotEquals": {"s3:ExistingObjectTag/k": "v", "s3:ExistingObjectTag/j": "w"}}`)),
|
||||||
|
http.StatusForbidden, http.StatusServiceUnavailable,
|
||||||
|
},
|
||||||
|
|
||||||
|
// Allowed only for k=w: the decision still depends on the tags.
|
||||||
|
{"otherValue", newUser(tagStatement("Allow", "*", "w")), http.StatusForbidden, http.StatusServiceUnavailable},
|
||||||
|
|
||||||
|
// Allowed for k=v under another prefix only.
|
||||||
|
deniedByAnyTags("otherPrefix", newUser(tagStatement("Allow", "other/*", "v"))),
|
||||||
|
// Allowed and denied for k=v alike.
|
||||||
|
deniedByAnyTags("allowDeny", newUser(tagStatement("Allow", "*", "v"), tagStatement("Deny", "*", "v"))),
|
||||||
|
// Allowed only without a tag nc that is 1.
|
||||||
|
deniedByAnyTags("nullConflict", newUser(statement("Allow", "*",
|
||||||
|
`{"Null": {"s3:ExistingObjectTag/nc": "true"}, "StringEquals": {"s3:ExistingObjectTag/nc": "1"}}`))),
|
||||||
|
// Tagged with the caller's name, under another prefix only.
|
||||||
|
deniedByAnyTags("ownerPrefix", newUser(statement("Allow", "other/*", ownerCondition))),
|
||||||
|
// Denied unless the object carries db=1, and denied if it does.
|
||||||
|
deniedByAnyTags("denyBoth", newUser(statement("Allow", "*", ""),
|
||||||
|
statement("Deny", "*", `{"StringNotEquals": {"s3:ExistingObjectTag/db": "1"}}`),
|
||||||
|
statement("Deny", "*", `{"StringEquals": {"s3:ExistingObjectTag/db": "1"}}`))),
|
||||||
|
// The parent asks for k=v and the service account for k=x.
|
||||||
|
deniedByAnyTags("serviceAccountConflict", newServiceAccount(newUser(tagStatement("Allow", "*", "v")),
|
||||||
|
tagStatement("Allow", "*", "x"))),
|
||||||
|
}
|
||||||
|
|
||||||
|
const object, missing = "tagged", "missing"
|
||||||
|
if _, err := z.PutObject(ctx, bucket, object, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""),
|
||||||
|
ObjectOptions{UserDefined: map[string]string{xhttp.AmzObjectTagging: "k=v&j=w&np=1&ne=1&fa=1&du=1&target=ok&owner=" + owner.AccessKey}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Anonymous reads are allowed by the bucket policy while the object carries k=v.
|
||||||
|
if _, err := globalBucketMetadataSys.Update(ctx, bucket, bucketPolicyConfig, []byte(`{"Version": "2012-10-17", "Statement": [
|
||||||
|
{"Effect": "Allow", "Principal": {"AWS": ["*"]}, "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::`+bucket+`/*"],
|
||||||
|
"Condition": {"StringEquals": {"s3:ExistingObjectTag/k": "v"}}}]}`)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
type request struct {
|
||||||
|
name string
|
||||||
|
method, target string
|
||||||
|
header map[string]string
|
||||||
|
body []byte
|
||||||
|
}
|
||||||
|
objectURL := getGetObjectURL("", bucket, object)
|
||||||
|
get := request{"GET", http.MethodGet, objectURL, nil, nil}
|
||||||
|
head := request{"HEAD", http.MethodHead, objectURL, nil, nil}
|
||||||
|
taggingURL := makeTestTargetURL("", bucket, object, url.Values{"tagging": []string{""}})
|
||||||
|
taggingBody := []byte(`<Tagging><TagSet><Tag><Key>k</Key><Value>v</Value></Tag></TagSet></Tagging>`)
|
||||||
|
putTagging := request{"PutObjectTagging", http.MethodPut, taggingURL, nil, taggingBody}
|
||||||
|
getTagging := request{"GetObjectTagging", http.MethodGet, taggingURL, nil, nil}
|
||||||
|
deleteTagging := request{"DeleteObjectTagging", http.MethodDelete, taggingURL, nil, nil}
|
||||||
|
check := func(stage string, creds auth.Credentials, name string, req request, want int) {
|
||||||
|
t.Helper()
|
||||||
|
var rec *httptest.ResponseRecorder
|
||||||
|
if creds.AccessKey == "" {
|
||||||
|
anonymous, err := newTestRequest(req.method, req.target, 0, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, anonymous)
|
||||||
|
} else {
|
||||||
|
var body io.ReadSeeker
|
||||||
|
if req.body != nil {
|
||||||
|
body = bytes.NewReader(req.body)
|
||||||
|
}
|
||||||
|
header := make(map[string]string)
|
||||||
|
for k, v := range req.header {
|
||||||
|
header[k] = v
|
||||||
|
}
|
||||||
|
if creds.IsTemp() {
|
||||||
|
header[xhttp.AmzSecurityToken] = creds.SessionToken
|
||||||
|
}
|
||||||
|
signed, err := newTestSignedRequestV4(req.method, req.target, int64(len(req.body)), body, creds.AccessKey, creds.SecretKey, header)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, signed)
|
||||||
|
}
|
||||||
|
if rec.Code != want {
|
||||||
|
var apiErr APIErrorResponse
|
||||||
|
_ = xml.Unmarshal(rec.Body.Bytes(), &apiErr)
|
||||||
|
t.Errorf("%s: %s by %s: status %d %s, want %d", stage, req.name, name, rec.Code, apiErr.Code, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var anonymous auth.Credentials
|
||||||
|
for _, u := range users {
|
||||||
|
check("healthy", u.creds, u.name, get, u.healthy)
|
||||||
|
check("healthy", u.creds, u.name, getTagging, u.healthy)
|
||||||
|
}
|
||||||
|
check("healthy", anonymous, "anonymous", get, http.StatusOK)
|
||||||
|
// The tags were read and allow the request: the range error is the answer.
|
||||||
|
check("healthy", users[0].creds, users[0].name,
|
||||||
|
request{"GET bytes=100-200", http.MethodGet, objectURL, map[string]string{xhttp.Range: "bytes=100-200"}, nil},
|
||||||
|
http.StatusRequestedRangeNotSatisfiable)
|
||||||
|
// Without an object there are no tags: AccessDenied, as before.
|
||||||
|
missingURL := getGetObjectURL("", bucket, missing)
|
||||||
|
check("missing key", users[0].creds, users[0].name, request{"GET", http.MethodGet, missingURL, nil, nil}, http.StatusForbidden)
|
||||||
|
check("missing key", users[0].creds, users[0].name, request{"HEAD", http.MethodHead, missingURL, nil, nil}, http.StatusForbidden)
|
||||||
|
check("missing key", users[0].creds, users[0].name, request{"GetObjectTagging", http.MethodGet, missingURL + "?tagging", nil, nil}, http.StatusForbidden)
|
||||||
|
|
||||||
|
defer objectReadFault(z, bucket, object, 10)()
|
||||||
|
if _, err := z.GetObjectInfo(ctx, bucket, object, ObjectOptions{}); !isErrReadQuorum(err) {
|
||||||
|
t.Fatalf("GetObjectInfo with 10 of 16 drives failing: %v, want InsufficientReadQuorum", err)
|
||||||
|
}
|
||||||
|
for _, req := range []request{get, head, getTagging, putTagging, deleteTagging} {
|
||||||
|
for _, u := range users {
|
||||||
|
check("read quorum lost", u.creds, u.name, req, u.quorumLost)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
check("read quorum lost", anonymous, "anonymous", get, http.StatusServiceUnavailable)
|
||||||
|
check("read quorum lost", anonymous, "anonymous", head, http.StatusServiceUnavailable)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExistingObjectTagsFor(t *testing.T) {
|
||||||
|
values := map[string][]string{"username": {"alice"}, "sub": {"u1"}}
|
||||||
|
for _, tc := range []struct {
|
||||||
|
effect policy.Effect
|
||||||
|
conditions string
|
||||||
|
want string // the tags to set, encoded as in x-amz-tagging
|
||||||
|
ok bool // false: no tags for this statement
|
||||||
|
}{
|
||||||
|
{policy.Allow, `{"StringEquals": {"s3:ExistingObjectTag/k": "v"}}`, "k=v", true},
|
||||||
|
{policy.Allow, `{"StringEqualsIgnoreCase": {"s3:ExistingObjectTag/k": "V"}, "IpAddress": {"aws:SourceIp": "10.0.0.0/8"}}`, "k=V", true},
|
||||||
|
{policy.Allow, `{"ForAnyValue:StringEquals": {"s3:ExistingObjectTag/k": ["b", "a"]}}`, "k=a", true},
|
||||||
|
{policy.Allow, `{"StringLike": {"s3:ExistingObjectTag/k": "a?c*"}}`, "k=a_c", true},
|
||||||
|
{policy.Allow, `{"StringEquals": {"s3:ExistingObjectTag/b": "2", "s3:ExistingObjectTag/a": "1"}}`, "a=1&b=2", true},
|
||||||
|
// Passed without the tag.
|
||||||
|
{policy.Allow, `{"StringNotEquals": {"s3:ExistingObjectTag/k": "v"}}`, "", true},
|
||||||
|
{policy.Allow, `{"Null": {"s3:ExistingObjectTag/k": "true"}}`, "", true},
|
||||||
|
// Null "false" asks for the tag, with any value.
|
||||||
|
{policy.Allow, `{"Null": {"s3:ExistingObjectTag/k": "false"}}`, "k=", true},
|
||||||
|
{policy.Allow, `{"Null": {"s3:ExistingObjectTag/k": "false"}, "StringNotEquals": {"s3:ExistingObjectTag/k": ["", "w"]}}`, "k=_", true},
|
||||||
|
{policy.Allow, `{"Null": {"s3:ExistingObjectTag/k": "false"}, "StringNotLike": {"s3:ExistingObjectTag/k": "*w"}}`, "k=", true},
|
||||||
|
{policy.Allow, `{"ForAllValues:StringNotEquals": {"s3:ExistingObjectTag/k": "w"}}`, "k=", true},
|
||||||
|
{policy.Allow, `{"Null": {"s3:ExistingObjectTag/k": "true"}, "StringEquals": {"s3:ExistingObjectTag/k": "v"}}`, "", false},
|
||||||
|
// Policy variables take the values of the request.
|
||||||
|
{policy.Allow, `{"StringEquals": {"s3:ExistingObjectTag/owner": "${aws:username}"}}`, "owner=alice", true},
|
||||||
|
{policy.Allow, `{"StringLike": {"s3:ExistingObjectTag/owner": "${jwt:sub}-*"}}`, "owner=u1-", true},
|
||||||
|
{policy.Allow, `{"StringEquals": {"s3:ExistingObjectTag/owner": "${ldap:user}"}}`, "", false},
|
||||||
|
// A Deny is kept from matching.
|
||||||
|
{policy.Deny, `{"StringNotEquals": {"s3:ExistingObjectTag/k": "v"}}`, "k=v", true},
|
||||||
|
{policy.Deny, `{"Null": {"s3:ExistingObjectTag/k": "true"}}`, "k=", true},
|
||||||
|
{policy.Deny, `{"StringEquals": {"s3:ExistingObjectTag/k": "v"}}`, "", true},
|
||||||
|
{policy.Deny, `{"StringEquals": {"s3:prefix": "v"}}`, "", false},
|
||||||
|
// No tag tests.
|
||||||
|
{policy.Allow, `{"StringEquals": {"s3:prefix": "v"}}`, "", false},
|
||||||
|
{policy.Allow, "", "", false},
|
||||||
|
} {
|
||||||
|
var conditions condition.Functions // nil: a statement without a Condition
|
||||||
|
if tc.conditions != "" {
|
||||||
|
if err := json.Unmarshal([]byte(tc.conditions), &conditions); err != nil {
|
||||||
|
t.Fatalf("%s: %v", tc.conditions, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
candidates := existingObjectTagsCandidatesFor(tc.effect, conditions, values)
|
||||||
|
ok := len(candidates) > 0
|
||||||
|
got := ""
|
||||||
|
if ok {
|
||||||
|
encoded, err := tags.MapToObjectTags(candidates[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: %v", tc.conditions, err)
|
||||||
|
}
|
||||||
|
got = encoded.String()
|
||||||
|
}
|
||||||
|
if got != tc.want || ok != tc.ok {
|
||||||
|
t.Errorf("%s %s: got %q %v, want %q %v", tc.effect, tc.conditions, got, ok, tc.want, tc.ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Group-only and external identities need mapping lookups that normally
|
||||||
|
// fall back to storage. Candidate collection must stay within the cache even
|
||||||
|
// for those identities, while still finding their group/claim/role policy.
|
||||||
|
func TestUnreadTagPolicySetsCacheOnly(t *testing.T) {
|
||||||
|
ctx, sys, _ := prepareIAMRevisionFixture(t)
|
||||||
|
const policyName = "unread-tag-policy"
|
||||||
|
batchAuthzPolicy(ctx, t, sys, policyName, `{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::bucket/*"],"Condition":{"StringEquals":{"s3:ExistingObjectTag/k":"v"}}}`)
|
||||||
|
member := iamAdminUser(ctx, t, sys, "unread-group-member", "")
|
||||||
|
_, err := sys.AddUsersToGroup(ctx, "unread-group", []string{member.AccessKey})
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, err = sys.PolicyDBSet(ctx, "unread-group", policyName, regUser, true)
|
||||||
|
mustIAM(t, err)
|
||||||
|
role, err := arn.NewIAMRoleARN("unread-role", "")
|
||||||
|
mustIAM(t, err)
|
||||||
|
sys.rolesMap = map[arn.ARN]string{role: policyName}
|
||||||
|
claimUser := auth.Credentials{
|
||||||
|
AccessKey: "unread-sts", ParentUser: "external-parent", SessionToken: "token",
|
||||||
|
Expiration: UTCNow().Add(time.Hour), Claims: map[string]any{iamPolicyClaimNameOpenID(): policyName},
|
||||||
|
}
|
||||||
|
roleUser := claimUser
|
||||||
|
roleUser.Claims = map[string]any{roleArnClaim: role.String()}
|
||||||
|
original := sys.store.IAMStorageAPI
|
||||||
|
defer func() { sys.store.IAMStorageAPI = original }()
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
cred auth.Credentials
|
||||||
|
}{
|
||||||
|
{"group", member},
|
||||||
|
{"claims", claimUser},
|
||||||
|
{"role", roleUser},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
name := tc.cred.AccessKey
|
||||||
|
if tc.cred.ParentUser != "" {
|
||||||
|
name = tc.cred.ParentUser
|
||||||
|
}
|
||||||
|
probe := &batchBlockedIAMStore{
|
||||||
|
IAMStorageAPI: original, mapping: name, doc: policyName,
|
||||||
|
started: make(chan struct{}), release: make(chan struct{}),
|
||||||
|
}
|
||||||
|
close(probe.release) // report an accidental read without blocking the test
|
||||||
|
sys.store.IAMStorageAPI = probe
|
||||||
|
sets := sys.unreadTagPolicySets(tc.cred)
|
||||||
|
select {
|
||||||
|
case <-probe.started:
|
||||||
|
t.Error("candidate collection attempted a storage read")
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if len(sets) != 1 || len(sets[0]) != 1 {
|
||||||
|
t.Fatalf("expected the cached policy through %s, got %v", tc.name, sets)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
+25
-13
@@ -340,16 +340,6 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
|
|||||||
return s3Err
|
return s3Err
|
||||||
}
|
}
|
||||||
|
|
||||||
func checkRequestAuthTypeWithExistingTags(ctx context.Context, r *http.Request, action policy.Action, bucketName, objectName, existingTags string) (s3Err APIErrorCode) {
|
|
||||||
logger.GetReqInfo(ctx).BucketName = bucketName
|
|
||||||
logger.GetReqInfo(ctx).ObjectName = objectName
|
|
||||||
|
|
||||||
if s3Err = authenticateRequest(ctx, r, action); s3Err != ErrNone {
|
|
||||||
return s3Err
|
|
||||||
}
|
|
||||||
return authorizeRequestWithExistingTags(ctx, r, action, existingTags)
|
|
||||||
}
|
|
||||||
|
|
||||||
func checkRequestAuthTypeWithRequestTags(ctx context.Context, r *http.Request, action policy.Action, bucketName, objectName string, requestTags *string) (s3Err APIErrorCode) {
|
func checkRequestAuthTypeWithRequestTags(ctx context.Context, r *http.Request, action policy.Action, bucketName, objectName string, requestTags *string) (s3Err APIErrorCode) {
|
||||||
logger.GetReqInfo(ctx).BucketName = bucketName
|
logger.GetReqInfo(ctx).BucketName = bucketName
|
||||||
logger.GetReqInfo(ctx).ObjectName = objectName
|
logger.GetReqInfo(ctx).ObjectName = objectName
|
||||||
@@ -425,6 +415,19 @@ func authorizeRequest(ctx context.Context, r *http.Request, action policy.Action
|
|||||||
return authorizeRequestWithExistingTags(ctx, r, action, "")
|
return authorizeRequestWithExistingTags(ctx, r, action, "")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// authorizeBucketRequest authorizes a bucket action, such as s3:ListBucket,
|
||||||
|
// on the bucket of a request made for one of its objects.
|
||||||
|
func authorizeBucketRequest(ctx context.Context, r *http.Request, action policy.Action) APIErrorCode {
|
||||||
|
reqInfo := logger.GetReqInfo(ctx)
|
||||||
|
if reqInfo == nil {
|
||||||
|
return ErrAccessDenied
|
||||||
|
}
|
||||||
|
objectName := reqInfo.ObjectName
|
||||||
|
reqInfo.ObjectName = ""
|
||||||
|
defer func() { reqInfo.ObjectName = objectName }()
|
||||||
|
return authorizeRequest(ctx, r, action)
|
||||||
|
}
|
||||||
|
|
||||||
func deleteObjectAction(versionID string) policy.Action {
|
func deleteObjectAction(versionID string) policy.Action {
|
||||||
if versionID != "" {
|
if versionID != "" {
|
||||||
return policy.DeleteObjectVersionAction
|
return policy.DeleteObjectVersionAction
|
||||||
@@ -676,7 +679,15 @@ func setAuthMiddleware(h http.Handler) http.Handler {
|
|||||||
|
|
||||||
aType := getRequestAuthType(r)
|
aType := getRequestAuthType(r)
|
||||||
switch aType {
|
switch aType {
|
||||||
case authTypeSigned, authTypeSignedV2, authTypeStreamingSigned, authTypeStreamingSignedTrailer:
|
case authTypeSigned, authTypeSignedV2, authTypeStreamingSigned, authTypeStreamingSignedTrailer, authTypeStreamingUnsignedTrailer:
|
||||||
|
// STREAMING-UNSIGNED-PAYLOAD-TRAILER is valid for a genuinely
|
||||||
|
// anonymous request. Once Authorization is present, it is a
|
||||||
|
// signed REST-HEADER request and must obey the same date/skew
|
||||||
|
// bound as the other SigV4 streaming forms.
|
||||||
|
if aType == authTypeStreamingUnsignedTrailer && r.Header.Get(xhttp.Authorization) == "" {
|
||||||
|
h.ServeHTTP(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
// Verify if date headers are set, if not reject the request
|
// Verify if date headers are set, if not reject the request
|
||||||
amzDate, errCode := parseAmzDateHeader(r)
|
amzDate, errCode := parseAmzDateHeader(r)
|
||||||
if errCode != ErrNone {
|
if errCode != ErrNone {
|
||||||
@@ -774,11 +785,12 @@ func isPutActionAllowedWithRequestTags(ctx context.Context, atype authType, buck
|
|||||||
|
|
||||||
func isPutActionAllowedWithCred(bucketName, objectName string, r *http.Request, action policy.Action, requestTags *string, cred auth.Credentials, owner bool) APIErrorCode {
|
func isPutActionAllowedWithCred(bucketName, objectName string, r *http.Request, action policy.Action, requestTags *string, cred auth.Credentials, owner bool) APIErrorCode {
|
||||||
// Do not check for PutObjectRetentionAction permission,
|
// Do not check for PutObjectRetentionAction permission,
|
||||||
// if mode and retain until date are not set.
|
// if neither retention values nor a replica retention revision are set.
|
||||||
// Can happen when bucket has default lock config set
|
// Can happen when bucket has default lock config set
|
||||||
if action == policy.PutObjectRetentionAction &&
|
if action == policy.PutObjectRetentionAction &&
|
||||||
r.Header.Get(xhttp.AmzObjectLockMode) == "" &&
|
r.Header.Get(xhttp.AmzObjectLockMode) == "" &&
|
||||||
r.Header.Get(xhttp.AmzObjectLockRetainUntilDate) == "" {
|
r.Header.Get(xhttp.AmzObjectLockRetainUntilDate) == "" &&
|
||||||
|
r.Header.Get(xhttp.MinIOSourceObjectRetentionTimestamp) == "" {
|
||||||
return ErrNone
|
return ErrNone
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -330,7 +330,7 @@ func TestBatchReviewRotationKeepsMigrationCleanup(t *testing.T) {
|
|||||||
t.Errorf("rotation changed a draining source after its version was copied (rotation error: %v)", rotationErr)
|
t.Errorf("rotation changed a draining source after its version was copied (rotation error: %v)", rotationErr)
|
||||||
}
|
}
|
||||||
snapshot := []FileInfo{{VersionID: before.VersionID, ModTime: beforeInfo.ModTime}}
|
snapshot := []FileInfo{{VersionID: before.VersionID, ModTime: beforeInfo.ModTime}}
|
||||||
if err := z.removeMovedVersions(t.Context(), bucket, object, 0, snapshot, nil); err != nil {
|
if _, err := z.removeMovedVersions(t.Context(), bucket, object, 0, snapshot, nil); err != nil {
|
||||||
t.Fatalf("rotation prevented migration cleanup: %v", err)
|
t.Fatalf("rotation prevented migration cleanup: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: before.VersionID}); !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
|
if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, object, ObjectOptions{VersionID: before.VersionID}); !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
|
||||||
|
|||||||
+6
-10
@@ -478,15 +478,11 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
|
|||||||
getObjectInfoFn := objectAPI.GetObjectInfo
|
getObjectInfoFn := objectAPI.GetObjectInfo
|
||||||
|
|
||||||
var (
|
var (
|
||||||
hasLockEnabled bool
|
|
||||||
dsc ReplicateDecision
|
dsc ReplicateDecision
|
||||||
goi ObjectInfo
|
goi ObjectInfo
|
||||||
gerr error
|
gerr error
|
||||||
)
|
)
|
||||||
replicateDeletes := hasReplicationRules(ctx, bucket, deleteObjectsReq.Objects)
|
replicateDeletes := hasReplicationRules(ctx, bucket, deleteObjectsReq.Objects)
|
||||||
if rcfg, _ := globalBucketObjectLockSys.Get(bucket); rcfg.LockEnabled {
|
|
||||||
hasLockEnabled = true
|
|
||||||
}
|
|
||||||
|
|
||||||
type deleteResult struct {
|
type deleteResult struct {
|
||||||
delInfo DeletedObject
|
delInfo DeletedObject
|
||||||
@@ -537,10 +533,10 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
|
|||||||
VersionSuspended: vc.Suspended(),
|
VersionSuspended: vc.Suspended(),
|
||||||
}
|
}
|
||||||
|
|
||||||
// Physical version deletes in a locked bucket are checked by the
|
// Every physical version delete checks the version's own protection
|
||||||
// single-object path below, while its mutation lock is held. A lookup
|
// under its mutation lock. Bucket metadata may be stale or unavailable
|
||||||
// here could miss a later protection update or a protected pool copy.
|
// and cannot establish that a stored version is unprotected.
|
||||||
if hasLockEnabled && object.VersionID != "" {
|
if object.VersionID != "" {
|
||||||
if first, ok := objectsToDelete[object]; ok {
|
if first, ok := objectsToDelete[object]; ok {
|
||||||
duplicateOf[index] = first
|
duplicateOf[index] = first
|
||||||
} else {
|
} else {
|
||||||
@@ -605,7 +601,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
|
|||||||
bulk := make([]ObjectToDelete, 0, len(deleteList))
|
bulk := make([]ObjectToDelete, 0, len(deleteList))
|
||||||
bulkIndex := make([]int, 0, len(deleteList))
|
bulkIndex := make([]int, 0, len(deleteList))
|
||||||
for i, object := range deleteList {
|
for i, object := range deleteList {
|
||||||
if !hasLockEnabled || object.VersionID == "" {
|
if object.VersionID == "" {
|
||||||
bulk = append(bulk, object)
|
bulk = append(bulk, object)
|
||||||
bulkIndex = append(bulkIndex, i)
|
bulkIndex = append(bulkIndex, i)
|
||||||
continue
|
continue
|
||||||
@@ -663,7 +659,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
|
|||||||
// under the mutation lock, and a missing version has no stored identity.
|
// under the mutation lock, and a missing version has no stored identity.
|
||||||
dindex := objectsToDelete[deleteList[i]]
|
dindex := objectsToDelete[deleteList[i]]
|
||||||
if errs[i] == nil || isErrObjectNotFound(errs[i]) || isErrVersionNotFound(errs[i]) {
|
if errs[i] == nil || isErrObjectNotFound(errs[i]) || isErrVersionNotFound(errs[i]) {
|
||||||
if replicateDeletes && (!hasLockEnabled || deleteList[i].VersionID == "") {
|
if replicateDeletes && deleteList[i].VersionID == "" {
|
||||||
dObjects[i].ReplicationState = deleteList[i].ReplicationState()
|
dObjects[i].ReplicationState = deleteList[i].ReplicationState()
|
||||||
}
|
}
|
||||||
deleteResults[dindex].delInfo = dObjects[i]
|
deleteResults[dindex].delInfo = dObjects[i]
|
||||||
|
|||||||
@@ -0,0 +1,125 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/xml"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
)
|
||||||
|
|
||||||
|
type metadataCacheLoadBarrier struct {
|
||||||
|
ObjectLayer
|
||||||
|
key string
|
||||||
|
read, resume chan struct{}
|
||||||
|
once sync.Once
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *metadataCacheLoadBarrier) GetObjectNInfo(ctx context.Context, bucket, key string, rs *HTTPRangeSpec, h http.Header, opts ObjectOptions) (*GetObjectReader, error) {
|
||||||
|
r, err := b.ObjectLayer.GetObjectNInfo(ctx, bucket, key, rs, h, opts)
|
||||||
|
if bucket == minioMetaBucket && key == b.key {
|
||||||
|
b.once.Do(func() { close(b.read); <-b.resume })
|
||||||
|
}
|
||||||
|
return r, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBucketMetadataLoadPreservesConcurrentPublication(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t,
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, _ http.Handler, _ auth.Credentials, t *testing.T) {
|
||||||
|
// Disk has the old unlocked config; a concurrent publication enables
|
||||||
|
// lock and versioning after the cache miss has started loading it.
|
||||||
|
enableBucketObjectLock(t, bucket)
|
||||||
|
newer, err := globalBucketMetadataSys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
globalBucketMetadataSys.Remove(bucket)
|
||||||
|
barrier := &metadataCacheLoadBarrier{ObjectLayer: obj, key: pathJoin(bucketMetaPrefix, bucket, bucketMetadataFile), read: make(chan struct{}), resume: make(chan struct{})}
|
||||||
|
previous := newObjectLayerFn()
|
||||||
|
setObjectLayer(barrier)
|
||||||
|
defer setObjectLayer(previous)
|
||||||
|
release := sync.OnceFunc(func() { close(barrier.resume) })
|
||||||
|
defer release()
|
||||||
|
type result struct {
|
||||||
|
meta BucketMetadata
|
||||||
|
reloaded bool
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
done := make(chan result, 1)
|
||||||
|
go func() {
|
||||||
|
meta, reloaded, err := globalBucketMetadataSys.GetConfig(t.Context(), bucket)
|
||||||
|
done <- result{meta, reloaded, err}
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-barrier.read:
|
||||||
|
case <-time.After(10 * time.Second):
|
||||||
|
t.Fatal("metadata load did not reach disk")
|
||||||
|
}
|
||||||
|
globalBucketMetadataSys.Set(bucket, newer)
|
||||||
|
release()
|
||||||
|
select {
|
||||||
|
case got := <-done:
|
||||||
|
if got.err != nil || got.reloaded || got.meta.objectLockConfig == nil || got.meta.objectLockConfig.ObjectLockEnabled != "Enabled" {
|
||||||
|
t.Fatalf("%s: stale load replaced the publication: %+v", backend, got)
|
||||||
|
}
|
||||||
|
case <-time.After(10 * time.Second):
|
||||||
|
t.Fatal("metadata load did not finish")
|
||||||
|
}
|
||||||
|
cached, err := globalBucketMetadataSys.Get(bucket)
|
||||||
|
if err != nil || cached.objectLockConfig == nil || cached.objectLockConfig.ObjectLockEnabled != "Enabled" {
|
||||||
|
t.Fatalf("cached lock lost: %+v %v", cached, err)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBulkDeleteVersionLockWithStaleBucketCache(t *testing.T) {
|
||||||
|
for _, pools := range []int{1, 2} {
|
||||||
|
z, _ := bulkDeletePools(t, pools)
|
||||||
|
bucket, router, err := initAPIHandlerTest(t.Context(), z, nil, MakeBucketOptions{LockEnabled: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, protection := range batchLockKinds {
|
||||||
|
oi := batchPutVersion(t, z, bucket, protection, batchLockMeta(protection))
|
||||||
|
saved, err := globalBucketMetadataSys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
globalBucketMetadataSys.Set(bucket, newBucketMetadata(bucket))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, bulkDeleteLockRequest(t, bucket, protection, oi.VersionID))
|
||||||
|
globalBucketMetadataSys.Set(bucket, saved)
|
||||||
|
var response DeleteObjectsResponse
|
||||||
|
if err := xml.Unmarshal(rec.Body.Bytes(), &response); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusOK || len(response.Errors) != 1 || len(response.DeletedObjects) != 0 || !batchVersionExists(t, z, bucket, protection, oi.VersionID) {
|
||||||
|
t.Fatalf("pools=%d %s: stale bucket cache bypassed version lock: %s", pools, protection, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -697,7 +697,7 @@ func (sys *BucketMetadataSys) GetConfig(ctx context.Context, bucket string) (met
|
|||||||
}
|
}
|
||||||
|
|
||||||
val, err, _ := sys.group.Do(bucket, func() (val any, err error) {
|
val, err, _ := sys.group.Do(bucket, func() (val any, err error) {
|
||||||
meta, err = loadBucketMetadata(ctx, objAPI, bucket)
|
meta, err := loadBucketMetadata(ctx, objAPI, bucket)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if !sys.Initialized() {
|
if !sys.Initialized() {
|
||||||
// bucket metadata not yet initialized
|
// bucket metadata not yet initialized
|
||||||
@@ -707,13 +707,18 @@ func (sys *BucketMetadataSys) GetConfig(ctx context.Context, bucket string) (met
|
|||||||
return meta, err
|
return meta, err
|
||||||
})
|
})
|
||||||
meta, _ = val.(BucketMetadata)
|
meta, _ = val.(BucketMetadata)
|
||||||
|
sys.Lock()
|
||||||
|
defer sys.Unlock()
|
||||||
|
// A writer may have published a newer configuration while the disk read
|
||||||
|
// was in flight. Never replace that publication with the older load.
|
||||||
|
if current, ok := sys.metadataMap[bucket]; ok {
|
||||||
|
return current, false, nil
|
||||||
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return meta, false, err
|
return meta, false, err
|
||||||
}
|
}
|
||||||
sys.Lock()
|
|
||||||
sys.metadataMap[bucket] = meta
|
sys.metadataMap[bucket] = meta
|
||||||
sys.clearLoadFailure(bucket)
|
sys.clearLoadFailure(bucket)
|
||||||
sys.Unlock()
|
|
||||||
|
|
||||||
return meta, true, nil
|
return meta, true, nil
|
||||||
}
|
}
|
||||||
|
|||||||
+129
-5
@@ -354,6 +354,8 @@ func checkPutObjectLockAllowed(ctx context.Context, rq *http.Request, bucket, ob
|
|||||||
var legalHold objectlock.ObjectLegalHold
|
var legalHold objectlock.ObjectLegalHold
|
||||||
|
|
||||||
retentionRequested := objectlock.IsObjectLockRetentionRequested(rq.Header)
|
retentionRequested := objectlock.IsObjectLockRetentionRequested(rq.Header)
|
||||||
|
// A timestamp without a value is a replicated retention removal.
|
||||||
|
retentionRequested = retentionRequested || replicaTrusted && rq.Header.Get(xhttp.MinIOSourceObjectRetentionTimestamp) != ""
|
||||||
legalHoldRequested := objectlock.IsObjectLockLegalHoldRequested(rq.Header)
|
legalHoldRequested := objectlock.IsObjectLockLegalHoldRequested(rq.Header)
|
||||||
|
|
||||||
retentionCfg, err := globalBucketObjectLockSys.Get(bucket)
|
retentionCfg, err := globalBucketObjectLockSys.Get(bucket)
|
||||||
@@ -472,6 +474,103 @@ func storedObjectLockState(metadata map[string]string) objectLockState {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// reducePoolLockState selects each field independently using a total order,
|
||||||
|
// so neither pool enumeration nor the starting copy can change the result.
|
||||||
|
// Active COMPLIANCE takes precedence over revisions; among those copies the
|
||||||
|
// longest retention wins. Only expired COMPLIANCE may lose to a removal.
|
||||||
|
func reducePoolLockState(copies []PoolObjInfo, readState func(map[string]string) objectLockState) objectLockState {
|
||||||
|
var now time.Time
|
||||||
|
clockRead := false
|
||||||
|
state := readState(copies[0].ObjInfo.UserDefined)
|
||||||
|
for _, copy := range copies[1:] {
|
||||||
|
next := readState(copy.ObjInfo.UserDefined)
|
||||||
|
if !clockRead && (state.mode == string(objectlock.RetCompliance) || next.mode == string(objectlock.RetCompliance)) {
|
||||||
|
var err error
|
||||||
|
now, err = objectlock.UTCNowNTP()
|
||||||
|
if err != nil {
|
||||||
|
now = time.Time{} // An unavailable clock cannot prove retention expired.
|
||||||
|
}
|
||||||
|
clockRead = true
|
||||||
|
}
|
||||||
|
if comparePoolRetention(next, state, now) > 0 {
|
||||||
|
state.mode, state.retainUntil, state.retentionTimestamp = next.mode, next.retainUntil, next.retentionTimestamp
|
||||||
|
}
|
||||||
|
order := compareLockRevision(next.legalHoldTimestamp, state.legalHoldTimestamp)
|
||||||
|
// Equal or unordered conflicting holds resolve to ON, then OFF, then
|
||||||
|
// absence. A strictly newer valid OFF still releases a prior hold.
|
||||||
|
if order == 0 {
|
||||||
|
order = strings.Compare(next.legalHold, state.legalHold)
|
||||||
|
}
|
||||||
|
if order == 0 {
|
||||||
|
order = strings.Compare(next.legalHoldTimestamp, state.legalHoldTimestamp)
|
||||||
|
}
|
||||||
|
if order > 0 {
|
||||||
|
state.legalHold, state.legalHoldTimestamp = next.legalHold, next.legalHoldTimestamp
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return state
|
||||||
|
}
|
||||||
|
|
||||||
|
func compareLockRevision(a, b string) int {
|
||||||
|
ta, aok := lockRevisionTime(a)
|
||||||
|
tb, bok := lockRevisionTime(b)
|
||||||
|
if aok != bok {
|
||||||
|
if aok {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
if !aok {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return ta.Compare(tb)
|
||||||
|
}
|
||||||
|
|
||||||
|
func comparePoolRetention(a, b objectLockState, now time.Time) int {
|
||||||
|
au, ae := time.Parse(time.RFC3339Nano, a.retainUntil)
|
||||||
|
bu, be := time.Parse(time.RFC3339Nano, b.retainUntil)
|
||||||
|
ap := a.mode == string(objectlock.RetCompliance) && (ae != nil || now.IsZero() || au.After(now))
|
||||||
|
bp := b.mode == string(objectlock.RetCompliance) && (be != nil || now.IsZero() || bu.After(now))
|
||||||
|
if ap != bp {
|
||||||
|
if ap {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
if ap { // An unreadable COMPLIANCE deadline cannot be shortened either.
|
||||||
|
if (ae != nil) != (be != nil) {
|
||||||
|
if ae != nil {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
if ae == nil && au.Compare(bu) != 0 {
|
||||||
|
return au.Compare(bu)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if order := compareLockRevision(a.retentionTimestamp, b.retentionTimestamp); order != 0 {
|
||||||
|
return order
|
||||||
|
}
|
||||||
|
// Ties and legacy unordered metadata must also have a stable winner.
|
||||||
|
// A retained value takes precedence over an unordered absence.
|
||||||
|
if order := strings.Compare(a.mode, b.mode); order != 0 {
|
||||||
|
return order
|
||||||
|
}
|
||||||
|
if (ae != nil) != (be != nil) {
|
||||||
|
if ae != nil {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
if ae == nil && be == nil && au.Compare(bu) != 0 {
|
||||||
|
return au.Compare(bu)
|
||||||
|
}
|
||||||
|
if order := strings.Compare(a.retainUntil, b.retainUntil); order != 0 {
|
||||||
|
return order
|
||||||
|
}
|
||||||
|
return strings.Compare(a.retentionTimestamp, b.retentionTimestamp)
|
||||||
|
}
|
||||||
|
|
||||||
// olderThan reports whether a stored replication timestamp is missing,
|
// olderThan reports whether a stored replication timestamp is missing,
|
||||||
// unreadable, or earlier than the source timestamp, in which case the
|
// unreadable, or earlier than the source timestamp, in which case the
|
||||||
// replica update wins. A zero source timestamp never wins.
|
// replica update wins. A zero source timestamp never wins.
|
||||||
@@ -491,6 +590,25 @@ func (s objectLockState) legalHoldIsOlderThan(src time.Time) bool {
|
|||||||
return olderThan(s.legalHoldTimestamp, src)
|
return olderThan(s.legalHoldTimestamp, src)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A peer's ordering timestamp cannot override an unexpired COMPLIANCE lock.
|
||||||
|
// Otherwise it could first remove retention with a same-content retransmit,
|
||||||
|
// then replace the now-unprotected version in a second request.
|
||||||
|
func (s objectLockState) complianceWouldWeaken(incoming objectLockState) bool {
|
||||||
|
if s.mode != string(objectlock.RetCompliance) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
until, err := time.Parse(time.RFC3339Nano, s.retainUntil)
|
||||||
|
if err != nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
now, err := objectlock.UTCNowNTP()
|
||||||
|
if err == nil && !until.After(now) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
nextUntil, nextErr := time.Parse(time.RFC3339Nano, incoming.retainUntil)
|
||||||
|
return incoming.mode != string(objectlock.RetCompliance) || nextErr != nil || nextUntil.Before(until)
|
||||||
|
}
|
||||||
|
|
||||||
// restoreRetention and restoreLegalHold put the stored state back into
|
// restoreRetention and restoreLegalHold put the stored state back into
|
||||||
// metadata that was rebuilt from a request whose update was not applied.
|
// metadata that was rebuilt from a request whose update was not applied.
|
||||||
func (s objectLockState) restoreRetention(metadata map[string]string) {
|
func (s objectLockState) restoreRetention(metadata map[string]string) {
|
||||||
@@ -524,14 +642,20 @@ func (s objectLockState) restoreLegalHold(metadata map[string]string) {
|
|||||||
// (an older incoming value must not win over a newer stored one just because the
|
// (an older incoming value must not win over a newer stored one just because the
|
||||||
// read timed out).
|
// read timed out).
|
||||||
func replicaStoredLock(ctx context.Context, getObjectInfo GetObjectInfoFn, bucket, object, versionID string) (objectLockState, error) {
|
func replicaStoredLock(ctx context.Context, getObjectInfo GetObjectInfoFn, bucket, object, versionID string) (objectLockState, error) {
|
||||||
oi, err := getObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: versionID})
|
oi, err := replicaStoredObjectInfo(ctx, getObjectInfo, bucket, object, versionID)
|
||||||
|
return storedObjectLockState(oi.UserDefined), err
|
||||||
|
}
|
||||||
|
|
||||||
|
// The same destination lookup supplies both Object Lock and tag revisions.
|
||||||
|
func replicaStoredObjectInfo(ctx context.Context, getObjectInfo GetObjectInfoFn, bucket, object, versionID string) (ObjectInfo, error) {
|
||||||
|
oi, err := getObjectInfo(ctx, bucket, object, ObjectOptions{VersionID: versionID, ReplicationRequest: true})
|
||||||
switch {
|
switch {
|
||||||
case err == nil:
|
case err == nil:
|
||||||
return storedObjectLockState(oi.UserDefined), nil
|
return oi, nil
|
||||||
case isErrObjectNotFound(err) || isErrVersionNotFound(err):
|
case isErrObjectNotFound(err) || isErrVersionNotFound(err):
|
||||||
return objectLockState{}, nil
|
return ObjectInfo{}, nil
|
||||||
default:
|
default:
|
||||||
return objectLockState{}, err
|
return ObjectInfo{}, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -610,7 +734,7 @@ func reconcileStoredObjectLock(metadata map[string]string, stored objectLockStat
|
|||||||
incoming := storedObjectLockState(metadata)
|
incoming := storedObjectLockState(metadata)
|
||||||
|
|
||||||
incomingRetentionTS, _ := time.Parse(time.RFC3339Nano, incoming.retentionTimestamp)
|
incomingRetentionTS, _ := time.Parse(time.RFC3339Nano, incoming.retentionTimestamp)
|
||||||
if !stored.retentionIsOlderThan(incomingRetentionTS) {
|
if !stored.retentionIsOlderThan(incomingRetentionTS) || stored.complianceWouldWeaken(incoming) {
|
||||||
// The stored retention is at least as new as the incoming one (or the
|
// The stored retention is at least as new as the incoming one (or the
|
||||||
// incoming update is unordered): drop the incoming value and put the stored
|
// incoming update is unordered): drop the incoming value and put the stored
|
||||||
// state back, which may itself be a removal (value keys absent, timestamp
|
// state back, which may itself be a removal (value keys absent, timestamp
|
||||||
|
|||||||
+13
-2
@@ -213,16 +213,27 @@ func getConditionValuesWithTags(r *http.Request, lc string, cred auth.Credential
|
|||||||
switch authType {
|
switch authType {
|
||||||
case authTypeSignedV2, authTypePresignedV2:
|
case authTypeSignedV2, authTypePresignedV2:
|
||||||
signatureVersion = signV2Algorithm
|
signatureVersion = signV2Algorithm
|
||||||
case authTypeSigned, authTypePresigned, authTypeStreamingSigned, authTypePostPolicy:
|
case authTypeSigned, authTypePresigned, authTypeStreamingSigned,
|
||||||
|
authTypeStreamingSignedTrailer, authTypePostPolicy:
|
||||||
signatureVersion = signV4Algorithm
|
signatureVersion = signV4Algorithm
|
||||||
|
case authTypeStreamingUnsignedTrailer:
|
||||||
|
// This form is also valid for a genuinely anonymous request. Only
|
||||||
|
// expose SigV4 condition values when header authentication is present.
|
||||||
|
if r.Header.Get(xhttp.Authorization) != "" {
|
||||||
|
signatureVersion = signV4Algorithm
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var authtype string
|
var authtype string
|
||||||
switch authType {
|
switch authType {
|
||||||
case authTypePresignedV2, authTypePresigned:
|
case authTypePresignedV2, authTypePresigned:
|
||||||
authtype = "REST-QUERY-STRING"
|
authtype = "REST-QUERY-STRING"
|
||||||
case authTypeSignedV2, authTypeSigned, authTypeStreamingSigned:
|
case authTypeSignedV2, authTypeSigned, authTypeStreamingSigned, authTypeStreamingSignedTrailer:
|
||||||
authtype = "REST-HEADER"
|
authtype = "REST-HEADER"
|
||||||
|
case authTypeStreamingUnsignedTrailer:
|
||||||
|
if r.Header.Get(xhttp.Authorization) != "" {
|
||||||
|
authtype = "REST-HEADER"
|
||||||
|
}
|
||||||
case authTypePostPolicy:
|
case authTypePostPolicy:
|
||||||
authtype = "POST"
|
authtype = "POST"
|
||||||
}
|
}
|
||||||
|
|||||||
+112
-104
@@ -2773,127 +2773,135 @@ func scheduleReplication(ctx context.Context, oi ObjectInfo, o ObjectLayer, dsc
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// proxyTaggingToRepTarget proxies tagging requests to remote targets for
|
// taggingProxyTarget keeps the exact client and tags that were authorized.
|
||||||
// active-active replicated setups
|
// A second discovery must not introduce an unauthorized destination.
|
||||||
func proxyTaggingToRepTarget(ctx context.Context, bucket, object string, tags *tags.Tags, opts ObjectOptions, proxyTargets *madmin.BucketTargets) (proxy proxyResult) {
|
type taggingProxyTarget struct {
|
||||||
// this option is set when active-active replication is in place between site A -> B,
|
client *TargetClient
|
||||||
// and request hits site B that does not have the object yet.
|
tags *tags.Tags
|
||||||
if opts.ProxyRequest || (opts.ProxyHeaderSet && !opts.ProxyRequest) { // true only when site B sets MinIOSourceProxyRequest header
|
|
||||||
return proxy
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// readTaggingProxyTargets authorizes all present replicas before returning any
|
||||||
|
// tags or allowing any mutation. Missing replicas may simply lag replication;
|
||||||
|
// unreadable replicas must still fail closed.
|
||||||
|
func readTaggingProxyTargets(ctx context.Context, bucket, object string, opts ObjectOptions, proxyTargets *madmin.BucketTargets, authorize func(string, error) APIErrorCode) ([]taggingProxyTarget, proxyResult, APIErrorCode) {
|
||||||
|
var targets []taggingProxyTarget
|
||||||
|
for _, target := range proxyTargets.Targets {
|
||||||
|
tgt := globalBucketTargetSys.GetRemoteTargetClient(bucket, target.Arn)
|
||||||
|
if tgt == nil || globalBucketTargetSys.isOffline(tgt.EndpointURL()) || tgt.disableProxy {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
targets = append(targets, taggingProxyTarget{client: tgt})
|
||||||
|
}
|
||||||
|
if len(targets) == 0 {
|
||||||
|
err := RemoteTargetConnectionErr{Bucket: bucket, Err: errors.New("no available remote replication target")}
|
||||||
|
return nil, proxyResult{Err: err}, authorize("", err)
|
||||||
|
}
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
var wg sync.WaitGroup
|
||||||
errs := make([]error, len(proxyTargets.Targets))
|
errs := make([]error, len(targets))
|
||||||
for idx, t := range proxyTargets.Targets {
|
for i := range targets {
|
||||||
tgt := globalBucketTargetSys.GetRemoteTargetClient(bucket, t.Arn)
|
wg.Go(func() {
|
||||||
if tgt == nil || globalBucketTargetSys.isOffline(tgt.EndpointURL()) {
|
tgt := &targets[i]
|
||||||
continue
|
tgt.tags, errs[i] = tgt.client.GetObjectTagging(ctx, tgt.client.Bucket, object, minio.GetObjectTaggingOptions{
|
||||||
}
|
|
||||||
// if proxying explicitly disabled on remote target
|
|
||||||
if tgt.disableProxy {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
idx := idx
|
|
||||||
wg.Add(1)
|
|
||||||
go func(idx int, tgt *TargetClient) {
|
|
||||||
defer wg.Done()
|
|
||||||
var err error
|
|
||||||
if tags != nil {
|
|
||||||
popts := minio.PutObjectTaggingOptions{
|
|
||||||
VersionID: opts.VersionID,
|
VersionID: opts.VersionID,
|
||||||
Internal: minio.AdvancedObjectTaggingOptions{
|
Internal: minio.AdvancedObjectTaggingOptions{ReplicationProxyRequest: "true"},
|
||||||
ReplicationProxyRequest: "true",
|
})
|
||||||
},
|
})
|
||||||
}
|
|
||||||
err = tgt.PutObjectTagging(ctx, tgt.Bucket, object, tags, popts)
|
|
||||||
} else {
|
|
||||||
dopts := minio.RemoveObjectTaggingOptions{
|
|
||||||
VersionID: opts.VersionID,
|
|
||||||
Internal: minio.AdvancedObjectTaggingOptions{
|
|
||||||
ReplicationProxyRequest: "true",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
err = tgt.RemoveObjectTagging(ctx, tgt.Bucket, object, dopts)
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
errs[idx] = err
|
|
||||||
}
|
|
||||||
}(idx, tgt)
|
|
||||||
}
|
}
|
||||||
wg.Wait()
|
wg.Wait()
|
||||||
|
|
||||||
var (
|
var firstNotFound, readErr error
|
||||||
terr error
|
|
||||||
taggedCount int
|
|
||||||
)
|
|
||||||
for _, err := range errs {
|
for _, err := range errs {
|
||||||
if err == nil {
|
switch {
|
||||||
taggedCount++
|
case err == nil:
|
||||||
|
case isRemoteTaggingNotFound(err):
|
||||||
|
if firstNotFound == nil {
|
||||||
|
firstNotFound = err
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
if code := authorize("", err); code != ErrNone {
|
||||||
|
return nil, proxyResult{}, code
|
||||||
|
}
|
||||||
|
if readErr == nil {
|
||||||
|
readErr = err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
present := make([]taggingProxyTarget, 0, len(targets))
|
||||||
|
for i, tgt := range targets {
|
||||||
|
if errs[i] != nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if err != nil {
|
if tgt.tags == nil {
|
||||||
terr = err
|
tgt.tags, _ = tags.MapToObjectTags(nil)
|
||||||
}
|
}
|
||||||
|
if code := authorize(tgt.tags.String(), nil); code != ErrNone {
|
||||||
|
return nil, proxyResult{}, code
|
||||||
}
|
}
|
||||||
// don't return error if at least one target was tagged successfully
|
present = append(present, tgt)
|
||||||
if taggedCount == 0 && terr != nil {
|
|
||||||
proxy.Err = terr
|
|
||||||
}
|
}
|
||||||
return proxy
|
if readErr != nil {
|
||||||
|
return nil, proxyResult{Err: readErr}, ErrNone
|
||||||
|
}
|
||||||
|
if len(present) == 0 {
|
||||||
|
// With no object anywhere, authorize the empty tag set once before
|
||||||
|
// disclosing its absence. No write RPC can help a missing replica.
|
||||||
|
return nil, proxyResult{Err: firstNotFound}, authorize("", firstNotFound)
|
||||||
|
}
|
||||||
|
return present, proxyResult{}, ErrNone
|
||||||
|
}
|
||||||
|
|
||||||
|
// proxyTaggingToRepTarget proxies tagging requests to remote targets for
|
||||||
|
// active-active replicated setups.
|
||||||
|
func proxyTaggingToRepTarget(ctx context.Context, bucket, object string, tags *tags.Tags, opts ObjectOptions, proxyTargets *madmin.BucketTargets, authorize func(string, error) APIErrorCode) (proxy proxyResult, authErr APIErrorCode) {
|
||||||
|
if opts.ProxyRequest || (opts.ProxyHeaderSet && !opts.ProxyRequest) {
|
||||||
|
return proxy, ErrNone
|
||||||
|
}
|
||||||
|
targets, proxy, authErr := readTaggingProxyTargets(ctx, bucket, object, opts, proxyTargets, authorize)
|
||||||
|
if authErr != ErrNone || proxy.Err != nil {
|
||||||
|
return proxy, authErr
|
||||||
|
}
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
writeErrs := make([]error, len(targets))
|
||||||
|
for i, target := range targets {
|
||||||
|
wg.Go(func() {
|
||||||
|
tgt := target.client
|
||||||
|
if tags != nil {
|
||||||
|
writeErrs[i] = tgt.PutObjectTagging(ctx, tgt.Bucket, object, tags, minio.PutObjectTaggingOptions{
|
||||||
|
VersionID: opts.VersionID,
|
||||||
|
Internal: minio.AdvancedObjectTaggingOptions{ReplicationProxyRequest: "true"},
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
writeErrs[i] = tgt.RemoveObjectTagging(ctx, tgt.Bucket, object, minio.RemoveObjectTaggingOptions{
|
||||||
|
VersionID: opts.VersionID,
|
||||||
|
Internal: minio.AdvancedObjectTaggingOptions{ReplicationProxyRequest: "true"},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
var lastErr error
|
||||||
|
for _, err := range writeErrs {
|
||||||
|
if err == nil {
|
||||||
|
return proxy, ErrNone
|
||||||
|
}
|
||||||
|
lastErr = err
|
||||||
|
}
|
||||||
|
return proxyResult{Err: lastErr}, ErrNone
|
||||||
}
|
}
|
||||||
|
|
||||||
// proxyGetTaggingToRepTarget proxies get tagging requests to remote targets for
|
// proxyGetTaggingToRepTarget proxies get tagging requests to remote targets for
|
||||||
// active-active replicated setups
|
// active-active replicated setups.
|
||||||
func proxyGetTaggingToRepTarget(ctx context.Context, bucket, object string, opts ObjectOptions, proxyTargets *madmin.BucketTargets) (tgs *tags.Tags, proxy proxyResult) {
|
func proxyGetTaggingToRepTarget(ctx context.Context, bucket, object string, opts ObjectOptions, proxyTargets *madmin.BucketTargets, authorize func(string, error) APIErrorCode) (tgs *tags.Tags, proxy proxyResult, authErr APIErrorCode) {
|
||||||
// this option is set when active-active replication is in place between site A -> B,
|
if opts.ProxyRequest || (opts.ProxyHeaderSet && !opts.ProxyRequest) {
|
||||||
// and request hits site B that does not have the object yet.
|
return nil, proxy, ErrNone
|
||||||
if opts.ProxyRequest || (opts.ProxyHeaderSet && !opts.ProxyRequest) { // true only when site B sets MinIOSourceProxyRequest header
|
|
||||||
return nil, proxy
|
|
||||||
}
|
}
|
||||||
var wg sync.WaitGroup
|
targets, proxy, authErr := readTaggingProxyTargets(ctx, bucket, object, opts, proxyTargets, authorize)
|
||||||
errs := make([]error, len(proxyTargets.Targets))
|
if authErr != ErrNone || proxy.Err != nil {
|
||||||
tagSlc := make([]map[string]string, len(proxyTargets.Targets))
|
return nil, proxy, authErr
|
||||||
for idx, t := range proxyTargets.Targets {
|
|
||||||
tgt := globalBucketTargetSys.GetRemoteTargetClient(bucket, t.Arn)
|
|
||||||
if tgt == nil || globalBucketTargetSys.isOffline(tgt.EndpointURL()) {
|
|
||||||
continue
|
|
||||||
}
|
}
|
||||||
// if proxying explicitly disabled on remote target
|
return targets[0].tags, proxy, ErrNone
|
||||||
if tgt.disableProxy {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
idx := idx
|
|
||||||
wg.Add(1)
|
|
||||||
go func(idx int, tgt *TargetClient) {
|
|
||||||
defer wg.Done()
|
|
||||||
var err error
|
|
||||||
gopts := minio.GetObjectTaggingOptions{
|
|
||||||
VersionID: opts.VersionID,
|
|
||||||
Internal: minio.AdvancedObjectTaggingOptions{
|
|
||||||
ReplicationProxyRequest: "true",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
tgs, err = tgt.GetObjectTagging(ctx, tgt.Bucket, object, gopts)
|
|
||||||
if err != nil {
|
|
||||||
errs[idx] = err
|
|
||||||
} else {
|
|
||||||
tagSlc[idx] = tgs.ToMap()
|
|
||||||
}
|
|
||||||
}(idx, tgt)
|
|
||||||
}
|
|
||||||
wg.Wait()
|
|
||||||
for idx, err := range errs {
|
|
||||||
errCode := minio.ToErrorResponse(err).Code
|
|
||||||
if err != nil && errCode != "NoSuchKey" && errCode != "NoSuchVersion" {
|
|
||||||
return nil, proxyResult{Err: err}
|
|
||||||
}
|
|
||||||
if err == nil {
|
|
||||||
tgs, _ = tags.MapToObjectTags(tagSlc[idx])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(errs) == 1 {
|
|
||||||
proxy.Err = errs[0]
|
|
||||||
}
|
|
||||||
return tgs, proxy
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func scheduleReplicationDelete(ctx context.Context, dv DeletedObjectReplicationInfo, o ObjectLayer) {
|
func scheduleReplicationDelete(ctx context.Context, dv DeletedObjectReplicationInfo, o ObjectLayer) {
|
||||||
|
|||||||
@@ -1722,6 +1722,9 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if err := er.checkReplicaContent(ctx, bucket, object, minioMetaMultipartBucket, uploadIDPath, partsMetadata, onlineDisks, opts); err != nil {
|
||||||
|
return oi, toObjectErr(err, bucket, object)
|
||||||
|
}
|
||||||
er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename().
|
er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename().
|
||||||
|
|
||||||
defer func() {
|
defer func() {
|
||||||
|
|||||||
+34
-3
@@ -133,8 +133,25 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
|
|||||||
}
|
}
|
||||||
|
|
||||||
if dstOpts.ReplicaLockReconcile {
|
if dstOpts.ReplicaLockReconcile {
|
||||||
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(fi.Metadata))
|
stored := fi.ToObjectInfo(dstBucket, dstObject, dstOpts.Versioned || dstOpts.VersionSuspended)
|
||||||
reconcileStoredObjectTags(srcInfo.UserDefined, fi.Metadata[xhttp.AmzObjectTagging], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
|
if srcInfo.versionOnly || dstOpts.replicaObjectInfo != nil {
|
||||||
|
// fi describes the source of the new reference, not the version
|
||||||
|
// being replaced. A pooled lookup also preserves protection merged
|
||||||
|
// from another copy, even when this set holds a newer weak revision.
|
||||||
|
getInfo := er.getObjectInfo
|
||||||
|
if dstOpts.replicaObjectInfo != nil {
|
||||||
|
getInfo = dstOpts.replicaObjectInfo
|
||||||
|
}
|
||||||
|
stored, err = replicaStoredObjectInfo(ctx, func(ctx context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
|
||||||
|
opts.NoLock = true
|
||||||
|
return getInfo(ctx, bucket, object, opts)
|
||||||
|
}, dstBucket, dstObject, dstOpts.VersionID)
|
||||||
|
if err != nil {
|
||||||
|
return oi, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(stored.UserDefined))
|
||||||
|
reconcileStoredObjectTags(srcInfo.UserDefined, stored.UserTags, stored.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
|
||||||
}
|
}
|
||||||
|
|
||||||
filterOnlineDisksInplace(fi, metaArr, onlineDisks)
|
filterOnlineDisksInplace(fi, metaArr, onlineDisks)
|
||||||
@@ -148,6 +165,17 @@ func (er erasureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, d
|
|||||||
fi.IsLatest = true // we are creating a new version so this is latest.
|
fi.IsLatest = true // we are creating a new version so this is latest.
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if srcInfo.versionOnly && (dstOpts.ReplicaLockReconcile || dstOpts.ReplicationRequest) && !dstOpts.DataMovement {
|
||||||
|
candidate := slices.Clone(metaArr)
|
||||||
|
for i := range candidate {
|
||||||
|
candidate[i].VersionID = versionID
|
||||||
|
candidate[i].Metadata = srcInfo.UserDefined
|
||||||
|
}
|
||||||
|
readDisks, readMeta := shuffleDisksAndPartsMetadataByIndex(onlineDisks, candidate, fi)
|
||||||
|
if err := er.checkReplicaContent(ctx, dstBucket, dstObject, srcBucket, srcObject, readMeta, readDisks, dstOpts); err != nil {
|
||||||
|
return oi, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if !dstOpts.ReplicationRequest && !dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement &&
|
if !dstOpts.ReplicationRequest && !dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement &&
|
||||||
(versionID == "" || versionID == nullVersionID) {
|
(versionID == "" || versionID == nullVersionID) {
|
||||||
@@ -1722,6 +1750,9 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Rename the successfully written temporary object to final location.
|
// Rename the successfully written temporary object to final location.
|
||||||
|
if err := er.checkReplicaContent(ctx, bucket, object, minioMetaTmpBucket, tempObj, partsMetadata, onlineDisks, opts); err != nil {
|
||||||
|
return ObjectInfo{}, toObjectErr(err, bucket, object)
|
||||||
|
}
|
||||||
onlineDisks, versions, oldDataDir, err := renameData(ctx, onlineDisks, minioMetaTmpBucket, tempObj, partsMetadata, bucket, object, writeQuorum)
|
onlineDisks, versions, oldDataDir, err := renameData(ctx, onlineDisks, minioMetaTmpBucket, tempObj, partsMetadata, bucket, object, writeQuorum)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if errors.Is(err, errFileNotFound) {
|
if errors.Is(err, errFileNotFound) {
|
||||||
@@ -2560,7 +2591,7 @@ func (er erasureObjects) PutObjectTags(ctx context.Context, bucket, object strin
|
|||||||
|
|
||||||
fi.Metadata[xhttp.AmzObjectTagging] = tags
|
fi.Metadata[xhttp.AmzObjectTagging] = tags
|
||||||
fi.ReplicationState = opts.PutReplicationState()
|
fi.ReplicationState = opts.PutReplicationState()
|
||||||
stamp := monotonicTaggingTimestamp(opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
|
stamp := monotonicMetadataTimestamp(opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp], fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp])
|
||||||
maps.Copy(fi.Metadata, opts.UserDefined)
|
maps.Copy(fi.Metadata, opts.UserDefined)
|
||||||
if stamp != "" {
|
if stamp != "" {
|
||||||
fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
|
fi.Metadata[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
|
||||||
|
|||||||
@@ -271,11 +271,33 @@ func (z *erasureServerPools) applyMigrationExpiry(ctx context.Context, bucket, o
|
|||||||
// in the source until the ordinary expiry path has deleted them: that path
|
// in the source until the ordinary expiry path has deleted them: that path
|
||||||
// owns replication state and tier cleanup, and it re-evaluates the current
|
// owns replication state and tier cleanup, and it re-evaluates the current
|
||||||
// lifecycle rule. A migration cleanup must never turn an asynchronous ILM
|
// lifecycle rule. A migration cleanup must never turn an asynchronous ILM
|
||||||
// decision into an unconditional physical purge.
|
// decision into an unconditional physical purge. It returns how many source
|
||||||
func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) error {
|
// versions were removed, including versions deleted by ILM before cleanup.
|
||||||
|
func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) (int, error) {
|
||||||
|
err := z.retireMovedVersions(ctx, bucket, object, source, versions, expired)
|
||||||
|
removed, countErr := z.countAbsentMigrationVersions(ctx, bucket, object, source, versions)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return 0, ctx.Err()
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return removed, err
|
||||||
|
}
|
||||||
|
return removed, countErr
|
||||||
|
}
|
||||||
|
|
||||||
|
// retireMovedVersions does not infer physical progress from a pool-level read
|
||||||
|
// or delete result. Decommission verifies the entire walk snapshot once after
|
||||||
|
// this cleanup and tier GC, including versions concurrently deleted by clients.
|
||||||
|
func (z *erasureServerPools) retireMovedVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo, expired []migrationExpiredVersion) error {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
if err := z.applyMigrationExpiry(ctx, bucket, object, source, expired); err != nil {
|
if err := z.applyMigrationExpiry(ctx, bucket, object, source, expired); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if len(versions) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
object = encodeDirObject(object)
|
object = encodeDirObject(object)
|
||||||
lk := z.NewNSLock(bucket, object)
|
lk := z.NewNSLock(bucket, object)
|
||||||
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
|
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
|
||||||
@@ -289,6 +311,7 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
|
|||||||
if vid == "" {
|
if vid == "" {
|
||||||
vid = nullVersionID
|
vid = nullVersionID
|
||||||
}
|
}
|
||||||
|
expiredVersion := isExpiredVersion(expired, version)
|
||||||
copies, err := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{VersionID: vid, NoAuditLog: true})
|
copies, err := z.objectPoolInfos(ctx, bucket, object, ObjectOptions{VersionID: vid, NoAuditLog: true})
|
||||||
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
continue
|
continue
|
||||||
@@ -311,7 +334,7 @@ func (z *erasureServerPools) removeMovedVersions(ctx context.Context, bucket, ob
|
|||||||
if !src.ModTime.Equal(version.ModTime) || src.DeleteMarker != version.Deleted {
|
if !src.ModTime.Equal(version.ModTime) || src.DeleteMarker != version.Deleted {
|
||||||
return PreConditionFailed{}
|
return PreConditionFailed{}
|
||||||
}
|
}
|
||||||
if isExpiredVersion(expired, version) {
|
if expiredVersion {
|
||||||
// The logical expiry path owns this version. Keeping the source is
|
// The logical expiry path owns this version. Keeping the source is
|
||||||
// also the retry point if that path failed or the pool check races it.
|
// also the retry point if that path failed or the pool check races it.
|
||||||
continue
|
continue
|
||||||
@@ -370,19 +393,7 @@ func (z *erasureServerPools) healObjectInPool(ctx context.Context, er *erasureOb
|
|||||||
// mergePoolLockState resolves each independently ordered field. An ordered
|
// mergePoolLockState resolves each independently ordered field. An ordered
|
||||||
// removal is a value in its own right; an absent unordered field is not one.
|
// removal is a value in its own right; an absent unordered field is not one.
|
||||||
func mergePoolLockState(copies []PoolObjInfo) objectLockState {
|
func mergePoolLockState(copies []PoolObjInfo) objectLockState {
|
||||||
state := storedObjectLockState(copies[0].ObjInfo.UserDefined)
|
return reducePoolLockState(copies, storedObjectLockState)
|
||||||
for _, copy := range copies[1:] {
|
|
||||||
next := storedObjectLockState(copy.ObjInfo.UserDefined)
|
|
||||||
retentionTime, _ := time.Parse(time.RFC3339Nano, next.retentionTimestamp)
|
|
||||||
if state.retentionIsOlderThan(retentionTime) || (state.retentionTimestamp == "" && state.mode == "" && next.mode != "") {
|
|
||||||
state.mode, state.retainUntil, state.retentionTimestamp = next.mode, next.retainUntil, next.retentionTimestamp
|
|
||||||
}
|
|
||||||
holdTime, _ := time.Parse(time.RFC3339Nano, next.legalHoldTimestamp)
|
|
||||||
if state.legalHoldIsOlderThan(holdTime) || (state.legalHoldTimestamp == "" && state.legalHold == "" && next.legalHold != "") {
|
|
||||||
state.legalHold, state.legalHoldTimestamp = next.legalHold, next.legalHoldTimestamp
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return state
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func replaceObjectLockMetadata(metadata map[string]string, state objectLockState) {
|
func replaceObjectLockMetadata(metadata map[string]string, state objectLockState) {
|
||||||
@@ -534,10 +545,10 @@ func reconcileStoredObjectTags(metadata map[string]string, storedTags, storedTim
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Local tagging mutations must advance the revision they overwrite, even when
|
// Local metadata mutations must advance the revision they overwrite, even when
|
||||||
// a request's clock or lock acquisition order is behind the stored revision.
|
// a request's clock or lock acquisition order is behind the stored revision.
|
||||||
// Replica writes use reconcileStoredObjectTags instead of minting a revision.
|
// Replica writes use reconcileStoredObjectTags instead of minting a revision.
|
||||||
func monotonicTaggingTimestamp(incoming, stored string) string {
|
func monotonicMetadataTimestamp(incoming, stored string) string {
|
||||||
requested, err := time.Parse(time.RFC3339Nano, incoming)
|
requested, err := time.Parse(time.RFC3339Nano, incoming)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return incoming
|
return incoming
|
||||||
|
|||||||
@@ -0,0 +1,102 @@
|
|||||||
|
// Copyright (c) 2026 Denis Babchuk
|
||||||
|
//
|
||||||
|
// This file is part of MinIO Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A key below another key ("k/b" under "k") is a directory inside the
|
||||||
|
// directory of that key. The listing treats the directory of an object as a
|
||||||
|
// leaf, so a decommission walk does not see "k/b" while "k" is still in the
|
||||||
|
// pool. The walk has to be repeated once the keys above are gone, or the
|
||||||
|
// check after the bucket finds them and the decommission fails.
|
||||||
|
|
||||||
|
func keysBelowDecommission(t *testing.T, z *erasureServerPools) PoolDecommissionInfo {
|
||||||
|
t.Helper()
|
||||||
|
if err := z.StartDecommission(t.Context(), 0); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
z.doDecommissionInRoutine(t.Context(), 0)
|
||||||
|
z.poolMetaMutex.RLock()
|
||||||
|
defer z.poolMetaMutex.RUnlock()
|
||||||
|
return *z.poolMeta.Pools[0].Decommission
|
||||||
|
}
|
||||||
|
|
||||||
|
func keysBelowPools(t *testing.T) (*erasureServerPools, string, []string) {
|
||||||
|
t.Helper()
|
||||||
|
z, _ := migrationTestPools(t)
|
||||||
|
const bucket = "keys-below"
|
||||||
|
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{VersioningEnabled: true}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
keys := []string{"k", "k/b", "k/b/c"}
|
||||||
|
for _, key := range keys {
|
||||||
|
putConsistencyObject(t, z, bucket, key, 0, key, ObjectOptions{Versioned: true})
|
||||||
|
}
|
||||||
|
return z, bucket, keys
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecommissionMovesKeysBelowAKey(t *testing.T) {
|
||||||
|
z, bucket, keys := keysBelowPools(t)
|
||||||
|
if info := keysBelowDecommission(t, z); !info.Complete || info.Failed {
|
||||||
|
t.Errorf("decommission: complete %v, failed %v; want complete", info.Complete, info.Failed)
|
||||||
|
}
|
||||||
|
for _, key := range keys {
|
||||||
|
if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, key, ObjectOptions{}); !isErrObjectNotFound(err) {
|
||||||
|
t.Errorf("%s still in pool 0: %v", key, err)
|
||||||
|
}
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, key, ""); body != key {
|
||||||
|
t.Errorf("%s reads %q", key, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An ILM expiry can remove the parent key during a decommission walk. That
|
||||||
|
// removal must count as progress, or the next key below the parent remains
|
||||||
|
// hidden from the listing and the source pool is left non-empty.
|
||||||
|
func TestDecommissionContinuesAfterLifecycleRemovesParent(t *testing.T) {
|
||||||
|
z, _ := migrationTestPools(t)
|
||||||
|
const bucket = "decommission-ilm-keys-below"
|
||||||
|
if err := z.MakeBucket(t.Context(), bucket, MakeBucketOptions{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
meta, err := globalBucketMetadataSys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
meta.LifecycleConfigXML = []byte(`<LifecycleConfiguration><Rule><ID>expire-old</ID><Status>Enabled</Status><Filter><Prefix></Prefix></Filter><Expiration><Days>1</Days></Expiration></Rule></LifecycleConfiguration>`)
|
||||||
|
if err := meta.Save(t.Context(), z); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
globalBucketMetadataSys.Set(bucket, meta)
|
||||||
|
|
||||||
|
putConsistencyObject(t, z, bucket, "k", 0, "parent", ObjectOptions{MTime: UTCNow().Add(-72 * time.Hour)})
|
||||||
|
putConsistencyObject(t, z, bucket, "k/b", 0, "child", ObjectOptions{MTime: UTCNow()})
|
||||||
|
|
||||||
|
if info := keysBelowDecommission(t, z); !info.Complete || info.Failed {
|
||||||
|
t.Fatalf("decommission: complete %v, failed %v; want complete", info.Complete, info.Failed)
|
||||||
|
}
|
||||||
|
if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, "k/b", ObjectOptions{}); !isErrObjectNotFound(err) {
|
||||||
|
t.Fatalf("child still in source pool: %v", err)
|
||||||
|
}
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k/b", ""); body != "child" {
|
||||||
|
t.Fatalf("child reads %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -28,6 +28,8 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/dustin/go-humanize"
|
"github.com/dustin/go-humanize"
|
||||||
@@ -752,7 +754,34 @@ func (set *erasureObjects) listObjectsToDecommission(ctx context.Context, bi dec
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// decommissionPool moves bucket bi out of pool idx. A key below another key
|
||||||
|
// ("k/b" under "k") is a directory inside the directory of that key, and the
|
||||||
|
// listing does not descend into the directory of an object: a walk sees "k/b"
|
||||||
|
// only once "k" has left the pool. The walk is repeated while it removes
|
||||||
|
// source versions, so each pass moves the keys that the previous one
|
||||||
|
// uncovered. Each repeat needs a version removed from the pool, which takes
|
||||||
|
// no new writes, so the repeats end.
|
||||||
func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) error {
|
func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) error {
|
||||||
|
for {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
removed, err := z.decommissionPoolWalk(ctx, idx, pool, bi)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if err != nil || removed == 0 {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// decommissionPoolWalk walks bucket bi of pool idx once and returns the number
|
||||||
|
// of snapshot versions verified absent from every online source drive afterwards.
|
||||||
|
func (z *erasureServerPools) decommissionPoolWalk(ctx context.Context, idx int, pool *erasureSets, bi decomBucketInfo) (int64, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
ctx = logger.SetReqInfo(ctx, &logger.ReqInfo{})
|
ctx = logger.SetReqInfo(ctx, &logger.ReqInfo{})
|
||||||
|
|
||||||
const envDecomWorkers = "_MINIO_DECOMMISSION_WORKERS"
|
const envDecomWorkers = "_MINIO_DECOMMISSION_WORKERS"
|
||||||
@@ -766,9 +795,13 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
// add that many extra workers.
|
// add that many extra workers.
|
||||||
workerSize += len(pool.sets)
|
workerSize += len(pool.sets)
|
||||||
|
|
||||||
|
var removed atomic.Int64
|
||||||
|
var verifyErr error
|
||||||
|
var verifyOnce sync.Once
|
||||||
|
|
||||||
wk, err := workers.New(workerSize)
|
wk, err := workers.New(workerSize)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
var vc *versioning.Versioning
|
var vc *versioning.Versioning
|
||||||
@@ -779,24 +812,24 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
if bi.Name != minioMetaBucket {
|
if bi.Name != minioMetaBucket {
|
||||||
vc, err = globalBucketVersioningSys.Get(bi.Name)
|
vc, err = globalBucketVersioningSys.Get(bi.Name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check if the current bucket has a configured lifecycle policy
|
// Check if the current bucket has a configured lifecycle policy
|
||||||
lc, lifecycleUpdatedAt, err = globalBucketMetadataSys.GetLifecycleConfig(bi.Name)
|
lc, lifecycleUpdatedAt, err = globalBucketMetadataSys.GetLifecycleConfig(bi.Name)
|
||||||
if err != nil && !errors.Is(err, BucketLifecycleNotFound{Bucket: bi.Name}) {
|
if err != nil && !errors.Is(err, BucketLifecycleNotFound{Bucket: bi.Name}) {
|
||||||
return err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check if bucket is object locked.
|
// Check if bucket is object locked.
|
||||||
lr, err = globalBucketObjectLockSys.Get(bi.Name)
|
lr, err = globalBucketObjectLockSys.Get(bi.Name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
rcfg, err = getReplicationConfig(ctx, bi.Name)
|
rcfg, err = getReplicationConfig(ctx, bi.Name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return 0, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -855,7 +888,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
// to create the appropriate stack.
|
// to create the appropriate stack.
|
||||||
versionsSorter(fivs.Versions).reverse()
|
versionsSorter(fivs.Versions).reverse()
|
||||||
|
|
||||||
var decommissioned int
|
var candidates []FileInfo
|
||||||
var expired []migrationExpiredVersion
|
var expired []migrationExpiredVersion
|
||||||
// The key's history in all pools, read once for its versions.
|
// The key's history in all pools, read once for its versions.
|
||||||
var history migrationHistory
|
var history migrationHistory
|
||||||
@@ -863,8 +896,8 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
stopFn := globalDecommissionMetrics.log(decomMetricDecommissionObject, idx, bi.Name, version.Name, version.VersionID)
|
stopFn := globalDecommissionMetrics.log(decomMetricDecommissionObject, idx, bi.Name, version.Name, version.VersionID)
|
||||||
// Apply lifecycle rules on the objects that are expired.
|
// Apply lifecycle rules on the objects that are expired.
|
||||||
if action := filterLifecycle(bi.Name, version.Name, version, &history); action.Delete() {
|
if action := filterLifecycle(bi.Name, version.Name, version, &history); action.Delete() {
|
||||||
decommissioned++
|
|
||||||
if ilmDeletesVersionOnly(action) || (action.DeleteAll() && !version.IsRemote()) {
|
if ilmDeletesVersionOnly(action) || (action.DeleteAll() && !version.IsRemote()) {
|
||||||
|
candidates = append(candidates, version)
|
||||||
// Run the logical expiry before retiring the source.
|
// Run the logical expiry before retiring the source.
|
||||||
// A full-object expiry is safe here only for hot-tier
|
// A full-object expiry is safe here only for hot-tier
|
||||||
// metadata; transitioned objects retain their tier cleanup
|
// metadata; transitioned objects retain their tier cleanup
|
||||||
@@ -912,7 +945,6 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
// This can happen when rebalance stop races with ongoing rebalance workers.
|
// This can happen when rebalance stop races with ongoing rebalance workers.
|
||||||
// These rebalance failures can be ignored.
|
// These rebalance failures can be ignored.
|
||||||
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
|
||||||
ignore = true
|
|
||||||
stopFn(0, nil)
|
stopFn(0, nil)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -926,8 +958,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
z.poolMeta.CountItem(idx, 0, failure)
|
z.poolMeta.CountItem(idx, 0, failure)
|
||||||
z.poolMetaMutex.Unlock()
|
z.poolMetaMutex.Unlock()
|
||||||
if !failure {
|
if !failure {
|
||||||
// Success keep a count.
|
candidates = append(candidates, version)
|
||||||
decommissioned++
|
|
||||||
}
|
}
|
||||||
auditLogDecom(ctx, "DecomCopyDeleteMarker", bi.Name, version.Name, versionID, err)
|
auditLogDecom(ctx, "DecomCopyDeleteMarker", bi.Name, version.Name, versionID, err)
|
||||||
continue
|
continue
|
||||||
@@ -1013,16 +1044,32 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
if failure {
|
if failure {
|
||||||
break // break out on first error
|
break // break out on first error
|
||||||
}
|
}
|
||||||
decommissioned++
|
candidates = append(candidates, version)
|
||||||
}
|
}
|
||||||
|
|
||||||
// if all versions were decommissioned, then we can delete the object versions.
|
// Retire only successful moves and eligible expiries. A concurrently
|
||||||
if decommissioned == len(fivs.Versions) {
|
// deleted or failed sibling must not strand the successful copies.
|
||||||
|
// Skipped versions are checked for absence, never purged here.
|
||||||
|
{
|
||||||
stopFn := globalDecommissionMetrics.log(decomMetricDecommissionRemoveObject, idx, bi.Name, entry.name)
|
stopFn := globalDecommissionMetrics.log(decomMetricDecommissionRemoveObject, idx, bi.Name, entry.name)
|
||||||
err := z.removeMovedVersions(ctx, bi.Name, entry.name, idx, fivs.Versions, expired)
|
err := z.retireMovedVersions(ctx, bi.Name, entry.name, idx, candidates, expired)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
err = z.removeFreeVersions(ctx, bi.Name, entry.name, idx, fivs.FreeVersions)
|
err = z.removeFreeVersions(ctx, bi.Name, entry.name, idx, fivs.FreeVersions)
|
||||||
}
|
}
|
||||||
|
// One metadata-only read of each source drive per entry, regardless
|
||||||
|
// of its version count. No read error can become progress or trigger
|
||||||
|
// another walk; leave the bucket queued for an explicit retry.
|
||||||
|
snapshot := slices.Concat(fivs.Versions, fivs.FreeVersions)
|
||||||
|
n, countErr := z.countAbsentMigrationVersions(ctx, bi.Name, entry.name, idx, snapshot)
|
||||||
|
removed.Add(int64(n))
|
||||||
|
// The disposable usage cache is also excluded by the final
|
||||||
|
// verification. Its unreadability still never counts as progress.
|
||||||
|
if countErr != nil && (bi.Name != minioMetaBucket || !strings.Contains(entry.name, dataUsageCacheName)) {
|
||||||
|
verifyOnce.Do(func() { verifyErr = countErr })
|
||||||
|
if err == nil {
|
||||||
|
err = countErr
|
||||||
|
}
|
||||||
|
}
|
||||||
stopFn(0, err)
|
stopFn(0, err)
|
||||||
auditLogDecom(ctx, "DecomDeleteObject", bi.Name, entry.name, "", err)
|
auditLogDecom(ctx, "DecomDeleteObject", bi.Name, entry.name, "", err)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1057,12 +1104,19 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
|
|||||||
setN := humanize.Ordinal(setIdx + 1)
|
setN := humanize.Ordinal(setIdx + 1)
|
||||||
retryDur := time.Duration(rand.Float64() * float64(5*time.Second))
|
retryDur := time.Duration(rand.Float64() * float64(5*time.Second))
|
||||||
decomLogOnceIf(ctx, fmt.Errorf("listing objects from %s set failed with %v, retrying in %v", setN, err, retryDur), "decom-listing-failed"+setN)
|
decomLogOnceIf(ctx, fmt.Errorf("listing objects from %s set failed with %v, retrying in %v", setN, err, retryDur), "decom-listing-failed"+setN)
|
||||||
time.Sleep(retryDur)
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-time.After(retryDur):
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}(setIdx)
|
}(setIdx)
|
||||||
}
|
}
|
||||||
wk.Wait()
|
wk.Wait()
|
||||||
return nil
|
if err := ctx.Err(); err != nil {
|
||||||
|
return removed.Load(), err
|
||||||
|
}
|
||||||
|
return removed.Load(), verifyErr
|
||||||
}
|
}
|
||||||
|
|
||||||
//msgp:ignore decomMetrics
|
//msgp:ignore decomMetrics
|
||||||
@@ -1113,6 +1167,9 @@ func (m *decomMetrics) log(d decomMetric, poolIdx int, paths ...string) func(z i
|
|||||||
// decommission fails and a retry resumes with what is left. A canceled
|
// decommission fails and a retry resumes with what is left. A canceled
|
||||||
// context returns at once.
|
// context returns at once.
|
||||||
func (z *erasureServerPools) decommissionInBackground(ctx context.Context, idx int) error {
|
func (z *erasureServerPools) decommissionInBackground(ctx context.Context, idx int) error {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
pool := z.serverPools[idx]
|
pool := z.serverPools[idx]
|
||||||
z.poolMetaMutex.RLock()
|
z.poolMetaMutex.RLock()
|
||||||
pending := z.poolMeta.PendingBuckets(idx)
|
pending := z.poolMeta.PendingBuckets(idx)
|
||||||
@@ -1151,7 +1208,7 @@ func (z *erasureServerPools) decommissionInBackground(ctx context.Context, idx i
|
|||||||
if err := z.decommissionPool(ctx, idx, pool, bucket); err != nil {
|
if err := z.decommissionPool(ctx, idx, pool, bucket); err != nil {
|
||||||
stopFn(0, err)
|
stopFn(0, err)
|
||||||
if contextCanceled(ctx) {
|
if contextCanceled(ctx) {
|
||||||
return err
|
return ctx.Err()
|
||||||
}
|
}
|
||||||
failed(bucket, err)
|
failed(bucket, err)
|
||||||
continue
|
continue
|
||||||
@@ -1164,7 +1221,7 @@ func (z *erasureServerPools) decommissionInBackground(ctx context.Context, idx i
|
|||||||
if err := z.checkAfterDecomBucket(ctx, idx, bucket); err != nil {
|
if err := z.checkAfterDecomBucket(ctx, idx, bucket); err != nil {
|
||||||
stopFn(0, err)
|
stopFn(0, err)
|
||||||
if contextCanceled(ctx) {
|
if contextCanceled(ctx) {
|
||||||
return err
|
return ctx.Err()
|
||||||
}
|
}
|
||||||
failed(bucket, err)
|
failed(bucket, err)
|
||||||
continue
|
continue
|
||||||
@@ -1177,6 +1234,9 @@ func (z *erasureServerPools) decommissionInBackground(ctx context.Context, idx i
|
|||||||
}
|
}
|
||||||
z.poolMetaMutex.Unlock()
|
z.poolMetaMutex.Unlock()
|
||||||
}
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
return firstErr
|
return firstErr
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1211,34 +1271,74 @@ func (z *erasureServerPools) removeFreeVersions(ctx context.Context, bucket, obj
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkAfterDecomBucket verifies that a source bucket is physically empty
|
// checkAfterDecomBucket verifies that a source bucket's online drives are empty
|
||||||
// before it is removed from the resumable decommission queue. ILM decisions
|
// before it is removed from the resumable decommission queue. ILM decisions
|
||||||
// are deliberately not treated as an exemption: an expired version remains
|
// are deliberately not treated as an exemption: an expired version remains
|
||||||
// until the expiry path has performed its logical delete, replication, and
|
// until the expiry path has performed its logical delete, replication, and
|
||||||
// tier cleanup.
|
// tier cleanup.
|
||||||
func (z *erasureServerPools) checkAfterDecomBucket(ctx context.Context, idx int, bi decomBucketInfo) error {
|
func (z *erasureServerPools) checkAfterDecomBucket(ctx context.Context, idx int, bi decomBucketInfo) error {
|
||||||
versionsFound := 0
|
if err := ctx.Err(); err != nil {
|
||||||
for _, set := range z.serverPools[idx].sets {
|
|
||||||
if err := set.listObjectsToDecommission(ctx, bi, func(entry metaCacheEntry) {
|
|
||||||
if !entry.isObject() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if bi.Name == minioMetaBucket && strings.Contains(entry.name, dataUsageCacheName) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
fivs, err := entry.fileInfoVersions(bi.Name)
|
|
||||||
if err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
versionsFound += len(fivs.Versions) + len(fivs.FreeVersions)
|
|
||||||
}); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
for _, set := range z.serverPools[idx].sets {
|
||||||
|
disks := set.getDisks()
|
||||||
|
if len(disks) == 0 {
|
||||||
|
return toObjectErr(errErasureReadQuorum, bi.Name)
|
||||||
}
|
}
|
||||||
if versionsFound > 0 {
|
readQuorum := set.migrationReadQuorum(len(disks))
|
||||||
return fmt.Errorf("at least %d object(s)/version(s) were found in bucket `%s` after decommissioning", versionsFound, bi.Name)
|
onlineCount := 0
|
||||||
|
for _, disk := range disks {
|
||||||
|
if disk == nil {
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
return nil
|
// Quorum listings omit minority versions, and WalkDir can skip
|
||||||
|
// unreadable metadata. Inspect physical directory entries instead:
|
||||||
|
// even a corrupt or truncated xl.meta must keep this pool queued.
|
||||||
|
pending := []string{bi.Prefix}
|
||||||
|
diskOffline := false
|
||||||
|
for len(pending) > 0 {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
dir := pending[len(pending)-1]
|
||||||
|
pending = pending[:len(pending)-1]
|
||||||
|
entries, err := disk.ListDir(ctx, "", bi.Name, dir, -1)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if errors.Is(err, errFileNotFound) || errors.Is(err, errVolumeNotFound) {
|
||||||
|
continue // A concurrent deletion can remove the directory.
|
||||||
|
}
|
||||||
|
if errors.Is(err, errDiskNotFound) {
|
||||||
|
diskOffline = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, entry := range entries {
|
||||||
|
name := pathJoin(dir, entry)
|
||||||
|
if bi.Name == minioMetaBucket && strings.Contains(name, dataUsageCacheName) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if entry == xlStorageFormatFile || entry == xlStorageFormatFileV1 {
|
||||||
|
return fmt.Errorf("object metadata %s remains in bucket %s after decommissioning", name, bi.Name)
|
||||||
|
}
|
||||||
|
if strings.HasSuffix(entry, SlashSeparator) {
|
||||||
|
pending = append(pending, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if diskOffline {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
onlineCount++
|
||||||
|
}
|
||||||
|
if onlineCount < readQuorum {
|
||||||
|
return toObjectErr(errErasureReadQuorum, bi.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ctx.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (z *erasureServerPools) checkAfterDecom(ctx context.Context, idx int) error {
|
func (z *erasureServerPools) checkAfterDecom(ctx context.Context, idx int) error {
|
||||||
|
|||||||
@@ -1260,6 +1260,7 @@ func TestPoolsLifecycleCheckedExpirySafeguard(t *testing.T) {
|
|||||||
{"hold-equal-stamps", poolLockMetadata("", "ON", old, recent), objectLockState{legalHold: "OFF", legalHoldTimestamp: recent}, false},
|
{"hold-equal-stamps", poolLockMetadata("", "ON", old, recent), objectLockState{legalHold: "OFF", legalHoldTimestamp: recent}, false},
|
||||||
{"hold-missing-approved-stamp", poolLockMetadata("", "ON", old, old), objectLockState{legalHold: "OFF"}, false},
|
{"hold-missing-approved-stamp", poolLockMetadata("", "ON", old, old), objectLockState{legalHold: "OFF"}, false},
|
||||||
{"retention-newer-locally", poolLockMetadata("GOVERNANCE", "OFF", recent, old), objectLockState{retentionTimestamp: old, legalHold: "OFF", legalHoldTimestamp: old}, false},
|
{"retention-newer-locally", poolLockMetadata("GOVERNANCE", "OFF", recent, old), objectLockState{retentionTimestamp: old, legalHold: "OFF", legalHoldTimestamp: old}, false},
|
||||||
|
{"compliance-cannot-release", poolLockMetadata("COMPLIANCE", "OFF", old, old), objectLockState{retentionTimestamp: recent, legalHold: "OFF", legalHoldTimestamp: old}, false},
|
||||||
// Controls: the approval strictly supersedes the local protection.
|
// Controls: the approval strictly supersedes the local protection.
|
||||||
{"hold-released", poolLockMetadata("", "ON", old, old), objectLockState{legalHold: "OFF", legalHoldTimestamp: recent}, true},
|
{"hold-released", poolLockMetadata("", "ON", old, old), objectLockState{legalHold: "OFF", legalHoldTimestamp: recent}, true},
|
||||||
{"retention-released", poolLockMetadata("GOVERNANCE", "OFF", old, old), objectLockState{retentionTimestamp: recent, legalHold: "OFF", legalHoldTimestamp: old}, true},
|
{"retention-released", poolLockMetadata("GOVERNANCE", "OFF", old, old), objectLockState{retentionTimestamp: recent, legalHold: "OFF", legalHoldTimestamp: old}, true},
|
||||||
@@ -1666,7 +1667,7 @@ func TestPoolsMigrationLifecycleRefusedExpiryKeepsSource(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
version := FileInfo{Name: key, VersionID: id, ModTime: source.ModTime}
|
version := FileInfo{Name: key, VersionID: id, ModTime: source.ModTime}
|
||||||
err = s.z.removeMovedVersions(ctx, s.bucket, key, 0, []FileInfo{version}, []migrationExpiredVersion{{
|
_, err = s.z.removeMovedVersions(ctx, s.bucket, key, 0, []FileInfo{version}, []migrationExpiredVersion{{
|
||||||
version: version,
|
version: version,
|
||||||
event: lifecycle.Event{Action: lifecycle.DeleteAction},
|
event: lifecycle.Event{Action: lifecycle.DeleteAction},
|
||||||
lifecycleUpdatedAt: updatedAt,
|
lifecycleUpdatedAt: updatedAt,
|
||||||
|
|||||||
@@ -70,6 +70,8 @@ func TestPoolsLifecycleDuplicateLockMetadata(t *testing.T) {
|
|||||||
{name: "independent-fields", copies: [2]map[string]string{poolLockMetadata("GOVERNANCE", "OFF", recent, old), poolLockMetadata("", "ON", old, recent)}, mode: "GOVERNANCE", hold: "ON"},
|
{name: "independent-fields", copies: [2]map[string]string{poolLockMetadata("GOVERNANCE", "OFF", recent, old), poolLockMetadata("", "ON", old, recent)}, mode: "GOVERNANCE", hold: "ON"},
|
||||||
{name: "hold-released", copies: [2]map[string]string{poolLockMetadata("", "ON", old, old), poolLockMetadata("", "OFF", old, recent)}, hold: "OFF", delete: true},
|
{name: "hold-released", copies: [2]map[string]string{poolLockMetadata("", "ON", old, old), poolLockMetadata("", "OFF", old, recent)}, hold: "OFF", delete: true},
|
||||||
{name: "retention-released", copies: [2]map[string]string{poolLockMetadata("GOVERNANCE", "OFF", old, old), poolLockMetadata("", "OFF", recent, old)}, hold: "OFF", delete: true},
|
{name: "retention-released", copies: [2]map[string]string{poolLockMetadata("GOVERNANCE", "OFF", old, old), poolLockMetadata("", "OFF", recent, old)}, hold: "OFF", delete: true},
|
||||||
|
{name: "compliance-before-removal", copies: [2]map[string]string{poolLockMetadata("COMPLIANCE", "OFF", old, old), poolLockMetadata("", "OFF", recent, old)}, mode: "COMPLIANCE", hold: "OFF"},
|
||||||
|
{name: "compliance-after-removal", copies: [2]map[string]string{poolLockMetadata("", "OFF", recent, old), poolLockMetadata("COMPLIANCE", "OFF", old, old)}, mode: "COMPLIANCE", hold: "OFF"},
|
||||||
} {
|
} {
|
||||||
for _, batch := range []bool{false, true} {
|
for _, batch := range []bool{false, true} {
|
||||||
path := "single"
|
path := "single"
|
||||||
|
|||||||
@@ -587,20 +587,10 @@ func setLifecycleLockState(metadata map[string]string, state objectLockState) {
|
|||||||
set(ReservedMetadataPrefix+ObjectLockLegalHoldTimestamp, state.legalHoldTimestamp)
|
set(ReservedMetadataPrefix+ObjectLockLegalHoldTimestamp, state.legalHoldTimestamp)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newestLockState returns the newest hold and the newest retention of the
|
// newestLockState orders independently revised fields without weakening a
|
||||||
// copies of a version, which lockRevisionsOrdered found equal or ordered.
|
// COMPLIANCE retention that any copy still protects.
|
||||||
func newestLockState(copies []PoolObjInfo) objectLockState {
|
func newestLockState(copies []PoolObjInfo) objectLockState {
|
||||||
state := lifecycleLockState(copies[0].ObjInfo.UserDefined)
|
return reducePoolLockState(copies, lifecycleLockState)
|
||||||
for _, copy := range copies[1:] {
|
|
||||||
next := lifecycleLockState(copy.ObjInfo.UserDefined)
|
|
||||||
if lockRevisionNewer(next.legalHoldTimestamp, state.legalHoldTimestamp) {
|
|
||||||
state.legalHold, state.legalHoldTimestamp = next.legalHold, next.legalHoldTimestamp
|
|
||||||
}
|
|
||||||
if lockRevisionNewer(next.retentionTimestamp, state.retentionTimestamp) {
|
|
||||||
state.mode, state.retainUntil, state.retentionTimestamp = next.mode, next.retainUntil, next.retentionTimestamp
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return state
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkedExpiryRefused reports whether a set must keep a version whose ILM
|
// checkedExpiryRefused reports whether a set must keep a version whose ILM
|
||||||
@@ -623,7 +613,7 @@ func checkedExpiryRefused(lc lifecycle.Lifecycle, lr objectlock.Retention, rcfg
|
|||||||
if lockRevisionNewer(approved.legalHoldTimestamp, local.legalHoldTimestamp) {
|
if lockRevisionNewer(approved.legalHoldTimestamp, local.legalHoldTimestamp) {
|
||||||
obj.UserDefined[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = approved.legalHold
|
obj.UserDefined[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = approved.legalHold
|
||||||
}
|
}
|
||||||
if lockRevisionNewer(approved.retentionTimestamp, local.retentionTimestamp) {
|
if lockRevisionNewer(approved.retentionTimestamp, local.retentionTimestamp) && !local.complianceWouldWeaken(approved) {
|
||||||
obj.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] = approved.mode
|
obj.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] = approved.mode
|
||||||
obj.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = approved.retainUntil
|
obj.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = approved.retainUntil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -132,7 +132,7 @@ func TestPoolsMigrationCleanupRechecksSource(t *testing.T) {
|
|||||||
_, err := z.serverPools[1].DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: nullVersionID})
|
_, err := z.serverPools[1].DeleteObject(t.Context(), bucket, oi.Name, ObjectOptions{VersionID: nullVersionID})
|
||||||
mustIAM(t, err)
|
mustIAM(t, err)
|
||||||
}
|
}
|
||||||
err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot, nil)
|
_, err := z.removeMovedVersions(t.Context(), bucket, oi.Name, 0, snapshot, nil)
|
||||||
if change != "delete-uuid" && err == nil {
|
if change != "delete-uuid" && err == nil {
|
||||||
t.Fatal("cleanup accepted changed or uncertain state")
|
t.Fatal("cleanup accepted changed or uncertain state")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -414,6 +414,46 @@ func testMigrationLifecycleExpiry(t *testing.T, kind string) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Physical expiry advances the walk; creating a delete marker does not remove
|
||||||
|
// the addressed version and must not make a walk repeat without progress.
|
||||||
|
func TestPoolsMigrationLifecycleRemovalProgress(t *testing.T) {
|
||||||
|
for _, versioned := range []bool{false, true} {
|
||||||
|
status := versioning.State("")
|
||||||
|
name, wantRemoved := "physical-delete", 1
|
||||||
|
if versioned {
|
||||||
|
status = versioning.Enabled
|
||||||
|
name, wantRemoved = "delete-marker-only", 0
|
||||||
|
}
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
m := newMigrationILM(t, status, false, migrationILMExpireCurrent())
|
||||||
|
oi := m.put("obj-01", "v1", ObjectOptions{Versioned: versioned, MTime: UTCNow().Add(-72 * time.Hour)})
|
||||||
|
_, updatedAt, err := globalBucketMetadataSys.GetLifecycleConfig(m.bucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
version := FileInfo{Name: oi.Name, VersionID: oi.VersionID, ModTime: oi.ModTime}
|
||||||
|
removed, err := m.z.removeMovedVersions(t.Context(), m.bucket, oi.Name, 0, []FileInfo{version}, []migrationExpiredVersion{{
|
||||||
|
version: version,
|
||||||
|
event: lifecycle.Event{Action: lifecycle.DeleteAction},
|
||||||
|
lifecycleUpdatedAt: updatedAt,
|
||||||
|
source: lcEventSrc_Decom,
|
||||||
|
}})
|
||||||
|
if err != nil || removed != wantRemoved {
|
||||||
|
t.Fatalf("removed %d versions, err %v; want %d, nil", removed, err, wantRemoved)
|
||||||
|
}
|
||||||
|
if versioned {
|
||||||
|
if body := migrationTestBody(t, m.z.GetObjectNInfo, m.bucket, oi.Name, oi.VersionID); body != "obj-01 v1" {
|
||||||
|
t.Fatalf("noncurrent version reads %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
current, err := m.z.GetObjectInfo(t.Context(), m.bucket, oi.Name, ObjectOptions{})
|
||||||
|
if !isErrObjectNotFound(err) || current.DeleteMarker != versioned {
|
||||||
|
t.Fatalf("after expiry: delete marker %t, err %v", current.DeleteMarker, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// A lifecycle action found during the migration walk must not be applied
|
// A lifecycle action found during the migration walk must not be applied
|
||||||
// after the bucket's lifecycle configuration changes. The source copy stays
|
// after the bucket's lifecycle configuration changes. The source copy stays
|
||||||
// available for a fresh scan under the new policy.
|
// available for a fresh scan under the new policy.
|
||||||
|
|||||||
@@ -0,0 +1,601 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful,
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"slices"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/bucket/lifecycle"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The walk has already captured the version when its data read reaches this
|
||||||
|
// hook. Metadata-only checks made by a concurrent DELETE do not reenter it.
|
||||||
|
type migrationProgressReadDisk struct {
|
||||||
|
StorageAPI
|
||||||
|
bucket, object, version string
|
||||||
|
before func() error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d migrationProgressReadDisk) ReadVersion(ctx context.Context, orig, bucket, object, version string, opts ReadOptions) (FileInfo, error) {
|
||||||
|
if opts.ReadData && bucket == d.bucket && object == d.object && version == d.version {
|
||||||
|
if err := d.before(); err != nil {
|
||||||
|
return FileInfo{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return d.StorageAPI.ReadVersion(ctx, orig, bucket, object, version, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Buffer every drive's first walk before publishing any entry. Thus deleting
|
||||||
|
// the parent cannot accidentally let this same walk discover the child.
|
||||||
|
type migrationSnapshotWalk struct {
|
||||||
|
bucket string
|
||||||
|
drives int32
|
||||||
|
calls atomic.Int32
|
||||||
|
ready chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
type migrationSnapshotWalkDisk struct {
|
||||||
|
StorageAPI
|
||||||
|
snapshot *migrationSnapshotWalk
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d migrationSnapshotWalkDisk) WalkDir(ctx context.Context, opts WalkDirOptions, out io.Writer) error {
|
||||||
|
if opts.Bucket != d.snapshot.bucket {
|
||||||
|
return d.StorageAPI.WalkDir(ctx, opts, out)
|
||||||
|
}
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := d.StorageAPI.WalkDir(ctx, opts, &buf); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if d.snapshot.calls.Add(1) == d.snapshot.drives {
|
||||||
|
close(d.snapshot.ready)
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-d.snapshot.ready:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
_, err := io.Copy(out, &buf)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func migrationProgressSnapshot(t *testing.T, z *erasureServerPools, bucket, key string) *migrationSnapshotWalk {
|
||||||
|
t.Helper()
|
||||||
|
s := &migrationSnapshotWalk{bucket: bucket, drives: int32(len(z.serverPools[0].getHashedSet(key).getDisks())), ready: make(chan struct{})}
|
||||||
|
t.Cleanup(conditionalPutSwapDisks(z.serverPools[0], key, func(d StorageAPI) StorageAPI {
|
||||||
|
return migrationSnapshotWalkDisk{StorageAPI: d, snapshot: s}
|
||||||
|
}))
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func migrationProgressPrepare(t *testing.T, z *erasureServerPools, kind string) {
|
||||||
|
t.Helper()
|
||||||
|
migrationTestMover(t, z, 0, kind)
|
||||||
|
z.poolMetaMutex.Lock()
|
||||||
|
z.poolMeta.Pools[0].LastUpdate = UTCNow()
|
||||||
|
z.poolMetaMutex.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecommissionSplitMetadataDoesNotAdvance(t *testing.T) {
|
||||||
|
m := newMigrationILM(t, "", false, migrationILMExpireCurrent())
|
||||||
|
const key = "obj-01"
|
||||||
|
m.put(key, "old", ObjectOptions{MTime: UTCNow().Add(-96 * time.Hour)})
|
||||||
|
disks := m.z.serverPools[0].getHashedSet(key).getDisks()
|
||||||
|
old := make([][]byte, len(disks)/2)
|
||||||
|
for i := range old {
|
||||||
|
var err error
|
||||||
|
old[i], err = disks[i].ReadAll(t.Context(), m.bucket, pathJoin(key, xlStorageFormatFile))
|
||||||
|
mustIAM(t, err)
|
||||||
|
}
|
||||||
|
m.put(key, "new-and-longer", ObjectOptions{MTime: UTCNow().Add(-72 * time.Hour)})
|
||||||
|
for i := range old {
|
||||||
|
mustIAM(t, disks[i].WriteAll(t.Context(), m.bucket, pathJoin(key, xlStorageFormatFile), old[i]))
|
||||||
|
}
|
||||||
|
if _, err := m.z.serverPools[0].GetObjectInfo(t.Context(), m.bucket, key, ObjectOptions{VersionID: nullVersionID}); !isErrVersionNotFound(err) {
|
||||||
|
t.Fatalf("fixture must expose the ambiguous logical NotFound, got %v", err)
|
||||||
|
}
|
||||||
|
migrationProgressPrepare(t, m.z, "decommission")
|
||||||
|
walk := migrationProgressSnapshot(t, m.z, m.bucket, key)
|
||||||
|
bi := decomBucketInfo{Name: m.bucket}
|
||||||
|
// Check a single pass first, so the unfixed implementation fails rather
|
||||||
|
// than looping until the test timeout.
|
||||||
|
removed, err := m.z.decommissionPoolWalk(t.Context(), 0, m.z.serverPools[0], bi)
|
||||||
|
if err != nil || removed != 0 {
|
||||||
|
t.Fatalf("split metadata reported progress: removed=%d err=%v", removed, err)
|
||||||
|
}
|
||||||
|
mustIAM(t, m.z.decommissionPool(t.Context(), 0, m.z.serverPools[0], bi))
|
||||||
|
if got := walk.calls.Load(); got != 2*walk.drives {
|
||||||
|
t.Fatalf("unchanged metadata repeated the walk: calls=%d", got)
|
||||||
|
}
|
||||||
|
if err := m.z.checkAfterDecomBucket(t.Context(), 0, bi); err == nil {
|
||||||
|
t.Fatal("final verification accepted split metadata still on disk")
|
||||||
|
}
|
||||||
|
for _, disk := range disks {
|
||||||
|
if _, err := disk.ReadVersion(t.Context(), "", m.bucket, key, nullVersionID, ReadOptions{}); err != nil {
|
||||||
|
t.Fatalf("migration removed an ambiguous source revision: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPoolsMigrationConcurrentDeleteCleanup(t *testing.T) {
|
||||||
|
for _, kind := range []string{"decommission", "rebalance"} {
|
||||||
|
t.Run(kind, func(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
v1 := putConsistencyObject(t, z, bucket, "k", 0, "older", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
|
||||||
|
v2 := putConsistencyObject(t, z, bucket, "k", 0, "newer", ObjectOptions{Versioned: true})
|
||||||
|
var once sync.Once
|
||||||
|
var deleteErr error
|
||||||
|
before := func() error {
|
||||||
|
once.Do(func() {
|
||||||
|
_, deleteErr = z.DeleteObject(t.Context(), bucket, "k", ObjectOptions{Versioned: true, VersionID: v2.VersionID})
|
||||||
|
})
|
||||||
|
return deleteErr
|
||||||
|
}
|
||||||
|
restore := conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
return migrationProgressReadDisk{StorageAPI: d, bucket: bucket, object: "k", version: v2.VersionID, before: before}
|
||||||
|
})
|
||||||
|
defer restore()
|
||||||
|
if failed := migrationTestWalk(t, z, bucket, 0, kind); failed != 0 {
|
||||||
|
t.Fatalf("failed moves: %d", failed)
|
||||||
|
}
|
||||||
|
restore()
|
||||||
|
for _, v := range []ObjectInfo{v1, v2} {
|
||||||
|
for _, disk := range z.serverPools[0].getHashedSet("k").getDisks() {
|
||||||
|
_, err := disk.ReadVersion(t.Context(), "", bucket, "k", v.VersionID, ReadOptions{})
|
||||||
|
if !errors.Is(err, errFileVersionNotFound) && !errors.Is(err, errFileNotFound) {
|
||||||
|
t.Fatalf("source version %s remained: %v", v.VersionID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k", v1.VersionID); body != "older" {
|
||||||
|
t.Fatalf("moved version reads %q", body)
|
||||||
|
}
|
||||||
|
if _, err := z.GetObjectInfo(t.Context(), bucket, "k", ObjectOptions{VersionID: v2.VersionID}); !isErrVersionNotFound(err) {
|
||||||
|
t.Fatalf("concurrently deleted version resurrected: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPoolsMigrationSiblingFailureCleanup(t *testing.T) {
|
||||||
|
for _, kind := range []string{"decommission", "rebalance"} {
|
||||||
|
for _, loseDestination := range []bool{false, true} {
|
||||||
|
name := kind + "/destination-retained"
|
||||||
|
if loseDestination {
|
||||||
|
name = kind + "/destination-removed"
|
||||||
|
}
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
v1 := putConsistencyObject(t, z, bucket, "k", 0, "older", ObjectOptions{Versioned: true, MTime: UTCNow().Add(-time.Hour)})
|
||||||
|
v2 := putConsistencyObject(t, z, bucket, "k", 0, "newer", ObjectOptions{Versioned: true})
|
||||||
|
var once sync.Once
|
||||||
|
var deleteErr error
|
||||||
|
before := func() error {
|
||||||
|
once.Do(func() {
|
||||||
|
if loseDestination {
|
||||||
|
_, deleteErr = z.serverPools[1].DeleteObject(t.Context(), bucket, "k", ObjectOptions{VersionID: v1.VersionID})
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return errDiskNotFound // only the sibling's data read fails
|
||||||
|
}
|
||||||
|
restore := conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
return migrationProgressReadDisk{StorageAPI: d, bucket: bucket, object: "k", version: v2.VersionID, before: before}
|
||||||
|
})
|
||||||
|
defer restore()
|
||||||
|
migrationProgressPrepare(t, z, kind)
|
||||||
|
if kind == "rebalance" {
|
||||||
|
mustIAM(t, z.rebalanceBucket(t.Context(), bucket, 0))
|
||||||
|
} else {
|
||||||
|
_, err := z.decommissionPoolWalk(t.Context(), 0, z.serverPools[0], decomBucketInfo{Name: bucket})
|
||||||
|
mustIAM(t, err)
|
||||||
|
if err := z.checkAfterDecomBucket(t.Context(), 0, decomBucketInfo{Name: bucket}); err == nil {
|
||||||
|
t.Fatal("failed sibling must block final decommission verification")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
restore()
|
||||||
|
mustIAM(t, deleteErr)
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k", v2.VersionID); body != "newer" {
|
||||||
|
t.Fatalf("failed sibling reads %q", body)
|
||||||
|
}
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k", v1.VersionID); body != "older" {
|
||||||
|
t.Fatalf("successful candidate lost: %q", body)
|
||||||
|
}
|
||||||
|
_, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, "k", ObjectOptions{VersionID: v1.VersionID})
|
||||||
|
if loseDestination && err != nil || !loseDestination && !isErrVersionNotFound(err) {
|
||||||
|
t.Fatalf("source cleanup ignored destination state: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecommissionConcurrentParentDeleteRescansChildren(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
putConsistencyObject(t, z, bucket, "k", 0, "parent", ObjectOptions{})
|
||||||
|
putConsistencyObject(t, z, bucket, "k/b", 0, "child", ObjectOptions{})
|
||||||
|
migrationProgressSnapshot(t, z, bucket, "k")
|
||||||
|
var once sync.Once
|
||||||
|
var deleteErr error
|
||||||
|
t.Cleanup(conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
return migrationProgressReadDisk{StorageAPI: d, bucket: bucket, object: "k", version: nullVersionID, before: func() error {
|
||||||
|
once.Do(func() {
|
||||||
|
_, deleteErr = z.DeleteObject(t.Context(), bucket, "k", ObjectOptions{VersionID: nullVersionID})
|
||||||
|
})
|
||||||
|
return deleteErr
|
||||||
|
}}
|
||||||
|
}))
|
||||||
|
if failed := migrationTestWalk(t, z, bucket, 0, "decommission"); failed != 0 {
|
||||||
|
t.Fatalf("failed moves: %d", failed)
|
||||||
|
}
|
||||||
|
mustIAM(t, z.checkAfterDecomBucket(t.Context(), 0, decomBucketInfo{Name: bucket}))
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k/b", ""); body != "child" {
|
||||||
|
t.Fatalf("child reads %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecommissionDegradedPoolRescansChildren(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
putConsistencyObject(t, z, bucket, "k", 0, "parent", ObjectOptions{})
|
||||||
|
putConsistencyObject(t, z, bucket, "k/b", 0, "child", ObjectOptions{})
|
||||||
|
var wrapped int
|
||||||
|
t.Cleanup(conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
wrapped++
|
||||||
|
if wrapped == 1 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return d
|
||||||
|
}))
|
||||||
|
if failed := migrationTestWalk(t, z, bucket, 0, "decommission"); failed != 0 {
|
||||||
|
t.Fatalf("failed moves: %d", failed)
|
||||||
|
}
|
||||||
|
mustIAM(t, z.checkAfterDecomBucket(t.Context(), 0, decomBucketInfo{Name: bucket}))
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k/b", ""); body != "child" {
|
||||||
|
t.Fatalf("child reads %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFreeVersionDecommissionRescansChildren(t *testing.T) {
|
||||||
|
f := newFreeVersionTest(t, "")
|
||||||
|
f.bucket = "free-parent"
|
||||||
|
mustIAM(t, f.z.MakeBucket(t.Context(), f.bucket, MakeBucketOptions{VersioningEnabled: true}))
|
||||||
|
f.deleteVersion(f.tiered("k", "parent", UTCNow().Add(-time.Hour)))
|
||||||
|
f.put("k/b", "child", UTCNow())
|
||||||
|
migrationProgressSnapshot(t, f.z, f.bucket, "k")
|
||||||
|
migrationProgressPrepare(t, f.z, "decommission")
|
||||||
|
bi := decomBucketInfo{Name: f.bucket}
|
||||||
|
removed, err := f.z.decommissionPoolWalk(t.Context(), 0, f.z.serverPools[0], bi)
|
||||||
|
if err != nil || removed != 1 {
|
||||||
|
t.Fatalf("free-only parent removal: removed=%d err=%v", removed, err)
|
||||||
|
}
|
||||||
|
if err := f.z.checkAfterDecomBucket(t.Context(), 0, bi); err == nil {
|
||||||
|
t.Fatal("the newly exposed child must still block completion")
|
||||||
|
}
|
||||||
|
mustIAM(t, f.z.decommissionPool(t.Context(), 0, f.z.serverPools[0], bi))
|
||||||
|
mustIAM(t, f.z.checkAfterDecomBucket(t.Context(), 0, bi))
|
||||||
|
if body := migrationTestBody(t, f.z.GetObjectNInfo, f.bucket, "k/b", ""); body != "k/b child" {
|
||||||
|
t.Fatalf("child reads %q", body)
|
||||||
|
}
|
||||||
|
if !slices.Contains(f.tier.removed, f.remote("k", "parent")) {
|
||||||
|
t.Fatal("free version cleanup did not delete its remote object")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPoolsMigrationCancellation(t *testing.T) {
|
||||||
|
for _, kind := range []string{"decommission", "rebalance"} {
|
||||||
|
for _, phase := range []string{"before-walk", "during-read", "deadline"} {
|
||||||
|
name := kind + "/" + phase
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
oi := putConsistencyObject(t, z, bucket, "k", 0, "source", ObjectOptions{Versioned: true})
|
||||||
|
migrationProgressPrepare(t, z, kind)
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
wantErr := context.Canceled
|
||||||
|
if phase == "deadline" {
|
||||||
|
cancel()
|
||||||
|
ctx, cancel = context.WithDeadline(t.Context(), time.Time{})
|
||||||
|
wantErr = context.DeadlineExceeded
|
||||||
|
}
|
||||||
|
defer cancel()
|
||||||
|
if phase == "during-read" {
|
||||||
|
t.Cleanup(conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
return migrationProgressReadDisk{StorageAPI: d, bucket: bucket, object: "k", version: oi.VersionID, before: func() error { cancel(); return ctx.Err() }}
|
||||||
|
}))
|
||||||
|
} else {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
if kind == "decommission" {
|
||||||
|
err = z.decommissionPool(ctx, 0, z.serverPools[0], decomBucketInfo{Name: bucket})
|
||||||
|
} else {
|
||||||
|
err = z.rebalanceBucket(ctx, bucket, 0)
|
||||||
|
}
|
||||||
|
if !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("cancellation returned %v", err)
|
||||||
|
}
|
||||||
|
if err := z.decommissionInBackground(ctx, 0); !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("canceled background migration returned %v", err)
|
||||||
|
}
|
||||||
|
if err := z.rebalanceBuckets(ctx, 0); !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("canceled background rebalance returned %v", err)
|
||||||
|
}
|
||||||
|
if n, err := z.countAbsentMigrationVersions(ctx, bucket, "k", 0, []FileInfo{{VersionID: oi.VersionID}}); n != 0 || !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("canceled progress verification: removed=%d err=%v", n, err)
|
||||||
|
}
|
||||||
|
if _, err := z.serverPools[0].GetObjectInfo(t.Context(), bucket, "k", ObjectOptions{VersionID: oi.VersionID}); err != nil {
|
||||||
|
t.Fatalf("canceled migration lost source: %v", err)
|
||||||
|
}
|
||||||
|
if err := z.checkAfterDecomBucket(ctx, 0, decomBucketInfo{Name: bucket}); !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("canceled verification returned %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type migrationProgressRawDisk struct {
|
||||||
|
StorageAPI
|
||||||
|
buf []byte
|
||||||
|
err error
|
||||||
|
reads *atomic.Int32
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d migrationProgressRawDisk) ReadXL(ctx context.Context, bucket, object string, data bool) (RawFileInfo, error) {
|
||||||
|
d.reads.Add(1)
|
||||||
|
if data {
|
||||||
|
return RawFileInfo{}, errors.New("progress check requested object data")
|
||||||
|
}
|
||||||
|
return RawFileInfo{Buf: d.buf}, d.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d migrationProgressRawDisk) ListDir(context.Context, string, string, string, int) ([]string, error) {
|
||||||
|
return nil, errFileNotFound
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPoolsMigrationPhysicalProgress(t *testing.T) {
|
||||||
|
f := newFreeVersionTest(t, "")
|
||||||
|
buf := f.xlMeta(0, "obj-01")
|
||||||
|
versions, err := getFileInfoVersions(buf, f.bucket, "obj-01", true)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if len(versions.Versions) != 3 || !versions.Versions[0].TierFreeVersion() {
|
||||||
|
t.Fatal("fixture must contain two live versions and one free version")
|
||||||
|
}
|
||||||
|
missing := FileInfo{VersionID: mustGetUUID()}
|
||||||
|
snapshot := append(slices.Clone(versions.Versions), missing)
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
buf []byte
|
||||||
|
err error
|
||||||
|
want int
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{name: "all-absent", err: errFileNotFound, want: 4},
|
||||||
|
{name: "missing-volume", err: errVolumeNotFound, want: 4},
|
||||||
|
{name: "minority-live-and-free", buf: buf, want: 1},
|
||||||
|
{name: "unavailable", err: errDiskNotFound, wantErr: true},
|
||||||
|
{name: "read-quorum", err: errErasureReadQuorum, wantErr: true},
|
||||||
|
{name: "corrupt-read", err: errFileCorrupt, wantErr: true},
|
||||||
|
{name: "corrupt-metadata", buf: []byte("broken xl.meta"), wantErr: true},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
var reads atomic.Int32
|
||||||
|
disks := make([]StorageAPI, 4)
|
||||||
|
for i := range disks {
|
||||||
|
disks[i] = migrationProgressRawDisk{err: errFileNotFound, reads: &reads}
|
||||||
|
}
|
||||||
|
disks[3] = migrationProgressRawDisk{buf: tc.buf, err: tc.err, reads: &reads}
|
||||||
|
er := erasureObjects{getDisks: func() []StorageAPI { return disks }}
|
||||||
|
n, err := er.countAbsentMigrationVersions(t.Context(), f.bucket, "obj-01", snapshot)
|
||||||
|
if n != tc.want || (err != nil) != tc.wantErr {
|
||||||
|
t.Fatalf("removed=%d err=%v, want %d / error=%t", n, err, tc.want, tc.wantErr)
|
||||||
|
}
|
||||||
|
if reads.Load() != int32(len(disks)) {
|
||||||
|
t.Fatalf("read count %d must not depend on version count", reads.Load())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
putConsistencyObject(t, f.z, f.bucket, "null", 0, "null-data", ObjectOptions{})
|
||||||
|
nullBuf := f.xlMeta(0, "null")
|
||||||
|
var reads atomic.Int32
|
||||||
|
er := erasureObjects{getDisks: func() []StorageAPI {
|
||||||
|
return []StorageAPI{migrationProgressRawDisk{buf: nullBuf, reads: &reads}}
|
||||||
|
}}
|
||||||
|
if n, err := er.countAbsentMigrationVersions(t.Context(), f.bucket, "null", []FileInfo{{VersionID: nullVersionID}, {VersionID: ""}}); n != 0 || err != nil {
|
||||||
|
t.Fatalf("null version normalization: removed=%d err=%v", n, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCountAbsentMigrationVersionsToleratesConfiguredOfflineDrives(t *testing.T) {
|
||||||
|
makeDisks := func(offlineCount int, asError bool) []StorageAPI {
|
||||||
|
disks := make([]StorageAPI, 4)
|
||||||
|
for i := range disks {
|
||||||
|
disks[i] = migrationProgressRawDisk{err: errFileNotFound, reads: &atomic.Int32{}}
|
||||||
|
}
|
||||||
|
for i := range offlineCount {
|
||||||
|
if asError {
|
||||||
|
disks[i] = migrationProgressRawDisk{err: errDiskNotFound, reads: &atomic.Int32{}}
|
||||||
|
} else {
|
||||||
|
disks[i] = nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return disks
|
||||||
|
}
|
||||||
|
snapshot := []FileInfo{{VersionID: mustGetUUID()}}
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
offline int
|
||||||
|
parity int
|
||||||
|
asError bool
|
||||||
|
wantCount int
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{name: "nil-drive-within-parity", offline: 1, parity: 1, wantCount: 1},
|
||||||
|
{name: "read-error-within-parity", offline: 1, parity: 1, asError: true, wantCount: 1},
|
||||||
|
{name: "half-offline-within-parity", offline: 2, parity: 2, wantCount: 1},
|
||||||
|
{name: "offline-beyond-parity", offline: 2, parity: 1, wantErr: true},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
disks := makeDisks(tc.offline, tc.asError)
|
||||||
|
er := erasureObjects{
|
||||||
|
setDriveCount: len(disks),
|
||||||
|
defaultParityCount: tc.parity,
|
||||||
|
getDisks: func() []StorageAPI { return disks },
|
||||||
|
}
|
||||||
|
got, err := er.countAbsentMigrationVersions(t.Context(), "bucket", "object", snapshot)
|
||||||
|
if got != tc.wantCount || (err != nil) != tc.wantErr {
|
||||||
|
t.Fatalf("removed=%d err=%v, want %d / error=%t", got, err, tc.wantCount, tc.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A successful quorum deletion can leave one source drive behind. It must not
|
||||||
|
// count as physical removal or pass the final check, nor cause a retry loop.
|
||||||
|
func TestDecommissionQuorumDeletionIsNotPhysicalProgress(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
oi := putConsistencyObject(t, z, bucket, "k", 0, "payload", ObjectOptions{Versioned: true})
|
||||||
|
move := migrationTestMover(t, z, 0, "decommission")
|
||||||
|
mustIAM(t, move(t.Context(), 0, bucket, migrationTestReader(t, z, 0, oi)))
|
||||||
|
first := true
|
||||||
|
restore := conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
if first {
|
||||||
|
first = false
|
||||||
|
return consistencyDeleteFaultDisk{StorageAPI: d, bucket: bucket, object: "k", version: oi.VersionID}
|
||||||
|
}
|
||||||
|
return d
|
||||||
|
})
|
||||||
|
defer restore()
|
||||||
|
snapshot := []FileInfo{{VersionID: oi.VersionID, ModTime: oi.ModTime}}
|
||||||
|
for range 2 {
|
||||||
|
n, err := z.removeMovedVersions(t.Context(), bucket, "k", 0, snapshot, nil)
|
||||||
|
if err != nil || n != 0 {
|
||||||
|
t.Fatalf("partial deletion reported progress: removed=%d err=%v", n, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := z.checkAfterDecomBucket(t.Context(), 0, decomBucketInfo{Name: bucket}); err == nil {
|
||||||
|
t.Fatal("final verification accepted a minority-drive source copy")
|
||||||
|
}
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k", oi.VersionID); body != "payload" {
|
||||||
|
t.Fatalf("destination copy lost: %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecommissionProgressReadFailureStopsWalk(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
putConsistencyObject(t, z, bucket, "k", 0, "payload", ObjectOptions{Versioned: true})
|
||||||
|
migrationProgressPrepare(t, z, "decommission")
|
||||||
|
walk := migrationProgressSnapshot(t, z, bucket, "k")
|
||||||
|
restore := conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
return tierReadFaultDisk{StorageAPI: d, bucket: bucket, object: "k", err: errDiskNotFound}
|
||||||
|
})
|
||||||
|
defer restore()
|
||||||
|
if err := z.decommissionPool(t.Context(), 0, z.serverPools[0], decomBucketInfo{Name: bucket}); err == nil {
|
||||||
|
t.Fatal("unreadable source progress check reported success")
|
||||||
|
}
|
||||||
|
if got := walk.calls.Load(); got != walk.drives {
|
||||||
|
t.Fatalf("verification failure repeated the walk: calls=%d", got)
|
||||||
|
}
|
||||||
|
restore()
|
||||||
|
if body := migrationTestBody(t, z.GetObjectNInfo, bucket, "k", ""); body != "payload" {
|
||||||
|
t.Fatalf("quorum move lost destination: %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type migrationProgressListFaultDisk struct {
|
||||||
|
StorageAPI
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d migrationProgressListFaultDisk) ListDir(context.Context, string, string, string, int) ([]string, error) {
|
||||||
|
return nil, errDiskAccessDenied
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecommissionFinalVerificationFailsClosed(t *testing.T) {
|
||||||
|
for _, state := range []string{"offline-drive", "directory-error", "corrupt-metadata", "truncated-metadata", "legacy-metadata"} {
|
||||||
|
t.Run(state, func(t *testing.T) {
|
||||||
|
z, bucket := migrationTestPools(t)
|
||||||
|
switch state {
|
||||||
|
case "offline-drive":
|
||||||
|
set := z.serverPools[0].getHashedSet("k")
|
||||||
|
maxOffline := set.defaultParityCount + 1
|
||||||
|
wrapped := 0
|
||||||
|
t.Cleanup(conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
wrapped++
|
||||||
|
if wrapped <= maxOffline {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return d
|
||||||
|
}))
|
||||||
|
case "directory-error":
|
||||||
|
t.Cleanup(conditionalPutSwapDisks(z.serverPools[0], "k", func(d StorageAPI) StorageAPI {
|
||||||
|
return migrationProgressListFaultDisk{StorageAPI: d}
|
||||||
|
}))
|
||||||
|
default:
|
||||||
|
// Real metadata on only one drive, without a readable quorum.
|
||||||
|
// This also covers errors that WalkDir deliberately skips.
|
||||||
|
disk := z.serverPools[0].getHashedSet("k").getDisks()[0]
|
||||||
|
file, data := xlStorageFormatFile, []byte("broken xl.meta")
|
||||||
|
if state == "truncated-metadata" {
|
||||||
|
data = nil
|
||||||
|
}
|
||||||
|
if state == "legacy-metadata" {
|
||||||
|
file = xlStorageFormatFileV1
|
||||||
|
}
|
||||||
|
mustIAM(t, disk.WriteAll(t.Context(), bucket, pathJoin("k", file), data))
|
||||||
|
}
|
||||||
|
if err := z.checkAfterDecomBucket(t.Context(), 0, decomBucketInfo{Name: bucket}); err == nil {
|
||||||
|
t.Fatal("unreadable storage was accepted as physically empty")
|
||||||
|
}
|
||||||
|
if n, err := z.countAbsentMigrationVersions(t.Context(), bucket, "k", 0, []FileInfo{{VersionID: mustGetUUID()}}); n != 0 || err == nil {
|
||||||
|
t.Fatalf("unreadable storage was counted as progress: removed=%d err=%v", n, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFreeVersionDecommissionPreservesLiveRemoteOwner(t *testing.T) {
|
||||||
|
f := newFreeVersionTest(t, "")
|
||||||
|
const key = "obj-02"
|
||||||
|
fv, tuple := f.freeRecord(0, key)
|
||||||
|
owner := putConsistencyObject(t, f.z, f.bucket, key, 1, "live-owner", ObjectOptions{Versioned: true})
|
||||||
|
infos, errs := readAllFileInfo(t.Context(), f.z.serverPools[1].getHashedSet(key).getDisks(), "", f.bucket, key, owner.VersionID, false, false)
|
||||||
|
mustIAM(t, errs[0])
|
||||||
|
fi := infos[0]
|
||||||
|
fi.TransitionStatus, fi.TransitionTier = lifecycle.TransitionComplete, tuple.Tier
|
||||||
|
fi.TransitionedObjName, fi.TransitionVersionID = tuple.Object, tuple.VersionID
|
||||||
|
mustIAM(t, f.z.serverPools[1].DecomTieredObject(t.Context(), f.bucket, key, fi, ObjectOptions{}))
|
||||||
|
if err := f.z.removeFreeVersions(t.Context(), f.bucket, key, 0, []FileInfo{fv}); !errors.Is(err, errTierStillReferenced) {
|
||||||
|
t.Fatalf("cleanup with a live remote owner: %v", err)
|
||||||
|
}
|
||||||
|
if n, err := f.z.countAbsentMigrationVersions(t.Context(), f.bucket, key, 0, []FileInfo{fv}); err != nil || n != 0 {
|
||||||
|
t.Fatalf("retained free record reported progress: removed=%d err=%v", n, err)
|
||||||
|
}
|
||||||
|
if slices.Contains(f.tier.removed, tuple.Object) {
|
||||||
|
t.Fatal("cleanup deleted a live remote object")
|
||||||
|
}
|
||||||
|
if err := f.z.checkAfterDecomBucket(t.Context(), 0, decomBucketInfo{Name: f.bucket}); err == nil {
|
||||||
|
t.Fatal("retained free record must block final decommission verification")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -17,7 +17,147 @@
|
|||||||
|
|
||||||
package cmd
|
package cmd
|
||||||
|
|
||||||
import xhttp "github.com/minio/minio/internal/http"
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/sync/errgroup"
|
||||||
|
)
|
||||||
|
|
||||||
|
func migrationVersionKey(versionID string) string {
|
||||||
|
if versionID == "" || versionID == nullVersionID {
|
||||||
|
return nullVersionID
|
||||||
|
}
|
||||||
|
return versionID
|
||||||
|
}
|
||||||
|
|
||||||
|
// migrationReadQuorum requires enough online drives to intersect every
|
||||||
|
// committed write quorum. All of those drives must be checked: a minority
|
||||||
|
// remnant still prevents progress. Unknown set geometry tolerates no outages.
|
||||||
|
func (er erasureObjects) migrationReadQuorum(diskCount int) int {
|
||||||
|
if er.setDriveCount <= 0 || er.setDriveCount != diskCount || er.defaultParityCount < 0 || er.defaultParityCount > er.setDriveCount/2 {
|
||||||
|
return diskCount
|
||||||
|
}
|
||||||
|
return er.defaultRQuorum()
|
||||||
|
}
|
||||||
|
|
||||||
|
// countAbsentMigrationVersions counts snapshot versions that no online source drive
|
||||||
|
// holds anymore, including free versions. A pool-level NotFound is insufficient:
|
||||||
|
// the ordinary read path can return it for inconsistent metadata. Only readable
|
||||||
|
// per-drive metadata (or physical absence) proves progress for another walk.
|
||||||
|
// The caller holds the object's writer lock while checking the source.
|
||||||
|
func (er erasureObjects) countAbsentMigrationVersions(ctx context.Context, bucket, object string, versions []FileInfo) (int, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if len(versions) == 0 {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
absent := make(map[string]struct{}, len(versions))
|
||||||
|
for _, version := range versions {
|
||||||
|
absent[migrationVersionKey(version.VersionID)] = struct{}{}
|
||||||
|
}
|
||||||
|
disks := er.getDisks()
|
||||||
|
if len(disks) == 0 {
|
||||||
|
return 0, toObjectErr(errErasureReadQuorum, bucket, object)
|
||||||
|
}
|
||||||
|
readQuorum := er.migrationReadQuorum(len(disks))
|
||||||
|
offline := make([]bool, len(disks))
|
||||||
|
stored := make([][]FileInfo, len(disks))
|
||||||
|
g := errgroup.WithNErrs(len(disks))
|
||||||
|
for i := range disks {
|
||||||
|
g.Go(func() error {
|
||||||
|
if disks[i] == nil {
|
||||||
|
offline[i] = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
raw, err := disks[i].ReadXL(ctx, bucket, object, false)
|
||||||
|
if errors.Is(err, errDiskNotFound) {
|
||||||
|
offline[i] = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if errors.Is(err, errFileNotFound) {
|
||||||
|
// ReadXL also returns NotFound for an empty xl.meta, and may
|
||||||
|
// ignore legacy metadata. Confirm physical absence before using
|
||||||
|
// that result as progress. This does not descend into child keys.
|
||||||
|
// Both reads run in parallel across drives, once per object.
|
||||||
|
entries, err := disks[i].ListDir(ctx, "", bucket, object, -1)
|
||||||
|
if errors.Is(err, errDiskNotFound) {
|
||||||
|
offline[i] = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil && !errors.Is(err, errFileNotFound) && !errors.Is(err, errVolumeNotFound) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry == xlStorageFormatFile || entry == xlStorageFormatFileV1 {
|
||||||
|
return errFileCorrupt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if errors.Is(err, errVolumeNotFound) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// allParts=false omits part details, not free versions. Decode every
|
||||||
|
// version before concluding absence, so corrupt metadata fails closed.
|
||||||
|
fivs, err := getAllFileInfoVersions(raw.Buf, bucket, object, false)
|
||||||
|
stored[i] = fivs.Versions
|
||||||
|
return err
|
||||||
|
}, i)
|
||||||
|
}
|
||||||
|
errs := g.Wait()
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
for i, err := range errs {
|
||||||
|
if err != nil {
|
||||||
|
return 0, toObjectErr(err, bucket, object)
|
||||||
|
}
|
||||||
|
for _, version := range stored[i] {
|
||||||
|
delete(absent, migrationVersionKey(version.VersionID))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
offlineCount := 0
|
||||||
|
for _, unavailable := range offline {
|
||||||
|
if unavailable {
|
||||||
|
offlineCount++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(disks)-offlineCount < readQuorum {
|
||||||
|
return 0, toObjectErr(errErasureReadQuorum, bucket, object)
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
return len(absent), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// countAbsentMigrationVersions checks ignored walk entries without authorizing
|
||||||
|
// their deletion. A concurrent client delete may have uncovered child keys.
|
||||||
|
func (z *erasureServerPools) countAbsentMigrationVersions(ctx context.Context, bucket, object string, source int, versions []FileInfo) (int, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if len(versions) == 0 {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
object = encodeDirObject(object)
|
||||||
|
lk := z.NewNSLock(bucket, object)
|
||||||
|
lkctx, err := lk.GetLock(ctx, globalOperationTimeout)
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return 0, ctx.Err()
|
||||||
|
}
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
defer lk.Unlock(lkctx)
|
||||||
|
return z.serverPools[source].getHashedSet(object).countAbsentMigrationVersions(lkctx.Context(), bucket, object, versions)
|
||||||
|
}
|
||||||
|
|
||||||
// migrationObjectMetadata restores tags removed from the read representation.
|
// migrationObjectMetadata restores tags removed from the read representation.
|
||||||
// Keep the original revision, including ordered empty states, for the existing
|
// Keep the original revision, including ordered empty states, for the existing
|
||||||
|
|||||||
@@ -590,9 +590,9 @@ func TestSinglePoolConditionalPutHTTP(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// An internal replica callback without an addressed version is not a public
|
// Replica overwrites must establish protection across every pool, including
|
||||||
// condition. Preserve its availability when another pool is unreadable. An
|
// the null version of an unversioned request. An unreadable pool may hold a
|
||||||
// addressed replica already requires all pools for lock/tag reconciliation.
|
// protected copy, so neither addressed nor unaddressed writes may replace it.
|
||||||
func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
|
func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
|
||||||
for _, addressed := range []bool{false, true} {
|
for _, addressed := range []bool{false, true} {
|
||||||
t.Run(fmt.Sprintf("addressed=%t", addressed), func(t *testing.T) {
|
t.Run(fmt.Sprintf("addressed=%t", addressed), func(t *testing.T) {
|
||||||
@@ -615,10 +615,9 @@ func TestPoolsConditionalPutReplicaAvailability(t *testing.T) {
|
|||||||
xhttp.MinIOSourceETag: fmt.Sprintf("%x", md5.Sum([]byte("replacement"))),
|
xhttp.MinIOSourceETag: fmt.Sprintf("%x", md5.Sum([]byte("replacement"))),
|
||||||
}
|
}
|
||||||
url := getPutObjectURL("", bucket, object)
|
url := getPutObjectURL("", bucket, object)
|
||||||
wantStatus, wantBody := http.StatusOK, "replacement"
|
wantStatus, wantBody := http.StatusServiceUnavailable, "old"
|
||||||
if addressed {
|
if addressed {
|
||||||
url += "?versionId=" + oi.VersionID
|
url += "?versionId=" + oi.VersionID
|
||||||
wantStatus, wantBody = http.StatusServiceUnavailable, "old"
|
|
||||||
}
|
}
|
||||||
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
|
put := multipartConditionRequest(t, router, http.MethodPut, url, "replacement", headers)
|
||||||
if put.Code != wantStatus {
|
if put.Code != wantStatus {
|
||||||
|
|||||||
@@ -376,6 +376,9 @@ func (z *erasureServerPools) IsPoolRebalancing(poolIndex int) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (z *erasureServerPools) rebalanceBuckets(ctx context.Context, poolIdx int) (err error) {
|
func (z *erasureServerPools) rebalanceBuckets(ctx context.Context, poolIdx int) (err error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
doneCh := make(chan error, 1)
|
doneCh := make(chan error, 1)
|
||||||
defer xioutil.SafeClose(doneCh)
|
defer xioutil.SafeClose(doneCh)
|
||||||
|
|
||||||
@@ -450,7 +453,7 @@ func (z *erasureServerPools) rebalanceBuckets(ctx context.Context, poolIdx int)
|
|||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
doneCh <- ctx.Err()
|
doneCh <- ctx.Err()
|
||||||
return err
|
return ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -474,6 +477,10 @@ func (z *erasureServerPools) rebalanceBuckets(ctx context.Context, poolIdx int)
|
|||||||
z.bucketRebalanceDone(bucket, poolIdx)
|
z.bucketRebalanceDone(bucket, poolIdx)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
doneCh <- err
|
||||||
|
return err
|
||||||
|
}
|
||||||
rebalanceLogEvent(GlobalContext, "Pool %d rebalancing is done", poolIdx+1)
|
rebalanceLogEvent(GlobalContext, "Pool %d rebalancing is done", poolIdx+1)
|
||||||
|
|
||||||
return err
|
return err
|
||||||
@@ -538,6 +545,9 @@ func (set *erasureObjects) listObjectsToRebalance(ctx context.Context, bucketNam
|
|||||||
|
|
||||||
// rebalanceBucket rebalances objects under bucket in poolIdx pool
|
// rebalanceBucket rebalances objects under bucket in poolIdx pool
|
||||||
func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string, poolIdx int) (err error) {
|
func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string, poolIdx int) (err error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
ctx = logger.SetReqInfo(ctx, &logger.ReqInfo{})
|
ctx = logger.SetReqInfo(ctx, &logger.ReqInfo{})
|
||||||
|
|
||||||
var vc *versioning.Versioning
|
var vc *versioning.Versioning
|
||||||
@@ -643,7 +653,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
|
|||||||
// to create the appropriate stack.
|
// to create the appropriate stack.
|
||||||
versionsSorter(fivs.Versions).reverse()
|
versionsSorter(fivs.Versions).reverse()
|
||||||
|
|
||||||
var rebalanced int
|
var candidates []FileInfo
|
||||||
var expired []migrationExpiredVersion
|
var expired []migrationExpiredVersion
|
||||||
// The key's history in all pools, read once for its versions.
|
// The key's history in all pools, read once for its versions.
|
||||||
var history migrationHistory
|
var history migrationHistory
|
||||||
@@ -658,12 +668,8 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
|
|||||||
|
|
||||||
// Apply lifecycle rules on the objects that are expired.
|
// Apply lifecycle rules on the objects that are expired.
|
||||||
if action := filterLifecycle(bucket, version.Name, version, &history); action.Delete() {
|
if action := filterLifecycle(bucket, version.Name, version, &history); action.Delete() {
|
||||||
// Count the version, or the source copies of the moved
|
|
||||||
// ones stay behind as duplicates. Its own source copy
|
|
||||||
// goes with them when ILM deletes it alone; any other
|
|
||||||
// version stays until the expiry has run.
|
|
||||||
rebalanced++
|
|
||||||
if ilmDeletesVersionOnly(action) || action.DeleteAll() {
|
if ilmDeletesVersionOnly(action) || action.DeleteAll() {
|
||||||
|
candidates = append(candidates, version)
|
||||||
expired = append(expired, migrationExpiredVersion{
|
expired = append(expired, migrationExpiredVersion{
|
||||||
version: version,
|
version: version,
|
||||||
event: lifecycle.Event{Action: action},
|
event: lifecycle.Event{Action: action},
|
||||||
@@ -706,7 +712,6 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
|
|||||||
// This can happen when rebalance stop races with ongoing rebalance workers.
|
// This can happen when rebalance stop races with ongoing rebalance workers.
|
||||||
// These rebalance failures can be ignored.
|
// These rebalance failures can be ignored.
|
||||||
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isDataMovementOverWriteErr(err) {
|
||||||
ignore = true
|
|
||||||
stopFn(0, nil)
|
stopFn(0, nil)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -716,7 +721,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
|
|||||||
failure = err != nil
|
failure = err != nil
|
||||||
if !failure {
|
if !failure {
|
||||||
z.updatePoolStats(poolIdx, bucket, version)
|
z.updatePoolStats(poolIdx, bucket, version)
|
||||||
rebalanced++
|
candidates = append(candidates, version)
|
||||||
}
|
}
|
||||||
auditLogRebalance(ctx, "Rebalance:DeleteMarker", bucket, version.Name, versionID, err)
|
auditLogRebalance(ctx, "Rebalance:DeleteMarker", bucket, version.Name, versionID, err)
|
||||||
continue
|
continue
|
||||||
@@ -773,13 +778,14 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
|
|||||||
break // break out on first error
|
break // break out on first error
|
||||||
}
|
}
|
||||||
z.updatePoolStats(poolIdx, bucket, version)
|
z.updatePoolStats(poolIdx, bucket, version)
|
||||||
rebalanced++
|
candidates = append(candidates, version)
|
||||||
}
|
}
|
||||||
|
|
||||||
// if all versions were rebalanced, we can delete the object versions.
|
// A skipped or failed sibling does not prevent retiring successful
|
||||||
if rebalanced == len(fivs.Versions) {
|
// moves. Cleanup still rechecks each candidate and its destination.
|
||||||
|
if len(candidates) > 0 {
|
||||||
stopFn := globalRebalanceMetrics.log(rebalanceMetricRebalanceRemoveObject, poolIdx, bucket, entry.name)
|
stopFn := globalRebalanceMetrics.log(rebalanceMetricRebalanceRemoveObject, poolIdx, bucket, entry.name)
|
||||||
err := z.removeMovedVersions(ctx, bucket, entry.name, poolIdx, fivs.Versions, expired)
|
_, err := z.removeMovedVersions(ctx, bucket, entry.name, poolIdx, candidates, expired)
|
||||||
stopFn(0, err)
|
stopFn(0, err)
|
||||||
auditLogRebalance(ctx, "Rebalance:DeleteObject", bucket, entry.name, "", err)
|
auditLogRebalance(ctx, "Rebalance:DeleteObject", bucket, entry.name, "", err)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -806,7 +812,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
|
|||||||
}
|
}
|
||||||
|
|
||||||
wk.Wait()
|
wk.Wait()
|
||||||
return nil
|
return ctx.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
type rebalSaveOpts uint8
|
type rebalSaveOpts uint8
|
||||||
|
|||||||
@@ -1195,6 +1195,7 @@ func (z *erasureServerPools) PutObject(ctx context.Context, bucket string, objec
|
|||||||
if opts.replicaObjectInfo == nil && (opts.ReplicaLockReconcile || !opts.ReplicationRequest) {
|
if opts.replicaObjectInfo == nil && (opts.ReplicaLockReconcile || !opts.ReplicationRequest) {
|
||||||
opts.replicaObjectInfo = z.replicaObjectInfo
|
opts.replicaObjectInfo = z.replicaObjectInfo
|
||||||
}
|
}
|
||||||
|
opts.replicaContentGuard = z.guardReplicaContent
|
||||||
oi, err := z.serverPools[idx].PutObject(ctx, bucket, object, data, opts)
|
oi, err := z.serverPools[idx].PutObject(ctx, bucket, object, data, opts)
|
||||||
if err == nil && opts.ReplicaLockReconcile && !opts.DataMovement {
|
if err == nil && opts.ReplicaLockReconcile && !opts.DataMovement {
|
||||||
err = z.retireReplicaCopies(ctx, bucket, object, idx, oi)
|
err = z.retireReplicaCopies(ctx, bucket, object, idx, oi)
|
||||||
@@ -1722,6 +1723,7 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
|
|||||||
stored := mergedPoolObjectInfo(copies)
|
stored := mergedPoolObjectInfo(copies)
|
||||||
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(stored.UserDefined))
|
reconcileStoredObjectLock(srcInfo.UserDefined, storedObjectLockState(stored.UserDefined))
|
||||||
reconcileStoredObjectTags(srcInfo.UserDefined, stored.UserTags, stored.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
|
reconcileStoredObjectTags(srcInfo.UserDefined, stored.UserTags, stored.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
|
||||||
|
dstOpts.replicaObjectInfo = z.replicaObjectInfo
|
||||||
idx := copies[0].Index
|
idx := copies[0].Index
|
||||||
oi, err := z.serverPools[idx].CopyObject(ctx, srcBucket, srcObject, dstBucket, dstObject, srcInfo, srcOpts, dstOpts)
|
oi, err := z.serverPools[idx].CopyObject(ctx, srcBucket, srcObject, dstBucket, dstObject, srcInfo, srcOpts, dstOpts)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -1749,6 +1751,9 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
|
|||||||
return objInfo, err
|
return objInfo, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if !z.SinglePool() {
|
||||||
|
dstOpts.replicaContentGuard = z.guardReplicaContent
|
||||||
|
}
|
||||||
if !z.SinglePool() && (dstOpts.ReplicaLockReconcile || !dstOpts.ReplicationRequest) {
|
if !z.SinglePool() && (dstOpts.ReplicaLockReconcile || !dstOpts.ReplicationRequest) {
|
||||||
dstOpts.replicaObjectInfo = z.replicaObjectInfo
|
dstOpts.replicaObjectInfo = z.replicaObjectInfo
|
||||||
if dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement {
|
if dstOpts.ReplicaLockReconcile && !dstOpts.DataMovement {
|
||||||
@@ -1799,6 +1804,7 @@ func (z *erasureServerPools) CopyObject(ctx context.Context, srcBucket, srcObjec
|
|||||||
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
|
ReplicaLockReconcile: dstOpts.ReplicaLockReconcile,
|
||||||
DataMovement: dstOpts.DataMovement,
|
DataMovement: dstOpts.DataMovement,
|
||||||
replicaObjectInfo: dstOpts.replicaObjectInfo,
|
replicaObjectInfo: dstOpts.replicaObjectInfo,
|
||||||
|
replicaContentGuard: dstOpts.replicaContentGuard,
|
||||||
}
|
}
|
||||||
|
|
||||||
return z.serverPools[poolIdx].PutObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, putOpts)
|
return z.serverPools[poolIdx].PutObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, putOpts)
|
||||||
@@ -2524,6 +2530,7 @@ func (z *erasureServerPools) CompleteMultipartUpload(ctx context.Context, bucket
|
|||||||
opts.replicaObjectInfo = z.replicaObjectInfo
|
opts.replicaObjectInfo = z.replicaObjectInfo
|
||||||
}
|
}
|
||||||
|
|
||||||
|
opts.replicaContentGuard = z.guardReplicaContent
|
||||||
for idx, pool := range z.serverPools {
|
for idx, pool := range z.serverPools {
|
||||||
if z.IsSuspended(idx) {
|
if z.IsSuspended(idx) {
|
||||||
continue
|
continue
|
||||||
@@ -3367,7 +3374,7 @@ func (z *erasureServerPools) PutObjectTags(ctx context.Context, bucket, object s
|
|||||||
// revision beyond all copies, so the returned value and every pool agree.
|
// revision beyond all copies, so the returned value and every pool agree.
|
||||||
if stamp := opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]; stamp != "" {
|
if stamp := opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]; stamp != "" {
|
||||||
for _, copy := range copies {
|
for _, copy := range copies {
|
||||||
stamp = monotonicTaggingTimestamp(stamp, copy.ObjInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
|
stamp = monotonicMetadataTimestamp(stamp, copy.ObjInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp])
|
||||||
}
|
}
|
||||||
opts.UserDefined = cloneMSS(opts.UserDefined)
|
opts.UserDefined = cloneMSS(opts.UserDefined)
|
||||||
opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
|
opts.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = stamp
|
||||||
|
|||||||
@@ -0,0 +1,196 @@
|
|||||||
|
// Copyright (c) 2026 PGSTY
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful,
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/zip"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/madmin-go/v3"
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/policy"
|
||||||
|
)
|
||||||
|
|
||||||
|
func iamImportArchive(t *testing.T, assets map[string]any) []byte {
|
||||||
|
t.Helper()
|
||||||
|
var buf bytes.Buffer
|
||||||
|
zw := zip.NewWriter(&buf)
|
||||||
|
for name, value := range assets {
|
||||||
|
w, err := zw.Create(pathJoin(iamAssetsDir, name))
|
||||||
|
mustIAM(t, err)
|
||||||
|
mustIAM(t, json.NewEncoder(w).Encode(value))
|
||||||
|
}
|
||||||
|
mustIAM(t, zw.Close())
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func iamImportCall(t *testing.T, data []byte) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodPut, "http://localhost/minio/admin/v3/import-iam-v2", int64(len(data)), bytes.NewReader(data), globalActiveCred.AccessKey, globalActiveCred.SecretKey, nil)
|
||||||
|
mustIAM(t, err)
|
||||||
|
mustIAM(t, req.ParseForm())
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
adminAPIHandlers{}.ImportIAMV2(rec, req.WithContext(newContext(req, rec, "ImportIAM")))
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("import-iam-v2: %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIAMImportExportState(t *testing.T) {
|
||||||
|
for _, target := range []string{"fresh", "existing"} {
|
||||||
|
t.Run(target, func(t *testing.T) {
|
||||||
|
ctx, sys := iamAdminFixture(t)
|
||||||
|
parent := iamAdminUser(ctx, t, sys, "import-parent", "")
|
||||||
|
for _, status := range []string{statusDisabled, statusEnabled} {
|
||||||
|
group := "import-" + status
|
||||||
|
_, err := sys.AddUsersToGroup(ctx, group, []string{parent.AccessKey})
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, err = sys.SetGroupStatus(ctx, group, status == statusEnabled)
|
||||||
|
mustIAM(t, err)
|
||||||
|
}
|
||||||
|
_, err := sys.PolicyDBSet(ctx, "import-disabled", "readwrite", regUser, true)
|
||||||
|
mustIAM(t, err)
|
||||||
|
sp, err := policy.ParseConfig(strings.NewReader(svcAccEditPolicy))
|
||||||
|
mustIAM(t, err)
|
||||||
|
expires := UTCNow().Add(time.Hour).Truncate(time.Second)
|
||||||
|
_, _, err = sys.NewServiceAccount(ctx, parent.AccessKey, nil, newServiceAccountOpts{
|
||||||
|
accessKey: "import-service", secretKey: "import-service-secret", status: auth.AccountOff,
|
||||||
|
sessionPolicy: sp, expiration: &expires, name: "disabled service", claims: map[string]any{"identity-attribute": "preserved"},
|
||||||
|
})
|
||||||
|
mustIAM(t, err)
|
||||||
|
exported := iamAdminCall(t, globalActiveCred, http.MethodGet, "/minio/admin/v3/export-iam", nil, nil, adminAPIHandlers{}.ExportIAM)
|
||||||
|
if exported.Code != http.StatusOK {
|
||||||
|
t.Fatalf("export-iam: %d %s", exported.Code, exported.Body.String())
|
||||||
|
}
|
||||||
|
if target == "fresh" {
|
||||||
|
ctx, sys = iamAdminFixture(t)
|
||||||
|
} else {
|
||||||
|
// Import must also replace an existing enabled account/group and
|
||||||
|
// restore an enabled group that is disabled on the target.
|
||||||
|
_, err = sys.UpdateServiceAccount(ctx, "import-service", updateServiceAccountOpts{status: auth.AccountOn})
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, err = sys.SetGroupStatus(ctx, "import-disabled", true)
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, err = sys.SetGroupStatus(ctx, "import-enabled", false)
|
||||||
|
mustIAM(t, err)
|
||||||
|
}
|
||||||
|
rec := iamImportCall(t, exported.Body.Bytes())
|
||||||
|
var result madmin.ImportIAMResult
|
||||||
|
mustIAM(t, json.Unmarshal(rec.Body.Bytes(), &result))
|
||||||
|
if len(result.Failed.Groups) != 0 || len(result.Failed.ServiceAccounts) != 0 || len(result.Added.ServiceAccounts) != 1 {
|
||||||
|
t.Fatalf("import did not restore the exported entities: %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
mustIAM(t, sys.store.LoadIAMCache(ctx, false))
|
||||||
|
for _, status := range []string{statusDisabled, statusEnabled} {
|
||||||
|
group, err := sys.GetGroupDescription("import-" + status)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if group.Status != status || len(group.Members) != 1 || group.Members[0] != parent.AccessKey {
|
||||||
|
t.Fatalf("group status or members changed: %+v", group)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if sys.IsAllowed(policy.Args{AccountName: parent.AccessKey, Action: policy.GetObjectAction, BucketName: "edit", ObjectName: "object"}) {
|
||||||
|
t.Error("the disabled imported group grants access")
|
||||||
|
}
|
||||||
|
cr, gotPolicy, err := sys.GetServiceAccount(ctx, "import-service")
|
||||||
|
mustIAM(t, err)
|
||||||
|
if cr.Status != auth.AccountOff || !cr.Expiration.Equal(expires) || cr.Name != "disabled service" || gotPolicy == nil || !gotPolicy.Equals(*sp) {
|
||||||
|
t.Fatal("import changed the service account status, expiration, name or inline policy")
|
||||||
|
}
|
||||||
|
if _, valid, err := sys.CheckKey(ctx, "import-service"); err != nil || valid {
|
||||||
|
t.Fatalf("imported disabled credential is usable: valid=%v err=%v", valid, err)
|
||||||
|
}
|
||||||
|
var stored UserIdentity
|
||||||
|
mustIAM(t, sys.store.loadIAMConfig(ctx, &stored, getUserIdentityPath("import-service", svcUser)))
|
||||||
|
raw, err := auth.ExtractClaims(stored.Credentials.SessionToken, stored.Credentials.SecretKey)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if _, ok := raw.Map()[sessionPolicyNameExtracted]; ok || raw.Map()["identity-attribute"] != "preserved" {
|
||||||
|
t.Fatal("import polluted the raw JWT or lost identity claims")
|
||||||
|
}
|
||||||
|
// Old exports may contain a derived claim and omit the policy version.
|
||||||
|
snapshot, err := serviceAccountSnapshot(stored)
|
||||||
|
mustIAM(t, err)
|
||||||
|
snapshot.Claims[sessionPolicyNameExtracted] = "stale decoded policy"
|
||||||
|
snapshot.SessionPolicy = []byte(strings.ReplaceAll(svcAccEditPolicy, `"Version":"2012-10-17",`, ""))
|
||||||
|
rec = iamImportCall(t, iamImportArchive(t, map[string]any{allSvcAcctsFile: map[string]madmin.SRSvcAccCreate{"import-service": snapshot}}))
|
||||||
|
mustIAM(t, json.Unmarshal(rec.Body.Bytes(), &result))
|
||||||
|
if len(result.Failed.ServiceAccounts) != 0 {
|
||||||
|
t.Fatalf("legacy snapshot import failed: %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
_, gotPolicy, err = sys.GetServiceAccount(ctx, "import-service")
|
||||||
|
mustIAM(t, err)
|
||||||
|
if gotPolicy == nil || !gotPolicy.Equals(*sp) {
|
||||||
|
t.Fatal("import did not normalize the inline policy")
|
||||||
|
}
|
||||||
|
mustIAM(t, sys.store.loadIAMConfig(ctx, &stored, getUserIdentityPath("import-service", svcUser)))
|
||||||
|
raw, err = auth.ExtractClaims(stored.Credentials.SessionToken, stored.Credentials.SecretKey)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if _, ok := raw.Map()[sessionPolicyNameExtracted]; ok {
|
||||||
|
t.Fatal("legacy import signed a derived claim into the JWT")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type iamImportGroupFailureStore struct {
|
||||||
|
IAMStorageAPI
|
||||||
|
statusOnly bool
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *iamImportGroupFailureStore) saveIAMConfig(ctx context.Context, item any, path string, opts ...options) error {
|
||||||
|
mutation, _ := ctx.Value(iamGroupMutationKey{}).(iamGroupMutation)
|
||||||
|
if path == getGroupInfoPath("import-failure") && mutation.StatusOnly == s.statusOnly {
|
||||||
|
return s.err
|
||||||
|
}
|
||||||
|
return s.IAMStorageAPI.saveIAMConfig(ctx, item, path, opts...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIAMImportGroupFailure(t *testing.T) {
|
||||||
|
for _, operation := range []string{"members", "status"} {
|
||||||
|
t.Run(operation, func(t *testing.T) {
|
||||||
|
_, sys := iamAdminFixture(t)
|
||||||
|
injected := madmin.ErrorResponse{Code: "InjectedGroupWrite", Message: operation + " write failed"}
|
||||||
|
sys.store.IAMStorageAPI = &iamImportGroupFailureStore{IAMStorageAPI: sys.store.IAMStorageAPI, statusOnly: operation == "status", err: injected}
|
||||||
|
rec := iamImportCall(t, iamImportArchive(t, map[string]any{allGroupsFile: map[string]GroupInfo{
|
||||||
|
"import-failure": {Status: statusDisabled},
|
||||||
|
}}))
|
||||||
|
var result struct {
|
||||||
|
Added madmin.IAMEntities `json:"added"`
|
||||||
|
Failed struct {
|
||||||
|
Groups []struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Error json.RawMessage `json:"error"`
|
||||||
|
} `json:"groups"`
|
||||||
|
} `json:"failed"`
|
||||||
|
}
|
||||||
|
mustIAM(t, json.Unmarshal(rec.Body.Bytes(), &result))
|
||||||
|
wantError, err := json.Marshal(injected)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if len(result.Added.Groups) != 0 || len(result.Failed.Groups) != 1 || result.Failed.Groups[0].Name != "import-failure" || !bytes.Equal(result.Failed.Groups[0].Error, wantError) {
|
||||||
|
t.Fatalf("import did not report the actual %s error: %s", operation, rec.Body.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -172,14 +172,41 @@ func (c *SiteReplicationSys) replicationItem(ctx context.Context, item madmin.SR
|
|||||||
key = item.SvcAccChange.Update.AccessKey
|
key = item.SvcAccChange.Update.AccessKey
|
||||||
}
|
}
|
||||||
if key != "" {
|
if key != "" {
|
||||||
r, err := loadIAMRevision(ctx, globalIAMSys.store, getUserIdentityPath(key, svcUser))
|
// A create or an edit goes out as the account stored now: the
|
||||||
|
// whole record, under its revision, whatever the caller's item
|
||||||
|
// says. A peer orders items by revision and drops an older one,
|
||||||
|
// so an edit sent as the fields it changed would lose every
|
||||||
|
// earlier edit that did not reach the peer, or reaches it late.
|
||||||
|
// A later write that won before sending goes out in place of
|
||||||
|
// this one.
|
||||||
|
stored := make(map[string]UserIdentity, 1)
|
||||||
|
err := globalIAMSys.store.loadUser(ctx, key, svcUser, stored)
|
||||||
|
if err != nil && !errors.Is(err, errNoSuchUser) {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
u, ok := stored[key]
|
||||||
|
if !ok {
|
||||||
|
// Nothing is sent for an account that is deleted, expired or
|
||||||
|
// of a revoked parent meanwhile; a deletion replicates itself.
|
||||||
|
// A stored token that cannot be read is an error, and the
|
||||||
|
// account is not taken for sent.
|
||||||
|
var raw UserIdentity
|
||||||
|
err := globalIAMSys.store.loadIAMConfig(ctx, &raw, getUserIdentityPath(key, svcUser))
|
||||||
|
if err == nil && !raw.Deleted && !raw.Credentials.IsExpired() {
|
||||||
|
_, err = extractJWTClaims(raw)
|
||||||
|
}
|
||||||
|
if err != nil && !errors.Is(err, errConfigNotFound) {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
return out, errIAMStaleUpdate
|
||||||
|
}
|
||||||
|
snapshot, err := serviceAccountSnapshot(u)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return out, err
|
return out, err
|
||||||
}
|
}
|
||||||
if r.Deleted || r.timestamp().After(item.UpdatedAt) {
|
out.SvcAccChange = &madmin.SRSvcAccChange{Create: &snapshot}
|
||||||
return out, errIAMStaleUpdate
|
out.UpdatedAt = u.UpdatedAt
|
||||||
}
|
out.RevokedBefore = u.RevokedBefore
|
||||||
out.RevokedBefore = r.RevokedBefore
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if item.Type == madmin.SRIAMItemGroupInfo && item.GroupInfo != nil && !item.GroupInfo.UpdateReq.IsRemove {
|
if item.Type == madmin.SRIAMItemGroupInfo && item.GroupInfo != nil && !item.GroupInfo.UpdateReq.IsRemove {
|
||||||
|
|||||||
+6
-19
@@ -20,7 +20,6 @@ package cmd
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/base64"
|
"encoding/base64"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"maps"
|
"maps"
|
||||||
@@ -2934,27 +2933,15 @@ func updatedServiceAccountCredentials(ctx context.Context, accessKey string, cr
|
|||||||
// the embedded one stays. Only a policy that was given, and is empty,
|
// the embedded one stays. Only a policy that was given, and is empty,
|
||||||
// removes it: the account then has all its parent is allowed, which is
|
// removes it: the account then has all its parent is allowed, which is
|
||||||
// usually more.
|
// usually more.
|
||||||
reset := opts.sessionPolicy != nil && opts.sessionPolicy.Version == "" && len(opts.sessionPolicy.Statements) == 0
|
if opts.sessionPolicy != nil {
|
||||||
if _, ok := m.Lookup(policy.SessionPolicyName); ok && reset {
|
policyBuf, err := marshalServiceAccountPolicy(opts.sessionPolicy)
|
||||||
m.Delete(policy.SessionPolicyName)
|
|
||||||
m.Set(iamPolicyClaimNameSA(), inheritedPolicyType)
|
|
||||||
}
|
|
||||||
|
|
||||||
if opts.sessionPolicy != nil { // session policies is being updated
|
|
||||||
if err := opts.sessionPolicy.Validate(); err != nil {
|
|
||||||
return cr, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if opts.sessionPolicy.Version != "" && len(opts.sessionPolicy.Statements) > 0 {
|
|
||||||
policyBuf, err := json.Marshal(opts.sessionPolicy)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return cr, err
|
return cr, err
|
||||||
}
|
}
|
||||||
|
if len(policyBuf) == 0 {
|
||||||
if len(policyBuf) > maxSVCSessionPolicySize {
|
m.Delete(policy.SessionPolicyName)
|
||||||
return cr, errSessionPolicyTooLarge
|
m.Set(iamPolicyClaimNameSA(), inheritedPolicyType)
|
||||||
}
|
} else {
|
||||||
|
|
||||||
// Overwrite session policy claims.
|
// Overwrite session policy claims.
|
||||||
m.Set(policy.SessionPolicyName, base64.StdEncoding.EncodeToString(policyBuf))
|
m.Set(policy.SessionPolicyName, base64.StdEncoding.EncodeToString(policyBuf))
|
||||||
m.Set(iamPolicyClaimNameSA(), embeddedPolicyType)
|
m.Set(iamPolicyClaimNameSA(), embeddedPolicyType)
|
||||||
|
|||||||
@@ -4,9 +4,12 @@
|
|||||||
package cmd
|
package cmd
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"reflect"
|
||||||
"runtime"
|
"runtime"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -25,6 +28,133 @@ import (
|
|||||||
|
|
||||||
const svcAccEditPolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::edit/*"]}]}`
|
const svcAccEditPolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::edit/*"]}]}`
|
||||||
|
|
||||||
|
func TestIAMServiceAccountPolicySemantics(t *testing.T) {
|
||||||
|
for _, expiring := range []bool{false, true} {
|
||||||
|
t.Run(fmt.Sprintf("expiring=%v", expiring), func(t *testing.T) {
|
||||||
|
f := newGroupRefreshFixture(t)
|
||||||
|
readPolicy, err := policy.ParseConfig(strings.NewReader(svcAccEditPolicy))
|
||||||
|
mustIAM(t, err)
|
||||||
|
writePolicy, err := policy.ParseConfig(strings.NewReader(strings.ReplaceAll(svcAccEditPolicy, "s3:GetObject", "s3:PutObject")))
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, err = f.sys.PolicyDBSet(f.ctx, f.parent.AccessKey, "readwrite", regUser, false)
|
||||||
|
mustIAM(t, err)
|
||||||
|
for _, action := range []policy.Action{policy.GetObjectAction, policy.PutObjectAction} {
|
||||||
|
if !f.sys.IsAllowed(policy.Args{AccountName: f.parent.AccessKey, Action: action, BucketName: "edit", ObjectName: "object"}) {
|
||||||
|
t.Fatal("control: the parent must allow reads and writes to the test object")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
noVersion, invalidVersion, oversized := *writePolicy, *writePolicy, *writePolicy
|
||||||
|
noVersion.Version = ""
|
||||||
|
invalidVersion.Version = "2008-10-17"
|
||||||
|
oversized.ID = policy.ID(strings.Repeat("x", maxSVCSessionPolicySize))
|
||||||
|
var expiration *time.Time
|
||||||
|
if expiring {
|
||||||
|
later := UTCNow().Add(time.Hour)
|
||||||
|
expiration = &later
|
||||||
|
}
|
||||||
|
for i, tc := range []struct {
|
||||||
|
name string
|
||||||
|
input *policy.Policy
|
||||||
|
invalid bool
|
||||||
|
}{
|
||||||
|
{"nil", nil, false},
|
||||||
|
{"empty", &policy.Policy{}, false},
|
||||||
|
{"empty-statements", &policy.Policy{Statements: []policy.Statement{}}, false},
|
||||||
|
{"empty-with-version", &policy.Policy{Version: policy.DefaultVersion, Statements: []policy.Statement{}}, false},
|
||||||
|
{"missing-version", &noVersion, false},
|
||||||
|
{"version", writePolicy, false},
|
||||||
|
{"invalid-version", &invalidVersion, true},
|
||||||
|
{"empty-invalid-version", &policy.Policy{Version: "2008-10-17"}, true},
|
||||||
|
{"invalid-statement", &policy.Policy{Statements: []policy.Statement{{}}}, true},
|
||||||
|
{"oversized", &oversized, true},
|
||||||
|
} {
|
||||||
|
for _, edit := range []bool{false, true} {
|
||||||
|
t.Run(fmt.Sprintf("%s/edit=%v", tc.name, edit), func(t *testing.T) {
|
||||||
|
key := fmt.Sprintf("policy-%d-%v", i, edit)
|
||||||
|
inputBefore, err := json.Marshal(tc.input)
|
||||||
|
mustIAM(t, err)
|
||||||
|
var before UserIdentity
|
||||||
|
if edit {
|
||||||
|
_, _, err = f.sys.NewServiceAccount(f.ctx, f.parent.AccessKey, nil, newServiceAccountOpts{
|
||||||
|
accessKey: key, secretKey: "policy-valid-secret", sessionPolicy: readPolicy, expiration: expiration,
|
||||||
|
})
|
||||||
|
mustIAM(t, err)
|
||||||
|
mustIAM(t, f.sys.store.loadIAMConfig(f.ctx, &before, getUserIdentityPath(key, svcUser)))
|
||||||
|
_, err = f.sys.UpdateServiceAccount(f.ctx, key, updateServiceAccountOpts{sessionPolicy: tc.input, description: "edited"})
|
||||||
|
} else {
|
||||||
|
// Snapshots can carry a stale copy of the policy in claims.
|
||||||
|
var claims map[string]any
|
||||||
|
if tc.input != nil {
|
||||||
|
claims = map[string]any{policy.SessionPolicyName: base64.StdEncoding.EncodeToString([]byte(svcAccEditPolicy))}
|
||||||
|
}
|
||||||
|
_, _, err = f.sys.NewServiceAccount(f.ctx, f.parent.AccessKey, nil, newServiceAccountOpts{
|
||||||
|
accessKey: key, secretKey: "policy-valid-secret", sessionPolicy: tc.input, expiration: expiration,
|
||||||
|
claims: claims,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if (err != nil) != tc.invalid {
|
||||||
|
t.Fatalf("operation error = %v, want rejected=%v", err, tc.invalid)
|
||||||
|
}
|
||||||
|
inputAfter, err := json.Marshal(tc.input)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if !bytes.Equal(inputBefore, inputAfter) {
|
||||||
|
t.Fatal("operation changed the caller's policy")
|
||||||
|
}
|
||||||
|
var stored UserIdentity
|
||||||
|
loadErr := f.sys.store.loadIAMConfig(f.ctx, &stored, getUserIdentityPath(key, svcUser))
|
||||||
|
if tc.invalid {
|
||||||
|
if edit {
|
||||||
|
mustIAM(t, loadErr)
|
||||||
|
if !reflect.DeepEqual(stored, before) {
|
||||||
|
t.Fatal("rejected edit changed the stored account")
|
||||||
|
}
|
||||||
|
} else if loadErr != errConfigNotFound {
|
||||||
|
t.Fatalf("rejected create left an account: %v", loadErr)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
mustIAM(t, loadErr)
|
||||||
|
claims, err := auth.ExtractClaims(stored.Credentials.SessionToken, stored.Credentials.SecretKey)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if _, ok := claims.Map()[sessionPolicyNameExtracted]; ok {
|
||||||
|
t.Fatal("raw JWT contains the derived policy claim")
|
||||||
|
}
|
||||||
|
var want *policy.Policy
|
||||||
|
if tc.input != nil && !tc.input.IsEmpty() {
|
||||||
|
want = writePolicy
|
||||||
|
} else if tc.input == nil && edit {
|
||||||
|
want = readPolicy
|
||||||
|
}
|
||||||
|
got, err := embeddedServiceAccountPolicy(claims)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if (got == nil) != (want == nil) || (got != nil && !got.Equals(*want)) {
|
||||||
|
t.Fatalf("stored policy = %+v, want %+v", got, want)
|
||||||
|
}
|
||||||
|
if want == nil {
|
||||||
|
if _, ok := claims.Lookup(policy.SessionPolicyName); ok || claims.Map()[iamPolicyClaimNameSA()] != inheritedPolicyType {
|
||||||
|
t.Fatal("inheritance must remove the embedded policy claim")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mustIAM(t, f.sys.store.LoadIAMCache(f.ctx, false))
|
||||||
|
u, valid, err := f.sys.CheckKey(f.ctx, key)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if !valid || u.Credentials.IsTemp() != expiring {
|
||||||
|
t.Fatal("credential validity or expiration changed")
|
||||||
|
}
|
||||||
|
for _, action := range []policy.Action{policy.GetObjectAction, policy.PutObjectAction} {
|
||||||
|
allowed := f.sys.IsAllowed(policy.Args{AccountName: key, Claims: u.Credentials.Claims, Action: action, BucketName: "edit", ObjectName: "object"})
|
||||||
|
wantAllowed := want == nil || (want == readPolicy && action == policy.GetObjectAction) || (want == writePolicy && action == policy.PutObjectAction)
|
||||||
|
if allowed != wantAllowed {
|
||||||
|
t.Errorf("%s allowed=%v, want %v", action, allowed, wantAllowed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestIAMServiceAccountEditKeepsPolicy(t *testing.T) {
|
func TestIAMServiceAccountEditKeepsPolicy(t *testing.T) {
|
||||||
f := newGroupRefreshFixture(t)
|
f := newGroupRefreshFixture(t)
|
||||||
want, err := policy.ParseConfig(strings.NewReader(svcAccEditPolicy))
|
want, err := policy.ParseConfig(strings.NewReader(svcAccEditPolicy))
|
||||||
@@ -110,6 +240,7 @@ func TestIAMServiceAccountEditKeepsPolicy(t *testing.T) {
|
|||||||
{"peer-status", madmin.SRSvcAccUpdate{Status: auth.AccountOff}, true, func(c auth.Credentials) bool { return c.Status == auth.AccountOff }},
|
{"peer-status", madmin.SRSvcAccUpdate{Status: auth.AccountOff}, true, func(c auth.Credentials) bool { return c.Status == auth.AccountOff }},
|
||||||
{"peer-null", madmin.SRSvcAccUpdate{Description: "described", SessionPolicy: madmin.SRSessionPolicy("null")}, true, func(c auth.Credentials) bool { return c.Description == "described" }},
|
{"peer-null", madmin.SRSvcAccUpdate{Description: "described", SessionPolicy: madmin.SRSessionPolicy("null")}, true, func(c auth.Credentials) bool { return c.Description == "described" }},
|
||||||
{"peer-empty", madmin.SRSvcAccUpdate{SessionPolicy: madmin.SRSessionPolicy(`{"Statement":[]}`)}, false, nil},
|
{"peer-empty", madmin.SRSvcAccUpdate{SessionPolicy: madmin.SRSessionPolicy(`{"Statement":[]}`)}, false, nil},
|
||||||
|
{"peer-empty-v", madmin.SRSvcAccUpdate{SessionPolicy: madmin.SRSessionPolicy(`{"Version":"2012-10-17","Statement":[]}`)}, false, nil},
|
||||||
} {
|
} {
|
||||||
t.Run(edit.name, func(t *testing.T) {
|
t.Run(edit.name, func(t *testing.T) {
|
||||||
key := "edit-" + edit.name
|
key := "edit-" + edit.name
|
||||||
|
|||||||
@@ -0,0 +1,530 @@
|
|||||||
|
// Copyright (c) 2026 PGSTY
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/zip"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"path"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/madmin-go/v3"
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
"github.com/pgsty/silo-pkg/v3/policy"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Site replication sends a service account as the record its site has
|
||||||
|
// stored, whole. A peer that applies the last item it was sent then has
|
||||||
|
// the account of the sending site, whatever it received or lost before,
|
||||||
|
// and whatever the sending server holds in its cache.
|
||||||
|
|
||||||
|
const svcAccSnapshotPolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::snapshot/*"]}]}`
|
||||||
|
|
||||||
|
// svcAccRecord is what a site has of a service account.
|
||||||
|
type svcAccRecord struct {
|
||||||
|
Parent, Secret, Status, Name, Description, Policy string
|
||||||
|
Expiration, UpdatedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// svcAccPolicyText is a policy document in the one form it is compared in;
|
||||||
|
// an account without an embedded policy has "".
|
||||||
|
func svcAccPolicyText(t *testing.T, document []byte) string {
|
||||||
|
t.Helper()
|
||||||
|
if len(document) == 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
p, err := policy.ParseConfig(bytes.NewReader(document))
|
||||||
|
mustIAM(t, err)
|
||||||
|
text, err := json.Marshal(p)
|
||||||
|
mustIAM(t, err)
|
||||||
|
return string(text)
|
||||||
|
}
|
||||||
|
|
||||||
|
// svcAccStored is the account as a server that starts now loads it.
|
||||||
|
func svcAccStored(t *testing.T, f *groupRefreshFixture, key string) svcAccRecord {
|
||||||
|
t.Helper()
|
||||||
|
u, ok := f.stored(t, key)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("service account %s is not stored", key)
|
||||||
|
}
|
||||||
|
cr := u.Credentials
|
||||||
|
rawClaims, err := auth.ExtractClaims(cr.SessionToken, cr.SecretKey)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if _, ok := rawClaims.Map()[sessionPolicyNameExtracted]; ok {
|
||||||
|
t.Fatal("stored JWT contains the derived policy claim")
|
||||||
|
}
|
||||||
|
claims, err := getClaimsFromTokenWithSecret(cr.SessionToken, cr.SecretKey)
|
||||||
|
mustIAM(t, err)
|
||||||
|
rec := svcAccRecord{Parent: cr.ParentUser, Secret: cr.SecretKey, Status: cr.Status, Name: cr.Name, Description: cr.Description, Expiration: cr.Expiration, UpdatedAt: u.UpdatedAt}
|
||||||
|
if sp, ok := claims.Lookup(policy.SessionPolicyName); ok && claims.MapClaims[iamPolicyClaimNameSA()] == embeddedPolicyType {
|
||||||
|
document, err := base64.StdEncoding.DecodeString(sp)
|
||||||
|
mustIAM(t, err)
|
||||||
|
rec.Policy = svcAccPolicyText(t, document)
|
||||||
|
}
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// svcAccSite is a site that replicates to one peer, which records the IAM
|
||||||
|
// items it is sent.
|
||||||
|
type svcAccSite struct {
|
||||||
|
*groupRefreshFixture
|
||||||
|
mu sync.Mutex
|
||||||
|
sent []iamReplicationItem
|
||||||
|
}
|
||||||
|
|
||||||
|
func newSvcAccSite(t *testing.T) *svcAccSite {
|
||||||
|
t.Helper()
|
||||||
|
s := &svcAccSite{groupRefreshFixture: newGroupRefreshFixture(t)}
|
||||||
|
globalObjLayerMutex.Lock()
|
||||||
|
globalObjectAPI = s.obj
|
||||||
|
globalObjLayerMutex.Unlock()
|
||||||
|
|
||||||
|
peer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method == http.MethodPut && strings.HasSuffix(r.URL.Path, iamRevisionPeerPath) {
|
||||||
|
var batch iamRevisionBatch
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&batch); err != nil {
|
||||||
|
t.Error(err)
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
s.sent = append(s.sent, batch.Items...)
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(iamRevisionResponse{iamRevisionStatus: iamRevisionStatus{Version: iamRevisionProtocol, Node: "node", Instance: "boot", Digest: "fixture"}})
|
||||||
|
}))
|
||||||
|
globalSiteReplicatorCred.Set("snapshot-site-signing-key")
|
||||||
|
globalSiteReplicationSys.Lock()
|
||||||
|
globalSiteReplicationSys.enabled = true
|
||||||
|
globalSiteReplicationSys.state = srState{ServiceAccountAccessKey: s.parent.AccessKey, Peers: s.peers(peer.URL)}
|
||||||
|
globalSiteReplicationSys.Unlock()
|
||||||
|
t.Cleanup(func() {
|
||||||
|
s.leave()
|
||||||
|
peer.Close()
|
||||||
|
})
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *svcAccSite) peers(endpoint string) map[string]madmin.PeerInfo {
|
||||||
|
return map[string]madmin.PeerInfo{
|
||||||
|
globalDeploymentID(): {Name: "local", DeploymentID: globalDeploymentID()},
|
||||||
|
"remote": {Name: "remote", DeploymentID: "remote", Endpoint: endpoint},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// leave ends the replication of this process, for the site that comes next.
|
||||||
|
func (s *svcAccSite) leave() {
|
||||||
|
globalSiteReplicationSys.Lock()
|
||||||
|
globalSiteReplicationSys.enabled = false
|
||||||
|
globalSiteReplicationSys.state = srState{}
|
||||||
|
globalSiteReplicationSys.Unlock()
|
||||||
|
globalSiteReplicatorCred.Set("")
|
||||||
|
}
|
||||||
|
|
||||||
|
// items are the creates and edits of account key the peer was sent, in the
|
||||||
|
// order of sending.
|
||||||
|
func (s *svcAccSite) items(key string) []iamReplicationItem {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
var items []iamReplicationItem
|
||||||
|
for _, item := range s.sent {
|
||||||
|
c := item.SvcAccChange
|
||||||
|
if c != nil && ((c.Create != nil && c.Create.AccessKey == key) || (c.Update != nil && c.Update.AccessKey == key)) {
|
||||||
|
items = append(items, item)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return items
|
||||||
|
}
|
||||||
|
|
||||||
|
// create and edit are an administrator's requests to this site.
|
||||||
|
func (s *svcAccSite) create(t *testing.T, key string) {
|
||||||
|
t.Helper()
|
||||||
|
rec := iamAdminCall(t, globalActiveCred, http.MethodPut, "/minio/admin/v3/add-service-account",
|
||||||
|
madmin.AddServiceAccountReq{TargetUser: s.parent.AccessKey, AccessKey: key, SecretKey: "snapshot-original-secret"}, nil, adminAPIHandlers{}.AddServiceAccount)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("add-service-account: %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *svcAccSite) edit(t *testing.T, key string, req madmin.UpdateServiceAccountReq) {
|
||||||
|
t.Helper()
|
||||||
|
rec := iamAdminCall(t, globalActiveCred, http.MethodPost, "/minio/admin/v3/update-service-account?accessKey="+key,
|
||||||
|
req, map[string]string{"accessKey": key}, adminAPIHandlers{}.UpdateServiceAccount)
|
||||||
|
if rec.Code != http.StatusNoContent {
|
||||||
|
t.Fatalf("update-service-account: %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// svcAccPeer is the site that applies items, in the order given, and what
|
||||||
|
// it then has of account key. It has the parent user, as a peer does.
|
||||||
|
func svcAccPeer(t *testing.T, from *svcAccSite, key string, items ...iamReplicationItem) svcAccRecord {
|
||||||
|
t.Helper()
|
||||||
|
from.leave()
|
||||||
|
peer := newGroupRefreshFixture(t)
|
||||||
|
for i, item := range items {
|
||||||
|
// As the peer reads it off the wire.
|
||||||
|
raw, err := json.Marshal(item)
|
||||||
|
mustIAM(t, err)
|
||||||
|
var received iamReplicationItem
|
||||||
|
mustIAM(t, json.Unmarshal(raw, &received))
|
||||||
|
if create := received.SvcAccChange.Create; create != nil {
|
||||||
|
if _, ok := create.Claims[sessionPolicyNameExtracted]; ok {
|
||||||
|
t.Fatal("replication snapshot contains the derived policy claim")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := applyIAMReplicationItem(peer.ctx, received); err != nil {
|
||||||
|
t.Fatalf("the peer refused item %d: %v", i, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return svcAccStored(t, peer, key)
|
||||||
|
}
|
||||||
|
|
||||||
|
func wantSvcAccRecord(t *testing.T, where string, got, want svcAccRecord) {
|
||||||
|
t.Helper()
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("%s:\n got %+v\nwant %+v", where, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An edit of a service account reaches a peer whatever became of the edits
|
||||||
|
// before it: one that arrives late, or never, is not lost.
|
||||||
|
func TestSiteReplicationServiceAccountEdits(t *testing.T) {
|
||||||
|
const key = "snapshot-edits"
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
edits []madmin.UpdateServiceAccountReq
|
||||||
|
took func(svcAccRecord) bool // control: what the site has after them
|
||||||
|
}{
|
||||||
|
{"policy-secret-description", []madmin.UpdateServiceAccountReq{
|
||||||
|
{NewPolicy: json.RawMessage(svcAccSnapshotPolicy)},
|
||||||
|
{NewSecretKey: "snapshot-rotated-secret"},
|
||||||
|
{NewDescription: "described"},
|
||||||
|
}, func(r svcAccRecord) bool {
|
||||||
|
return r.Policy != "" && r.Secret == "snapshot-rotated-secret" && r.Description == "described"
|
||||||
|
}},
|
||||||
|
{"secret-disable", []madmin.UpdateServiceAccountReq{
|
||||||
|
{NewPolicy: json.RawMessage(svcAccSnapshotPolicy), NewSecretKey: "snapshot-rotated-secret"},
|
||||||
|
{NewStatus: string(madmin.AccountDisabled)},
|
||||||
|
}, func(r svcAccRecord) bool {
|
||||||
|
return r.Policy != "" && r.Secret == "snapshot-rotated-secret" && r.Status == auth.AccountOff
|
||||||
|
}},
|
||||||
|
// The last edit removes the embedded policy: the peer gets an
|
||||||
|
// account without one, and the description set before.
|
||||||
|
{"policy-removed", []madmin.UpdateServiceAccountReq{
|
||||||
|
{NewPolicy: json.RawMessage(svcAccSnapshotPolicy)},
|
||||||
|
{NewDescription: "described"},
|
||||||
|
{NewPolicy: json.RawMessage(`{"Statement":[]}`)},
|
||||||
|
}, func(r svcAccRecord) bool { return r.Policy == "" && r.Description == "described" }},
|
||||||
|
{"policy-removed-with-version", []madmin.UpdateServiceAccountReq{
|
||||||
|
{NewPolicy: json.RawMessage(svcAccSnapshotPolicy)},
|
||||||
|
{NewPolicy: json.RawMessage(`{"Version":"2012-10-17","Statement":[]}`), NewName: "named"},
|
||||||
|
{NewDescription: "described"},
|
||||||
|
}, func(r svcAccRecord) bool {
|
||||||
|
return r.Policy == "" && r.Name == "named" && r.Description == "described"
|
||||||
|
}},
|
||||||
|
{"policy-missing-version", []madmin.UpdateServiceAccountReq{
|
||||||
|
{NewPolicy: json.RawMessage(svcAccSnapshotPolicy)},
|
||||||
|
{NewPolicy: json.RawMessage(strings.ReplaceAll(strings.ReplaceAll(svcAccSnapshotPolicy, `"Version":"2012-10-17",`, ""), "s3:GetObject", "s3:PutObject"))},
|
||||||
|
{NewDescription: "described"},
|
||||||
|
}, func(r svcAccRecord) bool {
|
||||||
|
return r.Policy == svcAccPolicyText(t, []byte(strings.ReplaceAll(svcAccSnapshotPolicy, "s3:GetObject", "s3:PutObject"))) && r.Description == "described"
|
||||||
|
}},
|
||||||
|
} {
|
||||||
|
for _, delivery := range []string{"late", "lost"} {
|
||||||
|
t.Run(tc.name+"/"+delivery, func(t *testing.T) {
|
||||||
|
site := newSvcAccSite(t)
|
||||||
|
site.create(t, key)
|
||||||
|
for _, edit := range tc.edits {
|
||||||
|
site.edit(t, key, edit)
|
||||||
|
}
|
||||||
|
sent := site.items(key)
|
||||||
|
if len(sent) != 1+len(tc.edits) {
|
||||||
|
t.Fatalf("the site sent %d items for the account, want one for the create and each edit", len(sent))
|
||||||
|
}
|
||||||
|
want := svcAccStored(t, site.groupRefreshFixture, key)
|
||||||
|
if !tc.took(want) {
|
||||||
|
t.Fatalf("control: the site did not apply the edits: %+v", want)
|
||||||
|
}
|
||||||
|
// The peer has the account and gets the last edit. The edits
|
||||||
|
// before it were lost, or reach it after the last.
|
||||||
|
last := len(sent) - 1
|
||||||
|
deliver := []iamReplicationItem{sent[0], sent[last]}
|
||||||
|
if delivery == "late" {
|
||||||
|
deliver = append(deliver, sent[1:last]...)
|
||||||
|
}
|
||||||
|
wantSvcAccRecord(t, "peer", svcAccPeer(t, site, key, deliver...), want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Two requests edit the account through this site at once, and the
|
||||||
|
// first comes to replicate when the second is stored.
|
||||||
|
t.Run("concurrent", func(t *testing.T) {
|
||||||
|
site := newSvcAccSite(t)
|
||||||
|
site.create(t, key)
|
||||||
|
sp, err := policy.ParseConfig(strings.NewReader(svcAccSnapshotPolicy))
|
||||||
|
mustIAM(t, err)
|
||||||
|
first, err := site.sys.UpdateServiceAccount(site.ctx, key, updateServiceAccountOpts{sessionPolicy: sp})
|
||||||
|
mustIAM(t, err)
|
||||||
|
second, err := site.sys.UpdateServiceAccount(site.ctx, key, updateServiceAccountOpts{description: "described"})
|
||||||
|
mustIAM(t, err)
|
||||||
|
for _, at := range []time.Time{first, second} {
|
||||||
|
mustIAM(t, globalSiteReplicationSys.IAMChangeHook(site.ctx, madmin.SRIAMItem{
|
||||||
|
Type: madmin.SRIAMItemSvcAcc,
|
||||||
|
SvcAccChange: &madmin.SRSvcAccChange{Update: &madmin.SRSvcAccUpdate{AccessKey: key}},
|
||||||
|
UpdatedAt: at,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
want := svcAccStored(t, site.groupRefreshFixture, key)
|
||||||
|
if want.Policy == "" || want.Description != "described" {
|
||||||
|
t.Fatalf("control: the site did not apply the edits: %+v", want)
|
||||||
|
}
|
||||||
|
wantSvcAccRecord(t, "peer", svcAccPeer(t, site, key, site.items(key)...), want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Another server of the site restricted, rotated and disabled the account,
|
||||||
|
// and this server's cache has not heard of it. What this server sends, and
|
||||||
|
// exports, is the account that is stored.
|
||||||
|
func TestSiteReplicationServiceAccountStaleCache(t *testing.T) {
|
||||||
|
const key = "snapshot-stale"
|
||||||
|
// prepare returns the site, the stored account and what it sent for the
|
||||||
|
// create.
|
||||||
|
prepare := func(t *testing.T) (*svcAccSite, svcAccRecord, iamReplicationItem) {
|
||||||
|
t.Helper()
|
||||||
|
site := newSvcAccSite(t)
|
||||||
|
site.create(t, key)
|
||||||
|
created := site.items(key)
|
||||||
|
if len(created) != 1 {
|
||||||
|
t.Fatalf("the site sent %d items for the create", len(created))
|
||||||
|
}
|
||||||
|
sp, err := policy.ParseConfig(strings.NewReader(svcAccSnapshotPolicy))
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, err = site.sibling(t).UpdateServiceAccount(site.ctx, key, updateServiceAccountOpts{
|
||||||
|
sessionPolicy: sp, secretKey: "snapshot-rotated-secret", status: auth.AccountOff, description: "restricted",
|
||||||
|
})
|
||||||
|
mustIAM(t, err)
|
||||||
|
if cached, _ := site.sys.store.GetUser(key); cached.Credentials.SecretKey != "snapshot-original-secret" || cached.Credentials.Status != auth.AccountOn {
|
||||||
|
t.Fatal("control: this server's cache must not have heard of the edit")
|
||||||
|
}
|
||||||
|
want := svcAccStored(t, site.groupRefreshFixture, key)
|
||||||
|
if want.Policy == "" || want.Secret != "snapshot-rotated-secret" || want.Status != auth.AccountOff {
|
||||||
|
t.Fatalf("control: the edit is not stored: %+v", want)
|
||||||
|
}
|
||||||
|
return site, want, created[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("heal", func(t *testing.T) {
|
||||||
|
site, want, created := prepare(t)
|
||||||
|
local := madmin.UserInfo{Status: madmin.AccountStatus(want.Status), UpdatedAt: want.UpdatedAt}
|
||||||
|
remote := madmin.UserInfo{Status: madmin.AccountStatus(auth.AccountOn), UpdatedAt: created.UpdatedAt}
|
||||||
|
info := srStatusInfo{Sites: globalSiteReplicationSys.state.Peers, UserStats: map[string]map[string]srUserStatsSummary{key: {
|
||||||
|
globalDeploymentID(): {userInfo: srUserInfo{UserInfo: local}},
|
||||||
|
"remote": {SRUserStatsSummary: madmin.SRUserStatsSummary{UserInfoMismatch: true}, userInfo: srUserInfo{UserInfo: remote}},
|
||||||
|
}}}
|
||||||
|
mustIAM(t, globalSiteReplicationSys.healUsers(site.ctx, site.obj, key, info))
|
||||||
|
sent := site.items(key)
|
||||||
|
if len(sent) != 2 {
|
||||||
|
t.Fatalf("heal sent %d items for the account, want 1", len(sent)-1)
|
||||||
|
}
|
||||||
|
wantSvcAccRecord(t, "healed peer", svcAccPeer(t, site, key, sent...), want)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("sync", func(t *testing.T) {
|
||||||
|
site, want, _ := prepare(t)
|
||||||
|
mustIAM(t, globalSiteReplicationSys.syncToAllPeers(site.ctx, madmin.SRAddOptions{}))
|
||||||
|
sent := site.items(key)
|
||||||
|
if len(sent) != 2 {
|
||||||
|
t.Fatalf("sync sent %d items for the account, want 1", len(sent)-1)
|
||||||
|
}
|
||||||
|
wantSvcAccRecord(t, "new peer", svcAccPeer(t, site, key, sent[1]), want)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("export", func(t *testing.T) {
|
||||||
|
_, want, _ := prepare(t)
|
||||||
|
rec := iamAdminCall(t, globalActiveCred, http.MethodGet, "/minio/admin/v3/export-iam", nil, nil, adminAPIHandlers{}.ExportIAM)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("export-iam: %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
archive, err := zip.NewReader(bytes.NewReader(rec.Body.Bytes()), int64(rec.Body.Len()))
|
||||||
|
mustIAM(t, err)
|
||||||
|
file, err := archive.Open(path.Join(iamAssetsDir, allSvcAcctsFile))
|
||||||
|
mustIAM(t, err)
|
||||||
|
raw, err := io.ReadAll(file)
|
||||||
|
mustIAM(t, err)
|
||||||
|
var accounts map[string]madmin.SRSvcAccCreate
|
||||||
|
mustIAM(t, json.Unmarshal(raw, &accounts))
|
||||||
|
a, ok := accounts[key]
|
||||||
|
if !ok || a.Expiration == nil {
|
||||||
|
t.Fatalf("the export has no account %s with an expiration (present=%v)", key, ok)
|
||||||
|
}
|
||||||
|
got := svcAccRecord{Parent: a.Parent, Secret: a.SecretKey, Status: a.Status, Name: a.Name, Description: a.Description, Policy: svcAccPolicyText(t, a.SessionPolicy), Expiration: *a.Expiration}
|
||||||
|
want.UpdatedAt = time.Time{} // an export has no revisions
|
||||||
|
wantSvcAccRecord(t, "export", got, want)
|
||||||
|
if a.Claims[iamPolicyClaimNameSA()] != embeddedPolicyType {
|
||||||
|
t.Errorf("exported claims say %v, want %s", a.Claims[iamPolicyClaimNameSA()], embeddedPolicyType)
|
||||||
|
}
|
||||||
|
if _, ok := a.Claims[sessionPolicyNameExtracted]; ok {
|
||||||
|
t.Error("exported claims contain the derived policy claim")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIAMServiceAccountSnapshotClaims(t *testing.T) {
|
||||||
|
f := newGroupRefreshFixture(t)
|
||||||
|
sp, err := policy.ParseConfig(strings.NewReader(svcAccSnapshotPolicy))
|
||||||
|
mustIAM(t, err)
|
||||||
|
_, _, err = f.sys.NewServiceAccount(f.ctx, f.parent.AccessKey, nil, newServiceAccountOpts{
|
||||||
|
accessKey: "snapshot-claims", secretKey: "snapshot-valid-secret", sessionPolicy: sp,
|
||||||
|
claims: map[string]any{"identity-attribute": "preserved"},
|
||||||
|
})
|
||||||
|
mustIAM(t, err)
|
||||||
|
u, ok := f.stored(t, "snapshot-claims")
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("service account missing")
|
||||||
|
}
|
||||||
|
cr := u.Credentials
|
||||||
|
rawClaims, err := auth.ExtractClaims(cr.SessionToken, cr.SecretKey)
|
||||||
|
mustIAM(t, err)
|
||||||
|
derived, ok := cr.Claims[sessionPolicyNameExtracted]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("control: loaded claims must contain the derived policy")
|
||||||
|
}
|
||||||
|
snapshot, err := serviceAccountSnapshot(u)
|
||||||
|
mustIAM(t, err)
|
||||||
|
if !reflect.DeepEqual(snapshot.Claims, rawClaims.Map()) {
|
||||||
|
t.Fatal("snapshot claims differ from the raw JWT")
|
||||||
|
}
|
||||||
|
if u.Credentials.SessionToken != cr.SessionToken || u.Credentials.Claims[sessionPolicyNameExtracted] != derived {
|
||||||
|
t.Fatal("snapshot changed the source token or its cached claims")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The parent was deleted and created again on both sites. The token of a
|
||||||
|
// service account created afterwards names that revocation, and a peer
|
||||||
|
// refuses a credential of the parent that does not: the create must carry
|
||||||
|
// the claims of the token.
|
||||||
|
func TestSiteReplicationServiceAccountOfRecreatedParent(t *testing.T) {
|
||||||
|
const key = "snapshot-reparent"
|
||||||
|
site := newSvcAccSite(t)
|
||||||
|
mustIAM(t, site.sys.DeleteUser(site.ctx, site.parent.AccessKey, false))
|
||||||
|
iamAdminUser(site.ctx, t, site.sys, site.parent.AccessKey, "refresh-parent")
|
||||||
|
revoked, err := loadIAMRevision(site.ctx, site.sys.store, getUserIdentityPath(site.parent.AccessKey, regUser))
|
||||||
|
mustIAM(t, err)
|
||||||
|
if revoked.RevokedBefore.IsZero() {
|
||||||
|
t.Fatal("control: the parent must have a revocation boundary")
|
||||||
|
}
|
||||||
|
site.create(t, key)
|
||||||
|
sent := site.items(key)
|
||||||
|
if len(sent) != 1 {
|
||||||
|
t.Fatalf("the site sent %d items for the create", len(sent))
|
||||||
|
}
|
||||||
|
want := svcAccStored(t, site.groupRefreshFixture, key)
|
||||||
|
|
||||||
|
// The peer's parent is newer than the deletion, which leaves it live
|
||||||
|
// with the boundary: a parent that was deleted and created again.
|
||||||
|
site.leave()
|
||||||
|
peer := newGroupRefreshFixture(t)
|
||||||
|
mustIAM(t, peer.sys.DeleteUser(withIAMReplicationTime(peer.ctx, revoked.RevokedBefore), peer.parent.AccessKey, false))
|
||||||
|
if r, err := loadIAMRevision(peer.ctx, peer.sys.store, getUserIdentityPath(peer.parent.AccessKey, regUser)); err != nil || r.Deleted || !r.RevokedBefore.Equal(revoked.RevokedBefore) {
|
||||||
|
t.Fatalf("control: the peer's parent must be live with the boundary (err=%v deleted=%v)", err, r.Deleted)
|
||||||
|
}
|
||||||
|
mustIAM(t, applyIAMReplicationItem(peer.ctx, sent[0]))
|
||||||
|
if _, ok := peer.stored(t, key); !ok {
|
||||||
|
t.Fatal("the peer dropped the new service account")
|
||||||
|
}
|
||||||
|
wantSvcAccRecord(t, "peer", svcAccStored(t, peer, key), want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// An account that is deleted is not sent, and that is not an error. An
|
||||||
|
// account whose stored token cannot be read is not taken for deleted: the
|
||||||
|
// site signing key is not available here, and no token is read without it.
|
||||||
|
func TestSiteReplicationServiceAccountUnreadable(t *testing.T) {
|
||||||
|
const key = "snapshot-unreadable"
|
||||||
|
site := newSvcAccSite(t)
|
||||||
|
site.create(t, key)
|
||||||
|
item := madmin.SRIAMItem{Type: madmin.SRIAMItemSvcAcc, SvcAccChange: &madmin.SRSvcAccChange{Update: &madmin.SRSvcAccUpdate{AccessKey: key}}}
|
||||||
|
|
||||||
|
globalSiteReplicatorCred.Set("")
|
||||||
|
if _, err := globalSiteReplicationSys.replicationItem(site.ctx, item); err == nil || errors.Is(err, errIAMStaleUpdate) {
|
||||||
|
t.Errorf("an account whose token cannot be read must be an error, got %v", err)
|
||||||
|
}
|
||||||
|
if err := globalSiteReplicationSys.IAMChangeHook(site.ctx, item); err == nil {
|
||||||
|
t.Error("the hook reported an account as sent that it could not read")
|
||||||
|
}
|
||||||
|
globalSiteReplicatorCred.Set("snapshot-site-signing-key")
|
||||||
|
|
||||||
|
mustIAM(t, site.sys.DeleteServiceAccount(site.ctx, key, false))
|
||||||
|
if _, err := globalSiteReplicationSys.replicationItem(site.ctx, item); !errors.Is(err, errIAMStaleUpdate) {
|
||||||
|
t.Errorf("a deleted account must not be sent, got %v", err)
|
||||||
|
}
|
||||||
|
if sent := site.items(key); len(sent) != 1 {
|
||||||
|
t.Errorf("the site sent %d items for the account, want only the create", len(sent))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// svcAccPausedStore holds every load of a user until proceed is closed.
|
||||||
|
type svcAccPausedStore struct {
|
||||||
|
IAMStorageAPI
|
||||||
|
once sync.Once
|
||||||
|
loading, proceed chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *svcAccPausedStore) loadUser(ctx context.Context, user string, userType IAMUserType, m map[string]UserIdentity) error {
|
||||||
|
s.once.Do(func() { close(s.loading) })
|
||||||
|
<-s.proceed
|
||||||
|
return s.IAMStorageAPI.loadUser(ctx, user, userType, m)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The hook reads the account from storage without the lock of the
|
||||||
|
// replication configuration. Loading an identity takes that lock for
|
||||||
|
// reading; if the hook held it too, a change of the configuration that
|
||||||
|
// arrives in between would wait for the hook, and the hook for it.
|
||||||
|
func TestSiteReplicationHookReadsStorageUnlocked(t *testing.T) {
|
||||||
|
const key = "snapshot-unlocked"
|
||||||
|
site := newSvcAccSite(t)
|
||||||
|
site.create(t, key)
|
||||||
|
paused := &svcAccPausedStore{IAMStorageAPI: site.sys.store.IAMStorageAPI, loading: make(chan struct{}), proceed: make(chan struct{})}
|
||||||
|
site.sys.store.IAMStorageAPI = paused
|
||||||
|
|
||||||
|
sent := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
sent <- globalSiteReplicationSys.IAMChangeHook(site.ctx, madmin.SRIAMItem{
|
||||||
|
Type: madmin.SRIAMItemSvcAcc,
|
||||||
|
SvcAccChange: &madmin.SRSvcAccChange{Update: &madmin.SRSvcAccUpdate{AccessKey: key}},
|
||||||
|
})
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-paused.loading:
|
||||||
|
case err := <-sent:
|
||||||
|
t.Fatalf("the hook did not read the account from storage (err=%v)", err)
|
||||||
|
}
|
||||||
|
// A writer that had to wait here would never get the lock, nor the
|
||||||
|
// hook its next read lock behind that writer: try, do not wait.
|
||||||
|
if globalSiteReplicationSys.TryLock() {
|
||||||
|
globalSiteReplicationSys.Unlock() //nolint:staticcheck // only the lock is of interest
|
||||||
|
} else {
|
||||||
|
t.Error("the hook holds the lock of the replication configuration while it reads storage")
|
||||||
|
}
|
||||||
|
close(paused.proceed)
|
||||||
|
select {
|
||||||
|
case err := <-sent:
|
||||||
|
mustIAM(t, err)
|
||||||
|
case <-time.After(10 * time.Second):
|
||||||
|
t.Fatal("the hook did not return")
|
||||||
|
}
|
||||||
|
if got := site.items(key); len(got) != 2 {
|
||||||
|
t.Errorf("the site sent %d items for the account, want the create and the edit", len(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
+86
-39
@@ -1098,6 +1098,32 @@ type newServiceAccountOpts struct {
|
|||||||
claims map[string]any
|
claims map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// marshalServiceAccountPolicy validates a supplied policy and normalizes its
|
||||||
|
// version without changing the caller's policy. An empty policy means inheritance.
|
||||||
|
func marshalServiceAccountPolicy(sp *policy.Policy) ([]byte, error) {
|
||||||
|
if sp == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
p := *sp
|
||||||
|
if p.Version == "" {
|
||||||
|
p.Version = policy.DefaultVersion
|
||||||
|
}
|
||||||
|
if err := p.Validate(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if p.IsEmpty() {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
buf, err := json.Marshal(p)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(buf) > maxSVCSessionPolicySize {
|
||||||
|
return nil, errSessionPolicyTooLarge
|
||||||
|
}
|
||||||
|
return buf, nil
|
||||||
|
}
|
||||||
|
|
||||||
// NewServiceAccount - create a new service account
|
// NewServiceAccount - create a new service account
|
||||||
func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, groups []string, opts newServiceAccountOpts) (auth.Credentials, time.Time, error) {
|
func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, groups []string, opts newServiceAccountOpts) (auth.Credentials, time.Time, error) {
|
||||||
if !sys.Initialized() {
|
if !sys.Initialized() {
|
||||||
@@ -1115,20 +1141,10 @@ func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, gro
|
|||||||
return auth.Credentials{}, time.Time{}, auth.ErrNoAccessKeyWithSecretKey
|
return auth.Credentials{}, time.Time{}, auth.ErrNoAccessKeyWithSecretKey
|
||||||
}
|
}
|
||||||
|
|
||||||
var policyBuf []byte
|
policyBuf, err := marshalServiceAccountPolicy(opts.sessionPolicy)
|
||||||
if opts.sessionPolicy != nil {
|
|
||||||
err := opts.sessionPolicy.Validate()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return auth.Credentials{}, time.Time{}, err
|
return auth.Credentials{}, time.Time{}, err
|
||||||
}
|
}
|
||||||
policyBuf, err = json.Marshal(opts.sessionPolicy)
|
|
||||||
if err != nil {
|
|
||||||
return auth.Credentials{}, time.Time{}, err
|
|
||||||
}
|
|
||||||
if len(policyBuf) > maxSVCSessionPolicySize {
|
|
||||||
return auth.Credentials{}, time.Time{}, errSessionPolicyTooLarge
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// found newly requested service account, to be same as
|
// found newly requested service account, to be same as
|
||||||
// parentUser, reject such operations.
|
// parentUser, reject such operations.
|
||||||
@@ -1150,6 +1166,14 @@ func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, gro
|
|||||||
|
|
||||||
// Add all the necessary claims for the service account.
|
// Add all the necessary claims for the service account.
|
||||||
for k, v := range opts.claims {
|
for k, v := range opts.claims {
|
||||||
|
// Older snapshots may carry this cache-only, decoded policy claim.
|
||||||
|
if k == sessionPolicyNameExtracted {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// An explicit policy, including an empty one, owns this claim.
|
||||||
|
if k == policy.SessionPolicyName && opts.sessionPolicy != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
_, ok := m[k]
|
_, ok := m[k]
|
||||||
if !ok {
|
if !ok {
|
||||||
m[k] = v
|
m[k] = v
|
||||||
@@ -1162,7 +1186,6 @@ func (sys *IAMSys) NewServiceAccount(ctx context.Context, parentUser string, gro
|
|||||||
}
|
}
|
||||||
|
|
||||||
var accessKey, secretKey string
|
var accessKey, secretKey string
|
||||||
var err error
|
|
||||||
if len(opts.accessKey) > 0 || len(opts.secretKey) > 0 {
|
if len(opts.accessKey) > 0 || len(opts.secretKey) > 0 {
|
||||||
accessKey, secretKey = opts.accessKey, opts.secretKey
|
accessKey, secretKey = opts.accessKey, opts.secretKey
|
||||||
} else {
|
} else {
|
||||||
@@ -1311,22 +1334,65 @@ func (sys *IAMSys) getServiceAccount(ctx context.Context, accessKey string) (Use
|
|||||||
return UserIdentity{}, nil, errNoSuchServiceAccount
|
return UserIdentity{}, nil, errNoSuchServiceAccount
|
||||||
}
|
}
|
||||||
|
|
||||||
var embeddedPolicy *policy.Policy
|
embeddedPolicy, err := embeddedServiceAccountPolicy(jwtClaims)
|
||||||
|
if err != nil {
|
||||||
|
return UserIdentity{}, nil, err
|
||||||
|
}
|
||||||
|
return sa, embeddedPolicy, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// embeddedServiceAccountPolicy returns the policy embedded in the token of a
|
||||||
|
// service account, or nil when the account has the policies of its parent.
|
||||||
|
func embeddedServiceAccountPolicy(jwtClaims *jwt.MapClaims) (*policy.Policy, error) {
|
||||||
pt, ptok := jwtClaims.Lookup(iamPolicyClaimNameSA())
|
pt, ptok := jwtClaims.Lookup(iamPolicyClaimNameSA())
|
||||||
sp, spok := jwtClaims.Lookup(policy.SessionPolicyName)
|
sp, spok := jwtClaims.Lookup(policy.SessionPolicyName)
|
||||||
if ptok && spok && pt == embeddedPolicyType {
|
if !ptok || !spok || pt != embeddedPolicyType {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
policyBytes, err := base64.StdEncoding.DecodeString(sp)
|
policyBytes, err := base64.StdEncoding.DecodeString(sp)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return UserIdentity{}, nil, err
|
return nil, err
|
||||||
}
|
|
||||||
embeddedPolicy, err = policy.ParseConfig(bytes.NewReader(policyBytes))
|
|
||||||
if err != nil {
|
|
||||||
return UserIdentity{}, nil, err
|
|
||||||
}
|
}
|
||||||
|
return policy.ParseConfig(bytes.NewReader(policyBytes))
|
||||||
}
|
}
|
||||||
|
|
||||||
return sa, embeddedPolicy, nil
|
// serviceAccountSnapshot returns service account u, a record loaded from
|
||||||
|
// storage, as the create that replication, the first sync, heal and export
|
||||||
|
// carry. Its claims and its embedded policy are read from the token of that
|
||||||
|
// record, so that every part of the snapshot is of one revision: a server's
|
||||||
|
// cache may hold the account as it was before an edit made through another
|
||||||
|
// server.
|
||||||
|
func serviceAccountSnapshot(u UserIdentity) (madmin.SRSvcAccCreate, error) {
|
||||||
|
jwtClaims, err := extractJWTClaims(u)
|
||||||
|
if err != nil {
|
||||||
|
return madmin.SRSvcAccCreate{}, err
|
||||||
|
}
|
||||||
|
// Extraction adds a decoded policy for local authorization. Transport only
|
||||||
|
// the token's claims, so peers and imports do not sign this derived value.
|
||||||
|
jwtClaims.Delete(sessionPolicyNameExtracted)
|
||||||
|
embeddedPolicy, err := embeddedServiceAccountPolicy(jwtClaims)
|
||||||
|
if err != nil {
|
||||||
|
return madmin.SRSvcAccCreate{}, err
|
||||||
|
}
|
||||||
|
var policyJSON []byte
|
||||||
|
if embeddedPolicy != nil {
|
||||||
|
if policyJSON, err = json.Marshal(embeddedPolicy); err != nil {
|
||||||
|
return madmin.SRSvcAccCreate{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cr := u.Credentials
|
||||||
|
return madmin.SRSvcAccCreate{
|
||||||
|
Parent: cr.ParentUser,
|
||||||
|
AccessKey: cr.AccessKey,
|
||||||
|
SecretKey: cr.SecretKey,
|
||||||
|
Groups: cr.Groups,
|
||||||
|
Claims: jwtClaims.Map(),
|
||||||
|
SessionPolicy: policyJSON,
|
||||||
|
Status: cr.Status,
|
||||||
|
Name: cr.Name,
|
||||||
|
Description: cr.Description,
|
||||||
|
Expiration: &cr.Expiration,
|
||||||
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetTemporaryAccount - wrapper method to get information about a temporary account
|
// GetTemporaryAccount - wrapper method to get information about a temporary account
|
||||||
@@ -1398,25 +1464,6 @@ func (sys *IAMSys) getAccountWithClaims(ctx context.Context, accessKey string) (
|
|||||||
return acc, jwtClaims, nil
|
return acc, jwtClaims, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetClaimsForSvcAcc - gets the claims associated with the service account.
|
|
||||||
func (sys *IAMSys) GetClaimsForSvcAcc(ctx context.Context, accessKey string) (map[string]any, error) {
|
|
||||||
if !sys.Initialized() {
|
|
||||||
return nil, errServerNotInitialized
|
|
||||||
}
|
|
||||||
|
|
||||||
sa, ok := sys.store.GetUser(accessKey)
|
|
||||||
if !ok || !sa.Credentials.IsServiceAccount() {
|
|
||||||
return nil, errNoSuchServiceAccount
|
|
||||||
}
|
|
||||||
|
|
||||||
jwtClaims, err := extractJWTClaims(sa)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return jwtClaims.Map(), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// DeleteServiceAccount - delete a service account
|
// DeleteServiceAccount - delete a service account
|
||||||
func (sys *IAMSys) DeleteServiceAccount(ctx context.Context, accessKey string, notifyPeers bool) error {
|
func (sys *IAMSys) DeleteServiceAccount(ctx context.Context, accessKey string, notifyPeers bool) error {
|
||||||
if !sys.Initialized() {
|
if !sys.Initialized() {
|
||||||
|
|||||||
@@ -122,6 +122,8 @@ type ObjectOptions struct {
|
|||||||
|
|
||||||
// Pools-layer version resolver; the caller holds the shared object lock.
|
// Pools-layer version resolver; the caller holds the shared object lock.
|
||||||
replicaObjectInfo GetObjectInfoFn
|
replicaObjectInfo GetObjectInfoFn
|
||||||
|
// Verifies protected content in every pool before a replica commits.
|
||||||
|
replicaContentGuard replicaContentGuardFn
|
||||||
|
|
||||||
// RefuseLockedOverwrite refuses, under the destination's write lock, a
|
// RefuseLockedOverwrite refuses, under the destination's write lock, a
|
||||||
// write that would replace a stored version Object Lock protects: the
|
// write that would replace a stored version Object Lock protects: the
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
// Copyright (c) 2026 Denis Babchuk
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful,
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/xml"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
// GetObjectAttributes answers NoSuchKey with AccessDenied to a caller without
|
||||||
|
// s3:ListBucket on the bucket, as S3 does, so that a missing object cannot be
|
||||||
|
// told apart from a forbidden one. A server error, such as lost read quorum,
|
||||||
|
// must reach a caller allowed to read the object as the error it is.
|
||||||
|
func TestGetObjectAttributesReadQuorumLost(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
defer cancel()
|
||||||
|
obj, dirs, err := prepareErasure16(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
z := obj.(*erasureServerPools)
|
||||||
|
previous := newObjectLayerFn()
|
||||||
|
setObjectLayer(z)
|
||||||
|
defer func() {
|
||||||
|
z.Shutdown(context.Background())
|
||||||
|
removeRoots(dirs)
|
||||||
|
setObjectLayer(previous)
|
||||||
|
}()
|
||||||
|
bucket, router, err := initAPIHandlerTest(ctx, z, nil, MakeBucketOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
const object, missing = "object", "missing"
|
||||||
|
if _, err := z.PutObject(ctx, bucket, object, mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""), ObjectOptions{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
read := `{"Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectAttributes"], "Resource": ["arn:aws:s3:::` + bucket + `/*"]}`
|
||||||
|
withoutList := newDeleteAuthzPolicyUser(t, "", bucket, "["+read+"]")
|
||||||
|
withList := newDeleteAuthzPolicyUser(t, "", bucket, "["+read+
|
||||||
|
`, {"Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::`+bucket+`"]}]`)
|
||||||
|
|
||||||
|
check := func(stage, user string, creds auth.Credentials, key string, want int) {
|
||||||
|
t.Helper()
|
||||||
|
target := makeTestTargetURL("", bucket, key, url.Values{"attributes": []string{""}})
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodGet, target, 0, nil, creds.AccessKey, creds.SecretKey,
|
||||||
|
map[string]string{xhttp.AmzObjectAttributes: "ETag"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
if rec.Code != want {
|
||||||
|
var apiErr APIErrorResponse
|
||||||
|
_ = xml.Unmarshal(rec.Body.Bytes(), &apiErr)
|
||||||
|
t.Errorf("%s: GetObjectAttributes %s by %s: status %d %s, want %d", stage, key, user, rec.Code, apiErr.Code, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
check("healthy", "withoutList", withoutList, object, http.StatusOK)
|
||||||
|
check("healthy", "withList", withList, object, http.StatusOK)
|
||||||
|
check("healthy", "withoutList", withoutList, missing, http.StatusForbidden)
|
||||||
|
check("healthy", "withList", withList, missing, http.StatusNotFound)
|
||||||
|
|
||||||
|
defer objectReadFault(z, bucket, object, 10)()
|
||||||
|
check("read quorum lost", "withoutList", withoutList, object, http.StatusServiceUnavailable)
|
||||||
|
check("read quorum lost", "withList", withList, object, http.StatusServiceUnavailable)
|
||||||
|
}
|
||||||
+176
-54
@@ -395,12 +395,14 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Validate pre-conditions if any.
|
// Validate pre-conditions if any.
|
||||||
|
var storedTags *string // set once the object has been read far enough
|
||||||
opts.CheckPrecondFn = func(oi ObjectInfo) bool {
|
opts.CheckPrecondFn = func(oi ObjectInfo) bool {
|
||||||
if _, err := DecryptObjectInfo(&oi, r); err != nil {
|
if _, err := DecryptObjectInfo(&oi, r); err != nil {
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
storedTags = &oi.UserTags
|
||||||
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.GetObjectAction, oi.UserTags); s3Error != ErrNone {
|
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.GetObjectAction, oi.UserTags); s3Error != ErrNone {
|
||||||
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
return true
|
return true
|
||||||
@@ -445,8 +447,14 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
// validate if the request indeed was authorized, if it wasn't we need to return "ErrAccessDenied"
|
// validate if the request indeed was authorized, if it wasn't we need to return "ErrAccessDenied"
|
||||||
// instead of any namespace related error.
|
// instead of any namespace related error. Use the stored tags if the callback read them.
|
||||||
if s3Error := authorizeRequest(ctx, r, policy.GetObjectAction); s3Error != ErrNone {
|
var s3Error APIErrorCode
|
||||||
|
if storedTags != nil {
|
||||||
|
s3Error = authorizeRequestWithExistingTags(ctx, r, policy.GetObjectAction, *storedTags)
|
||||||
|
} else {
|
||||||
|
s3Error = authorizeRequestWithUnreadTags(ctx, r, policy.GetObjectAction, err)
|
||||||
|
}
|
||||||
|
if s3Error != ErrNone {
|
||||||
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -623,12 +631,15 @@ func (api objectAPIHandlers) getObjectAttributesHandler(ctx context.Context, obj
|
|||||||
|
|
||||||
objInfo, err := objectAPI.GetObjectInfo(ctx, bucket, object, opts)
|
objInfo, err := objectAPI.GetObjectInfo(ctx, bucket, object, opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s3Error = authorizeRequest(ctx, r, policy.ListBucketAction)
|
// As with S3, a caller without s3:ListBucket on the bucket gets
|
||||||
if s3Error == ErrNone {
|
// AccessDenied in place of NoSuchKey. A server error, such as lost
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
// read quorum, tells nothing about the object and reaches the caller,
|
||||||
return
|
// who may read the object, as it is.
|
||||||
|
apiErr := toAPIError(ctx, err)
|
||||||
|
if apiErr.HTTPStatusCode < http.StatusInternalServerError && authorizeBucketRequest(ctx, r, policy.ListBucketAction) != ErrNone {
|
||||||
|
apiErr = errorCodes.ToAPIErr(ErrAccessDenied)
|
||||||
}
|
}
|
||||||
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
|
writeErrorResponse(ctx, w, apiErr, r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -925,7 +936,14 @@ func (api objectAPIHandlers) headObjectHandler(ctx context.Context, objectAPI Ob
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.GetObjectAction, objInfo.UserTags); s3Error != ErrNone {
|
// objInfo carries the stored tags unless reading the object failed.
|
||||||
|
var s3Error APIErrorCode
|
||||||
|
if err != nil && !proxy.Proxy {
|
||||||
|
s3Error = authorizeRequestWithUnreadTags(ctx, r, policy.GetObjectAction, err)
|
||||||
|
} else {
|
||||||
|
s3Error = authorizeRequestWithExistingTags(ctx, r, policy.GetObjectAction, objInfo.UserTags)
|
||||||
|
}
|
||||||
|
if s3Error != ErrNone {
|
||||||
writeErrorResponseHeadersOnly(w, errorCodes.ToAPIErr(s3Error))
|
writeErrorResponseHeadersOnly(w, errorCodes.ToAPIErr(s3Error))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -1829,12 +1847,27 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
|
|||||||
|
|
||||||
srcInfo.PutObjReader = pReader
|
srcInfo.PutObjReader = pReader
|
||||||
|
|
||||||
// Object Lock state as stored on disk, captured before the metadata
|
// Reconcile against the addressed destination version under its write
|
||||||
// directive rebuilds the map. A replica update is applied only when its
|
// lock. The source can be a different object or version, including when
|
||||||
// source timestamp is newer than the stored one, and a stale update must
|
// the destination version has never been written.
|
||||||
// leave the stored state in place instead of erasing it.
|
getObjectInfo := func(_ context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
|
||||||
storedLock := storedObjectLockState(srcInfo.UserDefined)
|
opts.NoLock = true
|
||||||
storedTagTimestamp := srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
|
return objectAPI.GetObjectInfo(lockCtx, bucket, object, opts)
|
||||||
|
}
|
||||||
|
var storedInfo ObjectInfo
|
||||||
|
if replicaTrusted && !remoteCallRequired {
|
||||||
|
storedInfo, err = replicaStoredObjectInfo(lockCtx, getObjectInfo, dstBucket, dstObject, dstOpts.VersionID)
|
||||||
|
if err != nil {
|
||||||
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
storedLock := storedObjectLockState(storedInfo.UserDefined)
|
||||||
|
storedTagTimestamp := storedInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
|
||||||
|
localTagTimestamp := ""
|
||||||
|
if cpSrcDstSame {
|
||||||
|
localTagTimestamp = srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp]
|
||||||
|
}
|
||||||
|
|
||||||
srcInfo.UserDefined, err = getCpObjMetadataFromHeader(ctx, r, srcInfo.UserDefined, allowReplicationMetadata)
|
srcInfo.UserDefined, err = getCpObjMetadataFromHeader(ctx, r, srcInfo.UserDefined, allowReplicationMetadata)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1859,9 +1892,9 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
|
|||||||
// the captured state even if metadata REPLACE rebuilt the map.
|
// the captured state even if metadata REPLACE rebuilt the map.
|
||||||
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = srcTimestamp.UTC().Format(time.RFC3339Nano)
|
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = srcTimestamp.UTC().Format(time.RFC3339Nano)
|
||||||
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
|
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
|
||||||
reconcileStoredObjectTags(srcInfo.UserDefined, srcInfo.UserTags, storedTagTimestamp)
|
reconcileStoredObjectTags(srcInfo.UserDefined, storedInfo.UserTags, storedTagTimestamp)
|
||||||
} else {
|
} else {
|
||||||
srcInfo.UserDefined[xhttp.AmzObjectTagging] = srcInfo.UserTags
|
srcInfo.UserDefined[xhttp.AmzObjectTagging] = storedInfo.UserTags
|
||||||
if storedTagTimestamp != "" {
|
if storedTagTimestamp != "" {
|
||||||
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = storedTagTimestamp
|
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = storedTagTimestamp
|
||||||
} else {
|
} else {
|
||||||
@@ -1870,7 +1903,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
|
|||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
|
srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
|
||||||
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = UTCNow().Format(time.RFC3339Nano)
|
srcInfo.UserDefined[ReservedMetadataPrefixLower+TaggingTimestamp] = monotonicMetadataTimestamp(UTCNow().Format(time.RFC3339Nano), localTagTimestamp)
|
||||||
}
|
}
|
||||||
// SSE-C rotation snapshots reserved metadata before the tag decision. Its
|
// SSE-C rotation snapshots reserved metadata before the tag decision. Its
|
||||||
// later merge must not put the old timestamp back over the accepted state.
|
// later merge must not put the old timestamp back over the accepted state.
|
||||||
@@ -1878,14 +1911,12 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
|
|||||||
|
|
||||||
srcInfo.UserDefined = filterReplicationStatusMetadata(srcInfo.UserDefined)
|
srcInfo.UserDefined = filterReplicationStatusMetadata(srcInfo.UserDefined)
|
||||||
srcInfo.UserDefined = objectlock.FilterObjectLockMetadata(srcInfo.UserDefined, true, true)
|
srcInfo.UserDefined = objectlock.FilterObjectLockMetadata(srcInfo.UserDefined, true, true)
|
||||||
|
// COPY may have preserved the source's reserved timestamps. Only the
|
||||||
|
// addressed destination and the incoming update may order a replica.
|
||||||
|
delete(srcInfo.UserDefined, ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp)
|
||||||
|
delete(srcInfo.UserDefined, ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp)
|
||||||
retPerms := isPutActionAllowed(ctx, getRequestAuthType(r), dstBucket, dstObject, r, policy.PutObjectRetentionAction)
|
retPerms := isPutActionAllowed(ctx, getRequestAuthType(r), dstBucket, dstObject, r, policy.PutObjectRetentionAction)
|
||||||
holdPerms := isPutActionAllowed(ctx, getRequestAuthType(r), dstBucket, dstObject, r, policy.PutObjectLegalHoldAction)
|
holdPerms := isPutActionAllowed(ctx, getRequestAuthType(r), dstBucket, dstObject, r, policy.PutObjectLegalHoldAction)
|
||||||
// The destination is locked above: look it up without its lock, in the
|
|
||||||
// locks' context.
|
|
||||||
getObjectInfo := func(_ context.Context, bucket, object string, opts ObjectOptions) (ObjectInfo, error) {
|
|
||||||
opts.NoLock = true
|
|
||||||
return objectAPI.GetObjectInfo(lockCtx, bucket, object, opts)
|
|
||||||
}
|
|
||||||
|
|
||||||
// apply default bucket configuration/governance headers for dest side.
|
// apply default bucket configuration/governance headers for dest side.
|
||||||
retentionMode, retentionDate, legalHold, s3Err := checkPutObjectLockAllowed(ctx, r, dstBucket, dstObject, getObjectInfo, retPerms, holdPerms, replicaTrusted)
|
retentionMode, retentionDate, legalHold, s3Err := checkPutObjectLockAllowed(ctx, r, dstBucket, dstObject, getObjectInfo, retPerms, holdPerms, replicaTrusted)
|
||||||
@@ -3140,6 +3171,22 @@ func (api objectAPIHandlers) DeleteObjectHandler(w http.ResponseWriter, r *http.
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Check before delOpts reads bucket versioning: unavailable bucket metadata
|
||||||
|
// must fail the request before a force-delete can reach the object layer.
|
||||||
|
if forceDelete, _ := strconv.ParseBool(r.Header.Get(xhttp.MinIOForceDelete)); forceDelete {
|
||||||
|
rcfg, err := globalBucketObjectLockSys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if rcfg.LockEnabled {
|
||||||
|
apiErr := toAPIError(ctx, errInvalidArgument)
|
||||||
|
apiErr.Description = "force-delete is forbidden on Object Locking enabled buckets"
|
||||||
|
writeErrorResponse(ctx, w, apiErr, r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
opts, err := delOpts(ctx, r, bucket, object)
|
opts, err := delOpts(ctx, r, bucket, object)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
@@ -3160,14 +3207,6 @@ func (api objectAPIHandlers) DeleteObjectHandler(w http.ResponseWriter, r *http.
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
rcfg, _ := globalBucketObjectLockSys.Get(bucket)
|
|
||||||
if rcfg.LockEnabled && opts.DeletePrefix {
|
|
||||||
apiErr := toAPIError(ctx, errInvalidArgument)
|
|
||||||
apiErr.Description = "force-delete is forbidden on Object Locking enabled buckets"
|
|
||||||
writeErrorResponse(ctx, w, apiErr, r.URL)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
os := newObjSweeper(bucket, object).WithVersion(opts.VersionID).WithVersioning(opts.Versioned, opts.VersionSuspended)
|
os := newObjSweeper(bucket, object).WithVersion(opts.VersionID).WithVersioning(opts.Versioned, opts.VersionSuspended)
|
||||||
|
|
||||||
opts.SetEvalMetadataFn(func(oi *ObjectInfo, gerr error) (dsc ReplicateDecision, err error) {
|
opts.SetEvalMetadataFn(func(oi *ObjectInfo, gerr error) (dsc ReplicateDecision, err error) {
|
||||||
@@ -3366,7 +3405,7 @@ func (api objectAPIHandlers) PutObjectLegalHoldHandler(w http.ResponseWriter, r
|
|||||||
VersionID: opts.VersionID,
|
VersionID: opts.VersionID,
|
||||||
EvalMetadataFn: func(oi *ObjectInfo, gerr error) (ReplicateDecision, error) {
|
EvalMetadataFn: func(oi *ObjectInfo, gerr error) (ReplicateDecision, error) {
|
||||||
oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = strings.ToUpper(string(legalHold.Status))
|
oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockLegalHold)] = strings.ToUpper(string(legalHold.Status))
|
||||||
oi.UserDefined[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp] = UTCNow().Format(time.RFC3339Nano)
|
oi.UserDefined[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp] = monotonicMetadataTimestamp(UTCNow().Format(time.RFC3339Nano), oi.UserDefined[ReservedMetadataPrefixLower+ObjectLockLegalHoldTimestamp])
|
||||||
|
|
||||||
dsc := mustReplicate(ctx, bucket, object, oi.getMustReplicateOptions(replication.MetadataReplicationType, opts))
|
dsc := mustReplicate(ctx, bucket, object, oi.getMustReplicateOptions(replication.MetadataReplicationType, opts))
|
||||||
if dsc.ReplicateAny() {
|
if dsc.ReplicateAny() {
|
||||||
@@ -3543,7 +3582,7 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
|
|||||||
oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] = ""
|
oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] = ""
|
||||||
oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = ""
|
oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] = ""
|
||||||
}
|
}
|
||||||
oi.UserDefined[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp] = UTCNow().Format(time.RFC3339Nano)
|
oi.UserDefined[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp] = monotonicMetadataTimestamp(UTCNow().Format(time.RFC3339Nano), oi.UserDefined[ReservedMetadataPrefixLower+ObjectLockRetentionTimestamp])
|
||||||
dsc = mustReplicate(ctx, bucket, object, oi.getMustReplicateOptions(replication.MetadataReplicationType, opts))
|
dsc = mustReplicate(ctx, bucket, object, oi.getMustReplicateOptions(replication.MetadataReplicationType, opts))
|
||||||
if dsc.ReplicateAny() {
|
if dsc.ReplicateAny() {
|
||||||
oi.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] = UTCNow().Format(time.RFC3339Nano)
|
oi.UserDefined[ReservedMetadataPrefixLower+ReplicationTimestamp] = UTCNow().Format(time.RFC3339Nano)
|
||||||
@@ -3687,9 +3726,22 @@ func (api objectAPIHandlers) GetObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
proxytgts := getProxyTargets(ctx, bucket, object, opts)
|
proxytgts := getProxyTargets(ctx, bucket, object, opts)
|
||||||
if !proxytgts.Empty() {
|
if !proxytgts.Empty() {
|
||||||
|
if s3Error := authorizeRequestWithPotentialTags(ctx, r, policy.GetObjectTaggingAction); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
globalReplicationStats.Load().incProxy(bucket, getObjectTaggingAPI, false)
|
globalReplicationStats.Load().incProxy(bucket, getObjectTaggingAPI, false)
|
||||||
// proxy to replication target if site replication is in place.
|
// proxy to replication target if site replication is in place.
|
||||||
tags, gerr := proxyGetTaggingToRepTarget(ctx, bucket, object, opts, proxytgts)
|
tags, gerr, s3Error := proxyGetTaggingToRepTarget(ctx, bucket, object, opts, proxytgts, func(existingTags string, readErr error) APIErrorCode {
|
||||||
|
if readErr != nil && !isRemoteTaggingNotFound(readErr) {
|
||||||
|
return authorizeRequestWithUnreadTags(ctx, r, policy.GetObjectTaggingAction, readErr)
|
||||||
|
}
|
||||||
|
return authorizeRequestWithExistingTags(ctx, r, policy.GetObjectTaggingAction, existingTags)
|
||||||
|
})
|
||||||
|
if s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
if gerr.Err != nil || tags == nil {
|
if gerr.Err != nil || tags == nil {
|
||||||
globalReplicationStats.Load().incProxy(bucket, getObjectTaggingAPI, true)
|
globalReplicationStats.Load().incProxy(bucket, getObjectTaggingAPI, true)
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, gerr.Err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, gerr.Err), r.URL)
|
||||||
@@ -3698,10 +3750,18 @@ func (api objectAPIHandlers) GetObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
ot = tags
|
ot = tags
|
||||||
w.Header()[xhttp.MinIOTaggingProxied] = []string{"true"} // indicate that the request was proxied.
|
w.Header()[xhttp.MinIOTaggingProxied] = []string{"true"} // indicate that the request was proxied.
|
||||||
} else {
|
} else {
|
||||||
|
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.GetObjectTaggingAction, ""); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
if s3Error := authorizeRequestWithUnreadTags(ctx, r, policy.GetObjectTaggingAction, err); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -3736,6 +3796,18 @@ func (api objectAPIHandlers) GetObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
writeSuccessResponseXML(w, encodeResponse(otags))
|
writeSuccessResponseXML(w, encodeResponse(otags))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func isRemoteTaggingNotFound(err error) bool {
|
||||||
|
if err == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch miniogo.ToErrorResponse(err).Code {
|
||||||
|
case "NoSuchKey", "NoSuchVersion":
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// PutObjectTaggingHandler - PUT object tagging
|
// PutObjectTaggingHandler - PUT object tagging
|
||||||
func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *http.Request) {
|
func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *http.Request) {
|
||||||
ctx := newContext(r, w, "PutObjectTagging")
|
ctx := newContext(r, w, "PutObjectTagging")
|
||||||
@@ -3755,6 +3827,13 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
logger.GetReqInfo(ctx).BucketName = bucket
|
||||||
|
logger.GetReqInfo(ctx).ObjectName = object
|
||||||
|
if s3Error := authenticateRequest(ctx, r, policy.PutObjectTaggingAction); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
// Tags XML will not be bigger than 1MiB in size, fail if its bigger.
|
// Tags XML will not be bigger than 1MiB in size, fail if its bigger.
|
||||||
tags, err := tags.ParseObjectXML(io.LimitReader(r.Body, 1<<20))
|
tags, err := tags.ParseObjectXML(io.LimitReader(r.Body, 1<<20))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -3763,13 +3842,6 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
}
|
}
|
||||||
tagsStr := tags.String()
|
tagsStr := tags.String()
|
||||||
|
|
||||||
logger.GetReqInfo(ctx).BucketName = bucket
|
|
||||||
logger.GetReqInfo(ctx).ObjectName = object
|
|
||||||
if s3Error := authenticateRequest(ctx, r, policy.PutObjectTaggingAction); s3Error != ErrNone {
|
|
||||||
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// Set this such that authorization policies can be applied on the object
|
// Set this such that authorization policies can be applied on the object
|
||||||
// tags. This is derived from the request body, so it must be injected only
|
// tags. This is derived from the request body, so it must be injected only
|
||||||
// after signature verification: the SigV4 verifier now rejects unsigned
|
// after signature verification: the SigV4 verifier now rejects unsigned
|
||||||
@@ -3783,14 +3855,6 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
}
|
}
|
||||||
|
|
||||||
objInfo, err := objAPI.GetObjectInfo(ctx, bucket, object, opts)
|
objInfo, err := objAPI.GetObjectInfo(ctx, bucket, object, opts)
|
||||||
existingTags := ""
|
|
||||||
if err == nil {
|
|
||||||
existingTags = objInfo.UserTags
|
|
||||||
}
|
|
||||||
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.PutObjectTaggingAction, existingTags); s3Error != ErrNone {
|
|
||||||
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// if object is not found locally, but exists on peer site - proxy
|
// if object is not found locally, but exists on peer site - proxy
|
||||||
// the tagging request to peer site. The response to client will
|
// the tagging request to peer site. The response to client will
|
||||||
@@ -3798,9 +3862,21 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
proxytgts := getProxyTargets(ctx, bucket, object, opts)
|
proxytgts := getProxyTargets(ctx, bucket, object, opts)
|
||||||
if !proxytgts.Empty() {
|
if !proxytgts.Empty() {
|
||||||
|
if s3Error := authorizeRequestWithPotentialTags(ctx, r, policy.PutObjectTaggingAction); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
perr, s3Error := proxyTaggingToRepTarget(ctx, bucket, object, tags, opts, proxytgts, func(existingTags string, readErr error) APIErrorCode {
|
||||||
|
if readErr != nil && !isRemoteTaggingNotFound(readErr) {
|
||||||
|
return authorizeRequestWithUnreadTags(ctx, r, policy.PutObjectTaggingAction, readErr)
|
||||||
|
}
|
||||||
|
return authorizeRequestWithExistingTags(ctx, r, policy.PutObjectTaggingAction, existingTags)
|
||||||
|
})
|
||||||
|
if s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
globalReplicationStats.Load().incProxy(bucket, putObjectTaggingAPI, false)
|
globalReplicationStats.Load().incProxy(bucket, putObjectTaggingAPI, false)
|
||||||
// proxy to replication target if site replication is in place.
|
|
||||||
perr := proxyTaggingToRepTarget(ctx, bucket, object, tags, opts, proxytgts)
|
|
||||||
if perr.Err != nil {
|
if perr.Err != nil {
|
||||||
globalReplicationStats.Load().incProxy(bucket, putObjectTaggingAPI, true)
|
globalReplicationStats.Load().incProxy(bucket, putObjectTaggingAPI, true)
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, perr.Err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, perr.Err), r.URL)
|
||||||
@@ -3821,9 +3897,26 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// A local read failure must never be turned into a write grant. A
|
||||||
|
// server error may still be disclosed to an otherwise authorized
|
||||||
|
// caller for retry, while a missing object is evaluated as untagged.
|
||||||
|
var s3Error APIErrorCode
|
||||||
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
|
s3Error = authorizeRequestWithExistingTags(ctx, r, policy.PutObjectTaggingAction, "")
|
||||||
|
} else {
|
||||||
|
s3Error = authorizeRequestWithUnreadTags(ctx, r, policy.PutObjectTaggingAction, err)
|
||||||
|
}
|
||||||
|
if s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.PutObjectTaggingAction, objInfo.UserTags); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
dsc := mustReplicate(ctx, bucket, object, getMustReplicateOptions(objInfo.UserDefined, tagsStr, objInfo.ReplicationStatus, replication.MetadataReplicationType, opts))
|
dsc := mustReplicate(ctx, bucket, object, getMustReplicateOptions(objInfo.UserDefined, tagsStr, objInfo.ReplicationStatus, replication.MetadataReplicationType, opts))
|
||||||
stamp := UTCNow().Format(time.RFC3339Nano)
|
stamp := UTCNow().Format(time.RFC3339Nano)
|
||||||
@@ -3880,6 +3973,13 @@ func (api objectAPIHandlers) DeleteObjectTaggingHandler(w http.ResponseWriter, r
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
logger.GetReqInfo(ctx).BucketName = bucket
|
||||||
|
logger.GetReqInfo(ctx).ObjectName = object
|
||||||
|
if s3Error := authenticateRequest(ctx, r, policy.DeleteObjectTaggingAction); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
opts, err := getOpts(ctx, r, bucket, object)
|
opts, err := getOpts(ctx, r, bucket, object)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
@@ -3894,9 +3994,21 @@ func (api objectAPIHandlers) DeleteObjectTaggingHandler(w http.ResponseWriter, r
|
|||||||
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
proxytgts := getProxyTargets(ctx, bucket, object, opts)
|
proxytgts := getProxyTargets(ctx, bucket, object, opts)
|
||||||
if !proxytgts.Empty() {
|
if !proxytgts.Empty() {
|
||||||
|
if s3Error := authorizeRequestWithPotentialTags(ctx, r, policy.DeleteObjectTaggingAction); s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
perr, s3Error := proxyTaggingToRepTarget(ctx, bucket, object, nil, opts, proxytgts, func(existingTags string, readErr error) APIErrorCode {
|
||||||
|
if readErr != nil && !isRemoteTaggingNotFound(readErr) {
|
||||||
|
return authorizeRequestWithUnreadTags(ctx, r, policy.DeleteObjectTaggingAction, readErr)
|
||||||
|
}
|
||||||
|
return authorizeRequestWithExistingTags(ctx, r, policy.DeleteObjectTaggingAction, existingTags)
|
||||||
|
})
|
||||||
|
if s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
globalReplicationStats.Load().incProxy(bucket, removeObjectTaggingAPI, false)
|
globalReplicationStats.Load().incProxy(bucket, removeObjectTaggingAPI, false)
|
||||||
// proxy to replication target if active-active replication is in place.
|
|
||||||
perr := proxyTaggingToRepTarget(ctx, bucket, object, nil, opts, proxytgts)
|
|
||||||
if perr.Err != nil {
|
if perr.Err != nil {
|
||||||
globalReplicationStats.Load().incProxy(bucket, removeObjectTaggingAPI, true)
|
globalReplicationStats.Load().incProxy(bucket, removeObjectTaggingAPI, true)
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, perr.Err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, perr.Err), r.URL)
|
||||||
@@ -3918,12 +4030,22 @@ func (api objectAPIHandlers) DeleteObjectTaggingHandler(w http.ResponseWriter, r
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
var s3Error APIErrorCode
|
||||||
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
|
s3Error = authorizeRequestWithExistingTags(ctx, r, policy.DeleteObjectTaggingAction, "")
|
||||||
|
} else {
|
||||||
|
s3Error = authorizeRequestWithUnreadTags(ctx, r, policy.DeleteObjectTaggingAction, err)
|
||||||
|
}
|
||||||
|
if s3Error != ErrNone {
|
||||||
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
|
return
|
||||||
|
}
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Allow deleteObjectTagging if policy action is set
|
// Allow deleteObjectTagging if policy action is set
|
||||||
if s3Error := checkRequestAuthTypeWithExistingTags(ctx, r, policy.DeleteObjectTaggingAction, bucket, object, oi.UserTags); s3Error != ErrNone {
|
if s3Error := authorizeRequestWithExistingTags(ctx, r, policy.DeleteObjectTaggingAction, oi.UserTags); s3Error != ErrNone {
|
||||||
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,421 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/md5"
|
||||||
|
"encoding/base64"
|
||||||
|
"fmt"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPoolLockReductionOrderIndependent(t *testing.T) {
|
||||||
|
const old = objectLockTestStamp0900
|
||||||
|
const recent = objectLockTestStamp1000
|
||||||
|
const newest = objectLockTestStamp1100
|
||||||
|
compliance := objectLockState{mode: "COMPLIANCE", retainUntil: "2035-01-01T00:00:00Z", retentionTimestamp: old}
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
states [3]objectLockState
|
||||||
|
want objectLockState
|
||||||
|
}{
|
||||||
|
{"compliance-before-newer-removal", [3]objectLockState{compliance, {retentionTimestamp: recent}, {mode: "GOVERNANCE", retainUntil: "2036-01-01T00:00:00Z", retentionTimestamp: newest}}, compliance},
|
||||||
|
{"longer-compliance-before-newer-shortening", [3]objectLockState{compliance, {mode: "COMPLIANCE", retainUntil: "2034-01-01T00:00:00Z", retentionTimestamp: newest}, {retentionTimestamp: recent}}, compliance},
|
||||||
|
{"unordered-compliance", [3]objectLockState{{mode: "COMPLIANCE", retainUntil: compliance.retainUntil, retentionTimestamp: "invalid"}, {retentionTimestamp: newest}, {}}, objectLockState{mode: "COMPLIANCE", retainUntil: compliance.retainUntil, retentionTimestamp: "invalid"}},
|
||||||
|
{"expired-compliance-can-release", [3]objectLockState{{mode: "COMPLIANCE", retainUntil: "2000-01-01T00:00:00Z", retentionTimestamp: old}, {retentionTimestamp: newest}, {retentionTimestamp: recent}}, objectLockState{retentionTimestamp: newest}},
|
||||||
|
{"governance-can-release", [3]objectLockState{{mode: "GOVERNANCE", retainUntil: compliance.retainUntil, retentionTimestamp: old}, {retentionTimestamp: newest}, {}}, objectLockState{retentionTimestamp: newest}},
|
||||||
|
{"unordered-hold-conflict", [3]objectLockState{{legalHold: "OFF"}, {legalHold: "ON"}, {}}, objectLockState{legalHold: "ON"}},
|
||||||
|
{"newer-hold-release", [3]objectLockState{{legalHold: "ON", legalHoldTimestamp: old}, {legalHold: "OFF", legalHoldTimestamp: newest}, {legalHold: "ON", legalHoldTimestamp: "invalid"}}, objectLockState{legalHold: "OFF", legalHoldTimestamp: newest}},
|
||||||
|
{"independent-fields", [3]objectLockState{compliance, {retentionTimestamp: newest, legalHold: "ON", legalHoldTimestamp: recent}, {legalHold: "OFF", legalHoldTimestamp: old}}, objectLockState{mode: compliance.mode, retainUntil: compliance.retainUntil, retentionTimestamp: old, legalHold: "ON", legalHoldTimestamp: recent}},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
for _, order := range [][3]int{{0, 1, 2}, {0, 2, 1}, {1, 0, 2}, {1, 2, 0}, {2, 0, 1}, {2, 1, 0}} {
|
||||||
|
var copies []PoolObjInfo
|
||||||
|
for _, i := range order {
|
||||||
|
meta := make(map[string]string)
|
||||||
|
tc.states[i].restoreRetention(meta)
|
||||||
|
tc.states[i].restoreLegalHold(meta)
|
||||||
|
copies = append(copies, PoolObjInfo{Index: i, ObjInfo: ObjectInfo{UserDefined: meta}})
|
||||||
|
}
|
||||||
|
for name, reduce := range map[string]func([]PoolObjInfo) objectLockState{"merge": mergePoolLockState, "lifecycle": newestLockState} {
|
||||||
|
if got := reduce(copies); got != tc.want {
|
||||||
|
t.Errorf("%s order=%v: got %+v want %+v", name, order, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// All requests copy the same bytes. Revisions belong to the addressed
|
||||||
|
// destination, even if the source is the latest version of the same key.
|
||||||
|
func TestAPICopyReplicaDestinationState(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
for _, sameKey := range []bool{true, false} {
|
||||||
|
for _, missing := range []bool{false, true} {
|
||||||
|
for _, directive := range []string{copyDirective, replaceDirective} {
|
||||||
|
name := fmt.Sprintf("copy-destination/%t/%t/%s", sameKey, missing, directive)
|
||||||
|
source := name
|
||||||
|
if !sameKey {
|
||||||
|
source += "-source"
|
||||||
|
}
|
||||||
|
targetMeta := poolLockMetadata("GOVERNANCE", "ON", objectLockTestStamp1000, objectLockTestStamp1000)
|
||||||
|
targetMeta[xhttp.AmzObjectTagging] = "state=destination"
|
||||||
|
targetMeta[r5TagStamp] = objectLockTestStamp1000
|
||||||
|
targetID := mustGetUUID()
|
||||||
|
if !missing {
|
||||||
|
targetID = putObjectLockVersion(t, obj, bucket, name, targetMeta)
|
||||||
|
}
|
||||||
|
sourceMeta := poolLockMetadata("", "OFF", objectLockTestStamp1100, objectLockTestStamp1100)
|
||||||
|
sourceMeta[xhttp.AmzObjectTagging] = "state=source"
|
||||||
|
sourceMeta[r5TagStamp] = objectLockTestStamp1100
|
||||||
|
sourceID := putObjectLockVersion(t, obj, bucket, source, sourceMeta)
|
||||||
|
headers := map[string]string{
|
||||||
|
xhttp.AmzCopySource: "/" + bucket + "/" + source + "?versionId=" + sourceID,
|
||||||
|
xhttp.AmzMetadataDirective: directive, xhttp.AmzTagDirective: replaceDirective,
|
||||||
|
xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA",
|
||||||
|
}
|
||||||
|
// No source revision means no change, including no
|
||||||
|
// inheritance from a different source for a first write.
|
||||||
|
rec := r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/"+name+"?versionId="+targetID, "", headers)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s %s: %d %s", backend, name, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
want := storedObjectLockState(targetMeta)
|
||||||
|
wantTags, wantStamp := "state=destination", objectLockTestStamp1000
|
||||||
|
if missing {
|
||||||
|
want = objectLockState{}
|
||||||
|
wantTags = ""
|
||||||
|
wantStamp = ""
|
||||||
|
}
|
||||||
|
got := r5Stored(t, obj, bucket, name, targetID, wantTags, wantStamp)
|
||||||
|
if state := storedObjectLockState(got.UserDefined); state != want {
|
||||||
|
t.Fatalf("%s absent revision: got %+v want %+v", name, state, want)
|
||||||
|
}
|
||||||
|
// The incoming update is older than the source but may
|
||||||
|
// still win at a new destination. A source lookup loses it.
|
||||||
|
headers[xhttp.MinIOSourceObjectRetentionTimestamp] = objectLockTestStamp0930
|
||||||
|
headers[xhttp.MinIOSourceObjectLegalHoldTimestamp] = objectLockTestStamp0930
|
||||||
|
headers[xhttp.AmzObjectLockMode] = "GOVERNANCE"
|
||||||
|
headers[xhttp.AmzObjectLockRetainUntilDate] = objectLockTestRetainUntil
|
||||||
|
headers[xhttp.AmzObjectLockLegalHold] = "ON"
|
||||||
|
headers[xhttp.MinIOSourceTaggingTimestamp] = objectLockTestStamp0930
|
||||||
|
headers[xhttp.AmzObjectTagging] = "state=incoming"
|
||||||
|
rec = r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/"+name+"?versionId="+targetID, "", headers)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("incoming %s: %d %s", name, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if missing {
|
||||||
|
want = objectLockState{mode: "GOVERNANCE", retainUntil: "2030-01-01T00:00:00.000Z", retentionTimestamp: objectLockTestStamp0930, legalHold: "ON", legalHoldTimestamp: objectLockTestStamp0930}
|
||||||
|
wantTags = "state=incoming"
|
||||||
|
wantStamp = objectLockTestStamp0930
|
||||||
|
}
|
||||||
|
got = r5Stored(t, obj, bucket, name, targetID, wantTags, wantStamp)
|
||||||
|
if state := storedObjectLockState(got.UserDefined); state != want {
|
||||||
|
t.Fatalf("%s stale revision: got %+v want %+v", name, state, want)
|
||||||
|
}
|
||||||
|
// Authorized same-content release needs no governance
|
||||||
|
// bypass header; it must not be rejected by the data guard.
|
||||||
|
delete(headers, xhttp.AmzObjectLockMode)
|
||||||
|
delete(headers, xhttp.AmzObjectLockRetainUntilDate)
|
||||||
|
headers[xhttp.MinIOSourceObjectRetentionTimestamp] = "2026-09-03T10:30:00Z"
|
||||||
|
headers[xhttp.MinIOSourceObjectLegalHoldTimestamp] = "2026-09-03T10:30:00Z"
|
||||||
|
headers[xhttp.AmzObjectLockLegalHold] = "OFF"
|
||||||
|
rec = r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/"+name+"?versionId="+targetID, "", headers)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("release %s: %d %s", name, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
got = r5Stored(t, obj, bucket, name, targetID, wantTags, wantStamp)
|
||||||
|
want = objectLockState{retentionTimestamp: "2026-09-03T10:30:00Z", legalHold: "OFF", legalHoldTimestamp: "2026-09-03T10:30:00Z"}
|
||||||
|
if state := storedObjectLockState(got.UserDefined); state != want {
|
||||||
|
t.Fatalf("%s release: got %+v want %+v", name, state, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReplicaReferenceCopyDestinationState(t *testing.T) {
|
||||||
|
z, bucket := consistencyPools(t)
|
||||||
|
for _, missing := range []bool{false, true} {
|
||||||
|
key := fmt.Sprintf("reference-copy-%t", missing)
|
||||||
|
id := mustGetUUID()
|
||||||
|
targetMeta := poolLockMetadata("GOVERNANCE", "ON", objectLockTestStamp1000, objectLockTestStamp1000)
|
||||||
|
targetMeta[xhttp.AmzObjectTagging] = "state=destination"
|
||||||
|
targetMeta[r5TagStamp] = objectLockTestStamp1000
|
||||||
|
if !missing {
|
||||||
|
putConsistencyObject(t, z, bucket, key, 0, "data", ObjectOptions{Versioned: true, VersionID: id, UserDefined: targetMeta})
|
||||||
|
}
|
||||||
|
sourceMeta := poolLockMetadata("", "OFF", objectLockTestStamp1100, objectLockTestStamp1100)
|
||||||
|
sourceMeta[xhttp.AmzObjectTagging] = "state=source"
|
||||||
|
sourceMeta[r5TagStamp] = objectLockTestStamp1100
|
||||||
|
src := putConsistencyObject(t, z, bucket, key, 0, "data", ObjectOptions{Versioned: true, UserDefined: sourceMeta})
|
||||||
|
src.metadataOnly = true
|
||||||
|
src.UserDefined = make(map[string]string)
|
||||||
|
_, err := z.CopyObject(t.Context(), bucket, key, bucket, key, src, ObjectOptions{VersionID: src.VersionID}, ObjectOptions{Versioned: true, VersionID: id, ReplicaLockReconcile: true, ReplicationRequest: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
want := storedObjectLockState(targetMeta)
|
||||||
|
wantTags, wantStamp := "state=destination", objectLockTestStamp1000
|
||||||
|
if missing {
|
||||||
|
want = objectLockState{}
|
||||||
|
wantTags = ""
|
||||||
|
wantStamp = ""
|
||||||
|
}
|
||||||
|
got := r5Stored(t, z, bucket, key, id, wantTags, wantStamp)
|
||||||
|
if state := storedObjectLockState(got.UserDefined); state != want {
|
||||||
|
t.Fatalf("missing=%t: got %+v want %+v", missing, state, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The set must not undo the all-pools decision with a newer weak revision
|
||||||
|
// from its own copy before the protected competing copy is retired.
|
||||||
|
func TestPoolsReplicaComplianceMetadataCopyOrder(t *testing.T) {
|
||||||
|
z, bucket := consistencyPools(t)
|
||||||
|
for _, protectedPool := range []int{0, 1} {
|
||||||
|
key := fmt.Sprintf("compliance-copy-%d", protectedPool)
|
||||||
|
id := mustGetUUID()
|
||||||
|
stamp := UTCNow().Add(-time.Hour)
|
||||||
|
var source ObjectInfo
|
||||||
|
for pool := range 2 {
|
||||||
|
meta := poolLockMetadata("", "OFF", objectLockTestStamp1100, objectLockTestStamp1000)
|
||||||
|
if pool == protectedPool {
|
||||||
|
meta = poolLockMetadata("COMPLIANCE", "OFF", objectLockTestStamp0900, objectLockTestStamp1000)
|
||||||
|
}
|
||||||
|
oi := putConsistencyObject(t, z, bucket, key, pool, "data", ObjectOptions{Versioned: true, VersionID: id, MTime: stamp, UserDefined: meta})
|
||||||
|
if pool == 0 {
|
||||||
|
source = oi
|
||||||
|
}
|
||||||
|
}
|
||||||
|
source.metadataOnly = true
|
||||||
|
source.UserDefined = map[string]string{}
|
||||||
|
_, err := z.CopyObject(t.Context(), bucket, key, bucket, key, source, ObjectOptions{VersionID: id}, ObjectOptions{VersionID: id, Versioned: true, ReplicaLockReconcile: true, ReplicationRequest: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := readObjectLockFields(t, z, bucket, key, id)
|
||||||
|
if got.mode != "COMPLIANCE" || got.retainUntil != "2030-01-01T00:00:00Z" {
|
||||||
|
t.Fatalf("protected pool %d lost COMPLIANCE: %+v", protectedPool, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPIReplicaRetentionTimestampAuthorization(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, _ auth.Credentials, t *testing.T) {
|
||||||
|
basic := `"s3:GetObject","s3:PutObject","s3:ReplicateObject"`
|
||||||
|
noRetention := newObjectAttributesAuthzUser(t, backend, bucket, basic)
|
||||||
|
withRetention := newObjectAttributesAuthzUser(t, backend, bucket, basic+`,"s3:PutObjectRetention"`)
|
||||||
|
for _, operation := range []string{"put", "copy", "multipart"} {
|
||||||
|
for _, authorized := range []bool{false, true} {
|
||||||
|
for _, timestamp := range []bool{false, true} {
|
||||||
|
key := fmt.Sprintf("replica-permission/%s/%t/%t", operation, authorized, timestamp)
|
||||||
|
meta := poolLockMetadata("GOVERNANCE", "", objectLockTestStamp0900, "")
|
||||||
|
id := putObjectLockVersion(t, obj, bucket, key, meta)
|
||||||
|
headers := map[string]string{xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA"}
|
||||||
|
if timestamp {
|
||||||
|
headers[xhttp.MinIOSourceObjectRetentionTimestamp] = objectLockTestStamp1000
|
||||||
|
}
|
||||||
|
method, path, body := http.MethodPut, "/"+bucket+"/"+key+"?versionId="+id, "data"
|
||||||
|
switch operation {
|
||||||
|
case "copy":
|
||||||
|
headers[xhttp.AmzCopySource] = "/" + bucket + "/" + key + "?versionId=" + id
|
||||||
|
headers[xhttp.AmzMetadataDirective] = replaceDirective
|
||||||
|
body = ""
|
||||||
|
case "multipart":
|
||||||
|
method = http.MethodPost
|
||||||
|
path += "&uploads"
|
||||||
|
body = ""
|
||||||
|
}
|
||||||
|
cred := noRetention
|
||||||
|
if authorized {
|
||||||
|
cred = withRetention
|
||||||
|
}
|
||||||
|
rec := r5Request(t, router, cred, method, path, body, headers)
|
||||||
|
want := http.StatusOK
|
||||||
|
if timestamp && !authorized {
|
||||||
|
want = http.StatusForbidden
|
||||||
|
}
|
||||||
|
if rec.Code != want {
|
||||||
|
t.Fatalf("%s: %d want %d: %s", key, rec.Code, want, rec.Body.String())
|
||||||
|
}
|
||||||
|
if want == http.StatusForbidden || !timestamp {
|
||||||
|
got := readObjectLockFields(t, obj, bucket, key, id)
|
||||||
|
if got.mode != "GOVERNANCE" || got.retentionStamp != objectLockTestStamp0900 {
|
||||||
|
t.Fatalf("unrequested/denied retention changed: %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPILocalObjectLockTimestampsAdvance(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
future := UTCNow().Add(48 * time.Hour)
|
||||||
|
meta := poolLockMetadata("GOVERNANCE", "ON", future.Format(time.RFC3339Nano), future.Format(time.RFC3339Nano))
|
||||||
|
id := putObjectLockVersion(t, obj, bucket, "local-lock", meta)
|
||||||
|
for n := 1; n <= 2; n++ {
|
||||||
|
if status := putRetention(t, router, cred, bucket, "local-lock", id, "GOVERNANCE", time.Date(2035, 1, n, 0, 0, 0, 0, time.UTC), false); status != http.StatusOK {
|
||||||
|
t.Fatalf("%s retention: %d", backend, status)
|
||||||
|
}
|
||||||
|
body := []byte(`<LegalHold xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>OFF</Status></LegalHold>`)
|
||||||
|
sum := md5.Sum(body)
|
||||||
|
rec := retentionAuthzRequest(t, router, cred, http.MethodPut, makeTestTargetURL("", bucket, "local-lock", url.Values{"legal-hold": {""}, "versionId": {id}}), body, map[string]string{xhttp.ContentMD5: base64.StdEncoding.EncodeToString(sum[:])})
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s hold: %d %s", backend, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
got := readObjectLockFields(t, obj, bucket, "local-lock", id)
|
||||||
|
want := future.Add(time.Duration(n) * time.Nanosecond).Format(time.RFC3339Nano)
|
||||||
|
if got.retentionStamp != want || got.legalHoldStamp != want {
|
||||||
|
t.Fatalf("local mutation %d regressed: %+v want %s", n, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
type lockBoundaryReadFailure struct {
|
||||||
|
ObjectLayer
|
||||||
|
bucket, key, version string
|
||||||
|
metadataError error
|
||||||
|
destinationError error
|
||||||
|
deleteCalls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *lockBoundaryReadFailure) GetObjectNInfo(ctx context.Context, bucket, key string, rs *HTTPRangeSpec, h http.Header, opts ObjectOptions) (*GetObjectReader, error) {
|
||||||
|
if f.metadataError != nil && bucket == minioMetaBucket && strings.HasPrefix(key, pathJoin(bucketMetaPrefix, f.bucket)+"/") {
|
||||||
|
return nil, f.metadataError
|
||||||
|
}
|
||||||
|
return f.ObjectLayer.GetObjectNInfo(ctx, bucket, key, rs, h, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *lockBoundaryReadFailure) GetObjectInfo(ctx context.Context, bucket, key string, opts ObjectOptions) (ObjectInfo, error) {
|
||||||
|
if f.destinationError != nil && bucket == f.bucket && key == f.key && opts.VersionID == f.version {
|
||||||
|
return ObjectInfo{}, f.destinationError
|
||||||
|
}
|
||||||
|
return f.ObjectLayer.GetObjectInfo(ctx, bucket, key, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *lockBoundaryReadFailure) DeleteObject(ctx context.Context, bucket, key string, opts ObjectOptions) (ObjectInfo, error) {
|
||||||
|
f.deleteCalls++
|
||||||
|
return f.ObjectLayer.DeleteObject(ctx, bucket, key, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPICopyReplicaDestinationReadFailure(t *testing.T) {
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true}, objAPITest: func(obj ObjectLayer, backend, bucket string, _ http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
id := putObjectLockVersion(t, obj, bucket, "destination", poolLockMetadata("GOVERNANCE", "ON", objectLockTestStamp1000, objectLockTestStamp1000))
|
||||||
|
source := putObjectLockVersion(t, obj, bucket, "source", nil)
|
||||||
|
f := &lockBoundaryReadFailure{ObjectLayer: obj, bucket: bucket, key: "destination", version: id, destinationError: errErasureReadQuorum}
|
||||||
|
router := initTestAPIEndPoints(f, []string{"CopyObject"})
|
||||||
|
headers := map[string]string{xhttp.AmzCopySource: "/" + bucket + "/source?versionId=" + source, xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA"}
|
||||||
|
rec := r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/destination?versionId="+id, "", headers)
|
||||||
|
if rec.Code < 500 {
|
||||||
|
t.Fatalf("%s read failure allowed copy: %d %s", backend, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if got := readObjectLockFields(t, obj, bucket, "destination", id); got.legalHold != "ON" || got.mode != "GOVERNANCE" {
|
||||||
|
t.Fatalf("failed read changed state: %+v", got)
|
||||||
|
}
|
||||||
|
}})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPIForceDeleteLockConfigFailure(t *testing.T) {
|
||||||
|
for _, locked := range []bool{false, true} {
|
||||||
|
t.Run(fmt.Sprintf("locked=%t", locked), func(t *testing.T) {
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: locked}, objAPITest: func(obj ObjectLayer, backend, bucket string, _ http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
saved, err := globalBucketMetadataSys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
restoreInit := markBucketMetadataInitialized(t)
|
||||||
|
defer restoreInit()
|
||||||
|
previous := newObjectLayerFn()
|
||||||
|
defer setObjectLayer(previous)
|
||||||
|
defer globalBucketMetadataSys.Set(bucket, saved)
|
||||||
|
for _, failure := range []string{"quorum", "unavailable", "cache-miss"} {
|
||||||
|
key := "force-" + failure + "/data"
|
||||||
|
putObjectLockVersion(t, obj, bucket, key, nil)
|
||||||
|
f := &lockBoundaryReadFailure{ObjectLayer: obj, bucket: bucket}
|
||||||
|
globalBucketMetadataSys.Remove(bucket)
|
||||||
|
switch failure {
|
||||||
|
case "quorum":
|
||||||
|
f.metadataError = errErasureReadQuorum
|
||||||
|
case "unavailable":
|
||||||
|
f.metadataError = errServerNotInitialized
|
||||||
|
}
|
||||||
|
router := initTestAPIEndPoints(f, []string{"DeleteObject"})
|
||||||
|
rec := r5Request(t, router, cred, http.MethodDelete, "/"+bucket+"/force-"+failure, "", map[string]string{xhttp.MinIOForceDelete: "true"})
|
||||||
|
setObjectLayer(previous)
|
||||||
|
globalBucketMetadataSys.Set(bucket, saved)
|
||||||
|
allowed := failure == "cache-miss" && !locked
|
||||||
|
if allowed {
|
||||||
|
if rec.Code != http.StatusNoContent || f.deleteCalls != 1 {
|
||||||
|
t.Fatalf("%s missing unlocked cache: %d calls=%d %s", backend, rec.Code, f.deleteCalls, rec.Body.String())
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if rec.Code < 400 || f.deleteCalls != 0 {
|
||||||
|
t.Fatalf("%s %s: force-delete proceeded: %d calls=%d %s", backend, failure, rec.Code, f.deleteCalls, rec.Body.String())
|
||||||
|
}
|
||||||
|
if _, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{}); err != nil {
|
||||||
|
t.Fatalf("rejected force-delete lost data: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPILocalCopyTagTimestampAdvances(t *testing.T) {
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
stamp := UTCNow().Add(48 * time.Hour)
|
||||||
|
meta := map[string]string{r5TagStamp: stamp.Format(time.RFC3339Nano), xhttp.AmzObjectTagging: "key=old"}
|
||||||
|
_, err := obj.PutObject(t.Context(), bucket, "local-copy", mustGetPutObjReader(t, strings.NewReader("data"), 4, "", ""), ObjectOptions{UserDefined: maps.Clone(meta)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := r5Request(t, router, cred, http.MethodPut, "/"+bucket+"/local-copy", "", map[string]string{xhttp.AmzCopySource: "/" + bucket + "/local-copy", xhttp.AmzMetadataDirective: replaceDirective, xhttp.AmzTagDirective: replaceDirective})
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s copy: %d %s", backend, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
r5Stored(t, obj, bucket, "local-copy", "", "", stamp.Add(time.Nanosecond).Format(time.RFC3339Nano))
|
||||||
|
}})
|
||||||
|
}
|
||||||
@@ -0,0 +1,205 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/crypto"
|
||||||
|
"github.com/minio/minio/internal/hash/sha256"
|
||||||
|
xioutil "github.com/minio/minio/internal/ioutil"
|
||||||
|
)
|
||||||
|
|
||||||
|
type replicaContentGuardFn func(context.Context, string, string, ObjectInfo, func() (*GetObjectReader, error)) error
|
||||||
|
|
||||||
|
// verifyLockedReplica compares the actual data, never a sender's ETag or
|
||||||
|
// checksum assertion. A peer may retransmit a protected version to update its
|
||||||
|
// metadata, but may not replace its content. Call under the object write lock.
|
||||||
|
func verifyLockedReplica(ctx context.Context, incoming, stored ObjectInfo, openIncoming, openStored func() (*GetObjectReader, error)) error {
|
||||||
|
if !enforceRetentionForDeletion(ctx, stored) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
locked := ObjectLocked{Bucket: stored.Bucket, Object: stored.Name, VersionID: stored.VersionID}
|
||||||
|
if storedObjectLockState(stored.UserDefined).complianceWouldWeaken(storedObjectLockState(incoming.UserDefined)) {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
inSSEC, oldSSEC := crypto.SSEC.IsEncrypted(incoming.UserDefined), crypto.SSEC.IsEncrypted(stored.UserDefined)
|
||||||
|
if inSSEC != oldSSEC {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
if inSSEC {
|
||||||
|
inSize, inErr := incoming.GetActualSize()
|
||||||
|
oldSize, oldErr := stored.GetActualSize()
|
||||||
|
if inErr != nil || oldErr != nil || inSize != oldSize {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
// Without the customer key, compare ciphertext and its interpretation.
|
||||||
|
// Equal ciphertext with a changed seal or multipart framing is not the
|
||||||
|
// same object, even if the supplied ETag and size are unchanged.
|
||||||
|
for _, key := range []string{
|
||||||
|
crypto.MetaIV, crypto.MetaAlgorithm, crypto.MetaSealedKeySSEC,
|
||||||
|
ReservedMetadataPrefix + "compression",
|
||||||
|
} {
|
||||||
|
if incoming.UserDefined[key] != stored.UserDefined[key] {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if crypto.IsMultiPart(incoming.UserDefined) != crypto.IsMultiPart(stored.UserDefined) || incoming.isMultipart() != stored.isMultipart() {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
if crypto.IsMultiPart(incoming.UserDefined) {
|
||||||
|
if len(incoming.Parts) != len(stored.Parts) {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
for i, p := range incoming.Parts {
|
||||||
|
old := stored.Parts[i]
|
||||||
|
if p.Number != old.Number || p.Size != old.Size {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
in, err := openIncoming()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer in.Close()
|
||||||
|
old, err := openStored()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer old.Close()
|
||||||
|
ih, oh := sha256.New(), sha256.New()
|
||||||
|
inSize, err := io.Copy(ih, in)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
oldSize, err := io.Copy(oh, old)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if inSize != oldSize || !bytes.Equal(ih.Sum(nil), oh.Sum(nil)) {
|
||||||
|
return locked
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (er erasureObjects) guardReplicaContent(ctx context.Context, bucket, object string, incoming ObjectInfo, openIncoming func() (*GetObjectReader, error)) error {
|
||||||
|
opts := ObjectOptions{VersionID: incoming.VersionID, NoLock: true, NoAuditLog: true, ReplicationRequest: true}
|
||||||
|
if opts.VersionID == "" {
|
||||||
|
opts.VersionID = nullVersionID
|
||||||
|
}
|
||||||
|
stored, err := er.getObjectInfo(ctx, bucket, object, opts)
|
||||||
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) || isErrMethodNotAllowed(err) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return verifyLockedReplica(ctx, incoming, stored, openIncoming, func() (*GetObjectReader, error) {
|
||||||
|
return er.GetObjectNInfo(ctx, bucket, object, nil, http.Header{}, opts)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (z *erasureServerPools) guardReplicaContent(ctx context.Context, bucket, object string, incoming ObjectInfo, openIncoming func() (*GetObjectReader, error)) error {
|
||||||
|
opts := ObjectOptions{VersionID: incoming.VersionID, NoLock: true, NoAuditLog: true, ReplicationRequest: true}
|
||||||
|
if opts.VersionID == "" {
|
||||||
|
opts.VersionID = nullVersionID
|
||||||
|
}
|
||||||
|
copies, err := z.objectPoolInfos(ctx, bucket, object, opts)
|
||||||
|
if isErrObjectNotFound(err) || isErrVersionNotFound(err) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, copy := range copies {
|
||||||
|
if err := verifyLockedReplica(ctx, incoming, copy.ObjInfo, openIncoming, func() (*GetObjectReader, error) {
|
||||||
|
return z.serverPools[copy.Index].GetObjectNInfo(ctx, bucket, object, nil, http.Header{}, opts)
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkReplicaContent reads the staged version before rename commits it. Its
|
||||||
|
// data is decoded/decrypted just like a GET (raw ciphertext for SSE-C), so this
|
||||||
|
// covers multipart, compression and destination SSE-S3/KMS re-encryption.
|
||||||
|
// disks and metadata are in erasure shard order, as passed to renameData.
|
||||||
|
func (er erasureObjects) checkReplicaContent(ctx context.Context, bucket, object, stageBucket, stageObject string, metadata []FileInfo, disks []StorageAPI, opts ObjectOptions) error {
|
||||||
|
if (!opts.ReplicaLockReconcile && !opts.ReplicationRequest) || opts.DataMovement {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var fi FileInfo
|
||||||
|
for i, disk := range disks {
|
||||||
|
if disk != nil {
|
||||||
|
fi = metadata[i]
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
incoming := fi.ToObjectInfo(bucket, object, opts.Versioned || opts.VersionSuspended)
|
||||||
|
guard := opts.replicaContentGuard
|
||||||
|
if guard == nil {
|
||||||
|
guard = er.guardReplicaContent
|
||||||
|
}
|
||||||
|
return guard(ctx, bucket, object, incoming, func() (*GetObjectReader, error) {
|
||||||
|
// A marker-only peer write does not reconcile the stored protection.
|
||||||
|
// It therefore cannot replace a protected version, even with equal data.
|
||||||
|
if !opts.ReplicaLockReconcile {
|
||||||
|
return nil, ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: incoming.VersionID}
|
||||||
|
}
|
||||||
|
if incoming.IsCompressed() {
|
||||||
|
// A replica may assert the source's logical size. A compressed GET
|
||||||
|
// limits its output to that size, so verify it against the lengths
|
||||||
|
// recorded while consuming the uploaded parts before comparing GETs.
|
||||||
|
// Otherwise a forged zero size could hide a non-empty replacement.
|
||||||
|
var actual int64
|
||||||
|
for _, part := range incoming.Parts {
|
||||||
|
actual += part.ActualSize
|
||||||
|
}
|
||||||
|
declared, err := incoming.GetActualSize()
|
||||||
|
if err != nil || declared != actual {
|
||||||
|
return nil, ObjectLocked{Bucket: bucket, Object: decodeDirObject(object), VersionID: incoming.VersionID}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
readOpts := ObjectOptions{NoDecryption: crypto.SSEC.IsEncrypted(incoming.UserDefined)}
|
||||||
|
fn, off, length, err := NewGetObjectReader(nil, incoming, readOpts, http.Header{})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// The ordinary reader expects original drive order, not shard order.
|
||||||
|
readDisks, readMeta := make([]StorageAPI, len(disks)), make([]FileInfo, len(metadata))
|
||||||
|
for i, shard := range fi.Erasure.Distribution {
|
||||||
|
readDisks[i], readMeta[i] = disks[shard-1], metadata[shard-1]
|
||||||
|
readMeta[i].Erasure.Index = shard
|
||||||
|
}
|
||||||
|
pr, pw := xioutil.WaitPipe()
|
||||||
|
go func() {
|
||||||
|
pw.CloseWithError(er.getObjectWithFileInfo(ctx, stageBucket, stageObject, off, length, pw, fi, readMeta, readDisks))
|
||||||
|
}()
|
||||||
|
reader, err := fn(pr, http.Header{}, func() { pr.CloseWithError(nil) })
|
||||||
|
if err != nil {
|
||||||
|
pr.CloseWithError(err)
|
||||||
|
}
|
||||||
|
return reader, err
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -0,0 +1,314 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/xml"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
"github.com/minio/minio/internal/crypto"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
"github.com/minio/minio/internal/kms"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAPIReplicaPreservesProtectedContent(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
oldTLS, oldKMS := globalIsTLS, GlobalKMS
|
||||||
|
globalIsTLS, GlobalKMS = true, kms.NewStub("replica-content")
|
||||||
|
defer func() { globalIsTLS, GlobalKMS = oldTLS, oldKMS }()
|
||||||
|
restoreCompression := disableCompression()
|
||||||
|
defer restoreCompression()
|
||||||
|
request := func(method, url string, data []byte, headers map[string]string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
req, err := newTestSignedRequestV4(method, url, int64(len(data)), bytes.NewReader(data), cred.AccessKey, cred.SecretKey, headers)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
for _, encryption := range []string{"plain", "sse-s3", "sse-kms", "sse-c", "plain-compressed", "sse-s3-compressed"} {
|
||||||
|
restore := func() {}
|
||||||
|
if strings.HasSuffix(encryption, "-compressed") {
|
||||||
|
restore = setCopyChecksumCompression(true)
|
||||||
|
}
|
||||||
|
for _, multipart := range []bool{false, true} {
|
||||||
|
for _, protection := range []string{"COMPLIANCE", "LEGALHOLD", "GOVERNANCE"} {
|
||||||
|
key := fmt.Sprintf("protected/%s/%v/%s.txt", encryption, multipart, protection)
|
||||||
|
enc := map[string]string{}
|
||||||
|
switch encryption {
|
||||||
|
case "sse-s3", "sse-s3-compressed":
|
||||||
|
enc[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionAES
|
||||||
|
case "sse-kms":
|
||||||
|
enc[xhttp.AmzServerSideEncryption] = xhttp.AmzEncryptionKMS
|
||||||
|
enc[xhttp.AmzServerSideEncryptionKmsID] = "replica-content"
|
||||||
|
case "sse-c":
|
||||||
|
enc = ssecKeyHeaders(bytes.Repeat([]byte{0x37}, 32), false)
|
||||||
|
}
|
||||||
|
headers := maps.Clone(enc)
|
||||||
|
maps.Copy(headers, batchLockMeta(protection))
|
||||||
|
// Both inline and disk-backed writes are exercised.
|
||||||
|
body := bytes.Repeat([]byte("immutable content "), 20000)
|
||||||
|
switch protection {
|
||||||
|
case "LEGALHOLD":
|
||||||
|
body = body[:64]
|
||||||
|
case "GOVERNANCE":
|
||||||
|
body = nil
|
||||||
|
}
|
||||||
|
if !multipart {
|
||||||
|
putCopyChecksumSource(t, router, cred, bucket, key, body, headers)
|
||||||
|
} else {
|
||||||
|
initRec := request(http.MethodPost, getNewMultipartURL("", bucket, key), nil, headers)
|
||||||
|
if initRec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("source initiate: %d %s", initRec.Code, initRec.Body.String())
|
||||||
|
}
|
||||||
|
var init InitiateMultipartUploadResponse
|
||||||
|
if err := xml.Unmarshal(initRec.Body.Bytes(), &init); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
part := request(http.MethodPut, getPutObjectPartURL("", bucket, key, init.UploadID, "1"), body, enc)
|
||||||
|
if part.Code != http.StatusOK {
|
||||||
|
t.Fatalf("source part: %d %s", part.Code, part.Body.String())
|
||||||
|
}
|
||||||
|
complete, err := xml.Marshal(CompleteMultipartUpload{Parts: []CompletePart{{PartNumber: 1, ETag: canonicalizeETag(part.Header()[xhttp.ETag][0])}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := request(http.MethodPost, getCompleteMultipartUploadURL("", bucket, key, init.UploadID), complete, enc)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("source complete: %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
gr, err := obj.GetObjectNInfo(t.Context(), bucket, key, nil, http.Header{}, ObjectOptions{ReplicationRequest: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
src := gr.ObjInfo
|
||||||
|
wire, err := io.ReadAll(gr)
|
||||||
|
gr.Close()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, altered := range []bool{true, false} {
|
||||||
|
data := bytes.Clone(wire)
|
||||||
|
if altered {
|
||||||
|
if len(data) == 0 {
|
||||||
|
data = []byte("x")
|
||||||
|
} else {
|
||||||
|
data[len(data)-1] ^= 0x20
|
||||||
|
}
|
||||||
|
}
|
||||||
|
opts, _, err := putReplicationOpts(t.Context(), "", src)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
hdrs := map[string]string{}
|
||||||
|
for k, vs := range opts.Header() {
|
||||||
|
if len(vs) > 0 {
|
||||||
|
hdrs[k] = vs[0]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hdrs[xhttp.MinIOSourceReplicationRequest] = "true"
|
||||||
|
hdrs[xhttp.AmzBucketReplicationStatus] = "REPLICA"
|
||||||
|
// Forge the old ETag and size along with a newer tag revision:
|
||||||
|
// the body comparison must still reject different bytes.
|
||||||
|
hdrs[xhttp.MinIOSourceETag] = src.ETag
|
||||||
|
hdrs[xhttp.MinIOSourceTaggingTimestamp] = UTCNow().Add(time.Minute).Format(time.RFC3339Nano)
|
||||||
|
hdrs[xhttp.AmzObjectTagging] = "replica=updated"
|
||||||
|
var rec *httptest.ResponseRecorder
|
||||||
|
if !multipart {
|
||||||
|
rec = request(http.MethodPut, getPutObjectURL("", bucket, key)+"?versionId="+src.VersionID, data, hdrs)
|
||||||
|
} else {
|
||||||
|
initRec := request(http.MethodPost, getNewMultipartURL("", bucket, key)+"&versionId="+src.VersionID, nil, hdrs)
|
||||||
|
if initRec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s %s initiate: %d %s", backend, key, initRec.Code, initRec.Body.String())
|
||||||
|
}
|
||||||
|
var init InitiateMultipartUploadResponse
|
||||||
|
if err := xml.Unmarshal(initRec.Body.Bytes(), &init); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
partHdrs := map[string]string{xhttp.MinIOSourceReplicationRequest: "true"}
|
||||||
|
partRec := request(http.MethodPut, getPutObjectPartURL("", bucket, key, init.UploadID, "1"), data, partHdrs)
|
||||||
|
if partRec.Code != http.StatusOK && (!altered || !strings.Contains(partRec.Body.String(), "XMinioObjectTampered")) {
|
||||||
|
t.Fatalf("%s %s part: %d %s", backend, key, partRec.Code, partRec.Body.String())
|
||||||
|
}
|
||||||
|
if partRec.Code != http.StatusOK {
|
||||||
|
rec = partRec
|
||||||
|
} else {
|
||||||
|
complete, err := xml.Marshal(CompleteMultipartUpload{Parts: []CompletePart{{PartNumber: 1, ETag: canonicalizeETag(partRec.Header()[xhttp.ETag][0])}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
actual, err := src.GetActualSize()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
partHdrs[xhttp.MinIOReplicationActualObjectSize] = strconv.FormatInt(actual, 10)
|
||||||
|
rec = request(http.MethodPost, getCompleteMultipartUploadURL("", bucket, key, init.UploadID), complete, partHdrs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if altered {
|
||||||
|
if rec.Code < 400 || !strings.Contains(rec.Body.String(), "WORM") && !strings.Contains(rec.Body.String(), "XMinioObjectTampered") {
|
||||||
|
t.Fatalf("%s %s: different content not refused: %d %s", backend, key, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
} else if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s %s: same content refused: %d %s", backend, key, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
getHdrs := map[string]string{}
|
||||||
|
if encryption == "sse-c" {
|
||||||
|
getHdrs = enc
|
||||||
|
}
|
||||||
|
got := request(http.MethodGet, getGetObjectURL("", bucket, key)+"?versionId="+src.VersionID, nil, getHdrs)
|
||||||
|
if got.Code != http.StatusOK || !bytes.Equal(got.Body.Bytes(), body) {
|
||||||
|
t.Fatalf("%s %s: protected content changed or became unreadable: %d", backend, key, got.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
restore()
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPoolsReplicaProtectedContentAcrossPools(t *testing.T) {
|
||||||
|
z, bucket := consistencyPools(t)
|
||||||
|
const key = "protected-across-pools"
|
||||||
|
id := mustGetUUID()
|
||||||
|
stamp := UTCNow().Add(-time.Hour)
|
||||||
|
putConsistencyObject(t, z, bucket, key, 0, "original", ObjectOptions{Versioned: true, VersionID: id, MTime: stamp})
|
||||||
|
putConsistencyObject(t, z, bucket, key, 1, "original", ObjectOptions{Versioned: true, VersionID: id, MTime: stamp, UserDefined: batchLockMeta("COMPLIANCE")})
|
||||||
|
_, err := z.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, strings.NewReader("tampered"), 8, "", ""), ObjectOptions{Versioned: true, VersionID: id, ReplicationRequest: true, ReplicaLockReconcile: true, UserDefined: map[string]string{}})
|
||||||
|
if _, ok := err.(ObjectLocked); !ok {
|
||||||
|
t.Fatalf("protected pool copy allowed overwrite: %v", err)
|
||||||
|
}
|
||||||
|
for i, pool := range z.serverPools {
|
||||||
|
gr, err := pool.GetObjectNInfo(t.Context(), bucket, key, nil, http.Header{}, ObjectOptions{VersionID: id})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
data, err := io.ReadAll(gr)
|
||||||
|
gr.Close()
|
||||||
|
if err != nil || string(data) != "original" {
|
||||||
|
t.Fatalf("pool %d: changed protected data: %q %v", i, data, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLockedSSECReplicaRejectsChangedSeal(t *testing.T) {
|
||||||
|
stored := ObjectInfo{Bucket: "bucket", Name: "key", Size: 0, UserDefined: batchLockMeta("LEGALHOLD")}
|
||||||
|
stored.UserDefined[crypto.MetaSealedKeySSEC] = "stored-seal"
|
||||||
|
incoming := stored
|
||||||
|
incoming.UserDefined = maps.Clone(stored.UserDefined)
|
||||||
|
incoming.UserDefined[crypto.MetaSealedKeySSEC] = "changed-seal"
|
||||||
|
open := func() (*GetObjectReader, error) {
|
||||||
|
t.Fatal("changed seal must be rejected before reading")
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if err := verifyLockedReplica(t.Context(), incoming, stored, open, open); err == nil {
|
||||||
|
t.Fatal("changed SSE-C seal accepted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPIReplicaCopyPreservesProtectedVersion(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
const key = "copy-protected-version"
|
||||||
|
original := batchPutVersion(t, obj, bucket, key, batchLockMeta("COMPLIANCE"))
|
||||||
|
data := []byte("different source version")
|
||||||
|
source, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(data), int64(len(data)), "", ""), ObjectOptions{Versioned: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
headers := map[string]string{
|
||||||
|
xhttp.AmzCopySource: url.QueryEscape("/"+bucket+"/"+key) + "?versionId=" + source.VersionID,
|
||||||
|
xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA",
|
||||||
|
xhttp.AmzMetadataDirective: replaceDirective,
|
||||||
|
}
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodPut, getCopyObjectURL("", bucket, key)+"?versionId="+original.VersionID, 0, nil, cred.AccessKey, cred.SecretKey, headers)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
if rec.Code < 400 || !strings.Contains(rec.Body.String(), "WORM") {
|
||||||
|
t.Fatalf("%s: copy replaced protected version: %d %s", backend, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
gr, err := obj.GetObjectNInfo(t.Context(), bucket, key, nil, http.Header{}, ObjectOptions{VersionID: original.VersionID})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got, err := io.ReadAll(gr)
|
||||||
|
gr.Close()
|
||||||
|
if err != nil || string(got) != "batch lock "+key {
|
||||||
|
t.Fatalf("%s: protected body changed: %q %v", backend, got, err)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPIReplicaCannotWeakenCompliance(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
const key = "compliance-removal"
|
||||||
|
oi := batchPutVersion(t, obj, bucket, key, batchLockMeta("COMPLIANCE"))
|
||||||
|
until := oi.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)]
|
||||||
|
for _, copyRequest := range []bool{false, true} {
|
||||||
|
headers := map[string]string{xhttp.MinIOSourceReplicationRequest: "true", xhttp.AmzBucketReplicationStatus: "REPLICA", xhttp.MinIOSourceObjectRetentionTimestamp: UTCNow().Add(time.Hour).Format(time.RFC3339Nano)}
|
||||||
|
body := []byte("batch lock " + key)
|
||||||
|
if copyRequest {
|
||||||
|
headers[xhttp.AmzCopySource] = url.QueryEscape("/"+bucket+"/"+key) + "?versionId=" + oi.VersionID
|
||||||
|
headers[xhttp.AmzMetadataDirective] = replaceDirective
|
||||||
|
body = nil
|
||||||
|
}
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucket, key)+"?versionId="+oi.VersionID, int64(len(body)), bytes.NewReader(body), cred.AccessKey, cred.SecretKey, headers)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s copy=%v: %d %s", backend, copyRequest, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
stored, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: oi.VersionID})
|
||||||
|
if err != nil || stored.UserDefined[strings.ToLower(xhttp.AmzObjectLockMode)] != "COMPLIANCE" || stored.UserDefined[strings.ToLower(xhttp.AmzObjectLockRetainUntilDate)] != until {
|
||||||
|
t.Fatalf("%s copy=%v: a replica cleared COMPLIANCE: %+v %v", backend, copyRequest, stored.UserDefined, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestReplicationControlsRequireSignedHeaders(t *testing.T) {
|
||||||
|
for _, header := range replicationRequestHeaders {
|
||||||
|
t.Run(header, func(t *testing.T) {
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodGet, "http://localhost/bucket/key", 0, nil, "access-key", "secret-key", nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Membership matters even for a present-empty internal header.
|
||||||
|
req.Header.Set(header, "")
|
||||||
|
if replicationHeadersSigned(req) {
|
||||||
|
t.Fatal("unsigned control granted replication trust")
|
||||||
|
}
|
||||||
|
if err := signRequestV4(req, "access-key", "secret-key"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !replicationHeadersSigned(req) {
|
||||||
|
t.Fatal("signed control rejected")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
for _, authorization := range []string{"", "AWS access-key:signature", "Credential=access-key"} {
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/bucket/key", nil)
|
||||||
|
req.Header.Set(xhttp.Authorization, authorization)
|
||||||
|
req.Header.Set(xhttp.MinIOSourceReplicationRequest, "true")
|
||||||
|
if replicationHeadersSigned(req) {
|
||||||
|
t.Fatalf("non-SigV4 authorization granted trust: %q", authorization)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "http://localhost/bucket/key", nil)
|
||||||
|
req.Header.Set(xhttp.MinIOSourceReplicationRequest, "true")
|
||||||
|
presignBoundaryRequest(t, req, UTCNow(), []string{"host", "x-minio-source-replication-request"}, auth.Credentials{AccessKey: "access-key", SecretKey: "secret-key"})
|
||||||
|
getRequestAuthType(req)
|
||||||
|
if replicationHeadersSigned(req) {
|
||||||
|
t.Fatal("presigned request granted peer trust")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPIUnsignedReplicationControlsAreIgnored(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, endpoints: []string{"PutObject"},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
for _, signedMarker := range []bool{false, true} {
|
||||||
|
headers := map[string]string{}
|
||||||
|
if signedMarker {
|
||||||
|
headers[xhttp.MinIOSourceReplicationRequest] = "true"
|
||||||
|
}
|
||||||
|
body := []byte("replication controls are authenticated")
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucket, "key"), int64(len(body)), bytes.NewReader(body), cred.AccessKey, cred.SecretKey, headers)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
req.Header.Set(xhttp.MinIOSourceReplicationRequest, "true")
|
||||||
|
req.Header.Set(xhttp.MinIOSourceMTime, "2100-01-01T00:00:00Z")
|
||||||
|
req.Header.Set(xhttp.MinIOSourceETag, "forged-source-etag")
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
oi, err := obj.GetObjectInfo(t.Context(), bucket, "key", ObjectOptions{})
|
||||||
|
if rec.Code != http.StatusOK || err != nil || oi.ModTime.Year() == 2100 || oi.ETag == "forged-source-etag" {
|
||||||
|
t.Fatalf("%s signedMarker=%v: unsigned replication state was accepted: HTTP=%d info=%+v err=%v", backend, signedMarker, rec.Code, oi, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -1850,6 +1850,12 @@ func testReplicaLockReconcileNullVersion(obj ObjectLayer, instanceType, bucketNa
|
|||||||
_ http.Handler, _ auth.Credentials, t *testing.T,
|
_ http.Handler, _ auth.Credentials, t *testing.T,
|
||||||
) {
|
) {
|
||||||
ctx := t.Context()
|
ctx := t.Context()
|
||||||
|
// A protected retransmit must preserve both the source ciphertext and
|
||||||
|
// its seal. Generate it once for the stored null version and its retry.
|
||||||
|
var ciphertext bytes.Buffer
|
||||||
|
if _, err := sio.Encrypt(&ciphertext, bytes.NewReader([]byte("data")), sio.Config{Key: bytes.Repeat([]byte{1}, 32)}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
// completeNullMPU runs an SSE-C replica multipart upload addressing the null
|
// completeNullMPU runs an SSE-C replica multipart upload addressing the null
|
||||||
// version (VersionSuspended) carrying uploadLock, through the reconcile.
|
// version (VersionSuspended) carrying uploadLock, through the reconcile.
|
||||||
@@ -1861,13 +1867,6 @@ func testReplicaLockReconcileNullVersion(obj ObjectLayer, instanceType, bucketNa
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// PutObjectPart now validates the DARE stream length (#119). This
|
|
||||||
// object-layer fixture needs an encrypted body before it can exercise
|
|
||||||
// the completion-time null-version metadata reconciliation.
|
|
||||||
var ciphertext bytes.Buffer
|
|
||||||
if _, err := sio.Encrypt(&ciphertext, bytes.NewReader([]byte("data")), sio.Config{Key: bytes.Repeat([]byte{1}, 32)}); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
part, err := obj.PutObjectPart(ctx, bucketName, object, res.UploadID, 1,
|
part, err := obj.PutObjectPart(ctx, bucketName, object, res.UploadID, 1,
|
||||||
mustGetPutObjReader(t, bytes.NewReader(ciphertext.Bytes()), int64(ciphertext.Len()), "", ""), ObjectOptions{})
|
mustGetPutObjReader(t, bytes.NewReader(ciphertext.Bytes()), int64(ciphertext.Len()), "", ""), ObjectOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1884,8 +1883,9 @@ func testReplicaLockReconcileNullVersion(obj ObjectLayer, instanceType, bucketNa
|
|||||||
object := "reconcile/null-vs-uuid"
|
object := "reconcile/null-vs-uuid"
|
||||||
// The null version holds a newer legal hold ON@10:00.
|
// The null version holds a newer legal hold ON@10:00.
|
||||||
if _, err := obj.PutObject(ctx, bucketName, object,
|
if _, err := obj.PutObject(ctx, bucketName, object,
|
||||||
mustGetPutObjReader(t, bytes.NewReader([]byte("data")), 4, "", ""),
|
mustGetPutObjReader(t, bytes.NewReader(ciphertext.Bytes()), int64(ciphertext.Len()), "", ""),
|
||||||
ObjectOptions{VersionSuspended: true, MTime: time.Date(2026, 9, 3, 10, 0, 0, 0, time.UTC), UserDefined: map[string]string{
|
ObjectOptions{VersionSuspended: true, MTime: time.Date(2026, 9, 3, 10, 0, 0, 0, time.UTC), UserDefined: map[string]string{
|
||||||
|
crypto.MetaSealedKeySSEC: "dummy-sealed-key",
|
||||||
strings.ToLower(xhttp.AmzObjectLockLegalHold): "ON",
|
strings.ToLower(xhttp.AmzObjectLockLegalHold): "ON",
|
||||||
ReservedMetadataPrefixLower + ObjectLockLegalHoldTimestamp: objectLockTestStamp1000,
|
ReservedMetadataPrefixLower + ObjectLockLegalHoldTimestamp: objectLockTestStamp1000,
|
||||||
}}); err != nil {
|
}}); err != nil {
|
||||||
|
|||||||
@@ -13,6 +13,8 @@ package cmd
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
|
||||||
objectreplication "github.com/minio/minio/internal/bucket/replication"
|
objectreplication "github.com/minio/minio/internal/bucket/replication"
|
||||||
xhttp "github.com/minio/minio/internal/http"
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
@@ -62,7 +64,7 @@ func isReplicaTrusted(ctx context.Context) bool {
|
|||||||
// grant must not turn client-controlled internal headers into trusted state.
|
// grant must not turn client-controlled internal headers into trusted state.
|
||||||
func replicationPermissionAllowed(ctx context.Context, r *http.Request, bucket, object string, action policy.Action) bool {
|
func replicationPermissionAllowed(ctx context.Context, r *http.Request, bucket, object string, action policy.Action) bool {
|
||||||
reqInfo := logger.GetReqInfo(ctx)
|
reqInfo := logger.GetReqInfo(ctx)
|
||||||
if reqInfo == nil || reqInfo.Cred.AccessKey == "" {
|
if reqInfo == nil || reqInfo.Cred.AccessKey == "" || !replicationHeadersSigned(r) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
reqInfo.BucketName = bucket
|
reqInfo.BucketName = bucket
|
||||||
@@ -70,6 +72,25 @@ func replicationPermissionAllowed(ctx context.Context, r *http.Request, bucket,
|
|||||||
return authorizeRequest(ctx, r, action) == ErrNone
|
return authorizeRequest(ctx, r, action) == ErrNone
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// replicationHeadersSigned binds every internal control to the authenticated
|
||||||
|
// header signature. Query grants and SigV2 requests cannot confer peer trust.
|
||||||
|
// Call only after the request's signature has been verified.
|
||||||
|
func replicationHeadersSigned(r *http.Request) bool {
|
||||||
|
if !isRequestSignatureV4(r) || isRequestPresignedSignatureV4(r) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
sig, errCode := parseSignV4(r.Header.Get(xhttp.Authorization), globalSite.Region(), serviceS3)
|
||||||
|
if errCode != ErrNone {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, name := range replicationRequestHeaders {
|
||||||
|
if _, present := r.Header[http.CanonicalHeaderKey(name)]; present && !slices.Contains(sig.SignedHeaders, strings.ToLower(name)) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
// evaluateReplicationTrust decides whether a request may carry replication
|
// evaluateReplicationTrust decides whether a request may carry replication
|
||||||
// semantics for the given action. A request that declares itself a replica
|
// semantics for the given action. A request that declares itself a replica
|
||||||
// without holding the replication permission is rejected. trusted reports that
|
// without holding the replication permission is rejected. trusted reports that
|
||||||
|
|||||||
+14
-20
@@ -144,13 +144,7 @@ func (api objectAPIHandlers) getObjectInArchiveFileHandler(ctx context.Context,
|
|||||||
|
|
||||||
zipInfo := zipObjInfo.ArchiveInfo(r.Header)
|
zipInfo := zipObjInfo.ArchiveInfo(r.Header)
|
||||||
if len(zipInfo) == 0 {
|
if len(zipInfo) == 0 {
|
||||||
opts.EncryptFn, err = zipObjInfo.metadataEncryptFn(r.Header)
|
zipInfo, _, err = readObjectZIPInfo(ctx, objectAPI, bucket, zipPath, opts)
|
||||||
if err != nil {
|
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
zipInfo, err = updateObjectMetadataWithZipInfo(ctx, objectAPI, bucket, zipPath, opts)
|
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
|
||||||
@@ -245,8 +239,7 @@ func listObjectsV2InArchive(ctx context.Context, objectAPI ObjectLayer, bucket,
|
|||||||
|
|
||||||
zipInfo := zipObjInfo.ArchiveInfo(h)
|
zipInfo := zipObjInfo.ArchiveInfo(h)
|
||||||
if len(zipInfo) == 0 {
|
if len(zipInfo) == 0 {
|
||||||
// Always update the latest version
|
zipInfo, _, err = readObjectZIPInfo(ctx, objectAPI, bucket, zipPath, ObjectOptions{})
|
||||||
zipInfo, err = updateObjectMetadataWithZipInfo(ctx, objectAPI, bucket, zipPath, ObjectOptions{})
|
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ListObjectsV2Info{}, err
|
return ListObjectsV2Info{}, err
|
||||||
@@ -437,12 +430,7 @@ func (api objectAPIHandlers) headObjectInArchiveFileHandler(ctx context.Context,
|
|||||||
|
|
||||||
zipInfo := zipObjInfo.ArchiveInfo(r.Header)
|
zipInfo := zipObjInfo.ArchiveInfo(r.Header)
|
||||||
if len(zipInfo) == 0 {
|
if len(zipInfo) == 0 {
|
||||||
opts.EncryptFn, err = zipObjInfo.metadataEncryptFn(r.Header)
|
zipInfo, _, err = readObjectZIPInfo(ctx, objectAPI, bucket, zipPath, opts)
|
||||||
if err != nil {
|
|
||||||
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
zipInfo, err = updateObjectMetadataWithZipInfo(ctx, objectAPI, bucket, zipPath, opts)
|
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
|
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
|
||||||
@@ -482,15 +470,22 @@ func (api objectAPIHandlers) headObjectInArchiveFileHandler(ctx context.Context,
|
|||||||
w.WriteHeader(http.StatusOK)
|
w.WriteHeader(http.StatusOK)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update the passed zip object metadata with the zip contents info, file name, modtime, size, etc.
|
// readObjectZIPInfo builds an index without modifying the object. A GET, HEAD
|
||||||
// The returned zip index will de decrypted.
|
// or archive listing must remain read-only even when no index was stored.
|
||||||
func updateObjectMetadataWithZipInfo(ctx context.Context, objectAPI ObjectLayer, bucket, object string, opts ObjectOptions) ([]byte, error) {
|
func readObjectZIPInfo(ctx context.Context, objectAPI ObjectLayer, bucket, object string, opts ObjectOptions) ([]byte, ObjectInfo, error) {
|
||||||
files, srcInfo, err := getFilesListFromZIPObject(ctx, objectAPI, bucket, object, opts)
|
files, srcInfo, err := getFilesListFromZIPObject(ctx, objectAPI, bucket, object, opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, srcInfo, err
|
||||||
}
|
}
|
||||||
files.OptimizeSize()
|
files.OptimizeSize()
|
||||||
zipInfo, err := files.Serialize()
|
zipInfo, err := files.Serialize()
|
||||||
|
return zipInfo, srcInfo, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// updateObjectMetadataWithZipInfo persists an index only for upload operations
|
||||||
|
// that explicitly request extraction. The returned index is decrypted.
|
||||||
|
func updateObjectMetadataWithZipInfo(ctx context.Context, objectAPI ObjectLayer, bucket, object string, opts ObjectOptions) ([]byte, error) {
|
||||||
|
zipInfo, srcInfo, err := readObjectZIPInfo(ctx, objectAPI, bucket, object, opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -500,7 +495,6 @@ func updateObjectMetadataWithZipInfo(ctx context.Context, objectAPI ObjectLayer,
|
|||||||
at = archiveTypeEnc
|
at = archiveTypeEnc
|
||||||
zipInfoStr = string(opts.EncryptFn(archiveTypeEnc, zipInfo))
|
zipInfoStr = string(opts.EncryptFn(archiveTypeEnc, zipInfo))
|
||||||
}
|
}
|
||||||
srcInfo.UserDefined[archiveTypeMetadataKey] = at
|
|
||||||
popts := ObjectOptions{
|
popts := ObjectOptions{
|
||||||
MTime: srcInfo.ModTime,
|
MTime: srcInfo.ModTime,
|
||||||
VersionID: srcInfo.VersionID,
|
VersionID: srcInfo.VersionID,
|
||||||
|
|||||||
@@ -0,0 +1,109 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/zip"
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAPIArchiveReadsDoNotPersistMetadata(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, makeBucketOptions: MakeBucketOptions{LockEnabled: true},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
var archive bytes.Buffer
|
||||||
|
zw := zip.NewWriter(&archive)
|
||||||
|
file, err := zw.Create("entry.txt")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err = file.Write([]byte("archive content")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err = zw.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
reader := newObjectAttributesAuthzUser(t, backend, bucket, `"s3:GetObject","s3:ListBucket"`)
|
||||||
|
policy := fmt.Appendf(nil, `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":["s3:GetObject","s3:ListBucket"],"Resource":["arn:aws:s3:::%s","arn:aws:s3:::%s/*"]}]}`, bucket, bucket)
|
||||||
|
if _, err = globalBucketMetadataSys.Update(t.Context(), bucket, bucketPolicyConfig, policy); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, anonymous := range []bool{false, true} {
|
||||||
|
for _, method := range []string{http.MethodGet, http.MethodHead, "LIST"} {
|
||||||
|
key := fmt.Sprintf("archive-%v-%s.zip", anonymous, method)
|
||||||
|
before, err := obj.PutObject(t.Context(), bucket, key, mustGetPutObjReader(t, bytes.NewReader(archive.Bytes()), int64(archive.Len()), "", ""), ObjectOptions{Versioned: true, UserDefined: batchLockMeta("COMPLIANCE")})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
before, err = obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: before.VersionID})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
url := getGetObjectURL("", bucket, key+"/entry.txt")
|
||||||
|
verb := method
|
||||||
|
if method == "LIST" {
|
||||||
|
verb = http.MethodGet
|
||||||
|
url = "/" + bucket + "?list-type=2&prefix=" + key + "/"
|
||||||
|
}
|
||||||
|
req, err := newTestSignedRequestV4(verb, url, 0, nil, reader.AccessKey, reader.SecretKey, map[string]string{xMinIOExtract: "true"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if anonymous {
|
||||||
|
req.Header.Del("Authorization")
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("%s anonymous=%v %s: %d %s", backend, anonymous, method, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if method == http.MethodGet && rec.Body.String() != "archive content" {
|
||||||
|
t.Fatalf("wrong archive entry: %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
if method == "LIST" && !bytes.Contains(rec.Body.Bytes(), []byte("entry.txt")) {
|
||||||
|
t.Fatal("archive listing omitted entry")
|
||||||
|
}
|
||||||
|
after, err := obj.GetObjectInfo(t.Context(), bucket, key, ObjectOptions{VersionID: before.VersionID})
|
||||||
|
if err != nil || !maps.Equal(before.UserDefined, after.UserDefined) || !after.ModTime.Equal(before.ModTime) || after.ETag != before.ETag {
|
||||||
|
t.Fatalf("%s anonymous=%v %s changed stored metadata: before=%v after=%v err=%v", backend, anonymous, method, before.UserDefined, after.UserDefined, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Explicit extraction during an authorized upload still builds and
|
||||||
|
// persists the index for subsequent reads.
|
||||||
|
req, err := newTestSignedRequestV4(http.MethodPut, getPutObjectURL("", bucket, "indexed.zip"), int64(archive.Len()), bytes.NewReader(archive.Bytes()), cred.AccessKey, cred.SecretKey, map[string]string{xMinIOExtract: "true"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
oi, err := obj.GetObjectInfo(t.Context(), bucket, "indexed.zip", ObjectOptions{})
|
||||||
|
if rec.Code != http.StatusOK || err != nil || len(oi.ArchiveInfo(nil)) == 0 {
|
||||||
|
t.Fatalf("explicit extraction lost index: %d %s %v", rec.Code, rec.Body.String(), err)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
+15
-76
@@ -1256,9 +1256,10 @@ func (c *SiteReplicationSys) IAMChangeHook(ctx context.Context, item madmin.SRIA
|
|||||||
// The IAM item has already been applied to the local cluster at this
|
// The IAM item has already been applied to the local cluster at this
|
||||||
// point, and only needs to be updated on all remote peer clusters.
|
// point, and only needs to be updated on all remote peer clusters.
|
||||||
|
|
||||||
c.RLock()
|
// What is sent is read from storage before the lock is taken: loading
|
||||||
defer c.RUnlock()
|
// an identity asks whether site replication is enabled, and a read lock
|
||||||
if !c.enabled {
|
// asked for a second time behind a waiting writer is never granted.
|
||||||
|
if !c.isEnabled() {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1284,6 +1285,12 @@ func (c *SiteReplicationSys) IAMChangeHook(ctx context.Context, item madmin.SRIA
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
c.RLock()
|
||||||
|
defer c.RUnlock()
|
||||||
|
if !c.enabled {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
cerr := c.concDo(nil, func(d string, p madmin.PeerInfo) error {
|
cerr := c.concDo(nil, func(d string, p madmin.PeerInfo) error {
|
||||||
admClient, err := c.getAdminClient(ctx, d)
|
admClient, err := c.getAdminClient(ctx, d)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -2365,40 +2372,10 @@ func (c *SiteReplicationSys) syncToAllPeers(ctx context.Context, addOpts madmin.
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
claims, err := globalIAMSys.GetClaimsForSvcAcc(ctx, acc.Credentials.AccessKey)
|
// The hook sends the account as it is stored (see replicationItem).
|
||||||
if err != nil {
|
|
||||||
return errSRBackendIssue(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, policy, err := globalIAMSys.GetServiceAccount(ctx, acc.Credentials.AccessKey)
|
|
||||||
if err != nil {
|
|
||||||
return errSRBackendIssue(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var policyJSON []byte
|
|
||||||
if policy != nil {
|
|
||||||
policyJSON, err = json.Marshal(policy)
|
|
||||||
if err != nil {
|
|
||||||
return wrapSRErr(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = c.IAMChangeHook(ctx, madmin.SRIAMItem{
|
err = c.IAMChangeHook(ctx, madmin.SRIAMItem{
|
||||||
Type: madmin.SRIAMItemSvcAcc,
|
Type: madmin.SRIAMItemSvcAcc,
|
||||||
SvcAccChange: &madmin.SRSvcAccChange{
|
SvcAccChange: &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{AccessKey: user}},
|
||||||
Create: &madmin.SRSvcAccCreate{
|
|
||||||
Parent: acc.Credentials.ParentUser,
|
|
||||||
AccessKey: user,
|
|
||||||
SecretKey: acc.Credentials.SecretKey,
|
|
||||||
Groups: acc.Credentials.Groups,
|
|
||||||
Claims: claims,
|
|
||||||
SessionPolicy: policyJSON,
|
|
||||||
Status: acc.Credentials.Status,
|
|
||||||
Name: acc.Credentials.Name,
|
|
||||||
Description: acc.Credentials.Description,
|
|
||||||
Expiration: &acc.Credentials.Expiration,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
UpdatedAt: acc.UpdatedAt,
|
UpdatedAt: acc.UpdatedAt,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -5485,49 +5462,11 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
|
|||||||
}
|
}
|
||||||
creds := u.Credentials
|
creds := u.Credentials
|
||||||
if creds.IsServiceAccount() {
|
if creds.IsServiceAccount() {
|
||||||
claims, err := globalIAMSys.GetClaimsForSvcAcc(ctx, creds.AccessKey)
|
// The hook sends the account as it is stored (see
|
||||||
if err != nil {
|
// replicationItem), not as this server has it cached.
|
||||||
replLogOnceIf(ctx,
|
|
||||||
fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
|
|
||||||
fmt.Sprintf("heal-user-%s", user))
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
_, policy, err := globalIAMSys.GetServiceAccount(ctx, creds.AccessKey)
|
|
||||||
if err != nil {
|
|
||||||
replLogOnceIf(ctx,
|
|
||||||
fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
|
|
||||||
fmt.Sprintf("heal-user-%s", user))
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
var policyJSON []byte
|
|
||||||
if policy != nil {
|
|
||||||
policyJSON, err = json.Marshal(policy)
|
|
||||||
if err != nil {
|
|
||||||
replLogOnceIf(ctx,
|
|
||||||
fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
|
|
||||||
fmt.Sprintf("heal-user-%s", user))
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := c.IAMChangeHook(ctx, madmin.SRIAMItem{
|
if err := c.IAMChangeHook(ctx, madmin.SRIAMItem{
|
||||||
Type: madmin.SRIAMItemSvcAcc,
|
Type: madmin.SRIAMItemSvcAcc,
|
||||||
SvcAccChange: &madmin.SRSvcAccChange{
|
SvcAccChange: &madmin.SRSvcAccChange{Create: &madmin.SRSvcAccCreate{AccessKey: creds.AccessKey}},
|
||||||
Create: &madmin.SRSvcAccCreate{
|
|
||||||
Parent: creds.ParentUser,
|
|
||||||
AccessKey: creds.AccessKey,
|
|
||||||
SecretKey: creds.SecretKey,
|
|
||||||
Groups: creds.Groups,
|
|
||||||
Claims: claims,
|
|
||||||
SessionPolicy: policyJSON,
|
|
||||||
Status: creds.Status,
|
|
||||||
Name: creds.Name,
|
|
||||||
Description: creds.Description,
|
|
||||||
Expiration: &creds.Expiration,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
UpdatedAt: lastUpdate,
|
UpdatedAt: lastUpdate,
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
replLogOnceIf(ctx,
|
replLogOnceIf(ctx,
|
||||||
|
|||||||
@@ -0,0 +1,200 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
//
|
||||||
|
// This file is part of Silo Object Storage stack
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAPIStreamingAuthenticationBoundaries(t *testing.T) {
|
||||||
|
defer DetectTestLeak(t)()
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{
|
||||||
|
t: t, endpoints: []string{"PutObject"},
|
||||||
|
objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, cred auth.Credentials, t *testing.T) {
|
||||||
|
data := []byte("authenticated streaming payload")
|
||||||
|
for _, mode := range []string{"valid", "missing-algorithm", "wrong-algorithm", "wrong-signature", "invalid-credential"} {
|
||||||
|
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, getPutObjectURL("", bucket, mode), data, UTCNow())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := signRequestV4(req, cred.AccessKey, cred.SecretKey); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
authorization := req.Header.Get(xhttp.Authorization)
|
||||||
|
switch mode {
|
||||||
|
case "missing-algorithm":
|
||||||
|
authorization = strings.TrimPrefix(authorization, signV4Algorithm+" ")
|
||||||
|
case "wrong-algorithm":
|
||||||
|
authorization = strings.Replace(authorization, signV4Algorithm, "INVALID", 1)
|
||||||
|
case "wrong-signature":
|
||||||
|
prefix, _, found := strings.Cut(authorization, "Signature=")
|
||||||
|
if !found {
|
||||||
|
t.Fatal("signed request has no signature")
|
||||||
|
}
|
||||||
|
authorization = prefix + "Signature=" + strings.Repeat("0", 64)
|
||||||
|
case "invalid-credential":
|
||||||
|
authorization = strings.Replace(authorization, cred.AccessKey, strings.Repeat("A", len(cred.AccessKey)), 1)
|
||||||
|
}
|
||||||
|
req.Header.Set(xhttp.Authorization, authorization)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
_, stored := obj.GetObjectInfo(t.Context(), bucket, mode, ObjectOptions{})
|
||||||
|
if mode == "valid" {
|
||||||
|
if rec.Code != http.StatusOK || stored != nil {
|
||||||
|
t.Fatalf("%s %s: %d %s, object: %v", backend, mode, rec.Code, rec.Body.String(), stored)
|
||||||
|
}
|
||||||
|
} else if rec.Code < 400 || stored == nil {
|
||||||
|
t.Errorf("%s %s: malformed authentication wrote an object: %d %s", backend, mode, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, trailers := range []bool{false, true} {
|
||||||
|
req, err := newTestStreamingSignedRequest(http.MethodPut, getPutObjectURL("", bucket, "tagged"), int64(len(data)), 64*1024, bytes.NewReader(data), cred.AccessKey, cred.SecretKey)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if trailers {
|
||||||
|
req.Header.Set(xhttp.AmzContentSha256, streamingContentSHA256Trailer)
|
||||||
|
req.Header.Set(awsTrailerHeader, "x-amz-checksum-crc32")
|
||||||
|
if err := signRequestV4(req, cred.AccessKey, cred.SecretKey); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
getRequestAuthType(req)
|
||||||
|
if _, _, _, _, code := calculateSeedSignature(req, trailers); code != ErrNone {
|
||||||
|
t.Fatalf("%s trailers=%v: valid seed rejected: %v", backend, trailers, code)
|
||||||
|
}
|
||||||
|
req.Header.Set(xhttp.AmzObjectTagging, "unsigned=injected")
|
||||||
|
if _, code := newSignV4ChunkedReader(req, trailers); code != ErrUnsignedHeaders {
|
||||||
|
t.Errorf("%s trailers=%v: unsigned tags accepted: %v", backend, trailers, code)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
if _, err := obj.GetObjectInfo(t.Context(), bucket, "tagged", ObjectOptions{}); rec.Code < 400 || err == nil {
|
||||||
|
t.Errorf("%s trailers=%v: injected metadata stored: %d", backend, trailers, rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A public bucket can still use the unsigned trailer decoder without
|
||||||
|
// claiming credentials; authorization remains the handler's job.
|
||||||
|
req, err := newStreamingUnsignedTrailerRequest(http.MethodPut, "/public/key", data, UTCNow())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rd, code := newUnsignedV4ChunkedReader(req, true)
|
||||||
|
if code != ErrNone {
|
||||||
|
t.Fatal(code)
|
||||||
|
}
|
||||||
|
got, err := io.ReadAll(rd)
|
||||||
|
rd.Close()
|
||||||
|
if err != nil || !bytes.Equal(got, data) {
|
||||||
|
t.Fatalf("anonymous decoder: %q %v", got, err)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStreamingTrailerConditionValues(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name, payload, authorization string
|
||||||
|
wantVersion, wantAuthType string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "signed trailer",
|
||||||
|
payload: streamingContentSHA256Trailer,
|
||||||
|
authorization: "AWS4-HMAC-SHA256 Credential=test",
|
||||||
|
wantVersion: signV4Algorithm,
|
||||||
|
wantAuthType: "REST-HEADER",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "authenticated unsigned trailer",
|
||||||
|
payload: unsignedPayloadTrailer,
|
||||||
|
authorization: "AWS4-HMAC-SHA256 Credential=test",
|
||||||
|
wantVersion: signV4Algorithm,
|
||||||
|
wantAuthType: "REST-HEADER",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "anonymous unsigned trailer",
|
||||||
|
payload: unsignedPayloadTrailer,
|
||||||
|
wantVersion: "",
|
||||||
|
wantAuthType: "",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
r := httptest.NewRequest(http.MethodPut, "http://minio.local/bucket/object", nil)
|
||||||
|
r.Header.Set(xhttp.AmzContentSha256, tc.payload)
|
||||||
|
if tc.authorization != "" {
|
||||||
|
r.Header.Set(xhttp.Authorization, tc.authorization)
|
||||||
|
}
|
||||||
|
values := getConditionValues(r, "", auth.Credentials{})
|
||||||
|
if got := strings.Join(values["signatureversion"], ""); got != tc.wantVersion {
|
||||||
|
t.Fatalf("signatureversion = %q, want %q", got, tc.wantVersion)
|
||||||
|
}
|
||||||
|
if got := strings.Join(values["authType"], ""); got != tc.wantAuthType {
|
||||||
|
t.Fatalf("authType = %q, want %q", got, tc.wantAuthType)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStreamingUnsignedTrailerDateGate(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
date string
|
||||||
|
authorized bool
|
||||||
|
wantCall bool
|
||||||
|
}{
|
||||||
|
{name: "anonymous without date", authorized: false, wantCall: true},
|
||||||
|
{name: "authenticated current", date: UTCNow().Format(iso8601Format), authorized: true, wantCall: true},
|
||||||
|
{name: "authenticated missing", authorized: true, wantCall: false},
|
||||||
|
{name: "authenticated malformed", date: "not-a-date", authorized: true, wantCall: false},
|
||||||
|
{name: "authenticated old", date: UTCNow().Add(-globalMaxSkewTime - time.Second).Format(iso8601Format), authorized: true, wantCall: false},
|
||||||
|
{name: "authenticated future", date: UTCNow().Add(globalMaxSkewTime + time.Second).Format(iso8601Format), authorized: true, wantCall: false},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
r := httptest.NewRequest(http.MethodPut, "http://minio.local/bucket/object", nil)
|
||||||
|
r.Header.Set(xhttp.AmzContentSha256, unsignedPayloadTrailer)
|
||||||
|
if tc.authorized {
|
||||||
|
r.Header.Set(xhttp.Authorization, "AWS4-HMAC-SHA256 Credential=test")
|
||||||
|
}
|
||||||
|
if tc.date != "" {
|
||||||
|
r.Header.Set(xhttp.AmzDate, tc.date)
|
||||||
|
}
|
||||||
|
|
||||||
|
called := false
|
||||||
|
h := setAuthMiddleware(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
called = true
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(rec, r)
|
||||||
|
if called != tc.wantCall {
|
||||||
|
t.Fatalf("handler called = %v, want %v; status=%d body=%q", called, tc.wantCall, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -127,6 +127,9 @@ func calculateSeedSignature(r *http.Request, trailers bool) (cred auth.Credentia
|
|||||||
if errCode != ErrNone {
|
if errCode != ErrNone {
|
||||||
return cred, "", "", time.Time{}, errCode
|
return cred, "", "", time.Time{}, errCode
|
||||||
}
|
}
|
||||||
|
if errCode := checkUnsignedHeaders(extractedSignedHeaders, r); errCode != ErrNone {
|
||||||
|
return cred, "", "", time.Time{}, errCode
|
||||||
|
}
|
||||||
|
|
||||||
cred, _, errCode = checkKeyValid(r, signV4Values.Credential.accessKey)
|
cred, _, errCode = checkKeyValid(r, signV4Values.Credential.accessKey)
|
||||||
if errCode != ErrNone {
|
if errCode != ErrNone {
|
||||||
|
|||||||
@@ -24,6 +24,8 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
)
|
)
|
||||||
|
|
||||||
// newUnsignedV4ChunkedReader returns a new s3UnsignedChunkedReader that translates
|
// newUnsignedV4ChunkedReader returns a new s3UnsignedChunkedReader that translates
|
||||||
@@ -42,8 +44,10 @@ func newUnsignedV4ChunkedReader(req *http.Request, trailer bool) (io.ReadCloser,
|
|||||||
if isRequestPresignedSignatureV4(req) {
|
if isRequestPresignedSignatureV4(req) {
|
||||||
return nil, ErrSignatureVersionNotSupported
|
return nil, ErrSignatureVersionNotSupported
|
||||||
}
|
}
|
||||||
if isRequestSignatureV4(req) {
|
// Any claimed header credential must pass the complete SigV4 parser. The
|
||||||
if errCode := reqSignatureV4Verify(req, globalSite.Region(), serviceS3); errCode != ErrNone {
|
// streaming auth classification does not require the algorithm prefix.
|
||||||
|
if req.Header.Get(xhttp.Authorization) != "" {
|
||||||
|
if errCode := doesSignatureMatch(unsignedPayloadTrailer, req, globalSite.Region(), serviceS3); errCode != ErrNone {
|
||||||
return nil, errCode
|
return nil, errCode
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,180 @@
|
|||||||
|
// Copyright (c) 2026 Feng Ruohang
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
//
|
||||||
|
// This program is free software: you can redistribute it and/or modify
|
||||||
|
// it under the terms of the GNU Affero General Public License as published by
|
||||||
|
// the Free Software Foundation, either version 3 of the License, or
|
||||||
|
// (at your option) any later version.
|
||||||
|
//
|
||||||
|
// This program is distributed in the hope that it will be useful,
|
||||||
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
// GNU Affero General Public License for more details.
|
||||||
|
//
|
||||||
|
// You should have received a copy of the GNU Affero General Public License
|
||||||
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/minio/madmin-go/v3"
|
||||||
|
"github.com/minio/minio/internal/auth"
|
||||||
|
"github.com/minio/minio/internal/bucket/replication"
|
||||||
|
xhttp "github.com/minio/minio/internal/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAPITaggingProxyAuthorization(t *testing.T) {
|
||||||
|
ExecObjectLayerAPITest(ExecObjectLayerAPITestArgs{t: t, objAPITest: func(obj ObjectLayer, backend, bucket string, router http.Handler, root auth.Credentials, t *testing.T) {
|
||||||
|
conditional := newDeleteAuthzPolicyUser(t, backend, bucket, fmt.Sprintf(`[
|
||||||
|
{"Effect":"Allow","Action":["s3:GetObjectTagging","s3:PutObjectTagging","s3:DeleteObjectTagging"],"Resource":["arn:aws:s3:::%s/*"]},
|
||||||
|
{"Effect":"Deny","Action":["s3:GetObjectTagging","s3:PutObjectTagging","s3:DeleteObjectTagging"],"Resource":["arn:aws:s3:::%s/*"],"Condition":{"StringEquals":{"s3:ExistingObjectTag/secret":"true"}}}
|
||||||
|
]`, bucket, bucket))
|
||||||
|
tagAllow := newDeleteAuthzPolicyUser(t, backend, bucket, fmt.Sprintf(`[
|
||||||
|
{"Effect":"Allow","Action":["s3:GetObjectTagging","s3:PutObjectTagging","s3:DeleteObjectTagging"],"Resource":["arn:aws:s3:::%s/*"],"Condition":{"StringEquals":{"s3:ExistingObjectTag/secret":"false"}}}
|
||||||
|
]`, bucket))
|
||||||
|
denied := newDeleteAuthzPolicyUser(t, backend, bucket, fmt.Sprintf(`[
|
||||||
|
{"Effect":"Deny","Action":["s3:*"],"Resource":["arn:aws:s3:::%s/*"]}
|
||||||
|
]`, bucket))
|
||||||
|
zero := iamAdminUser(t.Context(), t, globalIAMSys, "tagging-no-policy-"+mustGetUUID(), "")
|
||||||
|
versionID := mustGetUUID()
|
||||||
|
for _, method := range []string{http.MethodGet, http.MethodPut, http.MethodDelete} {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
tags []string
|
||||||
|
cred auth.Credentials
|
||||||
|
readStatus, writeStatus, wantStatus int
|
||||||
|
wantWrites int32
|
||||||
|
disabledFirst bool
|
||||||
|
firstReadStatus int
|
||||||
|
noReads bool
|
||||||
|
}{
|
||||||
|
{name: "deny-second-target", tags: []string{"false", "true"}, cred: conditional, wantStatus: http.StatusForbidden},
|
||||||
|
{name: "deny-first-target", tags: []string{"true", "false"}, cred: conditional, wantStatus: http.StatusForbidden},
|
||||||
|
{name: "all-authorized", tags: []string{"false", "false"}, cred: conditional, wantWrites: 2},
|
||||||
|
{name: "anonymous", tags: []string{"false"}, wantStatus: http.StatusForbidden, noReads: true},
|
||||||
|
{name: "invalid-credentials", tags: []string{"false"}, cred: auth.Credentials{AccessKey: root.AccessKey, SecretKey: "wrong-secret-key"}, wantStatus: http.StatusForbidden, noReads: true},
|
||||||
|
{name: "zero-permissions", tags: []string{"false"}, cred: zero, wantStatus: http.StatusForbidden, noReads: true},
|
||||||
|
{name: "zero-permissions-missing", tags: []string{"false"}, cred: zero, readStatus: http.StatusNotFound, wantStatus: http.StatusForbidden, noReads: true},
|
||||||
|
{name: "unconditional-deny", tags: []string{"false"}, cred: denied, wantStatus: http.StatusForbidden, noReads: true},
|
||||||
|
{name: "unreadable", tags: []string{"false"}, cred: root, readStatus: http.StatusServiceUnavailable, wantStatus: http.StatusServiceUnavailable},
|
||||||
|
{name: "missing-version", tags: []string{"false", "false"}, cred: root, readStatus: http.StatusNotFound, wantStatus: http.StatusNotFound},
|
||||||
|
{name: "tag-allow-missing-version", tags: []string{"false"}, cred: tagAllow, readStatus: http.StatusNotFound, wantStatus: http.StatusForbidden},
|
||||||
|
{name: "tag-allow-lagging-first", tags: []string{"false", "false"}, cred: tagAllow, firstReadStatus: http.StatusNotFound, wantWrites: 1},
|
||||||
|
{name: "tag-allow-lagging-second", tags: []string{"false", "false"}, cred: tagAllow, firstReadStatus: http.StatusOK, readStatus: http.StatusNotFound, wantWrites: 1},
|
||||||
|
{name: "tag-allow-present-untagged", tags: []string{"false", ""}, cred: tagAllow, wantStatus: http.StatusForbidden},
|
||||||
|
{name: "tag-allow-unreadable-first", tags: []string{"false", "false"}, cred: tagAllow, firstReadStatus: http.StatusServiceUnavailable, wantStatus: http.StatusServiceUnavailable},
|
||||||
|
{name: "tag-allow-unreadable-second", tags: []string{"false", "false"}, cred: tagAllow, firstReadStatus: http.StatusOK, readStatus: http.StatusServiceUnavailable, wantStatus: http.StatusServiceUnavailable},
|
||||||
|
{name: "disabled-target", tags: []string{"true", "false"}, cred: conditional, disabledFirst: true, wantWrites: 1},
|
||||||
|
{name: "no-available-target", tags: []string{"false"}, cred: tagAllow, disabledFirst: true, wantStatus: http.StatusServiceUnavailable, noReads: true},
|
||||||
|
{name: "skipped-target-cannot-mask-failure", tags: []string{"false", "false"}, cred: root, disabledFirst: true, writeStatus: http.StatusServiceUnavailable, wantStatus: http.StatusServiceUnavailable, wantWrites: 1},
|
||||||
|
} {
|
||||||
|
if method == http.MethodGet && tc.writeStatus != 0 {
|
||||||
|
continue // This case only exercises a remote mutation failure.
|
||||||
|
}
|
||||||
|
t.Run(backend+"/"+method+"/"+tc.name, func(t *testing.T) {
|
||||||
|
var reads, writes atomic.Int32
|
||||||
|
globalBucketTargetSys.Lock()
|
||||||
|
globalBucketTargetSys.targetsMap[bucket] = nil
|
||||||
|
globalBucketTargetSys.Unlock()
|
||||||
|
cfg := replication.Config{}
|
||||||
|
for i, value := range tc.tags {
|
||||||
|
readStatus := tc.readStatus
|
||||||
|
if i == 0 && tc.firstReadStatus != 0 {
|
||||||
|
readStatus = tc.firstReadStatus
|
||||||
|
}
|
||||||
|
target := newILMTestTarget(t)
|
||||||
|
registerILMTestTarget(t, obj, bucket, target, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Query().Get("versionId") != versionID || !r.URL.Query().Has("tagging") || r.Header.Get(xhttp.MinIOSourceProxyRequest) != "true" {
|
||||||
|
t.Errorf("remote request lost addressed version or proxy marker: %s", r.URL)
|
||||||
|
}
|
||||||
|
status := tc.writeStatus
|
||||||
|
if r.Method == http.MethodGet {
|
||||||
|
reads.Add(1)
|
||||||
|
status = readStatus
|
||||||
|
} else {
|
||||||
|
writes.Add(1)
|
||||||
|
if readStatus == http.StatusNotFound {
|
||||||
|
t.Error("tagging write reached a missing replica")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/xml")
|
||||||
|
if status != 0 && status != http.StatusOK {
|
||||||
|
w.WriteHeader(status)
|
||||||
|
code := "ServiceUnavailable"
|
||||||
|
if status == http.StatusNotFound {
|
||||||
|
code = "NoSuchVersion"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "<Error><Code>%s</Code><Message>injected remote result</Message></Error>", code)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
switch r.Method {
|
||||||
|
case http.MethodGet:
|
||||||
|
if value == "" {
|
||||||
|
fmt.Fprint(w, "<Tagging><TagSet></TagSet></Tagging>")
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(w, "<Tagging><TagSet><Tag><Key>secret</Key><Value>%s</Value></Tag></TagSet></Tagging>", value)
|
||||||
|
}
|
||||||
|
case http.MethodDelete:
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
globalBucketTargetSys.Lock()
|
||||||
|
globalBucketTargetSys.targetsMap[bucket][i].Credentials = &madmin.Credentials{AccessKey: "proxy-test", SecretKey: "proxy-test-secret"}
|
||||||
|
globalBucketTargetSys.arnRemotesMap[target.arn].Client.disableProxy = tc.disabledFirst && i == 0
|
||||||
|
globalBucketTargetSys.Unlock()
|
||||||
|
rule := configs[0].Rules[0]
|
||||||
|
rule.Priority = i + 1
|
||||||
|
rule.Destination = replication.Destination{ARN: target.arn, Bucket: target.bucket}
|
||||||
|
cfg.Rules = append(cfg.Rules, rule)
|
||||||
|
}
|
||||||
|
meta, err := globalBucketMetadataSys.Get(bucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
meta.replicationConfig = &cfg
|
||||||
|
globalBucketMetadataSys.Set(bucket, meta)
|
||||||
|
body := ""
|
||||||
|
if method == http.MethodPut {
|
||||||
|
body = "<Tagging><TagSet><Tag><Key>new</Key><Value>value</Value></Tag></TagSet></Tagging>"
|
||||||
|
}
|
||||||
|
path := "/" + bucket + "/remote-only?tagging&versionId=" + versionID
|
||||||
|
var req *http.Request
|
||||||
|
if tc.cred.AccessKey == "" {
|
||||||
|
req = httptest.NewRequest(method, path, strings.NewReader(body))
|
||||||
|
} else {
|
||||||
|
req, err = newTestSignedRequestV4(method, path, int64(len(body)), strings.NewReader(body), tc.cred.AccessKey, tc.cred.SecretKey, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(rec, req)
|
||||||
|
wantStatus := tc.wantStatus
|
||||||
|
if wantStatus == 0 {
|
||||||
|
wantStatus = http.StatusOK
|
||||||
|
if method == http.MethodDelete {
|
||||||
|
wantStatus = http.StatusNoContent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
wantWrites := tc.wantWrites
|
||||||
|
if method == http.MethodGet {
|
||||||
|
wantWrites = 0
|
||||||
|
}
|
||||||
|
if rec.Code != wantStatus || writes.Load() != wantWrites {
|
||||||
|
t.Errorf("status=%d writes=%d, want status=%d writes=%d: %s", rec.Code, writes.Load(), wantStatus, wantWrites, rec.Body.String())
|
||||||
|
}
|
||||||
|
if tc.noReads && reads.Load() != 0 {
|
||||||
|
t.Errorf("unauthorized request reached %d remote reads", reads.Load())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}})
|
||||||
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Silo Healthcheck
|
# Silo Healthcheck
|
||||||
|
|
||||||
Silo server exposes three un-authenticated, healthcheck endpoints liveness probe and a cluster probe at `/minio/health/live` and `/minio/health/cluster` respectively.
|
Silo server exposes four unauthenticated healthcheck endpoints: `/minio/health/live`, `/minio/health/ready`, `/minio/health/cluster`, and `/minio/health/cluster/read`.
|
||||||
|
|
||||||
## Native CLI probe
|
## Native CLI probe
|
||||||
|
|
||||||
@@ -76,7 +76,7 @@ readinessProbe:
|
|||||||
The reply is '200 OK' if cluster has write quorum if not it returns '503 Service Unavailable'.
|
The reply is '200 OK' if cluster has write quorum if not it returns '503 Service Unavailable'.
|
||||||
|
|
||||||
```
|
```
|
||||||
curl http://silo1:9001/minio/health/cluster
|
curl http://silo1:9000/minio/health/cluster
|
||||||
HTTP/1.1 503 Service Unavailable
|
HTTP/1.1 503 Service Unavailable
|
||||||
Accept-Ranges: bytes
|
Accept-Ranges: bytes
|
||||||
Content-Length: 0
|
Content-Length: 0
|
||||||
@@ -94,7 +94,7 @@ Date: Tue, 21 Jul 2020 00:36:14 GMT
|
|||||||
The reply is '200 OK' if cluster has read quorum if not it returns '503 Service Unavailable'.
|
The reply is '200 OK' if cluster has read quorum if not it returns '503 Service Unavailable'.
|
||||||
|
|
||||||
```
|
```
|
||||||
curl http://silo1:9001/minio/health/cluster/read
|
curl http://silo1:9000/minio/health/cluster/read
|
||||||
HTTP/1.1 503 Service Unavailable
|
HTTP/1.1 503 Service Unavailable
|
||||||
Accept-Ranges: bytes
|
Accept-Ranges: bytes
|
||||||
Content-Length: 0
|
Content-Length: 0
|
||||||
@@ -112,7 +112,7 @@ Date: Tue, 21 Jul 2020 00:36:14 GMT
|
|||||||
You may query the cluster probe endpoint to check if the node which received the request can be taken down for maintenance, if the server replies back '412 Precondition Failed' this means you will lose HA. '200 OK' means you are okay to proceed.
|
You may query the cluster probe endpoint to check if the node which received the request can be taken down for maintenance, if the server replies back '412 Precondition Failed' this means you will lose HA. '200 OK' means you are okay to proceed.
|
||||||
|
|
||||||
```
|
```
|
||||||
curl http://silo1:9001/minio/health/cluster?maintenance=true
|
curl http://silo1:9000/minio/health/cluster?maintenance=true
|
||||||
HTTP/1.1 412 Precondition Failed
|
HTTP/1.1 412 Precondition Failed
|
||||||
Accept-Ranges: bytes
|
Accept-Ranges: bytes
|
||||||
Content-Length: 0
|
Content-Length: 0
|
||||||
|
|||||||
+17
-3
@@ -34,9 +34,10 @@ helm install silo ./helm/silo \
|
|||||||
--set rootPassword='replace-with-a-long-random-secret'
|
--set rootPassword='replace-with-a-long-random-secret'
|
||||||
```
|
```
|
||||||
|
|
||||||
The default server and post-install image is `docker.io/pgsty/silo`. The image
|
The default server image is `docker.io/pgsty/silo` (which contains the `silo`
|
||||||
contains the `silo` server, `mcli`, and an `mc` compatibility symlink. It does
|
server, `mcli`, and an `mc` client compatibility symlink, without a `minio`
|
||||||
not contain a `minio` server binary.
|
server binary). Post-install resource jobs use `mcImage` (`docker.io/pgsty/mc`)
|
||||||
|
by default.
|
||||||
|
|
||||||
## Compatibility contract
|
## Compatibility contract
|
||||||
|
|
||||||
@@ -100,6 +101,19 @@ new chart invokes `silo server`; an old image does not provide that executable.
|
|||||||
The new image accepts a legacy first argv token of `minio` only to support old
|
The new image accepts a legacy first argv token of `minio` only to support old
|
||||||
chart commands during the forward migration.
|
chart commands during the forward migration.
|
||||||
|
|
||||||
|
When upgrading an existing Silo release to `RELEASE.2026-11-09T00-00-00Z` (chart
|
||||||
|
`7.0.4`):
|
||||||
|
|
||||||
|
- `helm upgrade --reuse-values` keeps the previously saved `image.tag` and
|
||||||
|
`mcImage.tag`; pass `--set-string image.tag=RELEASE.2026-11-09T00-00-00Z` and
|
||||||
|
`--set-string mcImage.tag=RELEASE.2026-11-09T00-00-00Z` explicitly.
|
||||||
|
- Drain or cancel active batch jobs before upgrading; `20261026` cannot resume
|
||||||
|
new submitter-bound batch manifests on downgrade.
|
||||||
|
- Single-pool clusters whose erasure sets do not all share the same pods/hosts
|
||||||
|
require a drained stop-all upgrade rather than a rolling pod restart, and
|
||||||
|
`POST ?restore` returns `503` while mixed versions are serving. See the
|
||||||
|
[20261109 release notes](https://silo.pgsty.com/blog/release/silo-20261109/#upgrade).
|
||||||
|
|
||||||
## Persistence and TLS
|
## Persistence and TLS
|
||||||
|
|
||||||
The default PersistentVolumeClaim mounts at `/export`. Use an existing claim
|
The default PersistentVolumeClaim mounts at `/export`. Use an existing claim
|
||||||
|
|||||||
@@ -1,43 +1,44 @@
|
|||||||
|
{{- $scheme := ternary "https" "http" .Values.tls.enabled -}}
|
||||||
{{- if eq .Values.service.type "ClusterIP" "NodePort" }}
|
{{- if eq .Values.service.type "ClusterIP" "NodePort" }}
|
||||||
Silo can be accessed via port {{ .Values.service.port }} on the following DNS name from within your cluster:
|
Silo can be accessed via port {{ .Values.service.port }} on the following DNS name from within your cluster:
|
||||||
{{ template "silo.fullname" . }}.{{ .Release.Namespace }}.{{ .Values.clusterDomain }}
|
{{ template "silo.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}
|
||||||
|
|
||||||
To access Silo from localhost, run the below commands:
|
To access Silo from localhost, run the commands below:
|
||||||
|
|
||||||
1. export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
|
1. export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
|
||||||
2. kubectl port-forward $POD_NAME 9000 --namespace {{ .Release.Namespace }}
|
2. kubectl port-forward $POD_NAME {{ .Values.service.port }}:{{ .Values.minioAPIPort | default 9000 }} --namespace {{ .Release.Namespace }}
|
||||||
|
|
||||||
Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/
|
Read more about port forwarding here: https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
|
||||||
|
|
||||||
You can now access Silo server on http://localhost:9000. Follow the below steps to connect to Silo server with mc client:
|
You can now access the Silo server on {{ $scheme }}://localhost:{{ .Values.service.port }}. Connect with the mcli client:
|
||||||
|
|
||||||
1. Download the Silo mc client - https://silo.pgsty.com/reference/minio-mc/#quickstart
|
1. Download mcli: https://silo.pgsty.com/download/#client
|
||||||
|
|
||||||
2. export MC_HOST_{{ template "silo.fullname" . }}_local=http://$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:{{ .Values.service.port }}
|
2. mcli alias set local {{ $scheme }}://localhost:{{ .Values.service.port }} "$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootUser}" | base64 --decode)" "$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)"
|
||||||
|
|
||||||
3. mc ls {{ template "silo.fullname" . }}_local
|
3. mcli ls local
|
||||||
|
|
||||||
{{- end }}
|
{{- end }}
|
||||||
{{- if eq .Values.service.type "LoadBalancer" }}
|
{{- if eq .Values.service.type "LoadBalancer" }}
|
||||||
Silo can be accessed via port {{ .Values.service.port }} on an external IP address. Get the service external IP address by:
|
Silo can be accessed via port {{ .Values.service.port }} on an external IP address. Get the service external IP address by:
|
||||||
kubectl get svc --namespace {{ .Release.Namespace }} -l app={{ template "silo.fullname" . }}
|
kubectl get svc --namespace {{ .Release.Namespace }} -l "app={{ template "silo.name" . }},release={{ .Release.Name }}"
|
||||||
|
|
||||||
Note that the public IP may take a couple of minutes to be available.
|
Note that the public IP may take a couple of minutes to become available.
|
||||||
|
|
||||||
You can now access Silo server on http://<External-IP>:9000. Follow the below steps to connect to Silo server with mc client:
|
You can now access the Silo server on {{ $scheme }}://<External-IP>:{{ .Values.service.port }}. Connect with the mcli client:
|
||||||
|
|
||||||
1. Download the Silo mc client - https://silo.pgsty.com/reference/minio-mc/#quickstart
|
1. Download mcli: https://silo.pgsty.com/download/#client
|
||||||
|
|
||||||
2. export MC_HOST_{{ template "silo.fullname" . }}_local=http://$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret {{ template "silo.secretName" . }} -o jsonpath="{.data.rootPassword}" | base64 --decode)@<External-IP>:{{ .Values.service.port }}
|
2. mcli alias set external {{ $scheme }}://<External-IP>:{{ .Values.service.port }} "$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootUser}" | base64 --decode)" "$(kubectl get secret {{ template "silo.secretName" . }} --namespace {{ .Release.Namespace }} -o jsonpath="{.data.rootPassword}" | base64 --decode)"
|
||||||
|
|
||||||
3. mc ls {{ template "silo.fullname" . }}
|
3. mcli ls external
|
||||||
|
|
||||||
Alternately, use a browser, an S3 SDK, or a MinIO-compatible SDK to access Silo - https://silo.pgsty.com/reference/minio-server/minio-server/
|
Alternately, use a browser, an S3 SDK, or a MinIO-compatible SDK to access Silo: https://silo.pgsty.com/reference/minio-server/
|
||||||
{{- end }}
|
{{- end }}
|
||||||
|
|
||||||
{{ if and (.Values.networkPolicy.enabled) (not .Values.networkPolicy.allowExternal) }}
|
{{ if and (.Values.networkPolicy.enabled) (not .Values.networkPolicy.allowExternal) }}
|
||||||
Note: Since NetworkPolicy is enabled, only pods with label
|
Note: Since NetworkPolicy is enabled, only pods with label
|
||||||
{{ template "silo.fullname" . }}-client=true"
|
{{ template "silo.fullname" . }}-client=true
|
||||||
will be able to connect to this Silo cluster.
|
will be able to connect to this Silo cluster.
|
||||||
{{- end }}
|
{{- end }}
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ fullnameOverride: ""
|
|||||||
##
|
##
|
||||||
clusterDomain: cluster.local
|
clusterDomain: cluster.local
|
||||||
|
|
||||||
## Set default image, imageTag, and imagePullPolicy. mode is used to indicate the
|
## Set default Silo server image repository, tag, and pull policy.
|
||||||
##
|
##
|
||||||
image:
|
image:
|
||||||
repository: pgsty/silo
|
repository: pgsty/silo
|
||||||
@@ -517,7 +517,7 @@ postJob:
|
|||||||
## Add environment variables for Silo. The MINIO_* names remain the stable
|
## Add environment variables for Silo. The MINIO_* names remain the stable
|
||||||
## configuration interface and are passed to the Silo container(s).
|
## configuration interface and are passed to the Silo container(s).
|
||||||
environment:
|
environment:
|
||||||
## Please refer for comprehensive list https://silo.pgsty.com/reference/minio-server/minio-server/
|
## See https://silo.pgsty.com/reference/minio-server/settings/ for available settings.
|
||||||
## MINIO_BROWSER: "off"
|
## MINIO_BROWSER: "off"
|
||||||
|
|
||||||
## The name of a secret in the same kubernetes namespace which contain secret values
|
## The name of a secret in the same kubernetes namespace which contain secret values
|
||||||
|
|||||||
Reference in New Issue
Block a user