Compare commits

..

79 Commits

Author SHA1 Message Date
Anis Eleuch 712fe1a8df fix: proxy requests to honor global transport
* fix: proxy requests to honor global transport 
Load the globalProxyEndpoint properly

also, currently, the proxy requests will fail silently for batch cancel
even if the proxy fails; instead,d properly send the corresponding error back
for such proxy failures if opted

* pass the transport to the GetProxyEnpoints function

---------

Co-authored-by: Praveen raj Mani <praveen@minio.io>
2025-02-03 22:03:04 +01:00
Klaus Post 4a319bedc9 Redact sensitive fields from DescribeBatchJob (#20881)
Redacts the following if set:

* replicate/credentials/secretKey
* replicate/credentials/sessionToken
* expire/notify/token
2025-02-03 08:56:26 -08:00
Klaus Post bdb3db6dad Add lock overload protection (#20876)
Reject new lock requests immediately when 1000 goroutines are queued 
for the local lock mutex.

We do not reject unlocking, refreshing, or maintenance; they add to the count.

The limit is set to allow for bursty behavior but prevent requests from 
overloading the server completely.
2025-01-31 11:54:34 -08:00
Klaus Post abb385af41 Check for valid checksum (#20878)
Add a few safety measures for checksums.
2025-01-28 16:59:23 -08:00
Harshavardhana 4ee62606e4 update govulncheck 2025-01-28 11:11:08 -08:00
Anis Eleuch 079d64c801 DeleteObjects: Send delete to all pools (#172) (#20821)
Currently, DeleteObjects() tries to find the object's pool before
sending a delete request. This only works well when an object has
multiple versions in different pools since looking for the pool does
not consider the version-id. When an S3 client wants to
remove a version-id that exists in pool 2, the delete request will be
directed to pool one because it has another version of the same object.

This commit will remove looking for pool logic and will send a delete
request to all pools in parallel. This should not cause any performance
regression in most of the cases since the object will unlikely exist
in only one pool, and the performance price will be similar to
getPoolIndex() in that case.
2025-01-28 08:57:18 -08:00
Klaus Post dcc000ae2c Allow URLs up to 32KB and improve parsing speed (#20874)
Before/after...
```
Benchmark_hasBadPathComponent/long-32          	   43936	     27232 ns/op	 146.89 MB/s	   32768 B/op	       1 allocs/op
Benchmark_hasBadPathComponent/long-32          	   89956	     13375 ns/op	 299.07 MB/s	       0 B/op	       0 allocs/op
```

* Remove unused.
2025-01-27 08:42:45 -08:00
Harshavardhana c5d19ecebb do not expose secret-key to lambda event handler (#20870) 2025-01-24 11:27:43 -08:00
Harshavardhana ed29a525b3 remove fips builds 2025-01-21 02:10:10 -08:00
Minio Trusted 020c46cd3c Update yaml files to latest version RELEASE.2025-01-20T14-49-07Z 2025-01-21 09:44:32 +00:00
Klaus Post 827004cd6d Add Full Object Checksums and CRC64-NVME (#20855)
Backport of AIStor PR 247.

Add support for full object checksums as described here:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

New checksum types are fully supported. Mint tests from https://github.com/minio/minio-go/pull/2026 are now passing.

Includes fixes from https://github.com/minio/minio/pull/20743 for mint tests.

Add using checksums as validation for object content. Fixes #20845 #20849

Fixes checksum replication (downstream PR 250)
2025-01-20 06:49:07 -08:00
Harshavardhana 779ec8f0d4 do not list buckets without local quorum (#20852)
ListBuckets() would result in listing buckets
without quorum, this PR fixes the behavior.
2025-01-19 15:13:17 -08:00
Minio Trusted 3d0f513ee2 Update yaml files to latest version RELEASE.2025-01-18T00-31-37Z 2025-01-18 01:38:52 +00:00
Harshavardhana 4b6eadbd80 update deps (#20851) 2025-01-17 16:31:37 -08:00
Shubhendu 6f47414b23 Correct bucket metrics name (#20823)
Earlier, cluster and bucket metrics were named 
`minio_usage_last_activity_nano_seconds`.

The bucket level is now named as 
`minio_bucket_usage_last_activity_nano_seconds`

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2025-01-17 13:12:21 -08:00
Harshavardhana 224a27992a update go-deps on debugging tools 2025-01-17 11:41:46 -08:00
Harshavardhana 232544e1d8 update inspect export, which was missed from #20846 2025-01-17 11:38:44 -08:00
Anis Eleuch dbcb71828d s3: Provide enough buffer when the object final size is unknown (#20847)
When compression is enabled, the final object size is not calculated in
that case, we need to make sure that the provided buffer is always
more significant than the shard size, the bitrot will always calculate 
the hash of blocks with shard size, except the last block.
2025-01-17 11:19:30 -08:00
Klaus Post b9196757fd Fix inconsistently written compressed files. (#20846)
Before https://github.com/minio/minio/pull/20575, files could pick up indices 
from unrelated files if no index was added.

This would result in these files not being consistent across a set.

When loading, search for the compression indicators and check if they 
are within the problematic date range, and clean up any parts that have 
an index but shouldn't.

The test validates that the signature matches the one in files stored without an index.

Bumps xlMetaVersion, so this check doesn't have to be made for future versions.
2025-01-17 11:17:18 -08:00
Andreas Auernhammer b4ac53d157 update github.com/minio/kms-go/kes to v0.3.1 (#20843)
Signed-off-by: Andreas Auernhammer <github@aead.dev>
2025-01-16 01:13:28 -08:00
Poorna 4952bdb770 decom: avoid skipping single delete markers for replication (#20836)
It is possible delete marker was received on old pool as decom
move in progress, this PR allows decom retry to ensure these
delete markers are moved to new pool so that decommission can
be completed.

Fixes #20819
2025-01-14 11:53:02 -08:00
Anis Eleuch 00b2ef2932 Bump golang.org/x/net to silence wrong vuln checker (#20814) 2025-01-08 16:39:24 +05:30
Klaus Post 4536ecfaa4 Add cpuio profiling potential crash workaround (#20809)
Add profiling potential crash wourkaround

Using admin traces could potentially crash the server (or handler more likely) due to upstream divide by 0: https://github.com/felixge/fgprof/pull/34

Ensure the profile always runs 100ms before stopping, so sample count isn't 0 (default sample rate ~10ms/sample, but allow for cpu starvation)
2025-01-06 21:21:54 +05:30
Harshavardhana 43a7402968 update helm release v5.4.0 2025-01-02 21:34:47 -08:00
Allan Roger Reid 330dca9a35 Add resiliency tests (#20786) 2024-12-20 20:24:45 -08:00
Klaus Post ddd137d317 ListObjectParts should return actual size (#20782)
Fixes #20781:

```
λ aws --endpoint-url http://127.0.0.1:9001 s3api list-parts --bucket testbucket --key test.testcompress --upload-id "ZDM0YzUwM2YtZWM1Zi00NWI2LTgxMzYtZTIwMGE3Yjc0Y2Y1LjYyMzgyMmFhLWU2N2QtNGUyYS04NDE1LWUzZDFlZmJmMWUyZHgxNzM0NjI1MjgyMDkyNzY4MDAw"
{
    "Parts": [
        {
            "PartNumber": 1,
            "LastModified": "2024-12-19T16:47:04.334000+00:00",
            "ETag": "\"7025f242f56479e06c435c0b500cdbb2\"",
            "Size": 2002
        }
    ],
    "ChecksumAlgorithm": "",
    "Initiator": {
        "ID": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4",
        "DisplayName": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"
    },
    "Owner": {
        "DisplayName": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4",
        "ID": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"
    },
    "StorageClass": "STANDARD"
}
```

(for whatever reason the python script generated a 2002 byte file ;)
2024-12-19 23:51:46 +05:30
Minio Trusted 06ddd8770e Update yaml files to latest version RELEASE.2024-12-18T13-15-44Z 2024-12-19 00:24:40 +00:00
Anis Eleuch 16f8cf1c52 heal: Include more use case of not healable but readable objects (#248) (#20776)
If one object has many parts where all parts are readable but some parts
are missing from some drives, this object can be sometimes un-healable,
which is wrong.

This commit will avoid reading from drives that have missing, corrupted or
outdated xl.meta. It will also check if any part is unreadable to avoid
healing in that case.
2024-12-18 05:15:44 -08:00
Mark Theunissen 01e520eb23 s3: Sanitize the source object name in CopyObject handler (#20774) 2024-12-17 07:01:07 -08:00
Harshavardhana 02f770a0c0 update all dependencies and use latest msgp (#20768) 2024-12-16 04:20:12 +05:30
dependabot[bot] 2f4c79bc0f Bump golang.org/x/crypto from 0.29.0 to 0.31.0 (#20767)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.29.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.29.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-14 23:04:23 -08:00
dependabot[bot] 969ee7dfbe Bump golang.org/x/crypto from 0.23.0 to 0.31.0 in /docs/debugging/inspect (#20760)
Bump golang.org/x/crypto in /docs/debugging/inspect

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.23.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.23.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-14 22:04:58 -08:00
Minio Trusted 5f0b086b05 Update yaml files to latest version RELEASE.2024-12-13T22-19-12Z 2024-12-15 03:52:57 +00:00
Artur Melanchyk 68b004a48f fix: specify size in some map allocations (#20764) 2024-12-13 14:19:12 -08:00
Harshavardhana 54ecce66f0 move readLock on uploadId at EOF in PutObjectPart (#258)
we do not need to hold the read locks at the higher
layer instead before reading the body, instead hold
the read locks properly at the time of renamePart()
for protection from racy part overwrites to compete
with concurrent completeMultipart().
2024-12-13 14:06:18 -08:00
Artur Melanchyk 2b008c598b fix: replace mutex with atomic (#20762) 2024-12-13 19:32:46 +05:30
dependabot[bot] 86d02b17cf Bump golang.org/x/crypto from 0.23.0 to 0.31.0 in /docs/debugging/s3-verify (#20757)
Bump golang.org/x/crypto in /docs/debugging/s3-verify

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.23.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.23.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-12 05:00:05 -08:00
Anis Eleuch c1a95a70ac heal: Move CheckParts from single handler to streaming RPC (#20755)
CheckParts call can take time to verify 10k parts of a single object in a single drive.
To avoid an internal dealine of one minute in the single handler RPC, this commit will
switch to streaming RPC instead.
2024-12-12 11:20:57 +05:30
Aditya Manthramurthy f246c9053f fix: Privilege escalation in IAM import API (#20756)
This API had missing permissions checking, allowing a user to change
their policy mapping by:

1. Craft iam-info.zip file: Update own user permission in
user_mappings.json
2. Upload it via `mc admin cluster iam import nobody iam-info.zip`

Here `nobody` can be a user with pretty much any kind of permission (but
not anonymous) and this ends up working.

Some more detailed steps - start from a fresh setup:

```
./minio server /tmp/d{1...4} &
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc admin user add myminio nobody nobody123
mc admin policy attach myminio readwrite nobody nobody123
mc alias set nobody http://localhost:9000 nobody nobody123

mc admin cluster iam export myminio
mkdir /tmp/x && mv myminio-iam-info.zip /tmp/x
cd /tmp/x
unzip myminio-iam-info.zip
echo '{"nobody":{"version":1,"policy":"consoleAdmin","updatedAt":"2024-08-13T19:47:10.1Z"}}' > \
      iam-assets/user_mappings.json
zip -r myminio-iam-info-updated.zip iam-assets/

mc admin cluster iam import nobody ./myminio-iam-info-updated.zip
mc admin service restart nobody
```
2024-12-12 07:39:40 +05:30
Cesar N. 9cdd204ae4 Upgrade Console version to v1.7.5 (#20748) 2024-12-11 16:23:53 +05:30
Harshavardhana 7b3eb9f7f8 fix: groups lookup performance issue with users with lots of groups (#20740)
fixes https://github.com/minio/minio/issues/20717
2024-12-11 16:23:28 +05:30
ebozduman d56ef8dbe1 Adds AIstore documentation link (#20738) 2024-12-11 16:22:17 +05:30
Mark Theunissen a248ed5ff5 Fixes for POST policy checks and the x-ignore implementation (#20674) 2024-12-11 16:21:34 +05:30
Klaus Post 5bb31e4883 Disable mint full object tests (#20743)
Remove expected failures from https://github.com/minio/minio-go/pull/2026
2024-12-09 18:59:22 -08:00
Taran Pelkey aff2a76d80 Return error when attempting to create a policy with commas in name (#20724) 2024-12-04 03:51:26 -08:00
Anis Eleuch eddbe6bca2 heal: Report bucket healing result correctly (#20721) 2024-12-04 04:42:25 +05:30
Anis Eleuch 734d1e320a heal: Single object heal to look for older versions as well (#203) (#20723)
`mc admin heal ALIAS/bucket/object` does not have any flag to heal
object noncurrent versions, this commit will make healing of the object
noncurrent versions implicitly asked.

This also fixes the 'mc admin heal ALIAS/bucket/object' that does not work 
correctly when the bucket is versioned. This has been broken since Apr 2023.
2024-12-04 04:42:04 +05:30
Anis Eleuch b8dab7b1a9 Set http server read/write timeout from --idle-timeout (#228) (#20715)
Golang http.Server will call SetReadDeadline overwriting the previous
deadline configuration set after a new connection Accept in the custom
listener code. Therefore, --idle-timeout was not correctly respected.

Make http.Server read/write timeout similar to --idle-timeout.
2024-12-02 18:51:17 +05:30
Klaus Post abd6bf060d Add 'X-Forwarded-For' to (s)FTP requests (#20709)
Fixes #20707
2024-11-29 18:25:37 +05:30
Alex f0d4ef604c Updated Console to v1.7.4 (#20693)
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2024-11-28 13:14:11 +05:30
Ramon de Klein 2712f75762 prevent IAM cleanup errors (#20691) 2024-11-28 13:13:51 +05:30
Krutika Dhananjay 4c46668da8 Add a test case for fix #20684 (#20688)
The test fails without the change.
Also, removed a duplicate test case involving lifecycle config with no rules.
2024-11-28 13:13:31 +05:30
Anis Eleuch 02e93fd6ba heal: Better reporting to mc with dangling/timeout errors (#20690)
The code assigns corrupted state to a drive for any unexpected error,
which is confusing for users. This change will make sure to assign
corrupted state only for corrupted parts or xl.meta. Use unknown state
with a explanation for any unexpected error, like canceled, deadline
errors, drive timeout, ...

Also make sure to return the bucket/object name when the object is not
found or marked not found by the heal dangling code.
2024-11-26 10:45:35 -08:00
Krutika Dhananjay 366876e98b Fix prefix validation in lifecycle rule (#20684) 2024-11-25 12:12:21 -08:00
Mark Theunissen d202fdd022 Add the policy name to the audit logs tags when doing policy-based API calls. Add retention settings to tags (#20638)
* Add the policy name to the audit log tags when doing policy-based API calls

* Audit log the retention settings requested in the API call

* Audit log of retention on PutObjectRetention API path too
2024-11-25 09:17:12 -08:00
Eng Zer Jun c07e5b49d4 refactor: replace experimental maps and slices with stdlib (#20679)
The experimental functions are now available in the standard library in
Go 1.23 [1].

[1]: https://go.dev/doc/go1.23#new-unique-package

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2024-11-25 09:10:22 -08:00
Aditya Manthramurthy 9a39f8ad4d fix: Remove User should fail for a service account (#20677)
The RemoveUser API only removes internal users, and it reports success
when it didnt find the internal user account for deletion. When provided
with a service account, it should not report success as that is misleading.
2024-11-21 18:24:04 -08:00
Nao Yonashiro 7e0c1c9413 feat: bump github.com/cosnicolaou/pbzip2 from 1.0.3 to 1.0.5 (#20671)
Update github.com/cosnicolaou/pbzip2 to latest version for
significant performance improvements. This update brings a 45%
reduction in processing time.
2024-11-20 18:53:52 -08:00
John Morales 485d833cd7 fix: API label casing and count value for +Inf bucket v2 metrics (#20656) 2024-11-18 11:17:52 -08:00
Klaus Post e8a476ef5a Keep larger merge buffers for RPC (#20654)
Keep larger merge buffers

When sending large messages >1K, the merge buffer would continuously be reallocated.

This could happen on listings, where blocks are typically 4->8K.

Keep merge buffer of up to 256KB.

Benchmark with 4096b messages:

```
benchmark                                          old ns/op     new ns/op     delta
BenchmarkRequests/servers=2/bytes/par=32-32        8271          6360          -23.10%
BenchmarkRequests/servers=2/bytes/par=64-32        7840          4731          -39.66%
BenchmarkRequests/servers=2/bytes/par=128-32       7291          4740          -34.99%
BenchmarkRequests/servers=2/bytes/par=256-32       7095          4580          -35.45%
BenchmarkRequests/servers=2/bytes/par=512-32       6757          4584          -32.16%
BenchmarkRequests/servers=2/bytes/par=1024-32      6429          4453          -30.74%

benchmark                                          old bytes     new bytes     delta
BenchmarkRequests/servers=2/bytes/par=32-32        12090         821           -93.21%
BenchmarkRequests/servers=2/bytes/par=64-32        17423         820           -95.29%
BenchmarkRequests/servers=2/bytes/par=128-32       18493         822           -95.56%
BenchmarkRequests/servers=2/bytes/par=256-32       18892         821           -95.65%
BenchmarkRequests/servers=2/bytes/par=512-32       19064         826           -95.67%
BenchmarkRequests/servers=2/bytes/par=1024-32      19038         842           -95.58%
```
2024-11-16 09:18:48 -08:00
Krutika Dhananjay 267f0ecea2 Fix 0 httpTimeout for logger webhook (#20653)
Previously, not setting http.Config.HTTPTimeout for logger webhook
was resulting in a timeout of 0, and causing "deadline exceeded"
errors in log webhook.

This change introduces a new env variable for configuring log webhook
timeout and more importantly sets it when the config is initialised.
2024-11-16 02:01:36 -08:00
Harshavardhana 4ee3434854 updating all dependencies as per regular cadence (#20646) 2024-11-14 12:33:18 -08:00
Anis Eleuch 0e9854372e heal/batch: Fix missing redirection to the first node (#20642)
Manual heal can return XMinioHealInvalidClientToken if the manual
healing is started in the first node, and the next mc call to get the
heal status is landed on another node. The reason is that redirection
based on the token ID is not able to redirect requests to the first node
due to a typo.

This also affects the batch cancel command if the batch is being done in
the first node, the user will never be able to cancel it due to the same
bug.
2024-11-13 04:07:28 -08:00
Klaus Post b5177993b3 Make DeadlineConn http.Listener compatible (#20635)
HTTP likes to slap an infinite read deadline on a connection and 
do a blocking read while the response is being written.

This effectively means that a reading deadline becomes the 
request-response deadline.

Instead of enforcing our timeout, we pass it through and keep 
"infinite deadline" is sticky on connections.

However, we still "record" when reads are aborted, so we never overwrite that.

The HTTP server should have `ReadTimeout` and `IdleTimeout` set for the deadline to be effective.

Use --idle-timeout for incoming connections.
2024-11-12 12:41:41 -08:00
Klaus Post 55f5c18fd9 Harden internode DeadlineConn (#20631)
Since DeadlineConn would send deadline updates directly upstream,
it would race with Read/Write operations. The stdlib will perform a read, 
but do an async SetReadDeadLine(unix(1)) to cancel the Read in 
`abortPendingRead`. In this case, the Read may override the 
deadline intended to cancel the read.

Stop updating deadlines if a deadline in the past is seen and when Close is called. 
A mutex now protects all upstream deadline calls to avoid races. 

This should fix the short-term buildup of...

```
365 @ 0x44112e 0x4756b9 0x475699 0x483525 0x732286 0x737407 0x73816b 0x479601
#	0x475698	sync.runtime_notifyListWait+0x138		runtime/sema.go:569
#	0x483524	sync.(*Cond).Wait+0x84				sync/cond.go:70
#	0x732285	net/http.(*connReader).abortPendingRead+0xa5	net/http/server.go:729
#	0x737406	net/http.(*response).finishRequest+0x86		net/http/server.go:1676
#	0x73816a	net/http.(*conn).serve+0x62a			net/http/server.go:2050
```

AFAICT Only affects internode calls that create a connection (non-grid).
2024-11-11 09:15:17 -08:00
Harshavardhana 8ce101c174 fix: LDAP service port number in tests 2024-11-11 07:01:29 -08:00
Klaus Post 4972735507 Fix lint issues from v1.62.0 upgrade (#20633)
* Fix lint issues from v1.62.0 upgrade

* Fix xlMetaV2TrimData version checks.
2024-11-11 06:51:43 -08:00
Minio Trusted e6ca6de194 Update yaml files to latest version RELEASE.2024-11-07T00-52-20Z 2024-11-07 23:58:42 +00:00
Harshavardhana cefc43e4da simplify the Get()/GetMultiple() re-use GetRaw() for both (#179)
Remember GetMultiple() must be used if your target is calling
PutMultiple(), without that the multiple events will not be
replayed.
2024-11-06 16:52:20 -08:00
Ramon de Klein 25e34fda5f decompress audit log properly before sending to remote target (#20619) 2024-11-06 13:25:24 -08:00
Erfan 4208d7af5a docs: remove redundant prometheus metric (#20618) 2024-11-06 07:44:21 -08:00
Klaus Post 8d42f37e4b Fix msgUnPath crash (#20614)
These are needed checks for the functions to be un-crashable with any input 
given to `msgUnPath` (tested with fuzzing).

Both conditions would result in a crash, which prevents that. Some 
additional upstream checks are needed.

Fixes #20610
2024-11-05 04:37:59 -08:00
dependabot[bot] 7cb4b5c636 Bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (#20611)
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.0...v4.5.1)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-05 02:22:36 -08:00
Harshavardhana 1615920f48 fix typos reported in CI/CD 2024-11-04 11:06:02 -08:00
Cesar N. 7ee42b3ff5 Update console package to v1.7.3 (#20606) 2024-11-04 08:47:08 -08:00
Harshavardhana a6f1e727fb add tests for ILM transition and healing (#166) (#20601)
This PR fixes a regression introduced in https://github.com/minio/minio/pull/19797
by restoring the healing ability of transitioned objects

Bonus: support for transitioned objects to carry original
The object name is for future reverse lookups if necessary.

Also fix parity calculation for tiered objects to n/2 for n/2 == (parity)
2024-10-31 15:10:24 -07:00
Aditya Manthramurthy c1fc7779ca Remove expires field from list objects metadata (#20600)
This field was always 0 regardless of whether the object had an expiry
so we are basically removing dead code.
2024-10-31 12:27:06 -07:00
Allan Roger Reid b3ab7546ee Fix typos in README.md (#20599) 2024-10-31 10:38:53 -07:00
Minio Trusted ad88a81e3d Update yaml files to latest version RELEASE.2024-10-29T16-01-48Z 2024-10-30 21:24:58 +00:00
164 changed files with 9370 additions and 2936 deletions
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
+39
View File
@@ -0,0 +1,39 @@
name: Resiliency Functional Tests
on:
pull_request:
branches:
- master
# This ensures that previous jobs for the PR are canceled when the PR is
# updated.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
build:
name: Go ${{ matrix.go-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
check-latest: true
- name: Build on ${{ matrix.os }}
if: matrix.os == 'ubuntu-latest'
env:
CGO_ENABLED: 0
GO111MODULE: on
run: |
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-resiliency
+2 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
- uses: actions/checkout@v4
@@ -39,3 +39,4 @@ jobs:
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make verify
make test-timeout
+1 -1
View File
@@ -61,7 +61,7 @@ jobs:
# are turned off - i.e. if ldap="", then ldap server is not enabled for
# the tests.
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
ldap: ["", "localhost:389"]
etcd: ["", "http://localhost:2379"]
openid: ["", "http://127.0.0.1:5556/dex"]
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
- name: setup-go-step
uses: actions/setup-go@v5
with:
go-version: 1.22.x
go-version: 1.23.x
- name: github sha short
id: vars
+2 -1
View File
@@ -21,7 +21,7 @@ jobs:
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
steps:
- uses: actions/checkout@v4
@@ -40,6 +40,7 @@ jobs:
sudo sysctl net.ipv6.conf.all.disable_ipv6=0
sudo sysctl net.ipv6.conf.default.disable_ipv6=0
make test-ilm
make test-ilm-transition
- name: Test PBAC
run: |
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
+3
View File
@@ -15,6 +15,9 @@ docker volume rm $(docker volume ls -f dangling=true) || true
## change working directory
cd .github/workflows/mint
## always pull latest
docker pull docker.io/minio/mint:edge
docker-compose -f minio-${MODE}.yaml up -d
sleep 1m
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
go-version: [1.22.x]
go-version: [1.23.x]
os: [ubuntu-latest]
steps:
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: 1.22.7
go-version: 1.23.5
- name: Get official govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest
shell: bash
+3346 -62
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.22-alpine as build
FROM golang:1.23-alpine as build
ARG TARGETARCH
ARG RELEASE
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.22-alpine AS build
FROM golang:1.23-alpine AS build
ARG TARGETARCH
ARG RELEASE
-69
View File
@@ -1,69 +0,0 @@
FROM golang:1.22-alpine AS build
ARG TARGETARCH
ARG RELEASE
ENV GOPATH=/go
ENV CGO_ENABLED=0
# Install curl and minisign
RUN apk add -U --no-cache ca-certificates && \
apk add -U --no-cache curl && \
go install aead.dev/minisign/cmd/minisign@v0.2.1
# Download minio binary and signature files
RUN curl -s -q https://dl.min.io/server/minio/release/linux-${TARGETARCH}/archive/minio.${RELEASE}.fips -o /go/bin/minio && \
curl -s -q https://dl.min.io/server/minio/release/linux-${TARGETARCH}/archive/minio.${RELEASE}.fips.minisig -o /go/bin/minio.minisig && \
curl -s -q https://dl.min.io/server/minio/release/linux-${TARGETARCH}/archive/minio.${RELEASE}.fips.sha256sum -o /go/bin/minio.sha256sum && \
chmod +x /go/bin/minio
# Download mc binary and signature files
RUN curl -s -q https://dl.min.io/client/mc/release/linux-${TARGETARCH}/mc.fips -o /go/bin/mc && \
curl -s -q https://dl.min.io/client/mc/release/linux-${TARGETARCH}/mc.fips.minisig -o /go/bin/mc.minisig && \
curl -s -q https://dl.min.io/client/mc/release/linux-${TARGETARCH}/mc.fips.sha256sum -o /go/bin/mc.sha256sum && \
chmod +x /go/bin/mc
RUN if [ "$TARGETARCH" = "amd64" ]; then \
curl -L -s -q https://github.com/moparisthebest/static-curl/releases/latest/download/curl-${TARGETARCH} -o /go/bin/curl; \
chmod +x /go/bin/curl; \
fi
# Verify binary signature using public key "RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGavRUN"
RUN minisign -Vqm /go/bin/minio -x /go/bin/minio.minisig -P RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav
FROM registry.access.redhat.com/ubi9/ubi-micro:latest
ARG RELEASE
LABEL name="MinIO" \
vendor="MinIO Inc <dev@min.io>" \
maintainer="MinIO Inc <dev@min.io>" \
version="${RELEASE}" \
release="${RELEASE}" \
summary="MinIO is a High Performance Object Storage, API compatible with Amazon S3 cloud storage service." \
description="MinIO object storage is fundamentally different. Designed for performance and the S3 API, it is 100% open-source. MinIO is ideal for large, private cloud environments with stringent security requirements and delivers mission-critical availability across a diverse range of workloads."
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key \
MINIO_ROOT_USER_FILE=access_key \
MINIO_ROOT_PASSWORD_FILE=secret_key \
MINIO_KMS_SECRET_KEY_FILE=kms_master_key \
MINIO_UPDATE_MINISIGN_PUBKEY="RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav" \
MINIO_CONFIG_ENV_FILE=config.env
RUN chmod -R 777 /usr/bin
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /go/bin/minio* /usr/bin/
COPY --from=build /go/bin/mc* /usr/bin/
COPY --from=build /go/bin/cur* /usr/bin/
COPY CREDITS /licenses/CREDITS
COPY LICENSE /licenses/LICENSE
COPY dockerscripts/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
EXPOSE 9000
VOLUME ["/data"]
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
CMD ["minio"]
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.22-alpine AS build
FROM golang:1.23-alpine AS build
ARG TARGETARCH
ARG RELEASE
+13 -1
View File
@@ -24,7 +24,7 @@ help: ## print this help
getdeps: ## fetch necessary dependencies
@mkdir -p ${GOPATH}/bin
@echo "Installing golangci-lint" && curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(GOLANGCI_DIR)
@echo "Installing msgp" && go install -v github.com/tinylib/msgp@v1.2.3-0.20241022140105-4558fbf3a223
@echo "Installing msgp" && go install -v github.com/tinylib/msgp@v1.2.5
@echo "Installing stringer" && go install -v golang.org/x/tools/cmd/stringer@latest
crosscompile: ## cross compile minio
@@ -60,6 +60,10 @@ test-ilm: install-race
@echo "Running ILM tests"
@env bash $(PWD)/docs/bucket/replication/setup_ilm_expiry_replication.sh
test-ilm-transition: install-race
@echo "Running ILM tiering tests with healing"
@env bash $(PWD)/docs/bucket/lifecycle/setup_ilm_transition.sh
test-pbac: install-race
@echo "Running bucket policies tests"
@env bash $(PWD)/docs/iam/policies/pbac-tests.sh
@@ -145,6 +149,10 @@ test-multipart: install-race ## test multipart
@echo "Test multipart behavior when part files are missing"
@(env bash $(PWD)/buildscripts/multipart-quorum-test.sh)
test-timeout: install-race ## test multipart
@echo "Test server timeout"
@(env bash $(PWD)/buildscripts/test-timeout.sh)
verify: install-race ## verify minio various setups
@echo "Verifying build with race"
@(env bash $(PWD)/buildscripts/verify-build.sh)
@@ -209,6 +217,10 @@ docker: build ## builds minio docker container
@echo "Building minio docker image '$(TAG)'"
@docker build -q --no-cache -t $(TAG) . -f Dockerfile
test-resiliency: build
@echo "Running resiliency tests"
@(DOCKER_COMPOSE_FILE=$(PWD)/docs/resiliency/docker-compose.yaml env bash $(PWD)/docs/resiliency/resiliency-tests.sh)
install-race: checks build-debugging ## builds minio to $(PWD)
@echo "Building minio binary with -race to './minio'"
@GORACE=history_size=7 CGO_ENABLED=1 go build -tags kqueue,dev -race -trimpath --ldflags "$(LDFLAGS)" -o $(PWD)/minio 1>/dev/null
+1 -1
View File
@@ -4,7 +4,7 @@
[![MinIO](https://raw.githubusercontent.com/minio/minio/master/.github/logo.svg?sanitize=true)](https://min.io)
MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.
MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads. To learn more about what MinIO is doing for AI storage, go to [AI storage documentation](https://min.io/solutions/object-storage-for-ai).
This README provides quickstart instructions on running MinIO on bare metal hardware, including container-based installations. For Kubernetes environments, use the [MinIO Kubernetes Operator](https://github.com/minio/operator/blob/master/README.md).
@@ -8,7 +8,7 @@
#
# This script assumes that LDAP server is at:
#
# `localhost:1389`
# `localhost:389`
#
# if this is not the case, set the environment variable
# `_MINIO_LDAP_TEST_SERVER`.
@@ -41,7 +41,7 @@ __init__() {
fi
if [ -z "$_MINIO_LDAP_TEST_SERVER" ]; then
export _MINIO_LDAP_TEST_SERVER=localhost:1389
export _MINIO_LDAP_TEST_SERVER=localhost:389
echo "Using default LDAP endpoint: $_MINIO_LDAP_TEST_SERVER"
fi
@@ -58,7 +58,7 @@ create_iam_content_in_old_minio() {
mc alias set old-minio http://localhost:9000 minioadmin minioadmin
mc ready old-minio
mc idp ldap add old-minio \
server_addr=localhost:1389 \
server_addr=localhost:389 \
server_insecure=on \
lookup_bind_dn=cn=admin,dc=min,dc=io \
lookup_bind_password=admin \
+137
View File
@@ -0,0 +1,137 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
WORK_DIR="$PWD/.verify-$RANDOM"
MINIO_CONFIG_DIR="$WORK_DIR/.minio"
MINIO=("$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" server)
if [ ! -x "$PWD/minio" ]; then
echo "minio executable binary not found in current directory"
exit 1
fi
if [ ! -x "$PWD/minio" ]; then
echo "minio executable binary not found in current directory"
exit 1
fi
trap 'catch $LINENO' ERR
function purge() {
rm -rf "$1"
}
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
fi
echo "Cleaning up instances of MinIO"
pkill minio || true
pkill -9 minio || true
purge "$WORK_DIR"
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
function gen_put_request() {
hdr_sleep=$1
body_sleep=$2
echo "PUT /testbucket/testobject HTTP/1.1"
sleep $hdr_sleep
echo "Host: foo-header"
echo "User-Agent: curl/8.2.1"
echo "Accept: */*"
echo "Content-Length: 30"
echo ""
sleep $body_sleep
echo "random line 0"
echo "random line 1"
echo ""
echo ""
}
function send_put_object_request() {
hdr_timeout=$1
body_timeout=$2
start=$(date +%s)
timeout 5m bash -c "gen_put_request $hdr_timeout $body_timeout | netcat 127.0.0.1 $start_port | read" || return -1
[ $(($(date +%s) - start)) -gt $((srv_hdr_timeout + srv_idle_timeout + 1)) ] && return -1
return 0
}
function test_minio_with_timeout() {
start_port=$1
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123
export MC_HOST_minio="http://minio:minio123@127.0.0.1:${start_port}/"
export MINIO_CI_CD=1
mkdir ${WORK_DIR}
C_PWD=${PWD}
if [ ! -x "$PWD/mc" ]; then
MC_BUILD_DIR="mc-$RANDOM"
if ! git clone --quiet https://github.com/minio/mc "$MC_BUILD_DIR"; then
echo "failed to download https://github.com/minio/mc"
purge "${MC_BUILD_DIR}"
exit 1
fi
(cd "${MC_BUILD_DIR}" && go build -o "$C_PWD/mc")
# remove mc source.
purge "${MC_BUILD_DIR}"
fi
"${MINIO[@]}" --address ":$start_port" --read-header-timeout ${srv_hdr_timeout}s --idle-timeout ${srv_idle_timeout}s "${WORK_DIR}/disk/" >"${WORK_DIR}/server1.log" 2>&1 &
pid=$!
disown $pid
sleep 1
if ! ps -p ${pid} 1>&2 >/dev/null; then
echo "server1 log:"
cat "${WORK_DIR}/server1.log"
echo "FAILED"
purge "$WORK_DIR"
exit 1
fi
set -e
"${PWD}/mc" mb minio/testbucket
"${PWD}/mc" anonymous set public minio/testbucket
# slow header writing
send_put_object_request 20 0 && exit -1
"${PWD}/mc" stat minio/testbucket/testobject && exit -1
# quick header write and slow bodywrite
send_put_object_request 0 40 && exit -1
"${PWD}/mc" stat minio/testbucket/testobject && exit -1
# quick header and body write
send_put_object_request 1 1 || exit -1
"${PWD}/mc" stat minio/testbucket/testobject || exit -1
}
function main() {
export start_port=$(shuf -i 10000-65000 -n 1)
export srv_hdr_timeout=5
export srv_idle_timeout=5
export -f gen_put_request
test_minio_with_timeout ${start_port}
}
main "$@"
+3 -3
View File
@@ -259,7 +259,7 @@ func (a adminAPIHandlers) RebalanceStart(w http.ResponseWriter, r *http.Request)
if ep := globalEndpoints[0].Endpoints[0]; !ep.IsLocal {
for nodeIdx, proxyEp := range globalProxyEndpoints {
if proxyEp.Endpoint.Host == ep.Host {
if proxyRequestByNodeIndex(ctx, w, r, nodeIdx) {
if proxied, success := proxyRequestByNodeIndex(ctx, w, r, nodeIdx, false); proxied && success {
return
}
}
@@ -330,7 +330,7 @@ func (a adminAPIHandlers) RebalanceStatus(w http.ResponseWriter, r *http.Request
if ep := globalEndpoints[0].Endpoints[0]; !ep.IsLocal {
for nodeIdx, proxyEp := range globalProxyEndpoints {
if proxyEp.Endpoint.Host == ep.Host {
if proxyRequestByNodeIndex(ctx, w, r, nodeIdx) {
if proxied, success := proxyRequestByNodeIndex(ctx, w, r, nodeIdx, false); proxied && success {
return
}
}
@@ -384,7 +384,7 @@ func proxyDecommissionRequest(ctx context.Context, defaultEndPoint Endpoint, w h
}
for nodeIdx, proxyEp := range globalProxyEndpoints {
if proxyEp.Endpoint.Host == host && !proxyEp.IsLocal {
if proxyRequestByNodeIndex(ctx, w, r, nodeIdx) {
if proxied, success := proxyRequestByNodeIndex(ctx, w, r, nodeIdx, false); proxied && success {
return true
}
}
+35 -45
View File
@@ -37,6 +37,7 @@ import (
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/auth"
"github.com/minio/minio/internal/config/dns"
"github.com/minio/minio/internal/logger"
"github.com/minio/mux"
xldap "github.com/minio/pkg/v3/ldap"
"github.com/minio/pkg/v3/policy"
@@ -65,6 +66,17 @@ func (a adminAPIHandlers) RemoveUser(w http.ResponseWriter, r *http.Request) {
return
}
// This API only supports removal of internal users not service accounts.
ok, _, err = globalIAMSys.IsServiceAccount(accessKey)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
if ok {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, errIAMActionNotAllowed), r.URL)
return
}
// When the user is root credential you are not allowed to
// remove the root user. Also you cannot delete yourself.
if accessKey == globalActiveCred.AccessKey || accessKey == cred.AccessKey {
@@ -1568,6 +1580,7 @@ func (a adminAPIHandlers) InfoCannedPolicy(w http.ResponseWriter, r *http.Reques
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, errTooManyPolicies), r.URL)
return
}
setReqInfoPolicyName(ctx, name)
policyDoc, err := globalIAMSys.InfoPolicy(name)
if err != nil {
@@ -1671,6 +1684,7 @@ func (a adminAPIHandlers) RemoveCannedPolicy(w http.ResponseWriter, r *http.Requ
vars := mux.Vars(r)
policyName := vars["name"]
setReqInfoPolicyName(ctx, policyName)
if err := globalIAMSys.DeletePolicy(ctx, policyName, true); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
@@ -1703,6 +1717,13 @@ func (a adminAPIHandlers) AddCannedPolicy(w http.ResponseWriter, r *http.Request
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminResourceInvalidArgument), r.URL)
return
}
setReqInfoPolicyName(ctx, policyName)
// Reject policy names with commas.
if strings.Contains(policyName, ",") {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrPolicyInvalidName), r.URL)
return
}
// Error out if Content-Length is missing.
if r.ContentLength <= 0 {
@@ -1768,6 +1789,7 @@ func (a adminAPIHandlers) SetPolicyForUserOrGroup(w http.ResponseWriter, r *http
policyName := vars["policyName"]
entityName := vars["userOrGroup"]
isGroup := vars["isGroup"] == "true"
setReqInfoPolicyName(ctx, policyName)
if !isGroup {
ok, _, err := globalIAMSys.IsTempUser(entityName)
@@ -1853,7 +1875,7 @@ func (a adminAPIHandlers) SetPolicyForUserOrGroup(w http.ResponseWriter, r *http
}))
}
// ListPolicyMappingEntities - GET /minio/admin/v3/idp/builtin/polciy-entities?policy=xxx&user=xxx&group=xxx
// ListPolicyMappingEntities - GET /minio/admin/v3/idp/builtin/policy-entities?policy=xxx&user=xxx&group=xxx
func (a adminAPIHandlers) ListPolicyMappingEntities(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
@@ -1955,6 +1977,7 @@ func (a adminAPIHandlers) AttachDetachPolicyBuiltin(w http.ResponseWriter, r *ht
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
setReqInfoPolicyName(ctx, strings.Join(addedOrRemoved, ","))
respBody := madmin.PolicyAssociationResp{
UpdatedAt: updatedAt,
@@ -2009,6 +2032,7 @@ func (a adminAPIHandlers) ExportIAM(w http.ResponseWriter, r *http.Request) {
// Get current object layer instance.
objectAPI, _ := validateAdminReq(ctx, w, r, policy.ExportIAMAction)
if objectAPI == nil {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL)
return
}
// Initialize a zip writer which will provide a zipped content
@@ -2231,17 +2255,13 @@ func (a adminAPIHandlers) ImportIAMV2(w http.ResponseWriter, r *http.Request) {
func (a adminAPIHandlers) importIAM(w http.ResponseWriter, r *http.Request, apiVer string) {
ctx := r.Context()
// Get current object layer instance.
objectAPI := newObjectLayerFn()
// Validate signature, permissions and get current object layer instance.
objectAPI, _ := validateAdminReq(ctx, w, r, policy.ImportIAMAction)
if objectAPI == nil || globalNotificationSys == nil {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL)
return
}
cred, owner, s3Err := validateAdminSignature(ctx, r, "")
if s3Err != ErrNone {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
return
}
data, err := io.ReadAll(r.Body)
if err != nil {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidRequest), r.URL)
@@ -2331,38 +2351,12 @@ func (a adminAPIHandlers) importIAM(w http.ResponseWriter, r *http.Request, apiV
return
}
if (cred.IsTemp() || cred.IsServiceAccount()) && cred.ParentUser == accessKey {
// Incoming access key matches parent user then we should
// reject password change requests.
writeErrorResponseJSON(ctx, w, importErrorWithAPIErr(ctx, ErrAddUserInvalidArgument, err, allUsersFile, accessKey), r.URL)
return
}
// Check if accessKey has beginning and end space characters, this only applies to new users.
if !exists && hasSpaceBE(accessKey) {
writeErrorResponseJSON(ctx, w, importErrorWithAPIErr(ctx, ErrAdminResourceInvalidArgument, err, allUsersFile, accessKey), r.URL)
return
}
checkDenyOnly := false
if accessKey == cred.AccessKey {
// Check that there is no explicit deny - otherwise it's allowed
// to change one's own password.
checkDenyOnly = true
}
if !globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.CreateUserAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
DenyOnly: checkDenyOnly,
}) {
writeErrorResponseJSON(ctx, w, importErrorWithAPIErr(ctx, ErrAccessDenied, err, allUsersFile, accessKey), r.URL)
return
}
if _, err = globalIAMSys.CreateUser(ctx, accessKey, ureq); err != nil {
failed.Users = append(failed.Users, madmin.IAMErrEntity{Name: accessKey, Error: err})
} else {
@@ -2462,17 +2456,6 @@ func (a adminAPIHandlers) importIAM(w http.ResponseWriter, r *http.Request, apiV
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminResourceInvalidArgument), r.URL)
return
}
if !globalIAMSys.IsAllowed(policy.Args{
AccountName: cred.AccessKey,
Groups: cred.Groups,
Action: policy.CreateServiceAccountAdminAction,
ConditionValues: getConditionValues(r, "", cred),
IsOwner: owner,
Claims: cred.Claims,
}) {
writeErrorResponseJSON(ctx, w, importErrorWithAPIErr(ctx, ErrAccessDenied, err, allSvcAcctsFile, user), r.URL)
return
}
updateReq := true
_, _, err = globalIAMSys.GetServiceAccount(ctx, svcAcctReq.AccessKey)
if err != nil {
@@ -2801,3 +2784,10 @@ func commonAddServiceAccount(r *http.Request, ldap bool) (context.Context, auth.
return ctx, cred, opts, createReq, targetUser, APIError{}
}
// setReqInfoPolicyName will set the given policyName as a tag on the context's request info,
// so that it appears in audit logs.
func setReqInfoPolicyName(ctx context.Context, policyName string) {
reqInfo := logger.GetReqInfo(ctx)
reqInfo.SetTags("policyName", policyName)
}
+6
View File
@@ -716,6 +716,12 @@ func (s *TestSuiteIAM) TestCannedPolicies(c *check) {
c.Fatalf("policy info err: %v", err)
}
// Check that policy with comma is rejected.
err = s.adm.AddCannedPolicy(ctx, "invalid,policy", policyBytes)
if err == nil {
c.Fatalf("invalid policy created successfully")
}
infoStr := string(info)
if !strings.Contains(infoStr, `"s3:PutObject"`) || !strings.Contains(infoStr, ":"+bucket+"/") {
c.Fatalf("policy contains unexpected content!")
+1 -1
View File
@@ -1320,7 +1320,7 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
}
// Analyze the heal token and route the request accordingly
token, success := proxyRequestByToken(ctx, w, r, hip.clientToken)
token, _, success := proxyRequestByToken(ctx, w, r, hip.clientToken, false)
if success {
return
}
+1 -10
View File
@@ -841,6 +841,7 @@ func (h *healSequence) healMinioSysMeta(objAPI ObjectLayer, metaPrefix string) f
// NOTE: Healing on meta is run regardless
// of any bucket being selected, this is to ensure that
// meta are always upto date and correct.
h.settings.Recursive = true
return objAPI.HealObjects(h.ctx, minioMetaBucket, metaPrefix, h.settings, func(bucket, object, versionID string, scanMode madmin.HealScanMode) error {
if h.isQuitting() {
return errHealStopSignalled
@@ -896,16 +897,6 @@ func (h *healSequence) healBucket(objAPI ObjectLayer, bucket string, bucketsOnly
return nil
}
if !h.settings.Recursive {
if h.object != "" {
if err := h.healObject(bucket, h.object, "", h.settings.ScanMode); err != nil {
return err
}
}
return nil
}
if err := objAPI.HealObjects(h.ctx, bucket, h.object, h.settings, h.healObject); err != nil {
return errFnHealFromAPIErr(h.ctx, err)
}
+2
View File
@@ -32,6 +32,8 @@ type DeletedObject struct {
DeleteMarkerMTime DeleteMarkerMTime `xml:"-"`
// MinIO extensions to support delete marker replication
ReplicationState ReplicationState `xml:"-"`
found bool // the object was found during deletion
}
// DeleteMarkerMTime is an embedded type containing time.Time for XML marshal
+7 -1
View File
@@ -213,6 +213,7 @@ const (
ErrPolicyAlreadyAttached
ErrPolicyNotAttached
ErrExcessData
ErrPolicyInvalidName
// Add new error codes here.
// SSE-S3/SSE-KMS related API errors
@@ -561,6 +562,11 @@ var errorCodes = errorCodeMap{
Description: "More data provided than indicated content length",
HTTPStatusCode: http.StatusBadRequest,
},
ErrPolicyInvalidName: {
Code: "PolicyInvalidName",
Description: "Policy name may not contain comma",
HTTPStatusCode: http.StatusBadRequest,
},
ErrPolicyTooLarge: {
Code: "PolicyTooLarge",
Description: "Policy exceeds the maximum allowed document size.",
@@ -623,7 +629,7 @@ var errorCodes = errorCodeMap{
},
ErrMissingContentMD5: {
Code: "MissingContentMD5",
Description: "Missing required header for this request: Content-Md5.",
Description: "Missing or invalid required header for this request: Content-Md5 or Amz-Content-Checksum",
HTTPStatusCode: http.StatusBadRequest,
},
ErrMissingSecurityHeader: {
+20 -19
View File
@@ -166,10 +166,11 @@ type Part struct {
Size int64
// Checksum values
ChecksumCRC32 string `xml:"ChecksumCRC32,omitempty"`
ChecksumCRC32C string `xml:"ChecksumCRC32C,omitempty"`
ChecksumSHA1 string `xml:"ChecksumSHA1,omitempty"`
ChecksumSHA256 string `xml:"ChecksumSHA256,omitempty"`
ChecksumCRC32 string `xml:"ChecksumCRC32,omitempty"`
ChecksumCRC32C string `xml:"ChecksumCRC32C,omitempty"`
ChecksumSHA1 string `xml:"ChecksumSHA1,omitempty"`
ChecksumSHA256 string `xml:"ChecksumSHA256,omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
}
// ListPartsResponse - format for list parts response.
@@ -192,6 +193,8 @@ type ListPartsResponse struct {
IsTruncated bool
ChecksumAlgorithm string
ChecksumType string
// List of parts.
Parts []Part `xml:"Part"`
}
@@ -413,10 +416,11 @@ type CompleteMultipartUploadResponse struct {
Key string
ETag string
ChecksumCRC32 string `xml:"ChecksumCRC32,omitempty"`
ChecksumCRC32C string `xml:"ChecksumCRC32C,omitempty"`
ChecksumSHA1 string `xml:"ChecksumSHA1,omitempty"`
ChecksumSHA256 string `xml:"ChecksumSHA256,omitempty"`
ChecksumCRC32 string `xml:"ChecksumCRC32,omitempty"`
ChecksumCRC32C string `xml:"ChecksumCRC32C,omitempty"`
ChecksumSHA1 string `xml:"ChecksumSHA1,omitempty"`
ChecksumSHA256 string `xml:"ChecksumSHA256,omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
}
// DeleteError structure.
@@ -593,9 +597,6 @@ func generateListVersionsResponse(ctx context.Context, bucket, prefix, marker, v
for k, v := range cleanReservedKeys(object.UserDefined) {
content.UserMetadata.Set(k, v)
}
if !object.Expires.IsZero() {
content.UserMetadata.Set("expires", object.Expires.Format(http.TimeFormat))
}
content.Internal = &ObjectInternalInfo{
K: object.DataBlocks,
M: object.ParityBlocks,
@@ -730,9 +731,6 @@ func generateListObjectsV2Response(ctx context.Context, bucket, prefix, token, n
for k, v := range cleanReservedKeys(object.UserDefined) {
content.UserMetadata.Set(k, v)
}
if !object.Expires.IsZero() {
content.UserMetadata.Set("expires", object.Expires.Format(http.TimeFormat))
}
content.Internal = &ObjectInternalInfo{
K: object.DataBlocks,
M: object.ParityBlocks,
@@ -799,11 +797,12 @@ func generateCompleteMultipartUploadResponse(bucket, key, location string, oi Ob
Bucket: bucket,
Key: key,
// AWS S3 quotes the ETag in XML, make sure we are compatible here.
ETag: "\"" + oi.ETag + "\"",
ChecksumSHA1: cs[hash.ChecksumSHA1.String()],
ChecksumSHA256: cs[hash.ChecksumSHA256.String()],
ChecksumCRC32: cs[hash.ChecksumCRC32.String()],
ChecksumCRC32C: cs[hash.ChecksumCRC32C.String()],
ETag: "\"" + oi.ETag + "\"",
ChecksumSHA1: cs[hash.ChecksumSHA1.String()],
ChecksumSHA256: cs[hash.ChecksumSHA256.String()],
ChecksumCRC32: cs[hash.ChecksumCRC32.String()],
ChecksumCRC32C: cs[hash.ChecksumCRC32C.String()],
ChecksumCRC64NVME: cs[hash.ChecksumCRC64NVME.String()],
}
return c
}
@@ -831,6 +830,7 @@ func generateListPartsResponse(partsInfo ListPartsInfo, encodingType string) Lis
listPartsResponse.IsTruncated = partsInfo.IsTruncated
listPartsResponse.NextPartNumberMarker = partsInfo.NextPartNumberMarker
listPartsResponse.ChecksumAlgorithm = partsInfo.ChecksumAlgorithm
listPartsResponse.ChecksumType = partsInfo.ChecksumType
listPartsResponse.Parts = make([]Part, len(partsInfo.Parts))
for index, part := range partsInfo.Parts {
@@ -843,6 +843,7 @@ func generateListPartsResponse(partsInfo ListPartsInfo, encodingType string) Lis
newPart.ChecksumCRC32C = part.ChecksumCRC32C
newPart.ChecksumSHA1 = part.ChecksumSHA1
newPart.ChecksumSHA256 = part.ChecksumSHA256
newPart.ChecksumCRC64NVME = part.ChecksumCRC64NVME
listPartsResponse.Parts[index] = newPart
}
return listPartsResponse
+200 -199
View File
File diff suppressed because one or more lines are too long
+13 -10
View File
@@ -289,6 +289,16 @@ type BatchJobExpire struct {
var _ yaml.Unmarshaler = &BatchJobExpire{}
// RedactSensitive will redact any sensitive information in b.
func (r *BatchJobExpire) RedactSensitive() {
if r == nil {
return
}
if r.NotificationCfg.Token != "" {
r.NotificationCfg.Token = redactedText
}
}
// UnmarshalYAML - BatchJobExpire extends default unmarshal to extract line, col information.
func (r *BatchJobExpire) UnmarshalYAML(val *yaml.Node) error {
type expireJob BatchJobExpire
@@ -435,14 +445,14 @@ func batchObjsForDelete(ctx context.Context, r *BatchJobExpire, ri *batchJobInfo
oiCache.Add(od, &exp.ObjectInfo)
}
var done bool
// DeleteObject(deletePrefix: true) to expire all versions of an object
for _, exp := range toExpireAll {
var success bool
for attempts := 1; attempts <= retryAttempts; attempts++ {
select {
case <-ctx.Done():
done = true
ri.trackMultipleObjectVersions(exp, success)
return
default:
}
stopFn := globalBatchJobsMetrics.trace(batchJobMetricExpire, ri.JobID, attempts)
@@ -459,14 +469,7 @@ func batchObjsForDelete(ctx context.Context, r *BatchJobExpire, ri *batchJobInfo
break
}
}
ri.trackMultipleObjectVersions(r.Bucket, exp, success)
if done {
break
}
}
if done {
return
ri.trackMultipleObjectVersions(exp, success)
}
// DeleteMultiple objects
+29 -2
View File
@@ -61,6 +61,8 @@ var globalBatchConfig batch.Config
const (
// Keep the completed/failed job stats 3 days before removing it
oldJobsExpiration = 3 * 24 * time.Hour
redactedText = "**REDACTED**"
)
// BatchJobRequest this is an internal data structure not for external consumption.
@@ -74,6 +76,29 @@ type BatchJobRequest struct {
ctx context.Context `msg:"-"`
}
// RedactSensitive will redact any sensitive information in b.
func (j *BatchJobRequest) RedactSensitive() {
j.Replicate.RedactSensitive()
j.Expire.RedactSensitive()
j.KeyRotate.RedactSensitive()
}
// RedactSensitive will redact any sensitive information in b.
func (r *BatchJobReplicateV1) RedactSensitive() {
if r == nil {
return
}
if r.Target.Creds.SecretKey != "" {
r.Target.Creds.SecretKey = redactedText
}
if r.Target.Creds.SessionToken != "" {
r.Target.Creds.SessionToken = redactedText
}
}
// RedactSensitive will redact any sensitive information in b.
func (r *BatchJobKeyRotateV1) RedactSensitive() {}
func notifyEndpoint(ctx context.Context, ri *batchJobInfo, endpoint, token string) error {
if endpoint == "" {
return nil
@@ -971,7 +996,7 @@ func (ri *batchJobInfo) updateAfter(ctx context.Context, api ObjectLayer, durati
// Note: to be used only with batch jobs that affect multiple versions through
// a single action. e.g batch-expire has an option to expire all versions of an
// object which matches the given filters.
func (ri *batchJobInfo) trackMultipleObjectVersions(bucket string, info ObjectInfo, success bool) {
func (ri *batchJobInfo) trackMultipleObjectVersions(info ObjectInfo, success bool) {
if success {
ri.Objects += int64(info.NumVersions)
} else {
@@ -1695,6 +1720,8 @@ func (a adminAPIHandlers) DescribeBatchJob(w http.ResponseWriter, r *http.Reques
return
}
// Remove sensitive fields.
req.RedactSensitive()
buf, err := yaml.Marshal(req)
if err != nil {
batchLogIf(ctx, err)
@@ -1802,7 +1829,7 @@ func (a adminAPIHandlers) CancelBatchJob(w http.ResponseWriter, r *http.Request)
return
}
if _, success := proxyRequestByToken(ctx, w, r, jobID); success {
if _, proxied, _ := proxyRequestByToken(ctx, w, r, jobID, true); proxied {
return
}
+11
View File
@@ -20,6 +20,7 @@ package cmd
import (
"bytes"
"context"
"errors"
"hash"
"io"
"sync"
@@ -37,12 +38,22 @@ type streamingBitrotWriter struct {
shardSize int64
canClose *sync.WaitGroup
byteBuf []byte
finished bool
}
func (b *streamingBitrotWriter) Write(p []byte) (int, error) {
if len(p) == 0 {
return 0, nil
}
if b.finished {
return 0, errors.New("bitrot write not allowed")
}
if int64(len(p)) > b.shardSize {
return 0, errors.New("unexpected bitrot buffer size")
}
if int64(len(p)) < b.shardSize {
b.finished = true
}
b.h.Reset()
b.h.Write(p)
hashBytes := b.h.Sum(nil)
+6 -1
View File
@@ -429,7 +429,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
// Content-Md5 is required should be set
// http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
if _, ok := r.Header[xhttp.ContentMD5]; !ok {
if !validateLengthAndChecksum(r) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL)
return
}
@@ -1338,6 +1338,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
}
opts.EncryptFn = metadataEncrypter(objectEncryptionKey)
pReader, err = pReader.WithEncryption(hashReader, &objectEncryptionKey)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
@@ -1809,6 +1810,10 @@ func (api objectAPIHandlers) PutBucketObjectLockConfigHandler(w http.ResponseWri
return
}
// Audit log tags.
reqInfo := logger.GetReqInfo(ctx)
reqInfo.SetTags("retention", config.String())
configData, err := xml.Marshal(config)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
+2 -3
View File
@@ -19,7 +19,6 @@ package cmd
import (
"encoding/xml"
"io"
"net/http"
"strconv"
"time"
@@ -53,7 +52,7 @@ func (api objectAPIHandlers) PutBucketLifecycleHandler(w http.ResponseWriter, r
bucket := vars["bucket"]
// PutBucketLifecycle always needs a Content-Md5
if _, ok := r.Header[xhttp.ContentMD5]; !ok {
if !validateLengthAndChecksum(r) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL)
return
}
@@ -70,7 +69,7 @@ func (api objectAPIHandlers) PutBucketLifecycleHandler(w http.ResponseWriter, r
return
}
bucketLifecycle, err := lifecycle.ParseLifecycleConfigWithID(io.LimitReader(r.Body, r.ContentLength))
bucketLifecycle, err := lifecycle.ParseLifecycleConfigWithID(r.Body)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
+10 -10
View File
@@ -243,26 +243,26 @@ func parseRequestToken(token string) (subToken string, nodeIndex int) {
return subToken, nodeIndex
}
func proxyRequestByToken(ctx context.Context, w http.ResponseWriter, r *http.Request, token string) (string, bool) {
subToken, nodeIndex := parseRequestToken(token)
if nodeIndex > 0 {
return subToken, proxyRequestByNodeIndex(ctx, w, r, nodeIndex)
func proxyRequestByToken(ctx context.Context, w http.ResponseWriter, r *http.Request, token string, returnErr bool) (subToken string, proxied bool, success bool) {
var nodeIndex int
if subToken, nodeIndex = parseRequestToken(token); nodeIndex >= 0 {
proxied, success = proxyRequestByNodeIndex(ctx, w, r, nodeIndex, returnErr)
}
return subToken, false
return
}
func proxyRequestByNodeIndex(ctx context.Context, w http.ResponseWriter, r *http.Request, index int) (success bool) {
func proxyRequestByNodeIndex(ctx context.Context, w http.ResponseWriter, r *http.Request, index int, returnErr bool) (proxied, success bool) {
if len(globalProxyEndpoints) == 0 {
return false
return
}
if index < 0 || index >= len(globalProxyEndpoints) {
return false
return
}
ep := globalProxyEndpoints[index]
if ep.IsLocal {
return false
return
}
return proxyRequest(ctx, w, r, ep)
return true, proxyRequest(ctx, w, r, ep, returnErr)
}
// ListObjectsV1Handler - GET Bucket (List Objects) Version 1.
+43 -16
View File
@@ -53,8 +53,6 @@ import (
"github.com/minio/minio/internal/once"
"github.com/tinylib/msgp/msgp"
"github.com/zeebo/xxh3"
"golang.org/x/exp/maps"
"golang.org/x/exp/slices"
)
const (
@@ -780,8 +778,9 @@ func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo, part
isSSEC := crypto.SSEC.IsEncrypted(objInfo.UserDefined)
for k, v := range objInfo.UserDefined {
_, isValidSSEHeader := validSSEReplicationHeaders[k]
// In case of SSE-C objects copy the allowed internal headers as well
if !isSSEC || !slices.Contains(maps.Keys(validSSEReplicationHeaders), k) {
if !isSSEC || !isValidSSEHeader {
if stringsHasPrefixFold(k, ReservedMetadataPrefixLower) {
continue
}
@@ -789,24 +788,29 @@ func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo, part
continue
}
}
if slices.Contains(maps.Keys(validSSEReplicationHeaders), k) {
if isValidSSEHeader {
meta[validSSEReplicationHeaders[k]] = v
} else {
meta[k] = v
}
}
if len(objInfo.Checksum) > 0 {
// Add encrypted CRC to metadata for SSE-C objects.
if isSSEC {
meta[ReplicationSsecChecksumHeader] = base64.StdEncoding.EncodeToString(objInfo.Checksum)
} else {
for _, pi := range objInfo.Parts {
if pi.Number == partNum {
for k, v := range pi.Checksums {
meta[k] = v
if objInfo.isMultipart() && partNum > 0 {
for _, pi := range objInfo.Parts {
if pi.Number == partNum {
for k, v := range pi.Checksums { // for PutObjectPart
meta[k] = v
}
}
}
} else {
for k, v := range getCRCMeta(objInfo, 0, nil) { // for PutObject/NewMultipartUpload
meta[k] = v
}
}
}
}
@@ -1667,7 +1671,7 @@ func replicateObjectWithMultipart(ctx context.Context, c *minio.Core, bucket, ob
cHeader := http.Header{}
cHeader.Add(xhttp.MinIOSourceReplicationRequest, "true")
if !isSSEC {
for k, v := range partInfo.Checksums {
for k, v := range getCRCMeta(objInfo, partInfo.Number, nil) {
cHeader.Add(k, v)
}
}
@@ -1691,12 +1695,13 @@ func replicateObjectWithMultipart(ctx context.Context, c *minio.Core, bucket, ob
return fmt.Errorf("ssec(%t): Part size mismatch: got %d, want %d", isSSEC, pInfo.Size, size)
}
uploadedParts = append(uploadedParts, minio.CompletePart{
PartNumber: pInfo.PartNumber,
ETag: pInfo.ETag,
ChecksumCRC32: pInfo.ChecksumCRC32,
ChecksumCRC32C: pInfo.ChecksumCRC32C,
ChecksumSHA1: pInfo.ChecksumSHA1,
ChecksumSHA256: pInfo.ChecksumSHA256,
PartNumber: pInfo.PartNumber,
ETag: pInfo.ETag,
ChecksumCRC32: pInfo.ChecksumCRC32,
ChecksumCRC32C: pInfo.ChecksumCRC32C,
ChecksumSHA1: pInfo.ChecksumSHA1,
ChecksumSHA256: pInfo.ChecksumSHA256,
ChecksumCRC64NVME: pInfo.ChecksumCRC64NVME,
})
}
userMeta := map[string]string{
@@ -1709,6 +1714,12 @@ func replicateObjectWithMultipart(ctx context.Context, c *minio.Core, bucket, ob
// really big value but its okay on heavily loaded systems. This is just tail end timeout.
cctx, ccancel := context.WithTimeout(ctx, 10*time.Minute)
defer ccancel()
if len(objInfo.Checksum) > 0 {
for k, v := range getCRCMeta(objInfo, 0, nil) {
userMeta[k] = v
}
}
_, err = c.CompleteMultipartUpload(cctx, bucket, object, uploadID, uploadedParts, minio.PutObjectOptions{
UserMetadata: userMeta,
Internal: minio.AdvancedPutOptions{
@@ -3754,3 +3765,19 @@ type validateReplicationDestinationOptions struct {
checkReadyErr sync.Map
}
func getCRCMeta(oi ObjectInfo, partNum int, h http.Header) map[string]string {
meta := make(map[string]string)
cs := oi.decryptChecksums(partNum, h)
for k, v := range cs {
cksum := hash.NewChecksumString(k, v)
if cksum == nil {
continue
}
if cksum.Valid() {
meta[cksum.Type.Key()] = v
}
meta[xhttp.AmzChecksumType] = cksum.Type.ObjType()
}
return meta
}
+7 -7
View File
@@ -108,18 +108,18 @@ func (l ReplicationLastHour) merge(o ReplicationLastHour) (merged ReplicationLas
// Add a new duration data
func (l *ReplicationLastHour) addsize(sz int64) {
min := time.Now().Unix() / 60
l.forwardTo(min)
winIdx := min % 60
l.Totals[winIdx].merge(AccElem{Total: min, Size: sz, N: 1})
l.LastMin = min
minutes := time.Now().Unix() / 60
l.forwardTo(minutes)
winIdx := minutes % 60
l.Totals[winIdx].merge(AccElem{Total: minutes, Size: sz, N: 1})
l.LastMin = minutes
}
// Merge all recorded counts of last hour into one
func (l *ReplicationLastHour) getTotal() AccElem {
var res AccElem
min := time.Now().Unix() / 60
l.forwardTo(min)
minutes := time.Now().Unix() / 60
l.forwardTo(minutes)
for _, elem := range l.Totals[:] {
res.merge(elem)
}
+571 -177
View File
@@ -1721,11 +1721,93 @@ func (z *dataUsageEntry) DecodeMsg(dc *msgp.Reader) (err error) {
if z.AllTierStats == nil {
z.AllTierStats = new(allTierStats)
}
err = z.AllTierStats.DecodeMsg(dc)
var zb0004 uint32
zb0004, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
for zb0004 > 0 {
zb0004--
field, err = dc.ReadMapKeyPtr()
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
switch msgp.UnsafeString(field) {
case "ts":
var zb0005 uint32
zb0005, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
if z.AllTierStats.Tiers == nil {
z.AllTierStats.Tiers = make(map[string]tierStats, zb0005)
} else if len(z.AllTierStats.Tiers) > 0 {
for key := range z.AllTierStats.Tiers {
delete(z.AllTierStats.Tiers, key)
}
}
for zb0005 > 0 {
zb0005--
var za0003 string
var za0004 tierStats
za0003, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
var zb0006 uint32
zb0006, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
for zb0006 > 0 {
zb0006--
field, err = dc.ReadMapKeyPtr()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
switch msgp.UnsafeString(field) {
case "ts":
za0004.TotalSize, err = dc.ReadUint64()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
return
}
case "nv":
za0004.NumVersions, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
return
}
case "no":
za0004.NumObjects, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
return
}
default:
err = dc.Skip()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
}
}
z.AllTierStats.Tiers[za0003] = za0004
}
default:
err = dc.Skip()
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
}
}
}
zb0001Mask |= 0x1
case "c":
@@ -1743,11 +1825,10 @@ func (z *dataUsageEntry) DecodeMsg(dc *msgp.Reader) (err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
return
}
@@ -1766,121 +1847,166 @@ func (z *dataUsageEntry) EncodeMsg(en *msgp.Writer) (err error) {
if err != nil {
return
}
if zb0001Len == 0 {
return
}
// write "ch"
err = en.Append(0xa2, 0x63, 0x68)
if err != nil {
return
}
err = z.Children.EncodeMsg(en)
if err != nil {
err = msgp.WrapError(err, "Children")
return
}
// write "sz"
err = en.Append(0xa2, 0x73, 0x7a)
if err != nil {
return
}
err = en.WriteInt64(z.Size)
if err != nil {
err = msgp.WrapError(err, "Size")
return
}
// write "os"
err = en.Append(0xa2, 0x6f, 0x73)
if err != nil {
return
}
err = en.WriteUint64(z.Objects)
if err != nil {
err = msgp.WrapError(err, "Objects")
return
}
// write "vs"
err = en.Append(0xa2, 0x76, 0x73)
if err != nil {
return
}
err = en.WriteUint64(z.Versions)
if err != nil {
err = msgp.WrapError(err, "Versions")
return
}
// write "dms"
err = en.Append(0xa3, 0x64, 0x6d, 0x73)
if err != nil {
return
}
err = en.WriteUint64(z.DeleteMarkers)
if err != nil {
err = msgp.WrapError(err, "DeleteMarkers")
return
}
// write "szs"
err = en.Append(0xa3, 0x73, 0x7a, 0x73)
if err != nil {
return
}
err = en.WriteArrayHeader(uint32(dataUsageBucketLen))
if err != nil {
err = msgp.WrapError(err, "ObjSizes")
return
}
for za0001 := range z.ObjSizes {
err = en.WriteUint64(z.ObjSizes[za0001])
if err != nil {
err = msgp.WrapError(err, "ObjSizes", za0001)
return
}
}
// write "vh"
err = en.Append(0xa2, 0x76, 0x68)
if err != nil {
return
}
err = en.WriteArrayHeader(uint32(dataUsageVersionLen))
if err != nil {
err = msgp.WrapError(err, "ObjVersions")
return
}
for za0002 := range z.ObjVersions {
err = en.WriteUint64(z.ObjVersions[za0002])
if err != nil {
err = msgp.WrapError(err, "ObjVersions", za0002)
return
}
}
if (zb0001Mask & 0x80) == 0 { // if not omitted
// write "ats"
err = en.Append(0xa3, 0x61, 0x74, 0x73)
// skip if no fields are to be emitted
if zb0001Len != 0 {
// write "ch"
err = en.Append(0xa2, 0x63, 0x68)
if err != nil {
return
}
if z.AllTierStats == nil {
err = en.WriteNil()
err = z.Children.EncodeMsg(en)
if err != nil {
err = msgp.WrapError(err, "Children")
return
}
// write "sz"
err = en.Append(0xa2, 0x73, 0x7a)
if err != nil {
return
}
err = en.WriteInt64(z.Size)
if err != nil {
err = msgp.WrapError(err, "Size")
return
}
// write "os"
err = en.Append(0xa2, 0x6f, 0x73)
if err != nil {
return
}
err = en.WriteUint64(z.Objects)
if err != nil {
err = msgp.WrapError(err, "Objects")
return
}
// write "vs"
err = en.Append(0xa2, 0x76, 0x73)
if err != nil {
return
}
err = en.WriteUint64(z.Versions)
if err != nil {
err = msgp.WrapError(err, "Versions")
return
}
// write "dms"
err = en.Append(0xa3, 0x64, 0x6d, 0x73)
if err != nil {
return
}
err = en.WriteUint64(z.DeleteMarkers)
if err != nil {
err = msgp.WrapError(err, "DeleteMarkers")
return
}
// write "szs"
err = en.Append(0xa3, 0x73, 0x7a, 0x73)
if err != nil {
return
}
err = en.WriteArrayHeader(uint32(dataUsageBucketLen))
if err != nil {
err = msgp.WrapError(err, "ObjSizes")
return
}
for za0001 := range z.ObjSizes {
err = en.WriteUint64(z.ObjSizes[za0001])
if err != nil {
return
}
} else {
err = z.AllTierStats.EncodeMsg(en)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
err = msgp.WrapError(err, "ObjSizes", za0001)
return
}
}
}
// write "c"
err = en.Append(0xa1, 0x63)
if err != nil {
return
}
err = en.WriteBool(z.Compacted)
if err != nil {
err = msgp.WrapError(err, "Compacted")
return
// write "vh"
err = en.Append(0xa2, 0x76, 0x68)
if err != nil {
return
}
err = en.WriteArrayHeader(uint32(dataUsageVersionLen))
if err != nil {
err = msgp.WrapError(err, "ObjVersions")
return
}
for za0002 := range z.ObjVersions {
err = en.WriteUint64(z.ObjVersions[za0002])
if err != nil {
err = msgp.WrapError(err, "ObjVersions", za0002)
return
}
}
if (zb0001Mask & 0x80) == 0 { // if not omitted
// write "ats"
err = en.Append(0xa3, 0x61, 0x74, 0x73)
if err != nil {
return
}
if z.AllTierStats == nil {
err = en.WriteNil()
if err != nil {
return
}
} else {
// map header, size 1
// write "ts"
err = en.Append(0x81, 0xa2, 0x74, 0x73)
if err != nil {
return
}
err = en.WriteMapHeader(uint32(len(z.AllTierStats.Tiers)))
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
for za0003, za0004 := range z.AllTierStats.Tiers {
err = en.WriteString(za0003)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
// map header, size 3
// write "ts"
err = en.Append(0x83, 0xa2, 0x74, 0x73)
if err != nil {
return
}
err = en.WriteUint64(za0004.TotalSize)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
return
}
// write "nv"
err = en.Append(0xa2, 0x6e, 0x76)
if err != nil {
return
}
err = en.WriteInt(za0004.NumVersions)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
return
}
// write "no"
err = en.Append(0xa2, 0x6e, 0x6f)
if err != nil {
return
}
err = en.WriteInt(za0004.NumObjects)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
return
}
}
}
}
// write "c"
err = en.Append(0xa1, 0x63)
if err != nil {
return
}
err = en.WriteBool(z.Compacted)
if err != nil {
err = msgp.WrapError(err, "Compacted")
return
}
}
return
}
@@ -1898,56 +2024,69 @@ func (z *dataUsageEntry) MarshalMsg(b []byte) (o []byte, err error) {
}
// variable map header, size zb0001Len
o = append(o, 0x80|uint8(zb0001Len))
if zb0001Len == 0 {
return
}
// string "ch"
o = append(o, 0xa2, 0x63, 0x68)
o, err = z.Children.MarshalMsg(o)
if err != nil {
err = msgp.WrapError(err, "Children")
return
}
// string "sz"
o = append(o, 0xa2, 0x73, 0x7a)
o = msgp.AppendInt64(o, z.Size)
// string "os"
o = append(o, 0xa2, 0x6f, 0x73)
o = msgp.AppendUint64(o, z.Objects)
// string "vs"
o = append(o, 0xa2, 0x76, 0x73)
o = msgp.AppendUint64(o, z.Versions)
// string "dms"
o = append(o, 0xa3, 0x64, 0x6d, 0x73)
o = msgp.AppendUint64(o, z.DeleteMarkers)
// string "szs"
o = append(o, 0xa3, 0x73, 0x7a, 0x73)
o = msgp.AppendArrayHeader(o, uint32(dataUsageBucketLen))
for za0001 := range z.ObjSizes {
o = msgp.AppendUint64(o, z.ObjSizes[za0001])
}
// string "vh"
o = append(o, 0xa2, 0x76, 0x68)
o = msgp.AppendArrayHeader(o, uint32(dataUsageVersionLen))
for za0002 := range z.ObjVersions {
o = msgp.AppendUint64(o, z.ObjVersions[za0002])
}
if (zb0001Mask & 0x80) == 0 { // if not omitted
// string "ats"
o = append(o, 0xa3, 0x61, 0x74, 0x73)
if z.AllTierStats == nil {
o = msgp.AppendNil(o)
} else {
o, err = z.AllTierStats.MarshalMsg(o)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
// skip if no fields are to be emitted
if zb0001Len != 0 {
// string "ch"
o = append(o, 0xa2, 0x63, 0x68)
o, err = z.Children.MarshalMsg(o)
if err != nil {
err = msgp.WrapError(err, "Children")
return
}
// string "sz"
o = append(o, 0xa2, 0x73, 0x7a)
o = msgp.AppendInt64(o, z.Size)
// string "os"
o = append(o, 0xa2, 0x6f, 0x73)
o = msgp.AppendUint64(o, z.Objects)
// string "vs"
o = append(o, 0xa2, 0x76, 0x73)
o = msgp.AppendUint64(o, z.Versions)
// string "dms"
o = append(o, 0xa3, 0x64, 0x6d, 0x73)
o = msgp.AppendUint64(o, z.DeleteMarkers)
// string "szs"
o = append(o, 0xa3, 0x73, 0x7a, 0x73)
o = msgp.AppendArrayHeader(o, uint32(dataUsageBucketLen))
for za0001 := range z.ObjSizes {
o = msgp.AppendUint64(o, z.ObjSizes[za0001])
}
// string "vh"
o = append(o, 0xa2, 0x76, 0x68)
o = msgp.AppendArrayHeader(o, uint32(dataUsageVersionLen))
for za0002 := range z.ObjVersions {
o = msgp.AppendUint64(o, z.ObjVersions[za0002])
}
if (zb0001Mask & 0x80) == 0 { // if not omitted
// string "ats"
o = append(o, 0xa3, 0x61, 0x74, 0x73)
if z.AllTierStats == nil {
o = msgp.AppendNil(o)
} else {
// map header, size 1
// string "ts"
o = append(o, 0x81, 0xa2, 0x74, 0x73)
o = msgp.AppendMapHeader(o, uint32(len(z.AllTierStats.Tiers)))
for za0003, za0004 := range z.AllTierStats.Tiers {
o = msgp.AppendString(o, za0003)
// map header, size 3
// string "ts"
o = append(o, 0x83, 0xa2, 0x74, 0x73)
o = msgp.AppendUint64(o, za0004.TotalSize)
// string "nv"
o = append(o, 0xa2, 0x6e, 0x76)
o = msgp.AppendInt(o, za0004.NumVersions)
// string "no"
o = append(o, 0xa2, 0x6e, 0x6f)
o = msgp.AppendInt(o, za0004.NumObjects)
}
}
}
// string "c"
o = append(o, 0xa1, 0x63)
o = msgp.AppendBool(o, z.Compacted)
}
// string "c"
o = append(o, 0xa1, 0x63)
o = msgp.AppendBool(o, z.Compacted)
return
}
@@ -2048,11 +2187,93 @@ func (z *dataUsageEntry) UnmarshalMsg(bts []byte) (o []byte, err error) {
if z.AllTierStats == nil {
z.AllTierStats = new(allTierStats)
}
bts, err = z.AllTierStats.UnmarshalMsg(bts)
var zb0004 uint32
zb0004, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
for zb0004 > 0 {
zb0004--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
switch msgp.UnsafeString(field) {
case "ts":
var zb0005 uint32
zb0005, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
if z.AllTierStats.Tiers == nil {
z.AllTierStats.Tiers = make(map[string]tierStats, zb0005)
} else if len(z.AllTierStats.Tiers) > 0 {
for key := range z.AllTierStats.Tiers {
delete(z.AllTierStats.Tiers, key)
}
}
for zb0005 > 0 {
var za0003 string
var za0004 tierStats
zb0005--
za0003, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
var zb0006 uint32
zb0006, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
for zb0006 > 0 {
zb0006--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
switch msgp.UnsafeString(field) {
case "ts":
za0004.TotalSize, bts, err = msgp.ReadUint64Bytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
return
}
case "nv":
za0004.NumVersions, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
return
}
case "no":
za0004.NumObjects, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
}
}
z.AllTierStats.Tiers[za0003] = za0004
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
}
}
}
zb0001Mask |= 0x1
case "c":
@@ -2070,11 +2291,10 @@ func (z *dataUsageEntry) UnmarshalMsg(bts []byte) (o []byte, err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
o = bts
return
}
@@ -2085,7 +2305,13 @@ func (z *dataUsageEntry) Msgsize() (s int) {
if z.AllTierStats == nil {
s += msgp.NilSize
} else {
s += z.AllTierStats.Msgsize()
s += 1 + 3 + msgp.MapHeaderSize
if z.AllTierStats.Tiers != nil {
for za0003, za0004 := range z.AllTierStats.Tiers {
_ = za0004
s += msgp.StringPrefixSize + len(za0003) + 1 + 3 + msgp.Uint64Size + 3 + msgp.IntSize + 3 + msgp.IntSize
}
}
}
s += 2 + msgp.BoolSize
return
@@ -2734,11 +2960,93 @@ func (z *dataUsageEntryV7) DecodeMsg(dc *msgp.Reader) (err error) {
if z.AllTierStats == nil {
z.AllTierStats = new(allTierStats)
}
err = z.AllTierStats.DecodeMsg(dc)
var zb0004 uint32
zb0004, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
for zb0004 > 0 {
zb0004--
field, err = dc.ReadMapKeyPtr()
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
switch msgp.UnsafeString(field) {
case "ts":
var zb0005 uint32
zb0005, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
if z.AllTierStats.Tiers == nil {
z.AllTierStats.Tiers = make(map[string]tierStats, zb0005)
} else if len(z.AllTierStats.Tiers) > 0 {
for key := range z.AllTierStats.Tiers {
delete(z.AllTierStats.Tiers, key)
}
}
for zb0005 > 0 {
zb0005--
var za0003 string
var za0004 tierStats
za0003, err = dc.ReadString()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
var zb0006 uint32
zb0006, err = dc.ReadMapHeader()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
for zb0006 > 0 {
zb0006--
field, err = dc.ReadMapKeyPtr()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
switch msgp.UnsafeString(field) {
case "ts":
za0004.TotalSize, err = dc.ReadUint64()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
return
}
case "nv":
za0004.NumVersions, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
return
}
case "no":
za0004.NumObjects, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
return
}
default:
err = dc.Skip()
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
}
}
z.AllTierStats.Tiers[za0003] = za0004
}
default:
err = dc.Skip()
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
}
}
}
zb0001Mask |= 0x1
case "c":
@@ -2756,11 +3064,10 @@ func (z *dataUsageEntryV7) DecodeMsg(dc *msgp.Reader) (err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
return
}
@@ -2861,11 +3168,93 @@ func (z *dataUsageEntryV7) UnmarshalMsg(bts []byte) (o []byte, err error) {
if z.AllTierStats == nil {
z.AllTierStats = new(allTierStats)
}
bts, err = z.AllTierStats.UnmarshalMsg(bts)
var zb0004 uint32
zb0004, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
for zb0004 > 0 {
zb0004--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
switch msgp.UnsafeString(field) {
case "ts":
var zb0005 uint32
zb0005, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
if z.AllTierStats.Tiers == nil {
z.AllTierStats.Tiers = make(map[string]tierStats, zb0005)
} else if len(z.AllTierStats.Tiers) > 0 {
for key := range z.AllTierStats.Tiers {
delete(z.AllTierStats.Tiers, key)
}
}
for zb0005 > 0 {
var za0003 string
var za0004 tierStats
zb0005--
za0003, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers")
return
}
var zb0006 uint32
zb0006, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
for zb0006 > 0 {
zb0006--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
switch msgp.UnsafeString(field) {
case "ts":
za0004.TotalSize, bts, err = msgp.ReadUint64Bytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "TotalSize")
return
}
case "nv":
za0004.NumVersions, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumVersions")
return
}
case "no":
za0004.NumObjects, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003, "NumObjects")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats", "Tiers", za0003)
return
}
}
}
z.AllTierStats.Tiers[za0003] = za0004
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
err = msgp.WrapError(err, "AllTierStats")
return
}
}
}
}
zb0001Mask |= 0x1
case "c":
@@ -2883,11 +3272,10 @@ func (z *dataUsageEntryV7) UnmarshalMsg(bts []byte) (o []byte, err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
if (zb0001Mask & 0x1) == 0 {
z.AllTierStats = nil
}
o = bts
return
}
@@ -2898,7 +3286,13 @@ func (z *dataUsageEntryV7) Msgsize() (s int) {
if z.AllTierStats == nil {
s += msgp.NilSize
} else {
s += z.AllTierStats.Msgsize()
s += 1 + 3 + msgp.MapHeaderSize
if z.AllTierStats.Tiers != nil {
for za0003, za0004 := range z.AllTierStats.Tiers {
_ = za0004
s += msgp.StringPrefixSize + len(za0003) + 1 + 3 + msgp.Uint64Size + 3 + msgp.IntSize + 3 + msgp.IntSize
}
}
}
s += 2 + msgp.BoolSize
return
+6 -6
View File
@@ -117,12 +117,12 @@ func (dt *dynamicTimeout) logEntry(duration time.Duration) {
// adjust changes the value of the dynamic timeout based on the
// previous results
func (dt *dynamicTimeout) adjust(entries [dynamicTimeoutLogSize]time.Duration) {
failures, max := 0, time.Duration(0)
failures, maxDur := 0, time.Duration(0)
for _, dur := range entries[:] {
if dur == maxDuration {
failures++
} else if dur > max {
max = dur
} else if dur > maxDur {
maxDur = dur
}
}
@@ -144,12 +144,12 @@ func (dt *dynamicTimeout) adjust(entries [dynamicTimeoutLogSize]time.Duration) {
} else if failPct < dynamicTimeoutDecreaseThresholdPct {
// We are hitting the timeout relatively few times,
// so decrease the timeout towards 25 % of maximum time spent.
max = max * 125 / 100
maxDur = maxDur * 125 / 100
timeout := atomic.LoadInt64(&dt.timeout)
if max < time.Duration(timeout) {
if maxDur < time.Duration(timeout) {
// Move 50% toward the max.
timeout = (int64(max) + timeout) / 2
timeout = (int64(maxDur) + timeout) / 2
}
if timeout < dt.minimum {
timeout = dt.minimum
+1 -1
View File
@@ -109,7 +109,7 @@ func kmsKeyIDFromMetadata(metadata map[string]string) string {
// be AWS S3 compliant.
//
// DecryptETags uses a KMS bulk decryption API, if available, which
// is more efficient than decrypting ETags sequentually.
// is more efficient than decrypting ETags sequentially.
func DecryptETags(ctx context.Context, k *kms.KMS, objects []ObjectInfo) error {
const BatchSize = 250 // We process the objects in batches - 250 is a reasonable default.
var (
+3 -3
View File
@@ -26,6 +26,7 @@ import (
"path/filepath"
"reflect"
"runtime"
"slices"
"sort"
"strconv"
"strings"
@@ -38,7 +39,6 @@ import (
"github.com/minio/minio/internal/mountinfo"
"github.com/minio/pkg/v3/env"
xnet "github.com/minio/pkg/v3/net"
"golang.org/x/exp/slices"
)
// EndpointType - enum for endpoint type.
@@ -1194,7 +1194,7 @@ func GetProxyEndpointLocalIndex(proxyEps []ProxyEndpoint) int {
}
// GetProxyEndpoints - get all endpoints that can be used to proxy list request.
func GetProxyEndpoints(endpointServerPools EndpointServerPools) []ProxyEndpoint {
func GetProxyEndpoints(endpointServerPools EndpointServerPools, transport http.RoundTripper) []ProxyEndpoint {
var proxyEps []ProxyEndpoint
proxyEpSet := set.NewStringSet()
@@ -1213,7 +1213,7 @@ func GetProxyEndpoints(endpointServerPools EndpointServerPools) []ProxyEndpoint
proxyEps = append(proxyEps, ProxyEndpoint{
Endpoint: endpoint,
Transport: globalRemoteTargetTransport,
Transport: transport,
})
}
}
+17 -27
View File
@@ -277,23 +277,21 @@ func partNeedsHealing(partErrs []int) bool {
return slices.IndexFunc(partErrs, func(i int) bool { return i != checkPartSuccess && i != checkPartUnknown }) > -1
}
func hasPartErr(partErrs []int) bool {
return slices.IndexFunc(partErrs, func(i int) bool { return i != checkPartSuccess }) > -1
func countPartNotSuccess(partErrs []int) (c int) {
for _, pe := range partErrs {
if pe != checkPartSuccess {
c++
}
}
return
}
// disksWithAllParts - This function needs to be called with
// []StorageAPI returned by listOnlineDisks. Returns,
//
// - disks which have all parts specified in the latest xl.meta.
//
// - slice of errors about the state of data files on disk - can have
// a not-found error or a hash-mismatch error.
func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetadata []FileInfo,
// checkObjectWithAllParts sets partsMetadata and onlineDisks when xl.meta is inexistant/corrupted or outdated
// it also checks if the status of each part (corrupted, missing, ok) in each drive
func checkObjectWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetadata []FileInfo,
errs []error, latestMeta FileInfo, filterByETag bool, bucket, object string,
scanMode madmin.HealScanMode,
) (availableDisks []StorageAPI, dataErrsByDisk map[int][]int, dataErrsByPart map[int][]int) {
availableDisks = make([]StorageAPI, len(onlineDisks))
) (dataErrsByDisk map[int][]int, dataErrsByPart map[int][]int) {
dataErrsByDisk = make(map[int][]int, len(onlineDisks))
for i := range onlineDisks {
dataErrsByDisk[i] = make([]int, len(latestMeta.Parts))
@@ -334,12 +332,12 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
metaErrs := make([]error, len(errs))
for i, onlineDisk := range onlineDisks {
for i := range onlineDisks {
if errs[i] != nil {
metaErrs[i] = errs[i]
continue
}
if onlineDisk == OfflineDisk {
if onlineDisks[i] == OfflineDisk {
metaErrs[i] = errDiskNotFound
continue
}
@@ -355,6 +353,7 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
if corrupted {
metaErrs[i] = errFileCorrupt
partsMetadata[i] = FileInfo{}
onlineDisks[i] = nil
continue
}
@@ -362,6 +361,7 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
if !meta.IsValid() {
partsMetadata[i] = FileInfo{}
metaErrs[i] = errFileCorrupt
onlineDisks[i] = nil
continue
}
@@ -372,6 +372,7 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
// might have the right erasure distribution.
partsMetadata[i] = FileInfo{}
metaErrs[i] = errFileCorrupt
onlineDisks[i] = nil
continue
}
}
@@ -392,8 +393,8 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
if metaErrs[i] != nil {
continue
}
meta := partsMetadata[i]
meta := partsMetadata[i]
if meta.Deleted || meta.IsRemote() {
continue
}
@@ -440,16 +441,5 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
dataErrsByDisk[disk][part] = dataErrsByPart[part][disk]
}
}
for i, onlineDisk := range onlineDisks {
if metaErrs[i] == nil && !hasPartErr(dataErrsByDisk[i]) {
// All parts verified, mark it as all data available.
availableDisks[i] = onlineDisk
} else {
// upon errors just make that disk's fileinfo invalid
partsMetadata[i] = FileInfo{}
}
}
return
}
+16 -33
View File
@@ -313,11 +313,11 @@ func TestListOnlineDisks(t *testing.T) {
t.Fatalf("Expected modTime to be equal to %v but was found to be %v",
test.expectedTime, modTime)
}
availableDisks, _, _ := disksWithAllParts(ctx, onlineDisks, partsMetadata,
_, _ = checkObjectWithAllParts(ctx, onlineDisks, partsMetadata,
test.errs, fi, false, bucket, object, madmin.HealDeepScan)
if test._tamperBackend != noTamper {
if tamperedIndex != -1 && availableDisks[tamperedIndex] != nil {
if tamperedIndex != -1 && onlineDisks[tamperedIndex] != nil {
t.Fatalf("Drive (%v) with part.1 missing is not a drive with available data",
erasureDisks[tamperedIndex])
}
@@ -495,11 +495,11 @@ func TestListOnlineDisksSmallObjects(t *testing.T) {
test.expectedTime, modTime)
}
availableDisks, _, _ := disksWithAllParts(ctx, onlineDisks, partsMetadata,
_, _ = checkObjectWithAllParts(ctx, onlineDisks, partsMetadata,
test.errs, fi, false, bucket, object, madmin.HealDeepScan)
if test._tamperBackend != noTamper {
if tamperedIndex != -1 && availableDisks[tamperedIndex] != nil {
if tamperedIndex != -1 && onlineDisks[tamperedIndex] != nil {
t.Fatalf("Drive (%v) with part.1 missing is not a drive with available data",
erasureDisks[tamperedIndex])
}
@@ -545,6 +545,7 @@ func TestDisksWithAllParts(t *testing.T) {
// Test 1: Test that all disks are returned without any failures with
// unmodified meta data
erasureDisks = s.getDisks()
partsMetadata, errs := readAllFileInfo(ctx, erasureDisks, "", bucket, object, "", false, true)
if err != nil {
t.Fatalf("Failed to read xl meta data %v", err)
@@ -557,14 +558,10 @@ func TestDisksWithAllParts(t *testing.T) {
erasureDisks, _, _ = listOnlineDisks(erasureDisks, partsMetadata, errs, readQuorum)
filteredDisks, _, dataErrsPerDisk := disksWithAllParts(ctx, erasureDisks, partsMetadata,
dataErrsPerDisk, _ := checkObjectWithAllParts(ctx, erasureDisks, partsMetadata,
errs, fi, false, bucket, object, madmin.HealDeepScan)
if len(filteredDisks) != len(erasureDisks) {
t.Errorf("Unexpected number of drives: %d", len(filteredDisks))
}
for diskIndex, disk := range filteredDisks {
for diskIndex, disk := range erasureDisks {
if partNeedsHealing(dataErrsPerDisk[diskIndex]) {
t.Errorf("Unexpected error: %v", dataErrsPerDisk[diskIndex])
}
@@ -575,17 +572,15 @@ func TestDisksWithAllParts(t *testing.T) {
}
// Test 2: Not synchronized modtime
erasureDisks = s.getDisks()
partsMetadataBackup := partsMetadata[0]
partsMetadata[0].ModTime = partsMetadata[0].ModTime.Add(-1 * time.Hour)
errs = make([]error, len(erasureDisks))
filteredDisks, _, _ = disksWithAllParts(ctx, erasureDisks, partsMetadata,
_, _ = checkObjectWithAllParts(ctx, erasureDisks, partsMetadata,
errs, fi, false, bucket, object, madmin.HealDeepScan)
if len(filteredDisks) != len(erasureDisks) {
t.Errorf("Unexpected number of drives: %d", len(filteredDisks))
}
for diskIndex, disk := range filteredDisks {
for diskIndex, disk := range erasureDisks {
if diskIndex == 0 && disk != nil {
t.Errorf("Drive not filtered as expected, drive: %d", diskIndex)
}
@@ -596,17 +591,15 @@ func TestDisksWithAllParts(t *testing.T) {
partsMetadata[0] = partsMetadataBackup // Revert before going to the next test
// Test 3: Not synchronized DataDir
erasureDisks = s.getDisks()
partsMetadataBackup = partsMetadata[1]
partsMetadata[1].DataDir = "foo-random"
errs = make([]error, len(erasureDisks))
filteredDisks, _, _ = disksWithAllParts(ctx, erasureDisks, partsMetadata,
_, _ = checkObjectWithAllParts(ctx, erasureDisks, partsMetadata,
errs, fi, false, bucket, object, madmin.HealDeepScan)
if len(filteredDisks) != len(erasureDisks) {
t.Errorf("Unexpected number of drives: %d", len(filteredDisks))
}
for diskIndex, disk := range filteredDisks {
for diskIndex, disk := range erasureDisks {
if diskIndex == 1 && disk != nil {
t.Errorf("Drive not filtered as expected, drive: %d", diskIndex)
}
@@ -617,6 +610,7 @@ func TestDisksWithAllParts(t *testing.T) {
partsMetadata[1] = partsMetadataBackup // Revert before going to the next test
// Test 4: key = disk index, value = part name with hash mismatch
erasureDisks = s.getDisks()
diskFailures := make(map[int]string)
diskFailures[0] = "part.1"
diskFailures[3] = "part.1"
@@ -637,29 +631,18 @@ func TestDisksWithAllParts(t *testing.T) {
}
errs = make([]error, len(erasureDisks))
filteredDisks, dataErrsPerDisk, _ = disksWithAllParts(ctx, erasureDisks, partsMetadata,
dataErrsPerDisk, _ = checkObjectWithAllParts(ctx, erasureDisks, partsMetadata,
errs, fi, false, bucket, object, madmin.HealDeepScan)
if len(filteredDisks) != len(erasureDisks) {
t.Errorf("Unexpected number of drives: %d", len(filteredDisks))
}
for diskIndex, disk := range filteredDisks {
for diskIndex := range erasureDisks {
if _, ok := diskFailures[diskIndex]; ok {
if disk != nil {
t.Errorf("Drive not filtered as expected, drive: %d", diskIndex)
}
if !partNeedsHealing(dataErrsPerDisk[diskIndex]) {
t.Errorf("Disk expected to be healed, driveIndex: %d", diskIndex)
}
} else {
if disk == nil {
t.Errorf("Drive erroneously filtered, driveIndex: %d", diskIndex)
}
if partNeedsHealing(dataErrsPerDisk[diskIndex]) {
t.Errorf("Disk not expected to be healed, driveIndex: %d", diskIndex)
}
}
}
}
+100 -77
View File
@@ -32,7 +32,7 @@ import (
"github.com/minio/minio/internal/grid"
"github.com/minio/minio/internal/logger"
"github.com/minio/pkg/v3/sync/errgroup"
"golang.org/x/exp/slices"
"github.com/puzpuzpuz/xsync/v3"
)
//go:generate stringer -type=healingMetric -trimprefix=healingMetric $GOFILE
@@ -45,7 +45,8 @@ const (
healingMetricCheckAbandonedParts
)
func (er erasureObjects) listAndHeal(ctx context.Context, bucket, prefix string, scanMode madmin.HealScanMode, healEntry func(string, metaCacheEntry, madmin.HealScanMode) error) error {
// List a prefix or a single object versions and heal
func (er erasureObjects) listAndHeal(ctx context.Context, bucket, prefix string, recursive bool, scanMode madmin.HealScanMode, healEntry func(string, metaCacheEntry, madmin.HealScanMode) error) error {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
@@ -78,11 +79,14 @@ func (er erasureObjects) listAndHeal(ctx context.Context, bucket, prefix string,
bucket: bucket,
path: path,
filterPrefix: filterPrefix,
recursive: true,
recursive: recursive,
forwardTo: "",
minDisks: 1,
reportNotFound: false,
agreed: func(entry metaCacheEntry) {
if !recursive && prefix != entry.name {
return
}
if err := healEntry(bucket, entry, scanMode); err != nil {
cancel()
}
@@ -94,7 +98,9 @@ func (er erasureObjects) listAndHeal(ctx context.Context, bucket, prefix string,
// proceed to heal nonetheless.
entry, _ = entries.firstFound()
}
if !recursive && prefix != entry.name {
return
}
if err := healEntry(bucket, *entry, scanMode); err != nil {
cancel()
return
@@ -112,9 +118,8 @@ func (er erasureObjects) listAndHeal(ctx context.Context, bucket, prefix string,
// listAllBuckets lists all buckets from all disks. It also
// returns the occurrence of each buckets in all disks
func listAllBuckets(ctx context.Context, storageDisks []StorageAPI, healBuckets map[string]VolInfo, readQuorum int) error {
func listAllBuckets(ctx context.Context, storageDisks []StorageAPI, healBuckets *xsync.MapOf[string, VolInfo], readQuorum int) error {
g := errgroup.WithNErrs(len(storageDisks))
var mu sync.Mutex
for index := range storageDisks {
index := index
g.Go(func() error {
@@ -126,6 +131,7 @@ func listAllBuckets(ctx context.Context, storageDisks []StorageAPI, healBuckets
if err != nil {
return err
}
for _, volInfo := range volsInfo {
// StorageAPI can send volume names which are
// incompatible with buckets - these are
@@ -133,53 +139,75 @@ func listAllBuckets(ctx context.Context, storageDisks []StorageAPI, healBuckets
if isReservedOrInvalidBucket(volInfo.Name, false) {
continue
}
mu.Lock()
if _, ok := healBuckets[volInfo.Name]; !ok {
healBuckets[volInfo.Name] = volInfo
}
mu.Unlock()
healBuckets.Compute(volInfo.Name, func(oldValue VolInfo, loaded bool) (newValue VolInfo, del bool) {
if loaded {
newValue = oldValue
newValue.count = oldValue.count + 1
return newValue, false
}
return VolInfo{
Name: volInfo.Name,
Created: volInfo.Created,
count: 1,
}, false
})
}
return nil
}, index)
}
return reduceReadQuorumErrs(ctx, g.Wait(), bucketMetadataOpIgnoredErrs, readQuorum)
if err := reduceReadQuorumErrs(ctx, g.Wait(), bucketMetadataOpIgnoredErrs, readQuorum); err != nil {
return err
}
healBuckets.Range(func(volName string, volInfo VolInfo) bool {
if volInfo.count < readQuorum {
healBuckets.Delete(volName)
}
return true
})
return nil
}
var errLegacyXLMeta = errors.New("legacy XL meta")
var errOutdatedXLMeta = errors.New("outdated XL meta")
var errPartMissingOrCorrupt = errors.New("part missing or corrupt")
var (
errLegacyXLMeta = errors.New("legacy XL meta")
errOutdatedXLMeta = errors.New("outdated XL meta")
errPartCorrupt = errors.New("part corrupt")
errPartMissing = errors.New("part missing")
)
// Only heal on disks where we are sure that healing is needed. We can expand
// this list as and when we figure out more errors can be added to this list safely.
func shouldHealObjectOnDisk(erErr error, partsErrs []int, meta FileInfo, latestMeta FileInfo) (bool, error) {
func shouldHealObjectOnDisk(erErr error, partsErrs []int, meta FileInfo, latestMeta FileInfo) (bool, bool, error) {
if errors.Is(erErr, errFileNotFound) || errors.Is(erErr, errFileVersionNotFound) || errors.Is(erErr, errFileCorrupt) {
return true, erErr
return true, true, erErr
}
if erErr == nil {
if meta.XLV1 {
// Legacy means heal always
// always check first.
return true, errLegacyXLMeta
return true, true, errLegacyXLMeta
}
if !latestMeta.Equals(meta) {
return true, errOutdatedXLMeta
return true, true, errOutdatedXLMeta
}
if !meta.Deleted && !meta.IsRemote() {
// If xl.meta was read fine but there may be problem with the part.N files.
for _, partErr := range partsErrs {
if slices.Contains([]int{
checkPartFileNotFound,
checkPartFileCorrupt,
}, partErr) {
return true, errPartMissingOrCorrupt
if partErr == checkPartFileNotFound {
return true, false, errPartMissing
}
if partErr == checkPartFileCorrupt {
return true, false, errPartCorrupt
}
}
}
return false, nil
return false, false, nil
}
return false, erErr
return false, false, erErr
}
const (
@@ -254,6 +282,21 @@ func (er *erasureObjects) auditHealObject(ctx context.Context, bucket, object, v
auditLogInternal(ctx, opts)
}
func objectErrToDriveState(reason error) string {
switch {
case reason == nil:
return madmin.DriveStateOk
case IsErr(reason, errDiskNotFound):
return madmin.DriveStateOffline
case IsErr(reason, errFileNotFound, errFileVersionNotFound, errVolumeNotFound, errPartMissing, errOutdatedXLMeta, errLegacyXLMeta):
return madmin.DriveStateMissing
case IsErr(reason, errFileCorrupt, errPartCorrupt):
return madmin.DriveStateCorrupt
default:
return fmt.Sprintf("%s (%s)", madmin.DriveStateUnknown, reason.Error())
}
}
// Heals an object by re-writing corrupt/missing erasure blocks.
func (er *erasureObjects) healObject(ctx context.Context, bucket string, object string, versionID string, opts madmin.HealOpts) (result madmin.HealResultItem, err error) {
dryRun := opts.DryRun
@@ -340,16 +383,7 @@ func (er *erasureObjects) healObject(ctx context.Context, bucket string, object
// No modtime quorum
filterDisksByETag := quorumETag != ""
// List of disks having all parts as per latest metadata.
// NOTE: do not pass in latestDisks to diskWithAllParts since
// the diskWithAllParts needs to reach the drive to ensure
// validity of the metadata content, we should make sure that
// we pass in disks as is for it to be verified. Once verified
// the disksWithAllParts() returns the actual disks that can be
// used here for reconstruction. This is done to ensure that
// we do not skip drives that have inconsistent metadata to be
// skipped from purging when they are stale.
availableDisks, dataErrsByDisk, dataErrsByPart := disksWithAllParts(ctx, onlineDisks, partsMetadata,
dataErrsByDisk, dataErrsByPart := checkObjectWithAllParts(ctx, onlineDisks, partsMetadata,
errs, latestMeta, filterDisksByETag, bucket, object, scanMode)
var erasure Erasure
@@ -371,26 +405,18 @@ func (er *erasureObjects) healObject(ctx context.Context, bucket string, object
// data state and a list of outdated disks on which data needs
// to be healed.
outDatedDisks := make([]StorageAPI, len(storageDisks))
disksToHealCount := 0
for i := range availableDisks {
yes, reason := shouldHealObjectOnDisk(errs[i], dataErrsByDisk[i], partsMetadata[i], latestMeta)
disksToHealCount, xlMetaToHealCount := 0, 0
for i := range onlineDisks {
yes, isMeta, reason := shouldHealObjectOnDisk(errs[i], dataErrsByDisk[i], partsMetadata[i], latestMeta)
if yes {
outDatedDisks[i] = storageDisks[i]
disksToHealCount++
if isMeta {
xlMetaToHealCount++
}
}
driveState := ""
switch {
case reason == nil:
driveState = madmin.DriveStateOk
case IsErr(reason, errDiskNotFound):
driveState = madmin.DriveStateOffline
case IsErr(reason, errFileNotFound, errFileVersionNotFound, errVolumeNotFound, errPartMissingOrCorrupt, errOutdatedXLMeta, errLegacyXLMeta):
driveState = madmin.DriveStateMissing
default:
// all remaining cases imply corrupt data/metadata
driveState = madmin.DriveStateCorrupt
}
driveState := objectErrToDriveState(reason)
result.Before.Drives = append(result.Before.Drives, madmin.HealDriveInfo{
UUID: "",
@@ -404,16 +430,6 @@ func (er *erasureObjects) healObject(ctx context.Context, bucket string, object
})
}
if isAllNotFound(errs) {
// File is fully gone, fileInfo is empty.
err := errFileNotFound
if versionID != "" {
err = errFileVersionNotFound
}
return er.defaultHealResult(FileInfo{}, storageDisks, storageEndpoints, errs,
bucket, object, versionID), err
}
if disksToHealCount == 0 {
// Nothing to heal!
return result, nil
@@ -425,13 +441,23 @@ func (er *erasureObjects) healObject(ctx context.Context, bucket string, object
return result, nil
}
cannotHeal := !latestMeta.XLV1 && !latestMeta.Deleted && disksToHealCount > latestMeta.Erasure.ParityBlocks
cannotHeal := !latestMeta.XLV1 && !latestMeta.Deleted && xlMetaToHealCount > latestMeta.Erasure.ParityBlocks
if cannotHeal && quorumETag != "" {
// This is an object that is supposed to be removed by the dangling code
// but we noticed that ETag is the same for all objects, let's give it a shot
cannotHeal = false
}
if !latestMeta.Deleted && !latestMeta.IsRemote() {
// check if there is a part that lost its quorum
for _, partErrs := range dataErrsByPart {
if countPartNotSuccess(partErrs) > latestMeta.Erasure.ParityBlocks {
cannotHeal = true
break
}
}
}
if cannotHeal {
// Allow for dangling deletes, on versions that have DataDir missing etc.
// this would end up restoring the correct readable versions.
@@ -470,16 +496,15 @@ func (er *erasureObjects) healObject(ctx context.Context, bucket string, object
tmpID := mustGetUUID()
migrateDataDir := mustGetUUID()
// Reorder so that we have data disks first and parity disks next.
if !latestMeta.Deleted && len(latestMeta.Erasure.Distribution) != len(availableDisks) {
err := fmt.Errorf("unexpected file distribution (%v) from available disks (%v), looks like backend disks have been manually modified refusing to heal %s/%s(%s)",
latestMeta.Erasure.Distribution, availableDisks, bucket, object, versionID)
healingLogOnceIf(ctx, err, "heal-object-available-disks")
if !latestMeta.Deleted && len(latestMeta.Erasure.Distribution) != len(onlineDisks) {
err := fmt.Errorf("unexpected file distribution (%v) from online disks (%v), looks like backend disks have been manually modified refusing to heal %s/%s(%s)",
latestMeta.Erasure.Distribution, onlineDisks, bucket, object, versionID)
healingLogOnceIf(ctx, err, "heal-object-online-disks")
return er.defaultHealResult(latestMeta, storageDisks, storageEndpoints, errs,
bucket, object, versionID), err
}
latestDisks := shuffleDisks(availableDisks, latestMeta.Erasure.Distribution)
latestDisks := shuffleDisks(onlineDisks, latestMeta.Erasure.Distribution)
if !latestMeta.Deleted && len(latestMeta.Erasure.Distribution) != len(outDatedDisks) {
err := fmt.Errorf("unexpected file distribution (%v) from outdated disks (%v), looks like backend disks have been manually modified refusing to heal %s/%s(%s)",
@@ -550,6 +575,10 @@ func (er *erasureObjects) healObject(ctx context.Context, bucket string, object
if disk == OfflineDisk {
continue
}
thisPartErrs := shuffleCheckParts(dataErrsByPart[partIndex], latestMeta.Erasure.Distribution)
if thisPartErrs[i] != checkPartSuccess {
continue
}
checksumInfo := copyPartsMetadata[i].Erasure.GetChecksumInfo(partNumber)
partPath := pathJoin(object, srcDataDir, fmt.Sprintf("part.%d", partNumber))
readers[i] = newBitrotReader(disk, copyPartsMetadata[i].Data, bucket, partPath, tillOffset, checksumAlgo,
@@ -817,13 +846,7 @@ func (er *erasureObjects) defaultHealResult(lfi FileInfo, storageDisks []Storage
})
continue
}
driveState := madmin.DriveStateCorrupt
switch errs[index] {
case errFileNotFound, errVolumeNotFound:
driveState = madmin.DriveStateMissing
case nil:
driveState = madmin.DriveStateOk
}
driveState := objectErrToDriveState(errs[index])
result.Before.Drives = append(result.Before.Drives, madmin.HealDriveInfo{
UUID: "",
Endpoint: storageEndpoints[index].String(),
+3 -3
View File
@@ -733,7 +733,7 @@ func TestHealingDanglingObject(t *testing.T) {
t.Fatalf("Expected versions 1, got %d", fileInfoPreHeal.NumVersions)
}
if err = objLayer.HealObjects(ctx, bucket, "", madmin.HealOpts{Remove: true},
if err = objLayer.HealObjects(ctx, bucket, "", madmin.HealOpts{Recursive: true, Remove: true},
func(bucket, object, vid string, scanMode madmin.HealScanMode) error {
_, err := objLayer.HealObject(ctx, bucket, object, vid, madmin.HealOpts{ScanMode: scanMode, Remove: true})
return err
@@ -780,7 +780,7 @@ func TestHealingDanglingObject(t *testing.T) {
t.Fatalf("Expected versions 1, got %d", fileInfoPreHeal.NumVersions)
}
if err = objLayer.HealObjects(ctx, bucket, "", madmin.HealOpts{Remove: true},
if err = objLayer.HealObjects(ctx, bucket, "", madmin.HealOpts{Recursive: true, Remove: true},
func(bucket, object, vid string, scanMode madmin.HealScanMode) error {
_, err := objLayer.HealObject(ctx, bucket, object, vid, madmin.HealOpts{ScanMode: scanMode, Remove: true})
return err
@@ -829,7 +829,7 @@ func TestHealingDanglingObject(t *testing.T) {
t.Fatalf("Expected versions 3, got %d", fileInfoPreHeal.NumVersions)
}
if err = objLayer.HealObjects(ctx, bucket, "", madmin.HealOpts{Remove: true},
if err = objLayer.HealObjects(ctx, bucket, "", madmin.HealOpts{Recursive: true, Remove: true},
func(bucket, object, vid string, scanMode madmin.HealScanMode) error {
_, err := objLayer.HealObject(ctx, bucket, object, vid, madmin.HealOpts{ScanMode: scanMode, Remove: true})
return err
+32 -32
View File
@@ -52,15 +52,15 @@ func reduceCommonVersions(diskVersions [][]byte, writeQuorum int) (versions []by
}
var commonVersions uint64
max := 0
maxCnt := 0
for versions, count := range diskVersionsCount {
if max < count {
max = count
if maxCnt < count {
maxCnt = count
commonVersions = versions
}
}
if max >= writeQuorum {
if maxCnt >= writeQuorum {
for _, versions := range diskVersions {
if binary.BigEndian.Uint64(versions) == commonVersions {
return versions
@@ -80,15 +80,15 @@ func reduceCommonDataDir(dataDirs []string, writeQuorum int) (dataDir string) {
dataDirsCount[ddir]++
}
max := 0
maxCnt := 0
for ddir, count := range dataDirsCount {
if max < count {
max = count
if maxCnt < count {
maxCnt = count
dataDir = ddir
}
}
if max >= writeQuorum {
if maxCnt >= writeQuorum {
return dataDir
}
@@ -115,20 +115,20 @@ func reduceErrs(errs []error, ignoredErrs []error) (maxCount int, maxErr error)
errorCounts[err]++
}
max := 0
maxCnt := 0
for err, count := range errorCounts {
switch {
case max < count:
max = count
case maxCnt < count:
maxCnt = count
maxErr = err
// Prefer `nil` over other error values with the same
// number of occurrences.
case max == count && err == nil:
case maxCnt == count && err == nil:
maxErr = err
}
}
return max, maxErr
return maxCnt, maxErr
}
// reduceQuorumErrs behaves like reduceErrs by only for returning
@@ -295,34 +295,34 @@ func shuffleDisksAndPartsMetadata(disks []StorageAPI, partsMetadata []FileInfo,
return shuffledDisks, shuffledPartsMetadata
}
// Return shuffled partsMetadata depending on distribution.
func shufflePartsMetadata(partsMetadata []FileInfo, distribution []int) (shuffledPartsMetadata []FileInfo) {
func shuffleWithDist[T any](input []T, distribution []int) []T {
if distribution == nil {
return partsMetadata
return input
}
shuffledPartsMetadata = make([]FileInfo, len(partsMetadata))
// Shuffle slice xl metadata for expected distribution.
for index := range partsMetadata {
shuffled := make([]T, len(input))
for index := range input {
blockIndex := distribution[index]
shuffledPartsMetadata[blockIndex-1] = partsMetadata[index]
shuffled[blockIndex-1] = input[index]
}
return shuffledPartsMetadata
return shuffled
}
// Return shuffled partsMetadata depending on distribution.
func shufflePartsMetadata(partsMetadata []FileInfo, distribution []int) []FileInfo {
return shuffleWithDist[FileInfo](partsMetadata, distribution)
}
// shuffleCheckParts - shuffle CheckParts slice depending on the
// erasure distribution.
func shuffleCheckParts(parts []int, distribution []int) []int {
return shuffleWithDist[int](parts, distribution)
}
// shuffleDisks - shuffle input disks slice depending on the
// erasure distribution. Return shuffled slice of disks with
// their expected distribution.
func shuffleDisks(disks []StorageAPI, distribution []int) (shuffledDisks []StorageAPI) {
if distribution == nil {
return disks
}
shuffledDisks = make([]StorageAPI, len(disks))
// Shuffle disks for expected distribution.
for index := range disks {
blockIndex := distribution[index]
shuffledDisks[blockIndex-1] = disks[index]
}
return shuffledDisks
func shuffleDisks(disks []StorageAPI, distribution []int) []StorageAPI {
return shuffleWithDist[StorageAPI](disks, distribution)
}
// evalDisks - returns a new slice of disks where nil is set if
+4 -2
View File
@@ -514,8 +514,10 @@ func listObjectParities(partsMetadata []FileInfo, errs []error) (parities []int)
if metadata.Deleted || metadata.Size == 0 {
parities[index] = totalShards / 2
} else if metadata.TransitionStatus == lifecycle.TransitionComplete {
// For tiered objects, read quorum is N/2+1 to ensure simple majority on xl.meta. It is not equal to EcM because the data integrity is entrusted with the warm tier.
parities[index] = totalShards - (totalShards/2 + 1)
// For tiered objects, read quorum is N/2+1 to ensure simple majority on xl.meta.
// It is not equal to EcM because the data integrity is entrusted with the warm tier.
// However, we never go below EcM, in case of a EcM=EcN setup.
parities[index] = max(totalShards-(totalShards/2+1), metadata.Erasure.ParityBlocks)
} else {
parities[index] = metadata.Erasure.ParityBlocks
}
+75 -38
View File
@@ -480,6 +480,7 @@ func (er erasureObjects) newMultipartUpload(ctx context.Context, bucket string,
if opts.WantChecksum != nil && opts.WantChecksum.Type.IsSet() {
userDefined[hash.MinIOMultipartChecksum] = opts.WantChecksum.Type.String()
userDefined[hash.MinIOMultipartChecksumType] = opts.WantChecksum.Type.ObjType()
}
modTime := opts.MTime
@@ -508,6 +509,7 @@ func (er erasureObjects) newMultipartUpload(ctx context.Context, bucket string,
return &NewMultipartUploadResult{
UploadID: uploadID,
ChecksumAlgo: userDefined[hash.MinIOMultipartChecksum],
ChecksumType: userDefined[hash.MinIOMultipartChecksumType],
}, nil
}
@@ -626,17 +628,13 @@ func (er erasureObjects) PutObjectPart(ctx context.Context, bucket, object, uplo
switch size := data.Size(); {
case size == 0:
buffer = make([]byte, 1) // Allocate at least a byte to reach EOF
case size == -1:
if size := data.ActualSize(); size > 0 && size < fi.Erasure.BlockSize {
// Account for padding and forced compression overhead and encryption.
buffer = make([]byte, data.ActualSize()+256+32+32, data.ActualSize()*2+512)
case size >= fi.Erasure.BlockSize || size == -1:
if int64(globalBytePoolCap.Load().Width()) < fi.Erasure.BlockSize {
buffer = make([]byte, fi.Erasure.BlockSize, 2*fi.Erasure.BlockSize)
} else {
buffer = globalBytePoolCap.Load().Get()
defer globalBytePoolCap.Load().Put(buffer)
}
case size >= fi.Erasure.BlockSize:
buffer = globalBytePoolCap.Load().Get()
defer globalBytePoolCap.Load().Put(buffer)
case size < fi.Erasure.BlockSize:
// No need to allocate fully fi.Erasure.BlockSize buffer if the incoming data is smaller.
buffer = make([]byte, size, 2*size+int64(fi.Erasure.ParityBlocks+fi.Erasure.DataBlocks-1))
@@ -744,6 +742,15 @@ func (er erasureObjects) PutObjectPart(ctx context.Context, bucket, object, uplo
ctx = plkctx.Context()
defer partIDLock.Unlock(plkctx)
// Read lock for upload id, only held while reading the upload metadata.
uploadIDRLock := er.NewNSLock(bucket, pathJoin(object, uploadID))
rlkctx, err := uploadIDRLock.GetRLock(ctx, globalOperationTimeout)
if err != nil {
return PartInfo{}, err
}
ctx = rlkctx.Context()
defer uploadIDRLock.RUnlock(rlkctx)
onlineDisks, err = er.renamePart(ctx, onlineDisks, minioMetaTmpBucket, tmpPartPath, minioMetaMultipartBucket, partPath, partFI, writeQuorum)
if err != nil {
if errors.Is(err, errFileNotFound) {
@@ -760,15 +767,16 @@ func (er erasureObjects) PutObjectPart(ctx context.Context, bucket, object, uplo
// Return success.
return PartInfo{
PartNumber: partInfo.Number,
ETag: partInfo.ETag,
LastModified: partInfo.ModTime,
Size: partInfo.Size,
ActualSize: partInfo.ActualSize,
ChecksumCRC32: partInfo.Checksums["CRC32"],
ChecksumCRC32C: partInfo.Checksums["CRC32C"],
ChecksumSHA1: partInfo.Checksums["SHA1"],
ChecksumSHA256: partInfo.Checksums["SHA256"],
PartNumber: partInfo.Number,
ETag: partInfo.ETag,
LastModified: partInfo.ModTime,
Size: partInfo.Size,
ActualSize: partInfo.ActualSize,
ChecksumCRC32: partInfo.Checksums["CRC32"],
ChecksumCRC32C: partInfo.Checksums["CRC32C"],
ChecksumSHA1: partInfo.Checksums["SHA1"],
ChecksumSHA256: partInfo.Checksums["SHA256"],
ChecksumCRC64NVME: partInfo.Checksums["CRC64NVME"],
}, nil
}
@@ -890,6 +898,7 @@ func (er erasureObjects) ListObjectParts(ctx context.Context, bucket, object, up
result.PartNumberMarker = partNumberMarker
result.UserDefined = cloneMSS(fi.Metadata)
result.ChecksumAlgorithm = fi.Metadata[hash.MinIOMultipartChecksum]
result.ChecksumType = fi.Metadata[hash.MinIOMultipartChecksumType]
if maxParts == 0 {
return result, nil
@@ -936,15 +945,16 @@ func (er erasureObjects) ListObjectParts(ctx context.Context, bucket, object, up
count := maxParts
for _, objPart := range objParts {
result.Parts = append(result.Parts, PartInfo{
PartNumber: objPart.Number,
LastModified: objPart.ModTime,
ETag: objPart.ETag,
Size: objPart.Size,
ActualSize: objPart.ActualSize,
ChecksumCRC32: objPart.Checksums["CRC32"],
ChecksumCRC32C: objPart.Checksums["CRC32C"],
ChecksumSHA1: objPart.Checksums["SHA1"],
ChecksumSHA256: objPart.Checksums["SHA256"],
PartNumber: objPart.Number,
LastModified: objPart.ModTime,
ETag: objPart.ETag,
Size: objPart.Size,
ActualSize: objPart.ActualSize,
ChecksumCRC32: objPart.Checksums["CRC32"],
ChecksumCRC32C: objPart.Checksums["CRC32C"],
ChecksumSHA1: objPart.Checksums["SHA1"],
ChecksumSHA256: objPart.Checksums["SHA256"],
ChecksumCRC64NVME: objPart.Checksums["CRC64NVME"],
})
count--
if count == 0 {
@@ -1126,12 +1136,12 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
// Checksum type set when upload started.
var checksumType hash.ChecksumType
if cs := fi.Metadata[hash.MinIOMultipartChecksum]; cs != "" {
checksumType = hash.NewChecksumType(cs)
checksumType = hash.NewChecksumType(cs, fi.Metadata[hash.MinIOMultipartChecksumType])
if opts.WantChecksum != nil && !opts.WantChecksum.Type.Is(checksumType) {
return oi, InvalidArgument{
Bucket: bucket,
Object: fi.Name,
Err: fmt.Errorf("checksum type mismatch"),
Err: fmt.Errorf("checksum type mismatch. got %q (%s) expected %q (%s)", checksumType.String(), checksumType.ObjType(), opts.WantChecksum.Type.String(), opts.WantChecksum.Type.ObjType()),
}
}
}
@@ -1211,6 +1221,9 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
// Allocate parts similar to incoming slice.
fi.Parts = make([]ObjectPartInfo, len(parts))
var checksum hash.Checksum
checksum.Type = checksumType
// Validate each part and then commit to disk.
for i, part := range parts {
partIdx := objectPartIndex(currentFI.Parts, part.PartNumber)
@@ -1244,10 +1257,11 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
}
}
wantCS := map[string]string{
hash.ChecksumCRC32.String(): part.ChecksumCRC32,
hash.ChecksumCRC32C.String(): part.ChecksumCRC32C,
hash.ChecksumSHA1.String(): part.ChecksumSHA1,
hash.ChecksumSHA256.String(): part.ChecksumSHA256,
hash.ChecksumCRC32.String(): part.ChecksumCRC32,
hash.ChecksumCRC32C.String(): part.ChecksumCRC32C,
hash.ChecksumSHA1.String(): part.ChecksumSHA1,
hash.ChecksumSHA256.String(): part.ChecksumSHA256,
hash.ChecksumCRC64NVME.String(): part.ChecksumCRC64NVME,
}
if wantCS[checksumType.String()] != crc {
return oi, InvalidPart{
@@ -1262,6 +1276,15 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
PartNumber: part.PartNumber,
}
}
if checksumType.FullObjectRequested() {
if err := checksum.AddPart(*cs, expPart.ActualSize); err != nil {
return oi, InvalidPart{
PartNumber: part.PartNumber,
ExpETag: "<nil>",
GotETag: err.Error(),
}
}
}
checksumCombined = append(checksumCombined, cs.Raw...)
}
@@ -1292,9 +1315,19 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
}
if opts.WantChecksum != nil {
err := opts.WantChecksum.Matches(checksumCombined, len(parts))
if err != nil {
return oi, err
if checksumType.FullObjectRequested() {
if opts.WantChecksum.Encoded != checksum.Encoded {
err := hash.ChecksumMismatch{
Want: opts.WantChecksum.Encoded,
Got: checksum.Encoded,
}
return oi, err
}
} else {
err := opts.WantChecksum.Matches(checksumCombined, len(parts))
if err != nil {
return oi, err
}
}
}
@@ -1308,14 +1341,18 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
if checksumType.IsSet() {
checksumType |= hash.ChecksumMultipart | hash.ChecksumIncludesMultipart
var cs *hash.Checksum
cs = hash.NewChecksumFromData(checksumType, checksumCombined)
fi.Checksum = cs.AppendTo(nil, checksumCombined)
checksum.Type = checksumType
if !checksumType.FullObjectRequested() {
checksum = *hash.NewChecksumFromData(checksumType, checksumCombined)
}
fi.Checksum = checksum.AppendTo(nil, checksumCombined)
if opts.EncryptFn != nil {
fi.Checksum = opts.EncryptFn("object-checksum", fi.Checksum)
}
}
delete(fi.Metadata, hash.MinIOMultipartChecksum) // Not needed in final object.
// Remove superfluous internal headers.
delete(fi.Metadata, hash.MinIOMultipartChecksum)
delete(fi.Metadata, hash.MinIOMultipartChecksumType)
// Save the final object size and modtime.
fi.Size = objectSize
+29 -9
View File
@@ -27,6 +27,8 @@ import (
"net/http"
"path"
"runtime"
"slices"
"sort"
"strconv"
"strings"
"sync"
@@ -1706,8 +1708,21 @@ func (er erasureObjects) DeleteObjects(ctx context.Context, bucket string, objec
}
dedupVersions := make([]FileInfoVersions, 0, len(versionsMap))
for _, version := range versionsMap {
dedupVersions = append(dedupVersions, version)
for _, fivs := range versionsMap {
// Removal of existing versions and adding a delete marker in the same
// request is supported. At the same time, we cannot allow adding
// two delete markers on top of any object. To avoid this situation,
// we will sort deletions to execute existing deletion first,
// then add only one delete marker if requested
sort.SliceStable(fivs.Versions, func(i, j int) bool {
return !fivs.Versions[i].Deleted
})
if idx := slices.IndexFunc(fivs.Versions, func(fi FileInfo) bool {
return fi.Deleted
}); idx > -1 {
fivs.Versions = fivs.Versions[:idx+1]
}
dedupVersions = append(dedupVersions, fivs)
}
// Initialize list of errors.
@@ -1732,12 +1747,6 @@ func (er erasureObjects) DeleteObjects(ctx context.Context, bucket string, objec
continue
}
for _, v := range dedupVersions[i].Versions {
if err == errFileNotFound || err == errFileVersionNotFound {
if !dobjects[v.Idx].DeleteMarker {
// Not delete marker, if not found, ok.
continue
}
}
delObjErrs[index][v.Idx] = err
}
}
@@ -1757,6 +1766,13 @@ func (er erasureObjects) DeleteObjects(ctx context.Context, bucket string, objec
}
}
err := reduceWriteQuorumErrs(ctx, diskErrs, objectOpIgnoredErrs, writeQuorums[objIndex])
if err == nil {
dobjects[objIndex].found = true
} else if isErrVersionNotFound(err) || isErrObjectNotFound(err) {
if !dobjects[objIndex].DeleteMarker {
err = nil
}
}
if objects[objIndex].VersionID != "" {
errs[objIndex] = toObjectErr(err, bucket, objects[objIndex].ObjectName, objects[objIndex].VersionID)
} else {
@@ -2376,7 +2392,11 @@ func (er erasureObjects) TransitionObject(ctx context.Context, bucket, object st
}()
var rv remoteVersionID
rv, err = tgtClient.Put(ctx, destObj, pr, fi.Size)
rv, err = tgtClient.PutWithMeta(ctx, destObj, pr, fi.Size, map[string]string{
"name": object, // preserve the original name of the object on the remote tier object metadata.
// this is just for future reverse lookup() purposes (applies only for new objects)
// does not apply retro-actively on already transitioned objects.
})
pr.CloseWithError(err)
if err != nil {
traceFn(ILMTransition, nil, err)
+69
View File
@@ -131,6 +131,75 @@ func TestErasureDeleteObjectBasic(t *testing.T) {
removeRoots(fsDirs)
}
func TestDeleteObjectsVersionedTwoPools(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
obj, fsDirs, err := prepareErasurePools()
if err != nil {
t.Fatal("Unable to initialize 'Erasure' object layer.", err)
}
// Remove all dirs.
for _, dir := range fsDirs {
defer os.RemoveAll(dir)
}
bucketName := "bucket"
objectName := "myobject"
err = obj.MakeBucket(ctx, bucketName, MakeBucketOptions{
VersioningEnabled: true,
})
if err != nil {
t.Fatal(err)
}
z, ok := obj.(*erasureServerPools)
if !ok {
t.Fatal("unexpected object layer type")
}
versions := make([]string, 2)
for i := range z.serverPools {
objInfo, err := z.serverPools[i].PutObject(ctx, bucketName, objectName,
mustGetPutObjReader(t, bytes.NewReader([]byte("abcd")), int64(len("abcd")), "", ""), ObjectOptions{
Versioned: true,
})
if err != nil {
t.Fatalf("Erasure Object upload failed: <ERROR> %s", err)
}
versions[i] = objInfo.VersionID
}
// Remove and check the version in the second pool, then
// remove and check the version in the first pool
for testIdx, vid := range []string{versions[1], versions[0]} {
names := []ObjectToDelete{
{
ObjectV: ObjectV{
ObjectName: objectName,
VersionID: vid,
},
},
}
_, delErrs := obj.DeleteObjects(ctx, bucketName, names, ObjectOptions{
Versioned: true,
})
for i := range delErrs {
if delErrs[i] != nil {
t.Errorf("Test %d: Failed to remove object `%v` with the error: `%v`", testIdx, names[i], delErrs[i])
}
_, statErr := obj.GetObjectInfo(ctx, bucketName, objectName, ObjectOptions{
VersionID: names[i].ObjectV.VersionID,
})
switch statErr.(type) {
case VersionNotFound:
default:
t.Errorf("Test %d: Object %s is not removed", testIdx, objectName)
}
}
}
}
func TestDeleteObjectsVersioned(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
+1 -1
View File
@@ -845,7 +845,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
// to decommission, just skip it, this also includes
// any other versions that have already expired.
remainingVersions := len(fivs.Versions) - expired
if version.Deleted && remainingVersions == 1 {
if version.Deleted && remainingVersions == 1 && rcfg == nil {
decommissioned++
stopFn(version.Size, errors.New("DELETE marked object with no other non-current versions will be skipped"))
continue
+2 -2
View File
@@ -32,9 +32,9 @@ func prepareErasurePools() (ObjectLayer, []string, error) {
pools := mustGetPoolEndpoints(0, fsDirs[:16]...)
pools = append(pools, mustGetPoolEndpoints(1, fsDirs[16:]...)...)
// Everything is fine, should return nil
objLayer, err := newErasureServerPools(context.Background(), pools)
objLayer, _, err := initObjectLayer(context.Background(), pools)
if err != nil {
removeRoots(fsDirs)
return nil, nil, err
}
return objLayer, fsDirs, nil
+42 -91
View File
@@ -256,7 +256,7 @@ func (z *erasureServerPools) NewNSLock(bucket string, objects ...string) RWLocke
// GetDisksID will return disks by their ID.
func (z *erasureServerPools) GetDisksID(ids ...string) []StorageAPI {
idMap := make(map[string]struct{})
idMap := make(map[string]struct{}, len(ids))
for _, id := range ids {
idMap[id] = struct{}{}
}
@@ -357,7 +357,7 @@ func (p serverPoolsAvailableSpace) TotalAvailable() uint64 {
// FilterMaxUsed will filter out any pools that has used percent bigger than max,
// unless all have that, in which case all are preserved.
func (p serverPoolsAvailableSpace) FilterMaxUsed(max int) {
func (p serverPoolsAvailableSpace) FilterMaxUsed(maxUsed int) {
// We aren't modifying p, only entries in it, so we don't need to receive a pointer.
if len(p) <= 1 {
// Nothing to do.
@@ -365,7 +365,7 @@ func (p serverPoolsAvailableSpace) FilterMaxUsed(max int) {
}
var ok bool
for _, z := range p {
if z.Available > 0 && z.MaxUsedPct < max {
if z.Available > 0 && z.MaxUsedPct < maxUsed {
ok = true
break
}
@@ -378,7 +378,7 @@ func (p serverPoolsAvailableSpace) FilterMaxUsed(max int) {
// Remove entries that are above.
for i, z := range p {
if z.Available > 0 && z.MaxUsedPct < max {
if z.Available > 0 && z.MaxUsedPct < maxUsed {
continue
}
p[i].Available = 0
@@ -1248,89 +1248,42 @@ func (z *erasureServerPools) DeleteObjects(ctx context.Context, bucket string, o
ctx = lkctx.Context()
defer multiDeleteLock.Unlock(lkctx)
// Fetch location of up to 10 objects concurrently.
poolObjIdxMap := map[int][]ObjectToDelete{}
origIndexMap := map[int][]int{}
dObjectsByPool := make([][]DeletedObject, len(z.serverPools))
dErrsByPool := make([][]error, len(z.serverPools))
// Always perform 1/10th of the number of objects per delete
concurrent := len(objects) / 10
if concurrent <= 10 {
// if we cannot get 1/10th then choose the number of
// objects as concurrent.
concurrent = len(objects)
}
var mu sync.Mutex
eg := errgroup.WithNErrs(len(objects)).WithConcurrency(concurrent)
for j, obj := range objects {
j := j
obj := obj
eg := errgroup.WithNErrs(len(z.serverPools)).WithConcurrency(len(z.serverPools))
for i, pool := range z.serverPools {
i := i
pool := pool
eg.Go(func() error {
pinfo, _, err := z.getPoolInfoExistingWithOpts(ctx, bucket, obj.ObjectName, ObjectOptions{
NoLock: true,
})
if err != nil {
if !isErrObjectNotFound(err) && !isErrVersionNotFound(err) {
derrs[j] = err
}
dobjects[j] = DeletedObject{
ObjectName: decodeDirObject(obj.ObjectName),
VersionID: obj.VersionID,
}
return nil
}
// Delete marker already present we are not going to create new delete markers.
if pinfo.ObjInfo.DeleteMarker && obj.VersionID == "" {
dobjects[j] = DeletedObject{
DeleteMarker: pinfo.ObjInfo.DeleteMarker,
DeleteMarkerVersionID: pinfo.ObjInfo.VersionID,
DeleteMarkerMTime: DeleteMarkerMTime{pinfo.ObjInfo.ModTime},
ObjectName: decodeDirObject(pinfo.ObjInfo.Name),
}
return nil
}
idx := pinfo.Index
mu.Lock()
defer mu.Unlock()
poolObjIdxMap[idx] = append(poolObjIdxMap[idx], obj)
origIndexMap[idx] = append(origIndexMap[idx], j)
dObjectsByPool[i], dErrsByPool[i] = pool.DeleteObjects(ctx, bucket, objects, opts)
return nil
}, j)
}, i)
}
eg.Wait() // wait to check all the pools.
if len(poolObjIdxMap) > 0 {
// Delete concurrently in all server pools.
var wg sync.WaitGroup
wg.Add(len(z.serverPools))
for idx, pool := range z.serverPools {
go func(idx int, pool *erasureSets) {
defer wg.Done()
objs := poolObjIdxMap[idx]
if len(objs) == 0 {
return
}
orgIndexes := origIndexMap[idx]
deletedObjects, errs := pool.DeleteObjects(ctx, bucket, objs, opts)
mu.Lock()
for i, derr := range errs {
if derr != nil {
derrs[orgIndexes[i]] = derr
}
deletedObjects[i].ObjectName = decodeDirObject(deletedObjects[i].ObjectName)
dobjects[orgIndexes[i]] = deletedObjects[i]
}
mu.Unlock()
}(idx, pool)
for i := range dobjects {
// Iterate over pools
for pool := range z.serverPools {
if dErrsByPool[pool][i] == nil && dObjectsByPool[pool][i].found {
// A fast exit when the object is found and removed
dobjects[i] = dObjectsByPool[pool][i]
derrs[i] = nil
break
}
if derrs[i] == nil {
// No error related to this object is found, assign this pool result
// whether it is nil because there is no object found or because of
// some other errors such erasure quorum errors.
dobjects[i] = dObjectsByPool[pool][i]
derrs[i] = dErrsByPool[pool][i]
}
}
wg.Wait()
}
for i := range dobjects {
dobjects[i].ObjectName = decodeDirObject(dobjects[i].ObjectName)
}
return dobjects, derrs
}
@@ -1858,16 +1811,6 @@ func (z *erasureServerPools) PutObjectPart(ctx context.Context, bucket, object,
return PartInfo{}, err
}
// Read lock for upload id.
// Only held while reading the upload metadata.
uploadIDRLock := z.NewNSLock(bucket, pathJoin(object, uploadID))
rlkctx, err := uploadIDRLock.GetRLock(ctx, globalOperationTimeout)
if err != nil {
return PartInfo{}, err
}
ctx = rlkctx.Context()
defer uploadIDRLock.RUnlock(rlkctx)
if z.SinglePool() {
return z.serverPools[0].PutObjectPart(ctx, bucket, object, uploadID, partID, data, opts)
}
@@ -2478,6 +2421,7 @@ func (z *erasureServerPools) Walk(ctx context.Context, bucket, prefix string, re
// HealObjectFn closure function heals the object.
type HealObjectFn func(bucket, object, versionID string, scanMode madmin.HealScanMode) error
// List a prefix or a single object versions and heal
func (z *erasureServerPools) HealObjects(ctx context.Context, bucket, prefix string, opts madmin.HealOpts, healObjectFn HealObjectFn) error {
healEntry := func(bucket string, entry metaCacheEntry, scanMode madmin.HealScanMode) error {
if entry.isDir() {
@@ -2541,7 +2485,7 @@ func (z *erasureServerPools) HealObjects(ctx context.Context, bucket, prefix str
go func(idx int, set *erasureObjects) {
defer wg.Done()
errs[idx] = set.listAndHeal(ctx, bucket, prefix, opts.ScanMode, healEntry)
errs[idx] = set.listAndHeal(ctx, bucket, prefix, opts.Recursive, opts.ScanMode, healEntry)
}(idx, set)
}
wg.Wait()
@@ -2593,15 +2537,22 @@ func (z *erasureServerPools) HealObject(ctx context.Context, bucket, object, ver
}
}
hr := madmin.HealResultItem{
Type: madmin.HealItemObject,
Bucket: bucket,
Object: object,
VersionID: versionID,
}
// At this stage, all errors are 'not found'
if versionID != "" {
return madmin.HealResultItem{}, VersionNotFound{
return hr, VersionNotFound{
Bucket: bucket,
Object: object,
VersionID: versionID,
}
}
return madmin.HealResultItem{}, ObjectNotFound{
return hr, ObjectNotFound{
Bucket: bucket,
Object: object,
}
+3 -7
View File
@@ -39,6 +39,7 @@ import (
"github.com/minio/minio/internal/logger"
"github.com/minio/pkg/v3/console"
"github.com/minio/pkg/v3/sync/errgroup"
"github.com/puzpuzpuz/xsync/v3"
)
// setsDsyncLockers is encapsulated type for Close()
@@ -701,9 +702,8 @@ func (s *erasureSets) getHashedSet(input string) (set *erasureObjects) {
}
// listDeletedBuckets lists deleted buckets from all disks.
func listDeletedBuckets(ctx context.Context, storageDisks []StorageAPI, delBuckets map[string]VolInfo, readQuorum int) error {
func listDeletedBuckets(ctx context.Context, storageDisks []StorageAPI, delBuckets *xsync.MapOf[string, VolInfo], readQuorum int) error {
g := errgroup.WithNErrs(len(storageDisks))
var mu sync.Mutex
for index := range storageDisks {
index := index
g.Go(func() error {
@@ -722,11 +722,7 @@ func listDeletedBuckets(ctx context.Context, storageDisks []StorageAPI, delBucke
vi, err := storageDisks[index].StatVol(ctx, pathJoin(minioMetaBucket, bucketMetaPrefix, deletedBucketsPrefix, volName))
if err == nil {
vi.Name = strings.TrimSuffix(volName, SlashSeparator)
mu.Lock()
if _, ok := delBuckets[volName]; !ok {
delBuckets[volName] = vi
}
mu.Unlock()
delBuckets.Store(volName, vi)
}
}
return nil
+8 -2
View File
@@ -24,6 +24,8 @@ import (
"errors"
"fmt"
"io"
"net"
"net/http"
"os"
"path"
"strings"
@@ -286,6 +288,10 @@ func (driver *ftpDriver) CheckPasswd(c *ftp.Context, username, password string)
}
func (driver *ftpDriver) getMinIOClient(ctx *ftp.Context) (*minio.Client, error) {
tr := http.RoundTripper(globalRemoteFTPClientTransport)
if host, _, err := net.SplitHostPort(ctx.Sess.RemoteAddr().String()); err == nil {
tr = forwardForTransport{tr: tr, fwd: host}
}
ui, ok := globalIAMSys.GetUser(context.Background(), ctx.Sess.LoginUser())
if !ok && !globalIAMSys.LDAPConfig.Enabled() {
return nil, errNoSuchUser
@@ -363,7 +369,7 @@ func (driver *ftpDriver) getMinIOClient(ctx *ftp.Context) (*minio.Client, error)
return minio.New(driver.endpoint, &minio.Options{
Creds: mcreds,
Secure: globalIsTLS,
Transport: globalRemoteFTPClientTransport,
Transport: tr,
})
}
@@ -377,7 +383,7 @@ func (driver *ftpDriver) getMinIOClient(ctx *ftp.Context) (*minio.Client, error)
return minio.New(driver.endpoint, &minio.Options{
Creds: credentials.NewStaticV4(ui.Credentials.AccessKey, ui.Credentials.SecretKey, ""),
Secure: globalIsTLS,
Transport: globalRemoteFTPClientTransport,
Transport: tr,
})
}
+32 -20
View File
@@ -22,19 +22,17 @@ import (
"net"
"net/http"
"path"
"path/filepath"
"runtime/debug"
"strings"
"sync/atomic"
"time"
"unicode"
"github.com/dustin/go-humanize"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/set"
"github.com/minio/minio/internal/grid"
xnet "github.com/minio/pkg/v3/net"
"golang.org/x/exp/maps"
"golang.org/x/exp/slices"
"github.com/minio/minio/internal/amztime"
"github.com/minio/minio/internal/config/dns"
@@ -75,7 +73,7 @@ const (
// and must not set by clients
func containsReservedMetadata(header http.Header) bool {
for key := range header {
if slices.Contains(maps.Keys(validSSEReplicationHeaders), key) {
if _, ok := validSSEReplicationHeaders[key]; ok {
return false
}
if stringsHasPrefixFold(key, ReservedMetadataPrefix) {
@@ -295,12 +293,6 @@ func parseAmzDateHeader(req *http.Request) (time.Time, APIErrorCode) {
return time.Time{}, ErrMissingDateHeader
}
// Bad path components to be rejected by the path validity handler.
const (
dotdotComponent = ".."
dotComponent = "."
)
func hasBadHost(host string) error {
if globalIsCICD && strings.TrimSpace(host) == "" {
// under CI/CD test setups ignore empty hosts as invalid hosts
@@ -313,21 +305,41 @@ func hasBadHost(host string) error {
// Check if the incoming path has bad path components,
// such as ".." and "."
func hasBadPathComponent(path string) bool {
if len(path) > 4096 {
// path cannot be greater than Linux PATH_MAX
// this is to avoid a busy loop, that can happen
// if the caller sends path of following style
// a/a/a/a/a/a/a/a...
n := len(path)
if n > 32<<10 {
// At 32K we are beyond reasonable.
return true
}
path = filepath.ToSlash(strings.TrimSpace(path)) // For windows '\' must be converted to '/'
for _, p := range strings.Split(path, SlashSeparator) {
switch strings.TrimSpace(p) {
case dotdotComponent:
i := 0
// Skip leading slashes (for sake of Windows \ is included as well)
for i < n && (path[i] == SlashSeparatorChar || path[i] == '\\') {
i++
}
for i < n {
// Find the next segment
start := i
for i < n && path[i] != SlashSeparatorChar && path[i] != '\\' {
i++
}
// Trim whitespace of segment
segmentStart, segmentEnd := start, i
for segmentStart < segmentEnd && unicode.IsSpace(rune(path[segmentStart])) {
segmentStart++
}
for segmentEnd > segmentStart && unicode.IsSpace(rune(path[segmentEnd-1])) {
segmentEnd--
}
// Check for ".." or "."
switch {
case segmentEnd-segmentStart == 2 && path[segmentStart] == '.' && path[segmentStart+1] == '.':
return true
case dotComponent:
case segmentEnd-segmentStart == 1 && path[segmentStart] == '.':
return true
}
i++
}
return false
}
+25
View File
@@ -22,6 +22,7 @@ import (
"net/http/httptest"
"net/url"
"strconv"
"strings"
"testing"
"github.com/minio/minio/internal/crypto"
@@ -184,3 +185,27 @@ func TestSSETLSHandler(t *testing.T) {
}
}
}
func Benchmark_hasBadPathComponent(t *testing.B) {
tests := []struct {
name string
input string
want bool
}{
{name: "empty", input: "", want: false},
{name: "backslashes", input: `\a\a\ \\ \\\\\\\`, want: false},
{name: "long", input: strings.Repeat("a/", 2000), want: false},
{name: "long-fail", input: strings.Repeat("a/", 2000) + "../..", want: true},
}
for _, tt := range tests {
t.Run(tt.name, func(b *testing.B) {
b.SetBytes(int64(len(tt.input)))
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if got := hasBadPathComponent(tt.input); got != tt.want {
t.Fatalf("hasBadPathComponent() = %v, want %v", got, tt.want)
}
}
})
}
}
+5 -2
View File
@@ -447,7 +447,7 @@ func getHostName(r *http.Request) (hostName string) {
}
// Proxy any request to an endpoint.
func proxyRequest(ctx context.Context, w http.ResponseWriter, r *http.Request, ep ProxyEndpoint) (success bool) {
func proxyRequest(ctx context.Context, w http.ResponseWriter, r *http.Request, ep ProxyEndpoint, returnErr bool) (success bool) {
success = true
// Make sure we remove any existing headers before
@@ -462,7 +462,10 @@ func proxyRequest(ctx context.Context, w http.ResponseWriter, r *http.Request, e
ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
success = false
if err != nil && !errors.Is(err, context.Canceled) {
replLogIf(GlobalContext, err)
proxyLogIf(GlobalContext, err)
}
if returnErr {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
}
},
})
+8 -2
View File
@@ -793,7 +793,10 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
for _, item := range listedConfigItems[stsListKey] {
userName := path.Dir(item)
// loadUser() will delete expired user during the load.
iamLogIf(ctx, iamOS.loadUser(ctx, userName, stsUser, stsAccountsFromStore))
err := iamOS.loadUser(ctx, userName, stsUser, stsAccountsFromStore)
if err != nil && !errors.Is(err, errNoSuchUser) {
iamLogIf(ctx, err)
}
// No need to return errors for failed expiration of STS users
}
@@ -801,7 +804,10 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
// (removed during loadUser() in the loop above) are removed from memory.
for _, item := range listedConfigItems[policyDBSTSUsersListKey] {
stsName := strings.TrimSuffix(item, ".json")
iamLogIf(ctx, iamOS.loadMappedPolicy(ctx, stsName, stsUser, false, stsAccPoliciesFromStore))
err := iamOS.loadMappedPolicy(ctx, stsName, stsUser, false, stsAccPoliciesFromStore)
if err != nil && !errors.Is(err, errNoSuchPolicy) {
iamLogIf(ctx, err)
}
// No need to return errors for failed expiration of STS users
}
+88 -12
View File
@@ -26,6 +26,7 @@ import (
"path"
"sort"
"strings"
"sync"
"time"
jsoniter "github.com/json-iterator/go"
@@ -37,6 +38,7 @@ import (
"github.com/minio/minio/internal/jwt"
"github.com/minio/pkg/v3/env"
"github.com/minio/pkg/v3/policy"
"github.com/minio/pkg/v3/sync/errgroup"
"github.com/puzpuzpuz/xsync/v3"
"golang.org/x/sync/singleflight"
)
@@ -357,6 +359,68 @@ func (c *iamCache) removeGroupFromMembershipsMap(group string) {
}
}
func (c *iamCache) policyDBGetGroups(store *IAMStoreSys, userPolicyPresent bool, groups ...string) ([]string, error) {
var policies []string
for _, group := range groups {
if store.getUsersSysType() == MinIOUsersSysType {
g, ok := c.iamGroupsMap[group]
if !ok {
continue
}
// Group is disabled, so we return no policy - this
// ensures the request is denied.
if g.Status == statusDisabled {
continue
}
}
policy, ok := c.iamGroupPolicyMap.Load(group)
if !ok {
continue
}
policies = append(policies, policy.toSlice()...)
}
found := len(policies) > 0
if found {
return policies, nil
}
if userPolicyPresent {
// if user mapping present and no group policies found
// rely on user policy for access, instead of fallback.
return nil, nil
}
var mu sync.Mutex
// no mappings found, fallback for all groups.
g := errgroup.WithNErrs(len(groups)).WithConcurrency(10) // load like 10 groups at a time.
for index := range groups {
index := index
g.Go(func() error {
err := store.loadMappedPolicy(context.TODO(), groups[index], regUser, true, c.iamGroupPolicyMap)
if err != nil && !errors.Is(err, errNoSuchPolicy) {
return err
}
if errors.Is(err, errNoSuchPolicy) {
return nil
}
policy, _ := c.iamGroupPolicyMap.Load(groups[index])
mu.Lock()
policies = append(policies, policy.toSlice()...)
mu.Unlock()
return nil
}, index)
}
err := errors.Join(g.Wait()...)
return policies, err
}
// policyDBGet - lower-level helper; does not take locks.
//
// If a group is passed, it returns policies associated with the group.
@@ -676,7 +740,8 @@ func (store *IAMStoreSys) LoadIAMCache(ctx context.Context, firstTime bool) erro
type IAMStoreSys struct {
IAMStorageAPI
group *singleflight.Group
group *singleflight.Group
policy *singleflight.Group
}
// HasWatcher - returns if the storage system has a watcher.
@@ -757,21 +822,32 @@ func (store *IAMStoreSys) PolicyDBGet(name string, groups ...string) ([]string,
cache := store.rlock()
defer store.runlock()
policies, _, err := cache.policyDBGet(store, name, false, false)
if err != nil {
return nil, err
}
userPolicyPresent := len(policies) > 0
for _, group := range groups {
ps, _, err := cache.policyDBGet(store, group, true, userPolicyPresent)
getPolicies := func() ([]string, error) {
policies, _, err := cache.policyDBGet(store, name, false, false)
if err != nil {
return nil, err
}
policies = append(policies, ps...)
}
return policies, nil
userPolicyPresent := len(policies) > 0
groupPolicies, err := cache.policyDBGetGroups(store, userPolicyPresent, groups...)
if err != nil {
return nil, err
}
policies = append(policies, groupPolicies...)
return policies, nil
}
if store.policy != nil {
val, err, _ := store.policy.Do(name, func() (interface{}, error) {
return getPolicies()
})
if err != nil {
return nil, err
}
return val.([]string), nil
}
return getPolicies()
}
// AddUsersToGroup - adds users to group, creating the group if needed.
+6 -1
View File
@@ -178,13 +178,18 @@ func (sys *IAMSys) initStore(objAPI ObjectLayer, etcdClient *etcd.Client) {
}
if etcdClient == nil {
var group *singleflight.Group
var (
group *singleflight.Group
policy *singleflight.Group
)
if env.Get("_MINIO_IAM_SINGLE_FLIGHT", config.EnableOn) == config.EnableOn {
group = &singleflight.Group{}
policy = &singleflight.Group{}
}
sys.store = &IAMStoreSys{
IAMStorageAPI: newIAMObjectStore(objAPI, sys.usersSysType),
group: group,
policy: policy,
}
} else {
sys.store = &IAMStoreSys{IAMStorageAPI: newIAMEtcdStore(etcdClient, sys.usersSysType)}
+152 -111
View File
@@ -24,11 +24,20 @@ import (
"fmt"
"strconv"
"sync"
"sync/atomic"
"time"
"github.com/minio/minio/internal/dsync"
)
// Reject new lock requests immediately when this many are queued
// for the local lock mutex.
// We do not block unlocking or maintenance, but they add to the count.
// The limit is set to allow for bursty behavior,
// but prevent requests to overload the server completely.
// Rejected clients are expected to retry.
const lockMutexWaitLimit = 1000
// lockRequesterInfo stores various info from the client for each lock that is requested.
type lockRequesterInfo struct {
Name string // name of the resource lock was requested for
@@ -52,9 +61,25 @@ func isWriteLock(lri []lockRequesterInfo) bool {
//
//msgp:ignore localLocker
type localLocker struct {
mutex sync.Mutex
lockMap map[string][]lockRequesterInfo
lockUID map[string]string // UUID -> resource map.
mutex sync.Mutex
waitMutex atomic.Int32
lockMap map[string][]lockRequesterInfo
lockUID map[string]string // UUID -> resource map.
// the following are updated on every cleanup defined in lockValidityDuration
readers atomic.Int32
writers atomic.Int32
lastCleanup atomic.Pointer[time.Time]
locksOverloaded atomic.Int64
}
// getMutex will lock the mutex.
// Call the returned function to unlock.
func (l *localLocker) getMutex() func() {
l.waitMutex.Add(1)
l.mutex.Lock()
l.waitMutex.Add(-1)
return l.mutex.Unlock
}
func (l *localLocker) String() string {
@@ -76,9 +101,16 @@ func (l *localLocker) Lock(ctx context.Context, args dsync.LockArgs) (reply bool
return false, fmt.Errorf("internal error: localLocker.Lock called with more than %d resources", maxDeleteList)
}
l.mutex.Lock()
defer l.mutex.Unlock()
// If we have too many waiting, reject this at once.
if l.waitMutex.Load() > lockMutexWaitLimit {
l.locksOverloaded.Add(1)
return false, nil
}
// Wait for mutex
defer l.getMutex()()
if ctx.Err() != nil {
return false, ctx.Err()
}
if !l.canTakeLock(args.Resources...) {
// Not all locks can be taken on resources,
// reject it completely.
@@ -117,9 +149,7 @@ func (l *localLocker) Unlock(_ context.Context, args dsync.LockArgs) (reply bool
return false, fmt.Errorf("internal error: localLocker.Unlock called with more than %d resources", maxDeleteList)
}
l.mutex.Lock()
defer l.mutex.Unlock()
err = nil
defer l.getMutex()()
for _, resource := range args.Resources {
lri, ok := l.lockMap[resource]
@@ -164,9 +194,17 @@ func (l *localLocker) RLock(ctx context.Context, args dsync.LockArgs) (reply boo
if len(args.Resources) != 1 {
return false, fmt.Errorf("internal error: localLocker.RLock called with more than one resource")
}
// If we have too many waiting, reject this at once.
if l.waitMutex.Load() > lockMutexWaitLimit {
l.locksOverloaded.Add(1)
return false, nil
}
l.mutex.Lock()
defer l.mutex.Unlock()
// Wait for mutex
defer l.getMutex()()
if ctx.Err() != nil {
return false, ctx.Err()
}
resource := args.Resources[0]
now := UTCNow()
lrInfo := lockRequesterInfo{
@@ -200,8 +238,7 @@ func (l *localLocker) RUnlock(_ context.Context, args dsync.LockArgs) (reply boo
return false, fmt.Errorf("internal error: localLocker.RUnlock called with more than one resource")
}
l.mutex.Lock()
defer l.mutex.Unlock()
defer l.getMutex()()
var lri []lockRequesterInfo
resource := args.Resources[0]
@@ -218,35 +255,28 @@ func (l *localLocker) RUnlock(_ context.Context, args dsync.LockArgs) (reply boo
}
type lockStats struct {
Total int
Writes int
Reads int
Total int
Writes int
Reads int
LockQueue int
LocksAbandoned int
LastCleanup *time.Time
}
func (l *localLocker) stats() lockStats {
l.mutex.Lock()
defer l.mutex.Unlock()
st := lockStats{Total: len(l.lockMap)}
for _, v := range l.lockMap {
if len(v) == 0 {
continue
}
entry := v[0]
if entry.Writer {
st.Writes++
} else {
st.Reads += len(v)
}
return lockStats{
Total: len(l.lockMap),
Reads: int(l.readers.Load()),
Writes: int(l.writers.Load()),
LockQueue: int(l.waitMutex.Load()),
LastCleanup: l.lastCleanup.Load(),
}
return st
}
type localLockMap map[string][]lockRequesterInfo
func (l *localLocker) DupLockMap() localLockMap {
l.mutex.Lock()
defer l.mutex.Unlock()
defer l.getMutex()()
lockCopy := make(map[string][]lockRequesterInfo, len(l.lockMap))
for k, v := range l.lockMap {
@@ -274,108 +304,110 @@ func (l *localLocker) IsLocal() bool {
}
func (l *localLocker) ForceUnlock(ctx context.Context, args dsync.LockArgs) (reply bool, err error) {
select {
case <-ctx.Done():
if ctx.Err() != nil {
return false, ctx.Err()
default:
l.mutex.Lock()
defer l.mutex.Unlock()
if len(args.UID) == 0 {
for _, resource := range args.Resources {
lris, ok := l.lockMap[resource]
if !ok {
continue
}
// Collect uids, so we don't mutate while we delete
uids := make([]string, 0, len(lris))
for _, lri := range lris {
uids = append(uids, lri.UID)
}
}
// Delete collected uids:
for _, uid := range uids {
lris, ok := l.lockMap[resource]
if !ok {
// Just to be safe, delete uuids.
for idx := 0; idx < maxDeleteList; idx++ {
mapID := formatUUID(uid, idx)
if _, ok := l.lockUID[mapID]; !ok {
break
}
delete(l.lockUID, mapID)
}
continue
}
l.removeEntry(resource, dsync.LockArgs{UID: uid}, &lris)
}
}
return true, nil
}
idx := 0
for {
mapID := formatUUID(args.UID, idx)
resource, ok := l.lockUID[mapID]
if !ok {
return idx > 0, nil
}
defer l.getMutex()()
if ctx.Err() != nil {
return false, ctx.Err()
}
if len(args.UID) == 0 {
for _, resource := range args.Resources {
lris, ok := l.lockMap[resource]
if !ok {
// Unexpected inconsistency, delete.
delete(l.lockUID, mapID)
idx++
continue
}
reply = true
l.removeEntry(resource, dsync.LockArgs{UID: args.UID}, &lris)
idx++
// Collect uids, so we don't mutate while we delete
uids := make([]string, 0, len(lris))
for _, lri := range lris {
uids = append(uids, lri.UID)
}
// Delete collected uids:
for _, uid := range uids {
lris, ok := l.lockMap[resource]
if !ok {
// Just to be safe, delete uuids.
for idx := 0; idx < maxDeleteList; idx++ {
mapID := formatUUID(uid, idx)
if _, ok := l.lockUID[mapID]; !ok {
break
}
delete(l.lockUID, mapID)
}
continue
}
l.removeEntry(resource, dsync.LockArgs{UID: uid}, &lris)
}
}
return true, nil
}
idx := 0
for {
mapID := formatUUID(args.UID, idx)
resource, ok := l.lockUID[mapID]
if !ok {
return idx > 0, nil
}
lris, ok := l.lockMap[resource]
if !ok {
// Unexpected inconsistency, delete.
delete(l.lockUID, mapID)
idx++
continue
}
reply = true
l.removeEntry(resource, dsync.LockArgs{UID: args.UID}, &lris)
idx++
}
}
func (l *localLocker) Refresh(ctx context.Context, args dsync.LockArgs) (refreshed bool, err error) {
select {
case <-ctx.Done():
if ctx.Err() != nil {
return false, ctx.Err()
default:
l.mutex.Lock()
defer l.mutex.Unlock()
}
// Check whether uid is still active.
resource, ok := l.lockUID[formatUUID(args.UID, 0)]
defer l.getMutex()()
if ctx.Err() != nil {
return false, ctx.Err()
}
// Check whether uid is still active.
resource, ok := l.lockUID[formatUUID(args.UID, 0)]
if !ok {
return false, nil
}
idx := 0
for {
lris, ok := l.lockMap[resource]
if !ok {
return false, nil
// Inconsistent. Delete UID.
delete(l.lockUID, formatUUID(args.UID, idx))
return idx > 0, nil
}
idx := 0
for {
lris, ok := l.lockMap[resource]
if !ok {
// Inconsistent. Delete UID.
delete(l.lockUID, formatUUID(args.UID, idx))
return idx > 0, nil
}
now := UTCNow()
for i := range lris {
if lris[i].UID == args.UID {
lris[i].TimeLastRefresh = now.UnixNano()
}
}
idx++
resource, ok = l.lockUID[formatUUID(args.UID, idx)]
if !ok {
// No more resources for UID, but we did update at least one.
return true, nil
now := UTCNow()
for i := range lris {
if lris[i].UID == args.UID {
lris[i].TimeLastRefresh = now.UnixNano()
}
}
idx++
resource, ok = l.lockUID[formatUUID(args.UID, idx)]
if !ok {
// No more resources for UID, but we did update at least one.
return true, nil
}
}
}
// Similar to removeEntry but only removes an entry only if the lock entry exists in map.
// Caller must hold 'l.mutex' lock.
func (l *localLocker) expireOldLocks(interval time.Duration) {
l.mutex.Lock()
defer l.mutex.Unlock()
defer l.getMutex()()
var readers, writers int32
for k, lris := range l.lockMap {
modified := false
for i := 0; i < len(lris); {
@@ -393,6 +425,11 @@ func (l *localLocker) expireOldLocks(interval time.Duration) {
lris = append(lris[:i], lris[i+1:]...)
// Check same i
} else {
if lri.Writer {
writers++
} else {
readers++
}
// Move to next
i++
}
@@ -401,6 +438,10 @@ func (l *localLocker) expireOldLocks(interval time.Duration) {
l.lockMap[k] = lris
}
}
t := time.Now()
l.lastCleanup.Store(&t)
l.readers.Store(readers)
l.writers.Store(writers)
}
func newLocker() *localLocker {
+124 -8
View File
@@ -3,6 +3,8 @@ package cmd
// Code generated by github.com/tinylib/msgp DO NOT EDIT.
import (
"time"
"github.com/tinylib/msgp/msgp"
)
@@ -506,6 +508,36 @@ func (z *lockStats) DecodeMsg(dc *msgp.Reader) (err error) {
err = msgp.WrapError(err, "Reads")
return
}
case "LockQueue":
z.LockQueue, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "LockQueue")
return
}
case "LocksAbandoned":
z.LocksAbandoned, err = dc.ReadInt()
if err != nil {
err = msgp.WrapError(err, "LocksAbandoned")
return
}
case "LastCleanup":
if dc.IsNil() {
err = dc.ReadNil()
if err != nil {
err = msgp.WrapError(err, "LastCleanup")
return
}
z.LastCleanup = nil
} else {
if z.LastCleanup == nil {
z.LastCleanup = new(time.Time)
}
*z.LastCleanup, err = dc.ReadTime()
if err != nil {
err = msgp.WrapError(err, "LastCleanup")
return
}
}
default:
err = dc.Skip()
if err != nil {
@@ -518,10 +550,10 @@ func (z *lockStats) DecodeMsg(dc *msgp.Reader) (err error) {
}
// EncodeMsg implements msgp.Encodable
func (z lockStats) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 3
func (z *lockStats) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 6
// write "Total"
err = en.Append(0x83, 0xa5, 0x54, 0x6f, 0x74, 0x61, 0x6c)
err = en.Append(0x86, 0xa5, 0x54, 0x6f, 0x74, 0x61, 0x6c)
if err != nil {
return
}
@@ -550,15 +582,52 @@ func (z lockStats) EncodeMsg(en *msgp.Writer) (err error) {
err = msgp.WrapError(err, "Reads")
return
}
// write "LockQueue"
err = en.Append(0xa9, 0x4c, 0x6f, 0x63, 0x6b, 0x51, 0x75, 0x65, 0x75, 0x65)
if err != nil {
return
}
err = en.WriteInt(z.LockQueue)
if err != nil {
err = msgp.WrapError(err, "LockQueue")
return
}
// write "LocksAbandoned"
err = en.Append(0xae, 0x4c, 0x6f, 0x63, 0x6b, 0x73, 0x41, 0x62, 0x61, 0x6e, 0x64, 0x6f, 0x6e, 0x65, 0x64)
if err != nil {
return
}
err = en.WriteInt(z.LocksAbandoned)
if err != nil {
err = msgp.WrapError(err, "LocksAbandoned")
return
}
// write "LastCleanup"
err = en.Append(0xab, 0x4c, 0x61, 0x73, 0x74, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x75, 0x70)
if err != nil {
return
}
if z.LastCleanup == nil {
err = en.WriteNil()
if err != nil {
return
}
} else {
err = en.WriteTime(*z.LastCleanup)
if err != nil {
err = msgp.WrapError(err, "LastCleanup")
return
}
}
return
}
// MarshalMsg implements msgp.Marshaler
func (z lockStats) MarshalMsg(b []byte) (o []byte, err error) {
func (z *lockStats) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 3
// map header, size 6
// string "Total"
o = append(o, 0x83, 0xa5, 0x54, 0x6f, 0x74, 0x61, 0x6c)
o = append(o, 0x86, 0xa5, 0x54, 0x6f, 0x74, 0x61, 0x6c)
o = msgp.AppendInt(o, z.Total)
// string "Writes"
o = append(o, 0xa6, 0x57, 0x72, 0x69, 0x74, 0x65, 0x73)
@@ -566,6 +635,19 @@ func (z lockStats) MarshalMsg(b []byte) (o []byte, err error) {
// string "Reads"
o = append(o, 0xa5, 0x52, 0x65, 0x61, 0x64, 0x73)
o = msgp.AppendInt(o, z.Reads)
// string "LockQueue"
o = append(o, 0xa9, 0x4c, 0x6f, 0x63, 0x6b, 0x51, 0x75, 0x65, 0x75, 0x65)
o = msgp.AppendInt(o, z.LockQueue)
// string "LocksAbandoned"
o = append(o, 0xae, 0x4c, 0x6f, 0x63, 0x6b, 0x73, 0x41, 0x62, 0x61, 0x6e, 0x64, 0x6f, 0x6e, 0x65, 0x64)
o = msgp.AppendInt(o, z.LocksAbandoned)
// string "LastCleanup"
o = append(o, 0xab, 0x4c, 0x61, 0x73, 0x74, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x75, 0x70)
if z.LastCleanup == nil {
o = msgp.AppendNil(o)
} else {
o = msgp.AppendTime(o, *z.LastCleanup)
}
return
}
@@ -605,6 +687,35 @@ func (z *lockStats) UnmarshalMsg(bts []byte) (o []byte, err error) {
err = msgp.WrapError(err, "Reads")
return
}
case "LockQueue":
z.LockQueue, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "LockQueue")
return
}
case "LocksAbandoned":
z.LocksAbandoned, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
err = msgp.WrapError(err, "LocksAbandoned")
return
}
case "LastCleanup":
if msgp.IsNil(bts) {
bts, err = msgp.ReadNilBytes(bts)
if err != nil {
return
}
z.LastCleanup = nil
} else {
if z.LastCleanup == nil {
z.LastCleanup = new(time.Time)
}
*z.LastCleanup, bts, err = msgp.ReadTimeBytes(bts)
if err != nil {
err = msgp.WrapError(err, "LastCleanup")
return
}
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
@@ -618,7 +729,12 @@ func (z *lockStats) UnmarshalMsg(bts []byte) (o []byte, err error) {
}
// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z lockStats) Msgsize() (s int) {
s = 1 + 6 + msgp.IntSize + 7 + msgp.IntSize + 6 + msgp.IntSize
func (z *lockStats) Msgsize() (s int) {
s = 1 + 6 + msgp.IntSize + 7 + msgp.IntSize + 6 + msgp.IntSize + 10 + msgp.IntSize + 15 + msgp.IntSize + 12
if z.LastCleanup == nil {
s += msgp.NilSize
} else {
s += msgp.TimeSize
}
return
}
+13 -3
View File
@@ -33,8 +33,12 @@ type lockRESTServer struct {
// RefreshHandler - refresh the current lock
func (l *lockRESTServer) RefreshHandler(args *dsync.LockArgs) (*dsync.LockResp, *grid.RemoteErr) {
// Add a timeout similar to what we expect upstream.
ctx, cancel := context.WithTimeout(context.Background(), dsync.DefaultTimeouts.RefreshCall)
defer cancel()
resp := lockRPCRefresh.NewResponse()
refreshed, err := l.ll.Refresh(context.Background(), *args)
refreshed, err := l.ll.Refresh(ctx, *args)
if err != nil {
return l.makeResp(resp, err)
}
@@ -46,8 +50,11 @@ func (l *lockRESTServer) RefreshHandler(args *dsync.LockArgs) (*dsync.LockResp,
// LockHandler - Acquires a lock.
func (l *lockRESTServer) LockHandler(args *dsync.LockArgs) (*dsync.LockResp, *grid.RemoteErr) {
// Add a timeout similar to what we expect upstream.
ctx, cancel := context.WithTimeout(context.Background(), dsync.DefaultTimeouts.Acquire)
defer cancel()
resp := lockRPCLock.NewResponse()
success, err := l.ll.Lock(context.Background(), *args)
success, err := l.ll.Lock(ctx, *args)
if err == nil && !success {
return l.makeResp(resp, errLockConflict)
}
@@ -65,8 +72,11 @@ func (l *lockRESTServer) UnlockHandler(args *dsync.LockArgs) (*dsync.LockResp, *
// RLockHandler - Acquires an RLock.
func (l *lockRESTServer) RLockHandler(args *dsync.LockArgs) (*dsync.LockResp, *grid.RemoteErr) {
// Add a timeout similar to what we expect upstream.
ctx, cancel := context.WithTimeout(context.Background(), dsync.DefaultTimeouts.Acquire)
defer cancel()
resp := lockRPCRLock.NewResponse()
success, err := l.ll.RLock(context.Background(), *args)
success, err := l.ll.RLock(ctx, *args)
if err == nil && !success {
err = errLockConflict
}
+12 -16
View File
@@ -29,6 +29,7 @@ import (
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
jsoniter "github.com/json-iterator/go"
@@ -182,8 +183,7 @@ func (o *listPathOptions) gatherResults(ctx context.Context, in <-chan metaCache
resultsDone := make(chan metaCacheEntriesSorted)
// Copy so we can mutate
resCh := resultsDone
var done bool
var mu sync.Mutex
var done atomic.Bool
resErr := io.EOF
go func() {
@@ -194,9 +194,7 @@ func (o *listPathOptions) gatherResults(ctx context.Context, in <-chan metaCache
// past limit
continue
}
mu.Lock()
returned = done
mu.Unlock()
returned = done.Load()
if returned {
resCh = nil
continue
@@ -250,9 +248,7 @@ func (o *listPathOptions) gatherResults(ctx context.Context, in <-chan metaCache
return func() (metaCacheEntriesSorted, error) {
select {
case <-ctx.Done():
mu.Lock()
done = true
mu.Unlock()
done.Store(true)
return metaCacheEntriesSorted{}, ctx.Err()
case r := <-resultsDone:
return r, resErr
@@ -626,18 +622,18 @@ func calcCommonWritesDeletes(infos []DiskInfo, readQuorum int) (commonWrite, com
}
filter := func(list []uint64) (commonCount uint64) {
max := 0
maxCnt := 0
signatureMap := map[uint64]int{}
for _, v := range list {
signatureMap[v]++
}
for ops, count := range signatureMap {
if max < count && commonCount < ops {
max = count
if maxCnt < count && commonCount < ops {
maxCnt = count
commonCount = ops
}
}
if max < readQuorum {
if maxCnt < readQuorum {
return 0
}
return commonCount
@@ -650,7 +646,7 @@ func calcCommonWritesDeletes(infos []DiskInfo, readQuorum int) (commonWrite, com
func calcCommonCounter(infos []DiskInfo, readQuorum int) (commonCount uint64) {
filter := func() (commonCount uint64) {
max := 0
maxCnt := 0
signatureMap := map[uint64]int{}
for _, info := range infos {
if info.Error != "" {
@@ -660,12 +656,12 @@ func calcCommonCounter(infos []DiskInfo, readQuorum int) (commonCount uint64) {
signatureMap[mutations]++
}
for ops, count := range signatureMap {
if max < count && commonCount < ops {
max = count
if maxCnt < count && commonCount < ops {
maxCnt = count
commonCount = ops
}
}
if max < readQuorum {
if maxCnt < readQuorum {
return 0
}
return commonCount
+17 -3
View File
@@ -685,6 +685,16 @@ func getUsageLastScanActivityMD() MetricDescription {
}
}
func getBucketUsageLastScanActivityMD() MetricDescription {
return MetricDescription{
Namespace: bucketMetricNamespace,
Subsystem: usageSubsystem,
Name: lastActivityTime,
Help: "Time elapsed (in nano seconds) since last scan activity",
Type: gaugeMetric,
}
}
func getBucketUsageQuotaTotalBytesMD() MetricDescription {
return MetricDescription{
Namespace: bucketMetricNamespace,
@@ -1875,13 +1885,17 @@ func getHistogramMetrics(hist *prometheus.HistogramVec, desc MetricDescription,
// add metrics with +Inf label
labels1 := make(map[string]string)
for _, lp := range dtoMetric.GetLabel() {
labels1[*lp.Name] = *lp.Value
if *lp.Name == "api" && toLowerAPILabels {
labels1[*lp.Name] = strings.ToLower(*lp.Value)
} else {
labels1[*lp.Name] = *lp.Value
}
}
labels1["le"] = fmt.Sprintf("%.3f", math.Inf(+1))
metrics = append(metrics, MetricV2{
Description: desc,
VariableLabels: labels1,
Value: dtoMetric.Counter.GetValue(),
Value: float64(dtoMetric.Histogram.GetSampleCount()),
})
}
return metrics
@@ -3239,7 +3253,7 @@ func getBucketUsageMetrics(opts MetricsGroupOpts) *MetricsGroupV2 {
}
metrics = append(metrics, MetricV2{
Description: getUsageLastScanActivityMD(),
Description: getBucketUsageLastScanActivityMD(),
Value: float64(time.Since(dataUsageInfo.LastUpdate)),
})
+133 -2
View File
@@ -18,13 +18,15 @@
package cmd
import (
"slices"
"strings"
"testing"
"time"
"github.com/prometheus/client_golang/prometheus"
)
func TestGetHistogramMetrics(t *testing.T) {
func TestGetHistogramMetrics_BucketCount(t *testing.T) {
histBuckets := []float64{0.05, 0.1, 0.25, 0.5, 0.75}
labels := []string{"GetObject", "PutObject", "CopyObject", "CompleteMultipartUpload"}
ttfbHist := prometheus.NewHistogramVec(
@@ -83,6 +85,135 @@ func TestGetHistogramMetrics(t *testing.T) {
metrics := getHistogramMetrics(ttfbHist, getBucketTTFBDistributionMD(), false)
// additional labels for +Inf for all histogram metrics
if expPoints := len(labels) * (len(histBuckets) + 1); expPoints != len(metrics) {
t.Fatalf("Expected %v data points but got %v", expPoints, len(metrics))
t.Fatalf("Expected %v data points when toLowerAPILabels=false but got %v", expPoints, len(metrics))
}
metrics = getHistogramMetrics(ttfbHist, getBucketTTFBDistributionMD(), true)
// additional labels for +Inf for all histogram metrics
if expPoints := len(labels) * (len(histBuckets) + 1); expPoints != len(metrics) {
t.Fatalf("Expected %v data points when toLowerAPILabels=true but got %v", expPoints, len(metrics))
}
}
func TestGetHistogramMetrics_Values(t *testing.T) {
histBuckets := []float64{0.50, 5.00}
labels := []string{"PutObject", "CopyObject"}
ttfbHist := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "s3_ttfb_seconds",
Help: "Time taken by requests served by current MinIO server instance",
Buckets: histBuckets,
},
[]string{"api"},
)
observations := []struct {
val float64
label string
}{
{
val: 0.02,
label: labels[0],
},
{
val: 0.19,
label: labels[1],
},
{
val: 0.31,
label: labels[1],
},
{
val: 0.61,
label: labels[0],
},
{
val: 6.79,
label: labels[1],
},
}
ticker := time.NewTicker(1 * time.Millisecond)
defer ticker.Stop()
for _, obs := range observations {
// Send observations once every 1ms, to simulate delay between
// observations. This is to test the channel based
// synchronization used internally.
select {
case <-ticker.C:
ttfbHist.With(prometheus.Labels{"api": obs.label}).Observe(obs.val)
}
}
// Accumulate regular-cased API label metrics for 'PutObject' for deeper verification
metrics := getHistogramMetrics(ttfbHist, getBucketTTFBDistributionMD(), false)
capitalPutObjects := make([]MetricV2, 0, len(histBuckets)+1)
for _, metric := range metrics {
if value := metric.VariableLabels["api"]; value == "PutObject" {
capitalPutObjects = append(capitalPutObjects, metric)
}
}
if expMetricsPerAPI := len(histBuckets) + 1; expMetricsPerAPI != len(capitalPutObjects) {
t.Fatalf("Expected %d api=PutObject metrics but got %d", expMetricsPerAPI, len(capitalPutObjects))
}
// Deterministic ordering
slices.SortFunc(capitalPutObjects, func(a MetricV2, b MetricV2) int {
le1 := a.VariableLabels["le"]
le2 := a.VariableLabels["le"]
return strings.Compare(le1, le2)
})
if le := capitalPutObjects[0].VariableLabels["le"]; le != "0.500" {
t.Errorf("Expected le='0.050' api=PutObject metrics but got '%v'", le)
}
if value := capitalPutObjects[0].Value; value != 1 {
t.Errorf("Expected le='0.050' api=PutObject value to be 1 but got '%v'", value)
}
if le := capitalPutObjects[1].VariableLabels["le"]; le != "5.000" {
t.Errorf("Expected le='5.000' api=PutObject metrics but got '%v'", le)
}
if value := capitalPutObjects[1].Value; value != 2 {
t.Errorf("Expected le='5.000' api=PutObject value to be 2 but got '%v'", value)
}
if le := capitalPutObjects[2].VariableLabels["le"]; le != "+Inf" {
t.Errorf("Expected le='+Inf' api=PutObject metrics but got '%v'", le)
}
if value := capitalPutObjects[2].Value; value != 2 {
t.Errorf("Expected le='+Inf' api=PutObject value to be 2 but got '%v'", value)
}
// Accumulate lower-cased API label metrics for 'copyobject' for deeper verification
metrics = getHistogramMetrics(ttfbHist, getBucketTTFBDistributionMD(), true)
lowerCopyObjects := make([]MetricV2, 0, len(histBuckets)+1)
for _, metric := range metrics {
if value := metric.VariableLabels["api"]; value == "copyobject" {
lowerCopyObjects = append(lowerCopyObjects, metric)
}
}
if expMetricsPerAPI := len(histBuckets) + 1; expMetricsPerAPI != len(lowerCopyObjects) {
t.Fatalf("Expected %d api=copyobject metrics but got %d", expMetricsPerAPI, len(lowerCopyObjects))
}
// Deterministic ordering
slices.SortFunc(lowerCopyObjects, func(a MetricV2, b MetricV2) int {
le1 := a.VariableLabels["le"]
le2 := a.VariableLabels["le"]
return strings.Compare(le1, le2)
})
if le := lowerCopyObjects[0].VariableLabels["le"]; le != "0.500" {
t.Errorf("Expected le='0.050' api=copyobject metrics but got '%v'", le)
}
if value := lowerCopyObjects[0].Value; value != 2 {
t.Errorf("Expected le='0.050' api=copyobject value to be 2 but got '%v'", value)
}
if le := lowerCopyObjects[1].VariableLabels["le"]; le != "5.000" {
t.Errorf("Expected le='5.000' api=copyobject metrics but got '%v'", le)
}
if value := lowerCopyObjects[1].Value; value != 2 {
t.Errorf("Expected le='5.000' api=copyobject value to be 2 but got '%v'", value)
}
if le := lowerCopyObjects[2].VariableLabels["le"]; le != "+Inf" {
t.Errorf("Expected le='+Inf' api=copyobject metrics but got '%v'", le)
}
if value := lowerCopyObjects[2].Value; value != 3 {
t.Errorf("Expected le='+Inf' api=copyobject value to be 3 but got '%v'", value)
}
}
+1 -1
View File
@@ -20,6 +20,7 @@ package cmd
import (
"context"
"fmt"
"slices"
"strings"
"sync"
@@ -27,7 +28,6 @@ import (
"github.com/minio/minio/internal/logger"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/exp/slices"
)
type collectorPath string
+28 -18
View File
@@ -419,6 +419,9 @@ type ListPartsInfo struct {
// ChecksumAlgorithm if set
ChecksumAlgorithm string
// ChecksumType if set
ChecksumType string
}
// Lookup - returns if uploadID is valid
@@ -597,10 +600,11 @@ type PartInfo struct {
ActualSize int64
// Checksum values
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC64NVME string
}
// CompletePart - represents the part that was completed, this is sent by the client
@@ -613,11 +617,14 @@ type CompletePart struct {
// Entity tag returned when the part was uploaded.
ETag string
Size int64
// Checksum values. Optional.
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC64NVME string
}
// CompleteMultipartUpload - represents list of parts which are completed, this is sent by the
@@ -630,6 +637,7 @@ type CompleteMultipartUpload struct {
type NewMultipartUploadResult struct {
UploadID string
ChecksumAlgo string
ChecksumType string
}
type getObjectAttributesResponse struct {
@@ -641,10 +649,11 @@ type getObjectAttributesResponse struct {
}
type objectAttributesChecksum struct {
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
}
type objectAttributesParts struct {
@@ -657,12 +666,13 @@ type objectAttributesParts struct {
}
type objectAttributesPart struct {
PartNumber int
Size int64
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
PartNumber int
Size int64
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
}
type objectAttributesErrorResponse struct {
+57 -12
View File
@@ -201,13 +201,16 @@ func (z *CompleteMultipartUpload) Msgsize() (s int) {
// MarshalMsg implements msgp.Marshaler
func (z *CompletePart) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 6
// map header, size 8
// string "PartNumber"
o = append(o, 0x86, 0xaa, 0x50, 0x61, 0x72, 0x74, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72)
o = append(o, 0x88, 0xaa, 0x50, 0x61, 0x72, 0x74, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72)
o = msgp.AppendInt(o, z.PartNumber)
// string "ETag"
o = append(o, 0xa4, 0x45, 0x54, 0x61, 0x67)
o = msgp.AppendString(o, z.ETag)
// string "Size"
o = append(o, 0xa4, 0x53, 0x69, 0x7a, 0x65)
o = msgp.AppendInt64(o, z.Size)
// string "ChecksumCRC32"
o = append(o, 0xad, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x43, 0x52, 0x43, 0x33, 0x32)
o = msgp.AppendString(o, z.ChecksumCRC32)
@@ -220,6 +223,9 @@ func (z *CompletePart) MarshalMsg(b []byte) (o []byte, err error) {
// string "ChecksumSHA256"
o = append(o, 0xae, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x53, 0x48, 0x41, 0x32, 0x35, 0x36)
o = msgp.AppendString(o, z.ChecksumSHA256)
// string "ChecksumCRC64NVME"
o = append(o, 0xb1, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x43, 0x52, 0x43, 0x36, 0x34, 0x4e, 0x56, 0x4d, 0x45)
o = msgp.AppendString(o, z.ChecksumCRC64NVME)
return
}
@@ -253,6 +259,12 @@ func (z *CompletePart) UnmarshalMsg(bts []byte) (o []byte, err error) {
err = msgp.WrapError(err, "ETag")
return
}
case "Size":
z.Size, bts, err = msgp.ReadInt64Bytes(bts)
if err != nil {
err = msgp.WrapError(err, "Size")
return
}
case "ChecksumCRC32":
z.ChecksumCRC32, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
@@ -277,6 +289,12 @@ func (z *CompletePart) UnmarshalMsg(bts []byte) (o []byte, err error) {
err = msgp.WrapError(err, "ChecksumSHA256")
return
}
case "ChecksumCRC64NVME":
z.ChecksumCRC64NVME, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "ChecksumCRC64NVME")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
@@ -291,7 +309,7 @@ func (z *CompletePart) UnmarshalMsg(bts []byte) (o []byte, err error) {
// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *CompletePart) Msgsize() (s int) {
s = 1 + 11 + msgp.IntSize + 5 + msgp.StringPrefixSize + len(z.ETag) + 14 + msgp.StringPrefixSize + len(z.ChecksumCRC32) + 15 + msgp.StringPrefixSize + len(z.ChecksumCRC32C) + 13 + msgp.StringPrefixSize + len(z.ChecksumSHA1) + 15 + msgp.StringPrefixSize + len(z.ChecksumSHA256)
s = 1 + 11 + msgp.IntSize + 5 + msgp.StringPrefixSize + len(z.ETag) + 5 + msgp.Int64Size + 14 + msgp.StringPrefixSize + len(z.ChecksumCRC32) + 15 + msgp.StringPrefixSize + len(z.ChecksumCRC32C) + 13 + msgp.StringPrefixSize + len(z.ChecksumSHA1) + 15 + msgp.StringPrefixSize + len(z.ChecksumSHA256) + 18 + msgp.StringPrefixSize + len(z.ChecksumCRC64NVME)
return
}
@@ -956,9 +974,9 @@ func (z *ListObjectsV2Info) Msgsize() (s int) {
// MarshalMsg implements msgp.Marshaler
func (z *ListPartsInfo) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 11
// map header, size 12
// string "Bucket"
o = append(o, 0x8b, 0xa6, 0x42, 0x75, 0x63, 0x6b, 0x65, 0x74)
o = append(o, 0x8c, 0xa6, 0x42, 0x75, 0x63, 0x6b, 0x65, 0x74)
o = msgp.AppendString(o, z.Bucket)
// string "Object"
o = append(o, 0xa6, 0x4f, 0x62, 0x6a, 0x65, 0x63, 0x74)
@@ -1001,6 +1019,9 @@ func (z *ListPartsInfo) MarshalMsg(b []byte) (o []byte, err error) {
// string "ChecksumAlgorithm"
o = append(o, 0xb1, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d)
o = msgp.AppendString(o, z.ChecksumAlgorithm)
// string "ChecksumType"
o = append(o, 0xac, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x54, 0x79, 0x70, 0x65)
o = msgp.AppendString(o, z.ChecksumType)
return
}
@@ -1125,6 +1146,12 @@ func (z *ListPartsInfo) UnmarshalMsg(bts []byte) (o []byte, err error) {
err = msgp.WrapError(err, "ChecksumAlgorithm")
return
}
case "ChecksumType":
z.ChecksumType, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "ChecksumType")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
@@ -1150,7 +1177,7 @@ func (z *ListPartsInfo) Msgsize() (s int) {
s += msgp.StringPrefixSize + len(za0002) + msgp.StringPrefixSize + len(za0003)
}
}
s += 18 + msgp.StringPrefixSize + len(z.ChecksumAlgorithm)
s += 18 + msgp.StringPrefixSize + len(z.ChecksumAlgorithm) + 13 + msgp.StringPrefixSize + len(z.ChecksumType)
return
}
@@ -1279,13 +1306,16 @@ func (z *MultipartInfo) Msgsize() (s int) {
// MarshalMsg implements msgp.Marshaler
func (z NewMultipartUploadResult) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 2
// map header, size 3
// string "UploadID"
o = append(o, 0x82, 0xa8, 0x55, 0x70, 0x6c, 0x6f, 0x61, 0x64, 0x49, 0x44)
o = append(o, 0x83, 0xa8, 0x55, 0x70, 0x6c, 0x6f, 0x61, 0x64, 0x49, 0x44)
o = msgp.AppendString(o, z.UploadID)
// string "ChecksumAlgo"
o = append(o, 0xac, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x41, 0x6c, 0x67, 0x6f)
o = msgp.AppendString(o, z.ChecksumAlgo)
// string "ChecksumType"
o = append(o, 0xac, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x54, 0x79, 0x70, 0x65)
o = msgp.AppendString(o, z.ChecksumType)
return
}
@@ -1319,6 +1349,12 @@ func (z *NewMultipartUploadResult) UnmarshalMsg(bts []byte) (o []byte, err error
err = msgp.WrapError(err, "ChecksumAlgo")
return
}
case "ChecksumType":
z.ChecksumType, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "ChecksumType")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
@@ -1333,7 +1369,7 @@ func (z *NewMultipartUploadResult) UnmarshalMsg(bts []byte) (o []byte, err error
// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z NewMultipartUploadResult) Msgsize() (s int) {
s = 1 + 9 + msgp.StringPrefixSize + len(z.UploadID) + 13 + msgp.StringPrefixSize + len(z.ChecksumAlgo)
s = 1 + 9 + msgp.StringPrefixSize + len(z.UploadID) + 13 + msgp.StringPrefixSize + len(z.ChecksumAlgo) + 13 + msgp.StringPrefixSize + len(z.ChecksumType)
return
}
@@ -1772,9 +1808,9 @@ func (z *ObjectInfo) Msgsize() (s int) {
// MarshalMsg implements msgp.Marshaler
func (z *PartInfo) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 9
// map header, size 10
// string "PartNumber"
o = append(o, 0x89, 0xaa, 0x50, 0x61, 0x72, 0x74, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72)
o = append(o, 0x8a, 0xaa, 0x50, 0x61, 0x72, 0x74, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72)
o = msgp.AppendInt(o, z.PartNumber)
// string "LastModified"
o = append(o, 0xac, 0x4c, 0x61, 0x73, 0x74, 0x4d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64)
@@ -1800,6 +1836,9 @@ func (z *PartInfo) MarshalMsg(b []byte) (o []byte, err error) {
// string "ChecksumSHA256"
o = append(o, 0xae, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x53, 0x48, 0x41, 0x32, 0x35, 0x36)
o = msgp.AppendString(o, z.ChecksumSHA256)
// string "ChecksumCRC64NVME"
o = append(o, 0xb1, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x43, 0x52, 0x43, 0x36, 0x34, 0x4e, 0x56, 0x4d, 0x45)
o = msgp.AppendString(o, z.ChecksumCRC64NVME)
return
}
@@ -1875,6 +1914,12 @@ func (z *PartInfo) UnmarshalMsg(bts []byte) (o []byte, err error) {
err = msgp.WrapError(err, "ChecksumSHA256")
return
}
case "ChecksumCRC64NVME":
z.ChecksumCRC64NVME, bts, err = msgp.ReadStringBytes(bts)
if err != nil {
err = msgp.WrapError(err, "ChecksumCRC64NVME")
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
@@ -1889,7 +1934,7 @@ func (z *PartInfo) UnmarshalMsg(bts []byte) (o []byte, err error) {
// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *PartInfo) Msgsize() (s int) {
s = 1 + 11 + msgp.IntSize + 13 + msgp.TimeSize + 5 + msgp.StringPrefixSize + len(z.ETag) + 5 + msgp.Int64Size + 11 + msgp.Int64Size + 14 + msgp.StringPrefixSize + len(z.ChecksumCRC32) + 15 + msgp.StringPrefixSize + len(z.ChecksumCRC32C) + 13 + msgp.StringPrefixSize + len(z.ChecksumSHA1) + 15 + msgp.StringPrefixSize + len(z.ChecksumSHA256)
s = 1 + 11 + msgp.IntSize + 13 + msgp.TimeSize + 5 + msgp.StringPrefixSize + len(z.ETag) + 5 + msgp.Int64Size + 11 + msgp.Int64Size + 14 + msgp.StringPrefixSize + len(z.ChecksumCRC32) + 15 + msgp.StringPrefixSize + len(z.ChecksumCRC32C) + 13 + msgp.StringPrefixSize + len(z.ChecksumSHA1) + 15 + msgp.StringPrefixSize + len(z.ChecksumSHA256) + 18 + msgp.StringPrefixSize + len(z.ChecksumCRC64NVME)
return
}
-7
View File
@@ -404,12 +404,7 @@ func putOptsFromHeaders(ctx context.Context, hdr http.Header, metadata map[strin
metadata = make(map[string]string)
}
wantCRC, err := hash.GetContentChecksum(hdr)
if err != nil {
return opts, fmt.Errorf("invalid/unknown checksum sent: %v", err)
}
etag := strings.TrimSpace(hdr.Get(xhttp.MinIOSourceETag))
if crypto.S3KMS.IsRequested(hdr) {
keyID, context, err := crypto.S3KMS.ParseHTTP(hdr)
if err != nil {
@@ -423,7 +418,6 @@ func putOptsFromHeaders(ctx context.Context, hdr http.Header, metadata map[strin
ServerSideEncryption: sseKms,
UserDefined: metadata,
MTime: mtime,
WantChecksum: wantCRC,
PreserveETag: etag,
}, nil
}
@@ -438,7 +432,6 @@ func putOptsFromHeaders(ctx context.Context, hdr http.Header, metadata map[strin
opts.ReplicationSourceRetentionTimestamp = retaintimestmp
opts.ReplicationSourceTaggingTimestamp = taggingtimestmp
opts.PreserveETag = etag
opts.WantChecksum = wantCRC
return opts, nil
}
+1 -1
View File
@@ -29,6 +29,7 @@ import (
"net/http"
"path"
"runtime"
"slices"
"strconv"
"strings"
"sync"
@@ -52,7 +53,6 @@ import (
"github.com/minio/pkg/v3/trie"
"github.com/minio/pkg/v3/wildcard"
"github.com/valyala/bytebufferpool"
"golang.org/x/exp/slices"
)
const (
+4 -4
View File
@@ -352,11 +352,11 @@ func isETagEqual(left, right string) bool {
// setPutObjHeaders sets all the necessary headers returned back
// upon a success Put/Copy/CompleteMultipart/Delete requests
// to activate delete only headers set delete as true
func setPutObjHeaders(w http.ResponseWriter, objInfo ObjectInfo, delete bool, h http.Header) {
func setPutObjHeaders(w http.ResponseWriter, objInfo ObjectInfo, del bool, h http.Header) {
// We must not use the http.Header().Set method here because some (broken)
// clients expect the ETag header key to be literally "ETag" - not "Etag" (case-sensitive).
// Therefore, we have to set the ETag directly as map entry.
if objInfo.ETag != "" && !delete {
if objInfo.ETag != "" && !del {
w.Header()[xhttp.ETag] = []string{`"` + objInfo.ETag + `"`}
}
@@ -364,13 +364,13 @@ func setPutObjHeaders(w http.ResponseWriter, objInfo ObjectInfo, delete bool, h
if objInfo.VersionID != "" && objInfo.VersionID != nullVersionID {
w.Header()[xhttp.AmzVersionID] = []string{objInfo.VersionID}
// If version is a deleted marker, set this header as well
if objInfo.DeleteMarker && delete { // only returned during delete object
if objInfo.DeleteMarker && del { // only returned during delete object
w.Header()[xhttp.AmzDeleteMarker] = []string{strconv.FormatBool(objInfo.DeleteMarker)}
}
}
if objInfo.Bucket != "" && objInfo.Name != "" {
if lc, err := globalLifecycleSys.Get(objInfo.Bucket); err == nil && !delete {
if lc, err := globalLifecycleSys.Get(objInfo.Bucket); err == nil && !del {
lc.SetPredictionHeaders(w, objInfo.ToLifecycleOpts())
}
}
+35 -24
View File
@@ -636,10 +636,11 @@ func (api objectAPIHandlers) getObjectAttributesHandler(ctx context.Context, obj
// AWS does not appear to append part number on this API call.
if len(chkSums) > 0 {
OA.Checksum = &objectAttributesChecksum{
ChecksumCRC32: strings.Split(chkSums["CRC32"], "-")[0],
ChecksumCRC32C: strings.Split(chkSums["CRC32C"], "-")[0],
ChecksumSHA1: strings.Split(chkSums["SHA1"], "-")[0],
ChecksumSHA256: strings.Split(chkSums["SHA256"], "-")[0],
ChecksumCRC32: strings.Split(chkSums["CRC32"], "-")[0],
ChecksumCRC32C: strings.Split(chkSums["CRC32C"], "-")[0],
ChecksumSHA1: strings.Split(chkSums["SHA1"], "-")[0],
ChecksumSHA256: strings.Split(chkSums["SHA256"], "-")[0],
ChecksumCRC64NVME: strings.Split(chkSums["CRC64NVME"], "-")[0],
}
}
}
@@ -678,12 +679,13 @@ func (api objectAPIHandlers) getObjectAttributesHandler(ctx context.Context, obj
OA.ObjectParts.NextPartNumberMarker = v.Number
OA.ObjectParts.Parts = append(OA.ObjectParts.Parts, &objectAttributesPart{
ChecksumSHA1: objInfo.Parts[i].Checksums["SHA1"],
ChecksumSHA256: objInfo.Parts[i].Checksums["SHA256"],
ChecksumCRC32: objInfo.Parts[i].Checksums["CRC32"],
ChecksumCRC32C: objInfo.Parts[i].Checksums["CRC32C"],
PartNumber: objInfo.Parts[i].Number,
Size: objInfo.Parts[i].Size,
ChecksumSHA1: objInfo.Parts[i].Checksums["SHA1"],
ChecksumSHA256: objInfo.Parts[i].Checksums["SHA256"],
ChecksumCRC32: objInfo.Parts[i].Checksums["CRC32"],
ChecksumCRC32C: objInfo.Parts[i].Checksums["CRC32C"],
ChecksumCRC64NVME: objInfo.Parts[i].Checksums["CRC64NVME"],
PartNumber: objInfo.Parts[i].Number,
Size: objInfo.Parts[i].Size,
})
}
}
@@ -1191,6 +1193,9 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
return
}
// Sanitize the source object name similar to NewMultipart and PutObject API
srcObject = trimLeadingSlash(srcObject)
if vid != "" && vid != nullVersionID {
_, err := uuid.Parse(vid)
if err != nil {
@@ -1899,6 +1904,13 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
var reader io.Reader
reader = rd
var opts ObjectOptions
opts, err = putOptsFromReq(ctx, r, bucket, object, metadata)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
actualSize := size
var idxCb func() []byte
if isCompressible(r.Header, object) && size > minCompressibleSize {
@@ -1915,6 +1927,8 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidChecksum), r.URL)
return
}
opts.WantChecksum = actualReader.Checksum()
// Set compression metrics.
var s2c io.ReadCloser
wantEncryption := crypto.Requested(r.Header)
@@ -1945,22 +1959,17 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if err := hashReader.AddChecksum(r, size < 0); err != nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidChecksum), r.URL)
return
if size >= 0 {
if err := hashReader.AddChecksum(r, false); err != nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidChecksum), r.URL)
return
}
opts.WantChecksum = hashReader.Checksum()
}
rawReader := hashReader
pReader := NewPutObjReader(rawReader)
var opts ObjectOptions
opts, err = putOptsFromReq(ctx, r, bucket, object, metadata)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
opts.IndexCB = idxCb
opts.WantChecksum = hashReader.Checksum()
if opts.PreserveETag != "" ||
r.Header.Get(xhttp.IfMatch) != "" ||
@@ -2724,7 +2733,7 @@ func (api objectAPIHandlers) PutObjectLegalHoldHandler(w http.ResponseWriter, r
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
return
}
if !hasContentMD5(r.Header) {
if !validateLengthAndChecksum(r) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL)
return
}
@@ -2734,7 +2743,7 @@ func (api objectAPIHandlers) PutObjectLegalHoldHandler(w http.ResponseWriter, r
return
}
legalHold, err := objectlock.ParseObjectLegalHold(io.LimitReader(r.Body, r.ContentLength))
legalHold, err := objectlock.ParseObjectLegalHold(r.Body)
if err != nil {
apiErr := errorCodes.ToAPIErr(ErrMalformedXML)
apiErr.Description = err.Error()
@@ -2882,7 +2891,7 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
return
}
if !hasContentMD5(r.Header) {
if !validateLengthAndChecksum(r) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL)
return
}
@@ -2899,6 +2908,8 @@ func (api objectAPIHandlers) PutObjectRetentionHandler(w http.ResponseWriter, r
writeErrorResponse(ctx, w, apiErr, r.URL)
return
}
reqInfo := logger.GetReqInfo(ctx)
reqInfo.SetTags("retention", objRetention.String())
opts, err := getOpts(ctx, r, bucket, object)
if err != nil {
+2 -2
View File
@@ -93,8 +93,8 @@ func getLambdaEventData(bucket, object string, cred auth.Credentials, r *http.Re
},
UserIdentity: levent.Identity{
Type: "IAMUser",
PrincipalID: cred.AccessKey,
AccessKeyID: cred.SecretKey,
PrincipalID: cred.ParentUser,
AccessKeyID: cred.AccessKey,
},
}
return eventData, nil
+8 -1
View File
@@ -214,7 +214,7 @@ func (api objectAPIHandlers) NewMultipartUploadHandler(w http.ResponseWriter, r
}
}
checksumType := hash.NewChecksumType(r.Header.Get(xhttp.AmzChecksumAlgo))
checksumType := hash.NewChecksumHeader(r.Header)
if checksumType.Is(hash.ChecksumInvalid) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidRequestParameter), r.URL)
return
@@ -233,6 +233,9 @@ func (api objectAPIHandlers) NewMultipartUploadHandler(w http.ResponseWriter, r
response := generateInitiateMultipartUploadResponse(bucket, object, res.UploadID)
if res.ChecksumAlgo != "" {
w.Header().Set(xhttp.AmzChecksumAlgo, res.ChecksumAlgo)
if res.ChecksumType != "" {
w.Header().Set(xhttp.AmzChecksumType, res.ChecksumType)
}
}
encodedSuccessResponse := encodeResponse(response)
@@ -1204,6 +1207,10 @@ func (api objectAPIHandlers) ListObjectPartsHandler(w http.ResponseWriter, r *ht
listPartsInfo.Parts[i].ETag = tryDecryptETag(objectEncryptionKey, p.ETag, kind == crypto.S3)
listPartsInfo.Parts[i].Size = p.ActualSize
}
} else if _, ok := listPartsInfo.UserDefined[ReservedMetadataPrefix+"compression"]; ok {
for i, p := range listPartsInfo.Parts {
listPartsInfo.Parts[i].Size = p.ActualSize
}
}
response := generateListPartsResponse(listPartsInfo, encodingType)
+17 -19
View File
@@ -60,20 +60,19 @@ func readDirFn(dirPath string, filter func(name string, typ os.FileMode) error)
if err != nil {
if err == syscall.ERROR_NO_MORE_FILES {
break
} else {
if isSysErrPathNotFound(err) {
return nil
}
err = osErrToFileErr(&os.PathError{
Op: "FindNextFile",
Path: dirPath,
Err: err,
})
if err == errFileNotFound {
return nil
}
return err
}
if isSysErrPathNotFound(err) {
return nil
}
err = osErrToFileErr(&os.PathError{
Op: "FindNextFile",
Path: dirPath,
Err: err,
})
if err == errFileNotFound {
return nil
}
return err
}
name := syscall.UTF16ToString(data.FileName[0:])
if name == "" || name == "." || name == ".." { // Useless names
@@ -136,13 +135,12 @@ func readDirWithOpts(dirPath string, opts readDirOpts) (entries []string, err er
if err != nil {
if err == syscall.ERROR_NO_MORE_FILES {
break
} else {
return nil, osErrToFileErr(&os.PathError{
Op: "FindNextFile",
Path: dirPath,
Err: err,
})
}
return nil, osErrToFileErr(&os.PathError{
Op: "FindNextFile",
Path: dirPath,
Err: err,
})
}
name := syscall.UTF16ToString(data.FileName[0:])
+5 -4
View File
@@ -56,6 +56,7 @@ var (
aoBucketInfo = grid.NewArrayOf[*BucketInfo](func() *BucketInfo { return &BucketInfo{} })
aoMetricsGroup = grid.NewArrayOf[*MetricV2](func() *MetricV2 { return &MetricV2{} })
madminBgHealState = grid.NewJSONPool[madmin.BgHealState]()
madminHealResultItem = grid.NewJSONPool[madmin.HealResultItem]()
madminCPUs = grid.NewJSONPool[madmin.CPUs]()
madminMemInfo = grid.NewJSONPool[madmin.MemInfo]()
madminNetInfo = grid.NewJSONPool[madmin.NetInfo]()
@@ -97,7 +98,7 @@ var (
getSysErrorsRPC = grid.NewSingleHandler[*grid.MSS, *grid.JSON[madmin.SysErrors]](grid.HandlerGetSysErrors, grid.NewMSS, madminSysErrors.NewJSON)
getSysServicesRPC = grid.NewSingleHandler[*grid.MSS, *grid.JSON[madmin.SysServices]](grid.HandlerGetSysServices, grid.NewMSS, madminSysServices.NewJSON)
headBucketRPC = grid.NewSingleHandler[*grid.MSS, *VolInfo](grid.HandlerHeadBucket, grid.NewMSS, func() *VolInfo { return &VolInfo{} })
healBucketRPC = grid.NewSingleHandler[*grid.MSS, grid.NoPayload](grid.HandlerHealBucket, grid.NewMSS, grid.NewNoPayload)
healBucketRPC = grid.NewSingleHandler[*grid.MSS, *grid.JSON[madmin.HealResultItem]](grid.HandlerHealBucket, grid.NewMSS, madminHealResultItem.NewJSON)
listBucketsRPC = grid.NewSingleHandler[*BucketOptions, *grid.Array[*BucketInfo]](grid.HandlerListBuckets, func() *BucketOptions { return &BucketOptions{} }, aoBucketInfo.New)
loadBucketMetadataRPC = grid.NewSingleHandler[*grid.MSS, grid.NoPayload](grid.HandlerLoadBucketMetadata, grid.NewMSS, grid.NewNoPayload).IgnoreNilConn()
loadGroupRPC = grid.NewSingleHandler[*grid.MSS, grid.NoPayload](grid.HandlerLoadGroup, grid.NewMSS, grid.NewNoPayload)
@@ -1258,21 +1259,21 @@ func (s *peerRESTServer) NetSpeedTestHandler(w http.ResponseWriter, r *http.Requ
peersLogIf(r.Context(), gob.NewEncoder(w).Encode(result))
}
func (s *peerRESTServer) HealBucketHandler(mss *grid.MSS) (np grid.NoPayload, nerr *grid.RemoteErr) {
func (s *peerRESTServer) HealBucketHandler(mss *grid.MSS) (np *grid.JSON[madmin.HealResultItem], nerr *grid.RemoteErr) {
bucket := mss.Get(peerS3Bucket)
if isMinioMetaBucket(bucket) {
return np, grid.NewRemoteErr(errInvalidArgument)
}
bucketDeleted := mss.Get(peerS3BucketDeleted) == "true"
_, err := healBucketLocal(context.Background(), bucket, madmin.HealOpts{
res, err := healBucketLocal(context.Background(), bucket, madmin.HealOpts{
Remove: bucketDeleted,
})
if err != nil {
return np, grid.NewRemoteErr(err)
}
return np, nil
return madminHealResultItem.NewJSONWith(&res), nil
}
func (s *peerRESTServer) ListBucketsHandler(opts *BucketOptions) (*grid.Array[*BucketInfo], *grid.RemoteErr) {
+16 -11
View File
@@ -21,6 +21,7 @@ import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strconv"
"sync/atomic"
@@ -29,7 +30,6 @@ import (
"github.com/minio/madmin-go/v3"
"github.com/minio/minio/internal/grid"
"github.com/minio/pkg/v3/sync/errgroup"
"golang.org/x/exp/slices"
)
var errPeerOffline = errors.New("peer is offline")
@@ -178,13 +178,24 @@ func (sys *S3PeerSys) HealBucket(ctx context.Context, bucket string, opts madmin
}
}
if healBucketErr := reduceWriteQuorumErrs(ctx, errs, bucketOpIgnoredErrs, len(errs)/2+1); healBucketErr != nil {
return madmin.HealResultItem{}, toObjectErr(healBucketErr, bucket)
}
res := madmin.HealResultItem{
Type: madmin.HealItemBucket,
Bucket: bucket,
SetCount: -1, // explicitly set an invalid value -1, for bucket heal scenario
}
for i, err := range errs {
if err == nil {
return healBucketResults[i], nil
res.Before.Drives = append(res.Before.Drives, healBucketResults[i].Before.Drives...)
res.After.Drives = append(res.After.Drives, healBucketResults[i].After.Drives...)
}
}
return madmin.HealResultItem{}, toObjectErr(errVolumeNotFound, bucket)
return res, nil
}
// ListBuckets lists buckets across all nodes and returns a consistent view:
@@ -355,14 +366,8 @@ func (client *remotePeerS3Client) HealBucket(ctx context.Context, bucket string,
ctx, cancel := context.WithTimeout(ctx, globalDriveConfig.GetMaxTimeout())
defer cancel()
_, err := healBucketRPC.Call(ctx, conn, mss)
// Initialize heal result info
return madmin.HealResultItem{
Type: madmin.HealItemBucket,
Bucket: bucket,
SetCount: -1, // explicitly set an invalid value -1, for bucket heal scenario
}, toStorageErr(err)
resp, err := healBucketRPC.Call(ctx, conn, mss)
return resp.ValueOrZero(), toStorageErr(err)
}
// GetBucketInfo returns bucket stat info from a peer
+15 -12
View File
@@ -23,6 +23,7 @@ import (
"github.com/minio/madmin-go/v3"
"github.com/minio/pkg/v3/sync/errgroup"
"github.com/puzpuzpuz/xsync/v3"
)
const (
@@ -101,7 +102,7 @@ func healBucketLocal(ctx context.Context, bucket string, opts madmin.HealOpts) (
for i := range beforeState {
res.Before.Drives = append(res.Before.Drives, madmin.HealDriveInfo{
UUID: "",
Endpoint: localDrives[i].String(),
Endpoint: localDrives[i].Endpoint().String(),
State: beforeState[i],
})
}
@@ -149,7 +150,7 @@ func healBucketLocal(ctx context.Context, bucket string, opts madmin.HealOpts) (
for i := range afterState {
res.After.Drives = append(res.After.Drives, madmin.HealDriveInfo{
UUID: "",
Endpoint: localDrives[i].String(),
Endpoint: localDrives[i].Endpoint().String(),
State: afterState[i],
})
}
@@ -164,7 +165,7 @@ func listBucketsLocal(ctx context.Context, opts BucketOptions) (buckets []Bucket
quorum := (len(localDrives) / 2)
buckets = make([]BucketInfo, 0, 32)
healBuckets := map[string]VolInfo{}
healBuckets := xsync.NewMapOf[string, VolInfo]()
// lists all unique buckets across drives.
if err := listAllBuckets(ctx, localDrives, healBuckets, quorum); err != nil {
@@ -172,7 +173,7 @@ func listBucketsLocal(ctx context.Context, opts BucketOptions) (buckets []Bucket
}
// include deleted buckets in listBuckets output
deletedBuckets := map[string]VolInfo{}
deletedBuckets := xsync.NewMapOf[string, VolInfo]()
if opts.Deleted {
// lists all deleted buckets across drives.
@@ -181,25 +182,27 @@ func listBucketsLocal(ctx context.Context, opts BucketOptions) (buckets []Bucket
}
}
for _, v := range healBuckets {
healBuckets.Range(func(_ string, volInfo VolInfo) bool {
bi := BucketInfo{
Name: v.Name,
Created: v.Created,
Name: volInfo.Name,
Created: volInfo.Created,
}
if vi, ok := deletedBuckets[v.Name]; ok {
if vi, ok := deletedBuckets.Load(volInfo.Name); ok {
bi.Deleted = vi.Created
}
buckets = append(buckets, bi)
}
return true
})
for _, v := range deletedBuckets {
if _, ok := healBuckets[v.Name]; !ok {
deletedBuckets.Range(func(_ string, v VolInfo) bool {
if _, ok := healBuckets.Load(v.Name); !ok {
buckets = append(buckets, BucketInfo{
Name: v.Name,
Deleted: v.Created,
})
}
}
return true
})
return buckets, nil
}
+12 -8
View File
@@ -56,10 +56,12 @@ func newPostPolicyBytesV4WithContentRange(credential, bucketName, objectKey stri
credentialConditionStr := fmt.Sprintf(`["eq", "$x-amz-credential", "%s"]`, credential)
// Add the meta-uuid string, set to 1234
uuidConditionStr := fmt.Sprintf(`["eq", "$x-amz-meta-uuid", "%s"]`, "1234")
// Add the content-encoding string, set to gzip.
contentEncodingConditionStr := fmt.Sprintf(`["eq", "$content-encoding", "%s"]`, "gzip")
// Combine all conditions into one string.
conditionStr := fmt.Sprintf(`"conditions":[%s, %s, %s, %s, %s, %s, %s]`, bucketConditionStr,
keyConditionStr, contentLengthCondStr, algorithmConditionStr, dateConditionStr, credentialConditionStr, uuidConditionStr)
conditionStr := fmt.Sprintf(`"conditions":[%s, %s, %s, %s, %s, %s, %s, %s]`, bucketConditionStr,
keyConditionStr, contentLengthCondStr, algorithmConditionStr, dateConditionStr, credentialConditionStr, uuidConditionStr, contentEncodingConditionStr)
retStr := "{"
retStr = retStr + expirationStr + ","
retStr += conditionStr
@@ -85,9 +87,11 @@ func newPostPolicyBytesV4(credential, bucketName, objectKey string, expiration t
credentialConditionStr := fmt.Sprintf(`["eq", "$x-amz-credential", "%s"]`, credential)
// Add the meta-uuid string, set to 1234
uuidConditionStr := fmt.Sprintf(`["eq", "$x-amz-meta-uuid", "%s"]`, "1234")
// Add the content-encoding string, set to gzip
contentEncodingConditionStr := fmt.Sprintf(`["eq", "$content-encoding", "%s"]`, "gzip")
// Combine all conditions into one string.
conditionStr := fmt.Sprintf(`"conditions":[%s, %s, %s, %s, %s, %s]`, bucketConditionStr, keyConditionStr, algorithmConditionStr, dateConditionStr, credentialConditionStr, uuidConditionStr)
conditionStr := fmt.Sprintf(`"conditions":[%s, %s, %s, %s, %s, %s, %s]`, bucketConditionStr, keyConditionStr, algorithmConditionStr, dateConditionStr, credentialConditionStr, uuidConditionStr, contentEncodingConditionStr)
retStr := "{"
retStr = retStr + expirationStr + ","
retStr += conditionStr
@@ -331,7 +335,7 @@ func testPostPolicyBucketHandler(obj ObjectLayer, instanceType string, t TestErr
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
dates: []interface{}{curTimePlus5Min.Format(iso8601TimeFormat), curTime.Format(iso8601DateFormat), curTime.Format(yyyymmdd)},
policy: `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], ["starts-with", "$key", "test/"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$x-amz-meta-uuid", "1234"]]}`,
policy: `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], ["starts-with", "$key", "test/"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$x-amz-meta-uuid", "1234"],["eq", "$content-encoding", "gzip"]]}`,
},
// Success case, no multipart filename.
{
@@ -341,7 +345,7 @@ func testPostPolicyBucketHandler(obj ObjectLayer, instanceType string, t TestErr
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
dates: []interface{}{curTimePlus5Min.Format(iso8601TimeFormat), curTime.Format(iso8601DateFormat), curTime.Format(yyyymmdd)},
policy: `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], ["starts-with", "$key", "test/"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$x-amz-meta-uuid", "1234"]]}`,
policy: `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], ["starts-with", "$key", "test/"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$x-amz-meta-uuid", "1234"],["eq", "$content-encoding", "gzip"]]}`,
noFilename: true,
},
// Success case, big body.
@@ -352,7 +356,7 @@ func testPostPolicyBucketHandler(obj ObjectLayer, instanceType string, t TestErr
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
dates: []interface{}{curTimePlus5Min.Format(iso8601TimeFormat), curTime.Format(iso8601DateFormat), curTime.Format(yyyymmdd)},
policy: `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], ["starts-with", "$key", "test/"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$x-amz-meta-uuid", "1234"]]}`,
policy: `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], ["starts-with", "$key", "test/"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$x-amz-meta-uuid", "1234"],["eq", "$content-encoding", "gzip"]]}`,
},
// Corrupted Base 64 result
{
@@ -447,7 +451,7 @@ func testPostPolicyBucketHandler(obj ObjectLayer, instanceType string, t TestErr
malformedBody: false,
ignoreContentLength: false,
},
// Failed with Content-Length not specified.
// Success with Content-Length not specified.
{
objectName: "test",
data: bytes.Repeat([]byte("a"), 1025),
@@ -547,7 +551,7 @@ func testPostPolicyBucketHandlerRedirect(obj ObjectLayer, instanceType string, t
rec := httptest.NewRecorder()
dates := []interface{}{curTimePlus5Min.Format(iso8601TimeFormat), curTime.Format(iso8601DateFormat), curTime.Format(yyyymmdd)}
policy := `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], {"success_action_redirect":"` + redirectURL.String() + `"},["starts-with", "$key", "test/"], ["eq", "$x-amz-meta-uuid", "1234"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"]]}`
policy := `{"expiration": "%s","conditions":[["eq", "$bucket", "` + bucketName + `"], {"success_action_redirect":"` + redirectURL.String() + `"},["starts-with", "$key", "test/"], ["eq", "$x-amz-meta-uuid", "1234"], ["eq", "$x-amz-algorithm", "AWS4-HMAC-SHA256"], ["eq", "$x-amz-date", "%s"], ["eq", "$x-amz-credential", "` + credentials.AccessKey + `/%s/us-east-1/s3/aws4_request"],["eq", "$content-encoding", "gzip"]]}`
// Generate the final policy document
policy = fmt.Sprintf(policy, dates...)
+51 -66
View File
@@ -53,18 +53,6 @@ var startsWithConds = map[string]bool{
"$x-amz-date": false,
}
var postPolicyIgnoreKeys = map[string]bool{
"Policy": true,
xhttp.AmzSignature: true,
xhttp.ContentEncoding: true,
http.CanonicalHeaderKey(xhttp.AmzChecksumAlgo): true,
http.CanonicalHeaderKey(xhttp.AmzChecksumCRC32): true,
http.CanonicalHeaderKey(xhttp.AmzChecksumCRC32C): true,
http.CanonicalHeaderKey(xhttp.AmzChecksumSHA1): true,
http.CanonicalHeaderKey(xhttp.AmzChecksumSHA256): true,
http.CanonicalHeaderKey(xhttp.AmzChecksumMode): true,
}
// Add policy conditionals.
const (
policyCondEqual = "eq"
@@ -232,19 +220,19 @@ func parsePostPolicyForm(r io.Reader) (PostPolicyForm, error) {
operator, matchType, value,
})
case policyCondContentLength:
min, err := toInteger(condt[1])
minLen, err := toInteger(condt[1])
if err != nil {
return parsedPolicy, err
}
max, err := toInteger(condt[2])
maxLen, err := toInteger(condt[2])
if err != nil {
return parsedPolicy, err
}
parsedPolicy.Conditions.ContentLengthRange = contentLengthRange{
Min: min,
Max: max,
Min: minLen,
Max: maxLen,
Valid: true,
}
default:
@@ -272,60 +260,57 @@ func checkPolicyCond(op string, input1, input2 string) bool {
return false
}
// S3 docs: "Each form field that you specify in a form (except x-amz-signature, file, policy, and field names
// that have an x-ignore- prefix) must appear in the list of conditions."
// https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html
// keyInPolicyExceptions - list of keys that, when present in the form, can be missing in the conditions of the policy.
var keyInPolicyExceptions = map[string]bool{
xhttp.AmzSignature: true,
"File": true,
"Policy": true,
// MinIO specific exceptions to the general S3 rule above.
encrypt.SseKmsKeyID: true,
encrypt.SseEncryptionContext: true,
encrypt.SseCustomerAlgorithm: true,
encrypt.SseCustomerKey: true,
encrypt.SseCustomerKeyMD5: true,
}
// checkPostPolicy - apply policy conditions and validate input values.
// (http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html)
// Note that content-length-range is checked in the API handler function PostPolicyBucketHandler.
// formValues is the already-canonicalized form values from the POST request.
func checkPostPolicy(formValues http.Header, postPolicyForm PostPolicyForm) error {
// Check if policy document expiry date is still not reached
if !postPolicyForm.Expiration.After(UTCNow()) {
return fmt.Errorf("Invalid according to Policy: Policy expired")
}
// check all formValues appear in postPolicyForm or return error. #https://github.com/minio/minio/issues/17391
checkHeader := map[string][]string{}
ignoreKeys := map[string]bool{}
for key, value := range formValues {
switch {
case ignoreKeys[key], postPolicyIgnoreKeys[key], strings.HasPrefix(key, encrypt.SseGenericHeader):
continue
case strings.HasPrefix(key, "X-Amz-Ignore-"):
ignoreKey := strings.Replace(key, "X-Amz-Ignore-", "", 1)
ignoreKeys[ignoreKey] = true
// if it have already
delete(checkHeader, ignoreKey)
default:
checkHeader[key] = value
}
}
// map to store the metadata
metaMap := make(map[string]string)
for _, policy := range postPolicyForm.Conditions.Policies {
if strings.HasPrefix(policy.Key, "$x-amz-meta-") {
formCanonicalName := http.CanonicalHeaderKey(strings.TrimPrefix(policy.Key, "$"))
metaMap[formCanonicalName] = policy.Value
}
}
// Check if any extra metadata field is passed as input
for key := range formValues {
if strings.HasPrefix(key, "X-Amz-Meta-") {
if _, ok := metaMap[key]; !ok {
return fmt.Errorf("Invalid according to Policy: Extra input fields: %s", key)
}
}
}
// Flag to indicate if all policies conditions are satisfied
var condPassed bool
// mustFindInPolicy is a map to list all the keys that we must find in the policy as
// we process it below. At the end of checkPostPolicy function, if any key is left in
// this map, that's an error.
mustFindInPolicy := make(map[string][]string, len(formValues))
for key, values := range formValues {
if keyInPolicyExceptions[key] || strings.HasPrefix(key, "X-Ignore-") {
continue
}
mustFindInPolicy[key] = values
}
// Iterate over policy conditions and check them against received form fields
for _, policy := range postPolicyForm.Conditions.Policies {
// Form fields names are in canonical format, convert conditions names
// to canonical for simplification purpose, so `$key` will become `Key`
formCanonicalName := http.CanonicalHeaderKey(strings.TrimPrefix(policy.Key, "$"))
// Operator for the current policy condition
op := policy.Operator
// Multiple values should not occur
if len(checkHeader[formCanonicalName]) >= 2 {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed: [%s, %s, %s]. FormValues have multiple values: [%s]", op, policy.Key, policy.Value, strings.Join(checkHeader[formCanonicalName], ", "))
// Multiple values are not allowed for a single form field
if len(mustFindInPolicy[formCanonicalName]) >= 2 {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed: [%s, %s, %s]. FormValues have multiple values: [%s]", op, policy.Key, policy.Value, strings.Join(mustFindInPolicy[formCanonicalName], ", "))
}
// If the current policy condition is known
if startsWithSupported, condFound := startsWithConds[policy.Key]; condFound {
// Check if the current condition supports starts-with operator
@@ -333,35 +318,35 @@ func checkPostPolicy(formValues http.Header, postPolicyForm PostPolicyForm) erro
return fmt.Errorf("Invalid according to Policy: Policy Condition failed")
}
// Check if current policy condition is satisfied
condPassed = checkPolicyCond(op, formValues.Get(formCanonicalName), policy.Value)
if !condPassed {
if !checkPolicyCond(op, formValues.Get(formCanonicalName), policy.Value) {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed")
}
} else if strings.HasPrefix(policy.Key, "$x-amz-meta-") || strings.HasPrefix(policy.Key, "$x-amz-") {
// This covers all conditions X-Amz-Meta-* and X-Amz-*
// Check if policy condition is satisfied
condPassed = checkPolicyCond(op, formValues.Get(formCanonicalName), policy.Value)
if !condPassed {
if !checkPolicyCond(op, formValues.Get(formCanonicalName), policy.Value) {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed: [%s, %s, %s]", op, policy.Key, policy.Value)
}
}
delete(checkHeader, formCanonicalName)
delete(mustFindInPolicy, formCanonicalName)
}
// For SignV2 - Signature/AWSAccessKeyId field will be ignored.
// For SignV2 - Signature/AWSAccessKeyId fields do not need to be in the policy
if _, ok := formValues[xhttp.AmzSignatureV2]; ok {
delete(checkHeader, xhttp.AmzSignatureV2)
for k := range checkHeader {
delete(mustFindInPolicy, xhttp.AmzSignatureV2)
for k := range mustFindInPolicy {
// case-insensitivity for AWSAccessKeyId
if strings.EqualFold(k, xhttp.AmzAccessKeyID) {
delete(checkHeader, k)
delete(mustFindInPolicy, k)
break
}
}
}
if len(checkHeader) != 0 {
logKeys := make([]string, 0, len(checkHeader))
for key := range checkHeader {
// Check mustFindInPolicy to see if any key is left, if so, it was not found in policy and we return an error.
if len(mustFindInPolicy) != 0 {
logKeys := make([]string, 0, len(mustFindInPolicy))
for key := range mustFindInPolicy {
logKeys = append(logKeys, key)
}
return fmt.Errorf("Each form field that you specify in a form must appear in the list of policy conditions. %q not specified in the policy.", strings.Join(logKeys, ", "))
+201 -62
View File
@@ -20,12 +20,12 @@ package cmd
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"strings"
"testing"
minio "github.com/minio/minio-go/v7"
xhttp "github.com/minio/minio/internal/http"
)
func TestParsePostPolicyForm(t *testing.T) {
@@ -78,6 +78,28 @@ func TestParsePostPolicyForm(t *testing.T) {
}
}
type formValues struct {
http.Header
}
func newFormValues() formValues {
return formValues{make(http.Header)}
}
func (f formValues) Set(key, value string) formValues {
f.Header.Set(key, value)
return f
}
func (f formValues) Add(key, value string) formValues {
f.Header.Add(key, value)
return f
}
func (f formValues) Clone() formValues {
return formValues{f.Header.Clone()}
}
// Test Post Policy parsing and checking conditions
func TestPostPolicyForm(t *testing.T) {
pp := minio.NewPostPolicy()
@@ -85,76 +107,193 @@ func TestPostPolicyForm(t *testing.T) {
pp.SetContentType("image/jpeg")
pp.SetUserMetadata("uuid", "14365123651274")
pp.SetKeyStartsWith("user/user1/filename")
pp.SetContentLengthRange(1048579, 10485760)
pp.SetContentLengthRange(100, 999999) // not testable from this layer, condition is checked in the API handler.
pp.SetSuccessStatusAction("201")
pp.SetCondition("eq", "X-Amz-Credential", "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request")
pp.SetCondition("eq", "X-Amz-Algorithm", "AWS4-HMAC-SHA256")
pp.SetCondition("eq", xhttp.AmzDate, "20160727T000000Z")
defaultFormVals := newFormValues()
defaultFormVals.Set("Bucket", "testbucket")
defaultFormVals.Set("Content-Type", "image/jpeg")
defaultFormVals.Set(xhttp.AmzMetaUUID, "14365123651274")
defaultFormVals.Set("Key", "user/user1/filename/${filename}/myfile.txt")
defaultFormVals.Set("X-Amz-Credential", "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request")
defaultFormVals.Set("X-Amz-Algorithm", "AWS4-HMAC-SHA256")
defaultFormVals.Set(xhttp.AmzDate, "20160727T000000Z")
defaultFormVals.Set("Success_action_status", "201")
policyCondFailedErr := "Invalid according to Policy: Policy Condition failed"
type testCase struct {
Bucket string
Key string
XAmzDate string
XAmzAlgorithm string
XAmzCredential string
XAmzMetaUUID string
ContentType string
SuccessActionStatus string
Policy string
Expired bool
expectedErr error
name string
fv formValues
expired bool
wantErr string
}
// Test case just contains fields we override from defaultFormVals.
testCases := []testCase{
// Everything is fine with this test
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: nil},
// Expired policy document
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", Expired: true, expectedErr: fmt.Errorf("Invalid according to Policy: Policy expired")},
// Different AMZ date
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "2017T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Key which doesn't start with user/user1/filename
{Bucket: "testbucket", Key: "myfile.txt", XAmzDate: "20160727T000000Z", XAmzMetaUUID: "14365123651274", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect bucket name.
{Bucket: "incorrect", Key: "user/user1/filename/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect key name
{Bucket: "testbucket", Key: "incorrect", XAmzDate: "20160727T000000Z", XAmzMetaUUID: "14365123651274", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect date
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "incorrect", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect ContentType
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "incorrect", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect Metadata
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "151274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed: [eq, $x-amz-meta-uuid, 14365123651274]")},
{
name: "happy path no errors",
fv: defaultFormVals.Clone(),
wantErr: "",
},
{
name: "expired policy document",
fv: defaultFormVals.Clone(),
expired: true,
wantErr: "Invalid according to Policy: Policy expired",
},
{
name: "different AMZ date",
fv: defaultFormVals.Clone().Set(xhttp.AmzDate, "2017T000000Z"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect date",
fv: defaultFormVals.Clone().Set(xhttp.AmzDate, "incorrect"),
wantErr: policyCondFailedErr,
},
{
name: "key which doesn't start with user/user1/filename",
fv: defaultFormVals.Clone().Set("Key", "myfile.txt"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect key name",
fv: defaultFormVals.Clone().Set("Key", "incorrect"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect bucket name",
fv: defaultFormVals.Clone().Set("Bucket", "incorrect"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect ContentType",
fv: defaultFormVals.Clone().Set(xhttp.ContentType, "incorrect"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect X-Amz-Algorithm",
fv: defaultFormVals.Clone().Set(xhttp.AmzAlgorithm, "incorrect"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect X-Amz-Credential",
fv: defaultFormVals.Clone().Set(xhttp.AmzCredential, "incorrect"),
wantErr: policyCondFailedErr,
},
{
name: "incorrect metadata uuid",
fv: defaultFormVals.Clone().Set(xhttp.AmzMetaUUID, "151274"),
wantErr: "Invalid according to Policy: Policy Condition failed: [eq, $x-amz-meta-uuid, 14365123651274]",
},
{
name: "unknown key XAmzMetaName is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(xhttp.AmzMetaName, "my-name"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Meta-Name" not specified in the policy.`,
},
{
name: "unknown key XAmzChecksumAlgo is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.AmzChecksumAlgo), "algo-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Checksum-Algorithm" not specified in the policy.`,
},
{
name: "unknown key XAmzChecksumCRC32 is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.AmzChecksumCRC32), "crc32-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Checksum-Crc32" not specified in the policy.`,
},
{
name: "unknown key XAmzChecksumCRC32C is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.AmzChecksumCRC32C), "crc32c-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Checksum-Crc32c" not specified in the policy.`,
},
{
name: "unknown key XAmzChecksumSHA1 is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.AmzChecksumSHA1), "sha1-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Checksum-Sha1" not specified in the policy.`,
},
{
name: "unknown key XAmzChecksumSHA256 is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.AmzChecksumSHA256), "sha256-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Checksum-Sha256" not specified in the policy.`,
},
{
name: "unknown key XAmzChecksumMode is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.AmzChecksumMode), "mode-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "X-Amz-Checksum-Mode" not specified in the policy.`,
},
{
name: "unknown key Content-Encoding is error as it does not appear in policy",
fv: defaultFormVals.Clone().Set(http.CanonicalHeaderKey(xhttp.ContentEncoding), "encoding-val"),
wantErr: `Each form field that you specify in a form must appear in the list of policy conditions. "Content-Encoding" not specified in the policy.`,
},
{
name: "many bucket values",
fv: defaultFormVals.Clone().Add("Bucket", "anotherbucket"),
wantErr: "Invalid according to Policy: Policy Condition failed: [eq, $bucket, testbucket]. FormValues have multiple values: [testbucket, anotherbucket]",
},
{
name: "XAmzSignature does not have to appear in policy",
fv: defaultFormVals.Clone().Set(xhttp.AmzSignature, "my-signature"),
},
{
name: "XIgnoreFoo does not have to appear in policy",
fv: defaultFormVals.Clone().Set("X-Ignore-Foo", "my-foo-value"),
},
{
name: "File does not have to appear in policy",
fv: defaultFormVals.Clone().Set("File", "file-value"),
},
{
name: "Signature does not have to appear in policy",
fv: defaultFormVals.Clone().Set(xhttp.AmzSignatureV2, "signature-value"),
},
{
name: "AWSAccessKeyID does not have to appear in policy",
fv: defaultFormVals.Clone().Set(xhttp.AmzAccessKeyID, "access").Set(xhttp.AmzSignatureV2, "signature-value"),
},
{
name: "any form value starting with X-Amz-Server-Side-Encryption- does not have to appear in policy",
fv: defaultFormVals.Clone().
Set(xhttp.AmzServerSideEncryptionKmsContext, "context-val").
Set(xhttp.AmzServerSideEncryptionCustomerAlgorithm, "algo-val"),
},
}
// Validate all the test cases.
for i, tt := range testCases {
formValues := make(http.Header)
formValues.Set("Bucket", tt.Bucket)
formValues.Set("Key", tt.Key)
formValues.Set("Content-Type", tt.ContentType)
formValues.Set("X-Amz-Date", tt.XAmzDate)
formValues.Set("X-Amz-Meta-Uuid", tt.XAmzMetaUUID)
formValues.Set("X-Amz-Algorithm", tt.XAmzAlgorithm)
formValues.Set("X-Amz-Credential", tt.XAmzCredential)
if tt.Expired {
// Expired already.
pp.SetExpires(UTCNow().AddDate(0, 0, -10))
} else {
// Expires in 10 days.
pp.SetExpires(UTCNow().AddDate(0, 0, 10))
}
formValues.Set("Policy", base64.StdEncoding.EncodeToString([]byte(pp.String())))
formValues.Set("Success_action_status", tt.SuccessActionStatus)
policyBytes, err := base64.StdEncoding.DecodeString(base64.StdEncoding.EncodeToString([]byte(pp.String())))
if err != nil {
t.Fatal(err)
}
// Run tests
for _, tt := range testCases {
t.Run(tt.name, func(t *testing.T) {
if tt.expired {
// Expired already.
pp.SetExpires(UTCNow().AddDate(0, 0, -10))
} else {
// Expires in 10 days.
pp.SetExpires(UTCNow().AddDate(0, 0, 10))
}
postPolicyForm, err := parsePostPolicyForm(bytes.NewReader(policyBytes))
if err != nil {
t.Fatal(err)
}
tt.fv.Set("Policy", base64.StdEncoding.EncodeToString([]byte(pp.String())))
err = checkPostPolicy(formValues, postPolicyForm)
if err != nil && tt.expectedErr != nil && err.Error() != tt.expectedErr.Error() {
t.Fatalf("Test %d:, Expected %s, got %s", i+1, tt.expectedErr.Error(), err.Error())
}
policyBytes, err := base64.StdEncoding.DecodeString(base64.StdEncoding.EncodeToString([]byte(pp.String())))
if err != nil {
t.Fatal(err)
}
postPolicyForm, err := parsePostPolicyForm(bytes.NewReader(policyBytes))
if err != nil {
t.Fatal(err)
}
errStr := ""
err = checkPostPolicy(tt.fv.Header, postPolicyForm)
if err != nil {
errStr = err.Error()
}
if errStr != tt.wantErr {
t.Errorf("test: '%s', want error: '%s', got error: '%s'", tt.name, tt.wantErr, errStr)
}
})
}
}
+6 -2
View File
@@ -31,6 +31,7 @@ import (
"os/signal"
"path/filepath"
"runtime"
"slices"
"strings"
"syscall"
"time"
@@ -53,7 +54,6 @@ import (
"github.com/minio/minio/internal/logger"
"github.com/minio/pkg/v3/certs"
"github.com/minio/pkg/v3/env"
"golang.org/x/exp/slices"
"gopkg.in/yaml.v2"
)
@@ -421,12 +421,14 @@ func serverHandleCmdArgs(ctxt serverCtxt) {
Interface: ctxt.Interface,
SendBufSize: ctxt.SendBufSize,
RecvBufSize: ctxt.RecvBufSize,
IdleTimeout: ctxt.IdleTimeout,
}
// allow transport to be HTTP/1.1 for proxying.
globalProxyEndpoints = GetProxyEndpoints(globalEndpoints)
globalInternodeTransport = NewInternodeHTTPTransport(ctxt.MaxIdleConnsPerHost)()
globalRemoteTargetTransport = NewRemoteTargetHTTPTransport(false)()
globalProxyEndpoints = GetProxyEndpoints(globalEndpoints, globalRemoteTargetTransport)
globalForwarder = handlers.NewForwarder(&handlers.Forwarder{
PassHost: true,
RoundTripper: globalRemoteTargetTransport,
@@ -878,6 +880,8 @@ func serverMain(ctx *cli.Context) {
UseHandler(setCriticalErrorHandler(corsHandler(handler))).
UseTLSConfig(newTLSConfig(getCert)).
UseIdleTimeout(globalServerCtxt.IdleTimeout).
UseReadTimeout(globalServerCtxt.IdleTimeout).
UseWriteTimeout(globalServerCtxt.IdleTimeout).
UseReadHeaderTimeout(globalServerCtxt.ReadHeaderTimeout).
UseBaseContext(GlobalContext).
UseCustomLogger(log.New(io.Discard, "", 0)). // Turn-off random logging by Go stdlib
+24 -3
View File
@@ -23,6 +23,7 @@ import (
"errors"
"fmt"
"io"
"net/http"
"os"
"path"
"strings"
@@ -45,6 +46,7 @@ const ftpMaxWriteOffset = 100 << 20
type sftpDriver struct {
permissions *ssh.Permissions
endpoint string
remoteIP string
}
//msgp:ignore sftpMetrics
@@ -89,8 +91,12 @@ func (m *sftpMetrics) log(s *sftp.Request, user string) func(sz int64, err error
// - sftp.Filewrite
// - sftp.Filelist
// - sftp.Filecmd
func NewSFTPDriver(perms *ssh.Permissions) sftp.Handlers {
handler := &sftpDriver{endpoint: fmt.Sprintf("127.0.0.1:%s", globalMinioPort), permissions: perms}
func NewSFTPDriver(perms *ssh.Permissions, remoteIP string) sftp.Handlers {
handler := &sftpDriver{
endpoint: fmt.Sprintf("127.0.0.1:%s", globalMinioPort),
permissions: perms,
remoteIP: remoteIP,
}
return sftp.Handlers{
FileGet: handler,
FilePut: handler,
@@ -99,16 +105,31 @@ func NewSFTPDriver(perms *ssh.Permissions) sftp.Handlers {
}
}
type forwardForTransport struct {
tr http.RoundTripper
fwd string
}
func (f forwardForTransport) RoundTrip(r *http.Request) (*http.Response, error) {
r.Header.Set("X-Forwarded-For", f.fwd)
return f.tr.RoundTrip(r)
}
func (f *sftpDriver) getMinIOClient() (*minio.Client, error) {
mcreds := credentials.NewStaticV4(
f.permissions.CriticalOptions["AccessKey"],
f.permissions.CriticalOptions["SecretKey"],
f.permissions.CriticalOptions["SessionToken"],
)
// Set X-Forwarded-For on all requests.
tr := http.RoundTripper(globalRemoteFTPClientTransport)
if f.remoteIP != "" {
tr = forwardForTransport{tr: tr, fwd: f.remoteIP}
}
return minio.New(f.endpoint, &minio.Options{
Creds: mcreds,
Secure: globalIsTLS,
Transport: globalRemoteFTPClientTransport,
Transport: tr,
})
}
+6 -1
View File
@@ -488,7 +488,12 @@ func startSFTPServer(args []string) {
sshConfig.AddHostKey(private)
handleSFTPSession := func(channel ssh.Channel, sconn *ssh.ServerConn) {
server := sftp.NewRequestServer(channel, NewSFTPDriver(sconn.Permissions), sftp.WithRSAllocator())
var remoteIP string
if host, _, err := net.SplitHostPort(sconn.RemoteAddr().String()); err == nil {
remoteIP = host
}
server := sftp.NewRequestServer(channel, NewSFTPDriver(sconn.Permissions, remoteIP), sftp.WithRSAllocator())
defer server.Close()
server.Serve()
}
+1 -1
View File
@@ -23,6 +23,7 @@ import (
"encoding/hex"
"io"
"net/http"
"slices"
"strconv"
"strings"
@@ -31,7 +32,6 @@ import (
xhttp "github.com/minio/minio/internal/http"
"github.com/minio/minio/internal/logger"
"github.com/minio/pkg/v3/policy"
"golang.org/x/exp/slices"
)
// http Header "x-amz-content-sha256" == "UNSIGNED-PAYLOAD" indicates that the
+1 -1
View File
@@ -60,7 +60,7 @@ const (
// getCanonicalHeaders generate a list of request headers with their values
func getCanonicalHeaders(signedHeaders http.Header) string {
var headers []string
vals := make(http.Header)
vals := make(http.Header, len(signedHeaders))
for k, vv := range signedHeaders {
k = strings.ToLower(k)
headers = append(headers, k)
+3 -2
View File
@@ -101,8 +101,6 @@ type VolsInfo []VolInfo
// VolInfo - represents volume stat information.
// The above means that any added/deleted fields are incompatible.
//
// The above means that any added/deleted fields are incompatible.
//
//msgp:tuple VolInfo
type VolInfo struct {
// Name of the volume.
@@ -110,6 +108,9 @@ type VolInfo struct {
// Date and time when the volume was created.
Created time.Time
// total VolInfo counts
count int
}
// FilesInfo represent a list of files, additionally
+279 -277
View File
@@ -813,11 +813,10 @@ func (z *DeleteOptions) DecodeMsg(dc *msgp.Reader) (err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.OldDataDir = ""
}
if (zb0001Mask & 0x1) == 0 {
z.OldDataDir = ""
}
return
}
@@ -836,61 +835,62 @@ func (z *DeleteOptions) EncodeMsg(en *msgp.Writer) (err error) {
if err != nil {
return
}
if zb0001Len == 0 {
return
}
// write "BaseOptions"
err = en.Append(0xab, 0x42, 0x61, 0x73, 0x65, 0x4f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73)
if err != nil {
return
}
// map header, size 0
_ = z.BaseOptions
err = en.Append(0x80)
if err != nil {
return
}
// write "r"
err = en.Append(0xa1, 0x72)
if err != nil {
return
}
err = en.WriteBool(z.Recursive)
if err != nil {
err = msgp.WrapError(err, "Recursive")
return
}
// write "i"
err = en.Append(0xa1, 0x69)
if err != nil {
return
}
err = en.WriteBool(z.Immediate)
if err != nil {
err = msgp.WrapError(err, "Immediate")
return
}
// write "u"
err = en.Append(0xa1, 0x75)
if err != nil {
return
}
err = en.WriteBool(z.UndoWrite)
if err != nil {
err = msgp.WrapError(err, "UndoWrite")
return
}
if (zb0001Mask & 0x10) == 0 { // if not omitted
// write "o"
err = en.Append(0xa1, 0x6f)
// skip if no fields are to be emitted
if zb0001Len != 0 {
// write "BaseOptions"
err = en.Append(0xab, 0x42, 0x61, 0x73, 0x65, 0x4f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73)
if err != nil {
return
}
err = en.WriteString(z.OldDataDir)
// map header, size 0
_ = z.BaseOptions
err = en.Append(0x80)
if err != nil {
err = msgp.WrapError(err, "OldDataDir")
return
}
// write "r"
err = en.Append(0xa1, 0x72)
if err != nil {
return
}
err = en.WriteBool(z.Recursive)
if err != nil {
err = msgp.WrapError(err, "Recursive")
return
}
// write "i"
err = en.Append(0xa1, 0x69)
if err != nil {
return
}
err = en.WriteBool(z.Immediate)
if err != nil {
err = msgp.WrapError(err, "Immediate")
return
}
// write "u"
err = en.Append(0xa1, 0x75)
if err != nil {
return
}
err = en.WriteBool(z.UndoWrite)
if err != nil {
err = msgp.WrapError(err, "UndoWrite")
return
}
if (zb0001Mask & 0x10) == 0 { // if not omitted
// write "o"
err = en.Append(0xa1, 0x6f)
if err != nil {
return
}
err = en.WriteString(z.OldDataDir)
if err != nil {
err = msgp.WrapError(err, "OldDataDir")
return
}
}
}
return
}
@@ -908,27 +908,28 @@ func (z *DeleteOptions) MarshalMsg(b []byte) (o []byte, err error) {
}
// variable map header, size zb0001Len
o = append(o, 0x80|uint8(zb0001Len))
if zb0001Len == 0 {
return
}
// string "BaseOptions"
o = append(o, 0xab, 0x42, 0x61, 0x73, 0x65, 0x4f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73)
// map header, size 0
_ = z.BaseOptions
o = append(o, 0x80)
// string "r"
o = append(o, 0xa1, 0x72)
o = msgp.AppendBool(o, z.Recursive)
// string "i"
o = append(o, 0xa1, 0x69)
o = msgp.AppendBool(o, z.Immediate)
// string "u"
o = append(o, 0xa1, 0x75)
o = msgp.AppendBool(o, z.UndoWrite)
if (zb0001Mask & 0x10) == 0 { // if not omitted
// string "o"
o = append(o, 0xa1, 0x6f)
o = msgp.AppendString(o, z.OldDataDir)
// skip if no fields are to be emitted
if zb0001Len != 0 {
// string "BaseOptions"
o = append(o, 0xab, 0x42, 0x61, 0x73, 0x65, 0x4f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73)
// map header, size 0
_ = z.BaseOptions
o = append(o, 0x80)
// string "r"
o = append(o, 0xa1, 0x72)
o = msgp.AppendBool(o, z.Recursive)
// string "i"
o = append(o, 0xa1, 0x69)
o = msgp.AppendBool(o, z.Immediate)
// string "u"
o = append(o, 0xa1, 0x75)
o = msgp.AppendBool(o, z.UndoWrite)
if (zb0001Mask & 0x10) == 0 { // if not omitted
// string "o"
o = append(o, 0xa1, 0x6f)
o = msgp.AppendString(o, z.OldDataDir)
}
}
return
}
@@ -1010,11 +1011,10 @@ func (z *DeleteOptions) UnmarshalMsg(bts []byte) (o []byte, err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.OldDataDir = ""
}
if (zb0001Mask & 0x1) == 0 {
z.OldDataDir = ""
}
o = bts
return
}
@@ -4331,11 +4331,10 @@ func (z *ReadMultipleReq) DecodeMsg(dc *msgp.Reader) (err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
return
}
@@ -4354,87 +4353,88 @@ func (z *ReadMultipleReq) EncodeMsg(en *msgp.Writer) (err error) {
if err != nil {
return
}
if zb0001Len == 0 {
return
}
// write "bk"
err = en.Append(0xa2, 0x62, 0x6b)
if err != nil {
return
}
err = en.WriteString(z.Bucket)
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
if (zb0001Mask & 0x2) == 0 { // if not omitted
// write "pr"
err = en.Append(0xa2, 0x70, 0x72)
// skip if no fields are to be emitted
if zb0001Len != 0 {
// write "bk"
err = en.Append(0xa2, 0x62, 0x6b)
if err != nil {
return
}
err = en.WriteString(z.Prefix)
err = en.WriteString(z.Bucket)
if err != nil {
err = msgp.WrapError(err, "Prefix")
err = msgp.WrapError(err, "Bucket")
return
}
}
// write "fl"
err = en.Append(0xa2, 0x66, 0x6c)
if err != nil {
return
}
err = en.WriteArrayHeader(uint32(len(z.Files)))
if err != nil {
err = msgp.WrapError(err, "Files")
return
}
for za0001 := range z.Files {
err = en.WriteString(z.Files[za0001])
if (zb0001Mask & 0x2) == 0 { // if not omitted
// write "pr"
err = en.Append(0xa2, 0x70, 0x72)
if err != nil {
return
}
err = en.WriteString(z.Prefix)
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
}
// write "fl"
err = en.Append(0xa2, 0x66, 0x6c)
if err != nil {
err = msgp.WrapError(err, "Files", za0001)
return
}
}
// write "ms"
err = en.Append(0xa2, 0x6d, 0x73)
if err != nil {
return
}
err = en.WriteInt64(z.MaxSize)
if err != nil {
err = msgp.WrapError(err, "MaxSize")
return
}
// write "mo"
err = en.Append(0xa2, 0x6d, 0x6f)
if err != nil {
return
}
err = en.WriteBool(z.MetadataOnly)
if err != nil {
err = msgp.WrapError(err, "MetadataOnly")
return
}
// write "ab"
err = en.Append(0xa2, 0x61, 0x62)
if err != nil {
return
}
err = en.WriteBool(z.AbortOn404)
if err != nil {
err = msgp.WrapError(err, "AbortOn404")
return
}
// write "mr"
err = en.Append(0xa2, 0x6d, 0x72)
if err != nil {
return
}
err = en.WriteInt(z.MaxResults)
if err != nil {
err = msgp.WrapError(err, "MaxResults")
return
err = en.WriteArrayHeader(uint32(len(z.Files)))
if err != nil {
err = msgp.WrapError(err, "Files")
return
}
for za0001 := range z.Files {
err = en.WriteString(z.Files[za0001])
if err != nil {
err = msgp.WrapError(err, "Files", za0001)
return
}
}
// write "ms"
err = en.Append(0xa2, 0x6d, 0x73)
if err != nil {
return
}
err = en.WriteInt64(z.MaxSize)
if err != nil {
err = msgp.WrapError(err, "MaxSize")
return
}
// write "mo"
err = en.Append(0xa2, 0x6d, 0x6f)
if err != nil {
return
}
err = en.WriteBool(z.MetadataOnly)
if err != nil {
err = msgp.WrapError(err, "MetadataOnly")
return
}
// write "ab"
err = en.Append(0xa2, 0x61, 0x62)
if err != nil {
return
}
err = en.WriteBool(z.AbortOn404)
if err != nil {
err = msgp.WrapError(err, "AbortOn404")
return
}
// write "mr"
err = en.Append(0xa2, 0x6d, 0x72)
if err != nil {
return
}
err = en.WriteInt(z.MaxResults)
if err != nil {
err = msgp.WrapError(err, "MaxResults")
return
}
}
return
}
@@ -4452,35 +4452,36 @@ func (z *ReadMultipleReq) MarshalMsg(b []byte) (o []byte, err error) {
}
// variable map header, size zb0001Len
o = append(o, 0x80|uint8(zb0001Len))
if zb0001Len == 0 {
return
// skip if no fields are to be emitted
if zb0001Len != 0 {
// string "bk"
o = append(o, 0xa2, 0x62, 0x6b)
o = msgp.AppendString(o, z.Bucket)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// string "pr"
o = append(o, 0xa2, 0x70, 0x72)
o = msgp.AppendString(o, z.Prefix)
}
// string "fl"
o = append(o, 0xa2, 0x66, 0x6c)
o = msgp.AppendArrayHeader(o, uint32(len(z.Files)))
for za0001 := range z.Files {
o = msgp.AppendString(o, z.Files[za0001])
}
// string "ms"
o = append(o, 0xa2, 0x6d, 0x73)
o = msgp.AppendInt64(o, z.MaxSize)
// string "mo"
o = append(o, 0xa2, 0x6d, 0x6f)
o = msgp.AppendBool(o, z.MetadataOnly)
// string "ab"
o = append(o, 0xa2, 0x61, 0x62)
o = msgp.AppendBool(o, z.AbortOn404)
// string "mr"
o = append(o, 0xa2, 0x6d, 0x72)
o = msgp.AppendInt(o, z.MaxResults)
}
// string "bk"
o = append(o, 0xa2, 0x62, 0x6b)
o = msgp.AppendString(o, z.Bucket)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// string "pr"
o = append(o, 0xa2, 0x70, 0x72)
o = msgp.AppendString(o, z.Prefix)
}
// string "fl"
o = append(o, 0xa2, 0x66, 0x6c)
o = msgp.AppendArrayHeader(o, uint32(len(z.Files)))
for za0001 := range z.Files {
o = msgp.AppendString(o, z.Files[za0001])
}
// string "ms"
o = append(o, 0xa2, 0x6d, 0x73)
o = msgp.AppendInt64(o, z.MaxSize)
// string "mo"
o = append(o, 0xa2, 0x6d, 0x6f)
o = msgp.AppendBool(o, z.MetadataOnly)
// string "ab"
o = append(o, 0xa2, 0x61, 0x62)
o = msgp.AppendBool(o, z.AbortOn404)
// string "mr"
o = append(o, 0xa2, 0x6d, 0x72)
o = msgp.AppendInt(o, z.MaxResults)
return
}
@@ -4569,11 +4570,10 @@ func (z *ReadMultipleReq) UnmarshalMsg(bts []byte) (o []byte, err error) {
}
}
// Clear omitted fields.
if zb0001Mask != 0x1 {
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
if (zb0001Mask & 0x1) == 0 {
z.Prefix = ""
}
o = bts
return
}
@@ -4691,82 +4691,83 @@ func (z *ReadMultipleResp) EncodeMsg(en *msgp.Writer) (err error) {
if err != nil {
return
}
if zb0001Len == 0 {
return
}
// write "bk"
err = en.Append(0xa2, 0x62, 0x6b)
if err != nil {
return
}
err = en.WriteString(z.Bucket)
if err != nil {
err = msgp.WrapError(err, "Bucket")
return
}
if (zb0001Mask & 0x2) == 0 { // if not omitted
// write "pr"
err = en.Append(0xa2, 0x70, 0x72)
// skip if no fields are to be emitted
if zb0001Len != 0 {
// write "bk"
err = en.Append(0xa2, 0x62, 0x6b)
if err != nil {
return
}
err = en.WriteString(z.Prefix)
err = en.WriteString(z.Bucket)
if err != nil {
err = msgp.WrapError(err, "Prefix")
err = msgp.WrapError(err, "Bucket")
return
}
}
// write "fl"
err = en.Append(0xa2, 0x66, 0x6c)
if err != nil {
return
}
err = en.WriteString(z.File)
if err != nil {
err = msgp.WrapError(err, "File")
return
}
// write "ex"
err = en.Append(0xa2, 0x65, 0x78)
if err != nil {
return
}
err = en.WriteBool(z.Exists)
if err != nil {
err = msgp.WrapError(err, "Exists")
return
}
if (zb0001Mask & 0x10) == 0 { // if not omitted
// write "er"
err = en.Append(0xa2, 0x65, 0x72)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// write "pr"
err = en.Append(0xa2, 0x70, 0x72)
if err != nil {
return
}
err = en.WriteString(z.Prefix)
if err != nil {
err = msgp.WrapError(err, "Prefix")
return
}
}
// write "fl"
err = en.Append(0xa2, 0x66, 0x6c)
if err != nil {
return
}
err = en.WriteString(z.Error)
err = en.WriteString(z.File)
if err != nil {
err = msgp.WrapError(err, "Error")
err = msgp.WrapError(err, "File")
return
}
// write "ex"
err = en.Append(0xa2, 0x65, 0x78)
if err != nil {
return
}
err = en.WriteBool(z.Exists)
if err != nil {
err = msgp.WrapError(err, "Exists")
return
}
if (zb0001Mask & 0x10) == 0 { // if not omitted
// write "er"
err = en.Append(0xa2, 0x65, 0x72)
if err != nil {
return
}
err = en.WriteString(z.Error)
if err != nil {
err = msgp.WrapError(err, "Error")
return
}
}
// write "d"
err = en.Append(0xa1, 0x64)
if err != nil {
return
}
err = en.WriteBytes(z.Data)
if err != nil {
err = msgp.WrapError(err, "Data")
return
}
// write "m"
err = en.Append(0xa1, 0x6d)
if err != nil {
return
}
err = en.WriteTime(z.Modtime)
if err != nil {
err = msgp.WrapError(err, "Modtime")
return
}
}
// write "d"
err = en.Append(0xa1, 0x64)
if err != nil {
return
}
err = en.WriteBytes(z.Data)
if err != nil {
err = msgp.WrapError(err, "Data")
return
}
// write "m"
err = en.Append(0xa1, 0x6d)
if err != nil {
return
}
err = en.WriteTime(z.Modtime)
if err != nil {
err = msgp.WrapError(err, "Modtime")
return
}
return
}
@@ -4788,34 +4789,35 @@ func (z *ReadMultipleResp) MarshalMsg(b []byte) (o []byte, err error) {
}
// variable map header, size zb0001Len
o = append(o, 0x80|uint8(zb0001Len))
if zb0001Len == 0 {
return
// skip if no fields are to be emitted
if zb0001Len != 0 {
// string "bk"
o = append(o, 0xa2, 0x62, 0x6b)
o = msgp.AppendString(o, z.Bucket)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// string "pr"
o = append(o, 0xa2, 0x70, 0x72)
o = msgp.AppendString(o, z.Prefix)
}
// string "fl"
o = append(o, 0xa2, 0x66, 0x6c)
o = msgp.AppendString(o, z.File)
// string "ex"
o = append(o, 0xa2, 0x65, 0x78)
o = msgp.AppendBool(o, z.Exists)
if (zb0001Mask & 0x10) == 0 { // if not omitted
// string "er"
o = append(o, 0xa2, 0x65, 0x72)
o = msgp.AppendString(o, z.Error)
}
// string "d"
o = append(o, 0xa1, 0x64)
o = msgp.AppendBytes(o, z.Data)
// string "m"
o = append(o, 0xa1, 0x6d)
o = msgp.AppendTime(o, z.Modtime)
}
// string "bk"
o = append(o, 0xa2, 0x62, 0x6b)
o = msgp.AppendString(o, z.Bucket)
if (zb0001Mask & 0x2) == 0 { // if not omitted
// string "pr"
o = append(o, 0xa2, 0x70, 0x72)
o = msgp.AppendString(o, z.Prefix)
}
// string "fl"
o = append(o, 0xa2, 0x66, 0x6c)
o = msgp.AppendString(o, z.File)
// string "ex"
o = append(o, 0xa2, 0x65, 0x78)
o = msgp.AppendBool(o, z.Exists)
if (zb0001Mask & 0x10) == 0 { // if not omitted
// string "er"
o = append(o, 0xa2, 0x65, 0x72)
o = msgp.AppendString(o, z.Error)
}
// string "d"
o = append(o, 0xa1, 0x64)
o = msgp.AppendBytes(o, z.Data)
// string "m"
o = append(o, 0xa1, 0x6d)
o = msgp.AppendTime(o, z.Modtime)
return
}
+7 -3
View File
@@ -464,16 +464,20 @@ func (client *storageRESTClient) WriteAll(ctx context.Context, volume string, pa
// CheckParts - stat all file parts.
func (client *storageRESTClient) CheckParts(ctx context.Context, volume string, path string, fi FileInfo) (*CheckPartsResp, error) {
var resp *CheckPartsResp
resp, err := storageCheckPartsRPC.Call(ctx, client.gridConn, &CheckPartsHandlerParams{
st, err := storageCheckPartsRPC.Call(ctx, client.gridConn, &CheckPartsHandlerParams{
DiskID: *client.diskID.Load(),
Volume: volume,
FilePath: path,
FI: fi,
})
if err != nil {
return nil, err
return nil, toStorageErr(err)
}
return resp, nil
err = st.Results(func(r *CheckPartsResp) error {
resp = r
return nil
})
return resp, toStorageErr(err)
}
// RenameData - rename source path to destination path atomically, metadata and data file.
+11 -7
View File
@@ -56,7 +56,7 @@ type storageRESTServer struct {
}
var (
storageCheckPartsRPC = grid.NewSingleHandler[*CheckPartsHandlerParams, *CheckPartsResp](grid.HandlerCheckParts2, func() *CheckPartsHandlerParams { return &CheckPartsHandlerParams{} }, func() *CheckPartsResp { return &CheckPartsResp{} })
storageCheckPartsRPC = grid.NewStream[*CheckPartsHandlerParams, grid.NoPayload, *CheckPartsResp](grid.HandlerCheckParts3, func() *CheckPartsHandlerParams { return &CheckPartsHandlerParams{} }, nil, func() *CheckPartsResp { return &CheckPartsResp{} })
storageDeleteFileRPC = grid.NewSingleHandler[*DeleteFileHandlerParams, grid.NoPayload](grid.HandlerDeleteFile, func() *DeleteFileHandlerParams { return &DeleteFileHandlerParams{} }, grid.NewNoPayload).AllowCallRequestPool(true)
storageDeleteVersionRPC = grid.NewSingleHandler[*DeleteVersionHandlerParams, grid.NoPayload](grid.HandlerDeleteVersion, func() *DeleteVersionHandlerParams { return &DeleteVersionHandlerParams{} }, grid.NewNoPayload)
storageDiskInfoRPC = grid.NewSingleHandler[*DiskInfoOptions, *DiskInfo](grid.HandlerDiskInfo, func() *DiskInfoOptions { return &DiskInfoOptions{} }, func() *DiskInfo { return &DiskInfo{} }).WithSharedResponse().AllowCallRequestPool(true)
@@ -456,16 +456,20 @@ func (s *storageRESTServer) UpdateMetadataHandler(p *MetadataHandlerParams) (gri
return grid.NewNPErr(s.getStorage().UpdateMetadata(context.Background(), volume, filePath, p.FI, p.UpdateOpts))
}
// CheckPartsHandler - check if a file metadata exists.
func (s *storageRESTServer) CheckPartsHandler(p *CheckPartsHandlerParams) (*CheckPartsResp, *grid.RemoteErr) {
// CheckPartsHandler - check if a file parts exists.
func (s *storageRESTServer) CheckPartsHandler(ctx context.Context, p *CheckPartsHandlerParams, out chan<- *CheckPartsResp) *grid.RemoteErr {
if !s.checkID(p.DiskID) {
return nil, grid.NewRemoteErr(errDiskNotFound)
return grid.NewRemoteErr(errDiskNotFound)
}
volume := p.Volume
filePath := p.FilePath
resp, err := s.getStorage().CheckParts(context.Background(), volume, filePath, p.FI)
return resp, grid.NewRemoteErr(err)
resp, err := s.getStorage().CheckParts(ctx, volume, filePath, p.FI)
if err != nil {
return grid.NewRemoteErr(err)
}
out <- resp
return grid.NewRemoteErr(err)
}
func (s *storageRESTServer) WriteAllHandler(p *WriteAllHandlerParams) (grid.NoPayload, *grid.RemoteErr) {
@@ -1382,7 +1386,7 @@ func registerStorageRESTHandlers(router *mux.Router, endpointServerPools Endpoin
logger.FatalIf(storageRenameDataRPC.Register(gm, server.RenameDataHandler, endpoint.Path), "unable to register handler")
logger.FatalIf(storageRenameDataInlineRPC.Register(gm, server.RenameDataInlineHandler, endpoint.Path), "unable to register handler")
logger.FatalIf(storageDeleteFileRPC.Register(gm, server.DeleteFileHandler, endpoint.Path), "unable to register handler")
logger.FatalIf(storageCheckPartsRPC.Register(gm, server.CheckPartsHandler, endpoint.Path), "unable to register handler")
logger.FatalIf(storageCheckPartsRPC.RegisterNoInput(gm, server.CheckPartsHandler, endpoint.Path), "unable to register handler")
logger.FatalIf(storageReadVersionRPC.Register(gm, server.ReadVersionHandlerWS, endpoint.Path), "unable to register handler")
logger.FatalIf(storageWriteMetadataRPC.Register(gm, server.WriteMetadataHandler, endpoint.Path), "unable to register handler")
logger.FatalIf(storageUpdateMetadataRPC.Register(gm, server.UpdateMetadataHandler, endpoint.Path), "unable to register handler")
+1 -1
View File
@@ -24,6 +24,7 @@ import (
"io"
"os"
"reflect"
"slices"
"strings"
"testing"
"time"
@@ -34,7 +35,6 @@ import (
cr "github.com/minio/minio-go/v7/pkg/credentials"
"github.com/minio/minio-go/v7/pkg/set"
"github.com/minio/pkg/v3/ldap"
"golang.org/x/exp/slices"
)
func runAllIAMSTSTests(suite *TestSuiteIAM, c *check) {
+4 -4
View File
@@ -2247,12 +2247,12 @@ func getEndpointsLocalAddr(endpointServerPools EndpointServerPools) string {
}
// fetches a random number between range min-max.
func getRandomRange(min, max int, seed int64) int {
func getRandomRange(minN, maxN int, seed int64) int {
// special value -1 means no explicit seeding.
if seed != -1 {
rand.Seed(seed)
if seed == -1 {
return rand.New(rand.NewSource(time.Now().UnixNano())).Intn(maxN-minN) + minN
}
return rand.Intn(max-min) + min
return rand.New(rand.NewSource(seed)).Intn(maxN-minN) + minN
}
// Randomizes the order of bytes in the byte array
+29 -4
View File
@@ -20,7 +20,9 @@ package cmd
import (
"bytes"
"context"
"crypto/md5"
"crypto/tls"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
@@ -254,10 +256,28 @@ func xmlDecoder(body io.Reader, v interface{}, size int64) error {
return err
}
// hasContentMD5 returns true if Content-MD5 header is set.
func hasContentMD5(h http.Header) bool {
_, ok := h[xhttp.ContentMD5]
return ok
// validateLengthAndChecksum returns if a content checksum is set,
// and will replace r.Body with a reader that checks the provided checksum
func validateLengthAndChecksum(r *http.Request) bool {
if mdFive := r.Header.Get(xhttp.ContentMD5); mdFive != "" {
want, err := base64.StdEncoding.DecodeString(mdFive)
if err != nil {
return false
}
r.Body = hash.NewChecker(r.Body, md5.New(), want, r.ContentLength)
return true
}
cs, err := hash.GetContentChecksum(r.Header)
if err != nil {
return false
}
if cs == nil || !cs.Type.IsSet() {
return false
}
if cs.Valid() && !cs.Type.Trailing() {
r.Body = hash.NewChecker(r.Body, cs.Type.Hasher(), cs.Raw, r.ContentLength)
}
return true
}
// http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
@@ -411,7 +431,12 @@ func startProfiler(profilerType string) (minioProfiler, error) {
return nil, err
}
stop := fgprof.Start(f, fgprof.FormatPprof)
startedAt := time.Now()
prof.stopFn = func() ([]byte, error) {
if elapsed := time.Since(startedAt); elapsed < 100*time.Millisecond {
// Light hack around https://github.com/felixge/fgprof/pull/34
time.Sleep(100*time.Millisecond - elapsed)
}
err := stop()
if err != nil {
return nil, err
+11 -1
View File
@@ -26,6 +26,7 @@ import (
"strings"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
@@ -59,10 +60,15 @@ func (az *warmBackendAzure) getDest(object string) string {
return destObj
}
func (az *warmBackendAzure) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
func (az *warmBackendAzure) PutWithMeta(ctx context.Context, object string, r io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
azMeta := map[string]*string{}
for k, v := range meta {
azMeta[k] = to.Ptr(v)
}
resp, err := az.clnt.UploadStream(ctx, az.Bucket, az.getDest(object), io.LimitReader(r, length), &azblob.UploadStreamOptions{
Concurrency: 4,
AccessTier: az.tier(), // set tier if specified
Metadata: azMeta,
})
if err != nil {
return "", azureToObjectError(err, az.Bucket, az.getDest(object))
@@ -74,6 +80,10 @@ func (az *warmBackendAzure) Put(ctx context.Context, object string, r io.Reader,
return remoteVersionID(vid), nil
}
func (az *warmBackendAzure) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
return az.PutWithMeta(ctx, object, r, length, map[string]string{})
}
func (az *warmBackendAzure) Get(ctx context.Context, object string, rv remoteVersionID, opts WarmBackendGetOpts) (r io.ReadCloser, err error) {
if opts.startOffset < 0 {
return nil, InvalidRange{}
+12 -5
View File
@@ -47,16 +47,17 @@ func (gcs *warmBackendGCS) getDest(object string) string {
return destObj
}
// FIXME: add support for remote version ID in GCS remote tier and remove this.
// Currently it's a no-op.
func (gcs *warmBackendGCS) Put(ctx context.Context, key string, data io.Reader, length int64) (remoteVersionID, error) {
func (gcs *warmBackendGCS) PutWithMeta(ctx context.Context, key string, data io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
object := gcs.client.Bucket(gcs.Bucket).Object(gcs.getDest(key))
// TODO: set storage class
w := object.NewWriter(ctx)
if gcs.StorageClass != "" {
w.ObjectAttrs.StorageClass = gcs.StorageClass
}
w.ObjectAttrs.Metadata = meta
if _, err := xioutil.Copy(w, data); err != nil {
return "", gcsToObjectError(err, gcs.Bucket, key)
}
if _, err := xioutil.Copy(w, data); err != nil {
return "", gcsToObjectError(err, gcs.Bucket, key)
}
@@ -64,6 +65,12 @@ func (gcs *warmBackendGCS) Put(ctx context.Context, key string, data io.Reader,
return "", w.Close()
}
// FIXME: add support for remote version ID in GCS remote tier and remove this.
// Currently it's a no-op.
func (gcs *warmBackendGCS) Put(ctx context.Context, key string, data io.Reader, length int64) (remoteVersionID, error) {
return gcs.PutWithMeta(ctx, key, data, length, map[string]string{})
}
func (gcs *warmBackendGCS) Get(ctx context.Context, key string, rv remoteVersionID, opts WarmBackendGetOpts) (r io.ReadCloser, err error) {
// GCS storage decompresses a gzipped object by default and returns the data.
// Refer to https://cloud.google.com/storage/docs/transcoding#decompressive_transcoding
+6 -1
View File
@@ -78,7 +78,7 @@ func optimalPartSize(objectSize int64) (partSize int64, err error) {
return partSize, nil
}
func (m *warmBackendMinIO) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
func (m *warmBackendMinIO) PutWithMeta(ctx context.Context, object string, r io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
partSize, err := optimalPartSize(length)
if err != nil {
return remoteVersionID(""), err
@@ -87,10 +87,15 @@ func (m *warmBackendMinIO) Put(ctx context.Context, object string, r io.Reader,
StorageClass: m.StorageClass,
PartSize: uint64(partSize),
DisableContentSha256: true,
UserMetadata: meta,
})
return remoteVersionID(res.VersionID), m.ToObjectError(err, object)
}
func (m *warmBackendMinIO) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
return m.PutWithMeta(ctx, object, r, length, map[string]string{})
}
func newWarmBackendMinIO(conf madmin.TierMinIO, tier string) (*warmBackendMinIO, error) {
// Validation of credentials
if conf.AccessKey == "" || conf.SecretKey == "" {

Some files were not shown because too many files have changed in this diff Show More