count metrics properly for any failures during drive heal (#20193 )

or via `mc admin heal --set 1 --pool 1`
Add fmt-gen subcommand (#20192 )
2026-11-03 14:22:12 +03:00 · 2024-07-30 22:46:26 -07:00 · 2024-07-30 15:59:48 -07:00 · 2024-07-30 15:28:46 -07:00 · 2024-07-30 12:01:06 -07:00 · 2024-07-29 18:56:40 -07:00
138 changed files with 4697 additions and 2125 deletions
@@ -3,6 +3,8 @@ extend-exclude = [
    ".git/",
    "docs/",
    "CREDITS",
+    "go.mod",
+    "go.sum",
 ]
 ignore-hidden = false

@@ -17,6 +19,7 @@ extend-ignore-re = [
    "MIIDBTCCAe2gAwIBAgIQWHw7h.*",
    'http\.Header\{"X-Amz-Server-Side-Encryptio":',
    "ZoEoZdLlzVbOlT9rbhD7ZN7TLyiYXSAlB79uGEge",
+    "ERRO:",
 ]

 [default.extend-words]
@@ -13275,376 +13275,6 @@ Mozilla Public License, version 2.0
   be used to construe this License against a Contributor.


-10. Versions of the License
-
-10.1. New Versions
-
-      Mozilla Foundation is the license steward. Except as provided in Section
-      10.3, no one other than the license steward has the right to modify or
-      publish new versions of this License. Each version will be given a
-      distinguishing version number.
-
-10.2. Effect of New Versions
-
-      You may distribute the Covered Software under the terms of the version
-      of the License under which You originally received the Covered Software,
-      or under the terms of any subsequent version published by the license
-      steward.
-
-10.3. Modified Versions
-
-      If you create software not governed by this License, and you want to
-      create a new license for such software, you may create and use a
-      modified version of this License if you rename the license and remove
-      any references to the name of the license steward (except to note that
-      such modified license differs from this License).
-
-10.4. Distributing Source Code Form that is Incompatible With Secondary
-      Licenses If You choose to distribute Source Code Form that is
-      Incompatible With Secondary Licenses under the terms of this version of
-      the License, the notice described in Exhibit B of this License must be
-      attached.
-
-Exhibit A - Source Code Form License Notice
-
-      This Source Code Form is subject to the
-      terms of the Mozilla Public License, v.
-      2.0. If a copy of the MPL was not
-      distributed with this file, You can
-      obtain one at
-      http://mozilla.org/MPL/2.0/.
-
-If it is not possible or desirable to put the notice in a particular file,
-then You may include the notice in a location (such as a LICENSE file in a
-relevant directory) where a recipient would be likely to look for such a
-notice.
-
-You may add additional accurate notices of copyright ownership.
-
-Exhibit B - "Incompatible With Secondary Licenses" Notice
-
-      This Source Code Form is "Incompatible
-      With Secondary Licenses", as defined by
-      the Mozilla Public License, v. 2.0.
-
-================================================================
-
-github.com/hashicorp/golang-lru/v2
-https://github.com/hashicorp/golang-lru/v2
----------------------------------------------------------------
-Copyright (c) 2014 HashiCorp, Inc.
-
-Mozilla Public License, version 2.0
-
-1. Definitions
-
-1.1. "Contributor"
-
-     means each individual or legal entity that creates, contributes to the
-     creation of, or owns Covered Software.
-
-1.2. "Contributor Version"
-
-     means the combination of the Contributions of others (if any) used by a
-     Contributor and that particular Contributor's Contribution.
-
-1.3. "Contribution"
-
-     means Covered Software of a particular Contributor.
-
-1.4. "Covered Software"
-
-     means Source Code Form to which the initial Contributor has attached the
-     notice in Exhibit A, the Executable Form of such Source Code Form, and
-     Modifications of such Source Code Form, in each case including portions
-     thereof.
-
-1.5. "Incompatible With Secondary Licenses"
-     means
-
-     a. that the initial Contributor has attached the notice described in
-        Exhibit B to the Covered Software; or
-
-     b. that the Covered Software was made available under the terms of
-        version 1.1 or earlier of the License, but not also under the terms of
-        a Secondary License.
-
-1.6. "Executable Form"
-
-     means any form of the work other than Source Code Form.
-
-1.7. "Larger Work"
-
-     means a work that combines Covered Software with other material, in a
-     separate file or files, that is not Covered Software.
-
-1.8. "License"
-
-     means this document.
-
-1.9. "Licensable"
-
-     means having the right to grant, to the maximum extent possible, whether
-     at the time of the initial grant or subsequently, any and all of the
-     rights conveyed by this License.
-
-1.10. "Modifications"
-
-     means any of the following:
-
-     a. any file in Source Code Form that results from an addition to,
-        deletion from, or modification of the contents of Covered Software; or
-
-     b. any new file in Source Code Form that contains any Covered Software.
-
-1.11. "Patent Claims" of a Contributor
-
-      means any patent claim(s), including without limitation, method,
-      process, and apparatus claims, in any patent Licensable by such
-      Contributor that would be infringed, but for the grant of the License,
-      by the making, using, selling, offering for sale, having made, import,
-      or transfer of either its Contributions or its Contributor Version.
-
-1.12. "Secondary License"
-
-      means either the GNU General Public License, Version 2.0, the GNU Lesser
-      General Public License, Version 2.1, the GNU Affero General Public
-      License, Version 3.0, or any later versions of those licenses.
-
-1.13. "Source Code Form"
-
-      means the form of the work preferred for making modifications.
-
-1.14. "You" (or "Your")
-
-      means an individual or a legal entity exercising rights under this
-      License. For legal entities, "You" includes any entity that controls, is
-      controlled by, or is under common control with You. For purposes of this
-      definition, "control" means (a) the power, direct or indirect, to cause
-      the direction or management of such entity, whether by contract or
-      otherwise, or (b) ownership of more than fifty percent (50%) of the
-      outstanding shares or beneficial ownership of such entity.
-
-
-2. License Grants and Conditions
-
-2.1. Grants
-
-     Each Contributor hereby grants You a world-wide, royalty-free,
-     non-exclusive license:
-
-     a. under intellectual property rights (other than patent or trademark)
-        Licensable by such Contributor to use, reproduce, make available,
-        modify, display, perform, distribute, and otherwise exploit its
-        Contributions, either on an unmodified basis, with Modifications, or
-        as part of a Larger Work; and
-
-     b. under Patent Claims of such Contributor to make, use, sell, offer for
-        sale, have made, import, and otherwise transfer either its
-        Contributions or its Contributor Version.
-
-2.2. Effective Date
-
-     The licenses granted in Section 2.1 with respect to any Contribution
-     become effective for each Contribution on the date the Contributor first
-     distributes such Contribution.
-
-2.3. Limitations on Grant Scope
-
-     The licenses granted in this Section 2 are the only rights granted under
-     this License. No additional rights or licenses will be implied from the
-     distribution or licensing of Covered Software under this License.
-     Notwithstanding Section 2.1(b) above, no patent license is granted by a
-     Contributor:
-
-     a. for any code that a Contributor has removed from Covered Software; or
-
-     b. for infringements caused by: (i) Your and any other third party's
-        modifications of Covered Software, or (ii) the combination of its
-        Contributions with other software (except as part of its Contributor
-        Version); or
-
-     c. under Patent Claims infringed by Covered Software in the absence of
-        its Contributions.
-
-     This License does not grant any rights in the trademarks, service marks,
-     or logos of any Contributor (except as may be necessary to comply with
-     the notice requirements in Section 3.4).
-
-2.4. Subsequent Licenses
-
-     No Contributor makes additional grants as a result of Your choice to
-     distribute the Covered Software under a subsequent version of this
-     License (see Section 10.2) or under the terms of a Secondary License (if
-     permitted under the terms of Section 3.3).
-
-2.5. Representation
-
-     Each Contributor represents that the Contributor believes its
-     Contributions are its original creation(s) or it has sufficient rights to
-     grant the rights to its Contributions conveyed by this License.
-
-2.6. Fair Use
-
-     This License is not intended to limit any rights You have under
-     applicable copyright doctrines of fair use, fair dealing, or other
-     equivalents.
-
-2.7. Conditions
-
-     Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in
-     Section 2.1.
-
-
-3. Responsibilities
-
-3.1. Distribution of Source Form
-
-     All distribution of Covered Software in Source Code Form, including any
-     Modifications that You create or to which You contribute, must be under
-     the terms of this License. You must inform recipients that the Source
-     Code Form of the Covered Software is governed by the terms of this
-     License, and how they can obtain a copy of this License. You may not
-     attempt to alter or restrict the recipients' rights in the Source Code
-     Form.
-
-3.2. Distribution of Executable Form
-
-     If You distribute Covered Software in Executable Form then:
-
-     a. such Covered Software must also be made available in Source Code Form,
-        as described in Section 3.1, and You must inform recipients of the
-        Executable Form how they can obtain a copy of such Source Code Form by
-        reasonable means in a timely manner, at a charge no more than the cost
-        of distribution to the recipient; and
-
-     b. You may distribute such Executable Form under the terms of this
-        License, or sublicense it under different terms, provided that the
-        license for the Executable Form does not attempt to limit or alter the
-        recipients' rights in the Source Code Form under this License.
-
-3.3. Distribution of a Larger Work
-
-     You may create and distribute a Larger Work under terms of Your choice,
-     provided that You also comply with the requirements of this License for
-     the Covered Software. If the Larger Work is a combination of Covered
-     Software with a work governed by one or more Secondary Licenses, and the
-     Covered Software is not Incompatible With Secondary Licenses, this
-     License permits You to additionally distribute such Covered Software
-     under the terms of such Secondary License(s), so that the recipient of
-     the Larger Work may, at their option, further distribute the Covered
-     Software under the terms of either this License or such Secondary
-     License(s).
-
-3.4. Notices
-
-     You may not remove or alter the substance of any license notices
-     (including copyright notices, patent notices, disclaimers of warranty, or
-     limitations of liability) contained within the Source Code Form of the
-     Covered Software, except that You may alter any license notices to the
-     extent required to remedy known factual inaccuracies.
-
-3.5. Application of Additional Terms
-
-     You may choose to offer, and to charge a fee for, warranty, support,
-     indemnity or liability obligations to one or more recipients of Covered
-     Software. However, You may do so only on Your own behalf, and not on
-     behalf of any Contributor. You must make it absolutely clear that any
-     such warranty, support, indemnity, or liability obligation is offered by
-     You alone, and You hereby agree to indemnify every Contributor for any
-     liability incurred by such Contributor as a result of warranty, support,
-     indemnity or liability terms You offer. You may include additional
-     disclaimers of warranty and limitations of liability specific to any
-     jurisdiction.
-
-4. Inability to Comply Due to Statute or Regulation
-
-   If it is impossible for You to comply with any of the terms of this License
-   with respect to some or all of the Covered Software due to statute,
-   judicial order, or regulation then You must: (a) comply with the terms of
-   this License to the maximum extent possible; and (b) describe the
-   limitations and the code they affect. Such description must be placed in a
-   text file included with all distributions of the Covered Software under
-   this License. Except to the extent prohibited by statute or regulation,
-   such description must be sufficiently detailed for a recipient of ordinary
-   skill to be able to understand it.
-
-5. Termination
-
-5.1. The rights granted under this License will terminate automatically if You
-     fail to comply with any of its terms. However, if You become compliant,
-     then the rights granted under this License from a particular Contributor
-     are reinstated (a) provisionally, unless and until such Contributor
-     explicitly and finally terminates Your grants, and (b) on an ongoing
-     basis, if such Contributor fails to notify You of the non-compliance by
-     some reasonable means prior to 60 days after You have come back into
-     compliance. Moreover, Your grants from a particular Contributor are
-     reinstated on an ongoing basis if such Contributor notifies You of the
-     non-compliance by some reasonable means, this is the first time You have
-     received notice of non-compliance with this License from such
-     Contributor, and You become compliant prior to 30 days after Your receipt
-     of the notice.
-
-5.2. If You initiate litigation against any entity by asserting a patent
-     infringement claim (excluding declaratory judgment actions,
-     counter-claims, and cross-claims) alleging that a Contributor Version
-     directly or indirectly infringes any patent, then the rights granted to
-     You by any and all Contributors for the Covered Software under Section
-     2.1 of this License shall terminate.
-
-5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user
-     license agreements (excluding distributors and resellers) which have been
-     validly granted by You or Your distributors under this License prior to
-     termination shall survive termination.
-
-6. Disclaimer of Warranty
-
-   Covered Software is provided under this License on an "as is" basis,
-   without warranty of any kind, either expressed, implied, or statutory,
-   including, without limitation, warranties that the Covered Software is free
-   of defects, merchantable, fit for a particular purpose or non-infringing.
-   The entire risk as to the quality and performance of the Covered Software
-   is with You. Should any Covered Software prove defective in any respect,
-   You (not any Contributor) assume the cost of any necessary servicing,
-   repair, or correction. This disclaimer of warranty constitutes an essential
-   part of this License. No use of  any Covered Software is authorized under
-   this License except under this disclaimer.
-
-7. Limitation of Liability
-
-   Under no circumstances and under no legal theory, whether tort (including
-   negligence), contract, or otherwise, shall any Contributor, or anyone who
-   distributes Covered Software as permitted above, be liable to You for any
-   direct, indirect, special, incidental, or consequential damages of any
-   character including, without limitation, damages for lost profits, loss of
-   goodwill, work stoppage, computer failure or malfunction, or any and all
-   other commercial damages or losses, even if such party shall have been
-   informed of the possibility of such damages. This limitation of liability
-   shall not apply to liability for death or personal injury resulting from
-   such party's negligence to the extent applicable law prohibits such
-   limitation. Some jurisdictions do not allow the exclusion or limitation of
-   incidental or consequential damages, so this exclusion and limitation may
-   not apply to You.
-
-8. Litigation
-
-   Any litigation relating to this License may be brought only in the courts
-   of a jurisdiction where the defendant maintains its principal place of
-   business and such litigation shall be governed by laws of that
-   jurisdiction, without reference to its conflict-of-law provisions. Nothing
-   in this Section shall prevent a party's ability to bring cross-claims or
-   counter-claims.
-
-9. Miscellaneous
-
-   This License represents the complete agreement concerning the subject
-   matter hereof. If any provision of this License is held to be
-   unenforceable, such provision shall be reformed only to the extent
-   necessary to make it enforceable. Any law or regulation which provides that
-   the language of a contract shall be construed against the drafter shall not
-   be used to construe this License against a Contributor.
-
-
 10. Versions of the License

 10.1. New Versions
@@ -16499,12 +16129,6 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI

 ================================================================

-github.com/mattn/go-localereader
-https://github.com/mattn/go-localereader
----------------------------------------------------------------
-All rights reserved proprietary
-================================================================
-
 github.com/mattn/go-runewidth
 https://github.com/mattn/go-runewidth
 ----------------------------------------------------------------
@@ -16801,12 +16425,6 @@ SOFTWARE.

 ================================================================

-github.com/minio/colorjson
-https://github.com/minio/colorjson
----------------------------------------------------------------
-All rights reserved proprietary
-================================================================
-
 github.com/minio/console
 https://github.com/minio/console
 ----------------------------------------------------------------
@@ -17474,12 +17092,6 @@ For more information on this, and how to apply and follow the GNU AGPL, see

 ================================================================

-github.com/minio/csvparser
-https://github.com/minio/csvparser
----------------------------------------------------------------
-All rights reserved proprietary
-================================================================
-
 github.com/minio/dnscache
 https://github.com/minio/dnscache
 ----------------------------------------------------------------
@@ -18175,12 +17787,6 @@ For more information on this, and how to apply and follow the GNU AGPL, see

 ================================================================

-github.com/minio/filepath
-https://github.com/minio/filepath
----------------------------------------------------------------
-All rights reserved proprietary
-================================================================
-
 github.com/minio/highwayhash
 https://github.com/minio/highwayhash
 ----------------------------------------------------------------
@@ -101,7 +101,7 @@ function fail() {
 }

 function check_online() {
-	if ! grep -q 'Status:' ${WORK_DIR}/dist-minio-*.log; then
+	if ! grep -q 'API:' ${WORK_DIR}/dist-minio-*.log; then
 		echo "1"
 	fi
 }
@@ -78,7 +78,7 @@ function start_minio_3_node() {
 }

 function check_heal() {
-	if ! grep -q 'Status:' ${WORK_DIR}/dist-minio-*.log; then
+	if ! grep -q 'API:' ${WORK_DIR}/dist-minio-*.log; then
 		return 1
 	fi

@@ -552,7 +552,7 @@ func (a adminAPIHandlers) ListAccessKeysLDAPBulk(w http.ResponseWriter, r *http.
 		dnList = append(dnList, selfDN)
 	}

-	accessKeyMap := make(map[string]madmin.ListAccessKeysLDAPResp)
+	var ldapUserList []string
 	if isAll {
 		ldapUsers, err := globalIAMSys.ListLDAPUsers(ctx)
 		if err != nil {
@@ -560,7 +560,7 @@ func (a adminAPIHandlers) ListAccessKeysLDAPBulk(w http.ResponseWriter, r *http.
 			return
 		}
 		for user := range ldapUsers {
-			accessKeyMap[user] = madmin.ListAccessKeysLDAPResp{}
+			ldapUserList = append(ldapUserList, user)
 		}
 	} else {
 		for _, userDN := range dnList {
@@ -573,7 +573,7 @@ func (a adminAPIHandlers) ListAccessKeysLDAPBulk(w http.ResponseWriter, r *http.
 			if foundResult == nil {
 				continue
 			}
-			accessKeyMap[foundResult.NormDN] = madmin.ListAccessKeysLDAPResp{}
+			ldapUserList = append(ldapUserList, foundResult.NormDN)
 		}
 	}

@@ -598,9 +598,12 @@ func (a adminAPIHandlers) ListAccessKeysLDAPBulk(w http.ResponseWriter, r *http.
 		return
 	}

-	for dn, accessKeys := range accessKeyMap {
+	accessKeyMap := make(map[string]madmin.ListAccessKeysLDAPResp)
+	for _, internalDN := range ldapUserList {
+		externalDN := globalIAMSys.LDAPConfig.DecodeDN(internalDN)
+		accessKeys := madmin.ListAccessKeysLDAPResp{}
 		if listSTSKeys {
-			stsKeys, err := globalIAMSys.ListSTSAccounts(ctx, dn)
+			stsKeys, err := globalIAMSys.ListSTSAccounts(ctx, internalDN)
 			if err != nil {
 				writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
 				return
@@ -614,13 +617,12 @@ func (a adminAPIHandlers) ListAccessKeysLDAPBulk(w http.ResponseWriter, r *http.
 			}
 			// if only STS keys, skip if user has no STS keys
 			if !listServiceAccounts && len(stsKeys) == 0 {
-				delete(accessKeyMap, dn)
 				continue
 			}
 		}

 		if listServiceAccounts {
-			serviceAccounts, err := globalIAMSys.ListServiceAccounts(ctx, dn)
+			serviceAccounts, err := globalIAMSys.ListServiceAccounts(ctx, internalDN)
 			if err != nil {
 				writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
 				return
@@ -634,11 +636,10 @@ func (a adminAPIHandlers) ListAccessKeysLDAPBulk(w http.ResponseWriter, r *http.
 			}
 			// if only service accounts, skip if user has no service accounts
 			if !listSTSKeys && len(serviceAccounts) == 0 {
-				delete(accessKeyMap, dn)
 				continue
 			}
 		}
-		accessKeyMap[dn] = accessKeys
+		accessKeyMap[externalDN] = accessKeys
 	}

 	data, err := json.Marshal(accessKeyMap)
@@ -2308,7 +2308,7 @@ func (a adminAPIHandlers) ImportIAM(w http.ResponseWriter, r *http.Request) {
 					// clean import.
 					err := globalIAMSys.DeleteServiceAccount(ctx, svcAcctReq.AccessKey, true)
 					if err != nil {
-						delErr := fmt.Errorf("failed to delete existing service account(%s) before importing it: %w", svcAcctReq.AccessKey, err)
+						delErr := fmt.Errorf("failed to delete existing service account (%s) before importing it: %w", svcAcctReq.AccessKey, err)
 						writeErrorResponseJSON(ctx, w, importError(ctx, delErr, allSvcAcctsFile, user), r.URL)
 						return
 					}
@@ -848,9 +848,10 @@ func (a adminAPIHandlers) DataUsageInfoHandler(w http.ResponseWriter, r *http.Re
 }

 func lriToLockEntry(l lockRequesterInfo, now time.Time, resource, server string) *madmin.LockEntry {
+	t := time.Unix(0, l.Timestamp)
 	entry := &madmin.LockEntry{
-		Timestamp:  l.Timestamp,
-		Elapsed:    now.Sub(l.Timestamp),
+		Timestamp:  t,
+		Elapsed:    now.Sub(t),
 		Resource:   resource,
 		ServerList: []string{server},
 		Source:     l.Source,
@@ -2186,7 +2187,7 @@ func (a adminAPIHandlers) KMSCreateKeyHandler(w http.ResponseWriter, r *http.Req
 	writeSuccessResponseHeadersOnly(w)
 }

-// KMSKeyStatusHandler - GET /minio/admin/v3/kms/status
+// KMSStatusHandler - GET /minio/admin/v3/kms/status
 func (a adminAPIHandlers) KMSStatusHandler(w http.ResponseWriter, r *http.Request) {
 	ctx := r.Context()

@@ -463,6 +463,7 @@ func TestTopLockEntries(t *testing.T) {
 			Owner:      lri.Owner,
 			ID:         lri.UID,
 			Quorum:     lri.Quorum,
+			Timestamp:  time.Unix(0, lri.Timestamp),
 		})
 	}

@@ -761,6 +761,15 @@ func (h *healSequence) queueHealTask(source healSource, healType madmin.HealItem
 		return nil
 	}

+	countOKDrives := func(drives []madmin.HealDriveInfo) (count int) {
+		for _, drive := range drives {
+			if drive.State == madmin.DriveStateOk {
+				count++
+			}
+		}
+		return count
+	}
+
 	// task queued, now wait for the response.
 	select {
 	case res := <-task.respCh:
@@ -781,6 +790,11 @@ func (h *healSequence) queueHealTask(source healSource, healType madmin.HealItem
 		if res.err != nil {
 			res.result.Detail = res.err.Error()
 		}
+		if res.result.ParityBlocks > 0 && res.result.DataBlocks > 0 && res.result.DataBlocks > res.result.ParityBlocks {
+			if got := countOKDrives(res.result.After.Drives); got < res.result.ParityBlocks {
+				res.result.Detail = fmt.Sprintf("quorum loss - expected %d minimum, got drive states in OK %d", res.result.ParityBlocks, got)
+			}
+		}
 		return h.pushHealResultItem(res.result)
 	case <-h.ctx.Done():
 		return nil
@@ -362,7 +362,7 @@ func initAutoHeal(ctx context.Context, objAPI ObjectLayer) {

 func getLocalDisksToHeal() (disksToHeal Endpoints) {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()
 	for _, disk := range localDrives {
 		_, err := disk.DiskInfo(context.Background(), DiskInfoOptions{})
@@ -36,6 +36,7 @@ import (
 	"github.com/minio/pkg/v3/env"
 	"github.com/minio/pkg/v3/wildcard"
 	"github.com/minio/pkg/v3/workers"
+	"github.com/minio/pkg/v3/xtime"
 	"gopkg.in/yaml.v3"
 )

@@ -116,7 +117,7 @@ func (p BatchJobExpirePurge) Validate() error {
 // BatchJobExpireFilter holds all the filters currently supported for batch replication
 type BatchJobExpireFilter struct {
 	line, col     int
-	OlderThan     time.Duration       `yaml:"olderThan,omitempty" json:"olderThan"`
+	OlderThan     xtime.Duration      `yaml:"olderThan,omitempty" json:"olderThan"`
 	CreatedBefore *time.Time          `yaml:"createdBefore,omitempty" json:"createdBefore"`
 	Tags          []BatchJobKV        `yaml:"tags,omitempty" json:"tags"`
 	Metadata      []BatchJobKV        `yaml:"metadata,omitempty" json:"metadata"`
@@ -162,7 +163,7 @@ func (ef BatchJobExpireFilter) Matches(obj ObjectInfo, now time.Time) bool {
 	if len(ef.Name) > 0 && !wildcard.Match(ef.Name, obj.Name) {
 		return false
 	}
-	if ef.OlderThan > 0 && now.Sub(obj.ModTime) <= ef.OlderThan {
+	if ef.OlderThan > 0 && now.Sub(obj.ModTime) <= ef.OlderThan.D() {
 		return false
 	}

@@ -552,22 +553,25 @@ func (r *BatchJobExpire) Start(ctx context.Context, api ObjectLayer, job BatchJo
 	go func() {
 		saveTicker := time.NewTicker(10 * time.Second)
 		defer saveTicker.Stop()
-		for {
+		quit := false
+		after := time.Minute
+		for !quit {
 			select {
 			case <-saveTicker.C:
-				// persist in-memory state to disk after every 10secs.
-				batchLogIf(ctx, ri.updateAfter(ctx, api, 10*time.Second, job))
-
 			case <-ctx.Done():
-				// persist in-memory state immediately before exiting due to context cancellation.
-				batchLogIf(ctx, ri.updateAfter(ctx, api, 0, job))
-				return
-
+				quit = true
 			case <-saverQuitCh:
-				// persist in-memory state immediately to disk.
-				batchLogIf(ctx, ri.updateAfter(ctx, api, 0, job))
-				return
+				quit = true
 			}
+
+			if quit {
+				// save immediately if we are quitting
+				after = 0
+			}
+
+			ctx, cancel := context.WithTimeout(GlobalContext, 30*time.Second) // independent context
+			batchLogIf(ctx, ri.updateAfter(ctx, api, after, job))
+			cancel()
 		}
 	}()

@@ -584,7 +588,7 @@ func (r *BatchJobExpire) Start(ctx context.Context, api ObjectLayer, job BatchJo
 		versionsCount int
 		toDel         []expireObjInfo
 	)
-	failed := true
+	failed := false
 	for result := range results {
 		if result.Err != nil {
 			failed = true
@@ -306,7 +306,7 @@ func (z *BatchJobExpireFilter) DecodeMsg(dc *msgp.Reader) (err error) {
 		}
 		switch msgp.UnsafeString(field) {
 		case "OlderThan":
-			z.OlderThan, err = dc.ReadDuration()
+			err = z.OlderThan.DecodeMsg(dc)
 			if err != nil {
 				err = msgp.WrapError(err, "OlderThan")
 				return
@@ -433,7 +433,7 @@ func (z *BatchJobExpireFilter) EncodeMsg(en *msgp.Writer) (err error) {
 	if err != nil {
 		return
 	}
-	err = en.WriteDuration(z.OlderThan)
+	err = z.OlderThan.EncodeMsg(en)
 	if err != nil {
 		err = msgp.WrapError(err, "OlderThan")
 		return
@@ -544,7 +544,11 @@ func (z *BatchJobExpireFilter) MarshalMsg(b []byte) (o []byte, err error) {
 	// map header, size 8
 	// string "OlderThan"
 	o = append(o, 0x88, 0xa9, 0x4f, 0x6c, 0x64, 0x65, 0x72, 0x54, 0x68, 0x61, 0x6e)
-	o = msgp.AppendDuration(o, z.OlderThan)
+	o, err = z.OlderThan.MarshalMsg(o)
+	if err != nil {
+		err = msgp.WrapError(err, "OlderThan")
+		return
+	}
 	// string "CreatedBefore"
 	o = append(o, 0xad, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x42, 0x65, 0x66, 0x6f, 0x72, 0x65)
 	if z.CreatedBefore == nil {
@@ -613,7 +617,7 @@ func (z *BatchJobExpireFilter) UnmarshalMsg(bts []byte) (o []byte, err error) {
 		}
 		switch msgp.UnsafeString(field) {
 		case "OlderThan":
-			z.OlderThan, bts, err = msgp.ReadDurationBytes(bts)
+			bts, err = z.OlderThan.UnmarshalMsg(bts)
 			if err != nil {
 				err = msgp.WrapError(err, "OlderThan")
 				return
@@ -734,7 +738,7 @@ func (z *BatchJobExpireFilter) UnmarshalMsg(bts []byte) (o []byte, err error) {

 // Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
 func (z *BatchJobExpireFilter) Msgsize() (s int) {
-	s = 1 + 10 + msgp.DurationSize + 14
+	s = 1 + 10 + z.OlderThan.Msgsize() + 14
 	if z.CreatedBefore == nil {
 		s += msgp.NilSize
 	} else {
@@ -20,7 +20,7 @@ package cmd
 import (
 	"testing"

-	"gopkg.in/yaml.v2"
+	"gopkg.in/yaml.v3"
 )

 func TestParseBatchJobExpire(t *testing.T) {
@@ -32,7 +32,7 @@ expire: # Expire objects that match a condition
  rules:
    - type: object  # regular objects with zero or more older versions
      name: NAME # match object names that satisfy the wildcard expression.
-      olderThan: 70h # match objects older than this value
+      olderThan: 7d10h # match objects older than this value
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      tags:
        - key: name
@@ -64,7 +64,7 @@ expire: # Expire objects that match a condition
    delay: 500ms # least amount of delay between each retry
 `
 	var job BatchJobRequest
-	err := yaml.UnmarshalStrict([]byte(expireYaml), &job)
+	err := yaml.Unmarshal([]byte(expireYaml), &job)
 	if err != nil {
 		t.Fatal("Failed to parse batch-job-expire yaml", err)
 	}
@@ -287,12 +287,12 @@ func (r *BatchJobReplicateV1) StartFromSource(ctx context.Context, api ObjectLay
 	isStorageClassOnly := len(r.Flags.Filter.Metadata) == 1 && strings.EqualFold(r.Flags.Filter.Metadata[0].Key, xhttp.AmzStorageClass)

 	skip := func(oi ObjectInfo) (ok bool) {
-		if r.Flags.Filter.OlderThan > 0 && time.Since(oi.ModTime) < r.Flags.Filter.OlderThan {
+		if r.Flags.Filter.OlderThan > 0 && time.Since(oi.ModTime) < r.Flags.Filter.OlderThan.D() {
 			// skip all objects that are newer than specified older duration
 			return true
 		}

-		if r.Flags.Filter.NewerThan > 0 && time.Since(oi.ModTime) >= r.Flags.Filter.NewerThan {
+		if r.Flags.Filter.NewerThan > 0 && time.Since(oi.ModTime) >= r.Flags.Filter.NewerThan.D() {
 			// skip all objects that are older than specified newer duration
 			return true
 		}
@@ -540,7 +540,7 @@ func toObjectInfo(bucket, object string, objInfo miniogo.ObjectInfo) ObjectInfo
 	return oi
 }

-func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLayer, remoteClnt *minio.Client, entries []ObjectInfo) error {
+func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLayer, remoteClnt *minio.Client, entries []ObjectInfo, prefix string) error {
 	input := make(chan minio.SnowballObject, 1)
 	opts := minio.SnowballOptions{
 		Opts:     minio.PutObjectOptions{},
@@ -562,6 +562,10 @@ func (r BatchJobReplicateV1) writeAsArchive(ctx context.Context, objAPI ObjectLa
 				continue
 			}

+			if prefix != "" {
+				entry.Name = pathJoin(prefix, entry.Name)
+			}
+
 			snowballObj := minio.SnowballObject{
 				// Create path to store objects within the bucket.
 				Key:       entry.Name,
@@ -725,7 +729,7 @@ const (

 	batchReplJobAPIVersion        = "v1"
 	batchReplJobDefaultRetries    = 3
-	batchReplJobDefaultRetryDelay = 250 * time.Millisecond
+	batchReplJobDefaultRetryDelay = time.Second
 )

 func getJobPath(job BatchJobRequest) string {
@@ -1016,18 +1020,18 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 	lastObject := ri.Object

 	delay := job.Replicate.Flags.Retry.Delay
-	if delay == 0 {
+	if delay < time.Second {
 		delay = batchReplJobDefaultRetryDelay
 	}
 	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

 	selectObj := func(info FileInfo) (ok bool) {
-		if r.Flags.Filter.OlderThan > 0 && time.Since(info.ModTime) < r.Flags.Filter.OlderThan {
+		if r.Flags.Filter.OlderThan > 0 && time.Since(info.ModTime) < r.Flags.Filter.OlderThan.D() {
 			// skip all objects that are newer than specified older duration
 			return false
 		}

-		if r.Flags.Filter.NewerThan > 0 && time.Since(info.ModTime) >= r.Flags.Filter.NewerThan {
+		if r.Flags.Filter.NewerThan > 0 && time.Since(info.ModTime) >= r.Flags.Filter.NewerThan.D() {
 			// skip all objects that are older than specified newer duration
 			return false
 		}
@@ -1115,7 +1119,7 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 			slowCh = make(chan itemOrErr[ObjectInfo], 100)
 		)

-		if !*r.Source.Snowball.Disable && r.Source.Type.isMinio() && r.Target.Type.isMinio() {
+		if r.Source.Snowball.Disable != nil && !*r.Source.Snowball.Disable && r.Source.Type.isMinio() && r.Target.Type.isMinio() {
 			go func() {
 				// Snowball currently needs the high level minio-go Client, not the Core one
 				cl, err := miniogo.New(u.Host, &miniogo.Options{
@@ -1125,7 +1129,7 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 					BucketLookup: lookupStyle(r.Target.Path),
 				})
 				if err != nil {
-					batchLogIf(ctx, err)
+					batchLogOnceIf(ctx, err, job.ID+"miniogo.New")
 					return
 				}

@@ -1135,8 +1139,8 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 				batch := make([]ObjectInfo, 0, *r.Source.Snowball.Batch)
 				writeFn := func(batch []ObjectInfo) {
 					if len(batch) > 0 {
-						if err := r.writeAsArchive(ctx, api, cl, batch); err != nil {
-							batchLogIf(ctx, err)
+						if err := r.writeAsArchive(ctx, api, cl, batch, r.Target.Prefix); err != nil {
+							batchLogOnceIf(ctx, err, job.ID+"writeAsArchive")
 							for _, b := range batch {
 								slowCh <- itemOrErr[ObjectInfo]{Item: b}
 							}
@@ -1144,7 +1148,7 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 							ri.trackCurrentBucketBatch(r.Source.Bucket, batch)
 							globalBatchJobsMetrics.save(job.ID, ri)
 							// persist in-memory state to disk after every 10secs.
-							batchLogIf(ctx, ri.updateAfter(ctx, api, 10*time.Second, job))
+							batchLogOnceIf(ctx, ri.updateAfter(ctx, api, 10*time.Second, job), job.ID+"updateAfter")
 						}
 					}
 				}
@@ -1204,7 +1208,7 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 		for res := range slowCh {
 			if res.Err != nil {
 				ri.Failed = true
-				batchLogIf(ctx, res.Err)
+				batchLogOnceIf(ctx, res.Err, job.ID+"res.Err")
 				continue
 			}
 			result := res.Item
@@ -1231,7 +1235,7 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 						return
 					}
 					stopFn(result, err)
-					batchLogIf(ctx, err)
+					batchLogOnceIf(ctx, err, job.ID+"ReplicateToTarget")
 					success = false
 				} else {
 					stopFn(result, nil)
@@ -1239,7 +1243,7 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 				ri.trackCurrentBucketObject(r.Source.Bucket, result, success, attempts)
 				globalBatchJobsMetrics.save(job.ID, ri)
 				// persist in-memory state to disk after every 10secs.
-				batchLogIf(ctx, ri.updateAfter(ctx, api, 10*time.Second, job))
+				batchLogOnceIf(ctx, ri.updateAfter(ctx, api, 10*time.Second, job), job.ID+"updateAfter2")

 				if wait := globalBatchConfig.ReplicationWait(); wait > 0 {
 					time.Sleep(wait)
@@ -1254,10 +1258,10 @@ func (r *BatchJobReplicateV1) Start(ctx context.Context, api ObjectLayer, job Ba

 		globalBatchJobsMetrics.save(job.ID, ri)
 		// persist in-memory state to disk.
-		batchLogIf(ctx, ri.updateAfter(ctx, api, 0, job))
+		batchLogOnceIf(ctx, ri.updateAfter(ctx, api, 0, job), job.ID+"updateAfter3")

 		if err := r.Notify(ctx, ri); err != nil {
-			batchLogIf(ctx, fmt.Errorf("unable to notify %v", err))
+			batchLogOnceIf(ctx, fmt.Errorf("unable to notify %v", err), job.ID+"notify")
 		}

 		cancel()
@@ -1567,6 +1571,9 @@ func (a adminAPIHandlers) ListBatchJobs(w http.ResponseWriter, r *http.Request)
 			writeErrorResponseJSON(ctx, w, toAPIError(ctx, result.Err), r.URL)
 			return
 		}
+		if strings.HasPrefix(result.Item.Name, batchJobReportsPrefix+slashSeparator) {
+			continue
+		}
 		req := &BatchJobRequest{}
 		if err := req.load(ctx, objectAPI, result.Item.Name); err != nil {
 			if !errors.Is(err, errNoSuchJob) {
@@ -1883,6 +1890,9 @@ func (j *BatchJobPool) resume(randomWait func() time.Duration) {
 			batchLogIf(j.ctx, result.Err)
 			continue
 		}
+		if strings.HasPrefix(result.Item.Name, batchJobReportsPrefix+slashSeparator) {
+			continue
+		}
 		// ignore batch-replicate.bin and batch-rotate.bin entries
 		if strings.HasSuffix(result.Item.Name, slashSeparator) {
 			continue
@@ -21,8 +21,8 @@ import (
 	"time"

 	miniogo "github.com/minio/minio-go/v7"
-
 	"github.com/minio/minio/internal/auth"
+	"github.com/minio/pkg/v3/xtime"
 )

 //go:generate msgp -file $GOFILE
@@ -65,12 +65,12 @@ import (

 // BatchReplicateFilter holds all the filters currently supported for batch replication
 type BatchReplicateFilter struct {
-	NewerThan     time.Duration `yaml:"newerThan,omitempty" json:"newerThan"`
-	OlderThan     time.Duration `yaml:"olderThan,omitempty" json:"olderThan"`
-	CreatedAfter  time.Time     `yaml:"createdAfter,omitempty" json:"createdAfter"`
-	CreatedBefore time.Time     `yaml:"createdBefore,omitempty" json:"createdBefore"`
-	Tags          []BatchJobKV  `yaml:"tags,omitempty" json:"tags"`
-	Metadata      []BatchJobKV  `yaml:"metadata,omitempty" json:"metadata"`
+	NewerThan     xtime.Duration `yaml:"newerThan,omitempty" json:"newerThan"`
+	OlderThan     xtime.Duration `yaml:"olderThan,omitempty" json:"olderThan"`
+	CreatedAfter  time.Time      `yaml:"createdAfter,omitempty" json:"createdAfter"`
+	CreatedBefore time.Time      `yaml:"createdBefore,omitempty" json:"createdBefore"`
+	Tags          []BatchJobKV   `yaml:"tags,omitempty" json:"tags"`
+	Metadata      []BatchJobKV   `yaml:"metadata,omitempty" json:"metadata"`
 }

 // BatchJobReplicateFlags various configurations for replication job definition currently includes
@@ -1409,13 +1409,13 @@ func (z *BatchReplicateFilter) DecodeMsg(dc *msgp.Reader) (err error) {
 		}
 		switch msgp.UnsafeString(field) {
 		case "NewerThan":
-			z.NewerThan, err = dc.ReadDuration()
+			err = z.NewerThan.DecodeMsg(dc)
 			if err != nil {
 				err = msgp.WrapError(err, "NewerThan")
 				return
 			}
 		case "OlderThan":
-			z.OlderThan, err = dc.ReadDuration()
+			err = z.OlderThan.DecodeMsg(dc)
 			if err != nil {
 				err = msgp.WrapError(err, "OlderThan")
 				return
@@ -1489,7 +1489,7 @@ func (z *BatchReplicateFilter) EncodeMsg(en *msgp.Writer) (err error) {
 	if err != nil {
 		return
 	}
-	err = en.WriteDuration(z.NewerThan)
+	err = z.NewerThan.EncodeMsg(en)
 	if err != nil {
 		err = msgp.WrapError(err, "NewerThan")
 		return
@@ -1499,7 +1499,7 @@ func (z *BatchReplicateFilter) EncodeMsg(en *msgp.Writer) (err error) {
 	if err != nil {
 		return
 	}
-	err = en.WriteDuration(z.OlderThan)
+	err = z.OlderThan.EncodeMsg(en)
 	if err != nil {
 		err = msgp.WrapError(err, "OlderThan")
 		return
@@ -1567,10 +1567,18 @@ func (z *BatchReplicateFilter) MarshalMsg(b []byte) (o []byte, err error) {
 	// map header, size 6
 	// string "NewerThan"
 	o = append(o, 0x86, 0xa9, 0x4e, 0x65, 0x77, 0x65, 0x72, 0x54, 0x68, 0x61, 0x6e)
-	o = msgp.AppendDuration(o, z.NewerThan)
+	o, err = z.NewerThan.MarshalMsg(o)
+	if err != nil {
+		err = msgp.WrapError(err, "NewerThan")
+		return
+	}
 	// string "OlderThan"
 	o = append(o, 0xa9, 0x4f, 0x6c, 0x64, 0x65, 0x72, 0x54, 0x68, 0x61, 0x6e)
-	o = msgp.AppendDuration(o, z.OlderThan)
+	o, err = z.OlderThan.MarshalMsg(o)
+	if err != nil {
+		err = msgp.WrapError(err, "OlderThan")
+		return
+	}
 	// string "CreatedAfter"
 	o = append(o, 0xac, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x41, 0x66, 0x74, 0x65, 0x72)
 	o = msgp.AppendTime(o, z.CreatedAfter)
@@ -1619,13 +1627,13 @@ func (z *BatchReplicateFilter) UnmarshalMsg(bts []byte) (o []byte, err error) {
 		}
 		switch msgp.UnsafeString(field) {
 		case "NewerThan":
-			z.NewerThan, bts, err = msgp.ReadDurationBytes(bts)
+			bts, err = z.NewerThan.UnmarshalMsg(bts)
 			if err != nil {
 				err = msgp.WrapError(err, "NewerThan")
 				return
 			}
 		case "OlderThan":
-			z.OlderThan, bts, err = msgp.ReadDurationBytes(bts)
+			bts, err = z.OlderThan.UnmarshalMsg(bts)
 			if err != nil {
 				err = msgp.WrapError(err, "OlderThan")
 				return
@@ -1694,7 +1702,7 @@ func (z *BatchReplicateFilter) UnmarshalMsg(bts []byte) (o []byte, err error) {

 // Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
 func (z *BatchReplicateFilter) Msgsize() (s int) {
-	s = 1 + 10 + msgp.DurationSize + 10 + msgp.DurationSize + 13 + msgp.TimeSize + 14 + msgp.TimeSize + 5 + msgp.ArrayHeaderSize
+	s = 1 + 10 + z.NewerThan.Msgsize() + 10 + z.OlderThan.Msgsize() + 13 + msgp.TimeSize + 14 + msgp.TimeSize + 5 + msgp.ArrayHeaderSize
 	for za0001 := range z.Tags {
 		s += z.Tags[za0001].Msgsize()
 	}
@@ -0,0 +1,100 @@
+// Copyright (c) 2015-2024 MinIO, Inc.
+//
+// This file is part of MinIO Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"testing"
+
+	"gopkg.in/yaml.v3"
+)
+
+func TestParseBatchJobReplicate(t *testing.T) {
+	replicateYaml := `
+replicate:
+  apiVersion: v1
+  # source of the objects to be replicated
+  source:
+    type: minio # valid values are "s3" or "minio"
+    bucket: mytest
+    prefix: object-prefix1 # 'PREFIX' is optional
+    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted
+    # Either the 'source' or 'remote' *must* be the "local" deployment
+#    endpoint: "http://127.0.0.1:9000"
+#    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
+#    credentials:
+#      accessKey: minioadmin # Required
+#      secretKey: minioadmin # Required
+#    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used
+    snowball: # automatically activated if the source is local
+      disable: true # optionally turn-off snowball archive transfer
+#      batch: 100 # upto this many objects per archive
+#      inmemory: true # indicates if the archive must be staged locally or in-memory
+#      compress: false # S2/Snappy compressed archive
+#      smallerThan: 5MiB # create archive for all objects smaller than 5MiB
+#      skipErrs: false # skips any source side read() errors
+
+  # target where the objects must be replicated
+  target:
+    type: minio # valid values are "s3" or "minio"
+    bucket: mytest
+    prefix: stage # 'PREFIX' is optional
+    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted
+
+    # Either the 'source' or 'remote' *must* be the "local" deployment
+    endpoint: "http://127.0.0.1:9001"
+    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
+    credentials:
+      accessKey: minioadmin
+      secretKey: minioadmin
+    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used
+
+  # NOTE: All flags are optional
+  # - filtering criteria only applies for all source objects match the criteria
+  # - configurable notification endpoints
+  # - configurable retries for the job (each retry skips successfully previously replaced objects)
+  flags:
+    filter:
+      newerThan: "7d10h31s" # match objects newer than this value (e.g. 7d10h31s)
+      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
+#      createdAfter: "date" # match objects created after "date"
+#      createdBefore: "date" # match objects created before "date"
+
+      ## NOTE: tags are not supported when "source" is remote.
+      tags:
+         - key: "name"
+           value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
+
+      metadata:
+         - key: "content-type"
+           value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
+
+#    notify:
+#      endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
+#      token: "Bearer xxxxx" # optional authentication token for the notification endpoint
+#
+#    retry:
+#      attempts: 10 # number of retries for the job before giving up
+#      delay: "500ms" # least amount of delay between each retry
+
+`
+	var job BatchJobRequest
+	err := yaml.Unmarshal([]byte(replicateYaml), &job)
+	if err != nil {
+		t.Fatal("Failed to parse batch-job-replicate yaml", err)
+	}
+}
@@ -389,6 +389,17 @@ func (r *BatchJobKeyRotateV1) Start(ctx context.Context, api ObjectLayer, job Ba
 					stopFn(result, err)
 					batchLogIf(ctx, err)
 					success = false
+					if attempts >= retryAttempts {
+						auditOptions := AuditLogOptions{
+							Event:     "KeyRotate",
+							APIName:   "StartBatchJob",
+							Bucket:    result.Bucket,
+							Object:    result.Name,
+							VersionID: result.VersionID,
+							Error:     err.Error(),
+						}
+						auditLogInternal(ctx, auditOptions)
+					}
 				} else {
 					stopFn(result, nil)
 				}
@@ -75,7 +75,7 @@ func (api objectAPIHandlers) PutBucketReplicationConfigHandler(w http.ResponseWr
 		writeErrorResponse(ctx, w, apiErr, r.URL)
 		return
 	}
-	sameTarget, apiErr := validateReplicationDestination(ctx, bucket, replicationConfig, true)
+	sameTarget, apiErr := validateReplicationDestination(ctx, bucket, replicationConfig, &validateReplicationDestinationOptions{CheckRemoteBucket: true})
 	if apiErr != noError {
 		writeErrorResponse(ctx, w, apiErr, r.URL)
 		return
@@ -559,7 +559,7 @@ func (api objectAPIHandlers) ValidateBucketReplicationCredsHandler(w http.Respon
 		lockEnabled = lcfg.Enabled()
 	}

-	sameTarget, apiErr := validateReplicationDestination(ctx, bucket, replicationConfig, true)
+	sameTarget, apiErr := validateReplicationDestination(ctx, bucket, replicationConfig, &validateReplicationDestinationOptions{CheckRemoteBucket: true})
 	if apiErr != noError {
 		writeErrorResponse(ctx, w, apiErr, r.URL)
 		return
@@ -91,7 +91,10 @@ func getReplicationConfig(ctx context.Context, bucketName string) (rc *replicati

 // validateReplicationDestination returns error if replication destination bucket missing or not configured
 // It also returns true if replication destination is same as this server.
-func validateReplicationDestination(ctx context.Context, bucket string, rCfg *replication.Config, checkRemote bool) (bool, APIError) {
+func validateReplicationDestination(ctx context.Context, bucket string, rCfg *replication.Config, opts *validateReplicationDestinationOptions) (bool, APIError) {
+	if opts == nil {
+		opts = &validateReplicationDestinationOptions{}
+	}
 	var arns []string
 	if rCfg.RoleArn != "" {
 		arns = append(arns, rCfg.RoleArn)
@@ -113,7 +116,7 @@ func validateReplicationDestination(ctx context.Context, bucket string, rCfg *re
 		if clnt == nil {
 			return sameTarget, toAPIError(ctx, BucketRemoteTargetNotFound{Bucket: bucket})
 		}
-		if checkRemote { // validate remote bucket
+		if opts.CheckRemoteBucket { // validate remote bucket
 			found, err := clnt.BucketExists(ctx, arn.Bucket)
 			if err != nil {
 				return sameTarget, errorCodes.ToAPIErrWithErr(ErrRemoteDestinationNotFoundError, err)
@@ -133,23 +136,29 @@ func validateReplicationDestination(ctx context.Context, bucket string, rCfg *re
 				}
 			}
 		}
+		// if checked bucket, then check the ready is unnecessary
+		if !opts.CheckRemoteBucket && opts.CheckReady {
+			endpoint := clnt.EndpointURL().String()
+			if errInt, ok := opts.checkReadyErr.Load(endpoint); !ok {
+				err = checkRemoteEndpoint(ctx, clnt.EndpointURL())
+				opts.checkReadyErr.Store(endpoint, err)
+			} else {
+				if errInt == nil {
+					err = nil
+				} else {
+					err = errInt.(error)
+				}
+			}
+			switch err.(type) {
+			case BucketRemoteIdenticalToSource:
+				return true, errorCodes.ToAPIErrWithErr(ErrBucketRemoteIdenticalToSource, fmt.Errorf("remote target endpoint %s is self referential", clnt.EndpointURL().String()))
+			default:
+			}
+		}
 		// validate replication ARN against target endpoint
-		c := globalBucketTargetSys.GetRemoteTargetClient(bucket, arnStr)
-		if c != nil {
-			if err := checkRemoteEndpoint(ctx, c.EndpointURL()); err != nil {
-				switch err.(type) {
-				case BucketRemoteIdenticalToSource:
-					return true, errorCodes.ToAPIErrWithErr(ErrBucketRemoteIdenticalToSource, fmt.Errorf("remote target endpoint %s is self referential", c.EndpointURL().String()))
-				default:
-				}
-			}
-			if c.EndpointURL().String() == clnt.EndpointURL().String() {
-				selfTarget, _ := isLocalHost(clnt.EndpointURL().Hostname(), clnt.EndpointURL().Port(), globalMinioPort)
-				if !sameTarget {
-					sameTarget = selfTarget
-				}
-				continue
-			}
+		selfTarget, _ := isLocalHost(clnt.EndpointURL().Hostname(), clnt.EndpointURL().Port(), globalMinioPort)
+		if !sameTarget {
+			sameTarget = selfTarget
 		}
 	}

@@ -765,21 +774,6 @@ func (m caseInsensitiveMap) Lookup(key string) (string, bool) {
 	return "", false
 }

-func getCRCMeta(oi ObjectInfo, partNum int, h http.Header) map[string]string {
-	meta := make(map[string]string)
-	cs := oi.decryptChecksums(partNum, h)
-	for k, v := range cs {
-		cksum := hash.NewChecksumString(k, v)
-		if cksum == nil {
-			continue
-		}
-		if cksum.Valid() {
-			meta[cksum.Type.Key()] = v
-		}
-	}
-	return meta
-}
-
 func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo, partNum int) (putOpts minio.PutObjectOptions, err error) {
 	meta := make(map[string]string)
 	isSSEC := crypto.SSEC.IsEncrypted(objInfo.UserDefined)
@@ -788,11 +782,6 @@ func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo, part
 		// In case of SSE-C objects copy the allowed internal headers as well
 		if !isSSEC || !slices.Contains(maps.Keys(validSSEReplicationHeaders), k) {
 			if stringsHasPrefixFold(k, ReservedMetadataPrefixLower) {
-				if strings.EqualFold(k, ReservedMetadataPrefixLower+"crc") {
-					for k, v := range getCRCMeta(objInfo, partNum, nil) {
-						meta[k] = v
-					}
-				}
 				continue
 			}
 			if isStandardHeader(k) {
@@ -811,8 +800,12 @@ func putReplicationOpts(ctx context.Context, sc string, objInfo ObjectInfo, part
 		if isSSEC {
 			meta[ReplicationSsecChecksumHeader] = base64.StdEncoding.EncodeToString(objInfo.Checksum)
 		} else {
-			for k, v := range getCRCMeta(objInfo, 0, nil) {
-				meta[k] = v
+			for _, pi := range objInfo.Parts {
+				if pi.Number == partNum {
+					for k, v := range pi.Checksums {
+						meta[k] = v
+					}
+				}
 			}
 		}
 	}
@@ -1306,7 +1299,7 @@ func (ri ReplicateObjectInfo) replicateObject(ctx context.Context, objectAPI Obj
 		HeaderSize: headerSize,
 	}
 	newCtx := ctx
-	if globalBucketMonitor.IsThrottled(bucket, tgt.ARN) {
+	if globalBucketMonitor.IsThrottled(bucket, tgt.ARN) && objInfo.Size < minLargeObjSize {
 		var cancel context.CancelFunc
 		newCtx, cancel = context.WithTimeout(ctx, throttleDeadline)
 		defer cancel()
@@ -1581,7 +1574,7 @@ applyAction:
 			HeaderSize: headerSize,
 		}
 		newCtx := ctx
-		if globalBucketMonitor.IsThrottled(bucket, tgt.ARN) {
+		if globalBucketMonitor.IsThrottled(bucket, tgt.ARN) && objInfo.Size < minLargeObjSize {
 			var cancel context.CancelFunc
 			newCtx, cancel = context.WithTimeout(ctx, throttleDeadline)
 			defer cancel()
@@ -1666,8 +1659,7 @@ func replicateObjectWithMultipart(ctx context.Context, c *minio.Core, bucket, ob
 		cHeader := http.Header{}
 		cHeader.Add(xhttp.MinIOSourceReplicationRequest, "true")
 		if !isSSEC {
-			crc := getCRCMeta(objInfo, partInfo.Number, nil) // No SSE-C keys here.
-			for k, v := range crc {
+			for k, v := range partInfo.Checksums {
 				cHeader.Add(k, v)
 			}
 		}
@@ -1803,15 +1795,18 @@ var (
 type ReplicationPool struct {
 	// atomic ops:
 	activeWorkers    int32
+	activeLrgWorkers int32
 	activeMRFWorkers int32

-	objLayer   ObjectLayer
-	ctx        context.Context
-	priority   string
-	maxWorkers int
-	mu         sync.RWMutex
-	mrfMU      sync.Mutex
-	resyncer   *replicationResyncer
+	objLayer    ObjectLayer
+	ctx         context.Context
+	priority    string
+	maxWorkers  int
+	maxLWorkers int
+
+	mu       sync.RWMutex
+	mrfMU    sync.Mutex
+	resyncer *replicationResyncer

 	// workers:
 	workers    []chan ReplicationWorkerOperation
@@ -1882,9 +1877,13 @@ func NewReplicationPool(ctx context.Context, o ObjectLayer, opts replicationPool
 	if maxWorkers > 0 && failedWorkers > maxWorkers {
 		failedWorkers = maxWorkers
 	}
+	maxLWorkers := LargeWorkerCount
+	if opts.MaxLWorkers > 0 {
+		maxLWorkers = opts.MaxLWorkers
+	}
 	pool := &ReplicationPool{
 		workers:         make([]chan ReplicationWorkerOperation, 0, workers),
-		lrgworkers:      make([]chan ReplicationWorkerOperation, 0, LargeWorkerCount),
+		lrgworkers:      make([]chan ReplicationWorkerOperation, 0, maxLWorkers),
 		mrfReplicaCh:    make(chan ReplicationWorkerOperation, 100000),
 		mrfWorkerKillCh: make(chan struct{}, failedWorkers),
 		resyncer:        newresyncer(),
@@ -1894,9 +1893,10 @@ func NewReplicationPool(ctx context.Context, o ObjectLayer, opts replicationPool
 		objLayer:        o,
 		priority:        priority,
 		maxWorkers:      maxWorkers,
+		maxLWorkers:     maxLWorkers,
 	}

-	pool.AddLargeWorkers()
+	pool.ResizeLrgWorkers(maxLWorkers, 0)
 	pool.ResizeWorkers(workers, 0)
 	pool.ResizeFailedWorkers(failedWorkers)
 	go pool.resyncer.PersistToDisk(ctx, o)
@@ -1975,23 +1975,8 @@ func (p *ReplicationPool) AddWorker(input <-chan ReplicationWorkerOperation, opT
 	}
 }

-// AddLargeWorkers adds a static number of workers to handle large uploads
-func (p *ReplicationPool) AddLargeWorkers() {
-	for i := 0; i < LargeWorkerCount; i++ {
-		p.lrgworkers = append(p.lrgworkers, make(chan ReplicationWorkerOperation, 100000))
-		i := i
-		go p.AddLargeWorker(p.lrgworkers[i])
-	}
-	go func() {
-		<-p.ctx.Done()
-		for i := 0; i < LargeWorkerCount; i++ {
-			xioutil.SafeClose(p.lrgworkers[i])
-		}
-	}()
-}
-
 // AddLargeWorker adds a replication worker to the static pool for large uploads.
-func (p *ReplicationPool) AddLargeWorker(input <-chan ReplicationWorkerOperation) {
+func (p *ReplicationPool) AddLargeWorker(input <-chan ReplicationWorkerOperation, opTracker *int32) {
 	for {
 		select {
 		case <-p.ctx.Done():
@@ -2002,11 +1987,23 @@ func (p *ReplicationPool) AddLargeWorker(input <-chan ReplicationWorkerOperation
 			}
 			switch v := oi.(type) {
 			case ReplicateObjectInfo:
+				if opTracker != nil {
+					atomic.AddInt32(opTracker, 1)
+				}
 				globalReplicationStats.incQ(v.Bucket, v.Size, v.DeleteMarker, v.OpType)
 				replicateObject(p.ctx, v, p.objLayer)
 				globalReplicationStats.decQ(v.Bucket, v.Size, v.DeleteMarker, v.OpType)
+				if opTracker != nil {
+					atomic.AddInt32(opTracker, -1)
+				}
 			case DeletedObjectReplicationInfo:
+				if opTracker != nil {
+					atomic.AddInt32(opTracker, 1)
+				}
 				replicateDelete(p.ctx, v, p.objLayer)
+				if opTracker != nil {
+					atomic.AddInt32(opTracker, -1)
+				}
 			default:
 				bugLogIf(p.ctx, fmt.Errorf("unknown replication type: %T", oi), "unknown-replicate-type")
 			}
@@ -2014,6 +2011,30 @@ func (p *ReplicationPool) AddLargeWorker(input <-chan ReplicationWorkerOperation
 	}
 }

+// ResizeLrgWorkers sets replication workers pool for large transfers(>=128MiB) to new size.
+// checkOld can be set to an expected value.
+// If the worker count changed
+func (p *ReplicationPool) ResizeLrgWorkers(n, checkOld int) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	if (checkOld > 0 && len(p.lrgworkers) != checkOld) || n == len(p.lrgworkers) || n < 1 {
+		// Either already satisfied or worker count changed while we waited for the lock.
+		return
+	}
+	for len(p.lrgworkers) < n {
+		input := make(chan ReplicationWorkerOperation, 100000)
+		p.lrgworkers = append(p.lrgworkers, input)
+
+		go p.AddLargeWorker(input, &p.activeLrgWorkers)
+	}
+	for len(p.lrgworkers) > n {
+		worker := p.lrgworkers[len(p.lrgworkers)-1]
+		p.lrgworkers = p.lrgworkers[:len(p.lrgworkers)-1]
+		xioutil.SafeClose(worker)
+	}
+}
+
 // ActiveWorkers returns the number of active workers handling replication traffic.
 func (p *ReplicationPool) ActiveWorkers() int {
 	return int(atomic.LoadInt32(&p.activeWorkers))
@@ -2024,6 +2045,11 @@ func (p *ReplicationPool) ActiveMRFWorkers() int {
 	return int(atomic.LoadInt32(&p.activeMRFWorkers))
 }

+// ActiveLrgWorkers returns the number of active workers handling traffic > 128MiB object size.
+func (p *ReplicationPool) ActiveLrgWorkers() int {
+	return int(atomic.LoadInt32(&p.activeLrgWorkers))
+}
+
 // ResizeWorkers sets replication workers pool to new size.
 // checkOld can be set to an expected value.
 // If the worker count changed
@@ -2049,7 +2075,7 @@ func (p *ReplicationPool) ResizeWorkers(n, checkOld int) {
 }

 // ResizeWorkerPriority sets replication failed workers pool size
-func (p *ReplicationPool) ResizeWorkerPriority(pri string, maxWorkers int) {
+func (p *ReplicationPool) ResizeWorkerPriority(pri string, maxWorkers, maxLWorkers int) {
 	var workers, mrfWorkers int
 	p.mu.Lock()
 	switch pri {
@@ -2076,11 +2102,15 @@ func (p *ReplicationPool) ResizeWorkerPriority(pri string, maxWorkers int) {
 	if maxWorkers > 0 && mrfWorkers > maxWorkers {
 		mrfWorkers = maxWorkers
 	}
+	if maxLWorkers <= 0 {
+		maxLWorkers = LargeWorkerCount
+	}
 	p.priority = pri
 	p.maxWorkers = maxWorkers
 	p.mu.Unlock()
 	p.ResizeWorkers(workers, 0)
 	p.ResizeFailedWorkers(mrfWorkers)
+	p.ResizeLrgWorkers(maxLWorkers, 0)
 }

 // ResizeFailedWorkers sets replication failed workers pool size
@@ -2124,9 +2154,18 @@ func (p *ReplicationPool) queueReplicaTask(ri ReplicateObjectInfo) {
 		h := xxh3.HashString(ri.Bucket + ri.Name)
 		select {
 		case <-p.ctx.Done():
-		case p.lrgworkers[h%LargeWorkerCount] <- ri:
+		case p.lrgworkers[h%uint64(len(p.lrgworkers))] <- ri:
 		default:
 			globalReplicationPool.queueMRFSave(ri.ToMRFEntry())
+			p.mu.RLock()
+			maxLWorkers := p.maxLWorkers
+			existing := len(p.lrgworkers)
+			p.mu.RUnlock()
+			maxLWorkers = min(maxLWorkers, LargeWorkerCount)
+			if p.ActiveLrgWorkers() < maxLWorkers {
+				workers := min(existing+1, maxLWorkers)
+				p.ResizeLrgWorkers(workers, existing)
+			}
 		}
 		return
 	}
@@ -2229,8 +2268,9 @@ func (p *ReplicationPool) queueReplicaDeleteTask(doi DeletedObjectReplicationInf
 }

 type replicationPoolOpts struct {
-	Priority   string
-	MaxWorkers int
+	Priority    string
+	MaxWorkers  int
+	MaxLWorkers int
 }

 func initBackgroundReplication(ctx context.Context, objectAPI ObjectLayer) {
@@ -3505,7 +3545,7 @@ func (p *ReplicationPool) persistToDrive(ctx context.Context, v MRFReplicateEntr
 	}

 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	for _, localDrive := range localDrives {
@@ -3572,7 +3612,7 @@ func (p *ReplicationPool) loadMRF() (mrfRec MRFReplicateEntries, err error) {
 	}

 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	for _, localDrive := range localDrives {
@@ -3693,3 +3733,12 @@ func (p *ReplicationPool) getMRF(ctx context.Context, bucket string) (ch <-chan

 	return mrfCh, nil
 }
+
+// validateReplicationDestinationOptions is used to configure the validation of the replication destination.
+// validateReplicationDestination uses this to configure the validation.
+type validateReplicationDestinationOptions struct {
+	CheckRemoteBucket bool
+	CheckReady        bool
+
+	checkReadyErr sync.Map
+}
@@ -685,16 +685,6 @@ func loadEnvVarsFromFiles() {
 		}
 	}

-	if env.IsSet(kms.EnvKMSSecretKeyFile) {
-		kmsSecret, err := readFromSecret(env.Get(kms.EnvKMSSecretKeyFile, ""))
-		if err != nil {
-			logger.Fatal(err, "Unable to read the KMS secret key inherited from secret file")
-		}
-		if kmsSecret != "" {
-			os.Setenv(kms.EnvKMSSecretKey, kmsSecret)
-		}
-	}
-
 	if env.IsSet(config.EnvConfigEnvFile) {
 		ekvs, err := minioEnvironFromFile(env.Get(config.EnvConfigEnvFile, ""))
 		if err != nil && !os.IsNotExist(err) {
@@ -881,6 +871,12 @@ func loadRootCredentials() {
 	} else {
 		globalActiveCred = auth.DefaultCredentials
 	}
+
+	var err error
+	globalNodeAuthToken, err = authenticateNode(globalActiveCred.AccessKey, globalActiveCred.SecretKey)
+	if err != nil {
+		logger.Fatal(err, "Unable to generate internode credentials")
+	}
 }

 // Initialize KMS global variable after valiadating and loading the configuration.
@@ -101,7 +101,7 @@ func initHelp() {
 		config.HelpKV{
 			Key:         config.SubnetSubSys,
 			Type:        "string",
-			Description: "register the cluster to MinIO SUBNET",
+			Description: "register Enterprise license for the cluster",
 			Optional:    true,
 		},
 		config.HelpKV{
@@ -1172,6 +1172,14 @@ func (o *ObjectInfo) decryptChecksums(part int, h http.Header) map[string]string
 	if len(data) == 0 {
 		return nil
 	}
+	if part > 0 && !crypto.SSEC.IsEncrypted(o.UserDefined) {
+		// already decrypted in ToObjectInfo for multipart objects
+		for _, pi := range o.Parts {
+			if pi.Number == part {
+				return pi.Checksums
+			}
+		}
+	}
 	if _, encrypted := crypto.IsEncrypted(o.UserDefined); encrypted {
 		decrypted, err := o.metadataDecrypter(h)("object-checksum", data)
 		if err != nil {
@@ -23,6 +23,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"testing"
 	"time"

@@ -147,6 +148,10 @@ func TestCommonTime(t *testing.T) {
 // TestListOnlineDisks - checks if listOnlineDisks and outDatedDisks
 // are consistent with each other.
 func TestListOnlineDisks(t *testing.T) {
+	if runtime.GOOS == globalWindowsOSName {
+		t.Skip()
+	}
+
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()

@@ -230,7 +235,7 @@ func TestListOnlineDisks(t *testing.T) {
 	}

 	object := "object"
-	data := bytes.Repeat([]byte("a"), smallFileThreshold*16)
+	data := bytes.Repeat([]byte("a"), smallFileThreshold*32)
 	z := obj.(*erasureServerPools)

 	erasureDisks, err := z.GetDisks(0, 0)
@@ -26,6 +26,7 @@ import (
 	"time"

 	"github.com/minio/minio/internal/amztime"
+	"github.com/minio/minio/internal/bucket/lifecycle"
 	"github.com/minio/minio/internal/bucket/replication"
 	"github.com/minio/minio/internal/crypto"
 	"github.com/minio/minio/internal/hash/sha256"
@@ -173,6 +174,7 @@ func (fi FileInfo) ToObjectInfo(bucket, object string, versioned bool) ObjectInf
 		}
 	}
 	objInfo.Checksum = fi.Checksum
+	objInfo.decryptPartsChecksums(nil)
 	objInfo.Inlined = fi.InlineData()
 	// Success.
 	return objInfo
@@ -456,6 +458,7 @@ func commonParity(parities []int, defaultParityCount int) int {
 }

 func listObjectParities(partsMetadata []FileInfo, errs []error) (parities []int) {
+	totalShards := len(partsMetadata)
 	parities = make([]int, len(partsMetadata))
 	for index, metadata := range partsMetadata {
 		if errs[index] != nil {
@@ -466,9 +469,13 @@ func listObjectParities(partsMetadata []FileInfo, errs []error) (parities []int)
 			parities[index] = -1
 			continue
 		}
+		//nolint:gocritic
 		// Delete marker or zero byte objects take highest parity.
 		if metadata.Deleted || metadata.Size == 0 {
-			parities[index] = len(partsMetadata) / 2
+			parities[index] = totalShards / 2
+		} else if metadata.TransitionStatus == lifecycle.TransitionComplete {
+			// For tiered objects, read quorum is N/2+1 to ensure simple majority on xl.meta. It is not equal to EcM because the data integrity is entrusted with the warm tier.
+			parities[index] = totalShards - (totalShards/2 + 1)
 		} else {
 			parities[index] = metadata.Erasure.ParityBlocks
 		}
@@ -19,6 +19,8 @@ package cmd

 import (
 	"context"
+	"fmt"
+	"slices"
 	"strconv"
 	"testing"
 	"time"
@@ -359,3 +361,124 @@ func TestSkipTierFreeVersion(t *testing.T) {
 		t.Fatal("Expected SkipTierFreeVersion to be set on FileInfo but wasn't")
 	}
 }
+
+func TestListObjectParities(t *testing.T) {
+	mkMetaArr := func(N, parity, agree int) []FileInfo {
+		fi := newFileInfo("obj-1", N-parity, parity)
+		fi.TransitionTier = "WARM-TIER"
+		fi.TransitionedObjName = mustGetUUID()
+		fi.TransitionStatus = "complete"
+		fi.Size = 1 << 20
+
+		metaArr := make([]FileInfo, N)
+		for i := range N {
+			fi.Erasure.Index = i + 1
+			metaArr[i] = fi
+			if i < agree {
+				continue
+			}
+			metaArr[i].TransitionTier, metaArr[i].TransitionedObjName = "", ""
+			metaArr[i].TransitionStatus = ""
+		}
+		return metaArr
+	}
+	mkParities := func(N, agreedParity, disagreedParity, agree int) []int {
+		ps := make([]int, N)
+		for i := range N {
+			if i < agree {
+				ps[i] = agreedParity
+				continue
+			}
+			ps[i] = disagreedParity // disagree
+		}
+		return ps
+	}
+
+	mkTest := func(N, parity, agree int) (res struct {
+		metaArr  []FileInfo
+		errs     []error
+		parities []int
+		parity   int
+	},
+	) {
+		res.metaArr = mkMetaArr(N, parity, agree)
+		res.parities = mkParities(N, N-(N/2+1), parity, agree)
+		res.errs = make([]error, N)
+		if agree >= N/2+1 { // simple majority consensus
+			res.parity = N - (N/2 + 1)
+		} else {
+			res.parity = -1
+		}
+		return res
+	}
+
+	nonTieredTest := func(N, parity, agree int) (res struct {
+		metaArr  []FileInfo
+		errs     []error
+		parities []int
+		parity   int
+	},
+	) {
+		fi := newFileInfo("obj-1", N-parity, parity)
+		fi.Size = 1 << 20
+		metaArr := make([]FileInfo, N)
+		parities := make([]int, N)
+		for i := range N {
+			fi.Erasure.Index = i + 1
+			metaArr[i] = fi
+			parities[i] = parity
+			if i < agree {
+				continue
+			}
+			metaArr[i].Erasure.Index = 0 // creates invalid fi on remaining drives
+			parities[i] = -1             // invalid fi are assigned parity -1
+		}
+		res.metaArr = metaArr
+		res.parities = parities
+		res.errs = make([]error, N)
+		if agree >= N-parity {
+			res.parity = parity
+		} else {
+			res.parity = -1
+		}
+
+		return res
+	}
+	tests := []struct {
+		metaArr  []FileInfo
+		errs     []error
+		parities []int
+		parity   int
+	}{
+		// More than simple majority consensus
+		mkTest(15, 3, 11),
+		// No simple majority consensus
+		mkTest(15, 3, 7),
+		// Exact simple majority consensus
+		mkTest(15, 3, 8),
+		// More than simple majority consensus
+		mkTest(16, 4, 11),
+		// No simple majority consensus
+		mkTest(16, 4, 8),
+		// Exact simple majority consensus
+		mkTest(16, 4, 9),
+		// non-tiered object require read quorum of EcM
+		nonTieredTest(15, 3, 12),
+		// non-tiered object with fewer than EcM in consensus
+		nonTieredTest(15, 3, 11),
+		// non-tiered object require read quorum of EcM
+		nonTieredTest(16, 4, 12),
+		// non-tiered object with fewer than EcM in consensus
+		nonTieredTest(16, 4, 11),
+	}
+	for i, test := range tests {
+		t.Run(fmt.Sprintf("Test %d", i+1), func(t *testing.T) {
+			if got := listObjectParities(test.metaArr, test.errs); !slices.Equal(got, test.parities) {
+				t.Fatalf("Expected parities %v but got %v", test.parities, got)
+			}
+			if got := commonParity(test.parities, len(test.metaArr)/2); got != test.parity {
+				t.Fatalf("Expected common parity %v but got %v", test.parity, got)
+			}
+		})
+	}
+}
@@ -103,10 +103,9 @@ func (er erasureObjects) checkUploadIDExists(ctx context.Context, bucket, object
 	return fi, partsMetadata, err
 }

-// Removes part.meta given by partName belonging to a multipart upload from minioMetaBucket
-func (er erasureObjects) removePartMeta(bucket, object, uploadID, dataDir string, partNumber int) {
-	uploadIDPath := er.getUploadIDDir(bucket, object, uploadID)
-	curpartPath := pathJoin(uploadIDPath, dataDir, fmt.Sprintf("part.%d", partNumber))
+// cleanMultipartPath removes all extraneous files and parts from the multipart folder, this is used per CompleteMultipart.
+// do not use this function outside of completeMultipartUpload()
+func (er erasureObjects) cleanupMultipartPath(ctx context.Context, paths ...string) {
 	storageDisks := er.getDisks()

 	g := errgroup.WithNErrs(len(storageDisks))
@@ -116,42 +115,7 @@ func (er erasureObjects) removePartMeta(bucket, object, uploadID, dataDir string
 		}
 		index := index
 		g.Go(func() error {
-			_ = storageDisks[index].Delete(context.TODO(), minioMetaMultipartBucket, curpartPath+".meta", DeleteOptions{
-				Recursive: false,
-				Immediate: false,
-			})
-
-			return nil
-		}, index)
-	}
-	g.Wait()
-}
-
-// Removes part given by partName belonging to a multipart upload from minioMetaBucket
-func (er erasureObjects) removeObjectPart(bucket, object, uploadID, dataDir string, partNumber int) {
-	uploadIDPath := er.getUploadIDDir(bucket, object, uploadID)
-	curpartPath := pathJoin(uploadIDPath, dataDir, fmt.Sprintf("part.%d", partNumber))
-	storageDisks := er.getDisks()
-
-	g := errgroup.WithNErrs(len(storageDisks))
-	for index, disk := range storageDisks {
-		if disk == nil {
-			continue
-		}
-		index := index
-		g.Go(func() error {
-			// Ignoring failure to remove parts that weren't present in CompleteMultipartUpload
-			// requests. xl.meta is the authoritative source of truth on which parts constitute
-			// the object. The presence of parts that don't belong in the object doesn't affect correctness.
-			_ = storageDisks[index].Delete(context.TODO(), minioMetaMultipartBucket, curpartPath, DeleteOptions{
-				Recursive: false,
-				Immediate: false,
-			})
-			_ = storageDisks[index].Delete(context.TODO(), minioMetaMultipartBucket, curpartPath+".meta", DeleteOptions{
-				Recursive: false,
-				Immediate: false,
-			})
-
+			_ = storageDisks[index].DeleteBulk(ctx, minioMetaMultipartBucket, paths...)
 			return nil
 		}, index)
 	}
@@ -159,7 +123,7 @@ func (er erasureObjects) removeObjectPart(bucket, object, uploadID, dataDir stri
 }

 // Clean-up the old multipart uploads. Should be run in a Go routine.
-func (er erasureObjects) cleanupStaleUploads(ctx context.Context, expiry time.Duration) {
+func (er erasureObjects) cleanupStaleUploads(ctx context.Context) {
 	// run multiple cleanup's local to this server.
 	var wg sync.WaitGroup
 	for _, disk := range er.getLocalDisks() {
@@ -167,7 +131,7 @@ func (er erasureObjects) cleanupStaleUploads(ctx context.Context, expiry time.Du
 			wg.Add(1)
 			go func(disk StorageAPI) {
 				defer wg.Done()
-				er.cleanupStaleUploadsOnDisk(ctx, disk, expiry)
+				er.cleanupStaleUploadsOnDisk(ctx, disk)
 			}(disk)
 		}
 	}
@@ -193,7 +157,7 @@ func (er erasureObjects) deleteAll(ctx context.Context, bucket, prefix string) {
 }

 // Remove the old multipart uploads on the given disk.
-func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk StorageAPI, expiry time.Duration) {
+func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk StorageAPI) {
 	drivePath := disk.Endpoint().Path

 	readDirFn(pathJoin(drivePath, minioMetaMultipartBucket), func(shaDir string, typ os.FileMode) error {
@@ -219,7 +183,7 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
 				modTime = fi.ModTime
 				wait()
 			}
-			if time.Since(modTime) < expiry {
+			if time.Since(modTime) < globalAPIConfig.getStaleUploadsExpiry() {
 				return nil
 			}
 			w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
@@ -238,7 +202,7 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
 			return nil
 		}
 		// Modtime is returned in the Created field. See (*xlStorage).StatVol
-		if time.Since(vi.Created) < expiry {
+		if time.Since(vi.Created) < globalAPIConfig.getStaleUploadsExpiry() {
 			return nil
 		}
 		w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
@@ -267,7 +231,7 @@ func (er erasureObjects) cleanupStaleUploadsOnDisk(ctx context.Context, disk Sto
 		w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
 		return w.Run(func() error {
 			wait := deleteMultipartCleanupSleeper.Timer(ctx)
-			if time.Since(vi.Created) > expiry {
+			if time.Since(vi.Created) > globalAPIConfig.getStaleUploadsExpiry() {
 				pathUUID := mustGetUUID()
 				targetPath := pathJoin(drivePath, minioMetaTmpDeletedBucket, pathUUID)

@@ -1359,10 +1323,10 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
 		}
 	}

+	paths := make([]string, 0, len(currentFI.Parts))
 	// Remove parts that weren't present in CompleteMultipartUpload request.
 	for _, curpart := range currentFI.Parts {
-		// Remove part.meta which is not needed anymore.
-		er.removePartMeta(bucket, object, uploadID, currentFI.DataDir, curpart.Number)
+		paths = append(paths, pathJoin(uploadIDPath, currentFI.DataDir, fmt.Sprintf("part.%d.meta", curpart.Number)))

 		if objectPartIndex(fi.Parts, curpart.Number) == -1 {
 			// Delete the missing part files. e.g,
@@ -1371,10 +1335,12 @@ func (er erasureObjects) CompleteMultipartUpload(ctx context.Context, bucket str
 			// Request 3: PutObjectPart 2
 			// Request 4: CompleteMultipartUpload --part 2
 			// N.B. 1st part is not present. This part should be removed from the storage.
-			er.removeObjectPart(bucket, object, uploadID, currentFI.DataDir, curpart.Number)
+			paths = append(paths, pathJoin(uploadIDPath, currentFI.DataDir, fmt.Sprintf("part.%d", curpart.Number)))
 		}
 	}

+	er.cleanupMultipartPath(ctx, paths...) // cleanup all part.N.meta, and skipped part.N's before final rename().
+
 	defer func() {
 		if err == nil {
 			er.deleteAll(context.Background(), minioMetaMultipartBucket, uploadIDPath)
@@ -283,7 +283,7 @@ func (er erasureObjects) GetObjectNInfo(ctx context.Context, bucket, object stri
 	}

 	if unlockOnDefer {
-		unlockOnDefer = fi.InlineData()
+		unlockOnDefer = fi.InlineData() || len(fi.Data) > 0
 	}

 	pr, pw := xioutil.WaitPipe()
@@ -908,6 +908,8 @@ func (er erasureObjects) getObjectFileInfo(ctx context.Context, bucket, object s
 		}

 		rw.Lock()
+		// when its a versioned bucket and empty versionID - at totalResp == setDriveCount
+		// we must use rawFileInfo to resolve versions to figure out the latest version.
 		if opts.VersionID == "" && totalResp == er.setDriveCount {
 			fi, onlineMeta, onlineDisks, modTime, etag, err = calcQuorum(pickLatestQuorumFilesInfo(ctx,
 				rawArr, errs, bucket, object, readData, opts.InclFreeVersions, true))
@@ -915,7 +917,7 @@ func (er erasureObjects) getObjectFileInfo(ctx context.Context, bucket, object s
 			fi, onlineMeta, onlineDisks, modTime, etag, err = calcQuorum(metaArr, errs)
 		}
 		rw.Unlock()
-		if err == nil && fi.InlineData() {
+		if err == nil && (fi.InlineData() || len(fi.Data) > 0) {
 			break
 		}
 	}
@@ -1399,7 +1401,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
 	writers := make([]io.Writer, len(onlineDisks))
 	var inlineBuffers []*bytes.Buffer
 	if shardFileSize >= 0 {
-		if !opts.Versioned && shardFileSize < inlineBlock {
+		if !opts.Versioned && shardFileSize <= inlineBlock {
 			inlineBuffers = make([]*bytes.Buffer, len(onlineDisks))
 		} else if shardFileSize < inlineBlock/8 {
 			inlineBuffers = make([]*bytes.Buffer, len(onlineDisks))
@@ -1407,7 +1409,7 @@ func (er erasureObjects) putObject(ctx context.Context, bucket string, object st
 	} else {
 		// If compressed, use actual size to determine.
 		if sz := erasure.ShardFileSize(data.ActualSize()); sz > 0 {
-			if !opts.Versioned && sz < inlineBlock {
+			if !opts.Versioned && sz <= inlineBlock {
 				inlineBuffers = make([]*bytes.Buffer, len(onlineDisks))
 			} else if sz < inlineBlock/8 {
 				inlineBuffers = make([]*bytes.Buffer, len(onlineDisks))
@@ -168,7 +168,7 @@ func newErasureServerPools(ctx context.Context, endpointServerPools EndpointServ

 	if !globalIsDistErasure {
 		globalLocalDrivesMu.Lock()
-		globalLocalDrives = localDrives
+		globalLocalDrivesMap = make(map[string]StorageAPI, len(localDrives))
 		for _, drive := range localDrives {
 			globalLocalDrivesMap[drive.Endpoint().String()] = drive
 		}
@@ -1526,14 +1526,12 @@ func (z *erasureServerPools) listObjectsGeneric(ctx context.Context, bucket, pre
 			loi.NextMarker = last.Name
 		}

-		if merged.lastSkippedEntry != "" {
-			if merged.lastSkippedEntry > loi.NextMarker {
-				// An object hidden by ILM was found during listing. Since the number of entries
-				// fetched from drives is limited, set IsTruncated to true to ask the s3 client
-				// to continue listing if it wishes in order to find if there is more objects.
-				loi.IsTruncated = true
-				loi.NextMarker = merged.lastSkippedEntry
-			}
+		if loi.IsTruncated && merged.lastSkippedEntry > loi.NextMarker {
+			// An object hidden by ILM was found during a truncated listing. Since the number of entries
+			// fetched from drives is limited by max-keys, we should use the last ILM filtered entry
+			// as a continuation token if it is lexially higher than the last visible object so that the
+			// next call of WalkDir() with the max-keys can reach new objects not seen previously.
+			loi.NextMarker = merged.lastSkippedEntry
 		}

 		if loi.NextMarker != "" {
@@ -2447,6 +2445,7 @@ const (
 type HealthOptions struct {
 	Maintenance    bool
 	DeploymentType string
+	NoLogging      bool
 }

 // HealthResult returns the current state of the system, also
@@ -2483,7 +2482,7 @@ func (hr HealthResult) String() string {
 		if i == 0 {
 			str.WriteString(")")
 		} else {
-			str.WriteString("), ")
+			str.WriteString(") | ")
 		}
 	}
 	return str.String()
@@ -2606,7 +2605,7 @@ func (z *erasureServerPools) Health(ctx context.Context, opts HealthOptions) Hea
 			})

 			healthy := erasureSetUpCount[poolIdx][setIdx].online >= poolWriteQuorums[poolIdx]
-			if !healthy {
+			if !healthy && !opts.NoLogging {
 				storageLogIf(logger.SetReqInfo(ctx, reqInfo),
 					fmt.Errorf("Write quorum could not be established on pool: %d, set: %d, expected write quorum: %d, drives-online: %d",
 						poolIdx, setIdx, poolWriteQuorums[poolIdx], erasureSetUpCount[poolIdx][setIdx].online), logger.FatalKind)
@@ -2614,7 +2613,7 @@ func (z *erasureServerPools) Health(ctx context.Context, opts HealthOptions) Hea
 			result.Healthy = result.Healthy && healthy

 			healthyRead := erasureSetUpCount[poolIdx][setIdx].online >= poolReadQuorums[poolIdx]
-			if !healthyRead {
+			if !healthyRead && !opts.NoLogging {
 				storageLogIf(logger.SetReqInfo(ctx, reqInfo),
 					fmt.Errorf("Read quorum could not be established on pool: %d, set: %d, expected read quorum: %d, drives-online: %d",
 						poolIdx, setIdx, poolReadQuorums[poolIdx], erasureSetUpCount[poolIdx][setIdx].online))
@@ -189,7 +189,7 @@ func (s *erasureSets) Legacy() (ok bool) {

 // connectDisks - attempt to connect all the endpoints, loads format
 // and re-arranges the disks in proper position.
-func (s *erasureSets) connectDisks() {
+func (s *erasureSets) connectDisks(log bool) {
 	defer func() {
 		s.lastConnectDisksOpTime = time.Now()
 	}()
@@ -224,7 +224,9 @@ func (s *erasureSets) connectDisks() {
 				if endpoint.IsLocal && errors.Is(err, errUnformattedDisk) {
 					globalBackgroundHealState.pushHealLocalDisks(endpoint)
 				} else if !errors.Is(err, errDriveIsRoot) {
-					printEndpointError(endpoint, err, true)
+					if log {
+						printEndpointError(endpoint, err, true)
+					}
 				}
 				return
 			}
@@ -260,13 +262,7 @@ func (s *erasureSets) connectDisks() {
 				if globalIsDistErasure {
 					globalLocalSetDrives[s.poolIndex][setIndex][diskIndex] = disk
 				}
-				for i, ldisk := range globalLocalDrives {
-					_, k, l := ldisk.GetDiskLoc()
-					if k == setIndex && l == diskIndex {
-						globalLocalDrives[i] = disk
-						break
-					}
-				}
+				globalLocalDrivesMap[disk.Endpoint().String()] = disk
 				globalLocalDrivesMu.Unlock()
 			}
 			s.erasureDisksMu.Unlock()
@@ -285,7 +281,7 @@ func (s *erasureSets) monitorAndConnectEndpoints(ctx context.Context, monitorInt
 	time.Sleep(time.Duration(r.Float64() * float64(time.Second)))

 	// Pre-emptively connect the disks if possible.
-	s.connectDisks()
+	s.connectDisks(false)

 	monitor := time.NewTimer(monitorInterval)
 	defer monitor.Stop()
@@ -299,7 +295,7 @@ func (s *erasureSets) monitorAndConnectEndpoints(ctx context.Context, monitorInt
 				console.Debugln("running drive monitoring")
 			}

-			s.connectDisks()
+			s.connectDisks(true)

 			// Reset the timer for next interval
 			monitor.Reset(monitorInterval)
@@ -529,7 +525,7 @@ func (s *erasureSets) cleanupStaleUploads(ctx context.Context) {
 					if set == nil {
 						return
 					}
-					set.cleanupStaleUploads(ctx, globalAPIConfig.getStaleUploadsExpiry())
+					set.cleanupStaleUploads(ctx)
 				}(set)
 			}
 			wg.Wait()
@@ -1133,13 +1129,7 @@ func (s *erasureSets) HealFormat(ctx context.Context, dryRun bool) (res madmin.H
 					if globalIsDistErasure {
 						globalLocalSetDrives[s.poolIndex][m][n] = disk
 					}
-					for i, ldisk := range globalLocalDrives {
-						_, k, l := ldisk.GetDiskLoc()
-						if k == m && l == n {
-							globalLocalDrives[i] = disk
-							break
-						}
-					}
+					globalLocalDrivesMap[disk.Endpoint().String()] = disk
 					globalLocalDrivesMu.Unlock()
 				}
 			}
@@ -0,0 +1,120 @@
+// Copyright (c) 2015-2024 MinIO, Inc.
+//
+// This file is part of MinIO Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//go:build fmtgen
+
+package cmd
+
+import (
+	"encoding/json"
+	"log"
+	"os"
+	"path/filepath"
+
+	"github.com/klauspost/compress/zip"
+	"github.com/minio/cli"
+)
+
+var fmtGenFlags = []cli.Flag{
+	cli.IntFlag{
+		Name:  "parity",
+		Usage: "specify erasure code parity",
+	},
+	cli.StringFlag{
+		Name:  "deployment-id",
+		Usage: "deployment-id of the MinIO cluster for which format.json is needed",
+	},
+	cli.StringFlag{
+		Name:   "address",
+		Value:  ":" + GlobalMinioDefaultPort,
+		Usage:  "bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname",
+		EnvVar: "MINIO_ADDRESS",
+	},
+}
+
+var fmtGenCmd = cli.Command{
+	Name:   "fmt-gen",
+	Usage:  "Generate format.json files for an erasure server pool",
+	Flags:  append(fmtGenFlags, GlobalFlags...),
+	Action: fmtGenMain,
+	CustomHelpTemplate: `NAME:
+  {{.HelpName}} - {{.Usage}}
+
+USAGE:
+  {{.HelpName}} {{if .VisibleFlags}}[FLAGS] {{end}}DIR1 [DIR2..]
+  {{.HelpName}} {{if .VisibleFlags}}[FLAGS] {{end}}DIR{1...64}
+  {{.HelpName}} {{if .VisibleFlags}}[FLAGS] {{end}}DIR{1...64} DIR{65...128}
+
+DIR:
+  DIR points to a directory on a filesystem. When you want to combine
+  multiple drives into a single large system, pass one directory per
+  filesystem separated by space. You may also use a '...' convention
+  to abbreviate the directory arguments. Remote directories in a
+  distributed setup are encoded as HTTP(s) URIs.
+{{if .VisibleFlags}}
+FLAGS:
+  {{range .VisibleFlags}}{{.}}
+  {{end}}{{end}}
+EXAMPLES:
+  1. Generate format.json.zip containing format.json files for all drives in a distributed MinIO server pool of 32 nodes with 32 drives each.
+     {{.Prompt}} {{.HelpName}} http://node{1...32}.example.com/mnt/export{1...32}
+
+`,
+}
+
+func fmtGenMain(ctxt *cli.Context) {
+	deploymentID := ctxt.String("deployment-id")
+	err := buildServerCtxt(ctxt, &globalServerCtxt)
+	if err != nil {
+		log.Fatalln(err)
+	}
+	handleCommonArgs(globalServerCtxt)
+	pools, _, err := createServerEndpoints(globalMinioAddr, globalServerCtxt.Layout.pools, globalServerCtxt.Layout.legacy)
+	if err != nil {
+		log.Fatalln(err)
+	}
+
+	zipFile, err := os.Create("format.json.zip")
+	if err != nil {
+		log.Fatalf("failed to create format.json.zip: %v", err)
+	}
+	defer zipFile.Close()
+	fmtZipW := zip.NewWriter(zipFile)
+	defer fmtZipW.Close()
+	for _, pool := range pools { // for each pool
+		setCount, setDriveCount := pool.SetCount, pool.DrivesPerSet
+		format := newFormatErasureV3(setCount, setDriveCount)
+		format.ID = deploymentID
+		for i := 0; i < setCount; i++ { // for each erasure set
+			for j := 0; j < setDriveCount; j++ {
+				newFormat := format.Clone()
+				newFormat.Erasure.This = format.Erasure.Sets[i][j]
+				if deploymentID != "" {
+					newFormat.ID = deploymentID
+				}
+				drive := pool.Endpoints[i*setDriveCount+j]
+				fmtBytes, err := json.Marshal(newFormat)
+				if err != nil {
+					//nolint:gocritic
+					log.Fatalf("failed to marshal format.json for %s: %v", drive.String(), err)
+				}
+				fmtJSON := filepath.Join(drive.Host, drive.Path, minioMetaBucket, "format.json")
+				embedFileInZip(fmtZipW, fmtJSON, fmtBytes, 0o600)
+			}
+		}
+	}
+}
@@ -1,6 +1,6 @@
-// Copyright (c) 2015-2022 MinIO, Inc.
+// Copyright (c) 2015-2024 MinIO, Inc.
 //
-// This file is part of MinIO Object Storage stack
+// # This file is part of MinIO Object Storage stack
 //
 // This program is free software: you can redistribute it and/or modify
 // it under the terms of the GNU Affero General Public License as published by
@@ -15,9 +15,10 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.

-//go:build go1.19
-// +build go1.19
+//go:build !fmtgen

-package fips
+package cmd

-const go19 = true
+import "github.com/minio/cli"
+
+var fmtGenCmd cli.Command
@@ -247,11 +247,14 @@ func guessIsRPCReq(req *http.Request) bool {
 	if req == nil {
 		return false
 	}
-	if req.Method == http.MethodGet && req.URL != nil && req.URL.Path == grid.RoutePath {
-		return true
+	if req.Method == http.MethodGet && req.URL != nil {
+		switch req.URL.Path {
+		case grid.RoutePath, grid.RouteLockPath:
+			return true
+		}
 	}

-	return req.Method == http.MethodPost &&
+	return (req.Method == http.MethodPost || req.Method == http.MethodGet) &&
 		strings.HasPrefix(req.URL.Path, minioReservedBucketPath+SlashSeparator)
 }

@@ -469,7 +472,7 @@ func setBucketForwardingMiddleware(h http.Handler) http.Handler {
 		}
 		if globalDNSConfig == nil || !globalBucketFederation ||
 			guessIsHealthCheckReq(r) || guessIsMetricsReq(r) ||
-			guessIsRPCReq(r) || guessIsLoginSTSReq(r) || isAdminReq(r) {
+			guessIsRPCReq(r) || guessIsLoginSTSReq(r) || isAdminReq(r) || isKMSReq(r) {
 			h.ServeHTTP(w, r)
 			return
 		}
@@ -51,9 +51,10 @@ func TestGuessIsRPC(t *testing.T) {
 	r = &http.Request{
 		Proto:  "HTTP/1.1",
 		Method: http.MethodGet,
+		URL:    u,
 	}
-	if guessIsRPCReq(r) {
-		t.Fatal("Test shouldn't report as net/rpc for a non net/rpc request.")
+	if !guessIsRPCReq(r) {
+		t.Fatal("Test shouldn't fail for a possible net/rpc request.")
 	}
 	r = &http.Request{
 		Proto:  "HTTP/1.1",
@@ -63,6 +64,14 @@ func TestGuessIsRPC(t *testing.T) {
 	if !guessIsRPCReq(r) {
 		t.Fatal("Grid RPC path not detected")
 	}
+	r = &http.Request{
+		Proto:  "HTTP/1.1",
+		Method: http.MethodGet,
+		URL:    &url.URL{Path: grid.RouteLockPath},
+	}
+	if !guessIsRPCReq(r) {
+		t.Fatal("Grid RPC path not detected")
+	}
 }

 var isHTTPHeaderSizeTooLargeTests = []struct {
@@ -150,6 +150,11 @@ type healEntryResult struct {

 // healErasureSet lists and heals all objects in a specific erasure set
 func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string, tracker *healingTracker) error {
+	bgSeq, found := globalBackgroundHealState.getHealSequenceByToken(bgHealingUUID)
+	if !found {
+		return errors.New("no local healing sequence initialized, unable to heal the drive")
+	}
+
 	scanMode := madmin.HealNormalScan

 	// Make sure to copy since `buckets slice`
@@ -163,11 +168,7 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
 	}

 	for _, bucket := range healBuckets {
-		_, err := objAPI.HealBucket(ctx, bucket, madmin.HealOpts{
-			Recreate: true,
-			ScanMode: scanMode,
-		})
-		if err != nil {
+		if err := bgSeq.healBucket(objAPI, bucket, true); err != nil {
 			// Log bucket healing error if any, we shall retry again.
 			healingLogIf(ctx, err)
 		}
@@ -264,10 +265,7 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
 		tracker.setBucket(bucket)
 		// Heal current bucket again in case if it is failed
 		// in the beginning of erasure set healing
-		if _, err := objAPI.HealBucket(ctx, bucket, madmin.HealOpts{
-			Recreate: true,
-			ScanMode: scanMode,
-		}); err != nil {
+		if err := bgSeq.healBucket(objAPI, bucket, true); err != nil {
 			// Set this such that when we return this function
 			// we let the caller retry this disk again for the
 			// buckets that failed healing.
@@ -366,6 +364,7 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
 				}
 				return false
 			case results <- result:
+				bgSeq.countScanned(madmin.HealItemObject)
 				return true
 			}
 		}
@@ -416,8 +415,10 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
 						return
 					}
 					result = healEntryFailure(0)
+					bgSeq.countFailed(madmin.HealItemObject)
 					healingLogIf(ctx, fmt.Errorf("unable to heal object %s/%s: %w", bucket, entry.name, err))
 				} else {
+					bgSeq.countHealed(madmin.HealItemObject)
 					result = healEntrySuccess(uint64(res.ObjectSize))
 				}

@@ -463,8 +464,10 @@ func (er *erasureObjects) healErasureSet(ctx context.Context, buckets []string,
 				}

 				if versionHealed {
+					bgSeq.countHealed(madmin.HealItemObject)
 					result = healEntrySuccess(uint64(version.Size))
 				} else {
+					bgSeq.countFailed(madmin.HealItemObject)
 					result = healEntryFailure(uint64(version.Size))
 					if version.VersionID != "" {
 						healingLogIf(ctx, fmt.Errorf("unable to heal object %s/%s-v(%s): %w", bucket, version.Name, version.VersionID, err))
@@ -310,6 +310,7 @@ var (
 	globalBootTime = UTCNow()

 	globalActiveCred         auth.Credentials
+	globalNodeAuthToken      string
 	globalSiteReplicatorCred siteReplicatorCred

 	// Captures if root credentials are set via ENV.
@@ -413,10 +414,9 @@ var (
 	globalServiceFreezeCnt int32
 	globalServiceFreezeMu  sync.Mutex // Updates.

-	// List of local drives to this node, this is only set during server startup,
-	// and is only mutated by HealFormat. Hold globalLocalDrivesMu to access.
-	globalLocalDrives    []StorageAPI
-	globalLocalDrivesMap = make(map[string]StorageAPI)
+	// Map of local drives to this node, this is set during server startup,
+	// disk reconnect and mutated by HealFormat. Hold globalLocalDrivesMu to access.
+	globalLocalDrivesMap map[string]StorageAPI
 	globalLocalDrivesMu  sync.RWMutex

 	globalDriveMonitoring = env.Get("_MINIO_DRIVE_ACTIVE_MONITORING", config.EnableOn) == config.EnableOn
@@ -31,9 +31,15 @@ import (
 // globalGrid is the global grid manager.
 var globalGrid atomic.Pointer[grid.Manager]

+// globalLockGrid is the global lock grid manager.
+var globalLockGrid atomic.Pointer[grid.Manager]
+
 // globalGridStart is a channel that will block startup of grid connections until closed.
 var globalGridStart = make(chan struct{})

+// globalLockGridStart is a channel that will block startup of lock grid connections until closed.
+var globalLockGridStart = make(chan struct{})
+
 func initGlobalGrid(ctx context.Context, eps EndpointServerPools) error {
 	hosts, local := eps.GridHosts()
 	lookupHost := globalDNSCache.LookupHost
@@ -55,9 +61,10 @@ func initGlobalGrid(ctx context.Context, eps EndpointServerPools) error {
 		AuthFn:       newCachedAuthToken(),
 		BlockConnect: globalGridStart,
 		// Record incoming and outgoing bytes.
-		Incoming: globalConnStats.incInternodeInputBytes,
-		Outgoing: globalConnStats.incInternodeOutputBytes,
-		TraceTo:  globalTrace,
+		Incoming:  globalConnStats.incInternodeInputBytes,
+		Outgoing:  globalConnStats.incInternodeOutputBytes,
+		TraceTo:   globalTrace,
+		RoutePath: grid.RoutePath,
 	})
 	if err != nil {
 		return err
@@ -65,3 +72,36 @@ func initGlobalGrid(ctx context.Context, eps EndpointServerPools) error {
 	globalGrid.Store(g)
 	return nil
 }
+
+func initGlobalLockGrid(ctx context.Context, eps EndpointServerPools) error {
+	hosts, local := eps.GridHosts()
+	lookupHost := globalDNSCache.LookupHost
+	g, err := grid.NewManager(ctx, grid.ManagerOptions{
+		// Pass Dialer for websocket grid, make sure we do not
+		// provide any DriveOPTimeout() function, as that is not
+		// useful over persistent connections.
+		Dialer: grid.ConnectWSWithRoutePath(
+			grid.ContextDialer(xhttp.DialContextWithLookupHost(lookupHost, xhttp.NewInternodeDialContext(rest.DefaultTimeout, globalTCPOptions.ForWebsocket()))),
+			newCachedAuthToken(),
+			&tls.Config{
+				RootCAs:          globalRootCAs,
+				CipherSuites:     fips.TLSCiphers(),
+				CurvePreferences: fips.TLSCurveIDs(),
+			}, grid.RouteLockPath),
+		Local:        local,
+		Hosts:        hosts,
+		AuthToken:    validateStorageRequestToken,
+		AuthFn:       newCachedAuthToken(),
+		BlockConnect: globalGridStart,
+		// Record incoming and outgoing bytes.
+		Incoming:  globalConnStats.incInternodeInputBytes,
+		Outgoing:  globalConnStats.incInternodeOutputBytes,
+		TraceTo:   globalTrace,
+		RoutePath: grid.RouteLockPath,
+	})
+	if err != nil {
+		return err
+	}
+	globalLockGrid.Store(g)
+	return nil
+}
@@ -39,14 +39,15 @@ import (
 type apiConfig struct {
 	mu sync.RWMutex

-	requestsDeadline      time.Duration
-	requestsPool          chan struct{}
-	clusterDeadline       time.Duration
-	listQuorum            string
-	corsAllowOrigins      []string
-	replicationPriority   string
-	replicationMaxWorkers int
-	transitionWorkers     int
+	requestsDeadline       time.Duration
+	requestsPool           chan struct{}
+	clusterDeadline        time.Duration
+	listQuorum             string
+	corsAllowOrigins       []string
+	replicationPriority    string
+	replicationMaxWorkers  int
+	replicationMaxLWorkers int
+	transitionWorkers      int

 	staleUploadsExpiry          time.Duration
 	staleUploadsCleanupInterval time.Duration
@@ -170,11 +171,12 @@ func (t *apiConfig) init(cfg api.Config, setDriveCounts []int, legacy bool) {
 	}
 	t.listQuorum = listQuorum
 	if globalReplicationPool != nil &&
-		(cfg.ReplicationPriority != t.replicationPriority || cfg.ReplicationMaxWorkers != t.replicationMaxWorkers) {
-		globalReplicationPool.ResizeWorkerPriority(cfg.ReplicationPriority, cfg.ReplicationMaxWorkers)
+		(cfg.ReplicationPriority != t.replicationPriority || cfg.ReplicationMaxWorkers != t.replicationMaxWorkers || cfg.ReplicationMaxLWorkers != t.replicationMaxLWorkers) {
+		globalReplicationPool.ResizeWorkerPriority(cfg.ReplicationPriority, cfg.ReplicationMaxWorkers, cfg.ReplicationMaxLWorkers)
 	}
 	t.replicationPriority = cfg.ReplicationPriority
 	t.replicationMaxWorkers = cfg.ReplicationMaxWorkers
+	t.replicationMaxLWorkers = cfg.ReplicationMaxLWorkers

 	// N B api.transition_workers will be deprecated
 	if globalTransitionState != nil {
@@ -381,14 +383,16 @@ func (t *apiConfig) getReplicationOpts() replicationPoolOpts {

 	if t.replicationPriority == "" {
 		return replicationPoolOpts{
-			Priority:   "auto",
-			MaxWorkers: WorkerMaxLimit,
+			Priority:    "auto",
+			MaxWorkers:  WorkerMaxLimit,
+			MaxLWorkers: LargeWorkerCount,
 		}
 	}

 	return replicationPoolOpts{
-		Priority:   t.replicationPriority,
-		MaxWorkers: t.replicationMaxWorkers,
+		Priority:    t.replicationPriority,
+		MaxWorkers:  t.replicationMaxWorkers,
+		MaxLWorkers: t.replicationMaxLWorkers,
 	}
 }

@@ -27,10 +27,6 @@ import (
 	"github.com/prometheus/client_golang/prometheus"
 )

-const (
-	apiGetObject = "GetObject"
-)
-
 // connStats - Network statistics
 // Count total input/output transferred bytes during
 // the server's life.
@@ -132,7 +128,7 @@ func (bh *bucketHTTPStats) updateHTTPStats(bucket, api string, w *xhttp.Response
 		return
 	}

-	if w != nil && api == apiGetObject {
+	if w != nil {
 		// Increment the prometheus http request response histogram with API, Bucket
 		bucketHTTPRequestsDuration.With(prometheus.Labels{
 			"api":    api,
@@ -437,9 +433,7 @@ func (st *HTTPStats) updateStats(api string, w *xhttp.ResponseRecorder) {
 	st.totalS3Requests.Inc(api)

 	// Increment the prometheus http request response histogram with appropriate label
-	if api == apiGetObject {
-		httpRequestsDuration.With(prometheus.Labels{"api": api}).Observe(w.TimeToFirstByte.Seconds())
-	}
+	httpRequestsDuration.With(prometheus.Labels{"api": api}).Observe(w.TimeToFirstByte.Seconds())

 	code := w.StatusCode

@@ -495,7 +495,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 		for _, item := range regUsersList {
 			userName := path.Dir(item)
 			if err := iamOS.loadUser(ctx, userName, regUser, cache.iamUsersMap); err != nil && err != errNoSuchUser {
-				return fmt.Errorf("unable to load the user `%s`: %w", userName, err)
+				return fmt.Errorf("unable to load the user: %w", err)
 			}
 		}
 		if took := time.Since(regUsersLoadStartTime); took > maxIAMLoadOpTime {
@@ -510,7 +510,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 		for _, item := range groupsList {
 			group := path.Dir(item)
 			if err := iamOS.loadGroup(ctx, group, cache.iamGroupsMap); err != nil && err != errNoSuchGroup {
-				return fmt.Errorf("unable to load the group `%s`: %w", group, err)
+				return fmt.Errorf("unable to load the group: %w", err)
 			}
 		}
 		if took := time.Since(groupsLoadStartTime); took > maxIAMLoadOpTime {
@@ -524,7 +524,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 	for _, item := range userPolicyMappingsList {
 		userName := strings.TrimSuffix(item, ".json")
 		if err := iamOS.loadMappedPolicy(ctx, userName, regUser, false, cache.iamUserPolicyMap); err != nil && !errors.Is(err, errNoSuchPolicy) {
-			return fmt.Errorf("unable to load the policy mapping for the user `%s`: %w", userName, err)
+			return fmt.Errorf("unable to load the policy mapping for the user: %w", err)
 		}
 	}
 	if took := time.Since(userPolicyMappingLoadStartTime); took > maxIAMLoadOpTime {
@@ -537,7 +537,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 	for _, item := range groupPolicyMappingsList {
 		groupName := strings.TrimSuffix(item, ".json")
 		if err := iamOS.loadMappedPolicy(ctx, groupName, regUser, true, cache.iamGroupPolicyMap); err != nil && !errors.Is(err, errNoSuchPolicy) {
-			return fmt.Errorf("unable to load the policy mapping for the group `%s`: %w", groupName, err)
+			return fmt.Errorf("unable to load the policy mapping for the group: %w", err)
 		}
 	}
 	if took := time.Since(groupPolicyMappingLoadStartTime); took > maxIAMLoadOpTime {
@@ -551,7 +551,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 	for _, item := range svcAccList {
 		userName := path.Dir(item)
 		if err := iamOS.loadUser(ctx, userName, svcUser, svcUsersMap); err != nil && err != errNoSuchUser {
-			return fmt.Errorf("unable to load the service account `%s`: %w", userName, err)
+			return fmt.Errorf("unable to load the service account: %w", err)
 		}
 	}
 	if took := time.Since(svcAccLoadStartTime); took > maxIAMLoadOpTime {
@@ -580,7 +580,7 @@ func (iamOS *IAMObjectStore) loadAllFromObjStore(ctx context.Context, cache *iam
 			// OIDC/AssumeRoleWithCustomToken/AssumeRoleWithCertificate).
 			err := iamOS.loadMappedPolicy(ctx, svcParent, stsUser, false, cache.iamSTSPolicyMap)
 			if err != nil && !errors.Is(err, errNoSuchPolicy) {
-				return fmt.Errorf("unable to load the policy mapping for the STS user `%s`: %w", svcParent, err)
+				return fmt.Errorf("unable to load the policy mapping for the STS user: %w", err)
 			}
 		}
 	}
@@ -2090,7 +2090,7 @@ func (store *IAMStoreSys) GetAllSTSUserMappings(userPredicate func(string) bool)

 // Assumes store is locked by caller. If userMap is empty, returns all user mappings.
 func (store *IAMStoreSys) listUserPolicyMappings(cache *iamCache, userMap map[string]set.StringSet,
-	userPredicate func(string) bool,
+	userPredicate func(string) bool, decodeFunc func(string) string,
 ) []madmin.UserPolicyEntities {
 	stsMap := xsync.NewMapOf[string, MappedPolicy]()
 	resMap := make(map[string]madmin.UserPolicyEntities, len(userMap))
@@ -2098,9 +2098,13 @@ func (store *IAMStoreSys) listUserPolicyMappings(cache *iamCache, userMap map[st
 	for user, groupSet := range userMap {
 		// Attempt to load parent user mapping for STS accounts
 		store.loadMappedPolicy(context.TODO(), user, stsUser, false, stsMap)
-		blankEntities := madmin.UserPolicyEntities{User: user}
+		decodeUser := user
+		if decodeFunc != nil {
+			decodeUser = decodeFunc(user)
+		}
+		blankEntities := madmin.UserPolicyEntities{User: decodeUser}
 		if !groupSet.IsEmpty() {
-			blankEntities.MemberOfMappings = store.listGroupPolicyMappings(cache, groupSet, nil)
+			blankEntities.MemberOfMappings = store.listGroupPolicyMappings(cache, groupSet, nil, decodeFunc)
 		}
 		resMap[user] = blankEntities
 	}
@@ -2116,7 +2120,11 @@ func (store *IAMStoreSys) listUserPolicyMappings(cache *iamCache, userMap map[st
 			if len(userMap) > 0 {
 				return true
 			}
-			entitiesWithMemberOf = madmin.UserPolicyEntities{User: user}
+			decodeUser := user
+			if decodeFunc != nil {
+				decodeUser = decodeFunc(user)
+			}
+			entitiesWithMemberOf = madmin.UserPolicyEntities{User: decodeUser}
 		}

 		ps := mappedPolicy.toSlice()
@@ -2155,7 +2163,7 @@ func (store *IAMStoreSys) listUserPolicyMappings(cache *iamCache, userMap map[st

 // Assumes store is locked by caller. If groups is empty, returns all group mappings.
 func (store *IAMStoreSys) listGroupPolicyMappings(cache *iamCache, groupsSet set.StringSet,
-	groupPredicate func(string) bool,
+	groupPredicate func(string) bool, decodeFunc func(string) string,
 ) []madmin.GroupPolicyEntities {
 	var r []madmin.GroupPolicyEntities

@@ -2168,10 +2176,15 @@ func (store *IAMStoreSys) listGroupPolicyMappings(cache *iamCache, groupsSet set
 			return true
 		}

+		decodeGroup := group
+		if decodeFunc != nil {
+			decodeGroup = decodeFunc(group)
+		}
+
 		ps := mappedPolicy.toSlice()
 		sort.Strings(ps)
 		r = append(r, madmin.GroupPolicyEntities{
-			Group:    group,
+			Group:    decodeGroup,
 			Policies: ps,
 		})
 		return true
@@ -2186,7 +2199,7 @@ func (store *IAMStoreSys) listGroupPolicyMappings(cache *iamCache, groupsSet set

 // Assumes store is locked by caller. If policies is empty, returns all policy mappings.
 func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.StringSet,
-	userPredicate, groupPredicate func(string) bool,
+	userPredicate, groupPredicate func(string) bool, decodeFunc func(string) string,
 ) []madmin.PolicyEntities {
 	policyToUsersMap := make(map[string]set.StringSet)
 	cache.iamUserPolicyMap.Range(func(user string, mappedPolicy MappedPolicy) bool {
@@ -2194,6 +2207,11 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 			return true
 		}

+		decodeUser := user
+		if decodeFunc != nil {
+			decodeUser = decodeFunc(user)
+		}
+
 		commonPolicySet := mappedPolicy.policySet()
 		if !queryPolSet.IsEmpty() {
 			commonPolicySet = commonPolicySet.Intersection(queryPolSet)
@@ -2201,9 +2219,9 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 		for _, policy := range commonPolicySet.ToSlice() {
 			s, ok := policyToUsersMap[policy]
 			if !ok {
-				policyToUsersMap[policy] = set.CreateStringSet(user)
+				policyToUsersMap[policy] = set.CreateStringSet(decodeUser)
 			} else {
-				s.Add(user)
+				s.Add(decodeUser)
 				policyToUsersMap[policy] = s
 			}
 		}
@@ -2217,6 +2235,11 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 				continue
 			}

+			decodeUser := user
+			if decodeFunc != nil {
+				decodeUser = decodeFunc(user)
+			}
+
 			var mappedPolicy MappedPolicy
 			store.loadIAMConfig(context.Background(), &mappedPolicy, getMappedPolicyPath(user, stsUser, false))

@@ -2227,9 +2250,9 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 			for _, policy := range commonPolicySet.ToSlice() {
 				s, ok := policyToUsersMap[policy]
 				if !ok {
-					policyToUsersMap[policy] = set.CreateStringSet(user)
+					policyToUsersMap[policy] = set.CreateStringSet(decodeUser)
 				} else {
-					s.Add(user)
+					s.Add(decodeUser)
 					policyToUsersMap[policy] = s
 				}
 			}
@@ -2244,6 +2267,11 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 					return true
 				}

+				decodeUser := user
+				if decodeFunc != nil {
+					decodeUser = decodeFunc(user)
+				}
+
 				commonPolicySet := mappedPolicy.policySet()
 				if !queryPolSet.IsEmpty() {
 					commonPolicySet = commonPolicySet.Intersection(queryPolSet)
@@ -2251,9 +2279,9 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 				for _, policy := range commonPolicySet.ToSlice() {
 					s, ok := policyToUsersMap[policy]
 					if !ok {
-						policyToUsersMap[policy] = set.CreateStringSet(user)
+						policyToUsersMap[policy] = set.CreateStringSet(decodeUser)
 					} else {
-						s.Add(user)
+						s.Add(decodeUser)
 						policyToUsersMap[policy] = s
 					}
 				}
@@ -2268,6 +2296,11 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 			return true
 		}

+		decodeGroup := group
+		if decodeFunc != nil {
+			decodeGroup = decodeFunc(group)
+		}
+
 		commonPolicySet := mappedPolicy.policySet()
 		if !queryPolSet.IsEmpty() {
 			commonPolicySet = commonPolicySet.Intersection(queryPolSet)
@@ -2275,9 +2308,9 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St
 		for _, policy := range commonPolicySet.ToSlice() {
 			s, ok := policyToGroupsMap[policy]
 			if !ok {
-				policyToGroupsMap[policy] = set.CreateStringSet(group)
+				policyToGroupsMap[policy] = set.CreateStringSet(decodeGroup)
 			} else {
-				s.Add(group)
+				s.Add(decodeGroup)
 				policyToGroupsMap[policy] = s
 			}
 		}
@@ -2318,7 +2351,7 @@ func (store *IAMStoreSys) listPolicyMappings(cache *iamCache, queryPolSet set.St

 // ListPolicyMappings - return users/groups mapped to policies.
 func (store *IAMStoreSys) ListPolicyMappings(q cleanEntitiesQuery,
-	userPredicate, groupPredicate func(string) bool,
+	userPredicate, groupPredicate func(string) bool, decodeFunc func(string) string,
 ) madmin.PolicyEntitiesResult {
 	cache := store.rlock()
 	defer store.runlock()
@@ -2328,13 +2361,13 @@ func (store *IAMStoreSys) ListPolicyMappings(q cleanEntitiesQuery,
 	isAllPoliciesQuery := len(q.Users) == 0 && len(q.Groups) == 0 && len(q.Policies) == 0

 	if len(q.Users) > 0 {
-		result.UserMappings = store.listUserPolicyMappings(cache, q.Users, userPredicate)
+		result.UserMappings = store.listUserPolicyMappings(cache, q.Users, userPredicate, decodeFunc)
 	}
 	if len(q.Groups) > 0 {
-		result.GroupMappings = store.listGroupPolicyMappings(cache, q.Groups, groupPredicate)
+		result.GroupMappings = store.listGroupPolicyMappings(cache, q.Groups, groupPredicate, decodeFunc)
 	}
 	if len(q.Policies) > 0 || isAllPoliciesQuery {
-		result.PolicyMappings = store.listPolicyMappings(cache, q.Policies, userPredicate, groupPredicate)
+		result.PolicyMappings = store.listPolicyMappings(cache, q.Policies, userPredicate, groupPredicate, decodeFunc)
 	}
 	return result
 }
@@ -217,6 +217,9 @@ func (sys *IAMSys) Load(ctx context.Context, firstTime bool) error {

 	if firstTime {
 		bootstrapTraceMsg(fmt.Sprintf("globalIAMSys.Load(): (duration: %s)", loadDuration))
+		if globalIsDistErasure {
+			logger.Info("IAM load(startup) finished. (duration: %s)", loadDuration)
+		}
 	}

 	select {
@@ -400,12 +403,12 @@ func (sys *IAMSys) periodicRoutines(ctx context.Context, baseInterval time.Durat
 			// Load all IAM items (except STS creds) periodically.
 			refreshStart := time.Now()
 			if err := sys.Load(ctx, false); err != nil {
-				iamLogIf(ctx, fmt.Errorf("Failure in periodic refresh for IAM (took %.2fs): %v", time.Since(refreshStart).Seconds(), err), logger.WarningKind)
+				iamLogIf(ctx, fmt.Errorf("Failure in periodic refresh for IAM (duration: %s): %v", time.Since(refreshStart), err), logger.WarningKind)
 			} else {
 				took := time.Since(refreshStart).Seconds()
 				if took > maxDurationSecondsForLog {
 					// Log if we took a lot of time to load.
-					logger.Info("IAM refresh took %.2fs", took)
+					logger.Info("IAM refresh took (duration: %s)", took)
 				}
 			}

@@ -841,7 +844,7 @@ func (sys *IAMSys) createCleanEntitiesQuery(q madmin.PolicyEntitiesQuery, ldap b
 		// Validate and normalize groups.
 		for _, group := range q.Groups {
 			lookupRes, underDN, _ := sys.LDAPConfig.GetValidatedGroupDN(nil, group)
-			if lookupRes != nil && !underDN {
+			if lookupRes != nil && underDN {
 				cleanQ.Groups.Add(lookupRes.NormDN)
 			}
 		}
@@ -871,7 +874,7 @@ func (sys *IAMSys) QueryLDAPPolicyEntities(ctx context.Context, q madmin.PolicyE
 	select {
 	case <-sys.configLoaded:
 		cleanQuery := sys.createCleanEntitiesQuery(q, true)
-		pe := sys.store.ListPolicyMappings(cleanQuery, sys.LDAPConfig.IsLDAPUserDN, sys.LDAPConfig.IsLDAPGroupDN)
+		pe := sys.store.ListPolicyMappings(cleanQuery, sys.LDAPConfig.IsLDAPUserDN, sys.LDAPConfig.IsLDAPGroupDN, sys.LDAPConfig.DecodeDN)
 		pe.Timestamp = UTCNow()
 		return &pe, nil
 	case <-ctx.Done():
@@ -955,7 +958,7 @@ func (sys *IAMSys) QueryPolicyEntities(ctx context.Context, q madmin.PolicyEntit
 				return !sys.LDAPConfig.IsLDAPGroupDN(s)
 			}
 		}
-		pe := sys.store.ListPolicyMappings(cleanQuery, userPredicate, groupPredicate)
+		pe := sys.store.ListPolicyMappings(cleanQuery, userPredicate, groupPredicate, nil)
 		pe.Timestamp = UTCNow()
 		return &pe, nil
 	case <-ctx.Done():
@@ -1439,11 +1442,13 @@ func (sys *IAMSys) updateGroupMembershipsForLDAP(ctx context.Context) {
 	// 1. Collect all LDAP users with active creds.
 	allCreds := sys.store.GetSTSAndServiceAccounts()
 	// List of unique LDAP (parent) user DNs that have active creds
-	var parentUsers []string
-	// Map of LDAP user to list of active credential objects
+	var parentUserActualDNList []string
+	// Map of LDAP user (internal representation) to list of active credential objects
 	parentUserToCredsMap := make(map[string][]auth.Credentials)
 	// DN to ldap username mapping for each LDAP user
-	parentUserToLDAPUsernameMap := make(map[string]string)
+	actualDNToLDAPUsernameMap := make(map[string]string)
+	// External (actual) LDAP DN to internal normalized representation
+	actualDNToParentUserMap := make(map[string]string)
 	for _, cred := range allCreds {
 		// Expired credentials don't need parent user updates.
 		if cred.IsExpired() {
@@ -1486,25 +1491,28 @@ func (sys *IAMSys) updateGroupMembershipsForLDAP(ctx context.Context) {
 				continue
 			}

-			ldapUsername, ok := jwtClaims.Lookup(ldapUserN)
-			if !ok {
+			ldapUsername, okUserN := jwtClaims.Lookup(ldapUserN)
+			ldapActualDN, okDN := jwtClaims.Lookup(ldapActualUser)
+			if !okUserN || !okDN {
 				// skip this cred - we dont have the
 				// username info needed
 				continue
 			}

 			// Collect each new cred.ParentUser into parentUsers
-			parentUsers = append(parentUsers, cred.ParentUser)
+			parentUserActualDNList = append(parentUserActualDNList, ldapActualDN)

 			// Update the ldapUsernameMap
-			parentUserToLDAPUsernameMap[cred.ParentUser] = ldapUsername
+			actualDNToLDAPUsernameMap[ldapActualDN] = ldapUsername
+
+			// Update the actualDNToParentUserMap
+			actualDNToParentUserMap[ldapActualDN] = cred.ParentUser
 		}
 		parentUserToCredsMap[cred.ParentUser] = append(parentUserToCredsMap[cred.ParentUser], cred)
-
 	}

 	// 2. Query LDAP server for groups of the LDAP users collected.
-	updatedGroups, err := sys.LDAPConfig.LookupGroupMemberships(parentUsers, parentUserToLDAPUsernameMap)
+	updatedGroups, err := sys.LDAPConfig.LookupGroupMemberships(parentUserActualDNList, actualDNToLDAPUsernameMap)
 	if err != nil {
 		// Log and return on error - perhaps it'll work the next time.
 		iamLogIf(GlobalContext, err)
@@ -1512,8 +1520,9 @@ func (sys *IAMSys) updateGroupMembershipsForLDAP(ctx context.Context) {
 	}

 	// 3. Update creds for those users whose groups are changed
-	for _, parentUser := range parentUsers {
-		currGroupsSet := updatedGroups[parentUser]
+	for _, parentActualDN := range parentUserActualDNList {
+		currGroupsSet := updatedGroups[parentActualDN]
+		parentUser := actualDNToParentUserMap[parentActualDN]
 		currGroups := currGroupsSet.ToSlice()
 		for _, cred := range parentUserToCredsMap[parentUser] {
 			gSet := set.CreateStringSet(cred.Groups...)
@@ -1571,11 +1580,11 @@ func (sys *IAMSys) NormalizeLDAPAccessKeypairs(ctx context.Context, accessKeyMap
 		// server and is under a configured base DN.
 		validatedParent, isUnderBaseDN, err := sys.LDAPConfig.GetValidatedUserDN(conn, parent)
 		if err != nil {
-			collectedErrors = append(collectedErrors, fmt.Errorf("could not validate `%s` exists in LDAP directory: %w", parent, err))
+			collectedErrors = append(collectedErrors, fmt.Errorf("could not validate parent exists in LDAP directory: %w", err))
 			continue
 		}
 		if validatedParent == nil || !isUnderBaseDN {
-			err := fmt.Errorf("DN `%s` was not found in the LDAP directory", parent)
+			err := fmt.Errorf("DN parent was not found in the LDAP directory")
 			collectedErrors = append(collectedErrors, err)
 			continue
 		}
@@ -1590,11 +1599,11 @@ func (sys *IAMSys) NormalizeLDAPAccessKeypairs(ctx context.Context, accessKeyMap
 			// configured base DN.
 			validatedGroup, _, err := sys.LDAPConfig.GetValidatedGroupDN(conn, group)
 			if err != nil {
-				collectedErrors = append(collectedErrors, fmt.Errorf("could not validate `%s` exists in LDAP directory: %w", group, err))
+				collectedErrors = append(collectedErrors, fmt.Errorf("could not validate group exists in LDAP directory: %w", err))
 				continue
 			}
 			if validatedGroup == nil {
-				err := fmt.Errorf("DN `%s` was not found in the LDAP directory", group)
+				err := fmt.Errorf("DN group was not found in the LDAP directory")
 				collectedErrors = append(collectedErrors, err)
 				continue
 			}
@@ -1684,7 +1693,7 @@ func (sys *IAMSys) NormalizeLDAPMappingImport(ctx context.Context, isGroup bool,
 			continue
 		}
 		if validatedDN == nil || !underBaseDN {
-			err := fmt.Errorf("DN `%s` was not found in the LDAP directory", k)
+			err := fmt.Errorf("DN was not found in the LDAP directory")
 			collectedErrors = append(collectedErrors, err)
 			continue
 		}
@@ -2027,7 +2036,7 @@ func (sys *IAMSys) PolicyDBUpdateLDAP(ctx context.Context, isAttach bool,
 		if dnResult == nil {
 			// dn not found - still attempt to detach if provided user is a DN.
 			if !isAttach && sys.LDAPConfig.IsLDAPUserDN(r.User) {
-				dn = r.User
+				dn = sys.LDAPConfig.QuickNormalizeDN(r.User)
 			} else {
 				err = errNoSuchUser
 				return
@@ -2044,7 +2053,7 @@ func (sys *IAMSys) PolicyDBUpdateLDAP(ctx context.Context, isAttach bool,
 		}
 		if dnResult == nil || !underBaseDN {
 			if !isAttach {
-				dn = r.Group
+				dn = sys.LDAPConfig.QuickNormalizeDN(r.Group)
 			} else {
 				err = errNoSuchGroup
 				return
@@ -24,10 +24,8 @@ import (

 	jwtgo "github.com/golang-jwt/jwt/v4"
 	jwtreq "github.com/golang-jwt/jwt/v4/request"
-	"github.com/hashicorp/golang-lru/v2/expirable"
 	"github.com/minio/minio/internal/auth"
 	xjwt "github.com/minio/minio/internal/jwt"
-	"github.com/minio/minio/internal/logger"
 	"github.com/minio/pkg/v3/policy"
 )

@@ -37,8 +35,8 @@ const (
 	// Default JWT token for web handlers is one day.
 	defaultJWTExpiry = 24 * time.Hour

-	// Inter-node JWT token expiry is 15 minutes.
-	defaultInterNodeJWTExpiry = 15 * time.Minute
+	// Inter-node JWT token expiry is 100 years approx.
+	defaultInterNodeJWTExpiry = 100 * 365 * 24 * time.Hour
 )

 var (
@@ -50,17 +48,10 @@ var (
 	errMalformedAuth      = errors.New("Malformed authentication input")
 )

-type cacheKey struct {
-	accessKey, secretKey, audience string
-}
-
-var cacheLRU = expirable.NewLRU[cacheKey, string](1000, nil, 15*time.Second)
-
-func authenticateNode(accessKey, secretKey, audience string) (string, error) {
+func authenticateNode(accessKey, secretKey string) (string, error) {
 	claims := xjwt.NewStandardClaims()
 	claims.SetExpiry(UTCNow().Add(defaultInterNodeJWTExpiry))
 	claims.SetAccessKey(accessKey)
-	claims.SetAudience(audience)

 	jwt := jwtgo.NewWithClaims(jwtgo.SigningMethodHS512, claims)
 	return jwt.SignedString([]byte(secretKey))
@@ -141,27 +132,9 @@ func metricsRequestAuthenticate(req *http.Request) (*xjwt.MapClaims, []string, b
 	return claims, groups, owner, nil
 }

-// newCachedAuthToken returns a token that is cached up to 15 seconds.
-// If globalActiveCred is updated it is reflected at once.
-func newCachedAuthToken() func(audience string) string {
-	fn := func(accessKey, secretKey, audience string) (s string, err error) {
-		k := cacheKey{accessKey: accessKey, secretKey: secretKey, audience: audience}
-
-		var ok bool
-		s, ok = cacheLRU.Get(k)
-		if !ok {
-			s, err = authenticateNode(accessKey, secretKey, audience)
-			if err != nil {
-				return "", err
-			}
-			cacheLRU.Add(k, s)
-		}
-		return s, nil
-	}
-	return func(audience string) string {
-		cred := globalActiveCred
-		token, err := fn(cred.AccessKey, cred.SecretKey, audience)
-		logger.CriticalIf(GlobalContext, err)
-		return token
+// newCachedAuthToken returns the cached token.
+func newCachedAuthToken() func() string {
+	return func() string {
+		return globalNodeAuthToken
 	}
 }
@@ -107,7 +107,7 @@ func BenchmarkParseJWTStandardClaims(b *testing.B) {
 	}

 	creds := globalActiveCred
-	token, err := authenticateNode(creds.AccessKey, creds.SecretKey, "")
+	token, err := authenticateNode(creds.AccessKey, creds.SecretKey)
 	if err != nil {
 		b.Fatal(err)
 	}
@@ -138,7 +138,7 @@ func BenchmarkParseJWTMapClaims(b *testing.B) {
 	}

 	creds := globalActiveCred
-	token, err := authenticateNode(creds.AccessKey, creds.SecretKey, "")
+	token, err := authenticateNode(creds.AccessKey, creds.SecretKey)
 	if err != nil {
 		b.Fatal(err)
 	}
@@ -176,7 +176,7 @@ func BenchmarkAuthenticateNode(b *testing.B) {
 		b.ResetTimer()
 		b.ReportAllocs()
 		for i := 0; i < b.N; i++ {
-			fn(creds.AccessKey, creds.SecretKey, "aud")
+			fn(creds.AccessKey, creds.SecretKey)
 		}
 	})
 	b.Run("cached", func(b *testing.B) {
@@ -184,7 +184,7 @@ func BenchmarkAuthenticateNode(b *testing.B) {
 		b.ResetTimer()
 		b.ReportAllocs()
 		for i := 0; i < b.N; i++ {
-			fn("aud")
+			fn()
 		}
 	})
 }
@@ -24,6 +24,7 @@ import (

 	"github.com/minio/kms-go/kes"
 	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/auth"
 	"github.com/minio/minio/internal/kms"
 	"github.com/minio/minio/internal/logger"
 	"github.com/minio/pkg/v3/policy"
@@ -56,7 +57,7 @@ func (a kmsAPIHandlers) KMSStatusHandler(w http.ResponseWriter, r *http.Request)
 	writeSuccessResponseJSON(w, resp)
 }

-// KMSMetricsHandler - POST /minio/kms/v1/metrics
+// KMSMetricsHandler - GET /minio/kms/v1/metrics
 func (a kmsAPIHandlers) KMSMetricsHandler(w http.ResponseWriter, r *http.Request) {
 	ctx := newContext(r, w, "KMSMetrics")
 	defer logger.AuditLog(ctx, w, r, mustGetClaimsFromToken(r))
@@ -83,7 +84,7 @@ func (a kmsAPIHandlers) KMSMetricsHandler(w http.ResponseWriter, r *http.Request
 	}
 }

-// KMSAPIsHandler - POST /minio/kms/v1/apis
+// KMSAPIsHandler - GET /minio/kms/v1/apis
 func (a kmsAPIHandlers) KMSAPIsHandler(w http.ResponseWriter, r *http.Request) {
 	ctx := newContext(r, w, "KMSAPIs")
 	defer logger.AuditLog(ctx, w, r, mustGetClaimsFromToken(r))
@@ -114,7 +115,7 @@ type versionResponse struct {
 	Version string `json:"version"`
 }

-// KMSVersionHandler - POST /minio/kms/v1/version
+// KMSVersionHandler - GET /minio/kms/v1/version
 func (a kmsAPIHandlers) KMSVersionHandler(w http.ResponseWriter, r *http.Request) {
 	ctx := newContext(r, w, "KMSVersion")
 	defer logger.AuditLog(ctx, w, r, mustGetClaimsFromToken(r))
@@ -159,7 +160,20 @@ func (a kmsAPIHandlers) KMSCreateKeyHandler(w http.ResponseWriter, r *http.Reque
 		return
 	}

-	if err := GlobalKMS.CreateKey(ctx, &kms.CreateKeyRequest{Name: r.Form.Get("key-id")}); err != nil {
+	keyID := r.Form.Get("key-id")
+
+	// Ensure policy allows the user to create this key name
+	cred, owner, s3Err := validateAdminSignature(ctx, r, "")
+	if s3Err != ErrNone {
+		writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
+		return
+	}
+	if !checkKMSActionAllowed(r, owner, cred, policy.KMSCreateKeyAction, keyID) {
+		writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
+		return
+	}
+
+	if err := GlobalKMS.CreateKey(ctx, &kms.CreateKeyRequest{Name: keyID}); err != nil {
 		writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
 		return
 	}
@@ -171,6 +185,9 @@ func (a kmsAPIHandlers) KMSListKeysHandler(w http.ResponseWriter, r *http.Reques
 	ctx := newContext(r, w, "KMSListKeys")
 	defer logger.AuditLog(ctx, w, r, mustGetClaimsFromToken(r))

+	// This only checks if the action (kms:ListKeys) is allowed, it does not check
+	// each key name against the policy's Resources. We check that below, once
+	// we have the list of key names from the KMS.
 	objectAPI, _ := validateAdminReq(ctx, w, r, policy.KMSListKeysAction)
 	if objectAPI == nil {
 		return
@@ -180,7 +197,7 @@ func (a kmsAPIHandlers) KMSListKeysHandler(w http.ResponseWriter, r *http.Reques
 		writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrKMSNotConfigured), r.URL)
 		return
 	}
-	names, _, err := GlobalKMS.ListKeyNames(ctx, &kms.ListRequest{
+	allKeyNames, _, err := GlobalKMS.ListKeyNames(ctx, &kms.ListRequest{
 		Prefix: r.Form.Get("pattern"),
 	})
 	if err != nil {
@@ -188,8 +205,24 @@ func (a kmsAPIHandlers) KMSListKeysHandler(w http.ResponseWriter, r *http.Reques
 		return
 	}

-	values := make([]kes.KeyInfo, 0, len(names))
-	for _, name := range names {
+	// Get the cred and owner for checking authz below.
+	cred, owner, s3Err := validateAdminSignature(ctx, r, "")
+	if s3Err != ErrNone {
+		writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
+		return
+	}
+
+	// Now we have all the key names, for each of them, check whether the policy grants permission for
+	// the user to list it.
+	keyNames := []string{}
+	for _, name := range allKeyNames {
+		if checkKMSActionAllowed(r, owner, cred, policy.KMSListKeysAction, name) {
+			keyNames = append(keyNames, name)
+		}
+	}
+
+	values := make([]kes.KeyInfo, 0, len(keyNames))
+	for _, name := range keyNames {
 		values = append(values, kes.KeyInfo{
 			Name: name,
 		})
@@ -224,6 +257,17 @@ func (a kmsAPIHandlers) KMSKeyStatusHandler(w http.ResponseWriter, r *http.Reque
 		KeyID: keyID,
 	}

+	// Ensure policy allows the user to get this key's status
+	cred, owner, s3Err := validateAdminSignature(ctx, r, "")
+	if s3Err != ErrNone {
+		writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL)
+		return
+	}
+	if !checkKMSActionAllowed(r, owner, cred, policy.KMSKeyStatusAction, keyID) {
+		writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), r.URL)
+		return
+	}
+
 	kmsContext := kms.Context{"MinIO admin API": "KMSKeyStatusHandler"} // Context for a test key operation
 	// 1. Generate a new key using the KMS.
 	key, err := GlobalKMS.GenerateKey(ctx, &kms.GenerateKeyRequest{Name: keyID, AssociatedData: kmsContext})
@@ -274,3 +318,16 @@ func (a kmsAPIHandlers) KMSKeyStatusHandler(w http.ResponseWriter, r *http.Reque
 	}
 	writeSuccessResponseJSON(w, resp)
 }
+
+// checkKMSActionAllowed checks for authorization for a specific action on a resource.
+func checkKMSActionAllowed(r *http.Request, owner bool, cred auth.Credentials, action policy.KMSAction, resource string) bool {
+	return globalIAMSys.IsAllowed(policy.Args{
+		AccountName:     cred.AccessKey,
+		Groups:          cred.Groups,
+		Action:          policy.Action(action),
+		ConditionValues: getConditionValues(r, "", cred),
+		IsOwner:         owner,
+		Claims:          cred.Claims,
+		BucketName:      resource, // overloading BucketName as that's what the policy engine uses to assemble a Resource.
+	})
+}
@@ -0,0 +1,851 @@
+// Copyright (c) 2015-2024 MinIO, Inc.
+//
+// This file is part of MinIO Object Storage stack
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"testing"
+
+	"github.com/minio/madmin-go/v3"
+	"github.com/minio/minio/internal/kms"
+	"github.com/minio/pkg/v3/policy"
+)
+
+const (
+	// KMS API paths
+	// For example: /minio/kms/v1/key/list?pattern=*
+	kmsURL           = kmsPathPrefix + kmsAPIVersionPrefix
+	kmsStatusPath    = kmsURL + "/status"
+	kmsMetricsPath   = kmsURL + "/metrics"
+	kmsAPIsPath      = kmsURL + "/apis"
+	kmsVersionPath   = kmsURL + "/version"
+	kmsKeyCreatePath = kmsURL + "/key/create"
+	kmsKeyListPath   = kmsURL + "/key/list"
+	kmsKeyStatusPath = kmsURL + "/key/status"
+
+	// Admin API paths
+	// For example: /minio/admin/v3/kms/status
+	adminURL              = adminPathPrefix + adminAPIVersionPrefix
+	kmsAdminStatusPath    = adminURL + "/kms/status"
+	kmsAdminKeyStatusPath = adminURL + "/kms/key/status"
+	kmsAdminKeyCreate     = adminURL + "/kms/key/create"
+)
+
+const (
+	userAccessKey = "miniofakeuseraccesskey"
+	userSecretKey = "miniofakeusersecret"
+)
+
+type kmsTestCase struct {
+	name   string
+	method string
+	path   string
+	query  map[string]string
+
+	// User credentials and policy for request
+	policy string
+	asRoot bool
+
+	// Wanted in response.
+	wantStatusCode int
+	wantKeyNames   []string
+	wantResp       []string
+}
+
+func TestKMSHandlersCreateKey(t *testing.T) {
+	adminTestBed, tearDown := setupKMSTest(t, true)
+	defer tearDown()
+
+	tests := []kmsTestCase{
+		// Create key test
+		{
+			name:   "create key as user with no policy want forbidden",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "new-test-key"},
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "create key as user with no resources specified want success",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "new-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:CreateKey"] }`,
+
+			wantStatusCode: http.StatusOK,
+		},
+		{
+			name:   "create key as user set policy to allow want success",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "second-new-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:CreateKey"],
+				"Resource": ["arn:minio:kms:::second-new-test-*"] }`,
+
+			wantStatusCode: http.StatusOK,
+		},
+		{
+			name:   "create key as user set policy to non matching resource want forbidden",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "third-new-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:CreateKey"],
+				"Resource": ["arn:minio:kms:::non-matching-key-name"] }`,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+	}
+	for testNum, test := range tests {
+		t.Run(fmt.Sprintf("%d %s", testNum+1, test.name), func(t *testing.T) {
+			execKMSTest(t, test, adminTestBed)
+		})
+	}
+}
+
+func TestKMSHandlersKeyStatus(t *testing.T) {
+	adminTestBed, tearDown := setupKMSTest(t, true)
+	defer tearDown()
+
+	tests := []kmsTestCase{
+		{
+			name:   "create a first key root user",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+		},
+		{
+			name:   "key status as root want success",
+			method: http.MethodGet,
+			path:   kmsKeyStatusPath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"abc-test-key"},
+		},
+		{
+			name:   "key status as user no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsKeyStatusPath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "key status as user legacy no resources specified want success",
+			method: http.MethodGet,
+			path:   kmsKeyStatusPath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:KeyStatus"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"abc-test-key"},
+		},
+		{
+			name:   "key status as user set policy to allow only one key",
+			method: http.MethodGet,
+			path:   kmsKeyStatusPath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:KeyStatus"],
+					"Resource": ["arn:minio:kms:::abc-test-*"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"abc-test-key"},
+		},
+		{
+			name:   "key status as user set policy to allow non-matching key",
+			method: http.MethodGet,
+			path:   kmsKeyStatusPath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:KeyStatus"],
+					"Resource": ["arn:minio:kms:::xyz-test-key"] }`,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+	}
+	for testNum, test := range tests {
+		t.Run(fmt.Sprintf("%d %s", testNum+1, test.name), func(t *testing.T) {
+			execKMSTest(t, test, adminTestBed)
+		})
+	}
+}
+
+func TestKMSHandlersAPIs(t *testing.T) {
+	adminTestBed, tearDown := setupKMSTest(t, true)
+	defer tearDown()
+
+	tests := []kmsTestCase{
+		// Version test
+		{
+			name:   "version as root want success",
+			method: http.MethodGet,
+			path:   kmsVersionPath,
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"version"},
+		},
+		{
+			name:   "version as user with no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsVersionPath,
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "version as user with policy ignores resource want success",
+			method: http.MethodGet,
+			path:   kmsVersionPath,
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:Version"],
+				"Resource": ["arn:minio:kms:::does-not-matter-it-is-ignored"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"version"},
+		},
+
+		// APIs test
+		{
+			name:   "apis as root want success",
+			method: http.MethodGet,
+			path:   kmsAPIsPath,
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"stub/path"},
+		},
+		{
+			name:   "apis as user with no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsAPIsPath,
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "apis as user with policy ignores resource want success",
+			method: http.MethodGet,
+			path:   kmsAPIsPath,
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:API"],
+				"Resource": ["arn:minio:kms:::does-not-matter-it-is-ignored"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"stub/path"},
+		},
+
+		// Metrics test
+		{
+			name:   "metrics as root want success",
+			method: http.MethodGet,
+			path:   kmsMetricsPath,
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"kms"},
+		},
+		{
+			name:   "metrics as user with no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsMetricsPath,
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "metrics as user with policy ignores resource want success",
+			method: http.MethodGet,
+			path:   kmsMetricsPath,
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:Metrics"],
+				"Resource": ["arn:minio:kms:::does-not-matter-it-is-ignored"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"kms"},
+		},
+
+		// Status tests
+		{
+			name:   "status as root want success",
+			method: http.MethodGet,
+			path:   kmsStatusPath,
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"MinIO builtin"},
+		},
+		{
+			name:   "status as user with no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsStatusPath,
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "status as user with policy ignores resource want success",
+			method: http.MethodGet,
+			path:   kmsStatusPath,
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:Status"],
+				"Resource": ["arn:minio:kms:::does-not-matter-it-is-ignored"]}`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"MinIO builtin"},
+		},
+	}
+	for testNum, test := range tests {
+		t.Run(fmt.Sprintf("%d %s", testNum+1, test.name), func(t *testing.T) {
+			execKMSTest(t, test, adminTestBed)
+		})
+	}
+}
+
+func TestKMSHandlersListKeys(t *testing.T) {
+	adminTestBed, tearDown := setupKMSTest(t, true)
+	defer tearDown()
+
+	tests := []kmsTestCase{
+		{
+			name:   "create a first key root user",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+		},
+		{
+			name:   "create a second key root user",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "xyz-test-key"},
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+		},
+
+		// List keys tests
+		{
+			name:   "list keys as root want all to be returned",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantKeyNames:   []string{"default-test-key", "abc-test-key", "xyz-test-key"},
+		},
+		{
+			name:   "list keys as user with no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "list keys as user with no resources specified want success",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["kms:ListKeys"]
+			}`,
+
+			wantStatusCode: http.StatusOK,
+			wantKeyNames:   []string{"default-test-key", "abc-test-key", "xyz-test-key"},
+		},
+		{
+			name:   "list keys as user set policy resource to allow only one key",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:ListKeys"],
+					"Resource": ["arn:minio:kms:::abc*"]}`,
+
+			wantStatusCode: http.StatusOK,
+			wantKeyNames:   []string{"abc-test-key"},
+		},
+		{
+			name:   "list keys as user set policy to allow only one key, use pattern that includes correct key",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "abc*"},
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:ListKeys"],
+					"Resource": ["arn:minio:kms:::abc*"]}`,
+
+			wantStatusCode: http.StatusOK,
+			wantKeyNames:   []string{"abc-test-key"},
+		},
+		{
+			name:   "list keys as user set policy to allow only one key, use pattern that excludes correct key",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "xyz*"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:ListKeys"],
+					"Resource": ["arn:minio:kms:::abc*"]}`,
+
+			wantStatusCode: http.StatusOK,
+			wantKeyNames:   []string{},
+		},
+		{
+			name:   "list keys as user set policy that has no matching key resources",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+					"Action": ["kms:ListKeys"],
+					"Resource": ["arn:minio:kms:::nonematch*"]}`,
+
+			wantStatusCode: http.StatusOK,
+			wantKeyNames:   []string{},
+		},
+		{
+			name:   "list keys as user set policy that allows listing but denies specific keys",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+			asRoot: false,
+
+			// It looks like this should allow listing any key that isn't "default-test-key", however
+			// the policy engine matches all Deny statements first, without regard to Resources (for KMS).
+			// This is for backwards compatibility where historically KMS statements ignored Resources.
+			policy: `{
+						"Effect": "Allow",
+						"Action": ["kms:ListKeys"]
+					},{
+						"Effect": "Deny",
+						"Action": ["kms:ListKeys"],
+						"Resource": ["arn:minio:kms:::default-test-key"]
+					}`,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+	}
+
+	for testNum, test := range tests {
+		t.Run(fmt.Sprintf("%d %s", testNum+1, test.name), func(t *testing.T) {
+			execKMSTest(t, test, adminTestBed)
+		})
+	}
+}
+
+func TestKMSHandlerAdminAPI(t *testing.T) {
+	adminTestBed, tearDown := setupKMSTest(t, true)
+	defer tearDown()
+
+	tests := []kmsTestCase{
+		// Create key tests
+		{
+			name:   "create a key root user",
+			method: http.MethodPost,
+			path:   kmsAdminKeyCreate,
+			query:  map[string]string{"key-id": "abc-test-key"},
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+		},
+		{
+			name:   "create key as user with no policy want forbidden",
+			method: http.MethodPost,
+			path:   kmsAdminKeyCreate,
+			query:  map[string]string{"key-id": "new-test-key"},
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "create key as user with no resources specified want success",
+			method: http.MethodPost,
+			path:   kmsAdminKeyCreate,
+			query:  map[string]string{"key-id": "new-test-key"},
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["admin:KMSCreateKey"] }`,
+
+			wantStatusCode: http.StatusOK,
+		},
+		{
+			name:   "create key as user set policy to non matching resource want success",
+			method: http.MethodPost,
+			path:   kmsAdminKeyCreate,
+			query:  map[string]string{"key-id": "third-new-test-key"},
+			asRoot: false,
+
+			// Admin actions ignore Resources
+			policy: `{"Effect": "Allow",
+				"Action": ["admin:KMSCreateKey"],
+				"Resource": ["arn:minio:kms:::this-is-disregarded"] }`,
+
+			wantStatusCode: http.StatusOK,
+		},
+
+		// Status tests
+		{
+			name:   "status as root want success",
+			method: http.MethodPost,
+			path:   kmsAdminStatusPath,
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"MinIO builtin"},
+		},
+		{
+			name:   "status as user with no policy want forbidden",
+			method: http.MethodPost,
+			path:   kmsAdminStatusPath,
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "status as user with policy ignores resource want success",
+			method: http.MethodPost,
+			path:   kmsAdminStatusPath,
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["admin:KMSKeyStatus"],
+				"Resource": ["arn:minio:kms:::does-not-matter-it-is-ignored"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"MinIO builtin"},
+		},
+
+		// Key status tests
+		{
+			name:   "key status as root want success",
+			method: http.MethodGet,
+			path:   kmsAdminKeyStatusPath,
+			asRoot: true,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"key-id"},
+		},
+		{
+			name:   "key status as user with no policy want forbidden",
+			method: http.MethodGet,
+			path:   kmsAdminKeyStatusPath,
+			asRoot: false,
+
+			wantStatusCode: http.StatusForbidden,
+			wantResp:       []string{"AccessDenied"},
+		},
+		{
+			name:   "key status as user with policy ignores resource want success",
+			method: http.MethodGet,
+			path:   kmsAdminKeyStatusPath,
+			asRoot: false,
+
+			policy: `{"Effect": "Allow",
+				"Action": ["admin:KMSKeyStatus"],
+				"Resource": ["arn:minio:kms:::does-not-matter-it-is-ignored"] }`,
+
+			wantStatusCode: http.StatusOK,
+			wantResp:       []string{"key-id"},
+		},
+	}
+
+	for testNum, test := range tests {
+		t.Run(fmt.Sprintf("%d %s", testNum+1, test.name), func(t *testing.T) {
+			execKMSTest(t, test, adminTestBed)
+		})
+	}
+}
+
+// execKMSTest runs a single test case for KMS handlers
+func execKMSTest(t *testing.T, test kmsTestCase, adminTestBed *adminErasureTestBed) {
+	var accessKey, secretKey string
+	if test.asRoot {
+		accessKey, secretKey = globalActiveCred.AccessKey, globalActiveCred.SecretKey
+	} else {
+		setupKMSUser(t, userAccessKey, userSecretKey, test.policy)
+		accessKey = userAccessKey
+		secretKey = userSecretKey
+	}
+
+	req := buildKMSRequest(t, test.method, test.path, accessKey, secretKey, test.query)
+	rec := httptest.NewRecorder()
+	adminTestBed.router.ServeHTTP(rec, req)
+
+	t.Logf("HTTP req: %s, resp code: %d, resp body: %s", req.URL.String(), rec.Code, rec.Body.String())
+
+	// Check status code
+	if rec.Code != test.wantStatusCode {
+		t.Errorf("want status code %d, got %d", test.wantStatusCode, rec.Code)
+	}
+
+	// Check returned key list is correct
+	if test.wantKeyNames != nil {
+		gotKeyNames := keyNamesFromListKeysResp(t, rec.Body.Bytes())
+		if len(test.wantKeyNames) != len(gotKeyNames) {
+			t.Fatalf("want keys len: %d, got len: %d", len(test.wantKeyNames), len(gotKeyNames))
+		}
+		for i, wantKeyName := range test.wantKeyNames {
+			if gotKeyNames[i] != wantKeyName {
+				t.Fatalf("want key name %s, in position %d, got %s", wantKeyName, i, gotKeyNames[i])
+			}
+		}
+	}
+
+	// Check generic text in the response
+	if test.wantResp != nil {
+		for _, want := range test.wantResp {
+			if !strings.Contains(rec.Body.String(), want) {
+				t.Fatalf("want response to contain %s, got %s", want, rec.Body.String())
+			}
+		}
+	}
+}
+
+// TestKMSHandlerNotConfiguredOrInvalidCreds tests KMS handlers for situations where KMS is not configured
+// or invalid credentials are provided.
+func TestKMSHandlerNotConfiguredOrInvalidCreds(t *testing.T) {
+	adminTestBed, tearDown := setupKMSTest(t, false)
+	defer tearDown()
+
+	tests := []struct {
+		name   string
+		method string
+		path   string
+		query  map[string]string
+	}{
+		{
+			name:   "GET status",
+			method: http.MethodGet,
+			path:   kmsStatusPath,
+		},
+		{
+			name:   "GET metrics",
+			method: http.MethodGet,
+			path:   kmsMetricsPath,
+		},
+		{
+			name:   "GET apis",
+			method: http.MethodGet,
+			path:   kmsAPIsPath,
+		},
+		{
+			name:   "GET version",
+			method: http.MethodGet,
+			path:   kmsVersionPath,
+		},
+		{
+			name:   "POST key create",
+			method: http.MethodPost,
+			path:   kmsKeyCreatePath,
+			query:  map[string]string{"key-id": "master-key-id"},
+		},
+		{
+			name:   "GET key list",
+			method: http.MethodGet,
+			path:   kmsKeyListPath,
+			query:  map[string]string{"pattern": "*"},
+		},
+		{
+			name:   "GET key status",
+			method: http.MethodGet,
+			path:   kmsKeyStatusPath,
+			query:  map[string]string{"key-id": "master-key-id"},
+		},
+	}
+
+	// Test when the GlobalKMS is not configured
+	for _, test := range tests {
+		t.Run(test.name+" not configured", func(t *testing.T) {
+			req := buildKMSRequest(t, test.method, test.path, "", "", test.query)
+			rec := httptest.NewRecorder()
+			adminTestBed.router.ServeHTTP(rec, req)
+			if rec.Code != http.StatusNotImplemented {
+				t.Errorf("want status code %d, got %d", http.StatusNotImplemented, rec.Code)
+			}
+		})
+	}
+
+	// Test when the GlobalKMS is configured but the credentials are invalid
+	GlobalKMS = kms.NewStub("default-test-key")
+	for _, test := range tests {
+		t.Run(test.name+" invalid credentials", func(t *testing.T) {
+			req := buildKMSRequest(t, test.method, test.path, userAccessKey, userSecretKey, test.query)
+			rec := httptest.NewRecorder()
+			adminTestBed.router.ServeHTTP(rec, req)
+			if rec.Code != http.StatusForbidden {
+				t.Errorf("want status code %d, got %d", http.StatusForbidden, rec.Code)
+			}
+		})
+	}
+}
+
+func setupKMSTest(t *testing.T, enableKMS bool) (*adminErasureTestBed, func()) {
+	adminTestBed, err := prepareAdminErasureTestBed(context.Background())
+	if err != nil {
+		t.Fatal(err)
+	}
+	registerKMSRouter(adminTestBed.router)
+
+	if enableKMS {
+		GlobalKMS = kms.NewStub("default-test-key")
+	}
+
+	tearDown := func() {
+		adminTestBed.TearDown()
+		GlobalKMS = nil
+	}
+	return adminTestBed, tearDown
+}
+
+func buildKMSRequest(t *testing.T, method, path, accessKey, secretKey string, query map[string]string) *http.Request {
+	if len(query) > 0 {
+		queryVal := url.Values{}
+		for k, v := range query {
+			queryVal.Add(k, v)
+		}
+		path = path + "?" + queryVal.Encode()
+	}
+
+	if accessKey == "" && secretKey == "" {
+		accessKey = globalActiveCred.AccessKey
+		secretKey = globalActiveCred.SecretKey
+	}
+
+	req, err := newTestSignedRequestV4(method, path, 0, nil, accessKey, secretKey, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return req
+}
+
+// setupKMSUser is a test helper that creates a new user with the provided access key and secret key
+// and applies the given policy to the user.
+func setupKMSUser(t *testing.T, accessKey, secretKey, p string) {
+	ctx := context.Background()
+	createUserParams := madmin.AddOrUpdateUserReq{
+		SecretKey: secretKey,
+		Status:    madmin.AccountEnabled,
+	}
+	_, err := globalIAMSys.CreateUser(ctx, accessKey, createUserParams)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	testKMSPolicyName := "testKMSPolicy"
+	if p != "" {
+		p = `{"Version":"2012-10-17","Statement":[` + p + `]}`
+		policyData, err := policy.ParseConfig(strings.NewReader(p))
+		if err != nil {
+			t.Fatal(err)
+		}
+		_, err = globalIAMSys.SetPolicy(ctx, testKMSPolicyName, *policyData)
+		if err != nil {
+			t.Fatal(err)
+		}
+		_, err = globalIAMSys.PolicyDBSet(ctx, accessKey, testKMSPolicyName, regUser, false)
+		if err != nil {
+			t.Fatal(err)
+		}
+	} else {
+		err = globalIAMSys.DeletePolicy(ctx, testKMSPolicyName, false)
+		if err != nil {
+			t.Fatal(err)
+		}
+		_, err = globalIAMSys.PolicyDBSet(ctx, accessKey, "", regUser, false)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+func keyNamesFromListKeysResp(t *testing.T, b []byte) []string {
+	var keyInfos []madmin.KMSKeyInfo
+	err := json.Unmarshal(b, &keyInfos)
+	if err != nil {
+		t.Fatalf("cannot unmarshal '%s', err: %v", b, err)
+	}
+
+	var gotKeyNames []string
+	for _, keyInfo := range keyInfos {
+		gotKeyNames = append(gotKeyNames, keyInfo.Name)
+	}
+	return gotKeyNames
+}
@@ -31,16 +31,16 @@ import (

 // lockRequesterInfo stores various info from the client for each lock that is requested.
 type lockRequesterInfo struct {
-	Name            string    // name of the resource lock was requested for
-	Writer          bool      // Bool whether write or read lock.
-	UID             string    // UID to uniquely identify request of client.
-	Timestamp       time.Time // Timestamp set at the time of initialization.
-	TimeLastRefresh time.Time // Timestamp for last lock refresh.
-	Source          string    // Contains line, function and filename requesting the lock.
-	Group           bool      // indicates if it was a group lock.
-	Owner           string    // Owner represents the UUID of the owner who originally requested the lock.
-	Quorum          int       // Quorum represents the quorum required for this lock to be active.
-	idx             int       `msg:"-"` // index of the lock in the lockMap.
+	Name            string // name of the resource lock was requested for
+	Writer          bool   // Bool whether write or read lock.
+	UID             string // UID to uniquely identify request of client.
+	Timestamp       int64  // Timestamp set at the time of initialization.
+	TimeLastRefresh int64  // Timestamp for last lock refresh.
+	Source          string // Contains line, function and filename requesting the lock.
+	Group           bool   // indicates if it was a group lock.
+	Owner           string // Owner represents the UUID of the owner who originally requested the lock.
+	Quorum          int    // Quorum represents the quorum required for this lock to be active.
+	idx             int    `msg:"-"` // index of the lock in the lockMap.
 }

 // isWriteLock returns whether the lock is a write or read lock.
@@ -87,6 +87,7 @@ func (l *localLocker) Lock(ctx context.Context, args dsync.LockArgs) (reply bool

 	// No locks held on the all resources, so claim write
 	// lock on all resources at once.
+	now := UTCNow()
 	for i, resource := range args.Resources {
 		l.lockMap[resource] = []lockRequesterInfo{
 			{
@@ -95,10 +96,10 @@ func (l *localLocker) Lock(ctx context.Context, args dsync.LockArgs) (reply bool
 				Source:          args.Source,
 				Owner:           args.Owner,
 				UID:             args.UID,
-				Timestamp:       UTCNow(),
-				TimeLastRefresh: UTCNow(),
+				Timestamp:       now.UnixNano(),
+				TimeLastRefresh: now.UnixNano(),
 				Group:           len(args.Resources) > 1,
-				Quorum:          args.Quorum,
+				Quorum:          *args.Quorum,
 				idx:             i,
 			},
 		}
@@ -108,7 +109,7 @@ func (l *localLocker) Lock(ctx context.Context, args dsync.LockArgs) (reply bool
 }

 func formatUUID(s string, idx int) string {
-	return s + strconv.Itoa(idx)
+	return concat(s, strconv.Itoa(idx))
 }

 func (l *localLocker) Unlock(_ context.Context, args dsync.LockArgs) (reply bool, err error) {
@@ -167,17 +168,19 @@ func (l *localLocker) RLock(ctx context.Context, args dsync.LockArgs) (reply boo
 	l.mutex.Lock()
 	defer l.mutex.Unlock()
 	resource := args.Resources[0]
+	now := UTCNow()
 	lrInfo := lockRequesterInfo{
 		Name:            resource,
 		Writer:          false,
 		Source:          args.Source,
 		Owner:           args.Owner,
 		UID:             args.UID,
-		Timestamp:       UTCNow(),
-		TimeLastRefresh: UTCNow(),
-		Quorum:          args.Quorum,
+		Timestamp:       now.UnixNano(),
+		TimeLastRefresh: now.UnixNano(),
+		Quorum:          *args.Quorum,
 	}
-	if lri, ok := l.lockMap[resource]; ok {
+	lri, ok := l.lockMap[resource]
+	if ok {
 		if reply = !isWriteLock(lri); reply {
 			// Unless there is a write lock
 			l.lockMap[resource] = append(l.lockMap[resource], lrInfo)
@@ -351,9 +354,10 @@ func (l *localLocker) Refresh(ctx context.Context, args dsync.LockArgs) (refresh
 				delete(l.lockUID, formatUUID(args.UID, idx))
 				return idx > 0, nil
 			}
+			now := UTCNow()
 			for i := range lris {
 				if lris[i].UID == args.UID {
-					lris[i].TimeLastRefresh = UTCNow()
+					lris[i].TimeLastRefresh = now.UnixNano()
 				}
 			}
 			idx++
@@ -376,7 +380,7 @@ func (l *localLocker) expireOldLocks(interval time.Duration) {
 		modified := false
 		for i := 0; i < len(lris); {
 			lri := &lris[i]
-			if time.Since(lri.TimeLastRefresh) > interval {
+			if time.Since(time.Unix(0, lri.TimeLastRefresh)) > interval {
 				delete(l.lockUID, formatUUID(lri.UID, lri.idx))
 				if len(lris) == 1 {
 					// Remove the write lock.
@@ -204,13 +204,13 @@ func (z *lockRequesterInfo) DecodeMsg(dc *msgp.Reader) (err error) {
 				return
 			}
 		case "Timestamp":
-			z.Timestamp, err = dc.ReadTime()
+			z.Timestamp, err = dc.ReadInt64()
 			if err != nil {
 				err = msgp.WrapError(err, "Timestamp")
 				return
 			}
 		case "TimeLastRefresh":
-			z.TimeLastRefresh, err = dc.ReadTime()
+			z.TimeLastRefresh, err = dc.ReadInt64()
 			if err != nil {
 				err = msgp.WrapError(err, "TimeLastRefresh")
 				return
@@ -288,7 +288,7 @@ func (z *lockRequesterInfo) EncodeMsg(en *msgp.Writer) (err error) {
 	if err != nil {
 		return
 	}
-	err = en.WriteTime(z.Timestamp)
+	err = en.WriteInt64(z.Timestamp)
 	if err != nil {
 		err = msgp.WrapError(err, "Timestamp")
 		return
@@ -298,7 +298,7 @@ func (z *lockRequesterInfo) EncodeMsg(en *msgp.Writer) (err error) {
 	if err != nil {
 		return
 	}
-	err = en.WriteTime(z.TimeLastRefresh)
+	err = en.WriteInt64(z.TimeLastRefresh)
 	if err != nil {
 		err = msgp.WrapError(err, "TimeLastRefresh")
 		return
@@ -361,10 +361,10 @@ func (z *lockRequesterInfo) MarshalMsg(b []byte) (o []byte, err error) {
 	o = msgp.AppendString(o, z.UID)
 	// string "Timestamp"
 	o = append(o, 0xa9, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70)
-	o = msgp.AppendTime(o, z.Timestamp)
+	o = msgp.AppendInt64(o, z.Timestamp)
 	// string "TimeLastRefresh"
 	o = append(o, 0xaf, 0x54, 0x69, 0x6d, 0x65, 0x4c, 0x61, 0x73, 0x74, 0x52, 0x65, 0x66, 0x72, 0x65, 0x73, 0x68)
-	o = msgp.AppendTime(o, z.TimeLastRefresh)
+	o = msgp.AppendInt64(o, z.TimeLastRefresh)
 	// string "Source"
 	o = append(o, 0xa6, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65)
 	o = msgp.AppendString(o, z.Source)
@@ -417,13 +417,13 @@ func (z *lockRequesterInfo) UnmarshalMsg(bts []byte) (o []byte, err error) {
 				return
 			}
 		case "Timestamp":
-			z.Timestamp, bts, err = msgp.ReadTimeBytes(bts)
+			z.Timestamp, bts, err = msgp.ReadInt64Bytes(bts)
 			if err != nil {
 				err = msgp.WrapError(err, "Timestamp")
 				return
 			}
 		case "TimeLastRefresh":
-			z.TimeLastRefresh, bts, err = msgp.ReadTimeBytes(bts)
+			z.TimeLastRefresh, bts, err = msgp.ReadInt64Bytes(bts)
 			if err != nil {
 				err = msgp.WrapError(err, "TimeLastRefresh")
 				return
@@ -466,7 +466,7 @@ func (z *lockRequesterInfo) UnmarshalMsg(bts []byte) (o []byte, err error) {

 // Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
 func (z *lockRequesterInfo) Msgsize() (s int) {
-	s = 1 + 5 + msgp.StringPrefixSize + len(z.Name) + 7 + msgp.BoolSize + 4 + msgp.StringPrefixSize + len(z.UID) + 10 + msgp.TimeSize + 16 + msgp.TimeSize + 7 + msgp.StringPrefixSize + len(z.Source) + 6 + msgp.BoolSize + 6 + msgp.StringPrefixSize + len(z.Owner) + 7 + msgp.IntSize
+	s = 1 + 5 + msgp.StringPrefixSize + len(z.Name) + 7 + msgp.BoolSize + 4 + msgp.StringPrefixSize + len(z.UID) + 10 + msgp.Int64Size + 16 + msgp.Int64Size + 7 + msgp.StringPrefixSize + len(z.Source) + 6 + msgp.BoolSize + 6 + msgp.StringPrefixSize + len(z.Owner) + 7 + msgp.IntSize
 	return
 }

@@ -32,6 +32,7 @@ import (
 func TestLocalLockerExpire(t *testing.T) {
 	wResources := make([]string, 1000)
 	rResources := make([]string, 1000)
+	quorum := 0
 	l := newLocker()
 	ctx := context.Background()
 	for i := range wResources {
@@ -40,7 +41,7 @@ func TestLocalLockerExpire(t *testing.T) {
 			Resources: []string{mustGetUUID()},
 			Source:    t.Name(),
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.Lock(ctx, arg)
 		if err != nil {
@@ -58,14 +59,14 @@ func TestLocalLockerExpire(t *testing.T) {
 			Resources: []string{name},
 			Source:    t.Name(),
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.RLock(ctx, arg)
 		if err != nil {
 			t.Fatal(err)
 		}
 		if !ok {
-			t.Fatal("did not get write lock")
+			t.Fatal("did not get read lock")
 		}
 		// RLock twice
 		ok, err = l.RLock(ctx, arg)
@@ -112,6 +113,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 	rUIDs := make([]string, 0, n*2)
 	l := newLocker()
 	ctx := context.Background()
+	quorum := 0
 	for i := range wResources {
 		names := [m]string{}
 		for j := range names {
@@ -123,7 +125,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 			Resources: names[:],
 			Source:    t.Name(),
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.Lock(ctx, arg)
 		if err != nil {
@@ -144,7 +146,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 			Resources: []string{name},
 			Source:    t.Name(),
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.RLock(ctx, arg)
 		if err != nil {
@@ -183,7 +185,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 			Resources: []string{name},
 			Source:    t.Name(),
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.RUnlock(ctx, arg)
 		if err != nil {
@@ -203,6 +205,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 	if len(l.lockUID) != len(rResources)+len(wResources)*m {
 		t.Fatalf("lockUID len, got %d, want %d + %d", len(l.lockUID), len(rResources), len(wResources)*m)
 	}
+
 	// RUnlock again, different uids
 	for i, name := range rResources {
 		arg := dsync.LockArgs{
@@ -210,7 +213,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 			Resources: []string{name},
 			Source:    "minio",
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.RUnlock(ctx, arg)
 		if err != nil {
@@ -238,7 +241,7 @@ func TestLocalLockerUnlock(t *testing.T) {
 			Resources: names[:],
 			Source:    "minio",
 			Owner:     "owner",
-			Quorum:    0,
+			Quorum:    &quorum,
 		}
 		ok, err := l.Unlock(ctx, arg)
 		if err != nil {
@@ -261,6 +264,7 @@ func TestLocalLockerUnlock(t *testing.T) {

 func Test_localLocker_expireOldLocksExpire(t *testing.T) {
 	rng := rand.New(rand.NewSource(0))
+	quorum := 0
 	// Numbers of unique locks
 	for _, locks := range []int{100, 1000, 1e6} {
 		if testing.Short() && locks > 100 {
@@ -289,7 +293,7 @@ func Test_localLocker_expireOldLocksExpire(t *testing.T) {
 								Resources: res,
 								Source:    hex.EncodeToString(tmp[:8]),
 								Owner:     hex.EncodeToString(tmp[8:]),
-								Quorum:    0,
+								Quorum:    &quorum,
 							})
 							if !ok || err != nil {
 								t.Fatal("failed:", err, ok)
@@ -311,7 +315,7 @@ func Test_localLocker_expireOldLocksExpire(t *testing.T) {
 					for _, v := range l.lockMap {
 						for i := range v {
 							if rng.Intn(2) == 0 {
-								v[i].TimeLastRefresh = expired
+								v[i].TimeLastRefresh = expired.UnixNano()
 							}
 						}
 					}
@@ -347,6 +351,7 @@ func Test_localLocker_expireOldLocksExpire(t *testing.T) {

 func Test_localLocker_RUnlock(t *testing.T) {
 	rng := rand.New(rand.NewSource(0))
+	quorum := 0
 	// Numbers of unique locks
 	for _, locks := range []int{1, 100, 1000, 1e6} {
 		if testing.Short() && locks > 100 {
@@ -375,7 +380,7 @@ func Test_localLocker_RUnlock(t *testing.T) {
 								Resources: res,
 								Source:    hex.EncodeToString(tmp[:8]),
 								Owner:     hex.EncodeToString(tmp[8:]),
-								Quorum:    0,
+								Quorum:    &quorum,
 							})
 							if !ok || err != nil {
 								t.Fatal("failed:", err, ok)
@@ -107,5 +107,5 @@ func newLockAPI(endpoint Endpoint) dsync.NetLocker {

 // Returns a lock rest client.
 func newlockRESTClient(ep Endpoint) *lockRESTClient {
-	return &lockRESTClient{globalGrid.Load().Connection(ep.GridHost())}
+	return &lockRESTClient{globalLockGrid.Load().Connection(ep.GridHost())}
 }
@@ -39,7 +39,7 @@ func TestLockRESTlient(t *testing.T) {

 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
-	err = initGlobalGrid(ctx, []PoolEndpoints{{Endpoints: Endpoints{endpoint, endpointLocal}}})
+	err = initGlobalLockGrid(ctx, []PoolEndpoints{{Endpoints: Endpoints{endpoint, endpointLocal}}})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -44,7 +44,7 @@ func createLockTestServer(ctx context.Context, t *testing.T) (string, *lockRESTS
 		},
 	}
 	creds := globalActiveCred
-	token, err := authenticateNode(creds.AccessKey, creds.SecretKey, "")
+	token, err := authenticateNode(creds.AccessKey, creds.SecretKey)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -63,15 +63,15 @@ func TestLockRpcServerRemoveEntry(t *testing.T) {
 		Owner:           "owner",
 		Writer:          true,
 		UID:             "0123-4567",
-		Timestamp:       UTCNow(),
-		TimeLastRefresh: UTCNow(),
+		Timestamp:       UTCNow().UnixNano(),
+		TimeLastRefresh: UTCNow().UnixNano(),
 	}
 	lockRequesterInfo2 := lockRequesterInfo{
 		Owner:           "owner",
 		Writer:          true,
 		UID:             "89ab-cdef",
-		Timestamp:       UTCNow(),
-		TimeLastRefresh: UTCNow(),
+		Timestamp:       UTCNow().UnixNano(),
+		TimeLastRefresh: UTCNow().UnixNano(),
 	}

 	locker.ll.lockMap["name"] = []lockRequesterInfo{
@@ -111,17 +111,17 @@ func newLockHandler(h grid.HandlerID) *grid.SingleHandler[*dsync.LockArgs, *dsyn
 }

 // registerLockRESTHandlers - register lock rest router.
-func registerLockRESTHandlers() {
+func registerLockRESTHandlers(gm *grid.Manager) {
 	lockServer := &lockRESTServer{
 		ll: newLocker(),
 	}

-	logger.FatalIf(lockRPCForceUnlock.Register(globalGrid.Load(), lockServer.ForceUnlockHandler), "unable to register handler")
-	logger.FatalIf(lockRPCRefresh.Register(globalGrid.Load(), lockServer.RefreshHandler), "unable to register handler")
-	logger.FatalIf(lockRPCLock.Register(globalGrid.Load(), lockServer.LockHandler), "unable to register handler")
-	logger.FatalIf(lockRPCUnlock.Register(globalGrid.Load(), lockServer.UnlockHandler), "unable to register handler")
-	logger.FatalIf(lockRPCRLock.Register(globalGrid.Load(), lockServer.RLockHandler), "unable to register handler")
-	logger.FatalIf(lockRPCRUnlock.Register(globalGrid.Load(), lockServer.RUnlockHandler), "unable to register handler")
+	logger.FatalIf(lockRPCForceUnlock.Register(gm, lockServer.ForceUnlockHandler), "unable to register handler")
+	logger.FatalIf(lockRPCRefresh.Register(gm, lockServer.RefreshHandler), "unable to register handler")
+	logger.FatalIf(lockRPCLock.Register(gm, lockServer.LockHandler), "unable to register handler")
+	logger.FatalIf(lockRPCUnlock.Register(gm, lockServer.UnlockHandler), "unable to register handler")
+	logger.FatalIf(lockRPCRLock.Register(gm, lockServer.RLockHandler), "unable to register handler")
+	logger.FatalIf(lockRPCRUnlock.Register(gm, lockServer.RUnlockHandler), "unable to register handler")

 	globalLockServer = lockServer.ll

@@ -107,6 +107,11 @@ func newApp(name string) *cli.App {

 	// registerCommand registers a cli command.
 	registerCommand := func(command cli.Command) {
+		// avoid registering commands which are not being built (via
+		// go:build tags)
+		if command.Name == "" {
+			return
+		}
 		commands = append(commands, command)
 		commandsTree.Insert(command.Name)
 	}
@@ -134,6 +139,7 @@ func newApp(name string) *cli.App {

 	// Register all commands.
 	registerCommand(serverCmd)
+	registerCommand(fmtGenCmd)

 	// Set up app.
 	cli.HelpFlag = cli.BoolFlag{
@@ -153,19 +153,7 @@ func (z *erasureServerPools) listPath(ctx context.Context, o *listPathOptions) (
 			} else {
 				// Continue listing
 				o.ID = c.id
-				go func(meta metacache) {
-					// Continuously update while we wait.
-					t := time.NewTicker(metacacheMaxClientWait / 10)
-					defer t.Stop()
-					select {
-					case <-ctx.Done():
-						// Request is done, stop updating.
-						return
-					case <-t.C:
-						meta.lastHandout = time.Now()
-						meta, _ = rpc.UpdateMetacacheListing(ctx, meta)
-					}
-				}(*c)
+				go c.keepAlive(ctx, rpc)
 			}
 		}
 	}
@@ -219,6 +207,9 @@ func (z *erasureServerPools) listPath(ctx context.Context, o *listPathOptions) (
 		o.ID = ""
 	}

+	if contextCanceled(ctx) {
+		return entries, ctx.Err()
+	}
 	// Do listing in-place.
 	// Create output for our results.
 	// Create filter for results.
@@ -449,5 +440,10 @@ func (z *erasureServerPools) listAndSave(ctx context.Context, o *listPathOptions
 		xioutil.SafeClose(saveCh)
 	}()

-	return filteredResults()
+	entries, err = filteredResults()
+	if err == nil {
+		// Check if listing recorded an error.
+		err = meta.getErr()
+	}
+	return entries, err
 }
@@ -805,6 +805,17 @@ func (m *metaCacheRPC) setErr(err string) {
 	*m.meta = meta
 }

+// getErr will return an error if the listing failed.
+// The error is not type safe.
+func (m *metaCacheRPC) getErr() error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.meta.status == scanStateError {
+		return errors.New(m.meta.error)
+	}
+	return nil
+}
+
 func (er *erasureObjects) saveMetaCacheStream(ctx context.Context, mc *metaCacheRPC, entries <-chan metaCacheEntry) (err error) {
 	o := mc.o
 	o.debugf(color.Green("saveMetaCacheStream:")+" with options: %#v", o)
@@ -24,6 +24,8 @@ import (
 	"path"
 	"strings"
 	"time"
+
+	"github.com/minio/pkg/v3/console"
 )

 type scanStatus uint8
@@ -97,6 +99,37 @@ func (m *metacache) worthKeeping() bool {
 	return true
 }

+// keepAlive will continuously update lastHandout until ctx is canceled.
+func (m metacache) keepAlive(ctx context.Context, rpc *peerRESTClient) {
+	// we intentionally operate on a copy of m, so we can update without locks.
+	t := time.NewTicker(metacacheMaxClientWait / 10)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			// Request is done, stop updating.
+			return
+		case <-t.C:
+			m.lastHandout = time.Now()
+
+			if m2, err := rpc.UpdateMetacacheListing(ctx, m); err == nil {
+				if m2.status != scanStateStarted {
+					if serverDebugLog {
+						console.Debugln("returning", m.id, "due to scan state", m2.status, time.Now().Format(time.RFC3339))
+					}
+					return
+				}
+				m = m2
+				if serverDebugLog {
+					console.Debugln("refreshed", m.id, time.Now().Format(time.RFC3339))
+				}
+			} else if serverDebugLog {
+				console.Debugln("error refreshing", m.id, time.Now().Format(time.RFC3339))
+			}
+		}
+	}
+}
+
 // baseDirFromPrefix will return the base directory given an object path.
 // For example an object with name prefix/folder/object.ext will return `prefix/folder/`.
 func baseDirFromPrefix(prefix string) string {
@@ -116,13 +149,17 @@ func baseDirFromPrefix(prefix string) string {
 // update cache with new status.
 // The updates are conditional so multiple callers can update with different states.
 func (m *metacache) update(update metacache) {
-	m.lastUpdate = UTCNow()
+	now := UTCNow()
+	m.lastUpdate = now

-	if m.lastHandout.After(m.lastHandout) {
-		m.lastHandout = UTCNow()
+	if update.lastHandout.After(m.lastHandout) {
+		m.lastHandout = update.lastUpdate
+		if m.lastHandout.After(now) {
+			m.lastHandout = now
+		}
 	}
 	if m.status == scanStateStarted && update.status == scanStateSuccess {
-		m.ended = UTCNow()
+		m.ended = now
 	}

 	if m.status == scanStateStarted && update.status != scanStateStarted {
@@ -138,7 +175,7 @@ func (m *metacache) update(update metacache) {
 	if m.error == "" && update.error != "" {
 		m.error = update.error
 		m.status = scanStateError
-		m.ended = UTCNow()
+		m.ended = now
 	}
 	m.fileNotFound = m.fileNotFound || update.fileNotFound
 }
@@ -262,7 +262,7 @@ func collectDriveMetrics(m madmin.RealtimeMetrics) {
 	latestDriveStatsMu.Unlock()

 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	for _, d := range localDrives {
@@ -38,7 +38,8 @@ const (

 // Standard env prometheus auth type
 const (
-	EnvPrometheusAuthType = "MINIO_PROMETHEUS_AUTH_TYPE"
+	EnvPrometheusAuthType    = "MINIO_PROMETHEUS_AUTH_TYPE"
+	EnvPrometheusOpenMetrics = "MINIO_PROMETHEUS_OPEN_METRICS"
 )

 type prometheusAuthType string
@@ -58,14 +59,15 @@ func registerMetricsRouter(router *mux.Router) {
 	if authType == prometheusPublic {
 		auth = NoAuthMiddleware
 	}
+
 	metricsRouter.Handle(prometheusMetricsPathLegacy, auth(metricsHandler()))
 	metricsRouter.Handle(prometheusMetricsV2ClusterPath, auth(metricsServerHandler()))
 	metricsRouter.Handle(prometheusMetricsV2BucketPath, auth(metricsBucketHandler()))
 	metricsRouter.Handle(prometheusMetricsV2NodePath, auth(metricsNodeHandler()))
 	metricsRouter.Handle(prometheusMetricsV2ResourcePath, auth(metricsResourceHandler()))

-	// Metrics v3!
-	metricsV3Server := newMetricsV3Server(authType)
+	// Metrics v3
+	metricsV3Server := newMetricsV3Server(auth)

 	// Register metrics v3 handler. It also accepts an optional query
 	// parameter `?list` - see handler for details.
@@ -19,6 +19,8 @@ package cmd

 import (
 	"context"
+
+	"github.com/minio/minio-go/v7/pkg/set"
 )

 const (
@@ -127,7 +129,8 @@ func loadAPIRequestsHTTPMetrics(ctx context.Context, m MetricValues, _ *metricsC
 // This is a `MetricsLoaderFn`.
 func loadAPIRequestsTTFBMetrics(ctx context.Context, m MetricValues, _ *metricsCache) error {
 	renameLabels := map[string]string{"api": "name"}
-	m.SetHistogram(apiRequestsTTFBSecondsDistribution, httpRequestsDuration, renameLabels, nil,
+	labelsFilter := map[string]set.StringSet{"api": set.CreateStringSet("GetObject")}
+	m.SetHistogram(apiRequestsTTFBSecondsDistribution, httpRequestsDuration, labelsFilter, renameLabels, nil,
 		"type", "s3")
 	return nil
 }
@@ -214,7 +217,8 @@ func loadBucketAPIHTTPMetrics(ctx context.Context, m MetricValues, _ *metricsCac
 // This is a `MetricsLoaderFn`.
 func loadBucketAPITTFBMetrics(ctx context.Context, m MetricValues, _ *metricsCache, buckets []string) error {
 	renameLabels := map[string]string{"api": "name"}
-	m.SetHistogram(apiRequestsTTFBSecondsDistribution, bucketHTTPRequestsDuration, renameLabels,
+	labelsFilter := map[string]set.StringSet{"api": set.CreateStringSet("GetObject")}
+	m.SetHistogram(apiRequestsTTFBSecondsDistribution, bucketHTTPRequestsDuration, labelsFilter, renameLabels,
 		buckets, "type", "s3")
 	return nil
 }
@@ -139,7 +139,7 @@ var (
 )

 // loadClusterUsageBucketMetrics - `MetricsLoaderFn` to load bucket usage metrics.
-func loadClusterUsageBucketMetrics(ctx context.Context, m MetricValues, c *metricsCache, buckets []string) error {
+func loadClusterUsageBucketMetrics(ctx context.Context, m MetricValues, c *metricsCache) error {
 	dataUsageInfo, err := c.dataUsageInfo.Get()
 	if err != nil {
 		metricsLogIf(ctx, err)
@@ -153,11 +153,7 @@ func loadClusterUsageBucketMetrics(ctx context.Context, m MetricValues, c *metri

 	m.Set(usageSinceLastUpdateSeconds, float64(time.Since(dataUsageInfo.LastUpdate)))

-	for _, bucket := range buckets {
-		usage, ok := dataUsageInfo.BucketsUsage[bucket]
-		if !ok {
-			continue
-		}
+	for bucket, usage := range dataUsageInfo.BucketsUsage {
 		quota, err := globalBucketQuotaSys.Get(ctx, bucket)
 		if err != nil {
 			// Log and continue if we are unable to retrieve metrics for this
@@ -23,9 +23,12 @@ import (
 	"net/http"
 	"slices"
 	"strings"
+	"sync"

+	"github.com/minio/minio/internal/config"
 	"github.com/minio/minio/internal/mcontext"
 	"github.com/minio/mux"
+	"github.com/minio/pkg/v3/env"
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
 )
@@ -33,41 +36,39 @@ import (
 type promLogger struct{}

 func (p promLogger) Println(v ...interface{}) {
-	s := make([]string, 0, len(v))
-	for _, val := range v {
-		s = append(s, fmt.Sprintf("%v", val))
-	}
-	err := fmt.Errorf("metrics handler error: %v", strings.Join(s, " "))
-	metricsLogIf(GlobalContext, err)
+	metricsLogIf(GlobalContext, fmt.Errorf("metrics handler error: %v", v))
 }

 type metricsV3Server struct {
 	registry *prometheus.Registry
 	opts     promhttp.HandlerOpts
-	authFn   func(http.Handler) http.Handler
+	auth     func(http.Handler) http.Handler

 	metricsData *metricsV3Collection
 }

-func newMetricsV3Server(authType prometheusAuthType) *metricsV3Server {
+var (
+	globalMetricsV3CollectorPaths []collectorPath
+	globalMetricsV3Once           sync.Once
+)
+
+func newMetricsV3Server(auth func(h http.Handler) http.Handler) *metricsV3Server {
 	registry := prometheus.NewRegistry()
-	authFn := AuthMiddleware
-	if authType == prometheusPublic {
-		authFn = NoAuthMiddleware
-	}
-
 	metricGroups := newMetricGroups(registry)
-
+	globalMetricsV3Once.Do(func() {
+		globalMetricsV3CollectorPaths = metricGroups.collectorPaths
+	})
 	return &metricsV3Server{
 		registry: registry,
 		opts: promhttp.HandlerOpts{
 			ErrorLog:            promLogger{},
-			ErrorHandling:       promhttp.HTTPErrorOnError,
+			ErrorHandling:       promhttp.ContinueOnError,
 			Registry:            registry,
 			MaxRequestsInFlight: 2,
+			EnableOpenMetrics:   env.Get(EnvPrometheusOpenMetrics, config.EnableOff) == config.EnableOn,
+			ProcessStartTime:    globalBootTime,
 		},
-		authFn: authFn,
-
+		auth:        auth,
 		metricsData: metricGroups,
 	}
 }
@@ -163,7 +164,7 @@ func (h *metricsV3Server) handle(path string, isListingRequest bool, buckets []s
 		http.Error(w, "Metrics Resource Not found", http.StatusNotFound)
 	})

-	// Require that metrics path has at least component.
+	// Require that metrics path has one component at least.
 	if path == "/" {
 		return notFoundHandler
 	}
@@ -221,7 +222,7 @@ func (h *metricsV3Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	pathComponents := mux.Vars(r)["pathComps"]
 	isListingRequest := r.Form.Has("list")

-	buckets := []string{}
+	var buckets []string
 	if strings.HasPrefix(pathComponents, "/bucket/") {
 		// bucket specific metrics, extract the bucket name from the path.
 		// it's the last part of the path. e.g. /bucket/api/<bucket-name>
@@ -246,5 +247,5 @@ func (h *metricsV3Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	})

 	// Add authentication
-	h.authFn(tracedHandler).ServeHTTP(w, r)
+	h.auth(tracedHandler).ServeHTTP(w, r)
 }
@@ -23,6 +23,7 @@ import (
 	"strings"
 	"sync"

+	"github.com/minio/minio-go/v7/pkg/set"
 	"github.com/minio/minio/internal/logger"
 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"
@@ -35,8 +36,8 @@ type collectorPath string
 // converted to snake-case (by replaced '/' and '-' with '_') and prefixed with
 // `minio_`.
 func (cp collectorPath) metricPrefix() string {
-	s := strings.TrimPrefix(string(cp), "/")
-	s = strings.ReplaceAll(s, "/", "_")
+	s := strings.TrimPrefix(string(cp), SlashSeparator)
+	s = strings.ReplaceAll(s, SlashSeparator, "_")
 	s = strings.ReplaceAll(s, "-", "_")
 	return "minio_" + s
 }
@@ -56,8 +57,8 @@ func (cp collectorPath) isDescendantOf(arg string) bool {
 	if len(arg) >= len(descendant) {
 		return false
 	}
-	if !strings.HasSuffix(arg, "/") {
-		arg += "/"
+	if !strings.HasSuffix(arg, SlashSeparator) {
+		arg += SlashSeparator
 	}
 	return strings.HasPrefix(descendant, arg)
 }
@@ -72,10 +73,11 @@ const (
 	GaugeMT
 	// HistogramMT - represents a histogram metric.
 	HistogramMT
-	// rangeL - represents a range label.
-	rangeL = "range"
 )

+// rangeL - represents a range label.
+const rangeL = "range"
+
 func (mt MetricType) String() string {
 	switch mt {
 	case CounterMT:
@@ -246,28 +248,38 @@ func (m *MetricValues) Set(name MetricName, value float64, labels ...string) {
 // SetHistogram - sets values for the given MetricName using the provided
 // histogram.
 //
+// `filterByLabels` is a map of label names to list of allowed label values to
+// filter by. Note that this filtering happens before any renaming of labels.
+//
 // `renameLabels` is a map of label names to rename. The keys are the original
 // label names and the values are the new label names.
 //
-// TODO: bucketFilter doc
+// `bucketFilter` is a list of bucket values to filter. If this is non-empty,
+// only metrics for the given buckets are added.
 //
 // `extraLabels` are additional labels to add to each metric. They are ordered
 // label name and value pairs.
 func (m *MetricValues) SetHistogram(name MetricName, hist *prometheus.HistogramVec,
-	renameLabels map[string]string, bucketFilter []string, extraLabels ...string,
+	filterByLabels map[string]set.StringSet, renameLabels map[string]string, bucketFilter []string,
+	extraLabels ...string,
 ) {
 	if _, ok := m.descriptors[name]; !ok {
 		panic(fmt.Sprintf("metric has no description: %s", name))
 	}
 	dummyDesc := MetricDescription{}
 	metricsV2 := getHistogramMetrics(hist, dummyDesc, false)
+mainLoop:
 	for _, metric := range metricsV2 {
+		for label, allowedValues := range filterByLabels {
+			if !allowedValues.Contains(metric.VariableLabels[label]) {
+				continue mainLoop
+			}
+		}
+
 		// If a bucket filter is provided, only add metrics for the given
 		// buckets.
-		if len(bucketFilter) > 0 {
-			if !slices.Contains(bucketFilter, metric.VariableLabels["bucket"]) {
-				continue
-			}
+		if len(bucketFilter) > 0 && !slices.Contains(bucketFilter, metric.VariableLabels["bucket"]) {
+			continue
 		}

 		labels := make([]string, 0, len(metric.VariableLabels)*2)
@@ -270,7 +270,7 @@ func newMetricGroups(r *prometheus.Registry) *metricsV3Collection {
 		loadClusterUsageObjectMetrics,
 	)

-	clusterUsageBucketsMG := NewBucketMetricsGroup(clusterUsageBucketsCollectorPath,
+	clusterUsageBucketsMG := NewMetricsGroup(clusterUsageBucketsCollectorPath,
 		[]MetricDescriptor{
 			usageSinceLastUpdateSecondsMD,
 			usageBucketTotalBytesMD,
@@ -222,6 +222,13 @@ func (d *naughtyDisk) CheckParts(ctx context.Context, volume string, path string
 	return d.disk.CheckParts(ctx, volume, path, fi)
 }

+func (d *naughtyDisk) DeleteBulk(ctx context.Context, volume string, paths ...string) (err error) {
+	if err := d.calcError(); err != nil {
+		return err
+	}
+	return d.disk.DeleteBulk(ctx, volume, paths...)
+}
+
 func (d *naughtyDisk) Delete(ctx context.Context, volume string, path string, deleteOpts DeleteOptions) (err error) {
 	if err := d.calcError(); err != nil {
 		return err
@@ -751,12 +751,6 @@ func isErrSignatureDoesNotMatch(err error) bool {
 	return errors.As(err, &signatureDoesNotMatch)
 }

-// isErrObjectNameInvalid - Check if error type is ObjectNameInvalid.
-func isErrObjectNameInvalid(err error) bool {
-	var invalidObject ObjectNameInvalid
-	return errors.As(err, &invalidObject)
-}
-
 // PreConditionFailed - Check if copy precondition failed
 type PreConditionFailed struct{}

@@ -34,6 +34,7 @@ import (
 	"sync"
 	"time"
 	"unicode/utf8"
+	"unsafe"

 	"github.com/google/uuid"
 	"github.com/klauspost/compress/s2"
@@ -246,6 +247,24 @@ func pathsJoinPrefix(prefix string, elem ...string) (paths []string) {
 	return paths
 }

+// string concat alternative to s1 + s2 with low overhead.
+func concat(ss ...string) string {
+	length := len(ss)
+	if length == 0 {
+		return ""
+	}
+	// create & allocate the memory in advance.
+	n := 0
+	for i := 0; i < length; i++ {
+		n += len(ss[i])
+	}
+	b := make([]byte, 0, n)
+	for i := 0; i < length; i++ {
+		b = append(b, ss[i]...)
+	}
+	return unsafe.String(unsafe.SliceData(b), n)
+}
+
 // pathJoin - like path.Join() but retains trailing SlashSeparator of the last element
 func pathJoin(elem ...string) string {
 	sb := bytebufferpool.Get()
@@ -20,8 +20,10 @@ package cmd
 import (
 	"bytes"
 	"context"
+	"encoding/hex"
 	"fmt"
 	"io"
+	"math/rand"
 	"net/http"
 	"net/http/httptest"
 	"path"
@@ -47,6 +49,43 @@ func pathJoinOld(elem ...string) string {
 	return path.Join(elem...) + trailingSlash
 }

+func concatNaive(ss ...string) string {
+	rs := ss[0]
+	for i := 1; i < len(ss); i++ {
+		rs += ss[i]
+	}
+	return rs
+}
+
+func benchmark(b *testing.B, data []string) {
+	b.Run("concat naive", func(b *testing.B) {
+		b.ResetTimer()
+		b.ReportAllocs()
+		for i := 0; i < b.N; i++ {
+			concatNaive(data...)
+		}
+	})
+	b.Run("concat fast", func(b *testing.B) {
+		b.ResetTimer()
+		b.ReportAllocs()
+		for i := 0; i < b.N; i++ {
+			concat(data...)
+		}
+	})
+}
+
+func BenchmarkConcatImplementation(b *testing.B) {
+	data := make([]string, 2)
+	rng := rand.New(rand.NewSource(0))
+	for i := 0; i < 2; i++ {
+		var tmp [16]byte
+		rng.Read(tmp[:])
+		data[i] = hex.EncodeToString(tmp[:])
+	}
+	b.ResetTimer()
+	benchmark(b, data)
+}
+
 func BenchmarkPathJoinOld(b *testing.B) {
 	b.Run("PathJoin", func(b *testing.B) {
 		b.ResetTimer()
@@ -248,10 +248,19 @@ func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Requ
 	}

 	// Check if the part number is correct.
-	if opts.PartNumber > 1 && opts.PartNumber > len(objInfo.Parts) {
-		// According to S3 we don't need to set any object information here.
-		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartNumber), r.URL)
-		return true
+	if opts.PartNumber > 1 {
+		partFound := false
+		for _, pi := range objInfo.Parts {
+			if pi.Number == opts.PartNumber {
+				partFound = true
+				break
+			}
+		}
+		if !partFound {
+			// According to S3 we don't need to set any object information here.
+			writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartNumber), r.URL)
+			return true
+		}
 	}

 	// If-None-Match : Return the object only if its entity tag (ETag) is different from the
@@ -501,8 +501,8 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 			reader *GetObjectReader
 			perr   error
 		)
-		// avoid proxying if version is a delete marker
-		if !isErrObjectNameInvalid(err) && !isErrMethodNotAllowed(err) && !(gr != nil && gr.ObjInfo.DeleteMarker) {
+
+		if (isErrObjectNotFound(err) || isErrVersionNotFound(err) || isErrReadQuorum(err)) && !(gr != nil && gr.ObjInfo.DeleteMarker) {
 			proxytgts := getProxyTargets(ctx, bucket, object, opts)
 			if !proxytgts.Empty() {
 				globalReplicationStats.incProxy(bucket, getObjectAPI, false)
@@ -1028,7 +1028,7 @@ func (api objectAPIHandlers) headObjectHandler(ctx context.Context, objectAPI Ob

 	objInfo, err := getObjectInfo(ctx, bucket, object, opts)
 	var proxy proxyResult
-	if err != nil && !objInfo.DeleteMarker && !isErrMethodNotAllowed(err) && !isErrObjectNameInvalid(err) {
+	if err != nil && !objInfo.DeleteMarker && (isErrObjectNotFound(err) || isErrVersionNotFound(err) || isErrReadQuorum(err)) {
 		// proxy HEAD to replication target if active-active replication configured on bucket
 		proxytgts := getProxyTargets(ctx, bucket, object, opts)
 		if !proxytgts.Empty() {
@@ -19,6 +19,7 @@ package cmd

 import (
 	"crypto/subtle"
+	"encoding/hex"
 	"io"
 	"net/http"
 	"net/url"
@@ -33,6 +34,7 @@ import (

 	"github.com/minio/minio/internal/auth"
 	levent "github.com/minio/minio/internal/config/lambda/event"
+	"github.com/minio/minio/internal/hash/sha256"
 	xhttp "github.com/minio/minio/internal/http"
 	"github.com/minio/minio/internal/logger"
 )
@@ -77,16 +79,13 @@ func getLambdaEventData(bucket, object string, cred auth.Credentials, r *http.Re
 		return levent.Event{}, err
 	}

-	token, err := authenticateNode(cred.AccessKey, cred.SecretKey, u.RawQuery)
-	if err != nil {
-		return levent.Event{}, err
-	}
+	ckSum := sha256.Sum256([]byte(cred.AccessKey + u.RawQuery))

 	eventData := levent.Event{
 		GetObjectContext: &levent.GetObjectContext{
 			InputS3URL:  u.String(),
 			OutputRoute: shortuuid.New(),
-			OutputToken: token,
+			OutputToken: hex.EncodeToString(ckSum[:]),
 		},
 		UserRequest: levent.UserRequest{
 			URL:     r.URL.String(),
@@ -664,7 +664,7 @@ var errUnsupportedSignal = fmt.Errorf("unsupported signal")

 func waitingDrivesNode() map[string]madmin.DiskMetrics {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	errs := make([]error, len(localDrives))
@@ -34,7 +34,7 @@ const (

 func healBucketLocal(ctx context.Context, bucket string, opts madmin.HealOpts) (res madmin.HealResultItem, err error) {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	// Initialize sync waitgroup.
@@ -158,7 +158,7 @@ func healBucketLocal(ctx context.Context, bucket string, opts madmin.HealOpts) (

 func listBucketsLocal(ctx context.Context, opts BucketOptions) (buckets []BucketInfo, err error) {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	quorum := (len(localDrives) / 2)
@@ -204,15 +204,17 @@ func listBucketsLocal(ctx context.Context, opts BucketOptions) (buckets []Bucket
 	return buckets, nil
 }

-func cloneDrives(drives []StorageAPI) []StorageAPI {
-	newDrives := make([]StorageAPI, len(drives))
-	copy(newDrives, drives)
-	return newDrives
+func cloneDrives(drives map[string]StorageAPI) []StorageAPI {
+	copyDrives := make([]StorageAPI, 0, len(drives))
+	for _, drive := range drives {
+		copyDrives = append(copyDrives, drive)
+	}
+	return copyDrives
 }

 func getBucketInfoLocal(ctx context.Context, bucket string, opts BucketOptions) (BucketInfo, error) {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	g := errgroup.WithNErrs(len(localDrives)).WithConcurrency(32)
@@ -261,7 +263,7 @@ func getBucketInfoLocal(ctx context.Context, bucket string, opts BucketOptions)

 func deleteBucketLocal(ctx context.Context, bucket string, opts DeleteBucketOptions) error {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	g := errgroup.WithNErrs(len(localDrives)).WithConcurrency(32)
@@ -299,7 +301,7 @@ func deleteBucketLocal(ctx context.Context, bucket string, opts DeleteBucketOpti

 func makeBucketLocal(ctx context.Context, bucket string, opts MakeBucketOptions) error {
 	globalLocalDrivesMu.RLock()
-	localDrives := cloneDrives(globalLocalDrives)
+	localDrives := cloneDrives(globalLocalDrivesMap)
 	globalLocalDrivesMu.RUnlock()

 	g := errgroup.WithNErrs(len(localDrives)).WithConcurrency(32)
@@ -166,13 +166,13 @@ func connectLoadInitFormats(verboseLogging bool, firstDisk bool, storageDisks []
 		if err != nil && !errors.Is(err, errXLBackend) && !errors.Is(err, errUnformattedDisk) {
 			if errors.Is(err, errDiskNotFound) && verboseLogging {
 				if globalEndpoints.NEndpoints() > 1 {
-					logger.Error("Unable to connect to %s: %v", endpoints[i], isServerResolvable(endpoints[i], time.Second))
+					logger.Info("Unable to connect to %s: %v, will be retried", endpoints[i], isServerResolvable(endpoints[i], time.Second))
 				} else {
 					logger.Fatal(err, "Unable to connect to %s: %v", endpoints[i], isServerResolvable(endpoints[i], time.Second))
 				}
 			} else {
 				if globalEndpoints.NEndpoints() > 1 {
-					logger.Error("Unable to use the drive %s: %v", endpoints[i], err)
+					logger.Info("Unable to use the drive %s: %v, will be retried", endpoints[i], err)
 				} else {
 					logger.Fatal(errInvalidArgument, "Unable to use the drive %s: %v", endpoints[i], err)
 				}
@@ -26,20 +26,28 @@ import (

 // Composed function registering routers for only distributed Erasure setup.
 func registerDistErasureRouters(router *mux.Router, endpointServerPools EndpointServerPools) {
+	var (
+		lockGrid   = globalLockGrid.Load()
+		commonGrid = globalGrid.Load()
+	)
+
 	// Register storage REST router only if its a distributed setup.
-	registerStorageRESTHandlers(router, endpointServerPools, globalGrid.Load())
+	registerStorageRESTHandlers(router, endpointServerPools, commonGrid)

 	// Register peer REST router only if its a distributed setup.
-	registerPeerRESTHandlers(router, globalGrid.Load())
+	registerPeerRESTHandlers(router, commonGrid)

 	// Register bootstrap REST router for distributed setups.
-	registerBootstrapRESTHandlers(globalGrid.Load())
+	registerBootstrapRESTHandlers(commonGrid)

 	// Register distributed namespace lock routers.
-	registerLockRESTHandlers()
+	registerLockRESTHandlers(lockGrid)
+
+	// Add lock grid to router
+	router.Handle(grid.RouteLockPath, adminMiddleware(lockGrid.Handler(storageServerRequestValidate), noGZFlag, noObjLayerFlag))

 	// Add grid to router
-	router.Handle(grid.RoutePath, adminMiddleware(globalGrid.Load().Handler(storageServerRequestValidate), noGZFlag, noObjLayerFlag))
+	router.Handle(grid.RoutePath, adminMiddleware(commonGrid.Handler(storageServerRequestValidate), noGZFlag, noObjLayerFlag))
 }

 // List of some generic middlewares which are applied for all incoming requests.
@@ -841,13 +841,14 @@ func serverMain(ctx *cli.Context) {

 	// Verify kernel release and version.
 	if oldLinux() {
-		warnings = append(warnings, color.YellowBold("- Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance"))
+		warnings = append(warnings, color.YellowBold("Detected Linux kernel version older than 4.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x linux kernel version for best performance"))
 	}

 	maxProcs := runtime.GOMAXPROCS(0)
 	cpuProcs := runtime.NumCPU()
 	if maxProcs < cpuProcs {
-		warnings = append(warnings, color.YellowBold("- Detected GOMAXPROCS(%d) < NumCPU(%d), please make sure to provide all PROCS to MinIO for optimal performance", maxProcs, cpuProcs))
+		warnings = append(warnings, color.YellowBold("Detected GOMAXPROCS(%d) < NumCPU(%d), please make sure to provide all PROCS to MinIO for optimal performance",
+			maxProcs, cpuProcs))
 	}

 	// Initialize grid
@@ -855,6 +856,11 @@ func serverMain(ctx *cli.Context) {
 		logger.FatalIf(initGlobalGrid(GlobalContext, globalEndpoints), "Unable to configure server grid RPC services")
 	})

+	// Initialize lock grid
+	bootstrapTrace("initLockGrid", func() {
+		logger.FatalIf(initGlobalLockGrid(GlobalContext, globalEndpoints), "Unable to configure server lock grid RPC services")
+	})
+
 	// Configure server.
 	bootstrapTrace("configureServer", func() {
 		handler, err := configureServerHandler(globalEndpoints)
@@ -862,7 +868,8 @@ func serverMain(ctx *cli.Context) {
 			logger.Fatal(config.ErrUnexpectedError(err), "Unable to configure one of server's RPC services")
 		}
 		// Allow grid to start after registering all services.
-		xioutil.SafeClose(globalGridStart)
+		close(globalGridStart)
+		close(globalLockGridStart)

 		httpServer := xhttp.NewServer(getServerListenAddrs()).
 			UseHandler(setCriticalErrorHandler(corsHandler(handler))).
@@ -921,16 +928,18 @@ func serverMain(ctx *cli.Context) {
 	}

 	bootstrapTrace("waitForQuorum", func() {
-		result := newObject.Health(context.Background(), HealthOptions{})
+		result := newObject.Health(context.Background(), HealthOptions{NoLogging: true})
 		for !result.HealthyRead {
 			if debugNoExit {
-				logger.Info("Not waiting for quorum since we are debugging.. possible cause unhealthy sets (%s)", result)
+				logger.Info("Not waiting for quorum since we are debugging.. possible cause unhealthy sets")
+				logger.Info(result.String())
 				break
 			}
 			d := time.Duration(r.Float64() * float64(time.Second))
-			logger.Info("Waiting for quorum READ healthcheck to succeed.. possible cause unhealthy sets (%s), retrying in %s", result, d)
+			logger.Info("Waiting for quorum READ healthcheck to succeed retrying in %s.. possible cause unhealthy sets", d)
+			logger.Info(result.String())
 			time.Sleep(d)
-			result = newObject.Health(context.Background(), HealthOptions{})
+			result = newObject.Health(context.Background(), HealthOptions{NoLogging: true})
 		}
 	})

@@ -953,11 +962,11 @@ func serverMain(ctx *cli.Context) {
 		}

 		if !globalServerCtxt.StrictS3Compat {
-			warnings = append(warnings, color.YellowBold("- Strict AWS S3 compatible incoming PUT, POST content payload validation is turned off, caution is advised do not use in production"))
+			warnings = append(warnings, color.YellowBold("Strict AWS S3 compatible incoming PUT, POST content payload validation is turned off, caution is advised do not use in production"))
 		}
 	})
 	if globalActiveCred.Equal(auth.DefaultCredentials) {
-		msg := fmt.Sprintf("- Detected default credentials '%s', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables",
+		msg := fmt.Sprintf("Detected default credentials '%s', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables",
 			globalActiveCred)
 		warnings = append(warnings, color.YellowBold(msg))
 	}
@@ -1103,18 +1112,12 @@ func serverMain(ctx *cli.Context) {
 		printStartupMessage(getAPIEndpoints(), err)

 		// Print a warning at the end of the startup banner so it is more noticeable
-		if newObject.BackendInfo().StandardSCParity == 0 {
-			warnings = append(warnings, color.YellowBold("- The standard parity is set to 0. This can lead to data loss."))
+		if newObject.BackendInfo().StandardSCParity == 0 && !globalIsErasureSD {
+			warnings = append(warnings, color.YellowBold("The standard parity is set to 0. This can lead to data loss."))
 		}
-		objAPI := newObjectLayerFn()
-		if objAPI != nil {
-			printStorageInfo(objAPI.StorageInfo(GlobalContext, true))
-		}
-		if len(warnings) > 0 {
-			logger.Info(color.Yellow("STARTUP WARNINGS:"))
-			for _, warn := range warnings {
-				logger.Info(warn)
-			}
+
+		for _, warn := range warnings {
+			logger.Warning(warn)
 		}
 	}()

@@ -23,7 +23,6 @@ import (
 	"net/url"
 	"strings"

-	"github.com/minio/madmin-go/v3"
 	"github.com/minio/minio/internal/color"
 	"github.com/minio/minio/internal/logger"
 	xnet "github.com/minio/pkg/v3/net"
@@ -37,7 +36,11 @@ func getFormatStr(strLen int, padding int) string {

 // Prints the formatted startup message.
 func printStartupMessage(apiEndpoints []string, err error) {
-	logger.Info(color.Bold(MinioBannerName))
+	banner := strings.Repeat("-", len(MinioBannerName))
+	if globalIsDistErasure {
+		logger.Startup(color.Bold(banner))
+	}
+	logger.Startup(color.Bold(MinioBannerName))
 	if err != nil {
 		if globalConsoleSys != nil {
 			globalConsoleSys.Send(GlobalContext, fmt.Sprintf("Server startup failed with '%v', some features may be missing", err))
@@ -47,7 +50,7 @@ func printStartupMessage(apiEndpoints []string, err error) {
 	if !globalSubnetConfig.Registered() {
 		var builder strings.Builder
 		startupBanner(&builder)
-		logger.Info(builder.String())
+		logger.Startup(builder.String())
 	}

 	strippedAPIEndpoints := stripStandardPorts(apiEndpoints, globalMinioHost)
@@ -61,6 +64,9 @@ func printStartupMessage(apiEndpoints []string, err error) {

 	// Prints documentation message.
 	printObjectAPIMsg()
+	if globalIsDistErasure {
+		logger.Startup(color.Bold(banner))
+	}
 }

 // Returns true if input is IPv6
@@ -113,21 +119,21 @@ func printServerCommonMsg(apiEndpoints []string) {

 	apiEndpointStr := strings.TrimSpace(strings.Join(apiEndpoints, "  "))
 	// Colorize the message and print.
-	logger.Info(color.Blue("API: ") + color.Bold(fmt.Sprintf("%s ", apiEndpointStr)))
+	logger.Startup(color.Blue("API: ") + color.Bold(fmt.Sprintf("%s ", apiEndpointStr)))
 	if color.IsTerminal() && (!globalServerCtxt.Anonymous && !globalServerCtxt.JSON && globalAPIConfig.permitRootAccess()) {
-		logger.Info(color.Blue("   RootUser: ") + color.Bold("%s ", cred.AccessKey))
-		logger.Info(color.Blue("   RootPass: ") + color.Bold("%s \n", cred.SecretKey))
+		logger.Startup(color.Blue("   RootUser: ") + color.Bold("%s ", cred.AccessKey))
+		logger.Startup(color.Blue("   RootPass: ") + color.Bold("%s \n", cred.SecretKey))
 		if region != "" {
-			logger.Info(color.Blue("   Region: ") + color.Bold("%s", fmt.Sprintf(getFormatStr(len(region), 2), region)))
+			logger.Startup(color.Blue("   Region: ") + color.Bold("%s", fmt.Sprintf(getFormatStr(len(region), 2), region)))
 		}
 	}

 	if globalBrowserEnabled {
 		consoleEndpointStr := strings.Join(stripStandardPorts(getConsoleEndpoints(), globalMinioConsoleHost), " ")
-		logger.Info(color.Blue("WebUI: ") + color.Bold(fmt.Sprintf("%s ", consoleEndpointStr)))
+		logger.Startup(color.Blue("WebUI: ") + color.Bold(fmt.Sprintf("%s ", consoleEndpointStr)))
 		if color.IsTerminal() && (!globalServerCtxt.Anonymous && !globalServerCtxt.JSON && globalAPIConfig.permitRootAccess()) {
-			logger.Info(color.Blue("   RootUser: ") + color.Bold("%s ", cred.AccessKey))
-			logger.Info(color.Blue("   RootPass: ") + color.Bold("%s ", cred.SecretKey))
+			logger.Startup(color.Blue("   RootUser: ") + color.Bold("%s ", cred.AccessKey))
+			logger.Startup(color.Blue("   RootPass: ") + color.Bold("%s ", cred.SecretKey))
 		}
 	}

@@ -137,7 +143,7 @@ func printServerCommonMsg(apiEndpoints []string) {

 // Prints startup message for Object API access, prints link to our SDK documentation.
 func printObjectAPIMsg() {
-	logger.Info(color.Blue("\nDocs: ") + "https://min.io/docs/minio/linux/index.html")
+	logger.Startup(color.Blue("\nDocs: ") + "https://min.io/docs/minio/linux/index.html")
 }

 func printLambdaTargets() {
@@ -149,7 +155,7 @@ func printLambdaTargets() {
 	for _, arn := range globalLambdaTargetList.List(globalSite.Region()) {
 		arnMsg += color.Bold(fmt.Sprintf("%s ", arn))
 	}
-	logger.Info(arnMsg + "\n")
+	logger.Startup(arnMsg + "\n")
 }

 // Prints bucket notification configurations.
@@ -168,7 +174,7 @@ func printEventNotifiers() {
 		arnMsg += color.Bold(fmt.Sprintf("%s ", arn))
 	}

-	logger.Info(arnMsg + "\n")
+	logger.Startup(arnMsg + "\n")
 }

 // Prints startup message for command line access. Prints link to our documentation
@@ -181,35 +187,9 @@ func printCLIAccessMsg(endPoint string, alias string) {

 	// Configure 'mc', following block prints platform specific information for minio client.
 	if color.IsTerminal() && (!globalServerCtxt.Anonymous && globalAPIConfig.permitRootAccess()) {
-		logger.Info(color.Blue("\nCLI: ") + mcQuickStartGuide)
+		logger.Startup(color.Blue("\nCLI: ") + mcQuickStartGuide)
 		mcMessage := fmt.Sprintf("$ mc alias set '%s' '%s' '%s' '%s'", alias,
 			endPoint, cred.AccessKey, cred.SecretKey)
-		logger.Info(fmt.Sprintf(getFormatStr(len(mcMessage), 3), mcMessage))
-	}
-}
-
-// Get formatted disk/storage info message.
-func getStorageInfoMsg(storageInfo StorageInfo) string {
-	var msg string
-	var mcMessage string
-	onlineDisks, offlineDisks := getOnlineOfflineDisksStats(storageInfo.Disks)
-	if storageInfo.Backend.Type == madmin.Erasure {
-		if offlineDisks.Sum() > 0 {
-			mcMessage = "Use `mc admin info` to look for latest server/drive info\n"
-		}
-
-		diskInfo := fmt.Sprintf(" %d Online, %d Offline. ", onlineDisks.Sum(), offlineDisks.Sum())
-		msg += color.Blue("Status:") + fmt.Sprintf(getFormatStr(len(diskInfo), 8), diskInfo)
-		if len(mcMessage) > 0 {
-			msg = fmt.Sprintf("%s %s", mcMessage, msg)
-		}
-	}
-	return msg
-}
-
-// Prints startup message of storage capacity and erasure information.
-func printStorageInfo(storageInfo StorageInfo) {
-	if msg := getStorageInfoMsg(storageInfo); msg != "" {
-		logger.Info(msg)
+		logger.Startup(fmt.Sprintf(getFormatStr(len(mcMessage), 3), mcMessage))
 	}
 }
@@ -21,32 +21,9 @@ import (
 	"context"
 	"os"
 	"reflect"
-	"strings"
 	"testing"
-
-	"github.com/minio/madmin-go/v3"
 )

-// Tests if we generate storage info.
-func TestStorageInfoMsg(t *testing.T) {
-	infoStorage := StorageInfo{}
-	infoStorage.Disks = []madmin.Disk{
-		{Endpoint: "http://127.0.0.1:9000/data/1/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9000/data/2/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9000/data/3/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9000/data/4/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9001/data/1/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9001/data/2/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9001/data/3/", State: madmin.DriveStateOk},
-		{Endpoint: "http://127.0.0.1:9001/data/4/", State: madmin.DriveStateOffline},
-	}
-	infoStorage.Backend.Type = madmin.Erasure
-
-	if msg := getStorageInfoMsg(infoStorage); !strings.Contains(msg, "7 Online, 1 Offline") {
-		t.Fatal("Unexpected storage info message, found:", msg)
-	}
-}
-
 // Tests stripping standard ports from apiEndpoints.
 func TestStripStandardPorts(t *testing.T) {
 	apiEndpoints := []string{"http://127.0.0.1:9000", "http://127.0.0.2:80", "https://127.0.0.3:443"}
@@ -35,6 +35,7 @@ import (
 	"time"

 	"github.com/dustin/go-humanize"
+	jwtgo "github.com/golang-jwt/jwt/v4"
 	"github.com/minio/minio-go/v7/pkg/set"
 	xhttp "github.com/minio/minio/internal/http"
 	"github.com/minio/pkg/v3/policy"
@@ -122,6 +123,9 @@ func runAllTests(suite *TestSuiteCommon, c *check) {
 	suite.TestObjectMultipartListError(c)
 	suite.TestObjectValidMD5(c)
 	suite.TestObjectMultipart(c)
+	suite.TestMetricsV3Handler(c)
+	suite.TestBucketSQSNotificationWebHook(c)
+	suite.TestBucketSQSNotificationAMQP(c)
 	suite.TearDownSuite(c)
 }

@@ -189,6 +193,36 @@ func (s *TestSuiteCommon) TearDownSuite(c *check) {
 	s.testServer.Stop()
 }

+const (
+	defaultPrometheusJWTExpiry = 100 * 365 * 24 * time.Hour
+)
+
+func (s *TestSuiteCommon) TestMetricsV3Handler(c *check) {
+	jwt := jwtgo.NewWithClaims(jwtgo.SigningMethodHS512, jwtgo.StandardClaims{
+		ExpiresAt: time.Now().UTC().Add(defaultPrometheusJWTExpiry).Unix(),
+		Subject:   s.accessKey,
+		Issuer:    "prometheus",
+	})
+
+	token, err := jwt.SignedString([]byte(s.secretKey))
+	c.Assert(err, nil)
+
+	for _, cpath := range globalMetricsV3CollectorPaths {
+		request, err := newTestSignedRequest(http.MethodGet, s.endPoint+minioReservedBucketPath+metricsV3Path+string(cpath),
+			0, nil, s.accessKey, s.secretKey, s.signer)
+		c.Assert(err, nil)
+
+		request.Header.Set("Authorization", "Bearer "+token)
+
+		// execute the request.
+		response, err := s.client.Do(request)
+		c.Assert(err, nil)
+
+		// assert the http response status code.
+		c.Assert(response.StatusCode, http.StatusOK)
+	}
+}
+
 func (s *TestSuiteCommon) TestBucketSQSNotificationWebHook(c *check) {
 	// Sample bucket notification.
 	bucketNotificationBuf := `<NotificationConfiguration><QueueConfiguration><Event>s3:ObjectCreated:Put</Event><Filter><S3Key><FilterRule><Name>prefix</Name><Value>images/</Value></FilterRule></S3Key></Filter><Id>1</Id><Queue>arn:minio:sqs:us-east-1:444455556666:webhook</Queue></QueueConfiguration></NotificationConfiguration>`
@@ -1129,7 +1129,7 @@ func (c *SiteReplicationSys) PeerBucketConfigureReplHandler(ctx context.Context,
 		if err != nil {
 			return err
 		}
-		sameTarget, apiErr := validateReplicationDestination(ctx, bucket, newReplicationConfig, true)
+		sameTarget, apiErr := validateReplicationDestination(ctx, bucket, newReplicationConfig, &validateReplicationDestinationOptions{CheckRemoteBucket: true})
 		if apiErr != noError {
 			return fmt.Errorf("bucket replication config validation error: %#v", apiErr)
 		}
@@ -4453,6 +4453,7 @@ func (c *SiteReplicationSys) healBuckets(ctx context.Context, objAPI ObjectLayer
 		return err
 	}
 	ilmExpiryCfgHealed := false
+	opts := validateReplicationDestinationOptions{CheckReady: true}
 	for _, bi := range buckets {
 		bucket := bi.Name
 		info, err := c.siteReplicationStatus(ctx, objAPI, madmin.SRStatusOptions{
@@ -4472,7 +4473,7 @@ func (c *SiteReplicationSys) healBuckets(ctx context.Context, objAPI ObjectLayer
 			c.healVersioningMetadata(ctx, objAPI, bucket, info)
 			c.healOLockConfigMetadata(ctx, objAPI, bucket, info)
 			c.healSSEMetadata(ctx, objAPI, bucket, info)
-			c.healBucketReplicationConfig(ctx, objAPI, bucket, info)
+			c.healBucketReplicationConfig(ctx, objAPI, bucket, info, &opts)
 			c.healBucketPolicies(ctx, objAPI, bucket, info)
 			c.healTagMetadata(ctx, objAPI, bucket, info)
 			c.healBucketQuotaConfig(ctx, objAPI, bucket, info)
@@ -5172,7 +5173,7 @@ func (c *SiteReplicationSys) healBucket(ctx context.Context, objAPI ObjectLayer,
 	return nil
 }

-func (c *SiteReplicationSys) healBucketReplicationConfig(ctx context.Context, objAPI ObjectLayer, bucket string, info srStatusInfo) error {
+func (c *SiteReplicationSys) healBucketReplicationConfig(ctx context.Context, objAPI ObjectLayer, bucket string, info srStatusInfo, opts *validateReplicationDestinationOptions) error {
 	bs := info.BucketStats[bucket]

 	c.RLock()
@@ -5226,7 +5227,7 @@ func (c *SiteReplicationSys) healBucketReplicationConfig(ctx context.Context, ob

 	if rcfg != nil && !replMismatch {
 		// validate remote targets on current cluster for this bucket
-		_, apiErr := validateReplicationDestination(ctx, bucket, rcfg, false)
+		_, apiErr := validateReplicationDestination(ctx, bucket, rcfg, opts)
 		if apiErr != noError {
 			replMismatch = true
 		}
@@ -5390,7 +5391,7 @@ func (c *SiteReplicationSys) healUserPolicies(ctx context.Context, objAPI Object
 			UpdatedAt: lastUpdate,
 		})
 		if err != nil {
-			replLogOnceIf(ctx, fmt.Errorf("Unable to heal IAM user policy mapping for %s from peer site %s -> site %s : %w", user, latestPeerName, peerName, err),
+			replLogOnceIf(ctx, fmt.Errorf("Unable to heal IAM user policy mapping from peer site %s -> site %s : %w", latestPeerName, peerName, err),
 				fmt.Sprintf("heal-user-policy-%s", user))
 		}
 	}
@@ -5455,7 +5456,7 @@ func (c *SiteReplicationSys) healGroupPolicies(ctx context.Context, objAPI Objec
 		})
 		if err != nil {
 			replLogOnceIf(ctx,
-				fmt.Errorf("Unable to heal IAM group policy mapping for %s from peer site %s -> site %s : %w", group, latestPeerName, peerName, err),
+				fmt.Errorf("Unable to heal IAM group policy mapping for from peer site %s -> site %s : %w", latestPeerName, peerName, err),
 				fmt.Sprintf("heal-group-policy-%s", group))
 		}
 	}
@@ -5518,7 +5519,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 			claims, err := globalIAMSys.GetClaimsForSvcAcc(ctx, creds.AccessKey)
 			if err != nil {
 				replLogOnceIf(ctx,
-					fmt.Errorf("Unable to heal service account %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+					fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
 					fmt.Sprintf("heal-user-%s", user))
 				continue
 			}
@@ -5526,7 +5527,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 			_, policy, err := globalIAMSys.GetServiceAccount(ctx, creds.AccessKey)
 			if err != nil {
 				replLogOnceIf(ctx,
-					fmt.Errorf("Unable to heal service account %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+					fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
 					fmt.Sprintf("heal-user-%s", user))
 				continue
 			}
@@ -5536,7 +5537,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 				policyJSON, err = json.Marshal(policy)
 				if err != nil {
 					replLogOnceIf(ctx,
-						fmt.Errorf("Unable to heal service account %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+						fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
 						fmt.Sprintf("heal-user-%s", user))
 					continue
 				}
@@ -5561,7 +5562,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 				UpdatedAt: lastUpdate,
 			}); err != nil {
 				replLogOnceIf(ctx,
-					fmt.Errorf("Unable to heal service account %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+					fmt.Errorf("Unable to heal service account from peer site %s -> %s : %w", latestPeerName, peerName, err),
 					fmt.Sprintf("heal-user-%s", user))
 			}
 			continue
@@ -5576,7 +5577,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 				// be applied.
 				if !errors.Is(err, errNoSuchUser) {
 					replLogOnceIf(ctx,
-						fmt.Errorf("Unable to heal temporary credentials %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+						fmt.Errorf("Unable to heal temporary credentials from peer site %s -> %s : %w", latestPeerName, peerName, err),
 						fmt.Sprintf("heal-user-%s", user))
 					continue
 				}
@@ -5596,7 +5597,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 				UpdatedAt: lastUpdate,
 			}); err != nil {
 				replLogOnceIf(ctx,
-					fmt.Errorf("Unable to heal temporary credentials %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+					fmt.Errorf("Unable to heal temporary credentials from peer site %s -> %s : %w", latestPeerName, peerName, err),
 					fmt.Sprintf("heal-user-%s", user))
 			}
 			continue
@@ -5614,7 +5615,7 @@ func (c *SiteReplicationSys) healUsers(ctx context.Context, objAPI ObjectLayer,
 			UpdatedAt: lastUpdate,
 		}); err != nil {
 			replLogOnceIf(ctx,
-				fmt.Errorf("Unable to heal user %s from peer site %s -> %s : %w", user, latestPeerName, peerName, err),
+				fmt.Errorf("Unable to heal user from peer site %s -> %s : %w", latestPeerName, peerName, err),
 				fmt.Sprintf("heal-user-%s", user))
 		}
 	}
@@ -5680,7 +5681,7 @@ func (c *SiteReplicationSys) healGroups(ctx context.Context, objAPI ObjectLayer,
 			UpdatedAt: lastUpdate,
 		}); err != nil {
 			replLogOnceIf(ctx,
-				fmt.Errorf("Unable to heal group %s from peer site %s -> site %s : %w", group, latestPeerName, peerName, err),
+				fmt.Errorf("Unable to heal group from peer site %s -> site %s : %w", latestPeerName, peerName, err),
 				fmt.Sprintf("heal-group-%s", group))
 		}
 	}
@@ -264,6 +264,24 @@ type FileInfo struct {
 	Versioned bool `msg:"vs"`
 }

+func (fi FileInfo) shardSize() int64 {
+	return ceilFrac(fi.Erasure.BlockSize, int64(fi.Erasure.DataBlocks))
+}
+
+// ShardFileSize - returns final erasure size from original size.
+func (fi FileInfo) ShardFileSize(totalLength int64) int64 {
+	if totalLength == 0 {
+		return 0
+	}
+	if totalLength == -1 {
+		return -1
+	}
+	numShards := totalLength / fi.Erasure.BlockSize
+	lastBlockSize := totalLength % fi.Erasure.BlockSize
+	lastShardSize := ceilFrac(lastBlockSize, int64(fi.Erasure.DataBlocks))
+	return numShards*fi.shardSize() + lastShardSize
+}
+
 // ShallowCopy - copies minimal information for READ MRF checks.
 func (fi FileInfo) ShallowCopy() (n FileInfo) {
 	n.Volume = fi.Volume
@@ -374,24 +392,24 @@ func newFileInfo(object string, dataBlocks, parityBlocks int) (fi FileInfo) {

 // ReadMultipleReq contains information of multiple files to read from disk.
 type ReadMultipleReq struct {
-	Bucket       string   // Bucket. Can be empty if multiple buckets.
-	Prefix       string   // Shared prefix of all files. Can be empty. Will be joined to filename without modification.
-	Files        []string // Individual files to read.
-	MaxSize      int64    // Return error if size is exceed.
-	MetadataOnly bool     // Read as XL meta and truncate data.
-	AbortOn404   bool     // Stop reading after first file not found.
-	MaxResults   int      // Stop after this many successful results. <= 0 means all.
+	Bucket       string   `msg:"bk"`           // Bucket. Can be empty if multiple buckets.
+	Prefix       string   `msg:"pr,omitempty"` // Shared prefix of all files. Can be empty. Will be joined to filename without modification.
+	Files        []string `msg:"fl"`           // Individual files to read.
+	MaxSize      int64    `msg:"ms"`           // Return error if size is exceed.
+	MetadataOnly bool     `msg:"mo"`           // Read as XL meta and truncate data.
+	AbortOn404   bool     `msg:"ab"`           // Stop reading after first file not found.
+	MaxResults   int      `msg:"mr"`           // Stop after this many successful results. <= 0 means all.
 }

 // ReadMultipleResp contains a single response from a ReadMultipleReq.
 type ReadMultipleResp struct {
-	Bucket  string    // Bucket as given by request.
-	Prefix  string    // Prefix as given by request.
-	File    string    // File name as given in request.
-	Exists  bool      // Returns whether the file existed on disk.
-	Error   string    // Returns any error when reading.
-	Data    []byte    // Contains all data of file.
-	Modtime time.Time // Modtime of file on disk.
+	Bucket  string    `msg:"bk"`           // Bucket as given by request.
+	Prefix  string    `msg:"pr,omitempty"` // Prefix as given by request.
+	File    string    `msg:"fl"`           // File name as given in request.
+	Exists  bool      `msg:"ex"`           // Returns whether the file existed on disk.
+	Error   string    `msg:"er,omitempty"` // Returns any error when reading.
+	Data    []byte    `msg:"d"`            // Contains all data of file.
+	Modtime time.Time `msg:"m"`            // Modtime of file on disk.
 }

 // DeleteVersionHandlerParams are parameters for DeleteVersionHandler
@@ -498,8 +516,8 @@ type WriteAllHandlerParams struct {
 //     only after as a 2-phase call, allowing the older dataDir to
 //     hang-around in-case we need some form of recovery.
 type RenameDataResp struct {
-	Sign       []byte
-	OldDataDir string // contains '<uuid>', it is designed to be passed as value to Delete(bucket, pathJoin(object, dataDir))
+	Sign       []byte `msg:"s"`
+	OldDataDir string `msg:"od"` // contains '<uuid>', it is designed to be passed as value to Delete(bucket, pathJoin(object, dataDir))
 }

 const (
@@ -516,15 +534,26 @@ const (

 // CheckPartsResp is a response of the storage CheckParts and VerifyFile APIs
 type CheckPartsResp struct {
-	Results []int
+	Results []int `msg:"r"`
 }

 // LocalDiskIDs - GetLocalIDs response.
 type LocalDiskIDs struct {
-	IDs []string
+	IDs []string `msg:"i"`
 }

 // ListDirResult - ListDir()'s response.
 type ListDirResult struct {
 	Entries []string `msg:"e"`
 }
+
+// DeleteBulkReq - send multiple paths in same delete request.
+type DeleteBulkReq struct {
+	Paths []string `msg:"p"`
+}
+
+// DeleteVersionsErrsResp - collection of delete errors
+// for bulk version deletes
+type DeleteVersionsErrsResp struct {
+	Errs []string `msg:"e"`
+}
@@ -348,6 +348,119 @@ func BenchmarkDecodeCheckPartsResp(b *testing.B) {
 	}
 }

+func TestMarshalUnmarshalDeleteBulkReq(t *testing.T) {
+	v := DeleteBulkReq{}
+	bts, err := v.MarshalMsg(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	left, err := v.UnmarshalMsg(bts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(left) > 0 {
+		t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
+	}
+
+	left, err = msgp.Skip(bts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(left) > 0 {
+		t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
+	}
+}
+
+func BenchmarkMarshalMsgDeleteBulkReq(b *testing.B) {
+	v := DeleteBulkReq{}
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		v.MarshalMsg(nil)
+	}
+}
+
+func BenchmarkAppendMsgDeleteBulkReq(b *testing.B) {
+	v := DeleteBulkReq{}
+	bts := make([]byte, 0, v.Msgsize())
+	bts, _ = v.MarshalMsg(bts[0:0])
+	b.SetBytes(int64(len(bts)))
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		bts, _ = v.MarshalMsg(bts[0:0])
+	}
+}
+
+func BenchmarkUnmarshalDeleteBulkReq(b *testing.B) {
+	v := DeleteBulkReq{}
+	bts, _ := v.MarshalMsg(nil)
+	b.ReportAllocs()
+	b.SetBytes(int64(len(bts)))
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		_, err := v.UnmarshalMsg(bts)
+		if err != nil {
+			b.Fatal(err)
+		}
+	}
+}
+
+func TestEncodeDecodeDeleteBulkReq(t *testing.T) {
+	v := DeleteBulkReq{}
+	var buf bytes.Buffer
+	msgp.Encode(&buf, &v)
+
+	m := v.Msgsize()
+	if buf.Len() > m {
+		t.Log("WARNING: TestEncodeDecodeDeleteBulkReq Msgsize() is inaccurate")
+	}
+
+	vn := DeleteBulkReq{}
+	err := msgp.Decode(&buf, &vn)
+	if err != nil {
+		t.Error(err)
+	}
+
+	buf.Reset()
+	msgp.Encode(&buf, &v)
+	err = msgp.NewReader(&buf).Skip()
+	if err != nil {
+		t.Error(err)
+	}
+}
+
+func BenchmarkEncodeDeleteBulkReq(b *testing.B) {
+	v := DeleteBulkReq{}
+	var buf bytes.Buffer
+	msgp.Encode(&buf, &v)
+	b.SetBytes(int64(buf.Len()))
+	en := msgp.NewWriter(msgp.Nowhere)
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		v.EncodeMsg(en)
+	}
+	en.Flush()
+}
+
+func BenchmarkDecodeDeleteBulkReq(b *testing.B) {
+	v := DeleteBulkReq{}
+	var buf bytes.Buffer
+	msgp.Encode(&buf, &v)
+	b.SetBytes(int64(buf.Len()))
+	rd := msgp.NewEndlessReader(buf.Bytes(), b)
+	dc := msgp.NewReader(rd)
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		err := v.DecodeMsg(dc)
+		if err != nil {
+			b.Fatal(err)
+		}
+	}
+}
+
 func TestMarshalUnmarshalDeleteFileHandlerParams(t *testing.T) {
 	v := DeleteFileHandlerParams{}
 	bts, err := v.MarshalMsg(nil)
@@ -687,6 +800,119 @@ func BenchmarkDecodeDeleteVersionHandlerParams(b *testing.B) {
 	}
 }

+func TestMarshalUnmarshalDeleteVersionsErrsResp(t *testing.T) {
+	v := DeleteVersionsErrsResp{}
+	bts, err := v.MarshalMsg(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	left, err := v.UnmarshalMsg(bts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(left) > 0 {
+		t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
+	}
+
+	left, err = msgp.Skip(bts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(left) > 0 {
+		t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
+	}
+}
+
+func BenchmarkMarshalMsgDeleteVersionsErrsResp(b *testing.B) {
+	v := DeleteVersionsErrsResp{}
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		v.MarshalMsg(nil)
+	}
+}
+
+func BenchmarkAppendMsgDeleteVersionsErrsResp(b *testing.B) {
+	v := DeleteVersionsErrsResp{}
+	bts := make([]byte, 0, v.Msgsize())
+	bts, _ = v.MarshalMsg(bts[0:0])
+	b.SetBytes(int64(len(bts)))
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		bts, _ = v.MarshalMsg(bts[0:0])
+	}
+}
+
+func BenchmarkUnmarshalDeleteVersionsErrsResp(b *testing.B) {
+	v := DeleteVersionsErrsResp{}
+	bts, _ := v.MarshalMsg(nil)
+	b.ReportAllocs()
+	b.SetBytes(int64(len(bts)))
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		_, err := v.UnmarshalMsg(bts)
+		if err != nil {
+			b.Fatal(err)
+		}
+	}
+}
+
+func TestEncodeDecodeDeleteVersionsErrsResp(t *testing.T) {
+	v := DeleteVersionsErrsResp{}
+	var buf bytes.Buffer
+	msgp.Encode(&buf, &v)
+
+	m := v.Msgsize()
+	if buf.Len() > m {
+		t.Log("WARNING: TestEncodeDecodeDeleteVersionsErrsResp Msgsize() is inaccurate")
+	}
+
+	vn := DeleteVersionsErrsResp{}
+	err := msgp.Decode(&buf, &vn)
+	if err != nil {
+		t.Error(err)
+	}
+
+	buf.Reset()
+	msgp.Encode(&buf, &v)
+	err = msgp.NewReader(&buf).Skip()
+	if err != nil {
+		t.Error(err)
+	}
+}
+
+func BenchmarkEncodeDeleteVersionsErrsResp(b *testing.B) {
+	v := DeleteVersionsErrsResp{}
+	var buf bytes.Buffer
+	msgp.Encode(&buf, &v)
+	b.SetBytes(int64(buf.Len()))
+	en := msgp.NewWriter(msgp.Nowhere)
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		v.EncodeMsg(en)
+	}
+	en.Flush()
+}
+
+func BenchmarkDecodeDeleteVersionsErrsResp(b *testing.B) {
+	v := DeleteVersionsErrsResp{}
+	var buf bytes.Buffer
+	msgp.Encode(&buf, &v)
+	b.SetBytes(int64(buf.Len()))
+	rd := msgp.NewEndlessReader(buf.Bytes(), b)
+	dc := msgp.NewReader(rd)
+	b.ReportAllocs()
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		err := v.DecodeMsg(dc)
+		if err != nil {
+			b.Fatal(err)
+		}
+	}
+}
+
 func TestMarshalUnmarshalDiskInfo(t *testing.T) {
 	v := DiskInfo{}
 	bts, err := v.MarshalMsg(nil)
@@ -81,6 +81,7 @@ type StorageAPI interface {
 	// Metadata operations
 	DeleteVersion(ctx context.Context, volume, path string, fi FileInfo, forceDelMarker bool, opts DeleteOptions) error
 	DeleteVersions(ctx context.Context, volume string, versions []FileInfoVersions, opts DeleteOptions) []error
+	DeleteBulk(ctx context.Context, volume string, paths ...string) error
 	WriteMetadata(ctx context.Context, origvolume, volume, path string, fi FileInfo) error
 	UpdateMetadata(ctx context.Context, volume, path string, fi FileInfo, opts UpdateMetadataOpts) error
 	ReadVersion(ctx context.Context, origvolume, volume, path, versionID string, opts ReadOptions) (FileInfo, error)
@@ -173,14 +173,28 @@ func (client *storageRESTClient) GetDiskLoc() (poolIdx, setIdx, diskIdx int) {
 	return client.endpoint.PoolIdx, client.endpoint.SetIdx, client.endpoint.DiskIdx
 }

-// Wrapper to restClient.Call to handle network errors, in case of network error the connection is disconnected
+// Wrapper to restClient.CallWithMethod to handle network errors, in case of network error the connection is disconnected
 // and a healthcheck routine gets invoked that would reconnect.
-func (client *storageRESTClient) call(ctx context.Context, method string, values url.Values, body io.Reader, length int64) (io.ReadCloser, error) {
+func (client *storageRESTClient) callGet(ctx context.Context, rpcMethod string, values url.Values, body io.Reader, length int64) (io.ReadCloser, error) {
 	if values == nil {
 		values = make(url.Values)
 	}
 	values.Set(storageRESTDiskID, *client.diskID.Load())
-	respBody, err := client.restClient.Call(ctx, method, values, body, length)
+	respBody, err := client.restClient.CallWithHTTPMethod(ctx, http.MethodGet, rpcMethod, values, body, length)
+	if err != nil {
+		return nil, toStorageErr(err)
+	}
+	return respBody, nil
+}
+
+// Wrapper to restClient.Call to handle network errors, in case of network error the connection is disconnected
+// and a healthcheck routine gets invoked that would reconnect.
+func (client *storageRESTClient) call(ctx context.Context, rpcMethod string, values url.Values, body io.Reader, length int64) (io.ReadCloser, error) {
+	if values == nil {
+		values = make(url.Values)
+	}
+	values.Set(storageRESTDiskID, *client.diskID.Load())
+	respBody, err := client.restClient.CallWithHTTPMethod(ctx, http.MethodPost, rpcMethod, values, body, length)
 	if err != nil {
 		return nil, toStorageErr(err)
 	}
@@ -499,7 +513,7 @@ var readMsgpReaderPool = sync.Pool{New: func() interface{} { return &msgp.Reader
 func msgpNewReader(r io.Reader) *msgp.Reader {
 	p := readMsgpReaderPool.Get().(*msgp.Reader)
 	if p.R == nil {
-		p.R = xbufio.NewReaderSize(r, 4<<10)
+		p.R = xbufio.NewReaderSize(r, 32<<10)
 	} else {
 		p.R.Reset(r)
 	}
@@ -526,7 +540,6 @@ func (client *storageRESTClient) ReadVersion(ctx context.Context, origvolume, vo
 			storageRESTFilePath:         path,
 			storageRESTVersionID:        versionID,
 			storageRESTInclFreeVersions: strconv.FormatBool(opts.InclFreeVersions),
-			storageRESTReadData:         strconv.FormatBool(opts.ReadData),
 			storageRESTHealing:          strconv.FormatBool(opts.Healing),
 		}))
 		if err != nil {
@@ -541,10 +554,9 @@ func (client *storageRESTClient) ReadVersion(ctx context.Context, origvolume, vo
 	values.Set(storageRESTFilePath, path)
 	values.Set(storageRESTVersionID, versionID)
 	values.Set(storageRESTInclFreeVersions, strconv.FormatBool(opts.InclFreeVersions))
-	values.Set(storageRESTReadData, strconv.FormatBool(opts.ReadData))
 	values.Set(storageRESTHealing, strconv.FormatBool(opts.Healing))

-	respBody, err := client.call(ctx, storageRESTMethodReadVersion, values, nil, -1)
+	respBody, err := client.callGet(ctx, storageRESTMethodReadVersion, values, nil, -1)
 	if err != nil {
 		return fi, err
 	}
@@ -568,7 +580,6 @@ func (client *storageRESTClient) ReadXL(ctx context.Context, volume string, path
 			storageRESTDiskID:   *client.diskID.Load(),
 			storageRESTVolume:   volume,
 			storageRESTFilePath: path,
-			storageRESTReadData: "false",
 		}))
 		if err != nil {
 			return rf, toStorageErr(err)
@@ -579,8 +590,8 @@ func (client *storageRESTClient) ReadXL(ctx context.Context, volume string, path
 	values := make(url.Values)
 	values.Set(storageRESTVolume, volume)
 	values.Set(storageRESTFilePath, path)
-	values.Set(storageRESTReadData, strconv.FormatBool(readData))
-	respBody, err := client.call(ctx, storageRESTMethodReadXL, values, nil, -1)
+
+	respBody, err := client.callGet(ctx, storageRESTMethodReadXL, values, nil, -1)
 	if err != nil {
 		return rf, toStorageErr(err)
 	}
@@ -617,9 +628,10 @@ func (client *storageRESTClient) ReadFileStream(ctx context.Context, volume, pat
 	values.Set(storageRESTFilePath, path)
 	values.Set(storageRESTOffset, strconv.Itoa(int(offset)))
 	values.Set(storageRESTLength, strconv.Itoa(int(length)))
-	respBody, err := client.call(ctx, storageRESTMethodReadFileStream, values, nil, -1)
+
+	respBody, err := client.callGet(ctx, storageRESTMethodReadFileStream, values, nil, -1)
 	if err != nil {
-		return nil, err
+		return nil, toStorageErr(err)
 	}
 	return respBody, nil
 }
@@ -638,7 +650,7 @@ func (client *storageRESTClient) ReadFile(ctx context.Context, volume string, pa
 		values.Set(storageRESTBitrotAlgo, "")
 		values.Set(storageRESTBitrotHash, "")
 	}
-	respBody, err := client.call(ctx, storageRESTMethodReadFile, values, nil, -1)
+	respBody, err := client.callGet(ctx, storageRESTMethodReadFile, values, nil, -1)
 	if err != nil {
 		return 0, err
 	}
@@ -725,7 +737,9 @@ func (client *storageRESTClient) DeleteVersions(ctx context.Context, volume stri
 	}

 	dErrResp := &DeleteVersionsErrsResp{}
-	if err = gob.NewDecoder(reader).Decode(dErrResp); err != nil {
+	decoder := msgpNewReader(reader)
+	defer readMsgpReaderPoolPut(decoder)
+	if err = dErrResp.DecodeMsg(decoder); err != nil {
 		for i := range errs {
 			errs[i] = toStorageErr(err)
 		}
@@ -733,7 +747,11 @@ func (client *storageRESTClient) DeleteVersions(ctx context.Context, volume stri
 	}

 	for i, dErr := range dErrResp.Errs {
-		errs[i] = toStorageErr(dErr)
+		if dErr != "" {
+			errs[i] = toStorageErr(errors.New(dErr))
+		} else {
+			errs[i] = nil
+		}
 	}

 	return errs
@@ -783,6 +801,26 @@ func (client *storageRESTClient) VerifyFile(ctx context.Context, volume, path st
 	return verifyResp, nil
 }

+func (client *storageRESTClient) DeleteBulk(ctx context.Context, volume string, paths ...string) (err error) {
+	values := make(url.Values)
+	values.Set(storageRESTVolume, volume)
+
+	req := &DeleteBulkReq{Paths: paths}
+	body, err := req.MarshalMsg(nil)
+	if err != nil {
+		return err
+	}
+
+	respBody, err := client.call(ctx, storageRESTMethodDeleteBulk, values, bytes.NewReader(body), int64(len(body)))
+	if err != nil {
+		return err
+	}
+	defer xhttp.DrainBody(respBody)
+
+	_, err = waitForHTTPResponse(respBody)
+	return toStorageErr(err)
+}
+
 func (client *storageRESTClient) StatInfoFile(ctx context.Context, volume, path string, glob bool) (stat []StatInfo, err error) {
 	values := make(url.Values)
 	values.Set(storageRESTVolume, volume)
@@ -20,7 +20,7 @@ package cmd
 //go:generate msgp -file $GOFILE -unexported

 const (
-	storageRESTVersion       = "v59" // Change ReadOptions inclFreeVersions
+	storageRESTVersion       = "v62" // Introduce DeleteBulk internode API.
 	storageRESTVersionPrefix = SlashSeparator + storageRESTVersion
 	storageRESTPrefix        = minioReservedBucketPath + "/storage"
 )
@@ -28,48 +28,49 @@ const (
 const (
 	storageRESTMethodHealth = "/health"

-	storageRESTMethodAppendFile     = "/appendfile"
-	storageRESTMethodCreateFile     = "/createfile"
-	storageRESTMethodWriteAll       = "/writeall"
-	storageRESTMethodReadVersion    = "/readversion"
-	storageRESTMethodReadXL         = "/readxl"
-	storageRESTMethodReadAll        = "/readall"
-	storageRESTMethodReadFile       = "/readfile"
-	storageRESTMethodReadFileStream = "/readfilestream"
-	storageRESTMethodListDir        = "/listdir"
-	storageRESTMethodDeleteVersions = "/deleteverions"
-	storageRESTMethodRenameFile     = "/renamefile"
-	storageRESTMethodVerifyFile     = "/verifyfile"
-	storageRESTMethodStatInfoFile   = "/statfile"
-	storageRESTMethodReadMultiple   = "/readmultiple"
-	storageRESTMethodCleanAbandoned = "/cleanabandoned"
+	storageRESTMethodAppendFile     = "/afile"
+	storageRESTMethodCreateFile     = "/cfile"
+	storageRESTMethodWriteAll       = "/wall"
+	storageRESTMethodReadVersion    = "/rver"
+	storageRESTMethodReadXL         = "/rxl"
+	storageRESTMethodReadAll        = "/rall"
+	storageRESTMethodReadFile       = "/rfile"
+	storageRESTMethodReadFileStream = "/rfilest"
+	storageRESTMethodListDir        = "/ls"
+	storageRESTMethodDeleteVersions = "/dvers"
+	storageRESTMethodRenameFile     = "/rfile"
+	storageRESTMethodVerifyFile     = "/vfile"
+	storageRESTMethodStatInfoFile   = "/sfile"
+	storageRESTMethodReadMultiple   = "/rmpl"
+	storageRESTMethodCleanAbandoned = "/cln"
+	storageRESTMethodDeleteBulk     = "/dblk"
 )

 const (
-	storageRESTVolume           = "volume"
-	storageRESTVolumes          = "volumes"
-	storageRESTDirPath          = "dir-path"
-	storageRESTFilePath         = "file-path"
-	storageRESTVersionID        = "version-id"
-	storageRESTReadData         = "read-data"
-	storageRESTHealing          = "healing"
-	storageRESTTotalVersions    = "total-versions"
-	storageRESTSrcVolume        = "source-volume"
-	storageRESTSrcPath          = "source-path"
-	storageRESTDstVolume        = "destination-volume"
-	storageRESTDstPath          = "destination-path"
+	storageRESTVolume           = "vol"
+	storageRESTVolumes          = "vols"
+	storageRESTDirPath          = "dpath"
+	storageRESTFilePath         = "fp"
+	storageRESTVersionID        = "vid"
+	storageRESTHealing          = "heal"
+	storageRESTTotalVersions    = "tvers"
+	storageRESTSrcVolume        = "svol"
+	storageRESTSrcPath          = "spath"
+	storageRESTDstVolume        = "dvol"
+	storageRESTDstPath          = "dpath"
 	storageRESTOffset           = "offset"
 	storageRESTLength           = "length"
 	storageRESTCount            = "count"
-	storageRESTBitrotAlgo       = "bitrot-algo"
-	storageRESTBitrotHash       = "bitrot-hash"
-	storageRESTDiskID           = "disk-id"
-	storageRESTForceDelete      = "force-delete"
+	storageRESTBitrotAlgo       = "balg"
+	storageRESTBitrotHash       = "bhash"
+	storageRESTDiskID           = "did"
+	storageRESTForceDelete      = "fdel"
 	storageRESTGlob             = "glob"
 	storageRESTMetrics          = "metrics"
-	storageRESTDriveQuorum      = "drive-quorum"
-	storageRESTOrigVolume       = "orig-volume"
-	storageRESTInclFreeVersions = "incl-free-versions"
+	storageRESTDriveQuorum      = "dquorum"
+	storageRESTOrigVolume       = "ovol"
+	storageRESTInclFreeVersions = "incl-fv"
+	storageRESTRange            = "rng"
 )

 type nsScannerOptions struct {
@@ -21,7 +21,6 @@ import (
 	"bufio"
 	"context"
 	"encoding/binary"
-	"encoding/gob"
 	"encoding/hex"
 	"errors"
 	"fmt"
@@ -29,7 +28,6 @@ import (
 	"net/http"
 	"os/user"
 	"path"
-	"runtime"
 	"runtime/debug"
 	"strconv"
 	"strings"
@@ -110,7 +108,7 @@ func (s *storageRESTServer) writeErrorResponse(w http.ResponseWriter, err error)
 const DefaultSkewTime = 15 * time.Minute

 // validateStorageRequestToken will validate the token against the provided audience.
-func validateStorageRequestToken(token, audience string) error {
+func validateStorageRequestToken(token string) error {
 	claims := xjwt.NewStandardClaims()
 	if err := xjwt.ParseWithStandardClaims(token, claims, []byte(globalActiveCred.SecretKey)); err != nil {
 		return errAuthentication
@@ -121,9 +119,6 @@ func validateStorageRequestToken(token, audience string) error {
 		return errAuthentication
 	}

-	if claims.Audience != audience {
-		return errAuthentication
-	}
 	return nil
 }

@@ -136,20 +131,24 @@ func storageServerRequestValidate(r *http.Request) error {
 		}
 		return errMalformedAuth
 	}
-	if err = validateStorageRequestToken(token, r.URL.RawQuery); err != nil {
+
+	if err = validateStorageRequestToken(token); err != nil {
 		return err
 	}

-	requestTimeStr := r.Header.Get("X-Minio-Time")
-	requestTime, err := time.Parse(time.RFC3339, requestTimeStr)
+	nanoTime, err := strconv.ParseInt(r.Header.Get("X-Minio-Time"), 10, 64)
 	if err != nil {
 		return errMalformedAuth
 	}
-	utcNow := UTCNow()
-	delta := requestTime.Sub(utcNow)
+
+	localTime := UTCNow()
+	remoteTime := time.Unix(0, nanoTime)
+
+	delta := remoteTime.Sub(localTime)
 	if delta < 0 {
 		delta *= -1
 	}
+
 	if delta > DefaultSkewTime {
 		return errSkewedAuthTime
 	}
@@ -375,10 +374,6 @@ func (s *storageRESTServer) ReadVersionHandlerWS(params *grid.MSS) (*FileInfo, *
 	volume := params.Get(storageRESTVolume)
 	filePath := params.Get(storageRESTFilePath)
 	versionID := params.Get(storageRESTVersionID)
-	readData, err := strconv.ParseBool(params.Get(storageRESTReadData))
-	if err != nil {
-		return nil, grid.NewRemoteErr(err)
-	}

 	healing, err := strconv.ParseBool(params.Get(storageRESTHealing))
 	if err != nil {
@@ -392,7 +387,7 @@ func (s *storageRESTServer) ReadVersionHandlerWS(params *grid.MSS) (*FileInfo, *

 	fi, err := s.getStorage().ReadVersion(context.Background(), origvolume, volume, filePath, versionID, ReadOptions{
 		InclFreeVersions: inclFreeVersions,
-		ReadData:         readData,
+		ReadData:         false,
 		Healing:          healing,
 	})
 	if err != nil {
@@ -410,11 +405,6 @@ func (s *storageRESTServer) ReadVersionHandler(w http.ResponseWriter, r *http.Re
 	volume := r.Form.Get(storageRESTVolume)
 	filePath := r.Form.Get(storageRESTFilePath)
 	versionID := r.Form.Get(storageRESTVersionID)
-	readData, err := strconv.ParseBool(r.Form.Get(storageRESTReadData))
-	if err != nil {
-		s.writeErrorResponse(w, err)
-		return
-	}
 	healing, err := strconv.ParseBool(r.Form.Get(storageRESTHealing))
 	if err != nil {
 		s.writeErrorResponse(w, err)
@@ -429,7 +419,7 @@ func (s *storageRESTServer) ReadVersionHandler(w http.ResponseWriter, r *http.Re

 	fi, err := s.getStorage().ReadVersion(r.Context(), origvolume, volume, filePath, versionID, ReadOptions{
 		InclFreeVersions: inclFreeVersions,
-		ReadData:         readData,
+		ReadData:         true,
 		Healing:          healing,
 	})
 	if err != nil {
@@ -506,15 +496,11 @@ func (s *storageRESTServer) ReadXLHandler(w http.ResponseWriter, r *http.Request
 	if !s.IsValid(w, r) {
 		return
 	}
+
 	volume := r.Form.Get(storageRESTVolume)
 	filePath := r.Form.Get(storageRESTFilePath)
-	readData, err := strconv.ParseBool(r.Form.Get(storageRESTReadData))
-	if err != nil {
-		s.writeErrorResponse(w, err)
-		return
-	}

-	rf, err := s.getStorage().ReadXL(r.Context(), volume, filePath, readData)
+	rf, err := s.getStorage().ReadXL(r.Context(), volume, filePath, true)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
@@ -528,14 +514,10 @@ func (s *storageRESTServer) ReadXLHandlerWS(params *grid.MSS) (*RawFileInfo, *gr
 	if !s.checkID(params.Get(storageRESTDiskID)) {
 		return nil, grid.NewRemoteErr(errDiskNotFound)
 	}
+
 	volume := params.Get(storageRESTVolume)
 	filePath := params.Get(storageRESTFilePath)
-	readData, err := strconv.ParseBool(params.Get(storageRESTReadData))
-	if err != nil {
-		return nil, grid.NewRemoteErr(err)
-	}
-
-	rf, err := s.getStorage().ReadXL(context.Background(), volume, filePath, readData)
+	rf, err := s.getStorage().ReadXL(context.Background(), volume, filePath, false)
 	if err != nil {
 		return nil, grid.NewRemoteErr(err)
 	}
@@ -593,45 +575,24 @@ func (s *storageRESTServer) ReadFileStreamHandler(w http.ResponseWriter, r *http
 	}
 	volume := r.Form.Get(storageRESTVolume)
 	filePath := r.Form.Get(storageRESTFilePath)
-	offset, err := strconv.Atoi(r.Form.Get(storageRESTOffset))
+	offset, err := strconv.ParseInt(r.Form.Get(storageRESTOffset), 10, 64)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
 	}
-	length, err := strconv.Atoi(r.Form.Get(storageRESTLength))
+	length, err := strconv.ParseInt(r.Form.Get(storageRESTLength), 10, 64)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
 	}

-	w.Header().Set(xhttp.ContentLength, strconv.Itoa(length))
-
-	rc, err := s.getStorage().ReadFileStream(r.Context(), volume, filePath, int64(offset), int64(length))
+	rc, err := s.getStorage().ReadFileStream(r.Context(), volume, filePath, offset, length)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
 	}
 	defer rc.Close()

-	rf, ok := w.(io.ReaderFrom)
-	if ok && runtime.GOOS != "windows" {
-		// Attempt to use splice/sendfile() optimization, A very specific behavior mentioned below is necessary.
-		// See https://github.com/golang/go/blob/f7c5cbb82087c55aa82081e931e0142783700ce8/src/net/sendfile_linux.go#L20
-		// Windows can lock up with this optimization, so we fall back to regular copy.
-		sr, ok := rc.(*sendFileReader)
-		if ok {
-			// Sendfile sends in 4MiB chunks per sendfile syscall which is more than enough
-			// for most setups.
-			_, err = rf.ReadFrom(sr.Reader)
-			if !xnet.IsNetworkOrHostDown(err, true) { // do not need to log disconnected clients
-				storageLogIf(r.Context(), err)
-			}
-			if err == nil || !errors.Is(err, xhttp.ErrNotImplemented) {
-				return
-			}
-		}
-	} // Fallback to regular copy
-
 	_, err = xioutil.Copy(w, rc)
 	if !xnet.IsNetworkOrHostDown(err, true) { // do not need to log disconnected clients
 		storageLogIf(r.Context(), err)
@@ -667,12 +628,6 @@ func (s *storageRESTServer) DeleteFileHandler(p *DeleteFileHandlerParams) (grid.
 	return grid.NewNPErr(s.getStorage().Delete(context.Background(), p.Volume, p.FilePath, p.Opts))
 }

-// DeleteVersionsErrsResp - collection of delete errors
-// for bulk version deletes
-type DeleteVersionsErrsResp struct {
-	Errs []error
-}
-
 // DeleteVersionsHandler - delete a set of a versions.
 func (s *storageRESTServer) DeleteVersionsHandler(w http.ResponseWriter, r *http.Request) {
 	if !s.IsValid(w, r) {
@@ -697,21 +652,20 @@ func (s *storageRESTServer) DeleteVersionsHandler(w http.ResponseWriter, r *http
 		}
 	}

-	dErrsResp := &DeleteVersionsErrsResp{Errs: make([]error, totalVersions)}
-
-	setEventStreamHeaders(w)
-	encoder := gob.NewEncoder(w)
 	done := keepHTTPResponseAlive(w)
-
 	opts := DeleteOptions{}
 	errs := s.getStorage().DeleteVersions(r.Context(), volume, versions, opts)
 	done(nil)
+
+	dErrsResp := &DeleteVersionsErrsResp{Errs: make([]string, totalVersions)}
 	for idx := range versions {
 		if errs[idx] != nil {
-			dErrsResp.Errs[idx] = StorageErr(errs[idx].Error())
+			dErrsResp.Errs[idx] = errs[idx].Error()
 		}
 	}
-	encoder.Encode(dErrsResp)
+
+	buf, _ := dErrsResp.MarshalMsg(nil)
+	w.Write(buf)
 }

 // RenameDataHandler - renames a meta object and data dir to destination.
@@ -1145,18 +1099,15 @@ func (s *storageRESTServer) VerifyFileHandler(w http.ResponseWriter, r *http.Req
 		return
 	}

-	setEventStreamHeaders(w)
-	encoder := gob.NewEncoder(w)
 	done := keepHTTPResponseAlive(w)
 	resp, err := s.getStorage().VerifyFile(r.Context(), volume, filePath, fi)
-	done(nil)
-
+	done(err)
 	if err != nil {
-		s.writeErrorResponse(w, err)
 		return
 	}

-	encoder.Encode(resp)
+	buf, _ := resp.MarshalMsg(nil)
+	w.Write(buf)
 }

 func checkDiskFatalErrs(errs []error) error {
@@ -1281,6 +1232,24 @@ func (s *storageRESTServer) StatInfoFile(w http.ResponseWriter, r *http.Request)
 	}
 }

+func (s *storageRESTServer) DeleteBulkHandler(w http.ResponseWriter, r *http.Request) {
+	if !s.IsValid(w, r) {
+		return
+	}
+
+	var req DeleteBulkReq
+	mr := msgpNewReader(r.Body)
+	defer readMsgpReaderPoolPut(mr)
+
+	if err := req.DecodeMsg(mr); err != nil {
+		s.writeErrorResponse(w, err)
+		return
+	}
+
+	volume := r.Form.Get(storageRESTVolume)
+	keepHTTPResponseAlive(w)(s.getStorage().DeleteBulk(r.Context(), volume, req.Paths...))
+}
+
 // ReadMultiple returns multiple files
 func (s *storageRESTServer) ReadMultiple(w http.ResponseWriter, r *http.Request) {
 	if !s.IsValid(w, r) {
@@ -1335,6 +1304,7 @@ func registerStorageRESTHandlers(router *mux.Router, endpointServerPools Endpoin
 		return collectInternodeStats(httpTraceHdrs(f))
 	}

+	globalLocalDrivesMap = make(map[string]StorageAPI)
 	globalLocalSetDrives = make([][][]StorageAPI, len(endpointServerPools))
 	for pool := range globalLocalSetDrives {
 		globalLocalSetDrives[pool] = make([][]StorageAPI, endpointServerPools[pool].SetCount)
@@ -1356,17 +1326,19 @@ func registerStorageRESTHandlers(router *mux.Router, endpointServerPools Endpoin

 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodHealth).HandlerFunc(h(server.HealthHandler))
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodAppendFile).HandlerFunc(h(server.AppendFileHandler))
-			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadVersion).HandlerFunc(h(server.ReadVersionHandler))
-			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadXL).HandlerFunc(h(server.ReadXLHandler))
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodCreateFile).HandlerFunc(h(server.CreateFileHandler))
-			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadFile).HandlerFunc(h(server.ReadFileHandler))
-			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadFileStream).HandlerFunc(h(server.ReadFileStreamHandler))
-
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodDeleteVersions).HandlerFunc(h(server.DeleteVersionsHandler))
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodVerifyFile).HandlerFunc(h(server.VerifyFileHandler))
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodStatInfoFile).HandlerFunc(h(server.StatInfoFile))
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodReadMultiple).HandlerFunc(h(server.ReadMultiple))
 			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodCleanAbandoned).HandlerFunc(h(server.CleanAbandonedDataHandler))
+			subrouter.Methods(http.MethodPost).Path(storageRESTVersionPrefix + storageRESTMethodDeleteBulk).HandlerFunc(h(server.DeleteBulkHandler))
+
+			subrouter.Methods(http.MethodGet).Path(storageRESTVersionPrefix + storageRESTMethodReadFileStream).HandlerFunc(h(server.ReadFileStreamHandler))
+			subrouter.Methods(http.MethodGet).Path(storageRESTVersionPrefix + storageRESTMethodReadVersion).HandlerFunc(h(server.ReadVersionHandler))
+			subrouter.Methods(http.MethodGet).Path(storageRESTVersionPrefix + storageRESTMethodReadXL).HandlerFunc(h(server.ReadXLHandler))
+			subrouter.Methods(http.MethodGet).Path(storageRESTVersionPrefix + storageRESTMethodReadFile).HandlerFunc(h(server.ReadFileHandler))
+
 			logger.FatalIf(storageListDirRPC.RegisterNoInput(gm, server.ListDirHandler, endpoint.Path), "unable to register handler")
 			logger.FatalIf(storageReadAllRPC.Register(gm, server.ReadAllHandler, endpoint.Path), "unable to register handler")
 			logger.FatalIf(storageWriteAllRPC.Register(gm, server.WriteAllHandler, endpoint.Path), "unable to register handler")
@@ -1408,7 +1380,6 @@ func registerStorageRESTHandlers(router *mux.Router, endpointServerPools Endpoin
 				globalLocalDrivesMu.Lock()
 				defer globalLocalDrivesMu.Unlock()

-				globalLocalDrives = append(globalLocalDrives, storage)
 				globalLocalDrivesMap[endpoint.String()] = storage
 				globalLocalSetDrives[endpoint.PoolIdx][endpoint.SetIdx][endpoint.DiskIdx] = storage
 				return true
@@ -315,6 +315,7 @@ func newStorageRESTHTTPServerClient(t testing.TB) *storageRESTClient {
 	url.Path = t.TempDir()

 	globalMinioHost, globalMinioPort = mustSplitHostPort(url.Host)
+	globalNodeAuthToken, _ = authenticateNode(globalActiveCred.AccessKey, globalActiveCred.SecretKey)

 	endpoint, err := NewEndpoint(url.String())
 	if err != nil {
@@ -36,12 +36,13 @@ func _() {
 	_ = x[storageMetricReadMultiple-25]
 	_ = x[storageMetricDeleteAbandonedParts-26]
 	_ = x[storageMetricDiskInfo-27]
-	_ = x[storageMetricLast-28]
+	_ = x[storageMetricDeleteBulk-28]
+	_ = x[storageMetricLast-29]
 }

-const _storageMetric_name = "MakeVolBulkMakeVolListVolsStatVolDeleteVolWalkDirListDirReadFileAppendFileCreateFileReadFileStreamRenameFileRenameDataCheckPartsDeleteDeleteVersionsVerifyFileWriteAllDeleteVersionWriteMetadataUpdateMetadataReadVersionReadXLReadAllStatInfoFileReadMultipleDeleteAbandonedPartsDiskInfoLast"
+const _storageMetric_name = "MakeVolBulkMakeVolListVolsStatVolDeleteVolWalkDirListDirReadFileAppendFileCreateFileReadFileStreamRenameFileRenameDataCheckPartsDeleteDeleteVersionsVerifyFileWriteAllDeleteVersionWriteMetadataUpdateMetadataReadVersionReadXLReadAllStatInfoFileReadMultipleDeleteAbandonedPartsDiskInfoDeleteBulkLast"

-var _storageMetric_index = [...]uint16{0, 11, 18, 26, 33, 42, 49, 56, 64, 74, 84, 98, 108, 118, 128, 134, 148, 158, 166, 179, 192, 206, 217, 223, 230, 242, 254, 274, 282, 286}
+var _storageMetric_index = [...]uint16{0, 11, 18, 26, 33, 42, 49, 56, 64, 74, 84, 98, 108, 118, 128, 134, 148, 158, 166, 179, 192, 206, 217, 223, 230, 242, 254, 274, 282, 292, 296}

 func (i storageMetric) String() string {
 	if i >= storageMetric(len(_storageMetric_index)-1) {
@@ -83,6 +83,8 @@ func TestMain(m *testing.M) {
 		SecretKey: auth.DefaultSecretKey,
 	}

+	globalNodeAuthToken, _ = authenticateNode(auth.DefaultAccessKey, auth.DefaultSecretKey)
+
 	// disable ENVs which interfere with tests.
 	for _, env := range []string{
 		crypto.EnvKMSAutoEncryption,
@@ -100,7 +102,7 @@ func TestMain(m *testing.M) {
 	// Disable printing console messages during tests.
 	color.Output = io.Discard
 	// Disable Error logging in testing.
-	logger.DisableErrorLog = true
+	logger.DisableLog = true

 	// Uncomment the following line to see trace logs during unit tests.
 	// logger.AddTarget(console.New())
@@ -70,6 +70,7 @@ const (
 	storageMetricReadMultiple
 	storageMetricDeleteAbandonedParts
 	storageMetricDiskInfo
+	storageMetricDeleteBulk

 	// .... add more

@@ -499,6 +500,16 @@ func (p *xlStorageDiskIDCheck) CheckParts(ctx context.Context, volume string, pa
 	})
 }

+func (p *xlStorageDiskIDCheck) DeleteBulk(ctx context.Context, volume string, paths ...string) (err error) {
+	ctx, done, err := p.TrackDiskHealth(ctx, storageMetricDeleteBulk, append([]string{volume}, paths...)...)
+	if err != nil {
+		return err
+	}
+	defer done(0, &err)
+
+	return p.storage.DeleteBulk(ctx, volume, paths...)
+}
+
 func (p *xlStorageDiskIDCheck) Delete(ctx context.Context, volume string, path string, deleteOpts DeleteOptions) (err error) {
 	ctx, done, err := p.TrackDiskHealth(ctx, storageMetricDelete, volume, path)
 	if err != nil {
@@ -1079,34 +1079,37 @@ func (s *xlStorage) deleteVersions(ctx context.Context, volume, path string, fis
 		return err
 	}

-	discard := true
+	s.RLock()
+	legacy := s.formatLegacy
+	s.RUnlock()

 	var legacyJSON bool
-	buf, _, err := s.readAllData(ctx, volume, volumeDir, pathJoin(volumeDir, path, xlStorageFormatFile), discard)
-	if err != nil {
-		if !errors.Is(err, errFileNotFound) {
-			return err
+	buf, err := xioutil.WithDeadline[[]byte](ctx, globalDriveConfig.GetMaxTimeout(), func(ctx context.Context) ([]byte, error) {
+		buf, _, err := s.readAllData(ctx, volume, volumeDir, pathJoin(volumeDir, path, xlStorageFormatFile))
+		if err != nil && !errors.Is(err, errFileNotFound) {
+			return nil, err
 		}

-		s.RLock()
-		legacy := s.formatLegacy
-		s.RUnlock()
-		if legacy {
-			buf, _, err = s.readAllData(ctx, volume, volumeDir, pathJoin(volumeDir, path, xlStorageFormatFileV1), discard)
+		if errors.Is(err, errFileNotFound) && legacy {
+			buf, _, err = s.readAllData(ctx, volume, volumeDir, pathJoin(volumeDir, path, xlStorageFormatFileV1))
 			if err != nil {
-				return err
+				return nil, err
 			}
 			legacyJSON = true
 		}
-	}

-	if len(buf) == 0 {
-		if errors.Is(err, errFileNotFound) && !skipAccessChecks(volume) {
-			if aerr := Access(volumeDir); aerr != nil && osIsNotExist(aerr) {
-				return errVolumeNotFound
+		if len(buf) == 0 {
+			if errors.Is(err, errFileNotFound) && !skipAccessChecks(volume) {
+				if aerr := Access(volumeDir); aerr != nil && osIsNotExist(aerr) {
+					return nil, errVolumeNotFound
+				}
+				return nil, errFileNotFound
 			}
 		}
-		return errFileNotFound
+		return buf, nil
+	})
+	if err != nil {
+		return err
 	}

 	if legacyJSON {
@@ -1180,10 +1183,7 @@ func (s *xlStorage) DeleteVersions(ctx context.Context, volume string, versions
 			errs[i] = ctx.Err()
 			continue
 		}
-		w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
-		if err := w.Run(func() error { return s.deleteVersions(ctx, volume, fiv.Name, fiv.Versions...) }); err != nil {
-			errs[i] = err
-		}
+		errs[i] = s.deleteVersions(ctx, volume, fiv.Name, fiv.Versions...)
 		diskHealthCheckOK(ctx, errs[i])
 	}

@@ -1214,7 +1214,7 @@ func (s *xlStorage) diskAlmostFilled() bool {
 	return (float64(info.Free)/float64(info.Used)) < almostFilledPercent || (float64(info.FreeInodes)/float64(info.UsedInodes)) < almostFilledPercent
 }

-func (s *xlStorage) moveToTrash(filePath string, recursive, immediatePurge bool) (err error) {
+func (s *xlStorage) moveToTrashNoDeadline(filePath string, recursive, immediatePurge bool) (err error) {
 	pathUUID := mustGetUUID()
 	targetPath := pathutil.Join(s.drivePath, minioMetaTmpDeletedBucket, pathUUID)

@@ -1267,10 +1267,16 @@ func (s *xlStorage) moveToTrash(filePath string, recursive, immediatePurge bool)
 			}
 		}
 	}
-
 	return nil
 }

+func (s *xlStorage) moveToTrash(filePath string, recursive, immediatePurge bool) (err error) {
+	w := xioutil.NewDeadlineWorker(globalDriveConfig.GetMaxTimeout())
+	return w.Run(func() (err error) {
+		return s.moveToTrashNoDeadline(filePath, recursive, immediatePurge)
+	})
+}
+
 // DeleteVersion - deletes FileInfo metadata for path at `xl.meta`. forceDelMarker
 // will force creating a new `xl.meta` to create a new delete marker
 func (s *xlStorage) DeleteVersion(ctx context.Context, volume, path string, fi FileInfo, forceDelMarker bool, opts DeleteOptions) (err error) {
@@ -1293,7 +1299,7 @@ func (s *xlStorage) DeleteVersion(ctx context.Context, volume, path string, fi F
 	}

 	var legacyJSON bool
-	buf, _, err := s.readAllData(ctx, volume, volumeDir, pathJoin(filePath, xlStorageFormatFile), true)
+	buf, _, err := s.readAllData(ctx, volume, volumeDir, pathJoin(filePath, xlStorageFormatFile))
 	if err != nil {
 		if !errors.Is(err, errFileNotFound) {
 			return err
@@ -1558,7 +1564,7 @@ func (s *xlStorage) readRaw(ctx context.Context, volume, volumeDir, filePath str

 	xlPath := pathJoin(filePath, xlStorageFormatFile)
 	if readData {
-		buf, dmTime, err = s.readAllData(ctx, volume, volumeDir, xlPath, false)
+		buf, dmTime, err = s.readAllData(ctx, volume, volumeDir, xlPath)
 	} else {
 		buf, dmTime, err = s.readMetadataWithDMTime(ctx, xlPath)
 		if err != nil {
@@ -1578,7 +1584,7 @@ func (s *xlStorage) readRaw(ctx context.Context, volume, volumeDir, filePath str
 	s.RUnlock()

 	if err != nil && errors.Is(err, errFileNotFound) && legacy {
-		buf, dmTime, err = s.readAllData(ctx, volume, volumeDir, pathJoin(filePath, xlStorageFormatFileV1), false)
+		buf, dmTime, err = s.readAllData(ctx, volume, volumeDir, pathJoin(filePath, xlStorageFormatFileV1))
 		if err != nil {
 			return nil, time.Time{}, err
 		}
@@ -1703,18 +1709,23 @@ func (s *xlStorage) ReadVersion(ctx context.Context, origvolume, volume, path, v
 			fi.Data = nil
 		}

+		attemptInline := fi.TransitionStatus == "" && fi.DataDir != "" && len(fi.Parts) == 1
 		// Reading data for small objects when
 		// - object has not yet transitioned
-		// - object size lesser than 128KiB
 		// - object has maximum of 1 parts
-		if fi.TransitionStatus == "" &&
-			fi.DataDir != "" && fi.Size <= smallFileThreshold &&
-			len(fi.Parts) == 1 {
-			partPath := fmt.Sprintf("part.%d", fi.Parts[0].Number)
-			dataPath := pathJoin(volumeDir, path, fi.DataDir, partPath)
-			fi.Data, _, err = s.readAllData(ctx, volume, volumeDir, dataPath, false)
-			if err != nil {
-				return FileInfo{}, err
+		if attemptInline {
+			inlineBlock := globalStorageClass.InlineBlock()
+			if inlineBlock <= 0 {
+				inlineBlock = 128 * humanize.KiByte
+			}
+
+			canInline := fi.ShardFileSize(fi.Parts[0].ActualSize) <= inlineBlock
+			if canInline {
+				dataPath := pathJoin(volumeDir, path, fi.DataDir, fmt.Sprintf("part.%d", fi.Parts[0].Number))
+				fi.Data, _, err = s.readAllData(ctx, volume, volumeDir, dataPath)
+				if err != nil {
+					return FileInfo{}, err
+				}
 			}
 		}
 	}
@@ -1722,7 +1733,7 @@ func (s *xlStorage) ReadVersion(ctx context.Context, origvolume, volume, path, v
 	return fi, nil
 }

-func (s *xlStorage) readAllData(ctx context.Context, volume, volumeDir string, filePath string, discard bool) (buf []byte, dmTime time.Time, err error) {
+func (s *xlStorage) readAllData(ctx context.Context, volume, volumeDir string, filePath string) (buf []byte, dmTime time.Time, err error) {
 	if filePath == "" {
 		return nil, dmTime, errFileNotFound
 	}
@@ -1766,14 +1777,6 @@ func (s *xlStorage) readAllData(ctx context.Context, volume, volumeDir string, f
 		}
 		return nil, dmTime, err
 	}
-
-	if discard {
-		// This discard is mostly true for DELETEEs
-		// so we need to make sure we do not keep
-		// page-cache references after.
-		defer disk.Fdatasync(f)
-	}
-
 	defer f.Close()

 	// Get size for precise allocation.
@@ -1825,7 +1828,7 @@ func (s *xlStorage) ReadAll(ctx context.Context, volume string, path string) (bu
 		return nil, err
 	}

-	buf, _, err = s.readAllData(ctx, volume, volumeDir, filePath, false)
+	buf, _, err = s.readAllData(ctx, volume, volumeDir, filePath)
 	return buf, err
 }

@@ -2062,7 +2065,6 @@ func (s *xlStorage) ReadFileStream(ctx context.Context, volume, path string, off
 			return nil, err
 		}
 	}
-
 	return &sendFileReader{Reader: io.LimitReader(file, length), Closer: file}, nil
 }

@@ -2423,7 +2425,41 @@ func (s *xlStorage) deleteFile(basePath, deletePath string, recursive, immediate
 	return nil
 }

-// DeleteFile - delete a file at path.
+// DeleteBulk - delete many files in bulk to trash.
+// this delete does not recursively delete empty
+// parents, if you need empty parent delete support
+// please use Delete() instead. This API is meant as
+// an optimization for Multipart operations.
+func (s *xlStorage) DeleteBulk(ctx context.Context, volume string, paths ...string) (err error) {
+	volumeDir, err := s.getVolDir(volume)
+	if err != nil {
+		return err
+	}
+
+	if !skipAccessChecks(volume) {
+		// Stat a volume entry.
+		if err = Access(volumeDir); err != nil {
+			return convertAccessError(err, errVolumeAccessDenied)
+		}
+	}
+
+	for _, fp := range paths {
+		// Following code is needed so that we retain SlashSeparator suffix if any in
+		// path argument.
+		filePath := pathJoin(volumeDir, fp)
+		if err = checkPathLength(filePath); err != nil {
+			return err
+		}
+
+		if err = s.moveToTrash(filePath, false, false); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// Delete - delete a file at path.
 func (s *xlStorage) Delete(ctx context.Context, volume string, path string, deleteOpts DeleteOptions) (err error) {
 	volumeDir, err := s.getVolDir(volume)
 	if err != nil {
@@ -2993,7 +3029,7 @@ func (s *xlStorage) ReadMultiple(ctx context.Context, req ReadMultipleReq, resp
 			if req.MetadataOnly {
 				data, mt, err = s.readMetadataWithDMTime(ctx, fullPath)
 			} else {
-				data, mt, err = s.readAllData(ctx, req.Bucket, volumeDir, fullPath, true)
+				data, mt, err = s.readAllData(ctx, req.Bucket, volumeDir, fullPath)
 			}
 			return err
 		}); err != nil {
@@ -3096,7 +3132,7 @@ func (s *xlStorage) CleanAbandonedData(ctx context.Context, volume string, path
 	}
 	baseDir := pathJoin(volumeDir, path+slashSeparator)
 	metaPath := pathutil.Join(baseDir, xlStorageFormatFile)
-	buf, _, err := s.readAllData(ctx, volume, volumeDir, metaPath, true)
+	buf, _, err := s.readAllData(ctx, volume, volumeDir, metaPath)
 	if err != nil {
 		return err
 	}
@@ -1,5 +1,4 @@
-//go:build windows || darwin || freebsd
-// +build windows darwin freebsd
+//go:build !unix || darwin || freebsd

 // Copyright (c) 2015-2021 MinIO, Inc.
 //
@@ -1,5 +1,4 @@
-//go:build !windows && !darwin && !freebsd
-// +build !windows,!darwin,!freebsd
+//go:build unix && !darwin && !freebsd

 // Copyright (c) 2015-2021 MinIO, Inc.
 //
@@ -22,12 +21,14 @@ package cmd

 import (
 	"os"
+	"syscall"
 )

 var (
 	// Disallow updating access times
-	readMode = os.O_RDONLY | 0x40000 // O_NOATIME
+	// Add non-block to avoid syscall to attempt to set epoll on files.
+	readMode = os.O_RDONLY | 0x40000 | syscall.O_NONBLOCK // O_NOATIME

 	// Write with data sync only used only for `xl.meta` writes
-	writeMode = 0x1000 // O_DSYNC
+	writeMode = 0x1000 | syscall.O_NONBLOCK // O_DSYNC
 )
@@ -2530,7 +2530,7 @@
              }
            ]
          },
-          "unit": "bytes"
+          "unit": "none"
        },
        "overrides": [
          {
@@ -157,9 +157,9 @@ For deployments with [bucket](https://min.io/docs/minio/linux/administration/buc

 | Name                                           | Description                                                                                                                                 |
 |:-----------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------|
-| `minio_notify_current_send_in_progress`        | Number of concurrent async Send calls active to all targets (deprecated, please use 'minio_notify_target_current_send_in_progress' instead) |
-| `minio_notify_events_errors_total`             | Events that were failed to be sent to the targets (deprecated, please use 'minio_notify_target_failed_events' instead)                      |
-| `minio_notify_events_sent_total`               | Total number of events sent to the targets (deprecated, please use 'minio_notify_target_total_events' instead)                              |
+| `minio_notify_current_send_in_progress`        | Number of concurrent async Send calls active to all targets (deprecated, please use `minio_notify_target_current_send_in_progress` instead) |
+| `minio_notify_events_errors_total`             | Events that were failed to be sent to the targets (deprecated, please use `minio_notify_target_failed_events` instead)                      |
+| `minio_notify_events_sent_total`               | Total number of events sent to the targets (deprecated, please use `minio_notify_target_total_events` instead)                              |
 | `minio_notify_events_skipped_total`            | Events that were skipped to be sent to the targets due to the in-memory queue being full                                                    |
 | `minio_notify_target_current_send_in_progress` | Number of concurrent async Send calls active to the target                                                                                  |
 | `minio_notify_target_queue_length`             | Number of events currently staged in the queue_dir configured for the target.                                                               |
@@ -254,7 +254,7 @@ For deployments with [bucket](https://min.io/docs/minio/linux/administration/buc
 | `minio_node_io_read_bytes`                 | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes.                  |
 | `minio_node_io_wchar_bytes`                | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar. |
 | `minio_node_io_write_bytes`                | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes.                |
-| `minio_node_process_cpu_total_seconds`     | Total user and system CPU time spent in seconds.                                                                |
+| `minio_node_process_cpu_total_seconds`     | Total user and system CPU time spent in seconds by the process.                                                |
 | `minio_node_process_resident_memory_bytes` | Resident memory size in bytes.                                                                                  |
 | `minio_node_process_virtual_memory_bytes`  | Virtual memory size in bytes.                                                                                   |
 | `minio_node_process_starttime_seconds`     | Start time for MinIO process per node, time in seconds since Unix epoc.                                         |
@@ -1,397 +1,417 @@
 # Metrics Version 3

-In metrics version 3, all metrics are available under the endpoint:
+In metrics version 3, all metrics are available under the following base endpoint:

 ```
 /minio/metrics/v3
 ```

-however, a specific path under this is required.
+To query metrics of a specific type, append the appropriate path to the base endpoint.
+Querying the base endpoint returns "404 Not Found."

 Metrics are organized into groups at paths **relative** to the top-level endpoint above.

+Metrics are also available using the [MinIO Admin Client](https://min.io/docs/minio/linux/reference/minio-mc-admin.html) and the `mc admin prometheus metrics` command. For more information, see [Metrics and Alerts](https://min.io/docs/minio/linux/operations/monitoring/metrics-and-alerts.html) in the MinIO Documentation.
+
 ## Metrics Request Handling

-Each endpoint below can be queried at different intervals as needed via a scrape configuration in Prometheus or a compatible metrics collection tool.
+Each endpoint can be queried as needed via a scrape configuration in Prometheus or a compatible metrics collection tool. You should schedule scrape operations so a prior scrape completes before the next one begins.

-For ease of configuration, each (non-empty) parent of the path serves all metric endpoints that are at descendant paths. For example, to query all system metrics one needs to only scrape `/minio/metrics/v3/system/`.
+For ease of configuration, each (non-empty) parent of the path serves all the metric endpoints at its child paths. For example, to query all system metrics scrape `/minio/metrics/v3/system/`.

-Some metrics are bucket specific. These will have a `/bucket` component in their path. As the number of buckets can be large, the metrics scrape operation needs to be provided with a specific list of buckets via the `bucket` query parameter. Only metrics for the given buckets will be returned (with the bucket label set). For example to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`. 
+Each metric endpoint may support multiple child endpoints. For example, the `/v3/system/` metric has multiple child groups as `/v3/system/<child>`. Querying the parent endpoint returns metrics for all child groups. Querying a child group returns only metrics for that child.

-Instead of a metrics scrape, it is also possible to list the metrics that would be returned by a path. This is done by adding a `?list` query parameter. The MinIO server will then list all possible metrics that could be returned. During an actual metrics scrape, only available metrics are returned - not all of them. With the `list` query parameter, the output format can be selected - just set the request `Content-Type` to `application/json` for JSON output, or `text/plain` for a simple markdown formatted table. The latter is the default.
+### Per-bucket Metrics

-## Request, System and Cluster Metrics
+Metrics with a `/bucket` component in the path return results for each specified bucket in the deployment. These endpoints **require** providing a list of buckets as a `bucket` query parameter.  The endpoint then returns only metrics for the given buckets, with the bucket name in a `bucket` label.

-At a high level metrics are grouped into three categories, listed in the following sub-sections. The path in each of the tables is relative to the top-level endpoint.
+For example, to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`. 

-### Request metrics 
+### List Available Metrics

-These are metrics about requests served by the (current) node.
+Instead of a metrics scrape, you can list the metrics that would be returned by a path by adding a `list` query parameter. The MinIO server then lists all available metrics that could be returned. Note that during an actual metrics scrape only metrics with available _values_ are returned. Metrics with null values are omitted from the scrape results.

-| Path            | Description                                      |
-|-----------------|--------------------------------------------------|
-| `/api/requests` | Metrics over all requests                        |
-| `/bucket/api`   | Metrics over all requests for a given bucket     |
-|                 |                                                  |
+To set the output format, set the request `Content-Type` to the desired format. Accepted values are `application/json` for JSON output or `text/plain` for a Markdown-formatted table. The default is Markdown.
+
+For example, the the following returns a list of all available bucket metrics:
+```
+/minio/metrics/v3/api/bucket?list
+```
+
+## Metric Categories
+
+At a high level, metrics are grouped into categories as described in the following sections. The path in each of the tables is relative to the top-level endpoint. Note that the standard GoCollector metrics are not shown.
+
+### Request metrics
+
+Metrics about requests served by the current node.
+
+| Path            | Description                                   |
+|-----------------|-----------------------------------------------|
+| `/api/requests` | Metrics over all requests.                    |
+| `/bucket/api`   | Metrics over all requests for a given bucket. |
+
+#### `/api/requests`
+
+| Name                                           | Description                                                                    | Labels                                       |
+|------------------------------------------------|--------------------------------------------------------------------------------|----------------------------------------------|
+| `minio_api_requests_rejected_auth_total`       | Total number of requests rejected for auth failure. <br><br>Type: counter      | `type`, `pool_index`, `server`               |
+| `minio_api_requests_rejected_header_total`     | Total number of requests rejected for invalid header. <br><br>Type: counter    | `type`, `pool_index`, `server`               |
+| `minio_api_requests_rejected_timestamp_total`  | Total number of requests rejected for invalid timestamp. <br><br>Type: counter | `type`, `pool_index`, `server`               |
+| `minio_api_requests_rejected_invalid_total`    | Total number of invalid requests. <br><br>Type: counter                        | `type`, `pool_index`, `server`               |
+| `minio_api_requests_waiting_total`             | Total number of requests in the waiting queue. <br><br>Type: gauge             | `type`, `pool_index`, `server`               |
+| `minio_api_requests_incoming_total`            | Total number of incoming requests. <br><br>Type: gauge                         | `type`, `pool_index`, `server`               |
+| `minio_api_requests_inflight_total`            | Total number of requests currently in flight. <br><br>Type: gauge              | `name`, `type`, `pool_index`, `server`       |
+| `minio_api_requests_total`                     | Total number of requests. <br><br>Type: counter                                | `name`, `type`, `pool_index`, `server`       |
+| `minio_api_requests_errors_total`              | Total number of requests with 4xx or 5xx errors. <br><br>Type: counter         | `name`, `type`, `pool_index`, `server`       |
+| `minio_api_requests_5xx_errors_total`          | Total number of requests with 5xx errors. <br><br>Type: counter                | `name`, `type`, `pool_index`, `server`       |
+| `minio_api_requests_4xx_errors_total`          | Total number of requests with 4xx errors. <br><br>Type: counter                | `name`, `type`, `pool_index`, `server`       |
+| `minio_api_requests_canceled_total`            | Total number of requests canceled by the client. <br><br>Type: counter         | `name`, `type`, `pool_index`, `server`       |
+| `minio_api_requests_ttfb_seconds_distribution` | Distribution of time to first byte across API calls. <br><br>Type: counter     | `name`, `type`, `le`, `pool_index`, `server` |
+| `minio_api_requests_traffic_sent_bytes`        | Total number of bytes sent. <br><br>Type: counter                              | `type`, `pool_index`, `server`               |
+| `minio_api_requests_traffic_received_bytes`    | Total number of bytes received. <br><br>Type: counter                          | `type`, `pool_index`, `server`               |
+
+#### `/bucket/api`
+
+| Name                                         | Description                                                                             | Labels                                                 |
+|----------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------------|
+| `minio_bucket_api_traffic_received_bytes`    | Total number of bytes sent for a bucket. <br><br>Type: counter                          | `bucket`, `type`, `server`, `pool_index`               |
+| `minio_bucket_api_traffic_sent_bytes`        | Total number of bytes received for a bucket. <br><br>Type: counter                      | `bucket`, `type`, `server`, `pool_index`               |
+| `minio_bucket_api_inflight_total`            | Total number of requests currently in flight for a bucket. <br><br>Type: gauge          | `bucket`, `name`, `type`, `server`, `pool_index`       |
+| `minio_bucket_api_total`                     | Total number of requests for a bucket. <br><br>Type: counter                            | `bucket`, `name`, `type`, `server`, `pool_index`       |
+| `minio_bucket_api_canceled_total`            | Total number of requests canceled by the client for a bucket. <br><br>Type: counter     | `bucket`, `name`, `type`, `server`, `pool_index`       |
+| `minio_bucket_api_4xx_errors_total`          | Total number of requests with 4xx errors for a bucket. <br><br>Type: counter            | `bucket`, `name`, `type`, `server`, `pool_index`       |
+| `minio_bucket_api_5xx_errors_total`          | Total number of requests with 5xx errors for a bucket. <br><br>Type: counter            | `bucket`, `name`, `type`, `server`, `pool_index`       |
+| `minio_bucket_api_ttfb_seconds_distribution` | Distribution of time to first byte across API calls for a bucket. <br><br>Type: counter | `bucket`, `name`, `le`, `type`, `server`, `pool_index` |

 ### Audit metrics

-These are metrics about the minio audit functionality
+Metrics about the MinIO audit functionality.

-| Path     | Description                            |
-|----------|----------------------------------------|
-| `/audit` | Metrics related to audit functionality |
+| Path     | Description                             |
+|----------|-----------------------------------------|
+| `/audit` | Metrics related to audit functionality. |

-### ILM metrics
+#### `/audit`

-These are metrics about the minio ILM functionality
-
-| Path   | Description                          |
-|--------|--------------------------------------|
-| `/ilm` | Metrics related to ILM functionality |
-
-### Logger webhook metrics
-
-These are metrics about the minio logger webhooks
-
-| Path              | Description                        |
-|-------------------|------------------------------------|
-| `/logger/webhook` | Metrics related to logger webhooks |
-
-### Notification metrics
-
-These are metrics about the minio notification functionality
-
-| Path     | Description                                          |
-|----------|------------------------------------------------------|
-| `/notification` | Metrics related to notification functionality |
-
-### Scanner metrics
-
-These are metrics about the minio scanner
-
-| Path       | Description                          |
-|------------|--------------------------------------|
-| `/scanner` | Metrics related to the MinIO scanner |
-
-### System metrics
-
-These are metrics about the minio process and the node.
-
-| Path                        | Description                                       |
-|-----------------------------|---------------------------------------------------|
-| `/system/drive`             | Metrics about drives on the system                |
-| `/system/memory`            | Metrics about memory on the system                |
-| `/system/network/internode` | Metrics about internode requests made by the node |
-| `/system/process`           | Standard process metrics                          |
-|                             |                                                   |
-
-### Debug metrics
-
-These are metrics for debugging
-
-| Path                        | Description                                       |
-|-----------------------------|---------------------------------------------------|
-| `/debug/go`                 | Standard Go lang metrics                          |
-|                             |                                                   |
+| Name                              | Description                                                                     | Labels                |
+|-----------------------------------|---------------------------------------------------------------------------------|-----------------------|
+| `minio_audit_failed_messages`     | Total number of messages that failed to send since start. <br><br>Type: counter | `target_id`, `server` |
+| `minio_audit_target_queue_length` | Number of unsent messages in queue for target. <br><br>Type: gauge              | `target_id`, `server` |
+| `minio_audit_total_messages`      | Total number of messages sent since start. <br><br>Type: counter                | `target_id`, `server` |

 ### Cluster metrics

-These present metrics about the whole MinIO cluster.
+Metrics about an entire MinIO cluster.

-| Path                     | Description                 |
-|--------------------------|-----------------------------|
-| `/cluster/health`        | Cluster health metrics      |
-| `/cluster/usage/objects` | Object statistics           |
-| `/cluster/usage/buckets` | Object statistics by bucket |
-| `/cluster/erasure-set`   | Erasure set metrics         |
-|                          |                             |
+| Path                     | Description                    |
+|--------------------------|--------------------------------|
+| `/cluster/config`        | Cluster configuration metrics. |
+| `/cluster/erasure-set`   | Erasure set metrics.           |
+| `/cluster/health`        | Cluster health metrics.        |
+| `/cluster/iam`           | Cluster iam metrics.           |
+| `/cluster/usage/buckets` | Object statistics by bucket.   |
+| `/cluster/usage/objects` | Object statistics.             |

-## Metrics Listing
+#### `/cluster/config`

-Each of the following sub-sections list metrics returned by each of the endpoints.
+| Name                                   | Description                                                  | Labels |
+|----------------------------------------|--------------------------------------------------------------|--------|
+| `minio_cluster_config_rrs_parity`      | Reduced redundancy storage class parity. <br><br>Type: gauge |        |
+| `minio_cluster_config_standard_parity` | Standard storage class parity. <br><br>Type: gauge           |        |

-The standard metrics group for GoCollector is not shown below.
+#### `/cluster/erasure-set`

-### `/api/requests`
+| Name                                             | Description                                                                                             | Labels              |
+|--------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------|
+| `minio_cluster_erasure_set_overall_write_quorum` | Overall write quorum across pools and sets. <br><br>Type: gauge                                         |                     |
+| `minio_cluster_erasure_set_overall_health`       | Overall health across pools and sets (1=healthy, 0=unhealthy). <br><br>Type: gauge                      |                     |
+| `minio_cluster_erasure_set_read_quorum`          | Read quorum for the erasure set in a pool. <br><br>Type: gauge                                          | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_write_quorum`         | Write quorum for the erasure set in a pool. <br><br>Type: gauge                                         | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_online_drives_count`  | Count of online drives in the erasure set in a pool. <br><br>Type: gauge                                | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_healing_drives_count` | Count of healing drives in the erasure set in a pool. <br><br>Type: gauge                               | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_health`               | Health of the erasure set in a pool (1=healthy, 0=unhealthy). <br><br>Type: gauge                       | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_read_tolerance`       | Number of drive failures that can be tolerated without disrupting read operations. <br><br>Type: gauge  | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_write_tolerance`      | Number of drive failures that can be tolerated without disrupting write operations. <br><br>Type: gauge | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_read_health`          | Health of the erasure set in a pool for read operations (1=healthy, 0=unhealthy). <br><br>Type: gauge   | `pool_id`, `set_id` |
+| `minio_cluster_erasure_set_write_health`         | Health of the erasure set in a pool for write operations (1=healthy, 0=unhealthy). <br><br>Type: gauge  | `pool_id`, `set_id` |

-| Name                                           | Type      | Help                                                    | Labels                           |
-|------------------------------------------------|-----------|---------------------------------------------------------|----------------------------------|
-| `minio_api_requests_rejected_auth_total`       | `counter` | Total number of requests rejected for auth failure      | `type,pool_index,server`         |
-| `minio_api_requests_rejected_header_total`     | `counter` | Total number of requests rejected for invalid header    | `type,pool_index,server`         |
-| `minio_api_requests_rejected_timestamp_total`  | `counter` | Total number of requests rejected for invalid timestamp | `type,pool_index,server`         |
-| `minio_api_requests_rejected_invalid_total`    | `counter` | Total number of invalid requests                        | `type,pool_index,server`         |
-| `minio_api_requests_waiting_total`             | `gauge`   | Total number of requests in the waiting queue           | `type,pool_index,server`         |
-| `minio_api_requests_incoming_total`            | `gauge`   | Total number of incoming requests                       | `type,pool_index,server`         |
-| `minio_api_requests_inflight_total`            | `gauge`   | Total number of requests currently in flight            | `name,type,pool_index,server`    |
-| `minio_api_requests_total`                     | `counter` | Total number of requests                                | `name,type,pool_index,server`    |
-| `minio_api_requests_errors_total`              | `counter` | Total number of requests with (4xx and 5xx) errors      | `name,type,pool_index,server`    |
-| `minio_api_requests_5xx_errors_total`          | `counter` | Total number of requests with 5xx errors                | `name,type,pool_index,server`    |
-| `minio_api_requests_4xx_errors_total`          | `counter` | Total number of requests with 4xx errors                | `name,type,pool_index,server`    |
-| `minio_api_requests_canceled_total`            | `counter` | Total number of requests canceled by the client         | `name,type,pool_index,server`    |
-| `minio_api_requests_ttfb_seconds_distribution` | `counter` | Distribution of time to first byte across API calls     | `name,type,le,pool_index,server` |
-| `minio_api_requests_traffic_sent_bytes`        | `counter` | Total number of bytes sent                              | `type,pool_index,server`         |
-| `minio_api_requests_traffic_received_bytes`    | `counter` | Total number of bytes received                          | `type,pool_index,server`         |
+#### `/cluster/health`

-### `/bucket/api`
+| Name                                               | Description                                                         | Labels |
+|----------------------------------------------------|---------------------------------------------------------------------|--------|
+| `minio_cluster_health_drives_offline_count`        | Count of offline drives in the cluster. <br><br>Type: gauge         |        |
+| `minio_cluster_health_drives_online_count`         | Count of online drives in the cluster. <br><br>Type: gauge          |        |
+| `minio_cluster_health_drives_count`                | Count of all drives in the cluster. <br><br>Type: gauge             |        |
+| `minio_cluster_health_nodes_offline_count`         | Count of offline nodes in the cluster. <br><br>Type: gauge          |        |
+| `minio_cluster_health_nodes_online_count`          | Count of online nodes in the cluster. <br><br>Type: gauge           |        |
+| `minio_cluster_health_capacity_raw_total_bytes`    | Total cluster raw storage capacity in bytes. <br><br>Type: gauge    |        |
+| `minio_cluster_health_capacity_raw_free_bytes`     | Total cluster raw storage free in bytes. <br><br>Type: gauge        |        |
+| `minio_cluster_health_capacity_usable_total_bytes` | Total cluster usable storage capacity in bytes. <br><br>Type: gauge |        |
+| `minio_cluster_health_capacity_usable_free_bytes`  | Total cluster usable storage free in bytes. <br><br>Type: gauge     |        |

-| Name                                         | Type      | Help                                                             | Labels                                  |
-|----------------------------------------------|-----------|------------------------------------------------------------------|-----------------------------------------|
-| `minio_bucket_api_traffic_received_bytes`    | `counter` | Total number of bytes sent for a bucket                          | `bucket,type,server,pool_index`         |
-| `minio_bucket_api_traffic_sent_bytes`        | `counter` | Total number of bytes received for a bucket                      | `bucket,type,server,pool_index`         |
-| `minio_bucket_api_inflight_total`            | `gauge`   | Total number of requests currently in flight for a bucket        | `bucket,name,type,server,pool_index`    |
-| `minio_bucket_api_total`                     | `counter` | Total number of requests for a bucket                            | `bucket,name,type,server,pool_index`    |
-| `minio_bucket_api_canceled_total`            | `counter` | Total number of requests canceled by the client for a bucket     | `bucket,name,type,server,pool_index`    |
-| `minio_bucket_api_4xx_errors_total`          | `counter` | Total number of requests with 4xx errors for a bucket            | `bucket,name,type,server,pool_index`    |
-| `minio_bucket_api_5xx_errors_total`          | `counter` | Total number of requests with 5xx errors for a bucket            | `bucket,name,type,server,pool_index`    |
-| `minio_bucket_api_ttfb_seconds_distribution` | `counter` | Distribution of time to first byte across API calls for a bucket | `bucket,name,le,type,server,pool_index` |
+#### `/cluster/iam`

-### `/bucket/replication`
+| Name                                                            | Description                                                                                                                                     | Labels |
+|-----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|--------|
+| `minio_cluster_iam_last_sync_duration_millis`                   | Last successful IAM data sync duration in milliseconds. <br><br>Type: counter                                                                   |        |
+| `minio_cluster_iam_plugin_authn_service_failed_requests_minute` | When plugin authentication is configured, returns failed requests count in the last full minute. <br><br>Type: counter                          |        |
+| `minio_cluster_iam_plugin_authn_service_last_fail_seconds`      | When plugin authentication is configured, returns time (in seconds) since the last failed request to the service. <br><br>Type: counter         |        |
+| `minio_cluster_iam_plugin_authn_service_last_succ_seconds`      | When plugin authentication is configured, returns time (in seconds) since the last successful request to the service. <br><br>Type: counter     |        |
+| `minio_cluster_iam_plugin_authn_service_succ_avg_rtt_ms_minute` | When plugin authentication is configured, returns average round-trip time of successful requests in the last full minute. <br><br>Type: counter |        |
+| `minio_cluster_iam_plugin_authn_service_succ_max_rtt_ms_minute` | When plugin authentication is configured, returns maximum round-trip time of successful requests in the last full minute. <br><br>Type: counter |        |
+| `minio_cluster_iam_plugin_authn_service_total_requests_minute`  | When plugin authentication is configured, returns total requests count in the last full minute. <br><br>Type: counter                           |        |
+| `minio_cluster_iam_since_last_sync_millis`                      | Time (in milliseconds) since last successful IAM data sync. <br><br>Type: counter                                                               |        |
+| `minio_cluster_iam_sync_failures`                               | Number of failed IAM data syncs since server start. <br><br>Type: counter                                                                       |        |
+| `minio_cluster_iam_sync_successes`                              | Number of successful IAM data syncs since server start. <br><br>Type: counter                                                                   |        |

-| Name                                                                | Type      | Help                                                                                        | Labels                                    |
-|---------------------------------------------------------------------|-----------|---------------------------------------------------------------------------------------------|-------------------------------------------|
-| `minio_bucket_replication_last_hour_failed_bytes`                   | `gauge`   | Total number of bytes failed at least once to replicate in the last hour on a bucket        | `bucket,server`                           |
-| `minio_bucket_replication_last_hour_failed_count`                   | `gauge`   | Total number of objects which failed replication in the last hour on a bucket               | `bucket,server`                           |
-| `minio_bucket_replication_last_minute_failed_bytes`                 | `gauge`   | Total number of bytes failed at least once to replicate in the last full minute on a bucket | `bucket,server`                           |
-| `minio_bucket_replication_last_minute_failed_count`                 | `gauge`   | Total number of objects which failed replication in the last full minute on a bucket        | `bucket,server`                           |
-| `minio_bucket_replication_latency_ms`                               | `gauge`   | Replication latency on a bucket in milliseconds                                             | `bucket,operation,range,targetArn,server` |
-| `minio_bucket_replication_proxied_delete_tagging_requests_total`    | `counter` | Number of DELETE tagging requests proxied to replication target                             | `bucket,server`                           |
-| `minio_bucket_replication_proxied_get_requests_failures`            | `counter` | Number of failures in GET requests proxied to replication target                            | `bucket,server`                           |
-| `minio_bucket_replication_proxied_get_requests_total`               | `counter` | Number of GET requests proxied to replication target                                        | `bucket,server`                           |
-| `minio_bucket_replication_proxied_get_tagging_requests_failures`    | `counter` | Number of failures in GET tagging requests proxied to replication target                    | `bucket,server`                           |
-| `minio_bucket_replication_proxied_get_tagging_requests_total`       | `counter` | Number of GET tagging requests proxied to replication target                                | `bucket,server`                           |
-| `minio_bucket_replication_proxied_head_requests_failures`           | `counter` | Number of failures in HEAD requests proxied to replication target                           | `bucket,server`                           |
-| `minio_bucket_replication_proxied_head_requests_total`              | `counter` | Number of HEAD requests proxied to replication target                                       | `bucket,server`                           |
-| `minio_bucket_replication_proxied_put_tagging_requests_failures`    | `counter` | Number of failures in PUT tagging requests proxied to replication target                    | `bucket,server`                           |
-| `minio_bucket_replication_proxied_put_tagging_requests_total`       | `counter` | Number of PUT tagging requests proxied to replication target                                | `bucket,server`                           |
-| `minio_bucket_replication_sent_bytes`                               | `counter` | Total number of bytes replicated to the target                                              | `bucket,server`                           |
-| `minio_bucket_replication_sent_count`                               | `counter` | Total number of objects replicated to the target                                            | `bucket,server`                           |
-| `minio_bucket_replication_total_failed_bytes`                       | `counter` | Total number of bytes failed at least once to replicate since server start                  | `bucket,server`                           |
-| `minio_bucket_replication_total_failed_count`                       | `counter` | Total number of objects which failed replication since server start                         | `bucket,server`                           |
-| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | `counter` | Number of failures in DELETE tagging requests proxied to replication target                 | `bucket,server`                           |
+#### `/cluster/usage/buckets` 

-### `/audit`
+| Name                                                            | Description                                                                          | Labels            |
+|-----------------------------------------------------------------|--------------------------------------------------------------------------------------|-------------------|
+| `minio_cluster_usage_buckets_since_last_update_seconds`         | Time since last update of usage metrics in seconds. <br><br>Type: gauge              |                   |
+| `minio_cluster_usage_buckets_total_bytes`                       | Total bucket size in bytes. <br><br>Type: gauge                                      | `bucket`          |
+| `minio_cluster_usage_buckets_objects_count`                     | Total object count in bucket. <br><br>Type: gauge                                    | `bucket`          |
+| `minio_cluster_usage_buckets_versions_count`                    | Total object versions count in bucket, including delete markers. <br><br>Type: gauge | `bucket`          |
+| `minio_cluster_usage_buckets_delete_markers_count`              | Total delete markers count in bucket. <br><br>Type: gauge                            | `bucket`          |
+| `minio_cluster_usage_buckets_quota_total_bytes`                 | Total bucket quota in bytes. <br><br>Type: gauge                                     | `bucket`          |
+| `minio_cluster_usage_buckets_object_size_distribution`          | Bucket object size distribution. <br><br>Type: gauge                                 | `range`, `bucket` |
+| `minio_cluster_usage_buckets_object_version_count_distribution` | Bucket object version count distribution. <br><br>Type: gauge                        | `range`, `bucket` |

-| Name                              | Type      | Help                                                     | Labels             |
-|-----------------------------------|-----------|----------------------------------------------------------|--------------------|
-| `minio_audit_failed_messages`     | `counter` | Total number of messages that failed to send since start | `target_id,server` |
-| `minio_audit_target_queue_length` | `gauge`   | Number of unsent messages in queue for target            | `target_id,server` |
-| `minio_audit_total_messages`      | `counter` | Total number of messages sent since start                | `target_id,server` |
+#### `/cluster/usage/objects`

-### `/system/drive`
+| Name                                                     | Description                                                                        | Labels  |
+|----------------------------------------------------------|------------------------------------------------------------------------------------|---------|
+| `minio_cluster_usage_objects_since_last_update_seconds`  | Time since last update of usage metrics in seconds. <br><br>Type: gauge            |         |
+| `minio_cluster_usage_objects_total_bytes`                | Total cluster usage in bytes. <br><br>Type: gauge                                  |         |
+| `minio_cluster_usage_objects_count`                      | Total cluster objects count. <br><br>Type: gauge                                   |         |
+| `minio_cluster_usage_objects_versions_count`             | Total cluster object versions count, including delete markers. <br><br>Type: gauge |         |
+| `minio_cluster_usage_objects_delete_markers_count`       | Total cluster delete markers count. <br><br>Type: gauge                            |         |
+| `minio_cluster_usage_objects_buckets_count`              | Total cluster buckets count. <br><br>Type: gauge                                   |         |
+| `minio_cluster_usage_objects_size_distribution`          | Cluster object size distribution. <br><br>Type: gauge                              | `range` |
+| `minio_cluster_usage_objects_version_count_distribution` | Cluster object version count distribution. <br><br>Type: gauge                     | `range` |

-| Name                                           | Type      | Help                                                               | Labels                                              |
-|------------------------------------------------|-----------|--------------------------------------------------------------------|-----------------------------------------------------|
-| `minio_system_drive_used_bytes`                | `gauge`   | Total storage used on a drive in bytes                             | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_free_bytes`                | `gauge`   | Total storage free on a drive in bytes                             | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_total_bytes`               | `gauge`   | Total storage available on a drive in bytes                        | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_used_inodes`               | `gauge`   | Total used inodes on a drive                                       | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_free_inodes`               | `gauge`   | Total free inodes on a drive                                       | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_total_inodes`              | `gauge`   | Total inodes available on a drive                                  | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_timeout_errors_total`      | `counter` | Total timeout errors on a drive                                    | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_io_errors_total`           | `counter` | Total I/O errors on a drive                                        | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_availability_errors_total` | `counter` | Total availability errors (I/O errors, timeouts) on a drive        | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_waiting_io`                | `gauge`   | Total waiting I/O operations on a drive                            | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_api_latency_micros`        | `gauge`   | Average last minute latency in µs for drive API storage operations | `drive,api,set_index,drive_index,pool_index,server` |
-| `minio_system_drive_offline_count`             | `gauge`   | Count of offline drives                                            | `pool_index,server`                                 |
-| `minio_system_drive_online_count`              | `gauge`   | Count of online drives                                             | `pool_index,server`                                 |
-| `minio_system_drive_count`                     | `gauge`   | Count of all drives                                                | `pool_index,server`                                 |
-| `minio_system_drive_health`                    | `gauge`   | Drive health (0 = offline, 1 = healthy, 2 = healing)               | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_reads_per_sec`             | `gauge`   | Reads per second on a drive                                        | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_reads_kb_per_sec`          | `gauge`   | Kilobytes read per second on a drive                               | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_reads_await`               | `gauge`   | Average time for read requests served on a drive                   | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_writes_per_sec`            | `gauge`   | Writes per second on a drive                                       | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_writes_kb_per_sec`         | `gauge`   | Kilobytes written per second on a drive                            | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_writes_await`              | `gauge`   | Average time for write requests served on a drive                  | `drive,set_index,drive_index,pool_index,server`     |
-| `minio_system_drive_perc_util`                 | `gauge`   | Percentage of time the disk was busy                               | `drive,set_index,drive_index,pool_index,server`     |
+### Debug metrics

-### `/system/memory`
+Standard Go runtime metrics.

-| Name                             | Type    | Help                               | Labels   |
-|----------------------------------|---------|------------------------------------|----------|
-| `minio_system_memory_used`       | `gauge` | Used memory on the node            | `server` |
-| `minio_system_memory_used_perc`  | `gauge` | Used memory percentage on the node | `server` |
-| `minio_system_memory_free`       | `gauge` | Free memory on the node            | `server` |
-| `minio_system_memory_total`      | `gauge` | Total memory on the node           | `server` |
-| `minio_system_memory_buffers`    | `gauge` | Buffers memory on the node         | `server` |
-| `minio_system_memory_cache`      | `gauge` | Cache memory on the node           | `server` |
-| `minio_system_memory_shared`     | `gauge` | Shared memory on the node          | `server` |
-| `minio_system_memory_available`  | `gauge` | Available memory on the node       | `server` |
+| Path        | Description         |
+|-------------|---------------------|
+| `/debug/go` | Go runtime metrics. |

-### `/system/cpu`
+### ILM metrics

-| Name                          | Type    | Help                               | Labels   |
-|-------------------------------|---------|------------------------------------|----------|
-| `minio_system_cpu_avg_idle`   | `gauge` | Average CPU idle time              | `server` |
-| `minio_system_cpu_avg_iowait` | `gauge` | Average CPU IOWait time            | `server` |
-| `minio_system_cpu_load`       | `gauge` | CPU load average 1min              | `server` |
-| `minio_system_cpu_load_perc`  | `gauge` | CPU load average 1min (percentage) | `server` |
-| `minio_system_cpu_nice`       | `gauge` | CPU nice time                      | `server` |
-| `minio_system_cpu_steal`      | `gauge` | CPU steal time                     | `server` |
-| `minio_system_cpu_system`     | `gauge` | CPU system time                    | `server` |
-| `minio_system_cpu_user`       | `gauge` | CPU user time                      | `server` |
+Metrics about the MinIO ILM functionality.

-### `/system/network/internode`
+| Path   | Description                           |
+|--------|---------------------------------------|
+| `/ilm` | Metrics related to ILM functionality. |

-| Name                                                 | Type      | Help                                                     | Labels              |
-|------------------------------------------------------|-----------|----------------------------------------------------------|---------------------|
-| `minio_system_network_internode_errors_total`        | `counter` | Total number of failed internode calls                   | `server,pool_index` |
-| `minio_system_network_internode_dial_errors_total`   | `counter` | Total number of internode TCP dial timeouts and errors   | `server,pool_index` |
-| `minio_system_network_internode_dial_avg_time_nanos` | `gauge`   | Average dial time of internodes TCP calls in nanoseconds | `server,pool_index` |
-| `minio_system_network_internode_sent_bytes_total`    | `counter` | Total number of bytes sent to other peer nodes           | `server,pool_index` |
-| `minio_system_network_internode_recv_bytes_total`    | `counter` | Total number of bytes received from other peer nodes     | `server,pool_index` |
+#### `/ilm`

-### `/system/process`
+| Name                                                  | Description                                                                                       | Labels   |
+|-------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------|
+| `minio_cluster_ilm_expiry_pending_tasks`              | Number of pending ILM expiry tasks in the queue. <br><br>Type: gauge                              | `server` |
+| `minio_cluster_ilm_transition_active_tasks`           | Number of active ILM transition tasks. <br><br>Type: gauge                                        | `server` |
+| `minio_cluster_ilm_transition_pending_tasks`          | Number of pending ILM transition tasks in the queue. <br><br>Type: gauge                          | `server` |
+| `minio_cluster_ilm_transition_missed_immediate_tasks` | Number of missed immediate ILM transition tasks. <br><br>Type: counter                            | `server` |
+| `minio_cluster_ilm_versions_scanned`                  | Total number of object versions checked for ILM actions since server start. <br><br>Type: counter | `server` |

-| Name                                               | Type      | Help                                                                                                           | Labels   |
-|----------------------------------------------------|-----------|----------------------------------------------------------------------------------------------------------------|----------|
-| `minio_system_process_locks_read_total`            | `gauge`   | Number of current READ locks on this peer                                                                      | `server` |
-| `minio_system_process_locks_write_total`           | `gauge`   | Number of current WRITE locks on this peer                                                                     | `server` |
-| `minio_system_process_cpu_total_seconds`           | `counter` | Total user and system CPU time spent in seconds                                                                | `server` |
-| `minio_system_process_go_routine_total`            | `gauge`   | Total number of go routines running                                                                            | `server` |
-| `minio_system_process_io_rchar_bytes`              | `counter` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar       | `server` |
-| `minio_system_process_io_read_bytes`               | `counter` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes                  | `server` |
-| `minio_system_process_io_wchar_bytes`              | `counter` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar | `server` |
-| `minio_system_process_io_write_bytes`              | `counter` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes                | `server` |
-| `minio_system_process_start_time_seconds`          | `gauge`   | Start time for MinIO process in seconds since Unix epoc                                                        | `server` |
-| `minio_system_process_uptime_seconds`              | `gauge`   | Uptime for MinIO process in seconds                                                                            | `server` |
-| `minio_system_process_file_descriptor_limit_total` | `gauge`   | Limit on total number of open file descriptors for the MinIO Server process                                    | `server` |
-| `minio_system_process_file_descriptor_open_total`  | `gauge`   | Total number of open file descriptors by the MinIO Server process                                              | `server` |
-| `minio_system_process_syscall_read_total`          | `counter` | Total read SysCalls to the kernel. /proc/[pid]/io syscr                                                        | `server` |
-| `minio_system_process_syscall_write_total`         | `counter` | Total write SysCalls to the kernel. /proc/[pid]/io syscw                                                       | `server` |
-| `minio_system_process_resident_memory_bytes`       | `gauge`   | Resident memory size in bytes                                                                                  | `server` |
-| `minio_system_process_virtual_memory_bytes`        | `gauge`   | Virtual memory size in bytes                                                                                   | `server` |
-| `minio_system_process_virtual_memory_max_bytes`    | `gauge`   | Maximum virtual memory size in bytes                                                                           | `server` |
+### Logger webhook metrics

-### `/cluster/health`
+Metrics about MinIO logger webhooks.

-| Name                                               | Type    | Help                                           | Labels |
-|----------------------------------------------------|---------|------------------------------------------------|--------|
-| `minio_cluster_health_drives_offline_count`        | `gauge` | Count of offline drives in the cluster         |        |
-| `minio_cluster_health_drives_online_count`         | `gauge` | Count of online drives in the cluster          |        |
-| `minio_cluster_health_drives_count`                | `gauge` | Count of all drives in the cluster             |        |
-| `minio_cluster_health_nodes_offline_count`         | `gauge` | Count of offline nodes in the cluster          |        |
-| `minio_cluster_health_nodes_online_count`          | `gauge` | Count of online nodes in the cluster           |        |
-| `minio_cluster_health_capacity_raw_total_bytes`    | `gauge` | Total cluster raw storage capacity in bytes    |        |
-| `minio_cluster_health_capacity_raw_free_bytes`     | `gauge` | Total cluster raw storage free in bytes        |        |
-| `minio_cluster_health_capacity_usable_total_bytes` | `gauge` | Total cluster usable storage capacity in bytes |        |
-| `minio_cluster_health_capacity_usable_free_bytes`  | `gauge` | Total cluster usable storage free in bytes     |        |
+| Path              | Description                         |
+|-------------------|-------------------------------------|
+| `/logger/webhook` | Metrics related to logger webhooks. |

-### `/cluster/config`
+#### `/logger/webhook`

-| Name                                   | Type    | Help                                           | Labels |
-|----------------------------------------|---------|------------------------------------------------|--------|
-| `minio_cluster_config_rrs_parity`      | `gauge` | Reduced redundancy storage class parity        |        |
-| `minio_cluster_config_standard_parity` | `gauge` | Standard storage class parity                  |        |
+| Name                                    | Description                                                         | Labels                       |
+|-----------------------------------------|---------------------------------------------------------------------|------------------------------|
+| `minio_logger_webhook_failed_messages`  | Number of messages that failed to send. <br><br>Type: counter       | `server`, `name`, `endpoint` |
+| `minio_logger_webhook_queue_length`     | Webhook queue length. <br><br>Type: gauge                           | `server`, `name`, `endpoint` |
+| `minio_logger_webhook_total_message`    | Total number of messages sent to this target. <br><br>Type: counter | `server`, `name`, `endpoint` |

-### `/cluster/usage/objects`
+### Notification metrics

-| Name                                                     | Type    | Help                                                           | Labels  |
-|----------------------------------------------------------|---------|----------------------------------------------------------------|---------|
-| `minio_cluster_usage_objects_since_last_update_seconds`  | `gauge` | Time since last update of usage metrics in seconds             |         |
-| `minio_cluster_usage_objects_total_bytes`                | `gauge` | Total cluster usage in bytes                                   |         |
-| `minio_cluster_usage_objects_count`                      | `gauge` | Total cluster objects count                                    |         |
-| `minio_cluster_usage_objects_versions_count`             | `gauge` | Total cluster object versions (including delete markers) count |         |
-| `minio_cluster_usage_objects_delete_markers_count`       | `gauge` | Total cluster delete markers count                             |         |
-| `minio_cluster_usage_objects_buckets_count`              | `gauge` | Total cluster buckets count                                    |         |
-| `minio_cluster_usage_objects_size_distribution`          | `gauge` | Cluster object size distribution                               | `range` |
-| `minio_cluster_usage_objects_version_count_distribution` | `gauge` | Cluster object version count distribution                      | `range` |
+Metrics about the MinIO notification functionality.

-### `/cluster/usage/buckets` 
+| Path            | Description                                    |
+|-----------------|------------------------------------------------|
+| `/notification` | Metrics related to notification functionality. |

-| Name                                                            | Type    | Help                                                             | Labels         |
-|-----------------------------------------------------------------|---------|------------------------------------------------------------------|----------------|
-| `minio_cluster_usage_buckets_since_last_update_seconds`         | `gauge` | Time since last update of usage metrics in seconds               |                |
-| `minio_cluster_usage_buckets_total_bytes`                       | `gauge` | Total bucket size in bytes                                       | `bucket`       |
-| `minio_cluster_usage_buckets_objects_count`                     | `gauge` | Total objects count in bucket                                    | `bucket`       |
-| `minio_cluster_usage_buckets_versions_count`                    | `gauge` | Total object versions (including delete markers) count in bucket | `bucket`       |
-| `minio_cluster_usage_buckets_delete_markers_count`              | `gauge` | Total delete markers count in bucket                             | `bucket`       |
-| `minio_cluster_usage_buckets_quota_total_bytes`                 | `gauge` | Total bucket quota in bytes                                      | `bucket`       |
-| `minio_cluster_usage_buckets_object_size_distribution`          | `gauge` | Bucket object size distribution                                  | `range,bucket` |
-| `minio_cluster_usage_buckets_object_version_count_distribution` | `gauge` | Bucket object version count distribution                         | `range,bucket` |
+#### `/notification`

-### `/cluster/erasure-set`
+| Name                                          | Description                                                                                           | Labels   |
+|-----------------------------------------------|-------------------------------------------------------------------------------------------------------|----------|
+| `minio_notification_current_send_in_progress` | Number of concurrent async Send calls active to all targets. <br><br>Type: counter                    | `server` |
+| `minio_notification_events_errors_total`      | Total number of events that failed to send to the targets. <br><br>Type: counter                      | `server` |
+| `minio_notification_events_sent_total`        | Total number of events sent to the targets. <br><br>Type: counter                                     | `server` |
+| `minio_notification_events_skipped_total`     | Number of events not sent to the targets due to the in-memory queue being full. <br><br>Type: counter | `server` |

-| Name                                             | Type    | Help                                                                              | Labels           |
-|--------------------------------------------------|---------|-----------------------------------------------------------------------------------|------------------|
-| `minio_cluster_erasure_set_overall_write_quorum` | `gauge` | Overall write quorum across pools and sets                                        |                  |
-| `minio_cluster_erasure_set_overall_health`       | `gauge` | Overall health across pools and sets (1=healthy, 0=unhealthy)                     |                  |
-| `minio_cluster_erasure_set_read_quorum`          | `gauge` | Read quorum for the erasure set in a pool                                         | `pool_id,set_id` |
-| `minio_cluster_erasure_set_write_quorum`         | `gauge` | Write quorum for the erasure set in a pool                                        | `pool_id,set_id` |
-| `minio_cluster_erasure_set_online_drives_count`  | `gauge` | Count of online drives in the erasure set in a pool                               | `pool_id,set_id` |
-| `minio_cluster_erasure_set_healing_drives_count` | `gauge` | Count of healing drives in the erasure set in a pool                              | `pool_id,set_id` |
-| `minio_cluster_erasure_set_health`               | `gauge` | Health of the erasure set in a pool (1=healthy, 0=unhealthy)                      | `pool_id,set_id` |
-| `minio_cluster_erasure_set_read_tolerance`       | `gauge` | No of drive failures that can be tolerated without disrupting read operations     | `pool_id,set_id` |
-| `minio_cluster_erasure_set_write_tolerance`      | `gauge` | No of drive failures that can be tolerated without disrupting write operations    | `pool_id,set_id` |
-| `minio_cluster_erasure_set_read_health`          | `gauge` | Health of the erasure set in a pool for read operations (1=healthy, 0=unhealthy)  | `pool_id,set_id` |
-| `minio_cluster_erasure_set_write_health`         | `gauge` | Health of the erasure set in a pool for write operations (1=healthy, 0=unhealthy) | `pool_id,set_id` |
+### Replication metrics

-### `/cluster/iam`
+Metrics about MinIO site and bucket replication.

-| Name                                                            | Type      | Help                                                                                                                     | Labels |
-|-----------------------------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------------|--------|
-| `minio_cluster_iam_last_sync_duration_millis`                   | `counter` | Last successful IAM data sync duration in milliseconds                                                                   |        |
-| `minio_cluster_iam_plugin_authn_service_failed_requests_minute` | `counter` | When plugin authentication is configured, returns failed requests count in the last full minute                          |        |
-| `minio_cluster_iam_plugin_authn_service_last_fail_seconds`      | `counter` | When plugin authentication is configured, returns time (in seconds) since the last failed request to the service         |        |
-| `minio_cluster_iam_plugin_authn_service_last_succ_seconds`      | `counter` | When plugin authentication is configured, returns time (in seconds) since the last successful request to the service     |        |
-| `minio_cluster_iam_plugin_authn_service_succ_avg_rtt_ms_minute` | `counter` | When plugin authentication is configured, returns average round-trip-time of successful requests in the last full minute |        |
-| `minio_cluster_iam_plugin_authn_service_succ_max_rtt_ms_minute` | `counter` | When plugin authentication is configured, returns maximum round-trip-time of successful requests in the last full minute |        |
-| `minio_cluster_iam_plugin_authn_service_total_requests_minute`  | `counter` | When plugin authentication is configured, returns total requests count in the last full minute                           |        |
-| `minio_cluster_iam_since_last_sync_millis`                      | `counter` | Time (in milliseconds) since last successful IAM data sync                                                               |        |
-| `minio_cluster_iam_sync_failures`                               | `counter` | Number of failed IAM data syncs since server start                                                                       |        |
-| `minio_cluster_iam_sync_successes`                              | `counter` | Number of successful IAM data syncs since server start                                                                   |        |
+| Path                  | Description                            |
+|-----------------------|----------------------------------------|
+| `/bucket/replication` | Metrics related to bucket replication. |
+| `/replication`        | Metrics related to site replication.   |

-### `/logger/webhook`
+#### `/replication`

-| Name                                    | Type      | Help                                         | Labels                 |
-|-----------------------------------------|-----------|----------------------------------------------|------------------------|
-| `minio_logger_webhook_failed_messages`  | `counter` | Number of messages that failed to send       | `server,name,endpoint` |
-| `minio_logger_webhook_queue_length`     | `gauge`   | Webhook queue length                         | `server,name,endpoint` |
-| `minio_logger_webhook_total_message`    | `counter` | Total number of messages sent to this target | `server,name,endpoint` |
+| Name                                              | Description                                                                                 | Labels   |
+|---------------------------------------------------|---------------------------------------------------------------------------------------------|----------|
+| `minio_replication_average_active_workers`        | Average number of active replication workers. <br><br>Type: gauge                           | `server` |
+| `minio_replication_average_queued_bytes`          | Average number of bytes queued for replication since server start. <br><br>Type: gauge      | `server` |
+| `minio_replication_average_queued_count`          | Average number of objects queued for replication since server start. <br><br>Type: gauge    | `server` |
+| `minio_replication_average_data_transfer_rate`    | Average replication data transfer rate in bytes/sec. <br><br>Type: gauge                    | `server` |
+| `minio_replication_current_active_workers`        | Total number of active replication workers. <br><br>Type: gauge                             | `server` |
+| `minio_replication_current_data_transfer_rate`    | Current replication data transfer rate in bytes/sec. <br><br>Type: gauge                    | `server` |
+| `minio_replication_last_minute_queued_bytes`      | Number of bytes queued for replication in the last full minute. <br><br>Type: gauge         | `server` |
+| `minio_replication_last_minute_queued_count`      | Number of objects queued for replication in the last full minute. <br><br>Type: gauge       | `server` |
+| `minio_replication_max_active_workers`            | Maximum number of active replication workers seen since server start. <br><br>Type: gauge   | `server` |
+| `minio_replication_max_queued_bytes`              | Maximum number of bytes queued for replication since server start. <br><br>Type: gauge      | `server` |
+| `minio_replication_max_queued_count`              | Maximum number of objects queued for replication since server start. <br><br>Type: gauge    | `server` |
+| `minio_replication_max_data_transfer_rate`        | Maximum replication data transfer rate in bytes/sec since server start. <br><br>Type: gauge | `server` |

-### `/replication`
+#### `/bucket/replication`

-| Name                                              | Type    | Help                                                                        | Labels   |
-|---------------------------------------------------|---------|-----------------------------------------------------------------------------|----------|
-| `minio_replication_average_active_workers`        | `gauge` | Average number of active replication workers                                | `server` |
-| `minio_replication_average_queued_bytes`          | `gauge` | Average number of bytes queued for replication since server start           | `server` |
-| `minio_replication_average_queued_count`          | `gauge` | Average number of objects queued for replication since server start         | `server` |
-| `minio_replication_average_data_transfer_rate`    | `gauge` | Average replication data transfer rate in bytes/sec                         | `server` |
-| `minio_replication_current_active_workers`        | `gauge` | Total number of active replication workers                                  | `server` |
-| `minio_replication_current_data_transfer_rate`    | `gauge` | Current replication data transfer rate in bytes/sec                         | `server` |
-| `minio_replication_last_minute_queued_bytes`      | `gauge` | Number of bytes queued for replication in the last full minute              | `server` |
-| `minio_replication_last_minute_queued_count`      | `gauge` | Number of objects queued for replication in the last full minute            | `server` |
-| `minio_replication_max_active_workers`            | `gauge` | Maximum number of active replication workers seen since server start        | `server` |
-| `minio_replication_max_queued_bytes`              | `gauge` | Maximum number of bytes queued for replication since server start           | `server` |
-| `minio_replication_max_queued_count`              | `gauge` | Maximum number of objects queued for replication since server start         | `server` |
-| `minio_replication_max_data_transfer_rate`        | `gauge` | Maximum replication data transfer rate in bytes/sec seen since server start | `server` |
+| Name                                                                | Description                                                                                                     | Labels                                                |
+|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
+| `minio_bucket_replication_last_hour_failed_bytes`                   | Total number of bytes on a bucket which failed to replicate at least once in the last hour. <br><br>Type: gauge | `bucket`, `server`                                    |
+| `minio_bucket_replication_last_hour_failed_count`                   | Total number of objects on a bucket which failed to replicate in the last hour. <br><br>Type: gauge             | `bucket`, `server`                                    |
+| `minio_bucket_replication_last_minute_failed_bytes`                 | Total number of bytes on a bucket which failed at least once in the last full minute. <br><br>Type: gauge       | `bucket`, `server`                                    |
+| `minio_bucket_replication_last_minute_failed_count`                 | Total number of objects on a bucket which failed to replicate in the last full minute. <br><br>Type: gauge      | `bucket`, `server`                                    |
+| `minio_bucket_replication_latency_ms`                               | Replication latency on a bucket in milliseconds. <br><br>Type: gauge                                            | `bucket`, `operation`, `range`, `targetArn`, `server` |
+| `minio_bucket_replication_proxied_delete_tagging_requests_total`    | Number of DELETE tagging requests proxied to replication target. <br><br>Type: counter                          | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_get_requests_failures`            | Number of failures in GET requests proxied to replication target. <br><br>Type: counter                         | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_get_requests_total`               | Number of GET requests proxied to replication target. <br><br>Type: counter                                     | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_get_tagging_requests_failures`    | Number of failures in GET tagging requests proxied to replication target. <br><br>Type: counter                 | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_get_tagging_requests_total`       | Number of GET tagging requests proxied to replication target. <br><br>Type: counter                             | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_head_requests_failures`           | Number of failures in HEAD requests proxied to replication target. <br><br>Type: counter                        | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_head_requests_total`              | Number of HEAD requests proxied to replication target. <br><br>Type: counter                                    | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_put_tagging_requests_failures`    | Number of failures in PUT tagging requests proxied to replication target. <br><br>Type: counter                 | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_put_tagging_requests_total`       | Number of PUT tagging requests proxied to replication target. <br><br>Type: counter                             | `bucket`, `server`                                    |
+| `minio_bucket_replication_sent_bytes`                               | Total number of bytes replicated to the target. <br><br>Type: counter                                           | `bucket`, `server`                                    |
+| `minio_bucket_replication_sent_count`                               | Total number of objects replicated to the target. <br><br>Type: counter                                         | `bucket`, `server`                                    |
+| `minio_bucket_replication_total_failed_bytes`                       | Total number of bytes failed to replicate at least once since server start. <br><br>Type: counter               | `bucket`, `server`                                    |
+| `minio_bucket_replication_total_failed_count`                       | Total number of objects that failed to replicate since server start. <br><br>Type: counter                      | `bucket`, `server`                                    |
+| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | Number of failures in DELETE tagging requests proxied to replication target. <br><br>Type: counter              | `bucket`, `server`                                    |

-### `/notification`
+### Scanner metrics

-| Name                                          | Type      | Help                                                                                     | Labels   |
-|-----------------------------------------------|-----------|------------------------------------------------------------------------------------------|----------|
-| `minio_notification_current_send_in_progress` | `counter` | Number of concurrent async Send calls active to all targets                              | `server` |
-| `minio_notification_events_errors_total`      | `counter` | Events that were failed to be sent to the targets                                        | `server` |
-| `minio_notification_events_sent_total`        | `counter` | Total number of events sent to the targets                                               | `server` |
-| `minio_notification_events_skipped_total`     | `counter` | Events that were skipped to be sent to the targets due to the in-memory queue being full | `server` |
+Metrics about the MinIO scanner.

-### `/scanner`
+| Path       | Description                           |
+|------------|---------------------------------------|
+| `/scanner` | Metrics related to the MinIO scanner. |

-| Name                                       | Type      | Help                                                       | Labels   |
-|--------------------------------------------|-----------|------------------------------------------------------------|----------|
-| `minio_scanner_bucket_scans_finished`      | `counter` | Total number of bucket scans finished since server start   | `server` |
-| `minio_scanner_bucket_scans_started`       | `counter` | Total number of bucket scans started since server start    | `server` |
-| `minio_scanner_directories_scanned`        | `counter` | Total number of directories scanned since server start     | `server` |
-| `minio_scanner_last_activity_seconds`      | `gauge`   | Time elapsed (in seconds) since last scan activity         | `server` |
-| `minio_scanner_objects_scanned`            | `counter` | Total number of unique objects scanned since server start  | `server` |
-| `minio_scanner_versions_scanned`           | `counter` | Total number of object versions scanned since server start | `server` |
+#### `/scanner`

-### `/ilm`
+| Name                                       | Description                                                                       | Labels   |
+|--------------------------------------------|-----------------------------------------------------------------------------------|----------|
+| `minio_scanner_bucket_scans_finished`      | Total number of bucket scans completed since server start. <br><br>Type: counter  | `server` |
+| `minio_scanner_bucket_scans_started`       | Total number of bucket scans started since server start. <br><br>Type: counter    | `server` |
+| `minio_scanner_directories_scanned`        | Total number of directories scanned since server start. <br><br>Type: counter     | `server` |
+| `minio_scanner_last_activity_seconds`      | Time elapsed (in seconds) since last scan activity. <br><br>Type: gauge           | `server` |
+| `minio_scanner_objects_scanned`            | Total number of unique objects scanned since server start. <br><br>Type: counter  | `server` |
+| `minio_scanner_versions_scanned`           | Total number of object versions scanned since server start. <br><br>Type: counter | `server` |

-| Name                                                  | Type      | Help                                                                       | Labels   |
-|-------------------------------------------------------|-----------|----------------------------------------------------------------------------|----------|
-| `minio_cluster_ilm_expiry_pending_tasks`              | `gauge`   | Number of pending ILM expiry tasks in the queue                            | `server` |
-| `minio_cluster_ilm_transition_active_tasks`           | `gauge`   | Number of active ILM transition tasks                                      | `server` |
-| `minio_cluster_ilm_transition_pending_tasks`          | `gauge`   | Number of pending ILM transition tasks in the queue                        | `server` |
-| `minio_cluster_ilm_transition_missed_immediate_tasks` | `counter` | Number of missed immediate ILM transition tasks                            | `server` |
-| `minio_cluster_ilm_versions_scanned`                  | `counter` | Total number of object versions checked for ILM actions since server start | `server` |
+### System metrics
+
+Metrics about the MinIO process and the node.
+
+| Path                        | Description                                        |
+|-----------------------------|----------------------------------------------------|
+| `/system/cpu`               | Metrics about CPUs on the system.                  |
+| `/system/drive`             | Metrics about drives on the system.                |
+| `/system/network/internode` | Metrics about internode requests made by the node. |
+| `/system/memory`            | Metrics about memory on the system.                |
+| `/system/process`           | Standard process metrics.                          |
+
+#### `/system/drive`
+
+| Name                                           | Description                                                                             | Labels                                                             |
+|------------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------------------------|
+| `minio_system_drive_used_bytes`                | Total storage used on a drive in bytes. <br><br>Type: gauge                             | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_free_bytes`                | Total storage free on a drive in bytes. <br><br>Type: gauge                             | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_total_bytes`               | Total storage available on a drive in bytes. <br><br>Type: gauge                        | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_used_inodes`               | Total used inodes on a drive. <br><br>Type: gauge                                       | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_free_inodes`               | Total free inodes on a drive. <br><br>Type: gauge                                       | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_total_inodes`              | Total inodes available on a drive. <br><br>Type: gauge                                  | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_timeout_errors_total`      | Total timeout errors on a drive. <br><br>Type: counter                                  | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_io_errors_total`           | Total I/O errors on a drive. <br><br>Type: counter                                      | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_availability_errors_total` | Total availability errors (I/O errors, timeouts) on a drive. <br><br>Type: counter      | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_waiting_io`                | Total waiting I/O operations on a drive. <br><br>Type: gauge                            | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_api_latency_micros`        | Average last minute latency in µs for drive API storage operations. <br><br>Type: gauge | `drive`, `api`, `set_index`, `drive_index`, `pool_index`, `server` |
+| `minio_system_drive_offline_count`             | Count of offline drives. <br><br>Type: gauge                                            | `pool_index`, `server`                                             |
+| `minio_system_drive_online_count`              | Count of online drives. <br><br>Type: gauge                                             | `pool_index`, `server`                                             |
+| `minio_system_drive_count`                     | Count of all drives. <br><br>Type: gauge                                                | `pool_index`, `server`                                             |
+| `minio_system_drive_health`                    | Drive health (0 = offline, 1 = healthy, 2 = healing). <br><br>Type: gauge               | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_reads_per_sec`             | Reads per second on a drive. <br><br>Type: gauge                                        | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_reads_kb_per_sec`          | Kilobytes read per second on a drive. <br><br>Type: gauge                               | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_reads_await`               | Average time for read requests served on a drive. <br><br>Type: gauge                   | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_writes_per_sec`            | Writes per second on a drive. <br><br>Type: gauge                                       | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_writes_kb_per_sec`         | Kilobytes written per second on a drive. <br><br>Type: gauge                            | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_writes_await`              | Average time for write requests served on a drive. <br><br>Type: gauge                  | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+| `minio_system_drive_perc_util`                 | Percentage of time the disk was busy. <br><br>Type: gauge                               | `drive`, `set_index`, `drive_index`, `pool_index`, `server`        |
+
+#### `/system/memory`
+
+| Name                             | Description                                             | Labels   |
+|----------------------------------|---------------------------------------------------------|----------|
+| `minio_system_memory_used`       | Used memory on the node. <br><br>Type: gauge            | `server` |
+| `minio_system_memory_used_perc`  | Used memory percentage on the node. <br><br>Type: gauge | `server` |
+| `minio_system_memory_free`       | Free memory on the node. <br><br>Type: gauge            | `server` |
+| `minio_system_memory_total`      | Total memory on the node. <br><br>Type: gauge           | `server` |
+| `minio_system_memory_buffers`    | Buffers memory on the node. <br><br>Type: gauge         | `server` |
+| `minio_system_memory_cache`      | Cache memory on the node. <br><br>Type: gauge           | `server` |
+| `minio_system_memory_shared`     | Shared memory on the node. <br><br>Type: gauge          | `server` |
+| `minio_system_memory_available`  | Available memory on the node. <br><br>Type: gauge       | `server` |
+
+#### `/system/cpu`
+
+| Name                          | Description                                             | Labels   |
+|-------------------------------|---------------------------------------------------------|----------|
+| `minio_system_cpu_avg_idle`   | Average CPU idle time. <br><br>Type: gauge              | `server` |
+| `minio_system_cpu_avg_iowait` | Average CPU IOWait time. <br><br>Type: gauge            | `server` |
+| `minio_system_cpu_load`       | CPU load average 1min. <br><br>Type: gauge              | `server` |
+| `minio_system_cpu_load_perc`  | CPU load average 1min (percentage). <br><br>Type: gauge | `server` |
+| `minio_system_cpu_nice`       | CPU nice time. <br><br>Type: gauge                      | `server` |
+| `minio_system_cpu_steal`      | CPU steal time. <br><br>Type: gauge                     | `server` |
+| `minio_system_cpu_system`     | CPU system time. <br><br>Type: gauge                    | `server` |
+| `minio_system_cpu_user`       | CPU user time. <br><br>Type: gauge                      | `server` |
+
+#### `/system/network/internode`
+
+| Name                                                 | Description                                                                   | Labels                 |
+|------------------------------------------------------|-------------------------------------------------------------------------------|------------------------|
+| `minio_system_network_internode_errors_total`        | Total number of failed internode calls. <br><br>Type: counter                 | `server`, `pool_index` |
+| `minio_system_network_internode_dial_errors_total`   | Total number of internode TCP dial timeouts and errors. <br><br>Type: counter | `server`, `pool_index` |
+| `minio_system_network_internode_dial_avg_time_nanos` | Average dial time of internodes TCP calls in nanoseconds. <br><br>Type: gauge | `server`, `pool_index` |
+| `minio_system_network_internode_sent_bytes_total`    | Total number of bytes sent to other peer nodes. <br><br>Type: counter         | `server`, `pool_index` |
+| `minio_system_network_internode_recv_bytes_total`    | Total number of bytes received from other peer nodes. <br><br>Type: counter   | `server`, `pool_index` |
+
+#### `/system/process`
+
+| Name                                               | Description                                                                                                                           | Labels   |
+|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|----------|
+| `minio_system_process_locks_read_total`            | Number of current READ locks on this peer. <br><br>Type: gauge                                                                        | `server` |
+| `minio_system_process_locks_write_total`           | Number of current WRITE locks on this peer. <br><br>Type: gauge                                                                       | `server` |
+| `minio_system_process_cpu_total_seconds`           | Total user and system CPU time spent in seconds. <br><br>Type: counter                                                                | `server` |
+| `minio_system_process_go_routine_total`            | Total number of go routines running. <br><br>Type: gauge                                                                              | `server` |
+| `minio_system_process_io_rchar_bytes`              | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar. <br><br>Type: counter       | `server` |
+| `minio_system_process_io_read_bytes`               | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes. <br><br>Type: counter                  | `server` |
+| `minio_system_process_io_wchar_bytes`              | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar. <br><br>Type: counter | `server` |
+| `minio_system_process_io_write_bytes`              | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes. <br><br>Type: counter                | `server` |
+| `minio_system_process_start_time_seconds`          | Start time for MinIO process in seconds since Unix epoch. <br><br>Type: gauge                                                         | `server` |
+| `minio_system_process_uptime_seconds`              | Uptime for MinIO process in seconds. <br><br>Type: gauge                                                                              | `server` |
+| `minio_system_process_file_descriptor_limit_total` | Limit on total number of open file descriptors for the MinIO Server process. <br><br>Type: gauge                                      | `server` |
+| `minio_system_process_file_descriptor_open_total`  | Total number of open file descriptors by the MinIO Server process. <br><br>Type: gauge                                                | `server` |
+| `minio_system_process_syscall_read_total`          | Total read SysCalls to the kernel. /proc/[pid]/io syscr. <br><br>Type: counter                                                        | `server` |
+| `minio_system_process_syscall_write_total`         | Total write SysCalls to the kernel. /proc/[pid]/io syscw. <br><br>Type: counter                                                       | `server` |
+| `minio_system_process_resident_memory_bytes`       | Resident memory size in bytes. <br><br>Type: gauge                                                                                    | `server` |
+| `minio_system_process_virtual_memory_bytes`        | Virtual memory size in bytes. <br><br>Type: gauge                                                                                     | `server` |
+| `minio_system_process_virtual_memory_max_bytes`    | Maximum virtual memory size in bytes. <br><br>Type: gauge                                                                             | `server` |
@@ -2,7 +2,7 @@ version: '3.7'

 # Settings and configurations that are common for all containers
 x-minio-common: &minio-common
-  image: quay.io/minio/minio:RELEASE.2024-07-04T14-25-45Z
+  image: quay.io/minio/minio:RELEASE.2024-07-29T22-14-52Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
@@ -65,8 +65,8 @@ echo "done"

 # Start MinIO instances
 echo -n "Starting MinIO instances ..."
-CI=on MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
-CI=on MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
+CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
+CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
 echo "done"

 if [ ! -f ./mc ]; then
@@ -99,7 +99,7 @@ sleep 30
 echo "Create bucket in source MinIO instance"
 ./mc mb minio1/test-bucket --insecure

-# Load objects to source site
+# Load objects to source site with checksum header
 echo "Loading objects to source MinIO instance"
 OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
 aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj --body /tmp/data/obj --no-verify-ssl --profile enterprise
@@ -176,4 +176,87 @@ if [ "${SRC_OBJ_2_ETAG}" != "${DEST_OBJ_2_ETAG}" ]; then
 	exit_1
 fi

+echo "Set default encryption on "
+
+# test if checksum header is replicated for encrypted objects
+# Enable SSE KMS for test-bucket bucket
+./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure
+
+# Load objects to source site with checksum header
+echo "Loading objects to source MinIO instance"
+OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
+aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj2 --body /tmp/data/obj --no-verify-ssl --profile enterprise
+
+split -n 10 /tmp/data/mpartobj
+CREATE_MPART_OUT=$(aws s3api --endpoint-url=https://localhost:9001 create-multipart-upload --bucket test-bucket --key mpartobj2 --checksum-algorithm SHA256 --no-verify-ssl --profile enterprise)
+UPLOAD_ID=$(echo "${CREATE_MPART_OUT}" | jq '.UploadId' | sed 's/"//g')
+
+PARTS=""
+for idx in {1..10}; do
+	F_SUFFIX=$(num_to_alpha "$idx")
+	PART_CHKSUM=$(openssl dgst -sha256 -binary <"xa${F_SUFFIX}" | base64)
+	UPLOAD_PART_OUT=$(aws s3api --endpoint-url=https://localhost:9001 upload-part --checksum-algorithm SHA256 --checksum-sha256 "${PART_CHKSUM}" --bucket test-bucket --key mpartobj2 --part-number "${idx}" --body "xa${F_SUFFIX}" --upload-id "${UPLOAD_ID}" --no-verify-ssl --profile enterprise)
+	PART_ETAG=$(echo "${UPLOAD_PART_OUT}" | jq '.ETag')
+	if [ "${idx}" == 10 ]; then
+		PARTS="${PARTS}{\"ETag\": ${PART_ETAG}, \"ChecksumSHA256\": \"${PART_CHKSUM}\", \"PartNumber\": ${idx}}"
+	else
+		PARTS="${PARTS}{\"ETag\": ${PART_ETAG}, \"ChecksumSHA256\": \"${PART_CHKSUM}\", \"PartNumber\": ${idx}},"
+	fi
+done
+
+echo "{\"Parts\":[${PARTS}]}" >fileparts.json
+jq <fileparts.json
+aws s3api --endpoint-url=https://localhost:9001 complete-multipart-upload --multipart-upload file://fileparts.json --bucket test-bucket --key mpartobj2 --upload-id "${UPLOAD_ID}" --no-verify-ssl --profile enterprise
+sleep 120
+
+# List the objects from replicated site
+echo "Objects from replicated site"
+./mc ls minio2/test-bucket --insecure
+count1=$(./mc ls minio2/test-bucket/obj2 --insecure | wc -l)
+if [ "${count1}" -ne 1 ]; then
+	echo "BUG: object minio1/test-bucket/obj2 not replicated"
+	exit_1
+fi
+count2=$(./mc ls minio2/test-bucket/mpartobj2 --insecure | wc -l)
+if [ "${count2}" -ne 1 ]; then
+	echo "BUG: object minio1/test-bucket/mpartobj2 not replicated"
+	exit_1
+fi
+
+# Stat the objects from source site
+echo "Stat minio1/test-bucket/obj2"
+SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
+SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
+SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
+SRC_OBJ_2_CHKSUM=$(echo "${SRC_OUT_2}" | jq '.ChecksumSHA256')
+SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
+SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
+
+# Stat the objects from replicated site
+echo "Stat minio2/test-bucket/obj2"
+DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
+DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
+DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
+DEST_OBJ_2_CHKSUM=$(echo "${DEST_OUT_2}" | jq '.ChecksumSHA256')
+DEST_OBJ_1_ETAG=$(echo "${DEST_OUT_1}" | jq '.ETag')
+DEST_OBJ_2_ETAG=$(echo "${DEST_OUT_2}" | jq '.ETag')
+
+# Check the replication of checksums and etags
+if [ "${SRC_OBJ_1_CHKSUM}" != "${DEST_OBJ_1_CHKSUM}" ]; then
+	echo "BUG: Checksums dont match for 'obj2'. Source: ${SRC_OBJ_1_CHKSUM}, Destination: ${DEST_OBJ_1_CHKSUM}"
+	exit_1
+fi
+if [ "${SRC_OBJ_2_CHKSUM}" != "${DEST_OBJ_2_CHKSUM}" ]; then
+	echo "BUG: Checksums dont match for 'mpartobj2'. Source: ${SRC_OBJ_2_CHKSUM}, Destination: ${DEST_OBJ_2_CHKSUM}"
+	exit_1
+fi
+if [ "${SRC_OBJ_1_ETAG}" != "${DEST_OBJ_1_ETAG}" ]; then
+	echo "BUG: Etags dont match for 'obj2'. Source: ${SRC_OBJ_1_ETAG}, Destination: ${DEST_OBJ_1_ETAG}"
+	exit_1
+fi
+if [ "${SRC_OBJ_2_ETAG}" != "${DEST_OBJ_2_ETAG}" ]; then
+	echo "BUG: Etags dont match for 'mpartobj2'. Source: ${SRC_OBJ_2_ETAG}, Destination: ${DEST_OBJ_2_ETAG}"
+	exit_1
+fi
+
 cleanup
@@ -31,8 +31,6 @@ MINIO_IDENTITY_OPENID_CLAIM_USERINFO        (on|off)    Enable fetching claims f
 MINIO_IDENTITY_OPENID_KEYCLOAK_REALM        (string)    Specify Keycloak 'realm' name, only honored if vendor was set to 'keycloak' as value, if no realm is specified 'master' is default
 MINIO_IDENTITY_OPENID_KEYCLOAK_ADMIN_URL    (string)    Specify Keycloak 'admin' REST API endpoint e.g. http://localhost:8080/auth/admin/
 MINIO_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC  (on|off)    Enable 'Host' header based dynamic redirect URI (default: 'off')
-MINIO_IDENTITY_OPENID_CLAIM_PREFIX          (string)    [DEPRECATED use 'claim_name'] JWT claim namespace prefix e.g. "customer1/"
-MINIO_IDENTITY_OPENID_REDIRECT_URI          (string)    [DEPRECATED use env 'MINIO_BROWSER_REDIRECT_URL'] Configure custom redirect_uri for OpenID login flow callback
 MINIO_IDENTITY_OPENID_COMMENT               (sentence)  optionally add a comment to this setting
 ```

--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Harshavardhana	a9dc061d84	count metrics properly for any failures during drive heal (#20193 ) or via `mc admin heal --set 1 --pool 1`	2024-07-30 22:46:26 -07:00
Krishnan Parthasarathi	01a8c09920	Add fmt-gen subcommand (#20192 ) fmt-gen subcommand is only available when built with build tag `fmtgen`.	2024-07-30 15:59:48 -07:00
Aditya Manthramurthy	4c8562bcec	Fix v2 metrics: Send all ttfb api labels (#20191 ) Fix a regression in #19733 where TTFB metrics for all APIs except GetObject were removed in v2 and v3 metrics. This causes breakage for existing v2 metrics users. Instead we continue to send TTFB for all APIs in V2 but only send for GetObject in V3.	2024-07-30 15:28:46 -07:00
Harshavardhana	f13c04629b	allow multipart uploads expiration to be dynamic (#20190 ) allow multipart uploads expiration to be dyamic It would seem like the new values will take effect only after a restart for changes in multipart_expiration. This PR fixes this by making it dynamic as it should have been.	2024-07-30 12:01:06 -07:00
Harshavardhana	80ff907d08	add DeleteBulk support, add sufficient deadlines per rename() (#20185 ) deadlines per moveToTrash() allows for a more granular timeout approach for syscalls, instead of an aggregate timeout. This PR also enhances multipart state cleanup to be optimal by removing 100's of multipart network rename() calls into single network call.	2024-07-29 18:56:40 -07:00
Minio Trusted	673df6d517	Update yaml files to latest version RELEASE.2024-07-29T22-14-52Z	2024-07-30 00:00:47 +00:00
Poorna	2d40433bc1	remove replication throttle deadline for objects > 128MiB (#20184 ) context deadline was introduced to avoid a slow transfer from blocking replication queue(s) shared by other buckets that may not be under throttling. This PR removes this context deadline for larger objects since they are anyway restricted to a limited set of workers. Otherwise, objects would get dequeued when the throttle limit is exceeded and cannot proceed within the deadline.	2024-07-29 15:14:52 -07:00
Andrea Longo	3bc39db34e	Restructure metrics v3 readme for docs use (#20114 )	2024-07-29 11:48:51 -07:00
Harshavardhana	a17f14f73a	separate lock from common grid to avoid epoll contention (#20180 ) epoll contention on TCP causes latency build-up when we have high volume ingress. This PR is an attempt to relieve this pressure. upstream issue https://github.com/golang/go/issues/65064 It seems to be a deeper problem; haven't yet tried the fix provide in this issue, but however this change without changing the compiler helps. Of course, this is a workaround for now, hoping for a more comprehensive fix from Go runtime.	2024-07-29 11:10:04 -07:00
Poorna	6651c655cb	fix replication of checksum when encryption is enabled (#20161 ) - Adding functional tests - Return checksum header on GET/HEAD, previously this was returning InvalidPartNumber error	2024-07-29 01:02:16 -07:00
Harshavardhana	3ae104edae	change Read* calls over net/http to move to http.MethodGet (#20173 ) - ReadVersion - ReadFile - ReadXL Further changes include to - Compact internode resource RPC paths - Compact internode query params To optimize on parsing by gorilla/mux as the length of this string increases latency in gorilla/mux - reduce to a meaningful string.	2024-07-29 01:00:12 -07:00
jiuker	c87a489514	fix: support prefix when batchJob replicate enable the snowball (#20178 )	2024-07-29 00:59:50 -07:00
Minio Trusted	a60267501d	Update yaml files to latest version RELEASE.2024-07-26T20-48-21Z	2024-07-27 10:43:28 +00:00
Poorna	641a56da0d	fix panic in replication queuing (#20169 ) Regression from #20077 ``` Jul 26 19:08:29 minio-dr-0101a minio[275423]: Error: grid handler (NSScanner) panic: runtime error: index out of range [4] with length 1 (errors.errorString) Jul 26 19:08:29 minio-dr-0101a minio[275423]: 33: internal/logger/logger.go:268:logger.LogIf() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 32: internal/grid/connection.go:50:grid.gridLogIf() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 31: internal/grid/muxserver.go:234:grid.(muxServer).handleRequests.func1() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 30: cmd/bucket-replication.go:2165:cmd.(ReplicationPool).queueReplicaTask() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 29: cmd/bucket-replication.go:3440:cmd.queueReplicationHeal() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 28: cmd/data-scanner.go:1396:cmd.(scannerItem).healReplication() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 27: cmd/data-scanner.go:1220:cmd.(scannerItem).applyActions() Jul 26 19:08:29 minio-dr-0101a minio[275423]: 26: cmd/xl-storage.go:627:cmd.(xlStorage).NSScanner.func2() ```	2024-07-26 13:48:21 -07:00
Klaus Post	59788e25c7	Update connection deadlines less frequently (#20166 ) Only set write deadline on connections every second. Combine the 2 write locations into 1.	2024-07-26 10:40:11 -07:00
Harshavardhana	a16193bb50	remove fdatasync() discard, we write with O_SYNC (#20168 ) fdatasync() discard for page-cached READs is not needed, it would seem like this can cause latencies in situations when things are loaded.	2024-07-26 10:27:56 -07:00
jiuker	132e7413ba	fix: check once ready for site-replication (#20149 )	2024-07-26 10:27:42 -07:00
Klaus Post	1966668066	Avoid Batch Replication Job log spam (#20158 ) Only print once per job and error location. Set default retry to default 1 second wait, and use as minimum.	2024-07-26 05:55:50 -07:00
Harshavardhana	064f36ca5a	move to GET for internal stream READs instead of POST (#20160 ) the main reason is to let Go net/http perform necessary book keeping properly, and in essential from consistency point of view its GETs all the way. Deprecate sendFile() as its buggy inside Go runtime.	2024-07-26 05:55:01 -07:00
Klaus Post	15b609ecea	Expose RPC reconnections and ping time (#20157 ) - Keeps track of reconnection count. - Keeps track of connection ping roundtrip times. Sends timestamp in ping message. - Allow ping without payload.	2024-07-25 14:07:21 -07:00
Krishnan Parthasarathi	4a1edfd9aa	Different read quorum for tiered objects (#20115 ) For a non-tiered object, MinIO requires that EcM (# of data blocks) of xl.meta agree, corresponding to the number of data blocks needed to read this object. OTOH, tiered objects have metadata in the hot tier and data in the warm tier. The data and its integrity are offloaded to the warm tier. This allows us to reduce the read quorum from EcM (typically > N/2, where N - erasure stripe width) to N/2 + 1. The simple majority of metadata ensures consensus on what the object is and where it is located.	2024-07-25 14:02:50 -07:00
Anis Eleuch	b7f319b62a	properly reload a fresh drive when found in a failed state during startup (#20145 ) When a drive is in a failed state when a single node multiple drives deployment is started, a replacement of a fresh disk will not be properly healed unless the user restarts the node. Fix this by always adding the new fresh disk to globalLocalDrivesMap. Also remove globalLocalDrives for simplification, a map to store local node drives can still be used since the order of local drives of a node is not defined.	2024-07-24 16:30:33 -07:00
Anis Eleuch	33c101544d	kms: Expose API when bucket federation is enabled (#20143 ) kms: Expose API available when bucket federation is enabled When bucket federation feature is enabled, KMS API will not work, such as `mc admin kms key list` The commit will fix the issue by disabling bucket forwarding when this is a KMS request.	2024-07-24 15:44:29 -07:00
Anis Eleuch	21cf29330e	grafana: Fix the unit in Open FDs panel (#20144 )	2024-07-24 07:51:03 -07:00
Harshavardhana	3b21bb5be8	use unixNanoTime instead of time.Time in lockRequestorInfo (#20140 ) Bonus: Skip Source, Quorum fields in lockArgs that are never sent during Unlock() phase.	2024-07-24 03:24:01 -07:00
Harshavardhana	6fe2b3f901	avoid sendFile() for ranges or object lengths < 4MiB (#20141 )	2024-07-24 03:22:50 -07:00
Taran Pelkey	b368d4cc13	Fix `updateGroupMembershipsForLDAP` behavior with unicode (#20137 )	2024-07-23 19:10:03 -07:00
Klaus Post	0680af7414	Set O_NONBLOCK for reads and writes on unix (#20133 ) Tracing syscalls, opening and reading an `xl.meta` looks like this: ``` openat(AT_FDCWD, "/mnt/drive1/ss8-old/testbucket/ObjSize4MiBThreads72/(554O51H/peTb(0iztdbTKw59.csv/xl.meta", O_RDONLY\|O_NOATIME\|O_CLOEXEC) = 34 <0.000> fcntl(34, F_GETFL) = 0x48000 (flags O_RDONLY\|O_LARGEFILE\|O_NOATIME) <0.000> fcntl(34, F_SETFL, O_RDONLY\|O_NONBLOCK\|O_LARGEFILE\|O_NOATIME) = 0 <0.000> epoll_ctl(4, EPOLL_CTL_ADD, 34, {events=EPOLLIN\|EPOLLOUT\|EPOLLRDHUP\|EPOLLET, data={u32=3172471557, u64=8145488475984499461}}) = -1 EPERM (Operation not permitted) <0.000> fcntl(34, F_GETFL) = 0x48800 (flags O_RDONLY\|O_NONBLOCK\|O_LARGEFILE\|O_NOATIME) <0.000> fcntl(34, F_SETFL, O_RDONLY\|O_LARGEFILE\|O_NOATIME) = 0 <0.000> fstat(34, {st_mode=S_IFREG\|0644, st_size=354, ...}) = 0 <0.000> read(34, "XL2 \1\0\3\0\306\0\0\1P\2\2\1\304$\225\304\20\0\0\0\0\0\0\0\0\0\0\0"..., 354) = 354 <0.000> close(34) = 0 <0.000> ``` Everything until `fstat` is the `os.Open` call. Looking at the code: https://github.com/golang/go/blob/master/src/os/file_unix.go#L212-L243 It seems for every file it "tries" to see if it is pollable. This causes `syscall.SetNonblock(fd, true)` to be called. This is the first `F_SETFL`. It then calls `f.pfd.Init("file", true)`. This will attempt to set it as pollable using `epoll_ctl`. This will always fail for files. It therefore calls `syscall.SetNonblock(fd, false)` resulting in the second `F_SETFL`. If we set the `O_NONBLOCK` call on the initial open, we should avoid the 4 `fcntl` syscalls per file. I don't see any way to avoid the `epoll_ctl` call, since kind is either `kindOpenFile` or `kindNonBlock`, so "pollable" will always be true. However avoiding 4 of 6 syscalls still seems worth it. This should not have any effect, since files will end up with "nonblock" anyway.	2024-07-23 09:36:24 -07:00
Harshavardhana	91805bcab6	add optimizations to bring performance on unversioned READS (#20128 ) allow non-inlined on disk to be inlined via an unversioned ReadVersion() call, we only need ReadXL() to resolve objects with multiple versions only. The choice of this block makes it to be dynamic and chosen by the user via `mc admin config set` Other bonus things - Start measuring internode TTFB performance. - Set TCP_NODELAY, TCP_CORK for low latency	2024-07-23 03:53:03 -07:00
Klaus Post	c0e2886e37	Tweak grid for less writes (#20129 ) Use `runtime.Gosched()` if we have less than maxMergeMessages and the queue is empty. Up maxMergeMessages to 50 to merge more messages into a single write. Add length check for an early bailout on readAllInto when we know packet length.	2024-07-23 03:28:14 -07:00
Andreas Auernhammer	4f5dded4d4	fips: enforce FIPS-compliant TLS ciphers in FIPS mode (#20131 ) This commit enforces FIPS-compliant TLS ciphers in FIPS mode by importing the `fipsonly` module. Otherwise, MinIO still accepts non-FIPS compliant TLS connections.	2024-07-23 03:11:25 -07:00
jiuker	b3a94c4e85	fix: Use xtime duration to parse batch job (#20117 )	2024-07-23 00:05:53 -07:00
Harshavardhana	8e618d45fc	remove unnecessary LRU for internode auth token (#20119 ) removes contentious usage of mutexes in LRU, which were never really reused in any manner; we do not need it. To trust hosts, the correct way is TLS certs; this PR completely removes this dependency, which has never been useful. ``` 0 0% 100% 25.83s 26.76% github.com/hashicorp/golang-lru/v2/expirable.(LRU[...]) 0 0% 100% 28.03s 29.04% github.com/hashicorp/golang-lru/v2/expirable.(LRU[...]) ``` Bonus: use `x-minio-time` as a nanosecond to avoid unnecessary parsing logic of time strings instead of using a more straightforward mechanism.	2024-07-22 00:04:48 -07:00
Harshavardhana	3ef59d2821	do not set KMSSecretKey env from KMSSecretKeyFile (#20122 ) fixes #20121	2024-07-21 14:39:15 -07:00
Harshavardhana	23db4958f5	fix tuned-adm command typo	2024-07-18 18:15:02 -07:00
Anis Eleuch	d9ee668b6d	s3: Fix wrong continuation token during listing with ILM enabled bucket (#20113 )	2024-07-18 13:37:34 -07:00
Anis Eleuch	2e5d792f0c	batch-expiry: Save progress regularly in the drives and at the end (#20098 ) - Also, fix failure reporting at the end. - Also, avoid parsing report objects when listing or resuming jobs, this does not cause any bugs, it is only printing, not useful errors.	2024-07-17 09:42:32 -07:00
Minio Trusted	b276651eaa	Update yaml files to latest version RELEASE.2024-07-16T23-46-41Z	2024-07-17 15:26:12 +00:00
Poorna	3535197f99	replication: proxy only on missing object or read quorum err (#20101 )	2024-07-16 16:46:41 -07:00
Frank Wessels	95f076340a	Update reedsolomon dependency with fix for Graviton4 processor (#20102 )	2024-07-16 12:27:21 -07:00
Mark Theunissen	698bb93a46	Allow a KMS Action to specify keys in the Resources of a policy (#20079 )	2024-07-16 07:03:03 -07:00
Minio Trusted	2584430141	Update yaml files to latest version RELEASE.2024-07-15T19-02-30Z	2024-07-15 22:10:04 +00:00
Klaus Post	ded373e600	Split handleMessages (cosmetic) (#20095 ) Split the read and write sides of handleMessages into two separate functions Cosmetic. The only non-copy-and-paste change is that `cancel(ErrDisconnected)` is moved into the defer on `readStream`.	2024-07-15 12:02:30 -07:00
Harshavardhana	e8c54c3d6c	add validation test for v3 metrics for all its endpoints (#20094 ) add unit test for v3 metrics for all its exposed endpoints Bonus: - support OpenMetrics encoding - adds boot time for prometheus - continueOnError is better to serve as much metrics as possible.	2024-07-15 09:28:02 -07:00
Shubhendu	f944a42886	Removed user and group details from logs (#20072 ) Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>	2024-07-14 11:12:07 -07:00
Harshavardhana	eff0ea43aa	fix: typo in BucketUsageMetrics group registration in v3 metrics (#20090 ) ``` curl http://localhost:9000/minio/metrics/v3/cluster/usage/buckets ``` Did not work as documented, due to the fact that there was a typo in the bucket usage metrics registration group. This endpoint is a cluster endpoint and does not require any `buckets` argument.	2024-07-14 11:11:42 -07:00
Minio Trusted	3b602bb532	Update yaml files to latest version RELEASE.2024-07-13T01-46-15Z	2024-07-13 02:08:28 +00:00
Alex	459985f0fa	Update Console to v1.6.3 (#20084 ) Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>	2024-07-12 18:46:15 -07:00
Harshavardhana	d0080046c2	allow sysfs tuning from tuned	2024-07-12 16:31:18 -07:00
Harshavardhana	7fcb428622	do not print unexpected logs (#20083 )	2024-07-12 13:51:54 -07:00
dependabot[bot]	4ea6f94ed8	Bump github.com/nats-io/nats-streaming-server from 0.24.3 to 0.24.6 (#20082 ) Bumps [github.com/nats-io/nats-streaming-server](https://github.com/nats-io/nats-streaming-server) from 0.24.3 to 0.24.6. - [Release notes](https://github.com/nats-io/nats-streaming-server/releases) - [Changelog](https://github.com/nats-io/nats-streaming-server/blob/main/.goreleaser.yml) - [Commits](https://github.com/nats-io/nats-streaming-server/compare/v0.24.3...v0.24.6) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-streaming-server dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-07-12 11:45:09 -07:00
Klaus Post	83adc2eebf	Fix ListObjects aborting after 3 minute on async request (#20074 ) When creating the async listing, if the first request does not return within 3 minutes, it is stopped, since it isn't being kept alive. Keep updating `lastHandout` while we are waiting for the initial request to be fulfilled.	2024-07-12 09:23:16 -07:00
Poorna	989c318a28	replication: make large workers configurable (#20077 ) This PR also improves throttling by reducing tokens requested from rate limiter based on available tokens to avoid exceeding throttle wait deadlines	2024-07-12 07:57:31 -07:00
Frank Wessels	ef802f2b2c	Updated dependencies for ARM SVE support (#20081 )	2024-07-12 07:36:29 -07:00
Taran Pelkey	f5d2fbc84c	Add DecodeDN and QuickNormalizeDN functions to LDAP config (#20076 )	2024-07-11 18:04:53 -07:00
Allan Roger Reid	e139673969	Audit failure in batch job key rotate (#20073 )	2024-07-11 16:13:15 -07:00
Harshavardhana	a8c6465f22	hide some deprecated fields from 'get' output (#20069 ) also update wording on `subnet license="" api_key=""`	2024-07-10 13:16:44 -07:00
Minio Trusted	27538e2d22	Update yaml files to latest version RELEASE.2024-07-10T18-41-49Z	2024-07-10 19:27:17 +00:00