Remove B2 gateway implementation (#9547)

S3 is now natively supported by B2 cloud storage provider
there is no reason to use specialized gateway for B2 anymore,
our current S3 gateway with caching would work with B2.

Resolves #8584

CREDITS

@@ -4487,214 +4487,6 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 ================================================================
 
-github.com/google/gofuzz
-https://github.com/google/gofuzz
-----------------------------------------------------------------
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-
-================================================================
-
 github.com/google/martian
 https://github.com/google/martian
 ----------------------------------------------------------------
@@ -10936,25 +10728,6 @@ THE SOFTWARE.
 
 ================================================================
 
-github.com/kurin/blazer
-https://github.com/kurin/blazer
-----------------------------------------------------------------
-Copyright 2016, the Blazer authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-================================================================
-
 github.com/lib/pq
 https://github.com/lib/pq
 ----------------------------------------------------------------
@@ -16956,6 +16729,36 @@ https://github.com/soheilhy/cmux
 
 ================================================================
 
+github.com/spaolacci/murmur3
+https://github.com/spaolacci/murmur3
+----------------------------------------------------------------
+Copyright 2013, Sébastien Paolacci.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+    * Neither the name of the library nor the
+      names of its contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+================================================================
+
 github.com/streadway/amqp
 https://github.com/streadway/amqp
 ----------------------------------------------------------------
@@ -17226,6 +17029,69 @@ SOFTWARE.
 
 ================================================================
 
+github.com/willf/bitset
+https://github.com/willf/bitset
+----------------------------------------------------------------
+Copyright (c) 2014 Will Fitzgerald. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+================================================================
+
+github.com/willf/bloom
+https://github.com/willf/bloom
+----------------------------------------------------------------
+Copyright (c) 2014 Will Fitzgerald. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+================================================================
+
 github.com/xdg/scram
 https://github.com/xdg/scram
 ----------------------------------------------------------------
@@ -17899,6 +17765,31 @@ THE SOFTWARE.
 
 ================================================================
 
+go.uber.org/tools
+https://go.uber.org/tools
+----------------------------------------------------------------
+Copyright (c) 2017 Uber Technologies, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+================================================================
+
 go.uber.org/zap
 https://go.uber.org/zap
 ----------------------------------------------------------------
@@ -17957,6 +17848,72 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 ================================================================
 
+golang.org/x/lint
+https://golang.org/x/lint
+----------------------------------------------------------------
+Copyright (c) 2013 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+================================================================
+
+golang.org/x/mod
+https://golang.org/x/mod
+----------------------------------------------------------------
+Copyright (c) 2009 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+================================================================
+
 golang.org/x/net
 https://golang.org/x/net
 ----------------------------------------------------------------
@@ -18188,6 +18145,39 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 ================================================================
 
+golang.org/x/xerrors
+https://golang.org/x/xerrors
+----------------------------------------------------------------
+Copyright (c) 2019 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+================================================================
+
 google.golang.org/api
 https://google.golang.org/api
 ----------------------------------------------------------------
@@ -20622,3 +20612,29 @@ https://gopkg.in/yaml.v2
 
 ================================================================
 
+honnef.co/go/tools
+https://honnef.co/go/tools
+----------------------------------------------------------------
+Copyright (c) 2016 Dominik Honnef
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+================================================================
+

browser/app/js/browser/selectors.js

@@ -1,5 +1,5 @@
 /*
- * MinIO Cloud Storage, (C) 2020 MinIO, Inc.
+ * MinIO Cloud Storage (C) 2020 MinIO, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -14,28 +14,11 @@
  * limitations under the License.
  */
 
-package tagging
+import { createSelector } from "reselect"
 
-import (
-	"encoding/xml"
+export const getServerInfo = state => state.browser.serverInfo
+
+export const hasServerPublicDomain = createSelector(
+  getServerInfo,
+  serverInfo => Boolean(serverInfo.info && serverInfo.info.domains && serverInfo.info.domains.length),
 )
-
-// TagSet - Set of tags under Tagging
-type TagSet struct {
-	XMLName xml.Name `xml:"TagSet"`
-	Tags    []Tag    `xml:"Tag"`
-}
-
-// ContainsDuplicateTag - returns true if duplicate keys are present in TagSet
-func (t TagSet) ContainsDuplicateTag() bool {
-	x := make(map[string]struct{}, len(t.Tags))
-
-	for _, t := range t.Tags {
-		if _, has := x[t.Key]; has {
-			return true
-		}
-		x[t.Key] = struct{}{}
-	}
-
-	return false
-}

browser/app/js/objects/ShareObjectModal.js

@@ -77,7 +77,8 @@ export class ShareObjectModal extends React.Component {
     hideShareObject()
   }
   render() {
-    const { shareObjectDetails, shareObject, hideShareObject } = this.props
+    const { shareObjectDetails, hideShareObject } = this.props
+    const url = `${window.location.protocol}//${shareObjectDetails.url}`
     return (
       <Modal
         show={true}
@@ -93,11 +94,12 @@ export class ShareObjectModal extends React.Component {
               type="text"
               ref={node => (this.copyTextInput = node)}
               readOnly="readOnly"
-              value={window.location.protocol + "//" + shareObjectDetails.url}
+              value={url}
               onClick={() => this.copyTextInput.select()}
             />
           </div>
-          <div
+          {shareObjectDetails.showExpiryDate && (
+            <div
             className="input-group"
             style={{ display: web.LoggedIn() ? "block" : "none" }}
           >
@@ -174,10 +176,11 @@ export class ShareObjectModal extends React.Component {
               </div>
             </div>
           </div>
+          )}
         </ModalBody>
         <div className="modal-footer">
           <CopyToClipboard
-            text={window.location.protocol + "//" + shareObjectDetails.url}
+            text={url}
             onCopy={this.onUrlCopied.bind(this)}
           >
             <button className="btn btn-success">Copy Link</button>

browser/app/js/objects/__tests__/ShareObjectModal.test.js

@@ -34,7 +34,7 @@ describe("ShareObjectModal", () => {
     shallow(
       <ShareObjectModal
         object={{ name: "obj1" }}
-        shareObjectDetails={{ show: true, object: "obj1", url: "test" }}
+        shareObjectDetails={{ show: true, object: "obj1", url: "test", showExpiryDate: true }}
       />
     )
   })
@@ -44,7 +44,7 @@ describe("ShareObjectModal", () => {
     const wrapper = shallow(
       <ShareObjectModal
         object={{ name: "obj1" }}
-        shareObjectDetails={{ show: true, object: "obj1", url: "test" }}
+        shareObjectDetails={{ show: true, object: "obj1", url: "test", showExpiryDate: true }}
         hideShareObject={hideShareObject}
       />
     )
@@ -59,7 +59,7 @@ describe("ShareObjectModal", () => {
     const wrapper = shallow(
       <ShareObjectModal
         object={{ name: "obj1" }}
-        shareObjectDetails={{ show: true, object: "obj1", url: "test" }}
+        shareObjectDetails={{ show: true, object: "obj1", url: "test", showExpiryDate: true }}
       />
     )
     expect(
@@ -76,7 +76,7 @@ describe("ShareObjectModal", () => {
     const wrapper = shallow(
       <ShareObjectModal
         object={{ name: "obj1" }}
-        shareObjectDetails={{ show: true, object: "obj1", url: "test" }}
+        shareObjectDetails={{ show: true, object: "obj1", url: "test", showExpiryDate: true }}
         hideShareObject={hideShareObject}
         showCopyAlert={showCopyAlert}
       />
@@ -89,8 +89,15 @@ describe("ShareObjectModal", () => {
   describe("Update expiry values", () => {
     const props = {
       object: { name: "obj1" },
-      shareObjectDetails: { show: true, object: "obj1", url: "test" }
+      shareObjectDetails: { show: true, object: "obj1", url: "test", showExpiryDate: true }
     }
+
+    it("should not show expiry values if shared with public link", () => {
+      const shareObjectDetails = { show: true, object: "obj1", url: "test", showExpiryDate: false }
+      const wrapper = shallow(<ShareObjectModal {...props} shareObjectDetails={shareObjectDetails} />)
+      expect(wrapper.find('.set-expire').exists()).toEqual(false)
+    })
+
     it("should have default expiry values", () => {
       const wrapper = shallow(<ShareObjectModal {...props} />)
       expect(wrapper.state("expiry")).toEqual({

browser/app/js/objects/__tests__/actions.test.js

@@ -34,6 +34,7 @@ jest.mock("../../web", () => ({
     .mockReturnValueOnce(false)
     .mockReturnValueOnce(true)
     .mockReturnValueOnce(true)
+    .mockReturnValueOnce(true)
     .mockReturnValueOnce(false),
   ListObjects: jest.fn(({ bucketName }) => {
     if (bucketName === "test-deny") {
@@ -69,7 +70,14 @@ jest.mock("../../web", () => ({
     })
     .mockImplementationOnce(() => {
       return Promise.resolve({ token: "test" })
-    })
+    }),
+  GetBucketPolicy: jest.fn(({ bucketName, prefix }) => {
+    if (!bucketName) {
+      return Promise.reject({ message: "Invalid bucket" })
+    }
+    if (bucketName === 'test-public') return Promise.resolve({ policy: 'readonly' })
+    return Promise.resolve({})
+  })
 }))
 
 const middlewares = [thunk]
@@ -295,7 +303,8 @@ describe("Objects actions", () => {
         type: "objects/SET_SHARE_OBJECT",
         show: true,
         object: "b.txt",
-        url: "test"
+        url: "test",
+        showExpiryDate: true
       }
     ]
     store.dispatch(actionsObjects.showShareObject("b.txt", "test"))
@@ -321,14 +330,16 @@ describe("Objects actions", () => {
   it("creates objects/SET_SHARE_OBJECT when object is shared", () => {
     const store = mockStore({
       buckets: { currentBucket: "bk1" },
-      objects: { currentPrefix: "pre1/" }
+      objects: { currentPrefix: "pre1/" },
+      browser: { serverInfo: {} },
     })
     const expectedActions = [
       {
         type: "objects/SET_SHARE_OBJECT",
         show: true,
         object: "a.txt",
-        url: "https://test.com/bk1/pre1/b.txt"
+        url: "https://test.com/bk1/pre1/b.txt",
+        showExpiryDate: true
       },
       {
         type: "alert/SET",
@@ -347,10 +358,42 @@ describe("Objects actions", () => {
       })
   })
 
+  it("creates objects/SET_SHARE_OBJECT when object is shared with public link", () => {
+    const store = mockStore({
+      buckets: { currentBucket: "test-public" },
+      objects: { currentPrefix: "pre1/" },
+      browser: { serverInfo: { info: { domains: ['public.com'] }} },
+    })
+    const expectedActions = [
+      {
+        type: "objects/SET_SHARE_OBJECT",
+        show: true,
+        object: "a.txt",
+        url: "public.com/test-public/pre1/a.txt",
+        showExpiryDate: false
+      },
+      {
+        type: "alert/SET",
+        alert: {
+          type: "success",
+          message: "Object shared.",
+          id: alertActions.alertId
+        }
+      }
+    ]
+    return store
+      .dispatch(actionsObjects.shareObject("a.txt", 1, 0, 0))
+      .then(() => {
+        const actions = store.getActions()
+        expect(actions).toEqual(expectedActions)
+      })
+  })
+
   it("creates alert/SET when shareObject is failed", () => {
     const store = mockStore({
       buckets: { currentBucket: "" },
-      objects: { currentPrefix: "pre1/" }
+      objects: { currentPrefix: "pre1/" },
+      browser: { serverInfo: {} },
     })
     const expectedActions = [
       {

browser/app/js/objects/actions.js

@@ -24,7 +24,6 @@ import {
 import { getCurrentBucket } from "../buckets/selectors"
 import { getCurrentPrefix, getCheckedList } from "./selectors"
 import * as alertActions from "../alert/actions"
-import * as bucketActions from "../buckets/actions"
 import {
   minioBrowserPrefix,
   SORT_BY_NAME,
@@ -33,6 +32,7 @@ import {
   SORT_ORDER_ASC,
   SORT_ORDER_DESC,
 } from "../constants"
+import { getServerInfo, hasServerPublicDomain } from '../browser/selectors'
 
 export const SET_LIST = "objects/SET_LIST"
 export const RESET_LIST = "objects/RESET_LIST"
@@ -222,19 +222,38 @@ export const deleteCheckedObjects = () => {
 
 export const shareObject = (object, days, hours, minutes) => {
   return function (dispatch, getState) {
+    const hasServerDomain = hasServerPublicDomain(getState())
     const currentBucket = getCurrentBucket(getState())
     const currentPrefix = getCurrentPrefix(getState())
     const objectName = `${currentPrefix}${object}`
     const expiry = days * 24 * 60 * 60 + hours * 60 * 60 + minutes * 60
     if (web.LoggedIn()) {
       return web
-        .PresignedGet({
-          host: location.host,
-          bucket: currentBucket,
-          object: objectName,
-          expiry: expiry,
+        .GetBucketPolicy({ bucketName: currentBucket, prefix: currentPrefix })
+        .catch(() => ({ policy: null }))
+        .then(({ policy }) => {
+          if (hasServerDomain && ['readonly', 'readwrite'].includes(policy)) {
+            const domain = getServerInfo(getState()).info.domains[0]
+            const url = `${domain}/${currentBucket}/${encodeURI(objectName)}`
+            dispatch(showShareObject(object, url, false))
+            dispatch(
+              alertActions.set({
+                type: "success",
+                message: "Object shared."
+              })
+            )
+          } else {
+            return web
+              .PresignedGet({
+                host: location.host,
+                bucket: currentBucket,
+                object: objectName,
+                expiry: expiry
+              })
+          }
         })
         .then((obj) => {
+          if (!obj) return
           dispatch(showShareObject(object, obj.url))
           dispatch(
             alertActions.set({
@@ -272,11 +291,12 @@ export const shareObject = (object, days, hours, minutes) => {
   }
 }
 
-export const showShareObject = (object, url) => ({
+export const showShareObject = (object, url, showExpiryDate = true) => ({
   type: SET_SHARE_OBJECT,
   show: true,
   object,
   url,
+  showExpiryDate,
 })
 
 export const hideShareObject = (object, url) => ({

browser/app/js/objects/reducer.js

@@ -89,7 +89,8 @@ export default (
         shareObject: {
           show: action.show,
           object: action.object,
-          url: action.url
+          url: action.url,
+          showExpiryDate: action.showExpiryDate
         }
       }
     case actionsObjects.CHECKED_LIST_ADD:

buildscripts/verify-build.sh

@@ -154,34 +154,6 @@ function run_test_erasure_sets() {
     return "$rv"
 }
 
-function run_test_dist_erasure_sets_ipv6()
-{
-    minio_pids=( $(start_minio_dist_erasure_sets_ipv6) )
-
-    export SERVER_ENDPOINT="[::1]:9000"
-
-    (cd "$WORK_DIR" && "$FUNCTIONAL_TESTS")
-    rv=$?
-
-    for pid in "${minio_pids[@]}"; do
-        kill "$pid"
-    done
-    sleep 3
-
-    if [ "$rv" -ne 0 ]; then
-        for i in $(seq 0 9); do
-            echo "server$i log:"
-            cat "$WORK_DIR/dist-minio-v6-900$i.log"
-        done
-    fi
-
-    for i in $(seq 0 9); do
-        rm -f "$WORK_DIR/dist-minio-v6-900$i.log"
-    done
-
-    return "$rv"
-}
-
 function run_test_zone_erasure_sets()
 {
     minio_pids=( $(start_minio_zone_erasure_sets) )

cmd/admin-handlers.go

@@ -688,13 +688,15 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
 
 	// Define a closure to start sending whitespace to client
 	// after 10s unless a response item comes in
-	keepConnLive := func(w http.ResponseWriter, respCh chan healResp) {
+	keepConnLive := func(w http.ResponseWriter, r *http.Request, respCh chan healResp) {
 		ticker := time.NewTicker(time.Second * 10)
 		defer ticker.Stop()
 		started := false
 	forLoop:
 		for {
 			select {
+			case <-r.Context().Done():
+				return
 			case <-ticker.C:
 				if !started {
 					// Start writing response to client
@@ -790,7 +792,7 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
 			respCh <- hr
 		}()
 	case hip.clientToken == "":
-		nh := newHealSequence(hip.bucket, hip.objPrefix, handlers.GetSourceIP(r), numDisks, hip.hs, hip.forceStart)
+		nh := newHealSequence(GlobalContext, hip.bucket, hip.objPrefix, handlers.GetSourceIP(r), numDisks, hip.hs, hip.forceStart)
 		go func() {
 			respBytes, apiErr, errMsg := globalAllHealState.LaunchNewHealSequence(nh)
 			hr := healResp{respBytes, apiErr, errMsg}
@@ -801,7 +803,7 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
 	// Due to the force-starting functionality, the Launch
 	// call above can take a long time - to keep the
 	// connection alive, we start sending whitespace
-	keepConnLive(w, respCh)
+	keepConnLive(w, r, respCh)
 }
 
 func (a adminAPIHandlers) BackgroundHealStatusHandler(w http.ResponseWriter, r *http.Request) {

cmd/admin-heal-ops.go

@@ -60,9 +60,8 @@ const (
 )
 
 var (
-	errHealIdleTimeout      = fmt.Errorf("healing results were not consumed for too long")
-	errHealPushStopNDiscard = fmt.Errorf("heal push stopped due to heal stop signal")
-	errHealStopSignalled    = fmt.Errorf("heal stop signaled")
+	errHealIdleTimeout   = fmt.Errorf("healing results were not consumed for too long")
+	errHealStopSignalled = fmt.Errorf("heal stop signaled")
 
 	errFnHealFromAPIErr = func(ctx context.Context, err error) error {
 		apiErr := toAPIError(ctx, err)
@@ -73,10 +72,6 @@ var (
 
 // healSequenceStatus - accumulated status of the heal sequence
 type healSequenceStatus struct {
-	// lock to update this structure as it is concurrently
-	// accessed
-	updateLock *sync.RWMutex
-
 	// summary and detail for failures
 	Summary       healStatusSummary `json:"Summary"`
 	FailureDetail string            `json:"Detail,omitempty"`
@@ -114,11 +109,9 @@ func initHealState() *allHealState {
 func (ahs *allHealState) periodicHealSeqsClean(ctx context.Context) {
 	// Launch clean-up routine to remove this heal sequence (after
 	// it ends) from the global state after timeout has elapsed.
-	ticker := time.NewTicker(time.Minute * 5)
-	defer ticker.Stop()
 	for {
 		select {
-		case <-ticker.C:
+		case <-time.After(time.Minute * 5):
 			now := UTCNow()
 			ahs.Lock()
 			for path, h := range ahs.healSeqMap {
@@ -202,9 +195,7 @@ func (ahs *allHealState) LaunchNewHealSequence(h *healSequence) (
 	existsAndLive := false
 	he, exists := ahs.getHealSequence(h.path)
 	if exists {
-		if !he.hasEnded() || len(he.currentStatus.Items) > 0 {
-			existsAndLive = true
-		}
+		existsAndLive = !he.hasEnded()
 	}
 
 	if existsAndLive {
@@ -277,8 +268,8 @@ func (ahs *allHealState) PopHealStatusJSON(path string,
 	}
 
 	// Take lock to access and update the heal-sequence
-	h.currentStatus.updateLock.Lock()
-	defer h.currentStatus.updateLock.Unlock()
+	h.mutex.Lock()
+	defer h.mutex.Unlock()
 
 	numItems := len(h.currentStatus.Items)
 
@@ -289,20 +280,18 @@ func (ahs *allHealState) PopHealStatusJSON(path string,
 		lastResultIndex = h.currentStatus.Items[numItems-1].ResultIndex
 	}
 
-	// After sending status to client, and before relinquishing
-	// the updateLock, reset Item to nil and record the result
-	// index sent to the client.
-	defer func(i int64) {
-		h.lastSentResultIndex = i
-		h.currentStatus.Items = nil
-	}(lastResultIndex)
+	h.lastSentResultIndex = lastResultIndex
 
 	jbytes, err := json.Marshal(h.currentStatus)
 	if err != nil {
+		h.currentStatus.Items = nil
+
 		logger.LogIf(h.ctx, err)
 		return nil, ErrInternalError
 	}
 
+	h.currentStatus.Items = nil
+
 	return jbytes, ErrNone
 }
 
@@ -352,9 +341,8 @@ type healSequence struct {
 	// completed
 	traverseAndHealDoneCh chan error
 
-	// channel to signal heal sequence to stop (e.g. from the
-	// heal-stop API)
-	stopSignalCh chan struct{}
+	// canceler to cancel heal sequence.
+	cancelCtx context.CancelFunc
 
 	// the last result index sent to client
 	lastSentResultIndex int64
@@ -380,12 +368,12 @@ type healSequence struct {
 
 // NewHealSequence - creates healSettings, assumes bucket and
 // objPrefix are already validated.
-func newHealSequence(bucket, objPrefix, clientAddr string,
+func newHealSequence(ctx context.Context, bucket, objPrefix, clientAddr string,
 	numDisks int, hs madmin.HealOpts, forceStart bool) *healSequence {
 
 	reqInfo := &logger.ReqInfo{RemoteHost: clientAddr, API: "Heal", BucketName: bucket}
 	reqInfo.AppendTags("prefix", objPrefix)
-	ctx := logger.SetReqInfo(GlobalContext, reqInfo)
+	ctx, cancel := context.WithCancel(logger.SetReqInfo(ctx, reqInfo))
 
 	return &healSequence{
 		respCh:         make(chan healResult),
@@ -402,10 +390,9 @@ func newHealSequence(bucket, objPrefix, clientAddr string,
 			Summary:      healNotStartedStatus,
 			HealSettings: hs,
 			NumDisks:     numDisks,
-			updateLock:   &sync.RWMutex{},
 		},
 		traverseAndHealDoneCh: make(chan error),
-		stopSignalCh:          make(chan struct{}),
+		cancelCtx:             cancel,
 		ctx:                   ctx,
 		scannedItemsMap:       make(map[madmin.HealItemType]int64),
 		healedItemsMap:        make(map[madmin.HealItemType]int64),
@@ -488,7 +475,7 @@ func (h *healSequence) gethealFailedItemsMap() map[string]int64 {
 // external signal)
 func (h *healSequence) isQuitting() bool {
 	select {
-	case <-h.stopSignalCh:
+	case <-h.ctx.Done():
 		return true
 	default:
 		return false
@@ -497,19 +484,15 @@ func (h *healSequence) isQuitting() bool {
 
 // check if the heal sequence has ended
 func (h *healSequence) hasEnded() bool {
-	h.currentStatus.updateLock.RLock()
-	summary := h.currentStatus.Summary
-	h.currentStatus.updateLock.RUnlock()
-	return summary == healStoppedStatus || summary == healFinishedStatus
+	h.mutex.RLock()
+	ended := len(h.currentStatus.Items) == 0 || h.currentStatus.Summary == healStoppedStatus || h.currentStatus.Summary == healFinishedStatus
+	h.mutex.RUnlock()
+	return ended
 }
 
 // stops the heal sequence - safe to call multiple times.
 func (h *healSequence) stop() {
-	select {
-	case <-h.stopSignalCh:
-	default:
-		close(h.stopSignalCh)
-	}
+	h.cancelCtx()
 }
 
 // pushHealResultItem - pushes a heal result item for consumption in
@@ -536,29 +519,27 @@ func (h *healSequence) pushHealResultItem(r madmin.HealResultItem) error {
 
 	var itemsLen int
 	for {
-		h.currentStatus.updateLock.Lock()
+		h.mutex.Lock()
 		itemsLen = len(h.currentStatus.Items)
 		if itemsLen == maxUnconsumedHealResultItems {
-			// unlock and wait to check again if we can push
-			h.currentStatus.updateLock.Unlock()
-
 			// wait for a second, or quit if an external
 			// stop signal is received or the
 			// unconsumedTimer fires.
 			select {
 			// Check after a second
 			case <-time.After(time.Second):
+				h.mutex.Unlock()
 				continue
 
-			case <-h.stopSignalCh:
+			case <-h.ctx.Done():
+				h.mutex.Unlock()
 				// discard result and return.
-				return errHealPushStopNDiscard
+				return errHealStopSignalled
 
-			// Timeout if no results consumed for too
-			// long.
+			// Timeout if no results consumed for too long.
 			case <-unconsumedTimer.C:
+				h.mutex.Unlock()
 				return errHealIdleTimeout
-
 			}
 		}
 		break
@@ -575,13 +556,7 @@ func (h *healSequence) pushHealResultItem(r madmin.HealResultItem) error {
 	h.currentStatus.Items = append(h.currentStatus.Items, r)
 
 	// release lock
-	h.currentStatus.updateLock.Unlock()
-
-	// This is a "safe" point for the heal sequence to quit if
-	// signaled externally.
-	if h.isQuitting() {
-		return errHealStopSignalled
-	}
+	h.mutex.Unlock()
 
 	return nil
 }
@@ -595,10 +570,10 @@ func (h *healSequence) pushHealResultItem(r madmin.HealResultItem) error {
 // the heal-sequence.
 func (h *healSequence) healSequenceStart() {
 	// Set status as running
-	h.currentStatus.updateLock.Lock()
+	h.mutex.Lock()
 	h.currentStatus.Summary = healRunningStatus
 	h.currentStatus.StartTime = UTCNow()
-	h.currentStatus.updateLock.Unlock()
+	h.mutex.Unlock()
 
 	if h.sourceCh == nil {
 		go h.traverseAndHeal()
@@ -608,25 +583,27 @@ func (h *healSequence) healSequenceStart() {
 
 	select {
 	case err, ok := <-h.traverseAndHealDoneCh:
+		if !ok {
+			return
+		}
+		h.mutex.Lock()
 		h.endTime = UTCNow()
-		h.currentStatus.updateLock.Lock()
-		defer h.currentStatus.updateLock.Unlock()
 		// Heal traversal is complete.
-		if ok {
+		if err == nil {
+			// heal traversal succeeded.
+			h.currentStatus.Summary = healFinishedStatus
+		} else {
 			// heal traversal had an error.
 			h.currentStatus.Summary = healStoppedStatus
 			h.currentStatus.FailureDetail = err.Error()
-		} else {
-			// heal traversal succeeded.
-			h.currentStatus.Summary = healFinishedStatus
 		}
-
-	case <-h.stopSignalCh:
+		h.mutex.Unlock()
+	case <-h.ctx.Done():
+		h.mutex.Lock()
 		h.endTime = UTCNow()
-		h.currentStatus.updateLock.Lock()
 		h.currentStatus.Summary = healStoppedStatus
 		h.currentStatus.FailureDetail = errHealStopSignalled.Error()
-		h.currentStatus.updateLock.Unlock()
+		h.mutex.Unlock()
 
 		// drain traverse channel so the traversal
 		// go-routine does not leak.
@@ -654,13 +631,20 @@ func (h *healSequence) queueHealTask(source healSource, healType madmin.HealItem
 	select {
 	case res := <-h.respCh:
 		if !h.reportProgress {
+			// Object might have been deleted, by the time heal
+			// was attempted, we should ignore this object and
+			// return success.
+			if isErrObjectNotFound(res.err) {
+				return nil
+			}
+
 			h.mutex.Lock()
 			defer h.mutex.Unlock()
 
 			// Progress is not reported in case of background heal processing.
 			// Instead we increment relevant counter based on the heal result
 			// for prometheus reporting.
-			if res.err != nil && !isErrObjectNotFound(res.err) {
+			if res.err != nil {
 				for _, d := range res.result.After.Drives {
 					// For failed items we report the endpoint and drive state
 					// This will help users take corrective actions for drives
@@ -670,7 +654,9 @@ func (h *healSequence) queueHealTask(source healSource, healType madmin.HealItem
 				// Only object type reported for successful healing
 				h.healedItemsMap[res.result.Type]++
 			}
-			return nil
+
+			// Report caller of any failure
+			return res.err
 		}
 		res.result.Type = healType
 		if res.err != nil {
@@ -688,10 +674,7 @@ func (h *healSequence) queueHealTask(source healSource, healType madmin.HealItem
 		return h.pushHealResultItem(res.result)
 	case <-h.ctx.Done():
 		return nil
-	case <-h.traverseAndHealDoneCh:
-		return nil
 	}
-
 }
 
 func (h *healSequence) healItemsFromSourceCh() error {
@@ -702,7 +685,10 @@ func (h *healSequence) healItemsFromSourceCh() error {
 
 	for {
 		select {
-		case source := <-h.sourceCh:
+		case source, ok := <-h.sourceCh:
+			if !ok {
+				return nil
+			}
 			var itemType madmin.HealItemType
 			switch {
 			case source.path == nopHeal:
@@ -721,8 +707,6 @@ func (h *healSequence) healItemsFromSourceCh() error {
 
 			h.scannedItemsMap[itemType]++
 			h.lastHealActivity = UTCNow()
-		case <-h.traverseAndHealDoneCh:
-			return nil
 		case <-h.ctx.Done():
 			return nil
 		}
@@ -731,7 +715,6 @@ func (h *healSequence) healItemsFromSourceCh() error {
 
 func (h *healSequence) healFromSourceCh() {
 	h.healItemsFromSourceCh()
-	close(h.traverseAndHealDoneCh)
 }
 
 func (h *healSequence) healItems(bucketsOnly bool) error {
@@ -763,13 +746,7 @@ func (h *healSequence) healItems(bucketsOnly bool) error {
 // two objects.
 func (h *healSequence) traverseAndHeal() {
 	bucketsOnly := false // Heals buckets and objects also.
-	if err := h.healItems(bucketsOnly); err != nil {
-		if h.isQuitting() {
-			err = errHealStopSignalled
-		}
-		h.traverseAndHealDoneCh <- err
-	}
-
+	h.traverseAndHealDoneCh <- h.healItems(bucketsOnly)
 	close(h.traverseAndHealDoneCh)
 }
 

cmd/api-errors.go

@@ -28,13 +28,13 @@ import (
 	"google.golang.org/api/googleapi"
 
 	minio "github.com/minio/minio-go/v6"
+	"github.com/minio/minio-go/v6/pkg/tags"
 	"github.com/minio/minio/cmd/config/etcd/dns"
 	"github.com/minio/minio/cmd/crypto"
 	"github.com/minio/minio/cmd/logger"
 	"github.com/minio/minio/pkg/auth"
 	"github.com/minio/minio/pkg/bucket/lifecycle"
 	objectlock "github.com/minio/minio/pkg/bucket/object/lock"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 	"github.com/minio/minio/pkg/event"
 	"github.com/minio/minio/pkg/hash"
@@ -157,6 +157,7 @@ const (
 	ErrInvalidRetentionDate
 	ErrPastObjectLockRetainDate
 	ErrUnknownWORMModeDirective
+	ErrBucketTaggingNotFound
 	ErrObjectLockInvalidHeaders
 	ErrInvalidTagDirective
 	// Add new error codes here.
@@ -777,6 +778,11 @@ var errorCodes = errorCodeMap{
 		Description:    "Bucket is missing ObjectLockConfiguration",
 		HTTPStatusCode: http.StatusBadRequest,
 	},
+	ErrBucketTaggingNotFound: {
+		Code:           "NoSuchTagSet",
+		Description:    "The TagSet does not exist",
+		HTTPStatusCode: http.StatusNotFound,
+	},
 	ErrObjectLockConfigurationNotFound: {
 		Code:           "ObjectLockConfigurationNotFoundError",
 		Description:    "Object Lock configuration does not exist for this bucket",
@@ -1896,7 +1902,7 @@ func toAPIError(ctx context.Context, err error) APIError {
 				Description:    e.Error(),
 				HTTPStatusCode: http.StatusBadRequest,
 			}
-		case tagging.Error:
+		case tags.Error:
 			apiErr = APIError{
 				Code:           e.Code(),
 				Description:    e.Error(),

cmd/api-response.go

@@ -601,7 +601,8 @@ func generateCompleteMultpartUploadResponse(bucket, key, location, etag string)
 		Location: location,
 		Bucket:   bucket,
 		Key:      key,
-		ETag:     etag,
+		// AWS S3 quotes the ETag in XML, make sure we are compatible here.
+		ETag: "\"" + etag + "\"",
 	}
 }
 

cmd/api-router.go

@@ -194,7 +194,7 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled, allowSSEKMS bool)
 		// GetBucketReplicationHandler - this is a dummy call.
 		bucket.Methods(http.MethodGet).HandlerFunc(
 			maxClients(collectAPIStats("getbucketreplication", httpTraceAll(api.GetBucketReplicationHandler)))).Queries("replication", "")
-		// GetBucketTaggingHandler - this is a dummy call.
+		// GetBucketTaggingHandler
 		bucket.Methods(http.MethodGet).HandlerFunc(
 			maxClients(collectAPIStats("getbuckettagging", httpTraceAll(api.GetBucketTaggingHandler)))).Queries("tagging", "")
 		//DeleteBucketWebsiteHandler
@@ -244,6 +244,9 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled, allowSSEKMS bool)
 		// PutBucketObjectLockConfig
 		bucket.Methods(http.MethodPut).HandlerFunc(
 			maxClients(collectAPIStats("putbucketobjectlockconfig", httpTraceAll(api.PutBucketObjectLockConfigHandler)))).Queries("object-lock", "")
+		// PutBucketTaggingHandler
+		bucket.Methods(http.MethodPut).HandlerFunc(
+			maxClients(collectAPIStats("putbuckettagging", httpTraceAll(api.PutBucketTaggingHandler)))).Queries("tagging", "")
 		// PutBucketVersioning
 		bucket.Methods(http.MethodPut).HandlerFunc(
 			maxClients(collectAPIStats("putbucketversioning", httpTraceAll(api.PutBucketVersioningHandler)))).Queries("versioning", "")

cmd/bucket-encryption-handlers.go

@@ -18,6 +18,7 @@ package cmd
 
 import (
 	"encoding/xml"
+	"fmt"
 	"io"
 	"net/http"
 
@@ -69,7 +70,12 @@ func (api objectAPIHandlers) PutBucketEncryptionHandler(w http.ResponseWriter, r
 	// Parse bucket encryption xml
 	encConfig, err := validateBucketSSEConfig(io.LimitReader(r.Body, maxBucketSSEConfigSize))
 	if err != nil {
-		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMalformedXML), r.URL, guessIsBrowserReq(r))
+		apiErr := APIError{
+			Code:           "MalformedXML",
+			Description:    fmt.Sprintf("%s (%s)", errorCodes[ErrMalformedXML].Description, err),
+			HTTPStatusCode: errorCodes[ErrMalformedXML].HTTPStatusCode,
+		}
+		writeErrorResponse(ctx, w, apiErr, r.URL, guessIsBrowserReq(r))
 		return
 	}
 
@@ -86,7 +92,7 @@ func (api objectAPIHandlers) PutBucketEncryptionHandler(w http.ResponseWriter, r
 	}
 
 	// Update the in-memory bucket encryption cache
-	globalBucketSSEConfigSys.Set(bucket, *encConfig)
+	globalBucketSSEConfigSys.Set(bucket, encConfig)
 
 	// Update peer MinIO servers of the updated bucket encryption config
 	globalNotificationSys.SetBucketSSEConfig(ctx, bucket, encConfig)
@@ -174,6 +180,7 @@ func (api objectAPIHandlers) DeleteBucketEncryptionHandler(w http.ResponseWriter
 
 	// Remove entry from the in-memory bucket encryption cache
 	globalBucketSSEConfigSys.Remove(bucket)
+
 	// Update peer MinIO servers of the updated bucket encryption config
 	globalNotificationSys.RemoveBucketSSEConfig(ctx, bucket)
 

cmd/bucket-encryption.go

@@ -31,13 +31,13 @@ import (
 // BucketSSEConfigSys - in-memory cache of bucket encryption config
 type BucketSSEConfigSys struct {
 	sync.RWMutex
-	bucketSSEConfigMap map[string]bucketsse.BucketSSEConfig
+	bucketSSEConfigMap map[string]*bucketsse.BucketSSEConfig
 }
 
 // NewBucketSSEConfigSys - Creates an empty in-memory bucket encryption configuration cache
 func NewBucketSSEConfigSys() *BucketSSEConfigSys {
 	return &BucketSSEConfigSys{
-		bucketSSEConfigMap: make(map[string]bucketsse.BucketSSEConfig),
+		bucketSSEConfigMap: make(map[string]*bucketsse.BucketSSEConfig),
 	}
 }
 
@@ -47,11 +47,12 @@ func (sys *BucketSSEConfigSys) load(buckets []BucketInfo, objAPI ObjectLayer) er
 		config, err := objAPI.GetBucketSSEConfig(GlobalContext, bucket.Name)
 		if err != nil {
 			if _, ok := err.(BucketSSEConfigNotFound); ok {
-				sys.Remove(bucket.Name)
+				continue
 			}
-			continue
+			// Quorum errors should be returned.
+			return err
 		}
-		sys.Set(bucket.Name, *config)
+		sys.Set(bucket.Name, config)
 	}
 
 	return nil
@@ -73,7 +74,7 @@ func (sys *BucketSSEConfigSys) Init(buckets []BucketInfo, objAPI ObjectLayer) er
 }
 
 // Get - gets bucket encryption config for the given bucket.
-func (sys *BucketSSEConfigSys) Get(bucket string) (config bucketsse.BucketSSEConfig, ok bool) {
+func (sys *BucketSSEConfigSys) Get(bucket string) (config *bucketsse.BucketSSEConfig, ok bool) {
 	// We don't cache bucket encryption config in gateway mode.
 	if globalIsGateway {
 		objAPI := newObjectLayerWithoutSafeModeFn()
@@ -85,7 +86,7 @@ func (sys *BucketSSEConfigSys) Get(bucket string) (config bucketsse.BucketSSECon
 		if err != nil {
 			return
 		}
-		return *cfg, true
+		return cfg, true
 	}
 
 	sys.Lock()
@@ -95,7 +96,7 @@ func (sys *BucketSSEConfigSys) Get(bucket string) (config bucketsse.BucketSSECon
 }
 
 // Set - sets bucket encryption config to given bucket name.
-func (sys *BucketSSEConfigSys) Set(bucket string, config bucketsse.BucketSSEConfig) {
+func (sys *BucketSSEConfigSys) Set(bucket string, config *bucketsse.BucketSSEConfig) {
 	// We don't cache bucket encryption config in gateway mode.
 	if globalIsGateway {
 		return
@@ -146,8 +147,8 @@ func removeBucketSSEConfig(ctx context.Context, objAPI ObjectLayer, bucket strin
 	// Path to bucket-encryption.xml for the given bucket.
 	configFile := path.Join(bucketConfigPrefix, bucket, bucketSSEConfig)
 
-	if err := objAPI.DeleteObject(ctx, minioMetaBucket, configFile); err != nil {
-		if _, ok := err.(ObjectNotFound); ok {
+	if err := deleteConfig(ctx, objAPI, configFile); err != nil {
+		if err == errConfigNotFound {
 			return BucketSSEConfigNotFound{Bucket: bucket}
 		}
 		return err

cmd/bucket-handlers.go

@@ -30,6 +30,7 @@ import (
 	"github.com/gorilla/mux"
 
 	"github.com/minio/minio-go/v6/pkg/set"
+	"github.com/minio/minio-go/v6/pkg/tags"
 	"github.com/minio/minio/cmd/config/etcd/dns"
 	"github.com/minio/minio/cmd/crypto"
 	xhttp "github.com/minio/minio/cmd/http"
@@ -46,6 +47,7 @@ import (
 const (
 	getBucketVersioningResponse = `<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>`
 	objectLockConfig            = "object-lock.xml"
+	bucketTaggingConfigFile     = "tagging.xml"
 )
 
 // Check if there are buckets on server without corresponding entry in etcd backend and
@@ -293,10 +295,12 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
 
 		for i := range bucketsInfo {
 			meta, err := loadBucketMetadata(ctx, objectAPI, bucketsInfo[i].Name)
-			logger.LogIf(ctx, err)
 			if err == nil {
 				bucketsInfo[i].Created = meta.Created
 			}
+			if err != errMetaDataConverted {
+				logger.LogIf(ctx, err)
+			}
 		}
 	}
 
@@ -589,8 +593,9 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
 			return
 		}
 		if objectLockEnabled {
-			globalBucketObjectLockConfig.Set(bucket, objectlock.Retention{})
-			globalNotificationSys.PutBucketObjectLockConfig(ctx, bucket, objectlock.Retention{})
+			ret := &objectlock.Retention{}
+			globalBucketObjectLockConfig.Set(bucket, ret)
+			globalNotificationSys.PutBucketObjectLockConfig(ctx, bucket, ret)
 		}
 	}
 
@@ -627,7 +632,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
 	bucket := mux.Vars(r)["bucket"]
 
 	// To detect if the client has disconnected.
-	r.Body = &detectDisconnect{r.Body, r.Context().Done()}
+	r.Body = &contextReader{r.Body, r.Context()}
 
 	// Require Content-Length to be set in the request
 	size := r.ContentLength
@@ -1069,8 +1074,8 @@ func (api objectAPIHandlers) PutBucketObjectLockConfigHandler(w http.ResponseWri
 		globalBucketObjectLockConfig.Set(bucket, retention)
 		globalNotificationSys.PutBucketObjectLockConfig(ctx, bucket, retention)
 	} else {
-		globalBucketObjectLockConfig.Remove(bucket)
-		globalNotificationSys.RemoveBucketObjectLockConfig(ctx, bucket)
+		globalBucketObjectLockConfig.Set(bucket, &objectlock.Retention{})
+		globalNotificationSys.PutBucketObjectLockConfig(ctx, bucket, &objectlock.Retention{})
 	}
 
 	// Write success response.
@@ -1102,7 +1107,7 @@ func (api objectAPIHandlers) GetBucketObjectLockConfigHandler(w http.ResponseWri
 	}
 
 	meta, err := loadBucketMetadata(ctx, objectAPI, bucket)
-	if err != nil {
+	if err != nil && err != errMetaDataConverted {
 		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
 		return
 	}
@@ -1128,3 +1133,113 @@ func (api objectAPIHandlers) GetBucketObjectLockConfigHandler(w http.ResponseWri
 	// Write success response.
 	writeSuccessResponseXML(w, configData)
 }
+
+// PutBucketTaggingHandler - PUT Bucket tagging.
+// ----------
+func (api objectAPIHandlers) PutBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
+	ctx := newContext(r, w, "PutBucketTagging")
+
+	defer logger.AuditLog(w, r, "PutBucketTagging", mustGetClaimsFromToken(r))
+
+	vars := mux.Vars(r)
+	bucket := vars["bucket"]
+
+	objectAPI := api.ObjectAPI()
+	if objectAPI == nil {
+		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
+		return
+	}
+
+	if s3Error := checkRequestAuthType(ctx, r, policy.PutBucketTaggingAction, bucket, ""); s3Error != ErrNone {
+		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
+		return
+	}
+	tags, err := tags.ParseBucketXML(io.LimitReader(r.Body, r.ContentLength))
+	if err != nil {
+		apiErr := errorCodes.ToAPIErr(ErrMalformedXML)
+		apiErr.Description = err.Error()
+		writeErrorResponse(ctx, w, apiErr, r.URL, guessIsBrowserReq(r))
+		return
+	}
+	data, err := xml.Marshal(tags)
+	if err != nil {
+		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
+		return
+	}
+	configFile := path.Join(bucketConfigPrefix, bucket, bucketTaggingConfigFile)
+	if err = saveConfig(ctx, objectAPI, configFile, data); err != nil {
+		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
+		return
+	}
+
+	// Write success response.
+	writeSuccessResponseHeadersOnly(w)
+}
+
+// GetBucketTaggingHandler - GET Bucket tagging.
+// ----------
+func (api objectAPIHandlers) GetBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
+	ctx := newContext(r, w, "GetBucketTagging")
+
+	defer logger.AuditLog(w, r, "GetBucketTagging", mustGetClaimsFromToken(r))
+
+	vars := mux.Vars(r)
+	bucket := vars["bucket"]
+
+	objectAPI := api.ObjectAPI()
+	if objectAPI == nil {
+		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
+		return
+	}
+	// check if user has permissions to perform this operation
+	if s3Error := checkRequestAuthType(ctx, r, policy.GetBucketTaggingAction, bucket, ""); s3Error != ErrNone {
+		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
+		return
+	}
+	configFile := path.Join(bucketConfigPrefix, bucket, bucketTaggingConfigFile)
+	configData, err := readConfig(ctx, objectAPI, configFile)
+	if err != nil {
+		var aerr APIError
+		if err == errConfigNotFound {
+			aerr = errorCodes.ToAPIErr(ErrBucketTaggingNotFound)
+		} else {
+			aerr = toAPIError(ctx, err)
+		}
+		writeErrorResponse(ctx, w, aerr, r.URL, guessIsBrowserReq(r))
+		return
+	}
+
+	// Write success response.
+	writeSuccessResponseXML(w, configData)
+}
+
+// DeleteBucketTaggingHandler - DELETE Bucket tagging.
+// ----------
+func (api objectAPIHandlers) DeleteBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
+	ctx := newContext(r, w, "DeleteBucketTagging")
+
+	defer logger.AuditLog(w, r, "DeleteBucketTagging", mustGetClaimsFromToken(r))
+
+	vars := mux.Vars(r)
+	bucket := vars["bucket"]
+
+	objectAPI := api.ObjectAPI()
+	if objectAPI == nil {
+		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
+		return
+	}
+
+	if s3Error := checkRequestAuthType(ctx, r, policy.PutBucketTaggingAction, bucket, ""); s3Error != ErrNone {
+		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
+		return
+	}
+
+	configFile := path.Join(bucketConfigPrefix, bucket, bucketTaggingConfigFile)
+	if err := deleteConfig(ctx, objectAPI, configFile); err != nil && err != errConfigNotFound {
+		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
+		return
+	}
+
+	// Write success response.
+	writeSuccessResponseHeadersOnly(w)
+}

cmd/bucket-lifecycle-handler.go

@@ -77,7 +77,7 @@ func (api objectAPIHandlers) PutBucketLifecycleHandler(w http.ResponseWriter, r
 		return
 	}
 
-	globalLifecycleSys.Set(bucket, *bucketLifecycle)
+	globalLifecycleSys.Set(bucket, bucketLifecycle)
 	globalNotificationSys.SetBucketLifecycle(ctx, bucket, bucketLifecycle)
 
 	// Success.

cmd/bucket-lifecycle.go

@@ -35,11 +35,11 @@ const (
 // LifecycleSys - Bucket lifecycle subsystem.
 type LifecycleSys struct {
 	sync.RWMutex
-	bucketLifecycleMap map[string]lifecycle.Lifecycle
+	bucketLifecycleMap map[string]*lifecycle.Lifecycle
 }
 
 // Set - sets lifecycle config to given bucket name.
-func (sys *LifecycleSys) Set(bucketName string, lifecycle lifecycle.Lifecycle) {
+func (sys *LifecycleSys) Set(bucketName string, lifecycle *lifecycle.Lifecycle) {
 	if globalIsGateway {
 		// no-op
 		return
@@ -52,7 +52,7 @@ func (sys *LifecycleSys) Set(bucketName string, lifecycle lifecycle.Lifecycle) {
 }
 
 // Get - gets lifecycle config associated to a given bucket name.
-func (sys *LifecycleSys) Get(bucketName string) (lc lifecycle.Lifecycle, ok bool) {
+func (sys *LifecycleSys) Get(bucketName string) (lc *lifecycle.Lifecycle, ok bool) {
 	if globalIsGateway {
 		// When gateway is enabled, no cached value
 		// is used to validate life cycle policies.
@@ -65,7 +65,7 @@ func (sys *LifecycleSys) Get(bucketName string) (lc lifecycle.Lifecycle, ok bool
 		if err != nil {
 			return
 		}
-		return *l, true
+		return l, true
 	}
 
 	sys.Lock()
@@ -104,8 +104,8 @@ func removeLifecycleConfig(ctx context.Context, objAPI ObjectLayer, bucketName s
 	// Construct path to lifecycle.xml for the given bucket.
 	configFile := path.Join(bucketConfigPrefix, bucketName, bucketLifecycleConfig)
 
-	if err := objAPI.DeleteObject(ctx, minioMetaBucket, configFile); err != nil {
-		if _, ok := err.(ObjectNotFound); ok {
+	if err := deleteConfig(ctx, objAPI, configFile); err != nil {
+		if err == errConfigNotFound {
 			return BucketLifecycleNotFound{Bucket: bucketName}
 		}
 		return err
@@ -116,7 +116,7 @@ func removeLifecycleConfig(ctx context.Context, objAPI ObjectLayer, bucketName s
 // NewLifecycleSys - creates new lifecycle system.
 func NewLifecycleSys() *LifecycleSys {
 	return &LifecycleSys{
-		bucketLifecycleMap: make(map[string]lifecycle.Lifecycle),
+		bucketLifecycleMap: make(map[string]*lifecycle.Lifecycle),
 	}
 }
 
@@ -139,15 +139,16 @@ func (sys *LifecycleSys) Init(buckets []BucketInfo, objAPI ObjectLayer) error {
 // Loads lifecycle policies for all buckets into LifecycleSys.
 func (sys *LifecycleSys) load(buckets []BucketInfo, objAPI ObjectLayer) error {
 	for _, bucket := range buckets {
-		config, err := objAPI.GetBucketLifecycle(GlobalContext, bucket.Name)
+		config, err := getLifecycleConfig(objAPI, bucket.Name)
 		if err != nil {
 			if _, ok := err.(BucketLifecycleNotFound); ok {
-				sys.Remove(bucket.Name)
+				continue
 			}
-			continue
+			// Quorum errors should be returned.
+			return err
 		}
 
-		sys.Set(bucket.Name, *config)
+		sys.Set(bucket.Name, config)
 	}
 
 	return nil

cmd/bucket-meta.go

@@ -115,13 +115,15 @@ func (b *bucketMetadata) convertLegacyLockconfig(ctx context.Context, objectAPI
 			return err
 		}
 
-		logger.LogIf(ctx, deleteConfig(ctx, objectAPI, configFile))
+		if err := deleteConfig(ctx, objectAPI, configFile); err != nil && !errors.Is(err, errConfigNotFound) {
+			logger.LogIf(ctx, err)
+		}
 		return nil
 	}
 
 	configData, err := readConfig(ctx, objectAPI, configFile)
 	if err != nil {
-		if err != errConfigNotFound {
+		if !errors.Is(err, errConfigNotFound) {
 			return err
 		}
 

cmd/bucket-object-lock.go

@@ -31,7 +31,7 @@ import (
 )
 
 // Similar to enforceRetentionBypassForDelete but for WebUI
-func enforceRetentionBypassForDeleteWeb(ctx context.Context, r *http.Request, bucket, object string, getObjectInfoFn GetObjectInfoFn) APIErrorCode {
+func enforceRetentionBypassForDeleteWeb(ctx context.Context, r *http.Request, bucket, object string, getObjectInfoFn GetObjectInfoFn, govBypassPerms bool) APIErrorCode {
 	opts, err := getOpts(ctx, r, bucket, object)
 	if err != nil {
 		return toAPIErrorCode(ctx, err)
@@ -80,7 +80,7 @@ func enforceRetentionBypassForDeleteWeb(ctx context.Context, r *http.Request, bu
 			// and must explicitly include x-amz-bypass-governance-retention:true
 			// as a request header with any request that requires overriding
 			// governance mode.
-			byPassSet := objectlock.IsObjectLockGovernanceBypassSet(r.Header)
+			byPassSet := govBypassPerms && objectlock.IsObjectLockGovernanceBypassSet(r.Header)
 			if !byPassSet {
 				t, err := objectlock.UTCNowNTP()
 				if err != nil {
@@ -91,6 +91,11 @@ func enforceRetentionBypassForDeleteWeb(ctx context.Context, r *http.Request, bu
 				if !ret.RetainUntilDate.Before(t) {
 					return ErrObjectLocked
 				}
+
+				if !govBypassPerms {
+					return ErrObjectLocked
+				}
+
 				return ErrNone
 			}
 		}
@@ -393,7 +398,7 @@ func initBucketObjectLockConfig(buckets []BucketInfo, objAPI ObjectLayer) error
 		configData, err := readConfig(ctx, objAPI, configFile)
 		if err != nil {
 			if errors.Is(err, errConfigNotFound) {
-				globalBucketObjectLockConfig.Set(bucket.Name, objectlock.Retention{})
+				globalBucketObjectLockConfig.Set(bucket.Name, &objectlock.Retention{})
 				continue
 			}
 			return err
@@ -403,7 +408,7 @@ func initBucketObjectLockConfig(buckets []BucketInfo, objAPI ObjectLayer) error
 		if err != nil {
 			return err
 		}
-		retention := objectlock.Retention{}
+		retention := &objectlock.Retention{}
 		if config.Rule != nil {
 			retention = config.ToRetention()
 		}

cmd/bucket-policy-handlers.go

@@ -92,7 +92,7 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
 		return
 	}
 
-	globalPolicySys.Set(bucket, *bucketPolicy)
+	globalPolicySys.Set(bucket, bucketPolicy)
 	globalNotificationSys.SetBucketPolicy(ctx, bucket, bucketPolicy)
 
 	// Success.

cmd/bucket-policy.go

@@ -39,11 +39,11 @@ import (
 // PolicySys - policy subsystem.
 type PolicySys struct {
 	sync.RWMutex
-	bucketPolicyMap map[string]policy.Policy
+	bucketPolicyMap map[string]*policy.Policy
 }
 
 // Set - sets policy to given bucket name.  If policy is empty, existing policy is removed.
-func (sys *PolicySys) Set(bucketName string, policy policy.Policy) {
+func (sys *PolicySys) Set(bucketName string, policy *policy.Policy) {
 	if globalIsGateway {
 		// Set policy is a non-op under gateway mode.
 		return
@@ -97,12 +97,12 @@ func (sys *PolicySys) IsAllowed(args policy.Args) bool {
 // Loads policies for all buckets into PolicySys.
 func (sys *PolicySys) load(buckets []BucketInfo, objAPI ObjectLayer) error {
 	for _, bucket := range buckets {
-		config, err := objAPI.GetBucketPolicy(GlobalContext, bucket.Name)
+		config, err := getPolicyConfig(objAPI, bucket.Name)
 		if err != nil {
 			if _, ok := err.(BucketPolicyNotFound); ok {
-				sys.Remove(bucket.Name)
+				continue
 			}
-			continue
+			return err
 		}
 		// This part is specifically written to handle migration
 		// when the Version string is empty, this was allowed
@@ -118,7 +118,7 @@ func (sys *PolicySys) load(buckets []BucketInfo, objAPI ObjectLayer) error {
 				return err
 			}
 		}
-		sys.Set(bucket.Name, *config)
+		sys.Set(bucket.Name, config)
 	}
 	return nil
 }
@@ -126,7 +126,7 @@ func (sys *PolicySys) load(buckets []BucketInfo, objAPI ObjectLayer) error {
 // Init - initializes policy system from policy.json of all buckets.
 func (sys *PolicySys) Init(buckets []BucketInfo, objAPI ObjectLayer) error {
 	if objAPI == nil {
-		return errInvalidArgument
+		return errServerNotInitialized
 	}
 
 	// In gateway mode, we don't need to load the policies
@@ -142,7 +142,7 @@ func (sys *PolicySys) Init(buckets []BucketInfo, objAPI ObjectLayer) error {
 // NewPolicySys - creates new policy system.
 func NewPolicySys() *PolicySys {
 	return &PolicySys{
-		bucketPolicyMap: make(map[string]policy.Policy),
+		bucketPolicyMap: make(map[string]*policy.Policy),
 	}
 }
 

cmd/bucket-quota.go

@@ -60,23 +60,50 @@ func (sys *BucketQuotaSys) Remove(bucketName string) {
 	sys.Unlock()
 }
 
-// Exists - bucketName has Quota config set
-func (sys *BucketQuotaSys) Exists(bucketName string) bool {
-	sys.RLock()
-	_, ok := sys.quotaMap[bucketName]
-	sys.RUnlock()
-	return ok
-}
-
-// Keys - list of buckets with quota configuration
-func (sys *BucketQuotaSys) Keys() []string {
+// Buckets - list of buckets with quota configuration
+func (sys *BucketQuotaSys) Buckets() []string {
 	sys.RLock()
 	defer sys.RUnlock()
-	var keys []string
+	var buckets []string
 	for k := range sys.quotaMap {
-		keys = append(keys, k)
+		buckets = append(buckets, k)
 	}
-	return keys
+	return buckets
+}
+
+// Init initialize bucket quota sys configuration with all buckets.
+func (sys *BucketQuotaSys) Init(buckets []BucketInfo, objAPI ObjectLayer) error {
+	if objAPI == nil {
+		return errServerNotInitialized
+	}
+
+	// In gateway mode, we do not support bucket quota.
+	// So, this is a no-op for gateway servers.
+	if globalIsGateway {
+		return nil
+	}
+
+	return sys.load(buckets, objAPI)
+}
+
+func (sys *BucketQuotaSys) load(buckets []BucketInfo, objAPI ObjectLayer) error {
+	for _, bucket := range buckets {
+		ctx := logger.SetReqInfo(GlobalContext, &logger.ReqInfo{BucketName: bucket.Name})
+		configFile := path.Join(bucketConfigPrefix, bucket.Name, bucketQuotaConfigFile)
+		configData, err := readConfig(ctx, objAPI, configFile)
+		if err != nil {
+			if errors.Is(err, errConfigNotFound) {
+				continue
+			}
+			return err
+		}
+		quotaCfg, err := parseBucketQuota(configData)
+		if err != nil {
+			return err
+		}
+		globalBucketQuotaSys.Set(bucket.Name, quotaCfg)
+	}
+	return nil
 }
 
 // NewBucketQuotaSys returns initialized BucketQuotaSys
@@ -131,26 +158,6 @@ func enforceBucketQuota(ctx context.Context, bucket string, size int64) error {
 	return globalBucketStorageCache.check(ctx, q, bucket, size)
 }
 
-func initBucketQuotaSys(buckets []BucketInfo, objAPI ObjectLayer) error {
-	for _, bucket := range buckets {
-		ctx := logger.SetReqInfo(GlobalContext, &logger.ReqInfo{BucketName: bucket.Name})
-		configFile := path.Join(bucketConfigPrefix, bucket.Name, bucketQuotaConfigFile)
-		configData, err := readConfig(ctx, objAPI, configFile)
-		if err != nil {
-			if errors.Is(err, errConfigNotFound) {
-				continue
-			}
-			return err
-		}
-		quotaCfg, err := parseBucketQuota(configData)
-		if err != nil {
-			return err
-		}
-		globalBucketQuotaSys.Set(bucket.Name, quotaCfg)
-	}
-	return nil
-}
-
 const (
 	bgQuotaInterval = 1 * time.Hour
 )
@@ -182,7 +189,7 @@ func enforceFIFOQuota(ctx context.Context, objectAPI ObjectLayer) error {
 	if env.Get(envDataUsageCrawlConf, config.EnableOn) == config.EnableOff {
 		return nil
 	}
-	for _, bucket := range globalBucketQuotaSys.Keys() {
+	for _, bucket := range globalBucketQuotaSys.Buckets() {
 		// Check if the current bucket has quota restrictions, if not skip it
 		cfg, ok := globalBucketQuotaSys.Get(bucket)
 		if !ok {

cmd/config-encrypted.go

@@ -36,42 +36,10 @@ func handleEncryptedConfigBackend(objAPI ObjectLayer, server bool) error {
 		return nil
 	}
 
-	// If its server mode or nas gateway, migrate the backend.
-	doneCh := make(chan struct{})
-
-	var encrypted bool
-	var err error
-
-	// Migrating Config backend needs a retry mechanism for
-	// the following reasons:
-	//  - Read quorum is lost just after the initialization
-	//    of the object layer.
-	retryTimerCh := newRetryTimerSimple(doneCh)
-	var stop bool
-
-	rquorum := InsufficientReadQuorum{}
-	wquorum := InsufficientWriteQuorum{}
-
-	for !stop {
-		select {
-		case <-retryTimerCh:
-			if encrypted, err = checkBackendEncrypted(objAPI); err != nil {
-				if errors.Is(err, errDiskNotFound) ||
-					errors.As(err, &rquorum) ||
-					isErrBucketNotFound(err) {
-					logger.Info("Waiting for config backend to be encrypted..")
-					continue
-				}
-				close(doneCh)
-				return err
-			}
-			stop = true
-		case <-globalOSSignalCh:
-			close(doneCh)
-			return fmt.Errorf("Config encryption process stopped gracefully")
-		}
+	encrypted, err := checkBackendEncrypted(objAPI)
+	if err != nil {
+		return fmt.Errorf("Unable to encrypt config %w", err)
 	}
-	close(doneCh)
 
 	if encrypted {
 		// backend is encrypted, but credentials are not specified
@@ -91,34 +59,12 @@ func handleEncryptedConfigBackend(objAPI ObjectLayer, server bool) error {
 		}
 	}
 
-	doneCh = make(chan struct{})
-	defer close(doneCh)
-
-	retryTimerCh = newRetryTimerSimple(doneCh)
-
-	// Migrating Config backend needs a retry mechanism for
-	// the following reasons:
-	//  - Read quorum is lost just after the initialization
-	//    of the object layer.
-	for {
-		select {
-		case <-retryTimerCh:
-			// Migrate IAM configuration
-			if err = migrateConfigPrefixToEncrypted(objAPI, globalOldCred, encrypted); err != nil {
-				if errors.Is(err, errDiskNotFound) ||
-					errors.As(err, &rquorum) ||
-					errors.As(err, &wquorum) ||
-					isErrBucketNotFound(err) {
-					logger.Info("Waiting for config backend to be encrypted..")
-					continue
-				}
-				return err
-			}
-			return nil
-		case <-globalOSSignalCh:
-			return fmt.Errorf("Config encryption process stopped gracefully")
-		}
+	// Migrate IAM configuration
+	if err = migrateConfigPrefixToEncrypted(objAPI, globalOldCred, encrypted); err != nil {
+		return fmt.Errorf("Unable to migrate all config at .minio.sys/config/: %w", err)
 	}
+
+	return nil
 }
 
 const (

cmd/config/errors.go

@@ -169,6 +169,12 @@ Example 1:
    $ minio server /data/minio/`,
 	)
 
+	ErrUnsupportedBackend = newErrFn(
+		"Unable to write to the backend",
+		"Please ensure your disk supports O_DIRECT",
+		"",
+	)
+
 	ErrUnableToWriteInBackend = newErrFn(
 		"Unable to write to the backend",
 		"Please ensure MinIO binary has write permissions for the backend",

cmd/config/identity/ldap/config.go

@@ -58,6 +58,7 @@ type Config struct {
 	stsExpiryDuration time.Duration // contains converted value
 	tlsSkipVerify     bool          // allows skipping TLS verification
 	serverInsecure    bool          // allows plain text connection to LDAP Server
+	serverStartTLS    bool          // allows plain text connection to LDAP Server
 	rootCAs           *x509.CertPool
 }
 
@@ -73,11 +74,13 @@ const (
 	GroupSearchBaseDN    = "group_search_base_dn"
 	TLSSkipVerify        = "tls_skip_verify"
 	ServerInsecure       = "server_insecure"
+	ServerStartTLS       = "server_starttls"
 
 	EnvServerAddr           = "MINIO_IDENTITY_LDAP_SERVER_ADDR"
 	EnvSTSExpiry            = "MINIO_IDENTITY_LDAP_STS_EXPIRY"
 	EnvTLSSkipVerify        = "MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY"
 	EnvServerInsecure       = "MINIO_IDENTITY_LDAP_SERVER_INSECURE"
+	EnvServerStartTLS       = "MINIO_IDENTITY_LDAP_SERVER_STARTTLS"
 	EnvUsernameFormat       = "MINIO_IDENTITY_LDAP_USERNAME_FORMAT"
 	EnvUsernameSearchFilter = "MINIO_IDENTITY_LDAP_USERNAME_SEARCH_FILTER"
 	EnvUsernameSearchBaseDN = "MINIO_IDENTITY_LDAP_USERNAME_SEARCH_BASE_DN"
@@ -129,6 +132,10 @@ var (
 			Key:   ServerInsecure,
 			Value: config.EnableOff,
 		},
+		config.KV{
+			Key:   ServerStartTLS,
+			Value: config.EnableOff,
+		},
 	}
 )
 
@@ -257,6 +264,18 @@ func (l *Config) Connect() (ldapConn *ldap.Conn, err error) {
 		return ldap.Dial("tcp", l.ServerAddr)
 	}
 
+	if l.serverStartTLS {
+		conn, err := ldap.Dial("tcp", l.ServerAddr)
+		if err != nil {
+			return nil, err
+		}
+		err = conn.StartTLS(&tls.Config{
+			InsecureSkipVerify: l.tlsSkipVerify,
+			RootCAs:            l.rootCAs,
+		})
+		return conn, err
+	}
+
 	return ldap.DialTLS("tcp", l.ServerAddr, &tls.Config{
 		InsecureSkipVerify: l.tlsSkipVerify,
 		RootCAs:            l.rootCAs,
@@ -303,6 +322,12 @@ func Lookup(kvs config.KVS, rootCAs *x509.CertPool) (l Config, err error) {
 			return l, err
 		}
 	}
+	if v := env.Get(EnvServerStartTLS, kvs.Get(ServerStartTLS)); v != "" {
+		l.serverStartTLS, err = config.ParseBool(v)
+		if err != nil {
+			return l, err
+		}
+	}
 	if v := env.Get(EnvTLSSkipVerify, kvs.Get(TLSSkipVerify)); v != "" {
 		l.tlsSkipVerify, err = config.ParseBool(v)
 		if err != nil {

cmd/crypto/kes.go

@@ -1,4 +1,4 @@
-// MinIO Cloud Storage, (C) 2019 MinIO, Inc.
+// MinIO Cloud Storage, (C) 2019-2020 MinIO, Inc.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -19,6 +19,7 @@ import (
 	"crypto/tls"
 	"crypto/x509"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"io/ioutil"
@@ -27,6 +28,10 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"time"
+
+	xhttp "github.com/minio/minio/cmd/http"
+	xnet "github.com/minio/minio/pkg/net"
 )
 
 // KesConfig contains the configuration required
@@ -112,6 +117,7 @@ func NewKes(cfg KesConfig) (KMS, error) {
 		Certificates: []tls.Certificate{cert},
 		RootCAs:      certPool,
 	}
+	cfg.Transport.ForceAttemptHTTP2 = true
 	return &kesService{
 		client: &kesClient{
 			addr: cfg.Endpoint,
@@ -208,6 +214,18 @@ type kesClient struct {
 	httpClient http.Client
 }
 
+// Response KES response struct
+type response struct {
+	Plaintext  []byte `json:"plaintext"`
+	Ciphertext []byte `json:"ciphertext,omitempty"`
+}
+
+// Request KES request struct
+type request struct {
+	Ciphertext []byte `json:"ciphertext,omitempty"`
+	Context    []byte `json:"context"`
+}
+
 // GenerateDataKey requests a new data key from the KES server.
 // On success, the KES server will respond with the plaintext key
 // and the ciphertext key as the plaintext key encrypted with
@@ -217,10 +235,7 @@ type kesClient struct {
 // such that you have to provide the same context when decrypting
 // the data key.
 func (c *kesClient) GenerateDataKey(name string, context []byte) ([]byte, []byte, error) {
-	type Request struct {
-		Context []byte `json:"context"`
-	}
-	body, err := json.Marshal(Request{
+	body, err := json.Marshal(request{
 		Context: context,
 	})
 	if err != nil {
@@ -228,25 +243,57 @@ func (c *kesClient) GenerateDataKey(name string, context []byte) ([]byte, []byte
 	}
 
 	url := fmt.Sprintf("%s/v1/key/generate/%s", c.addr, url.PathEscape(name))
-	resp, err := c.httpClient.Post(url, "application/json", bytes.NewReader(body))
+
+	const limit = 1 << 20 // A plaintext/ciphertext key pair will never be larger than 1 MB
+	resp, err := c.postRetry(url, bytes.NewReader(body), limit)
 	if err != nil {
 		return nil, nil, err
 	}
-	if resp.StatusCode != http.StatusOK {
-		return nil, nil, c.parseErrorResponse(resp)
-	}
-	defer resp.Body.Close()
 
-	type Response struct {
-		Plaintext  []byte `json:"plaintext"`
-		Ciphertext []byte `json:"ciphertext"`
+	return resp.Plaintext, resp.Ciphertext, nil
+}
+
+func (c *kesClient) post(url string, body io.Reader, limit int64) (*response, error) {
+	resp, err := c.httpClient.Post(url, "application/json", body)
+	if err != nil {
+		return nil, err
 	}
-	const limit = 1 << 20 // A plaintext/ciphertext key pair will never be larger than 1 MB
-	var response Response
-	if err = json.NewDecoder(io.LimitReader(resp.Body, limit)).Decode(&response); err != nil {
-		return nil, nil, err
+
+	// Drain the entire body to make sure we have re-use connections
+	defer xhttp.DrainBody(resp.Body)
+
+	if resp.StatusCode != http.StatusOK {
+		return nil, c.parseErrorResponse(resp)
+	}
+
+	response := &response{}
+	if err = json.NewDecoder(io.LimitReader(resp.Body, limit)).Decode(response); err != nil {
+		return nil, err
+	}
+	return response, nil
+}
+
+func (c *kesClient) postRetry(url string, body io.ReadSeeker, limit int64) (*response, error) {
+	for i := 0; ; i++ {
+		body.Seek(0, io.SeekStart) // seek to the beginning of the body.
+
+		response, err := c.post(url, body, limit)
+		if err == nil {
+			return response, nil
+		}
+
+		if !xnet.IsNetworkOrHostDown(err) && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) {
+			return nil, err
+		}
+
+		// retriable network errors.
+		remain := retryMax - i
+		if remain <= 0 {
+			return response, err
+		}
+
+		<-time.After(LinearJitterBackoff(retryWaitMin, retryWaitMax, i))
 	}
-	return response.Plaintext, response.Ciphertext, nil
 }
 
 // GenerateDataKey decrypts an encrypted data key with the key
@@ -256,11 +303,7 @@ func (c *kesClient) GenerateDataKey(name string, context []byte) ([]byte, []byte
 // The optional context must match the value you provided when
 // generating the data key.
 func (c *kesClient) DecryptDataKey(name string, ciphertext, context []byte) ([]byte, error) {
-	type Request struct {
-		Ciphertext []byte `json:"ciphertext"`
-		Context    []byte `json:"context"`
-	}
-	body, err := json.Marshal(Request{
+	body, err := json.Marshal(request{
 		Ciphertext: ciphertext,
 		Context:    context,
 	})
@@ -269,37 +312,26 @@ func (c *kesClient) DecryptDataKey(name string, ciphertext, context []byte) ([]b
 	}
 
 	url := fmt.Sprintf("%s/v1/key/decrypt/%s", c.addr, url.PathEscape(name))
-	resp, err := c.httpClient.Post(url, "application/json", bytes.NewReader(body))
+
+	const limit = 32 * 1024 // A data key will never be larger than 32 KB
+	resp, err := c.postRetry(url, bytes.NewReader(body), limit)
 	if err != nil {
 		return nil, err
 	}
-	if resp.StatusCode != http.StatusOK {
-		return nil, c.parseErrorResponse(resp)
-	}
-	defer resp.Body.Close()
-
-	type Response struct {
-		Plaintext []byte `json:"plaintext"`
-	}
-	const limit = 32 * 1024 // A data key will never be larger than 32 KB
-	var response Response
-	if err = json.NewDecoder(io.LimitReader(resp.Body, limit)).Decode(&response); err != nil {
-		return nil, err
-	}
-	return response.Plaintext, nil
+	return resp.Plaintext, nil
 }
 
 func (c *kesClient) parseErrorResponse(resp *http.Response) error {
 	if resp.Body == nil {
-		return nil
+		return Errorf("%s: no body", http.StatusText(resp.StatusCode))
 	}
-	defer resp.Body.Close()
 
 	const limit = 32 * 1024 // A (valid) error response will not be greater than 32 KB
 	var errMsg strings.Builder
 	if _, err := io.Copy(&errMsg, io.LimitReader(resp.Body, limit)); err != nil {
-		return err
+		return Errorf("%s: %s", http.StatusText(resp.StatusCode), err)
 	}
+
 	return Errorf("%s: %s", http.StatusText(resp.StatusCode), errMsg.String())
 }
 

cmd/crypto/retry.go

@@ -0,0 +1,65 @@
+// MinIO Cloud Storage, (C) 2020 MinIO, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package crypto
+
+import (
+	"math/rand"
+	"time"
+)
+
+// default retry configuration
+const (
+	retryWaitMin = 500 * time.Millisecond // minimum retry limit.
+	retryWaitMax = 3 * time.Second        // 3 secs worth of max retry.
+	retryMax     = 2
+)
+
+// LinearJitterBackoff provides the time.Duration for a caller to
+// perform linear backoff based on the attempt number and with jitter to
+// prevent a thundering herd.
+//
+// min and max here are *not* absolute values. The number to be multiplied by
+// the attempt number will be chosen at random from between them, thus they are
+// bounding the jitter.
+//
+// For instance:
+// * To get strictly linear backoff of one second increasing each retry, set
+// both to one second (1s, 2s, 3s, 4s, ...)
+// * To get a small amount of jitter centered around one second increasing each
+// retry, set to around one second, such as a min of 800ms and max of 1200ms
+// (892ms, 2102ms, 2945ms, 4312ms, ...)
+// * To get extreme jitter, set to a very wide spread, such as a min of 100ms
+// and a max of 20s (15382ms, 292ms, 51321ms, 35234ms, ...)
+func LinearJitterBackoff(min, max time.Duration, attemptNum int) time.Duration {
+	// attemptNum always starts at zero but we want to start at 1 for multiplication
+	attemptNum++
+
+	if max <= min {
+		// Unclear what to do here, or they are the same, so return min *
+		// attemptNum
+		return min * time.Duration(attemptNum)
+	}
+
+	// Seed rand; doing this every time is fine
+	rand := rand.New(rand.NewSource(int64(time.Now().Nanosecond())))
+
+	// Pick a random number that lies somewhere between the min and max and
+	// multiply by the attemptNum. attemptNum starts at zero so we always
+	// increment here. We first get a random percentage, then apply that to the
+	// difference between min and max, and add to min.
+	jitter := rand.Float64() * float64(max-min)
+	jitterMin := int64(jitter) + int64(min)
+	return time.Duration(jitterMin * int64(attemptNum))
+}

cmd/data-update-tracker.go

@@ -352,7 +352,9 @@ func (d *dataUpdateTracker) deserialize(src io.Reader, newerThan time.Time) erro
 	// Version
 	if _, err := io.ReadFull(src, tmp[:1]); err != nil {
 		if d.debug {
-			logger.LogIf(ctx, err)
+			if err != io.EOF {
+				logger.LogIf(ctx, err)
+			}
 		}
 		return err
 	}

cmd/disk-cache-backend.go

@@ -29,6 +29,7 @@ import (
 	"os"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/djherbis/atime"
@@ -37,7 +38,6 @@ import (
 	"github.com/minio/minio/cmd/logger"
 	"github.com/minio/minio/pkg/disk"
 	"github.com/minio/sio"
-	"go.uber.org/atomic"
 )
 
 const (
@@ -125,17 +125,16 @@ func (m *cacheMeta) ToObjectInfo(bucket, object string) (o ObjectInfo) {
 
 // represents disk cache struct
 type diskCache struct {
-	dir      string // caching directory
-	quotaPct int    // max usage in %
-	// mark false if drive is offline
-	online bool
-	// mutex to protect updates to online variable
-	onlineMutex   *sync.RWMutex
+	gcCounter uint64 // ref: https://golang.org/pkg/sync/atomic/#pkg-note-BUG
+	// is set to 0 if drive is offline
+	online uint32
+
+	dir           string // caching directory
+	quotaPct      int    // max usage in %
 	pool          sync.Pool
 	after         int // minimum accesses before an object is cached.
 	lowWatermark  int
 	highWatermark int
-	gcCounter     atomic.Uint64
 	// nsMutex namespace lock
 	nsMutex *nsLockMap
 	// Object functions pointing to the corresponding functions of backend implementation.
@@ -153,8 +152,7 @@ func newDiskCache(dir string, quotaPct, after, lowWatermark, highWatermark int)
 		after:         after,
 		lowWatermark:  lowWatermark,
 		highWatermark: highWatermark,
-		online:        true,
-		onlineMutex:   &sync.RWMutex{},
+		online:        1,
 		pool: sync.Pool{
 			New: func() interface{} {
 				b := disk.AlignedBlock(int(cacheBlkSize))
@@ -340,27 +338,25 @@ func (c *diskCache) purge(ctx context.Context) {
 }
 
 func (c *diskCache) incGCCounter() {
-	c.gcCounter.Add(uint64(1))
+	atomic.AddUint64(&c.gcCounter, 1)
 }
+
 func (c *diskCache) resetGCCounter() {
-	c.gcCounter.Store(uint64(0))
+	atomic.StoreUint64(&c.gcCounter, 0)
 }
+
 func (c *diskCache) gcCount() uint64 {
-	return c.gcCounter.Load()
+	return atomic.LoadUint64(&c.gcCounter)
 }
 
 // sets cache drive status
-func (c *diskCache) setOnline(status bool) {
-	c.onlineMutex.Lock()
-	c.online = status
-	c.onlineMutex.Unlock()
+func (c *diskCache) setOffline() {
+	atomic.StoreUint32(&c.online, 0)
 }
 
 // returns true if cache drive is online
 func (c *diskCache) IsOnline() bool {
-	c.onlineMutex.RLock()
-	defer c.onlineMutex.RUnlock()
-	return c.online
+	return atomic.LoadUint32(&c.online) != 0
 }
 
 // Stat returns ObjectInfo from disk cache
@@ -680,7 +676,8 @@ func (c *diskCache) Put(ctx context.Context, bucket, object string, data io.Read
 	}
 	n, err := c.bitrotWriteToCache(cachePath, cacheDataFile, reader, actualSize)
 	if IsErr(err, baseErrs...) {
-		c.setOnline(false)
+		// take the cache drive offline
+		c.setOffline()
 	}
 	if err != nil {
 		removeAll(cachePath)
@@ -727,7 +724,8 @@ func (c *diskCache) putRange(ctx context.Context, bucket, object string, data io
 	cacheFile := MustGetUUID()
 	n, err := c.bitrotWriteToCache(cachePath, cacheFile, reader, actualSize)
 	if IsErr(err, baseErrs...) {
-		c.setOnline(false)
+		// take the cache drive offline
+		c.setOffline()
 	}
 	if err != nil {
 		removeAll(cachePath)

cmd/disk-cache-stats.go

@@ -16,46 +16,44 @@
 
 package cmd
 
-import (
-	"go.uber.org/atomic"
-)
+import "sync/atomic"
 
 // CacheStats - represents bytes served from cache,
 // cache hits and cache misses.
 type CacheStats struct {
-	BytesServed atomic.Uint64
-	Hits        atomic.Uint64
-	Misses      atomic.Uint64
+	BytesServed uint64
+	Hits        uint64
+	Misses      uint64
 }
 
 // Increase total bytes served from cache
 func (s *CacheStats) incBytesServed(n int64) {
-	s.BytesServed.Add(uint64(n))
+	atomic.AddUint64(&s.BytesServed, uint64(n))
 }
 
 // Increase cache hit by 1
 func (s *CacheStats) incHit() {
-	s.Hits.Add(uint64(1))
+	atomic.AddUint64(&s.Hits, 1)
 }
 
 // Increase cache miss by 1
 func (s *CacheStats) incMiss() {
-	s.Misses.Add(uint64(1))
+	atomic.AddUint64(&s.Misses, 1)
 }
 
 // Get total bytes served
 func (s *CacheStats) getBytesServed() uint64 {
-	return s.BytesServed.Load()
+	return atomic.LoadUint64(&s.BytesServed)
 }
 
 // Get total cache hits
 func (s *CacheStats) getHits() uint64 {
-	return s.Hits.Load()
+	return atomic.LoadUint64(&s.Hits)
 }
 
 // Get total cache misses
 func (s *CacheStats) getMisses() uint64 {
-	return s.Misses.Load()
+	return atomic.LoadUint64(&s.Misses)
 }
 
 // Prepare new CacheStats structure

cmd/disk-cache_test.go

@@ -78,7 +78,7 @@ func TestGetCachedLoc(t *testing.T) {
 		// find cache drive where object would be hashed
 		index := c.hashIndex(bucketName, objectName)
 		// turn off drive by setting online status to false
-		c.cache[index].online = false
+		c.cache[index].setOffline()
 		cfs, err := c.getCacheLoc(bucketName, objectName)
 		if n == 1 && err == errDiskNotFound {
 			continue
@@ -118,7 +118,7 @@ func TestGetCacheMaxUse(t *testing.T) {
 		// find cache drive where object would be hashed
 		index := c.hashIndex(bucketName, objectName)
 		// turn off drive by setting online status to false
-		c.cache[index].online = false
+		c.cache[index].setOffline()
 		cb, err := c.getCacheLoc(bucketName, objectName)
 		if n == 1 && err == errDiskNotFound {
 			continue

cmd/dummy-handlers.go

@@ -21,7 +21,6 @@ import (
 	"net/http"
 
 	"github.com/gorilla/mux"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 )
 
@@ -59,12 +58,6 @@ func (api objectAPIHandlers) GetBucketReplicationHandler(w http.ResponseWriter,
 	w.(http.Flusher).Flush()
 }
 
-// DeleteBucketTaggingHandler - DELETE bucket tagging, a dummy api
-func (api objectAPIHandlers) DeleteBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
-	writeSuccessResponseHeadersOnly(w)
-	w.(http.Flusher).Flush()
-}
-
 // DeleteBucketWebsiteHandler - DELETE bucket website, a dummy api
 func (api objectAPIHandlers) DeleteBucketWebsiteHandler(w http.ResponseWriter, r *http.Request) {
 	writeSuccessResponseHeadersOnly(w)
@@ -130,41 +123,3 @@ func (api objectAPIHandlers) GetBucketCorsHandler(w http.ResponseWriter, r *http
 
 	w.(http.Flusher).Flush()
 }
-
-// GetBucketTaggingHandler - GET bucket tagging, a dummy api
-func (api objectAPIHandlers) GetBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
-	ctx := newContext(r, w, "GetBucketTagging")
-
-	vars := mux.Vars(r)
-	bucket := vars["bucket"]
-
-	objAPI := api.ObjectAPI()
-	if objAPI == nil {
-		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
-		return
-	}
-
-	// Allow getBucketTagging if policy action is set, since this is a dummy call
-	// we are simply re-purposing the bucketPolicyAction.
-	if s3Error := checkRequestAuthType(ctx, r, policy.GetBucketPolicyAction, bucket, ""); s3Error != ErrNone {
-		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
-		return
-	}
-
-	// Validate if bucket exists, before proceeding further...
-	_, err := objAPI.GetBucketInfo(ctx, bucket)
-	if err != nil {
-		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
-		return
-	}
-
-	tags := &tagging.Tagging{}
-	tags.TagSet.Tags = append(tags.TagSet.Tags, tagging.Tag{})
-
-	if err := xml.NewEncoder(w).Encode(tags); err != nil {
-		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
-		return
-	}
-
-	w.(http.Flusher).Flush()
-}

cmd/encryption-v1.go

@@ -247,22 +247,6 @@ func EncryptRequest(content io.Reader, r *http.Request, bucket, object string, m
 	return newEncryptReader(content, key, bucket, object, metadata, crypto.S3.IsRequested(r.Header))
 }
 
-// DecryptCopyRequest decrypts the object with the client provided key. It also removes
-// the client-side-encryption metadata from the object and sets the correct headers.
-func DecryptCopyRequest(client io.Writer, r *http.Request, bucket, object string, metadata map[string]string) (io.WriteCloser, error) {
-	var (
-		key []byte
-		err error
-	)
-	if crypto.SSECopy.IsRequested(r.Header) {
-		key, err = ParseSSECopyCustomerRequest(r.Header, metadata)
-		if err != nil {
-			return nil, err
-		}
-	}
-	return newDecryptWriter(client, key, bucket, object, 0, metadata)
-}
-
 func decryptObjectInfo(key []byte, bucket, object string, metadata map[string]string) ([]byte, error) {
 	switch {
 	default:
@@ -317,14 +301,6 @@ func decryptObjectInfo(key []byte, bucket, object string, metadata map[string]st
 	}
 }
 
-func newDecryptWriter(client io.Writer, key []byte, bucket, object string, seqNumber uint32, metadata map[string]string) (io.WriteCloser, error) {
-	objectEncryptionKey, err := decryptObjectInfo(key, bucket, object, metadata)
-	if err != nil {
-		return nil, err
-	}
-	return newDecryptWriterWithObjectKey(client, objectEncryptionKey, seqNumber, metadata)
-}
-
 func newDecryptWriterWithObjectKey(client io.Writer, objectEncryptionKey []byte, seqNumber uint32, metadata map[string]string) (io.WriteCloser, error) {
 	writer, err := sio.DecryptWriter(client, sio.Config{
 		Key:            objectEncryptionKey,
@@ -450,26 +426,6 @@ func DecryptBlocksRequestR(inputReader io.Reader, h http.Header, offset,
 	return w, nil
 }
 
-// DecryptRequestWithSequenceNumber decrypts the object with the client provided key. It also removes
-// the client-side-encryption metadata from the object and sets the correct headers.
-func DecryptRequestWithSequenceNumber(client io.Writer, r *http.Request, bucket, object string, seqNumber uint32, metadata map[string]string) (io.WriteCloser, error) {
-	if crypto.S3.IsEncrypted(metadata) {
-		return newDecryptWriter(client, nil, bucket, object, seqNumber, metadata)
-	}
-
-	key, err := ParseSSECustomerRequest(r)
-	if err != nil {
-		return nil, err
-	}
-	return newDecryptWriter(client, key, bucket, object, seqNumber, metadata)
-}
-
-// DecryptRequest decrypts the object with client provided key for SSE-C and SSE-S3. It also removes
-// the encryption metadata from the object and sets the correct headers.
-func DecryptRequest(client io.Writer, r *http.Request, bucket, object string, metadata map[string]string) (io.WriteCloser, error) {
-	return DecryptRequestWithSequenceNumber(client, r, bucket, object, 0, metadata)
-}
-
 // DecryptBlocksReader - decrypts multipart parts, while implementing
 // a io.Reader compatible interface.
 type DecryptBlocksReader struct {

cmd/encryption-v1_test.go

@@ -78,135 +78,6 @@ func TestEncryptRequest(t *testing.T) {
 	}
 }
 
-var decryptRequestTests = []struct {
-	bucket, object string
-	header         map[string]string
-	metadata       map[string]string
-	shouldFail     bool
-}{
-	{
-		bucket: "bucket",
-		object: "object",
-		header: map[string]string{
-			crypto.SSECAlgorithm: "AES256",
-			crypto.SSECKey:       "MzJieXRlc2xvbmdzZWNyZXRrZXltdXN0cHJvdmlkZWQ=",
-			crypto.SSECKeyMD5:    "7PpPLAK26ONlVUGOWlusfg==",
-		},
-		metadata: map[string]string{
-			crypto.SSESealAlgorithm: crypto.InsecureSealAlgorithm,
-			crypto.SSEIV:            "7nQqotA8xgrPx6QK7Ap3GCfjKitqJSrGP7xzgErSJlw=",
-			crypto.SSECSealedKey:    "EAAfAAAAAAD7v1hQq3PFRUHsItalxmrJqrOq6FwnbXNarxOOpb8jTWONPPKyM3Gfjkjyj6NCf+aB/VpHCLCTBA==",
-		},
-		shouldFail: false,
-	},
-	{
-		bucket: "bucket",
-		object: "object",
-		header: map[string]string{
-			crypto.SSECAlgorithm: "AES256",
-			crypto.SSECKey:       "MzJieXRlc2xvbmdzZWNyZXRrZXltdXN0cHJvdmlkZWQ=",
-			crypto.SSECKeyMD5:    "7PpPLAK26ONlVUGOWlusfg==",
-		},
-		metadata: map[string]string{
-			crypto.SSESealAlgorithm: crypto.SealAlgorithm,
-			crypto.SSEIV:            "qEqmsONcorqlcZXJxaw32H04eyXyXwUgjHzlhkaIYrU=",
-			crypto.SSECSealedKey:    "IAAfAIM14ugTGcM/dIrn4iQMrkl1sjKyeBQ8FBEvRebYj8vWvxG+0cJRpC6NXRU1wJN50JaUOATjO7kz0wZ2mA==",
-		},
-		shouldFail: false,
-	},
-	{
-		bucket: "bucket",
-		object: "object",
-		header: map[string]string{
-			crypto.SSECAlgorithm: "AES256",
-			crypto.SSECKey:       "XAm0dRrJsEsyPb1UuFNezv1bl9hxuYsgUVC/MUctE2k=",
-			crypto.SSECKeyMD5:    "bY4wkxQejw9mUJfo72k53A==",
-		},
-		metadata: map[string]string{
-			crypto.SSESealAlgorithm: "HMAC-SHA3",
-			crypto.SSEIV:            "XAm0dRrJsEsyPb1UuFNezv1bl9hxuYsgUVC/MUctE2k=",
-			crypto.SSECSealedKey:    "SY5E9AvI2tI7/nUrUAssIGE32Hcs4rR9z/CUuPqu5N4=",
-		},
-		shouldFail: true,
-	},
-	{
-		bucket: "bucket",
-		object: "object",
-		header: map[string]string{
-			crypto.SSECAlgorithm: "AES256",
-			crypto.SSECKey:       "XAm0dRrJsEsyPb1UuFNezv1bl9hxuYsgUVC/MUctE2k=",
-			crypto.SSECKeyMD5:    "bY4wkxQejw9mUJfo72k53A==",
-		},
-		metadata: map[string]string{
-			crypto.SSESealAlgorithm: crypto.InsecureSealAlgorithm,
-			crypto.SSEIV:            "RrJsEsyPb1UuFNezv1bl9hxuYsgUVC/MUctE2k=",
-			crypto.SSECSealedKey:    "SY5E9AvI2tI7/nUrUAssIGE32Hcs4rR9z/CUuPqu5N4=",
-		},
-		shouldFail: true,
-	},
-	{
-		bucket: "bucket",
-		object: "object",
-		header: map[string]string{
-			crypto.SSECAlgorithm: "AES256",
-			crypto.SSECKey:       "XAm0dRrJsEsyPb1UuFNezv1bl9hxuYsgUVC/MUctE2k=",
-			crypto.SSECKeyMD5:    "bY4wkxQejw9mUJfo72k53A==",
-		},
-		metadata: map[string]string{
-			crypto.SSESealAlgorithm: crypto.InsecureSealAlgorithm,
-			crypto.SSEIV:            "XAm0dRrJsEsyPb1UuFNezv1bl9ehxuYsgUVC/MUctE2k=",
-			crypto.SSECSealedKey:    "SY5E9AvI2tI7/nUrUAssIGE32Hds4rR9z/CUuPqu5N4=",
-		},
-		shouldFail: true,
-	},
-	{
-		bucket: "bucket",
-		object: "object-2",
-		header: map[string]string{
-			crypto.SSECAlgorithm: "AES256",
-			crypto.SSECKey:       "MzJieXRlc2xvbmdzZWNyZXRrZXltdXN0cHJvdmlkZWQ=",
-			crypto.SSECKeyMD5:    "7PpPLAK26ONlVUGOWlusfg==",
-		},
-		metadata: map[string]string{
-			crypto.SSESealAlgorithm: crypto.SealAlgorithm,
-			crypto.SSEIV:            "qEqmsONcorqlcZXJxaw32H04eyXyXwUgjHzlhkaIYrU=",
-			crypto.SSECSealedKey:    "IAAfAIM14ugTGcM/dIrn4iQMrkl1sjKyeBQ8FBEvRebYj8vWvxG+0cJRpC6NXRU1wJN50JaUOATjO7kz0wZ2mA==",
-		},
-		shouldFail: true,
-	},
-}
-
-func TestDecryptRequest(t *testing.T) {
-	defer func(flag bool) { globalIsSSL = flag }(globalIsSSL)
-	globalIsSSL = true
-	for i, test := range decryptRequestTests[1:] {
-		client := bytes.NewBuffer(nil)
-		req := &http.Request{Header: http.Header{}}
-		for k, v := range test.header {
-			req.Header.Set(k, v)
-		}
-		_, err := DecryptRequest(client, req, test.bucket, test.object, test.metadata)
-		if err != nil && !test.shouldFail {
-			t.Fatalf("Test %d: Failed to encrypt request: %v", i, err)
-		}
-		if err == nil && test.shouldFail {
-			t.Fatalf("Test %d: should fail but passed", i)
-		}
-		if key, ok := test.metadata[crypto.SSECKey]; ok {
-			t.Errorf("Test %d: Client provided key survived in metadata - key: %s", i, key)
-		}
-		if kdf, ok := test.metadata[crypto.SSESealAlgorithm]; ok && !test.shouldFail {
-			t.Errorf("Test %d: ServerSideEncryptionKDF should not be part of metadata: %v", i, kdf)
-		}
-		if iv, ok := test.metadata[crypto.SSEIV]; ok && !test.shouldFail {
-			t.Errorf("Test %d: crypto.SSEIV should not be part of metadata: %v", i, iv)
-		}
-		if mac, ok := test.metadata[crypto.SSECSealedKey]; ok && !test.shouldFail {
-			t.Errorf("Test %d: ServerSideEncryptionKeyMAC should not be part of metadata: %v", i, mac)
-		}
-	}
-}
-
 var decryptObjectInfoTests = []struct {
 	info    ObjectInfo
 	headers http.Header

cmd/format-fs.go

@@ -190,7 +190,7 @@ func initFormatFS(ctx context.Context, fsPath string) (rlk *lock.RLockedFile, er
 	fsFormatPath := pathJoin(fsPath, minioMetaBucket, formatConfigFile)
 
 	// Add a deployment ID, if it does not exist.
-	if err := formatFSFixDeploymentID(fsFormatPath); err != nil {
+	if err := formatFSFixDeploymentID(ctx, fsFormatPath); err != nil {
 		return nil, err
 	}
 
@@ -288,7 +288,7 @@ func formatFSGetDeploymentID(rlk *lock.RLockedFile) (id string, err error) {
 }
 
 // Generate a deployment ID if one does not exist already.
-func formatFSFixDeploymentID(fsFormatPath string) error {
+func formatFSFixDeploymentID(ctx context.Context, fsFormatPath string) error {
 	rlk, err := lock.RLockedOpenFile(fsFormatPath)
 	if err == nil {
 		// format.json can be empty in a rare condition when another
@@ -339,11 +339,12 @@ func formatFSFixDeploymentID(fsFormatPath string) error {
 		return time.Now().Round(time.Second).Sub(formatStartTime).String()
 	}
 
-	doneCh := make(chan struct{})
-	defer close(doneCh)
+	retryCtx, cancel := context.WithCancel(ctx)
+	// Indicate to our routine to exit cleanly upon return.
+	defer cancel()
 
 	var wlk *lock.LockedFile
-	retryCh := newRetryTimerSimple(doneCh)
+	retryCh := newRetryTimerSimple(retryCtx)
 	var stop bool
 	for !stop {
 		select {
@@ -358,7 +359,7 @@ func formatFSFixDeploymentID(fsFormatPath string) error {
 				return err
 			}
 			stop = true
-		case <-globalOSSignalCh:
+		case <-ctx.Done():
 			return fmt.Errorf("Initializing FS format stopped gracefully")
 		}
 	}

cmd/fs-v1.go

@@ -35,12 +35,12 @@ import (
 
 	jsoniter "github.com/json-iterator/go"
 	"github.com/minio/minio-go/v6/pkg/s3utils"
+	"github.com/minio/minio-go/v6/pkg/tags"
 	"github.com/minio/minio/cmd/config"
 	xhttp "github.com/minio/minio/cmd/http"
 	"github.com/minio/minio/cmd/logger"
 	bucketsse "github.com/minio/minio/pkg/bucket/encryption"
 	"github.com/minio/minio/pkg/bucket/lifecycle"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 	"github.com/minio/minio/pkg/color"
 	"github.com/minio/minio/pkg/lock"
@@ -131,8 +131,12 @@ func NewFSObjectLayer(fsPath string) (ObjectLayer, error) {
 	var err error
 	if fsPath, err = getValidPath(fsPath); err != nil {
 		if err == errMinDiskSize {
-			return nil, err
+			return nil, config.ErrUnableToWriteInBackend(err).Hint(err.Error())
+		} else if err == errUnsupportedDisk {
+			hint := fmt.Sprintf("'%s' does not support O_DIRECT flags, refusing to use", fsPath)
+			return nil, config.ErrUnsupportedBackend(err).Hint(hint)
 		}
+
 		// Show a descriptive error with a hint about how to fix it.
 		var username string
 		if u, err := user.Current(); err == nil {
@@ -1219,18 +1223,13 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
 }
 
 // GetObjectTag - get object tags from an existing object
-func (fs *FSObjects) GetObjectTag(ctx context.Context, bucket, object string) (tagging.Tagging, error) {
+func (fs *FSObjects) GetObjectTag(ctx context.Context, bucket, object string) (*tags.Tags, error) {
 	oi, err := fs.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
 	if err != nil {
-		return tagging.Tagging{}, err
+		return nil, err
 	}
 
-	tags, err := tagging.FromString(oi.UserTags)
-	if err != nil {
-		return tagging.Tagging{}, err
-	}
-
-	return tags, nil
+	return tags.ParseObjectTags(oi.UserTags)
 }
 
 // PutObjectTag - replace or add tags to an existing object

cmd/gateway-metrics.go

@@ -18,57 +18,56 @@ package cmd
 
 import (
 	"net/http"
-
-	"go.uber.org/atomic"
+	"sync/atomic"
 )
 
 // RequestStats - counts for Get and Head requests
 type RequestStats struct {
-	Get  atomic.Uint64 `json:"Get"`
-	Head atomic.Uint64 `json:"Head"`
-	Put  atomic.Uint64 `json:"Put"`
-	Post atomic.Uint64 `json:"Post"`
+	Get  uint64 `json:"Get"`
+	Head uint64 `json:"Head"`
+	Put  uint64 `json:"Put"`
+	Post uint64 `json:"Post"`
 }
 
 // Metrics - represents bytes served from backend
 // only implemented for S3 Gateway
 type Metrics struct {
-	bytesReceived atomic.Uint64
-	bytesSent     atomic.Uint64
+	bytesReceived uint64
+	bytesSent     uint64
 	requestStats  RequestStats
 }
 
 // IncBytesReceived - Increase total bytes received from gateway backend
 func (s *Metrics) IncBytesReceived(n uint64) {
-	s.bytesReceived.Add(n)
+	atomic.AddUint64(&s.bytesReceived, n)
 }
 
 // GetBytesReceived - Get total bytes received from gateway backend
 func (s *Metrics) GetBytesReceived() uint64 {
-	return s.bytesReceived.Load()
+	return atomic.LoadUint64(&s.bytesReceived)
 }
 
 // IncBytesSent - Increase total bytes sent to gateway backend
 func (s *Metrics) IncBytesSent(n uint64) {
-	s.bytesSent.Add(n)
+	atomic.AddUint64(&s.bytesSent, n)
 }
 
 // GetBytesSent - Get total bytes received from gateway backend
 func (s *Metrics) GetBytesSent() uint64 {
-	return s.bytesSent.Load()
+	return atomic.LoadUint64(&s.bytesSent)
 }
 
 // IncRequests - Increase request count sent to gateway backend by 1
 func (s *Metrics) IncRequests(method string) {
 	// Only increment for Head & Get requests, else no op
 	if method == http.MethodGet {
-		s.requestStats.Get.Add(1)
+		atomic.AddUint64(&s.requestStats.Get, 1)
 	} else if method == http.MethodHead {
-		s.requestStats.Head.Add(1)
+		atomic.AddUint64(&s.requestStats.Head, 1)
 	} else if method == http.MethodPut {
-		s.requestStats.Put.Add(1)
+		atomic.AddUint64(&s.requestStats.Put, 1)
 	} else if method == http.MethodPost {
-		s.requestStats.Post.Add(1)
+		atomic.AddUint64(&s.requestStats.Post, 1)
 	}
 }
 

cmd/gateway-unsupported.go

@@ -22,9 +22,9 @@ import (
 
 	"github.com/minio/minio/cmd/logger"
 
+	"github.com/minio/minio-go/v6/pkg/tags"
 	bucketsse "github.com/minio/minio/pkg/bucket/encryption"
 	"github.com/minio/minio/pkg/bucket/lifecycle"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 
 	"github.com/minio/minio/pkg/madmin"
@@ -205,9 +205,9 @@ func (a GatewayUnsupported) PutObjectTag(ctx context.Context, bucket, object str
 }
 
 // GetObjectTag - not implemented.
-func (a GatewayUnsupported) GetObjectTag(ctx context.Context, bucket, object string) (tagging.Tagging, error) {
+func (a GatewayUnsupported) GetObjectTag(ctx context.Context, bucket, object string) (*tags.Tags, error) {
 	logger.LogIf(ctx, NotImplemented{})
-	return tagging.Tagging{}, NotImplemented{}
+	return nil, NotImplemented{}
 }
 
 // DeleteObjectTag - not implemented.

cmd/gateway/b2/gateway-b2_test.go

@@ -1,146 +0,0 @@
-/*
- * MinIO Cloud Storage, (C) 2017 MinIO, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package b2
-
-import (
-	"fmt"
-	"testing"
-
-	minio "github.com/minio/minio/cmd"
-)
-
-// Test b2 object error.
-func TestB2ObjectError(t *testing.T) {
-	testCases := []struct {
-		params      []string
-		b2Err       error
-		expectedErr error
-	}{
-		{
-			[]string{}, nil, nil,
-		},
-		{
-			[]string{}, fmt.Errorf("Not *Error"), fmt.Errorf("Not *Error"),
-		},
-		{
-			[]string{}, fmt.Errorf("Non B2 Error"), fmt.Errorf("Non B2 Error"),
-		},
-	}
-
-	for i, testCase := range testCases {
-		actualErr := b2ToObjectError(testCase.b2Err, testCase.params...)
-		if actualErr != nil {
-			if actualErr.Error() != testCase.expectedErr.Error() {
-				t.Errorf("Test %d: Expected %s, got %s", i+1, testCase.expectedErr, actualErr)
-			}
-		}
-	}
-}
-
-// Test b2 msg code to object error.
-func TestB2MsgCodeToObjectError(t *testing.T) {
-	testCases := []struct {
-		params      []string
-		code        int
-		msgCode     string
-		msg         string
-		expectedErr error
-	}{
-		{
-			[]string{"bucket"},
-			1,
-			"duplicate_bucket_name",
-			"",
-			minio.BucketAlreadyOwnedByYou{
-				Bucket: "bucket",
-			},
-		},
-		{
-			[]string{"bucket"},
-			1,
-			"bad_request",
-			"",
-			minio.BucketNotFound{
-				Bucket: "bucket",
-			},
-		},
-		{
-			[]string{"bucket", "object"},
-			1,
-			"bad_request",
-			"",
-			minio.ObjectNameInvalid{
-				Bucket: "bucket",
-				Object: "object",
-			},
-		},
-		{
-			[]string{"bucket"},
-			1,
-			"bad_bucket_id",
-			"",
-			minio.BucketNotFound{Bucket: "bucket"},
-		},
-		{
-			[]string{"bucket", "object"},
-			1,
-			"file_not_present",
-			"",
-			minio.ObjectNotFound{
-				Bucket: "bucket",
-				Object: "object",
-			},
-		},
-		{
-			[]string{"bucket", "object"},
-			1,
-			"not_found",
-			"",
-			minio.ObjectNotFound{
-				Bucket: "bucket",
-				Object: "object",
-			},
-		},
-		{
-			[]string{"bucket"},
-			1,
-			"cannot_delete_non_empty_bucket",
-			"",
-			minio.BucketNotEmpty{
-				Bucket: "bucket",
-			},
-		},
-		{
-			[]string{"bucket", "object", "uploadID"},
-			1,
-			"",
-			"No active upload for",
-			minio.InvalidUploadID{
-				UploadID: "uploadID",
-			},
-		},
-	}
-
-	for i, testCase := range testCases {
-		actualErr := b2MsgCodeToObjectError(testCase.code, testCase.msgCode, testCase.msg, testCase.params...)
-		if actualErr != nil {
-			if actualErr.Error() != testCase.expectedErr.Error() {
-				t.Errorf("Test %d: Expected %s, got %s", i+1, testCase.expectedErr, actualErr)
-			}
-		}
-	}
-}

cmd/gateway/gateway.go

@@ -1,5 +1,5 @@
 /*
- * MinIO Cloud Storage, (C) 2017 MinIO, Inc.
+ * MinIO Cloud Storage, (C) 2017-2020 MinIO, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -17,15 +17,23 @@
 package gateway
 
 import (
-	// Import all gateways.
-	_ "github.com/minio/minio/cmd/gateway/azure"
-	_ "github.com/minio/minio/cmd/gateway/gcs"
-	_ "github.com/minio/minio/cmd/gateway/hdfs"
+	// Import all gateways please keep the order
+
+	// NAS
 	_ "github.com/minio/minio/cmd/gateway/nas"
+
+	// Azure
+	_ "github.com/minio/minio/cmd/gateway/azure"
+
+	// S3
 	_ "github.com/minio/minio/cmd/gateway/s3"
 
-	// B2 is specifically kept here to avoid re-ordering by goimports,
-	// please ask on github.com/minio/minio/issues before changing this.
-	_ "github.com/minio/minio/cmd/gateway/b2"
-	// Add your gateway here.
+	// HDFS
+	_ "github.com/minio/minio/cmd/gateway/hdfs"
+
+	// GCS (use only if you must, GCS already supports S3 API)
+	_ "github.com/minio/minio/cmd/gateway/gcs"
+	// gateway functionality is frozen, no new gateways are being implemented
+	// or considered for upstream inclusion at this point in time. if needed
+	// please keep a fork of the project.
 )

cmd/generic-handlers.go

@@ -491,7 +491,6 @@ var notImplementedBucketResourceNames = map[string]bool{
 	"metrics":        true,
 	"replication":    true,
 	"requestPayment": true,
-	"tagging":        true,
 	"versioning":     true,
 	"website":        true,
 }
@@ -680,9 +679,12 @@ func (f bucketForwardingHandler) ServeHTTP(w http.ResponseWriter, r *http.Reques
 
 	bucket, object := request2BucketObjectName(r)
 
-	// ListBucket requests should be handled at current endpoint as
-	// all buckets data can be fetched from here.
-	if r.Method == http.MethodGet && bucket == "" && object == "" {
+	// Requests in federated setups for STS type calls which are
+	// performed at '/' resource should be routed by the muxer,
+	// the assumption is simply such that requests without a bucket
+	// in a federated setup cannot be proxied, so serve them at
+	// current server.
+	if bucket == "" {
 		f.handler.ServeHTTP(w, r)
 		return
 	}

cmd/global-heal.go

@@ -18,7 +18,6 @@ package cmd
 
 import (
 	"context"
-	"sync"
 	"time"
 
 	"github.com/minio/minio/cmd/logger"
@@ -38,7 +37,7 @@ var leaderLockTimeout = newDynamicTimeout(time.Minute, time.Minute)
 func newBgHealSequence(numDisks int) *healSequence {
 
 	reqInfo := &logger.ReqInfo{API: "BackgroundHeal"}
-	ctx := logger.SetReqInfo(GlobalContext, reqInfo)
+	ctx, cancelCtx := context.WithCancel(logger.SetReqInfo(GlobalContext, reqInfo))
 
 	hs := madmin.HealOpts{
 		// Remove objects that do not have read-quorum
@@ -56,15 +55,13 @@ func newBgHealSequence(numDisks int) *healSequence {
 			Summary:      healNotStartedStatus,
 			HealSettings: hs,
 			NumDisks:     numDisks,
-			updateLock:   &sync.RWMutex{},
 		},
-		traverseAndHealDoneCh: make(chan error),
-		stopSignalCh:          make(chan struct{}),
-		ctx:                   ctx,
-		reportProgress:        false,
-		scannedItemsMap:       make(map[madmin.HealItemType]int64),
-		healedItemsMap:        make(map[madmin.HealItemType]int64),
-		healFailedItemsMap:    make(map[string]int64),
+		cancelCtx:          cancelCtx,
+		ctx:                ctx,
+		reportProgress:     false,
+		scannedItemsMap:    make(map[madmin.HealItemType]int64),
+		healedItemsMap:     make(map[madmin.HealItemType]int64),
+		healFailedItemsMap: make(map[string]int64),
 	}
 }
 
@@ -96,7 +93,12 @@ func healErasureSet(ctx context.Context, setIndex int, xlObj *xlObjects) error {
 		if ok {
 			break
 		}
-		time.Sleep(time.Second)
+		select {
+		case <-ctx.Done():
+			return nil
+		case <-time.After(time.Second):
+			continue
+		}
 	}
 
 	// Heal all buckets with all objects
@@ -155,7 +157,12 @@ func execLeaderTasks(ctx context.Context, z *xlZones) {
 		if ok {
 			break
 		}
-		time.Sleep(time.Second)
+		select {
+		case <-ctx.Done():
+			return
+		case <-time.After(time.Second):
+			continue
+		}
 	}
 	for {
 		select {

cmd/globals.go

@@ -218,7 +218,7 @@ var (
 
 	globalBucketObjectLockConfig = objectlock.NewBucketObjectLockConfig()
 
-	globalBucketQuotaSys     = NewBucketQuotaSys()
+	globalBucketQuotaSys     *BucketQuotaSys
 	globalBucketStorageCache bucketStorageCache
 
 	// Disk cache drives
@@ -291,6 +291,7 @@ var (
 func getGlobalInfo() (globalInfo map[string]interface{}) {
 	globalInfo = map[string]interface{}{
 		"serverRegion": globalServerRegion,
+		"domains":      globalDomainNames,
 		// Add more relevant global settings here.
 	}
 

cmd/handler-utils.go

@@ -33,7 +33,6 @@ import (
 	xhttp "github.com/minio/minio/cmd/http"
 	"github.com/minio/minio/cmd/logger"
 	"github.com/minio/minio/pkg/auth"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/handlers"
 	"github.com/minio/minio/pkg/madmin"
 )
@@ -161,26 +160,6 @@ func extractMetadataFromMap(ctx context.Context, v map[string][]string, m map[st
 	return nil
 }
 
-// extractTags extracts tag key and value from given http header. It then
-// - Parses the input format X-Amz-Tagging:"Key1=Value1&Key2=Value2" into a map[string]string
-// with entries in the format X-Amg-Tag-Key1:Value1, X-Amz-Tag-Key2:Value2
-// - Validates the tags
-// - Returns the Tag in original string format "Key1=Value1&Key2=Value2"
-func extractTags(ctx context.Context, tags string) (string, error) {
-	// Check if the metadata has tagging related header
-	if tags != "" {
-		tagging, err := tagging.FromString(tags)
-		if err != nil {
-			return "", err
-		}
-		if err := tagging.Validate(); err != nil {
-			return "", err
-		}
-		return tagging.String(), nil
-	}
-	return "", nil
-}
-
 // The Query string for the redirect URL the client is
 // redirected on successful upload.
 func getRedirectPostRawQuery(objInfo ObjectInfo) string {

cmd/http-stats.go

@@ -21,61 +21,61 @@ import (
 	"net/http"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/minio/minio/cmd/logger"
 	"github.com/prometheus/client_golang/prometheus"
-	"go.uber.org/atomic"
 )
 
 // ConnStats - Network statistics
 // Count total input/output transferred bytes during
 // the server's life.
 type ConnStats struct {
-	totalInputBytes  atomic.Uint64
-	totalOutputBytes atomic.Uint64
-	s3InputBytes     atomic.Uint64
-	s3OutputBytes    atomic.Uint64
+	totalInputBytes  uint64
+	totalOutputBytes uint64
+	s3InputBytes     uint64
+	s3OutputBytes    uint64
 }
 
 // Increase total input bytes
 func (s *ConnStats) incInputBytes(n int) {
-	s.totalInputBytes.Add(uint64(n))
+	atomic.AddUint64(&s.totalInputBytes, uint64(n))
 }
 
 // Increase total output bytes
 func (s *ConnStats) incOutputBytes(n int) {
-	s.totalOutputBytes.Add(uint64(n))
+	atomic.AddUint64(&s.totalOutputBytes, uint64(n))
 }
 
 // Return total input bytes
 func (s *ConnStats) getTotalInputBytes() uint64 {
-	return s.totalInputBytes.Load()
+	return atomic.LoadUint64(&s.totalInputBytes)
 }
 
 // Return total output bytes
 func (s *ConnStats) getTotalOutputBytes() uint64 {
-	return s.totalOutputBytes.Load()
+	return atomic.LoadUint64(&s.totalOutputBytes)
 }
 
 // Increase outbound input bytes
 func (s *ConnStats) incS3InputBytes(n int) {
-	s.s3InputBytes.Add(uint64(n))
+	atomic.AddUint64(&s.s3InputBytes, uint64(n))
 }
 
 // Increase outbound output bytes
 func (s *ConnStats) incS3OutputBytes(n int) {
-	s.s3OutputBytes.Add(uint64(n))
+	atomic.AddUint64(&s.s3OutputBytes, uint64(n))
 }
 
 // Return outbound input bytes
 func (s *ConnStats) getS3InputBytes() uint64 {
-	return s.s3InputBytes.Load()
+	return atomic.LoadUint64(&s.s3InputBytes)
 }
 
 // Return outbound output bytes
 func (s *ConnStats) getS3OutputBytes() uint64 {
-	return s.s3OutputBytes.Load()
+	return atomic.LoadUint64(&s.s3OutputBytes)
 }
 
 // Return connection stats (total input/output bytes and total s3 input/output bytes)
@@ -178,7 +178,7 @@ func (st *HTTPStats) updateStats(api string, r *http.Request, w *logger.Response
 		}
 	}
 
-	if r.Method == "GET" {
+	if r.Method == http.MethodGet {
 		// Increment the prometheus http request response histogram with appropriate label
 		httpRequestsDuration.With(prometheus.Labels{"api": api}).Observe(durationSecs)
 	}

cmd/merge-walk-pool.go

@@ -39,9 +39,9 @@ type mergeWalk struct {
 // mergeWalkPool's purpose is to maintain active mergeWalk go-routines in a map so that
 // it can be looked up across related list calls.
 type MergeWalkPool struct {
+	sync.Mutex
 	pool    map[listParams][]mergeWalk
 	timeOut time.Duration
-	lock    *sync.Mutex
 }
 
 // NewMergeWalkPool - initialize new tree walk pool.
@@ -49,7 +49,6 @@ func NewMergeWalkPool(timeout time.Duration) *MergeWalkPool {
 	tPool := &MergeWalkPool{
 		pool:    make(map[listParams][]mergeWalk),
 		timeOut: timeout,
-		lock:    &sync.Mutex{},
 	}
 	return tPool
 }
@@ -58,9 +57,9 @@ func NewMergeWalkPool(timeout time.Duration) *MergeWalkPool {
 // listParams, removes it from the pool, and returns the MergeWalkResult
 // channel.
 // Returns nil if listParams does not have an asccociated mergeWalk.
-func (t MergeWalkPool) Release(params listParams) ([]FileInfoCh, chan struct{}) {
-	t.lock.Lock()
-	defer t.lock.Unlock()
+func (t *MergeWalkPool) Release(params listParams) ([]FileInfoCh, chan struct{}) {
+	t.Lock()
+	defer t.Unlock()
 	walks, ok := t.pool[params] // Pick the valid walks.
 	if ok {
 		if len(walks) > 0 {
@@ -88,9 +87,9 @@ func (t MergeWalkPool) Release(params listParams) ([]FileInfoCh, chan struct{})
 // 2) Relase() signals the timer go-routine to end on endTimerCh.
 //    During listing the timer should not timeout and end the mergeWalk go-routine, hence the
 //    timer go-routine should be ended.
-func (t MergeWalkPool) Set(params listParams, resultChs []FileInfoCh, endWalkCh chan struct{}) {
-	t.lock.Lock()
-	defer t.lock.Unlock()
+func (t *MergeWalkPool) Set(params listParams, resultChs []FileInfoCh, endWalkCh chan struct{}) {
+	t.Lock()
+	defer t.Unlock()
 
 	// Should be a buffered channel so that Release() never blocks.
 	endTimerCh := make(chan struct{}, 1)
@@ -111,7 +110,7 @@ func (t MergeWalkPool) Set(params listParams, resultChs []FileInfoCh, endWalkCh
 		case <-time.After(t.timeOut):
 			// Timeout has expired. Remove the mergeWalk from mergeWalkPool and
 			// end the mergeWalk go-routine.
-			t.lock.Lock()
+			t.Lock()
 			walks, ok := t.pool[params]
 			if ok {
 				// Trick of filtering without allocating
@@ -135,7 +134,7 @@ func (t MergeWalkPool) Set(params listParams, resultChs []FileInfoCh, endWalkCh
 			}
 			// Signal the mergeWalk go-routine to die.
 			close(endWalkCh)
-			t.lock.Unlock()
+			t.Unlock()
 		case <-endTimerCh:
 			return
 		}

cmd/merge-walk-pool_test.go

@@ -80,22 +80,22 @@ func TestManyMergeWalksSameParam(t *testing.T) {
 			tw.Set(params, walkChs, endWalkCh)
 		}
 
-		tw.lock.Lock()
+		tw.Lock()
 		if walks, ok := tw.pool[params]; ok {
 			if len(walks) != 10 {
 				t.Error("There aren't as many walks as were Set")
 			}
 		}
-		tw.lock.Unlock()
+		tw.Unlock()
 		for i := 0; i < 10; i++ {
-			tw.lock.Lock()
+			tw.Lock()
 			if walks, ok := tw.pool[params]; ok {
 				// Before ith Release we should have 10-i treeWalk go-routines.
 				if 10-i != len(walks) {
 					t.Error("There aren't as many walks as were Set")
 				}
 			}
-			tw.lock.Unlock()
+			tw.Unlock()
 			tw.Release(params)
 		}
 	}

cmd/metrics.go

@@ -19,6 +19,7 @@ package cmd
 import (
 	"net/http"
 	"strings"
+	"sync/atomic"
 	"time"
 
 	"github.com/minio/minio/cmd/logger"
@@ -190,7 +191,7 @@ func gatewayMetricsPrometheus(ch chan<- prometheus.Metric) {
 			"Total number of requests made to "+globalGatewayName+" by current MinIO Gateway",
 			[]string{"method"}, nil),
 		prometheus.CounterValue,
-		float64(s.Get.Load()),
+		float64(atomic.LoadUint64(&s.Get)),
 		http.MethodGet,
 	)
 	ch <- prometheus.MustNewConstMetric(
@@ -199,7 +200,7 @@ func gatewayMetricsPrometheus(ch chan<- prometheus.Metric) {
 			"Total number of requests made to "+globalGatewayName+" by current MinIO Gateway",
 			[]string{"method"}, nil),
 		prometheus.CounterValue,
-		float64(s.Head.Load()),
+		float64(atomic.LoadUint64(&s.Head)),
 		http.MethodHead,
 	)
 	ch <- prometheus.MustNewConstMetric(
@@ -208,7 +209,7 @@ func gatewayMetricsPrometheus(ch chan<- prometheus.Metric) {
 			"Total number of requests made to "+globalGatewayName+" by current MinIO Gateway",
 			[]string{"method"}, nil),
 		prometheus.CounterValue,
-		float64(s.Put.Load()),
+		float64(atomic.LoadUint64(&s.Put)),
 		http.MethodPut,
 	)
 	ch <- prometheus.MustNewConstMetric(
@@ -217,7 +218,7 @@ func gatewayMetricsPrometheus(ch chan<- prometheus.Metric) {
 			"Total number of requests made to "+globalGatewayName+" by current MinIO Gateway",
 			[]string{"method"}, nil),
 		prometheus.CounterValue,
-		float64(s.Post.Load()),
+		float64(atomic.LoadUint64(&s.Post)),
 		http.MethodPost,
 	)
 }

cmd/notification.go

@@ -771,7 +771,7 @@ func (sys *NotificationSys) load(buckets []BucketInfo, objAPI ObjectLayer) error
 // Init - initializes notification system from notification.xml and listener.json of all buckets.
 func (sys *NotificationSys) Init(buckets []BucketInfo, objAPI ObjectLayer) error {
 	if objAPI == nil {
-		return errInvalidArgument
+		return errServerNotInitialized
 	}
 
 	// In gateway mode, notifications are not supported.
@@ -905,7 +905,7 @@ func (sys *NotificationSys) Send(args eventArgs) {
 }
 
 // PutBucketObjectLockConfig - put bucket object lock configuration to all peers.
-func (sys *NotificationSys) PutBucketObjectLockConfig(ctx context.Context, bucketName string, retention objectlock.Retention) {
+func (sys *NotificationSys) PutBucketObjectLockConfig(ctx context.Context, bucketName string, retention *objectlock.Retention) {
 	g := errgroup.WithNErrs(len(sys.peerClients))
 	for index, client := range sys.peerClients {
 		if client == nil {

cmd/object-api-common.go

@@ -42,7 +42,7 @@ const (
 )
 
 // Global object layer mutex, used for safely updating object layer.
-var globalObjLayerMutex *sync.RWMutex
+var globalObjLayerMutex sync.RWMutex
 
 // Global object layer, only accessed by globalObjectAPI.
 var globalObjectAPI ObjectLayer
@@ -50,11 +50,6 @@ var globalObjectAPI ObjectLayer
 //Global cacheObjects, only accessed by newCacheObjectsFn().
 var globalCacheObjectAPI CacheObjectLayer
 
-func init() {
-	// Initialize this once per server initialization.
-	globalObjLayerMutex = &sync.RWMutex{}
-}
-
 // Checks if the object is a directory, this logic uses
 // if size == 0 and object ends with SlashSeparator then
 // returns true.

cmd/object-api-errors.go

@@ -17,6 +17,7 @@
 package cmd
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"io"
@@ -106,6 +107,8 @@ func toObjectErr(err error, params ...string) error {
 		err = InsufficientWriteQuorum{}
 	case io.ErrUnexpectedEOF, io.ErrShortWrite:
 		err = IncompleteBody{}
+	case context.Canceled, context.DeadlineExceeded:
+		err = IncompleteBody{}
 	}
 	return err
 }

cmd/object-api-interface.go

@@ -22,9 +22,9 @@ import (
 	"net/http"
 
 	"github.com/minio/minio-go/v6/pkg/encrypt"
+	"github.com/minio/minio-go/v6/pkg/tags"
 	bucketsse "github.com/minio/minio/pkg/bucket/encryption"
 	"github.com/minio/minio/pkg/bucket/lifecycle"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 
 	"github.com/minio/minio/pkg/madmin"
@@ -138,6 +138,6 @@ type ObjectLayer interface {
 
 	// ObjectTagging operations
 	PutObjectTag(context.Context, string, string, string) error
-	GetObjectTag(context.Context, string, string) (tagging.Tagging, error)
+	GetObjectTag(context.Context, string, string) (*tags.Tags, error)
 	DeleteObjectTag(context.Context, string, string) error
 }

cmd/object-api-utils.go

@@ -18,6 +18,7 @@ package cmd
 
 import (
 	"bytes"
+	"context"
 	"encoding/hex"
 	"fmt"
 	"io"
@@ -842,16 +843,17 @@ func newS2CompressReader(r io.Reader) io.ReadCloser {
 	return pr
 }
 
-// Returns error if the cancelCh has been closed (indicating that S3 client has disconnected)
-type detectDisconnect struct {
+// Returns error if the context is canceled, indicating
+// either client has disconnected
+type contextReader struct {
 	io.ReadCloser
-	cancelCh <-chan struct{}
+	ctx context.Context
 }
 
-func (d *detectDisconnect) Read(p []byte) (int, error) {
+func (d *contextReader) Read(p []byte) (int, error) {
 	select {
-	case <-d.cancelCh:
-		return 0, io.ErrUnexpectedEOF
+	case <-d.ctx.Done():
+		return 0, d.ctx.Err()
 	default:
 		return d.ReadCloser.Read(p)
 	}

cmd/object-handlers-common.go

@@ -172,7 +172,7 @@ func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Requ
 	}
 
 	// Check if the part number is correct.
-	if opts.PartNumber > 0 && opts.PartNumber != len(objInfo.Parts) {
+	if opts.PartNumber > 0 && len(objInfo.Parts) > 0 && opts.PartNumber != len(objInfo.Parts) {
 		writeHeaders()
 		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrPreconditionFailed), r.URL, guessIsBrowserReq(r))
 		return true

cmd/object-handlers.go

@@ -34,13 +34,13 @@ import (
 	"github.com/gorilla/mux"
 	miniogo "github.com/minio/minio-go/v6"
 	"github.com/minio/minio-go/v6/pkg/encrypt"
+	"github.com/minio/minio-go/v6/pkg/tags"
 	"github.com/minio/minio/cmd/config/etcd/dns"
 	"github.com/minio/minio/cmd/config/storageclass"
 	"github.com/minio/minio/cmd/crypto"
 	xhttp "github.com/minio/minio/cmd/http"
 	"github.com/minio/minio/cmd/logger"
 	objectlock "github.com/minio/minio/pkg/bucket/object/lock"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 	"github.com/minio/minio/pkg/event"
 	"github.com/minio/minio/pkg/handlers"
@@ -647,24 +647,6 @@ func getCpObjMetadataFromHeader(ctx context.Context, r *http.Request, userMeta m
 	return defaultMeta, nil
 }
 
-// Extract tags relevant for an CopyObject operation based on conditional
-// header values specified in X-Amz-Tagging-Directive.
-func getCpObjTagsFromHeader(ctx context.Context, r *http.Request, tags string) (string, error) {
-	// if x-amz-tagging-directive says REPLACE then
-	// we extract tags from the input headers.
-	if isDirectiveReplace(r.Header.Get(xhttp.AmzTagDirective)) {
-		if tags := r.Header.Get(xhttp.AmzObjectTagging); tags != "" {
-			return extractTags(ctx, tags)
-		}
-		// If x-amz-tagging-directive is explicitly set to replace and  x-amz-tagging is not set.
-		// The S3 behavior is to unset the tags.
-		return "", nil
-	}
-
-	// Copy is default behavior if x-amz-tagging-directive is not set.
-	return tags, nil
-}
-
 // getRemoteInstanceTransport contains a singleton roundtripper.
 var getRemoteInstanceTransport http.RoundTripper
 var getRemoteInstanceTransportOnce sync.Once
@@ -1056,14 +1038,18 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
 		return
 	}
 
-	tags, err := getCpObjTagsFromHeader(ctx, r, srcInfo.UserTags)
-	if err != nil {
-		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
-		return
+	objTags := srcInfo.UserTags
+	// If x-amz-tagging-directive header is REPLACE, get passed tags.
+	if isDirectiveReplace(r.Header.Get(xhttp.AmzTagDirective)) {
+		objTags = r.Header.Get(xhttp.AmzObjectTagging)
+		if _, err := tags.ParseObjectTags(objTags); err != nil {
+			writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
+			return
+		}
 	}
 
-	if tags != "" {
-		srcInfo.UserDefined[xhttp.AmzObjectTagging] = tags
+	if objTags != "" {
+		srcInfo.UserDefined[xhttp.AmzObjectTagging] = objTags
 	}
 
 	srcInfo.UserDefined = objectlock.FilterObjectLockMetadata(srcInfo.UserDefined, true, true)
@@ -1209,7 +1195,7 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
 	}
 
 	// To detect if the client has disconnected.
-	r.Body = &detectDisconnect{r.Body, r.Context().Done()}
+	r.Body = &contextReader{r.Body, r.Context()}
 
 	// X-Amz-Copy-Source shouldn't be set for this call.
 	if _, ok := r.Header[xhttp.AmzCopySource]; ok {
@@ -1265,12 +1251,13 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
 		return
 	}
 
-	if tags := r.Header.Get(xhttp.AmzObjectTagging); tags != "" {
-		metadata[xhttp.AmzObjectTagging], err = extractTags(ctx, tags)
-		if err != nil {
+	if objTags := r.Header.Get(xhttp.AmzObjectTagging); objTags != "" {
+		if _, err := tags.ParseObjectTags(objTags); err != nil {
 			writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
 			return
 		}
+
+		metadata[xhttp.AmzObjectTagging] = objTags
 	}
 
 	if rAuthType == authTypeStreamingSigned {
@@ -1958,7 +1945,7 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
 	}
 
 	// To detect if the client has disconnected.
-	r.Body = &detectDisconnect{r.Body, r.Context().Done()}
+	r.Body = &contextReader{r.Body, r.Context()}
 
 	// X-Amz-Copy-Source shouldn't be set for this call.
 	if _, ok := r.Header[xhttp.AmzCopySource]; ok {
@@ -2994,14 +2981,14 @@ func (api objectAPIHandlers) PutObjectTaggingHandler(w http.ResponseWriter, r *h
 		return
 	}
 
-	tagging, err := tagging.ParseTagging(io.LimitReader(r.Body, r.ContentLength))
+	tags, err := tags.ParseObjectXML(io.LimitReader(r.Body, r.ContentLength))
 	if err != nil {
 		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
 		return
 	}
 
 	// Put object tags
-	err = objAPI.PutObjectTag(ctx, bucket, object, tagging.String())
+	err = objAPI.PutObjectTag(ctx, bucket, object, tags.String())
 	if err != nil {
 		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
 		return
@@ -3036,8 +3023,7 @@ func (api objectAPIHandlers) DeleteObjectTaggingHandler(w http.ResponseWriter, r
 	}
 
 	// Delete object tags
-	err = objAPI.DeleteObjectTag(ctx, bucket, object)
-	if err != nil {
+	if err = objAPI.DeleteObjectTag(ctx, bucket, object); err != nil && err != errConfigNotFound {
 		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
 		return
 	}

cmd/peer-rest-client.go

@@ -656,12 +656,12 @@ func (client *peerRESTClient) PutBucketNotification(bucket string, rulesMap even
 }
 
 // PutBucketObjectLockConfig - PUT bucket object lock configuration.
-func (client *peerRESTClient) PutBucketObjectLockConfig(bucket string, retention objectlock.Retention) error {
+func (client *peerRESTClient) PutBucketObjectLockConfig(bucket string, retention *objectlock.Retention) error {
 	values := make(url.Values)
 	values.Set(peerRESTBucket, bucket)
 
 	var reader bytes.Buffer
-	err := gob.NewEncoder(&reader).Encode(&retention)
+	err := gob.NewEncoder(&reader).Encode(retention)
 	if err != nil {
 		return err
 	}

cmd/peer-rest-server.go

@@ -675,13 +675,14 @@ func (s *peerRESTServer) SetBucketPolicyHandler(w http.ResponseWriter, r *http.R
 		s.writeErrorResponse(w, errors.New("Bucket name is missing"))
 		return
 	}
-	var policyData policy.Policy
+
 	if r.ContentLength < 0 {
 		s.writeErrorResponse(w, errInvalidArgument)
 		return
 	}
 
-	err := gob.NewDecoder(r.Body).Decode(&policyData)
+	var policyData = &policy.Policy{}
+	err := gob.NewDecoder(r.Body).Decode(policyData)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
@@ -716,13 +717,13 @@ func (s *peerRESTServer) SetBucketLifecycleHandler(w http.ResponseWriter, r *htt
 		s.writeErrorResponse(w, errors.New("Bucket name is missing"))
 		return
 	}
-	var lifecycleData lifecycle.Lifecycle
 	if r.ContentLength < 0 {
 		s.writeErrorResponse(w, errInvalidArgument)
 		return
 	}
 
-	err := gob.NewDecoder(r.Body).Decode(&lifecycleData)
+	var lifecycleData = &lifecycle.Lifecycle{}
+	err := gob.NewDecoder(r.Body).Decode(lifecycleData)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
@@ -758,13 +759,13 @@ func (s *peerRESTServer) SetBucketSSEConfigHandler(w http.ResponseWriter, r *htt
 		return
 	}
 
-	var encConfig bucketsse.BucketSSEConfig
 	if r.ContentLength < 0 {
 		s.writeErrorResponse(w, errInvalidArgument)
 		return
 	}
 
-	err := gob.NewDecoder(r.Body).Decode(&encConfig)
+	var encConfig = &bucketsse.BucketSSEConfig{}
+	err := gob.NewDecoder(r.Body).Decode(encConfig)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return
@@ -860,13 +861,13 @@ func (s *peerRESTServer) PutBucketObjectLockConfigHandler(w http.ResponseWriter,
 		return
 	}
 
-	var retention objectlock.Retention
 	if r.ContentLength < 0 {
 		s.writeErrorResponse(w, errInvalidArgument)
 		return
 	}
 
-	err := gob.NewDecoder(r.Body).Decode(&retention)
+	var retention = &objectlock.Retention{}
+	err := gob.NewDecoder(r.Body).Decode(retention)
 	if err != nil {
 		s.writeErrorResponse(w, err)
 		return

cmd/policy_test.go

@@ -41,10 +41,10 @@ func TestPolicySysSet(t *testing.T) {
 		},
 	}
 	case1Result := NewPolicySys()
-	case1Result.bucketPolicyMap["mybucket"] = case1Policy
+	case1Result.bucketPolicyMap["mybucket"] = &case1Policy
 
 	case2PolicySys := NewPolicySys()
-	case2PolicySys.bucketPolicyMap["mybucket"] = case1Policy
+	case2PolicySys.bucketPolicyMap["mybucket"] = &case1Policy
 	case2Policy := policy.Policy{
 		Version: policy.DefaultVersion,
 		Statements: []policy.Statement{
@@ -58,10 +58,10 @@ func TestPolicySysSet(t *testing.T) {
 		},
 	}
 	case2Result := NewPolicySys()
-	case2Result.bucketPolicyMap["mybucket"] = case2Policy
+	case2Result.bucketPolicyMap["mybucket"] = &case2Policy
 
 	case3PolicySys := NewPolicySys()
-	case3PolicySys.bucketPolicyMap["mybucket"] = case2Policy
+	case3PolicySys.bucketPolicyMap["mybucket"] = &case2Policy
 	case3Policy := policy.Policy{
 		ID:      "MyPolicyForMyBucket",
 		Version: policy.DefaultVersion,
@@ -81,7 +81,7 @@ func TestPolicySysSet(t *testing.T) {
 
 	for i, testCase := range testCases {
 		result := testCase.policySys
-		result.Set(testCase.bucketName, testCase.bucketPolicy)
+		result.Set(testCase.bucketName, &testCase.bucketPolicy)
 
 		if !reflect.DeepEqual(result, testCase.expectedResult) {
 			t.Fatalf("case %v: expected: %v, got: %v\n", i+1, testCase.expectedResult, result)
@@ -103,7 +103,7 @@ func TestPolicySysRemove(t *testing.T) {
 		},
 	}
 	case1PolicySys := NewPolicySys()
-	case1PolicySys.bucketPolicyMap["mybucket"] = case1Policy
+	case1PolicySys.bucketPolicyMap["mybucket"] = &case1Policy
 	case1Result := NewPolicySys()
 
 	case2Policy := policy.Policy{
@@ -119,9 +119,9 @@ func TestPolicySysRemove(t *testing.T) {
 		},
 	}
 	case2PolicySys := NewPolicySys()
-	case2PolicySys.bucketPolicyMap["mybucket"] = case2Policy
+	case2PolicySys.bucketPolicyMap["mybucket"] = &case2Policy
 	case2Result := NewPolicySys()
-	case2Result.bucketPolicyMap["mybucket"] = case2Policy
+	case2Result.bucketPolicyMap["mybucket"] = &case2Policy
 
 	case3PolicySys := NewPolicySys()
 	case3Result := NewPolicySys()
@@ -148,7 +148,7 @@ func TestPolicySysRemove(t *testing.T) {
 
 func TestPolicySysIsAllowed(t *testing.T) {
 	policySys := NewPolicySys()
-	policySys.Set("mybucket", policy.Policy{
+	policySys.Set("mybucket", &policy.Policy{
 		Version: policy.DefaultVersion,
 		Statements: []policy.Statement{
 			policy.NewStatement(

cmd/posix-errors.go

@@ -38,6 +38,11 @@ func isSysErrNoSpace(err error) bool {
 	return errors.Is(err, syscall.ENOSPC)
 }
 
+// Invalid argument, unsupported flags such as O_DIRECT
+func isSysErrInvalidArg(err error) bool {
+	return errors.Is(err, syscall.EINVAL)
+}
+
 // Input/output error
 func isSysErrIO(err error) bool {
 	return errors.Is(err, syscall.EIO)

cmd/posix.go

@@ -110,8 +110,11 @@ func checkPathLength(pathName string) error {
 
 	// Disallow more than 1024 characters on windows, there
 	// are no known name_max limits on Windows.
-	if runtime.GOOS == "windows" && len(pathName) > 1024 {
-		return nil
+	if runtime.GOOS == "windows" {
+		if len(pathName) <= 1024 {
+			return nil
+		}
+		return errFileNameTooLong
 	}
 
 	// On Unix we reject paths if they are just '.', '..' or '/'
@@ -174,13 +177,20 @@ func getValidPath(path string) (string, error) {
 	// check if backend is writable.
 	var rnd [8]byte
 	_, _ = rand.Read(rnd[:])
+
 	fn := pathJoin(path, ".writable-check-"+hex.EncodeToString(rnd[:])+".tmp")
-	file, err := os.Create(fn)
+	defer os.Remove(fn)
+
+	// open file in direct I/O and use default umask, this also verifies
+	// if direct i/o failed.
+	file, err := disk.OpenFileDirectIO(fn, os.O_CREATE|os.O_EXCL, 0666)
 	if err != nil {
+		if isSysErrInvalidArg(err) {
+			return path, errUnsupportedDisk
+		}
 		return path, err
 	}
 	file.Close()
-	os.Remove(fn)
 
 	return path, nil
 }
@@ -1276,6 +1286,8 @@ func (s *posix) CreateFile(volume, path string, fileSize int64, r io.Reader) (er
 			return errFileAccessDenied
 		case isSysErrIO(err):
 			return errFaultyDisk
+		case isSysErrInvalidArg(err):
+			return errUnsupportedDisk
 		default:
 			return err
 		}

cmd/retry.go

@@ -17,6 +17,7 @@
 package cmd
 
 import (
+	"context"
 	"math/rand"
 	"sync"
 	"time"
@@ -61,7 +62,7 @@ var globalRandomSource = rand.New(&lockedRandSource{
 // until the maximum retry attempts are reached. - this function is a fully
 // configurable version, meant for only advanced use cases. For the most part
 // one should use newRetryTimerSimple and newRetryTimer.
-func newRetryTimerWithJitter(unit time.Duration, cap time.Duration, jitter float64, doneCh chan struct{}) <-chan int {
+func newRetryTimerWithJitter(ctx context.Context, unit time.Duration, cap time.Duration, jitter float64) <-chan int {
 	attemptCh := make(chan int)
 
 	// normalize jitter to the range [0, 1.0]
@@ -100,7 +101,7 @@ func newRetryTimerWithJitter(unit time.Duration, cap time.Duration, jitter float
 			select { // Attempts starts.
 			case attemptCh <- nextBackoff:
 				nextBackoff++
-			case <-doneCh:
+			case <-ctx.Done():
 				// Stop the routine.
 				return
 			}
@@ -108,7 +109,7 @@ func newRetryTimerWithJitter(unit time.Duration, cap time.Duration, jitter float
 			// wait till next backoff time or till doneCh gets a message.
 			select {
 			case <-timer.C:
-			case <-doneCh:
+			case <-ctx.Done():
 				// stop the timer and return.
 				timer.Stop()
 				return
@@ -130,6 +131,6 @@ const (
 // newRetryTimerSimple creates a timer with exponentially increasing delays
 // until the maximum retry attempts are reached. - this function is a
 // simpler version with all default values.
-func newRetryTimerSimple(doneCh chan struct{}) <-chan int {
-	return newRetryTimerWithJitter(defaultRetryUnit, defaultRetryCap, MaxJitter, doneCh)
+func newRetryTimerSimple(ctx context.Context) <-chan int {
+	return newRetryTimerWithJitter(ctx, defaultRetryUnit, defaultRetryCap, MaxJitter)
 }

cmd/retry_test.go

@@ -1,5 +1,5 @@
 /*
- * MinIO Cloud Storage, (C) 2016 MinIO, Inc.
+ * Minio Cloud Storage, (C) 2016-2020 Minio, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -17,25 +17,26 @@
 package cmd
 
 import (
+	"context"
 	"testing"
 	"time"
 )
 
 // Tests for retry timer.
 func TestRetryTimerSimple(t *testing.T) {
-	doneCh := make(chan struct{})
-	attemptCh := newRetryTimerSimple(doneCh)
+	rctx, cancel := context.WithCancel(context.Background())
+	attemptCh := newRetryTimerSimple(rctx)
 	i := <-attemptCh
 	if i != 0 {
-		close(doneCh)
+		cancel()
 		t.Fatalf("Invalid attempt counter returned should be 0, found %d instead", i)
 	}
 	i = <-attemptCh
 	if i <= 0 {
-		close(doneCh)
+		cancel()
 		t.Fatalf("Invalid attempt counter returned should be greater than 0, found %d instead", i)
 	}
-	close(doneCh)
+	cancel()
 	_, ok := <-attemptCh
 	if ok {
 		t.Fatal("Attempt counter should be closed")
@@ -44,20 +45,23 @@ func TestRetryTimerSimple(t *testing.T) {
 
 // Test retry time with no jitter.
 func TestRetryTimerWithNoJitter(t *testing.T) {
-	doneCh := make(chan struct{})
+	rctx, cancel := context.WithCancel(context.Background())
+
 	// No jitter
-	attemptCh := newRetryTimerWithJitter(time.Millisecond, 5*time.Millisecond, NoJitter, doneCh)
+	attemptCh := newRetryTimerWithJitter(rctx, time.Millisecond, 5*time.Millisecond, NoJitter)
 	i := <-attemptCh
 	if i != 0 {
-		close(doneCh)
+		cancel()
 		t.Fatalf("Invalid attempt counter returned should be 0, found %d instead", i)
 	}
 	// Loop through the maximum possible attempt.
 	for i = range attemptCh {
 		if i == 30 {
-			close(doneCh)
+			break
 		}
 	}
+
+	cancel()
 	_, ok := <-attemptCh
 	if ok {
 		t.Fatal("Attempt counter should be closed")
@@ -66,15 +70,16 @@ func TestRetryTimerWithNoJitter(t *testing.T) {
 
 // Test retry time with Jitter greater than MaxJitter.
 func TestRetryTimerWithJitter(t *testing.T) {
-	doneCh := make(chan struct{})
+	rctx, cancel := context.WithCancel(context.Background())
+
 	// Jitter will be set back to 1.0
-	attemptCh := newRetryTimerWithJitter(defaultRetryUnit, defaultRetryCap, 2.0, doneCh)
+	attemptCh := newRetryTimerWithJitter(rctx, time.Second, 30*time.Second, 2.0)
 	i := <-attemptCh
 	if i != 0 {
-		close(doneCh)
+		cancel()
 		t.Fatalf("Invalid attempt counter returned should be 0, found %d instead", i)
 	}
-	close(doneCh)
+	cancel()
 	_, ok := <-attemptCh
 	if ok {
 		t.Fatal("Attempt counter should be closed")

cmd/server-main.go

@@ -155,55 +155,50 @@ func newAllSubsystems() {
 
 	// Create new bucket encryption subsystem
 	globalBucketSSEConfigSys = NewBucketSSEConfigSys()
+
+	// Create new bucket quota subsystem
+	globalBucketQuotaSys = NewBucketQuotaSys()
 }
 
-func initSafeMode(buckets []BucketInfo) (err error) {
+func initSafeMode() (err error) {
 	newObject := newObjectLayerWithoutSafeModeFn()
 
-	// Construct path to config/transaction.lock for locking
-	transactionConfigPrefix := minioConfigPrefix + "/transaction.lock"
-
 	// Make sure to hold lock for entire migration to avoid
 	// such that only one server should migrate the entire config
 	// at a given time, this big transaction lock ensures this
 	// appropriately. This is also true for rotation of encrypted
 	// content.
-	objLock := newObject.NewNSLock(GlobalContext, minioMetaBucket, transactionConfigPrefix)
-	if err = objLock.GetLock(globalOperationTimeout); err != nil {
-		return err
-	}
-
-	defer func(objLock RWLocker) {
-		objLock.Unlock()
+	txnLk := newObject.NewNSLock(GlobalContext, minioMetaBucket, minioConfigPrefix+"/transaction.lock")
+	defer func(txnLk RWLocker) {
+		txnLk.Unlock()
 
 		if err != nil {
 			var cerr config.Err
+			// For any config error, we don't need to drop into safe-mode
+			// instead its a user error and should be fixed by user.
 			if errors.As(err, &cerr) {
 				return
 			}
 
 			// Prints the formatted startup message in safe mode operation.
+			// Drops-into safe mode where users need to now manually recover
+			// the server.
 			printStartupSafeModeMessage(getAPIEndpoints(), err)
 
 			// Initialization returned error reaching safe mode and
 			// not proceeding waiting for admin action.
 			handleSignals()
 		}
-	}(objLock)
-
-	// Migrate all backend configs to encrypted backend configs, optionally
-	// handles rotating keys for encryption.
-	if err = handleEncryptedConfigBackend(newObject, true); err != nil {
-		return fmt.Errorf("Unable to handle encrypted backend for config, iam and policies: %w", err)
-	}
+	}(txnLk)
 
 	// ****  WARNING ****
 	// Migrating to encrypted backend should happen before initialization of any
 	// sub-systems, make sure that we do not move the above codeblock elsewhere.
+	// Create cancel context to control 'newRetryTimer' go routine.
+	retryCtx, cancel := context.WithCancel(GlobalContext)
 
-	// Validate and initialize all subsystems.
-	doneCh := make(chan struct{})
-	defer close(doneCh)
+	// Indicate to our routine to exit cleanly upon return.
+	defer cancel()
 
 	// Initializing sub-systems needs a retry mechanism for
 	// the following reasons:
@@ -211,43 +206,106 @@ func initSafeMode(buckets []BucketInfo) (err error) {
 	//    of the object layer.
 	//  - Write quorum not met when upgrading configuration
 	//    version is needed, migration is needed etc.
-	retryTimerCh := newRetryTimerSimple(doneCh)
-	for {
-		rquorum := InsufficientReadQuorum{}
-		wquorum := InsufficientWriteQuorum{}
-		bucketNotFound := BucketNotFound{}
-		var err error
-		select {
-		case n := <-retryTimerCh:
-			if err = initAllSubsystems(buckets, newObject); err != nil {
-				if errors.Is(err, errDiskNotFound) ||
-					errors.As(err, &rquorum) ||
-					errors.As(err, &wquorum) ||
-					errors.As(err, &bucketNotFound) {
-					if n < 5 {
-						logger.Info("Waiting for all sub-systems to be initialized..")
-					} else {
-						logger.Info("Waiting for all sub-systems to be initialized.. %v", err)
-					}
-					continue
-				}
-				return err
-			}
-			return nil
-		case <-globalOSSignalCh:
-			if err == nil {
-				return errors.New("Initializing sub-systems stopped gracefully")
-			}
-			return fmt.Errorf("Unable to initialize sub-systems: %w", err)
+	rquorum := InsufficientReadQuorum{}
+	wquorum := InsufficientWriteQuorum{}
+	for range newRetryTimerSimple(retryCtx) {
+		// let one of the server acquire the lock, if not let them timeout.
+		// which shall be retried again by this loop.
+		if err = txnLk.GetLock(leaderLockTimeout); err != nil {
+			logger.Info("Waiting for all MinIO sub-systems to be initialized.. trying to acquire lock")
+			continue
 		}
+
+		// These messages only meant primarily for distributed setup, so only log during distributed setup.
+		if globalIsDistXL {
+			logger.Info("Waiting for all MinIO sub-systems to be initialized.. lock acquired")
+		}
+
+		// Migrate all backend configs to encrypted backend configs, optionally
+		// handles rotating keys for encryption, if there is any retriable failure
+		// that shall be retried if there is an error.
+		if err = handleEncryptedConfigBackend(newObject, true); err == nil {
+			// Upon success migrating the config, initialize all sub-systems
+			// if all sub-systems initialized successfully return right away
+			if err = initAllSubsystems(newObject); err == nil {
+				// All successful return.
+				if globalIsDistXL {
+					// These messages only meant primarily for distributed setup, so only log during distributed setup.
+					logger.Info("All MinIO sub-systems initialized successfully")
+				}
+				return nil
+			}
+		}
+
+		// One of these retriable errors shall be retried.
+		if errors.Is(err, errDiskNotFound) ||
+			errors.Is(err, errConfigNotFound) ||
+			errors.Is(err, context.Canceled) ||
+			errors.Is(err, context.DeadlineExceeded) ||
+			errors.As(err, &rquorum) ||
+			errors.As(err, &wquorum) ||
+			isErrBucketNotFound(err) {
+			logger.Info("Waiting for all MinIO sub-systems to be initialized.. possible cause (%v)", err)
+			txnLk.Unlock() // Unlock the transaction lock and allow other nodes to acquire the lock if possible.
+			continue
+		}
+
+		// Any other unhandled return right here.
+		return fmt.Errorf("Unable to initialize sub-systems: %w", err)
 	}
+
+	// Return an error when retry is canceled or deadlined
+	if err = retryCtx.Err(); err != nil {
+		return fmt.Errorf("Unable to initialize sub-systems: %w", err)
+	}
+
+	// Retry was canceled successfully.
+	return errors.New("Initializing sub-systems stopped gracefully")
 }
 
-func initAllSubsystems(buckets []BucketInfo, newObject ObjectLayer) (err error) {
+func initAllSubsystems(newObject ObjectLayer) (err error) {
+	// %w is used by all error returns here to make sure
+	// we wrap the underlying error, make sure when you
+	// are modifying this code that you do so, if and when
+	// you want to add extra context to your error. This
+	// ensures top level retry works accordingly.
+	var buckets []BucketInfo
+	if globalIsDistXL || globalIsXL {
+		// List buckets to heal, and be re-used for loading configs.
+		buckets, err = newObject.ListBucketsHeal(GlobalContext)
+		if err != nil {
+			return fmt.Errorf("Unable to list buckets to heal: %w", err)
+		}
+		// Attempt a heal if possible and re-use the bucket names
+		// to reload their config.
+		wquorum := &InsufficientWriteQuorum{}
+		rquorum := &InsufficientReadQuorum{}
+		for _, bucket := range buckets {
+			if err = newObject.MakeBucketWithLocation(GlobalContext, bucket.Name, ""); err != nil {
+				if errors.As(err, &wquorum) || errors.As(err, &rquorum) {
+					// Retrun the error upwards for the caller to retry.
+					return fmt.Errorf("Unable to heal bucket: %w", err)
+				}
+				if _, ok := err.(BucketExists); !ok {
+					// ignore any other error and log for investigation.
+					logger.LogIf(GlobalContext, err)
+					continue
+				}
+				// Bucket already exists, nothing that needs to be done.
+			}
+		}
+	} else {
+		buckets, err = newObject.ListBuckets(GlobalContext)
+		if err != nil {
+			return fmt.Errorf("Unable to list buckets: %w", err)
+		}
+	}
+
 	// Initialize config system.
 	if err = globalConfigSys.Init(newObject); err != nil {
 		return fmt.Errorf("Unable to initialize config system: %w", err)
 	}
+
 	if globalEtcdClient != nil {
 		// ****  WARNING ****
 		// Migrating to encrypted backend on etcd should happen before initialization of
@@ -275,10 +333,6 @@ func initAllSubsystems(buckets []BucketInfo, newObject ObjectLayer) (err error)
 	if err = initBucketObjectLockConfig(buckets, newObject); err != nil {
 		return fmt.Errorf("Unable to initialize object lock system: %w", err)
 	}
-	// Initialize bucket quota system.
-	if err = initBucketQuotaSys(buckets, newObject); err != nil {
-		return fmt.Errorf("Unable to initialize bucket quota system: %w", err)
-	}
 
 	// Initialize lifecycle system.
 	if err = globalLifecycleSys.Init(buckets, newObject); err != nil {
@@ -290,6 +344,16 @@ func initAllSubsystems(buckets []BucketInfo, newObject ObjectLayer) (err error)
 		return fmt.Errorf("Unable to initialize bucket encryption subsystem: %w", err)
 	}
 
+	// Initialize bucket quota system.
+	if err = globalBucketQuotaSys.Init(buckets, newObject); err != nil {
+		return fmt.Errorf("Unable to initialize bucket quota system: %w", err)
+	}
+
+	// Populate existing buckets to the etcd backend
+	if globalDNSConfig != nil {
+		initFederatorBackend(buckets, newObject)
+	}
+
 	return nil
 }
 
@@ -454,13 +518,7 @@ func serverMain(ctx *cli.Context) {
 
 	go startBackgroundOps(GlobalContext, newObject)
 
-	// Calls New() and initializes all sub-systems.
-	buckets, err := newObject.ListBuckets(GlobalContext)
-	if err != nil {
-		logger.Fatal(err, "Unable to list buckets")
-	}
-
-	logger.FatalIf(initSafeMode(buckets), "Unable to initialize server switching into safe-mode")
+	logger.FatalIf(initSafeMode(), "Unable to initialize server switching into safe-mode")
 
 	if globalCacheConfig.Enabled {
 		// initialize the new disk cache objects.
@@ -473,11 +531,6 @@ func serverMain(ctx *cli.Context) {
 		globalObjLayerMutex.Unlock()
 	}
 
-	// Populate existing buckets to the etcd backend
-	if globalDNSConfig != nil {
-		initFederatorBackend(buckets, newObject)
-	}
-
 	// Disable safe mode operation, after all initialization is over.
 	globalObjLayerMutex.Lock()
 	globalSafeMode = false

cmd/storage-errors.go

@@ -25,6 +25,9 @@ var errCorruptedFormat = StorageErr("corrupted backend format, please join https
 // errUnformattedDisk - unformatted disk found.
 var errUnformattedDisk = StorageErr("unformatted disk found")
 
+// errUnsupporteDisk - when disk does not support O_DIRECT flag.
+var errUnsupportedDisk = StorageErr("disk does not support O_DIRECT")
+
 // errDiskFull - cannot create volume or files when disk is full.
 var errDiskFull = StorageErr("disk path full")
 
@@ -81,7 +84,7 @@ var errBitrotHashAlgoInvalid = StorageErr("bit-rot hash algorithm is invalid")
 var errCrossDeviceLink = StorageErr("Rename across devices not allowed, please fix your backend configuration")
 
 // errMinDiskSize - cannot create volume or files when disk size is less than threshold.
-var errMinDiskSize = StorageErr("The disk size is less than the minimum threshold")
+var errMinDiskSize = StorageErr("The disk size is less than 900MiB threshold")
 
 // errLessData - returned when less data available than what was requested.
 var errLessData = StorageErr("less data available than what was requested")

cmd/storage-rest-server.go

@@ -738,6 +738,12 @@ func registerStorageRESTHandlers(router *mux.Router, endpointZones EndpointZones
 			}
 			storage, err := newPosix(endpoint.Path)
 			if err != nil {
+				if err == errMinDiskSize {
+					logger.Fatal(config.ErrUnableToWriteInBackend(err).Hint(err.Error()), "Unable to initialize backend")
+				} else if err == errUnsupportedDisk {
+					hint := fmt.Sprintf("'%s' does not support O_DIRECT flags, refusing to use", endpoint.Path)
+					logger.Fatal(config.ErrUnsupportedBackend(err).Hint(hint), "Unable to initialize backend")
+				}
 				// Show a descriptive error with a hint about how to fix it.
 				var username string
 				if u, err := user.Current(); err == nil {

cmd/test-utils_test.go

@@ -1750,7 +1750,7 @@ func ExecObjectLayerAPIAnonTest(t *testing.T, obj ObjectLayer, testName, bucketN
 	if err := obj.SetBucketPolicy(context.Background(), bucketName, bucketPolicy); err != nil {
 		t.Fatalf("unexpected error. %v", err)
 	}
-	globalPolicySys.Set(bucketName, *bucketPolicy)
+	globalPolicySys.Set(bucketName, bucketPolicy)
 	defer globalPolicySys.Remove(bucketName)
 
 	// now call the handler again with the unsigned/anonymous request, it should be accepted.

cmd/tree-walk-pool.go

@@ -55,9 +55,9 @@ type treeWalk struct {
 // treeWalkPool's purpose is to maintain active treeWalk go-routines in a map so that
 // it can be looked up across related list calls.
 type TreeWalkPool struct {
+	sync.Mutex
 	pool    map[listParams][]treeWalk
 	timeOut time.Duration
-	lock    *sync.Mutex
 }
 
 // NewTreeWalkPool - initialize new tree walk pool.
@@ -65,7 +65,6 @@ func NewTreeWalkPool(timeout time.Duration) *TreeWalkPool {
 	tPool := &TreeWalkPool{
 		pool:    make(map[listParams][]treeWalk),
 		timeOut: timeout,
-		lock:    &sync.Mutex{},
 	}
 	return tPool
 }
@@ -74,9 +73,9 @@ func NewTreeWalkPool(timeout time.Duration) *TreeWalkPool {
 // listParams, removes it from the pool, and returns the TreeWalkResult
 // channel.
 // Returns nil if listParams does not have an asccociated treeWalk.
-func (t TreeWalkPool) Release(params listParams) (resultCh chan TreeWalkResult, endWalkCh chan struct{}) {
-	t.lock.Lock()
-	defer t.lock.Unlock()
+func (t *TreeWalkPool) Release(params listParams) (resultCh chan TreeWalkResult, endWalkCh chan struct{}) {
+	t.Lock()
+	defer t.Unlock()
 	walks, ok := t.pool[params] // Pick the valid walks.
 	if ok {
 		if len(walks) > 0 {
@@ -104,9 +103,9 @@ func (t TreeWalkPool) Release(params listParams) (resultCh chan TreeWalkResult,
 // 2) Relase() signals the timer go-routine to end on endTimerCh.
 //    During listing the timer should not timeout and end the treeWalk go-routine, hence the
 //    timer go-routine should be ended.
-func (t TreeWalkPool) Set(params listParams, resultCh chan TreeWalkResult, endWalkCh chan struct{}) {
-	t.lock.Lock()
-	defer t.lock.Unlock()
+func (t *TreeWalkPool) Set(params listParams, resultCh chan TreeWalkResult, endWalkCh chan struct{}) {
+	t.Lock()
+	defer t.Unlock()
 
 	// Should be a buffered channel so that Release() never blocks.
 	endTimerCh := make(chan struct{}, 1)
@@ -125,7 +124,7 @@ func (t TreeWalkPool) Set(params listParams, resultCh chan TreeWalkResult, endWa
 		case <-time.After(t.timeOut):
 			// Timeout has expired. Remove the treeWalk from treeWalkPool and
 			// end the treeWalk go-routine.
-			t.lock.Lock()
+			t.Lock()
 			walks, ok := t.pool[params]
 			if ok {
 				// Trick of filtering without allocating
@@ -149,7 +148,7 @@ func (t TreeWalkPool) Set(params listParams, resultCh chan TreeWalkResult, endWa
 			}
 			// Signal the treeWalk go-routine to die.
 			close(endWalkCh)
-			t.lock.Unlock()
+			t.Unlock()
 		case <-endTimerCh:
 			return
 		}

cmd/tree-walk-pool_test.go

@@ -80,22 +80,22 @@ func TestManyWalksSameParam(t *testing.T) {
 			tw.Set(params, resultCh, endWalkCh)
 		}
 
-		tw.lock.Lock()
+		tw.Lock()
 		if walks, ok := tw.pool[params]; ok {
 			if len(walks) != 10 {
 				t.Error("There aren't as many walks as were Set")
 			}
 		}
-		tw.lock.Unlock()
+		tw.Unlock()
 		for i := 0; i < 10; i++ {
-			tw.lock.Lock()
+			tw.Lock()
 			if walks, ok := tw.pool[params]; ok {
 				// Before ith Release we should have 10-i treeWalk go-routines.
 				if 10-i != len(walks) {
 					t.Error("There aren't as many walks as were Set")
 				}
 			}
-			tw.lock.Unlock()
+			tw.Unlock()
 			tw.Release(params)
 		}
 	}

cmd/typed-errors.go

@@ -92,3 +92,6 @@ var errIAMActionNotAllowed = errors.New("Specified IAM action is not allowed und
 
 // error returned when access is denied.
 var errAccessDenied = errors.New("Do not have enough permissions to access this resource")
+
+// error returned when object is locked.
+var errLockedObject = errors.New("Object is WORM protected and cannot be overwritten or deleted")

cmd/web-handlers.go

@@ -667,10 +667,8 @@ next:
 	for _, objectName := range args.Objects {
 		// If not a directory, remove the object.
 		if !HasSuffix(objectName, SlashSeparator) && objectName != "" {
-			// Check for permissions only in the case of
-			// non-anonymous login. For anonymous login, policy has already
-			// been checked.
-			govBypassPerms := ErrAccessDenied
+			// Check permissions for non-anonymous user.
+			govBypassPerms := false
 			if authErr != errNoAuthToken {
 				if !globalIAMSys.IsAllowed(iampolicy.Args{
 					AccountName:     claims.AccessKey,
@@ -692,22 +690,12 @@ next:
 					ObjectName:      objectName,
 					Claims:          claims.Map(),
 				}) {
-					govBypassPerms = ErrNone
-				}
-				if globalIAMSys.IsAllowed(iampolicy.Args{
-					AccountName:     claims.AccessKey,
-					Action:          iampolicy.GetBucketObjectLockConfigurationAction,
-					BucketName:      args.BucketName,
-					ConditionValues: getConditionValues(r, "", claims.AccessKey, claims.Map()),
-					IsOwner:         owner,
-					ObjectName:      objectName,
-					Claims:          claims.Map(),
-				}) {
-					govBypassPerms = ErrNone
+					govBypassPerms = true
 				}
 			}
+
 			if authErr == errNoAuthToken {
-				// Check if object is allowed to be deleted anonymously
+				// Check if object is allowed to be deleted anonymously.
 				if !globalPolicySys.IsAllowed(policy.Args{
 					Action:          policy.DeleteObjectAction,
 					BucketName:      args.BucketName,
@@ -726,31 +714,17 @@ next:
 					IsOwner:         false,
 					ObjectName:      objectName,
 				}) {
-					govBypassPerms = ErrNone
-				}
-
-				// Check if object is allowed to be deleted anonymously
-				if globalPolicySys.IsAllowed(policy.Args{
-					Action:          policy.GetBucketObjectLockConfigurationAction,
-					BucketName:      args.BucketName,
-					ConditionValues: getConditionValues(r, "", "", nil),
-					IsOwner:         false,
-					ObjectName:      objectName,
-				}) {
-					govBypassPerms = ErrNone
+					govBypassPerms = true
 				}
 			}
-			if govBypassPerms != ErrNone {
+
+			apiErr := enforceRetentionBypassForDeleteWeb(ctx, r, args.BucketName, objectName, getObjectInfo, govBypassPerms)
+			if apiErr == ErrObjectLocked {
+				return toJSONError(ctx, errLockedObject)
+			}
+			if apiErr != ErrNone && apiErr != ErrNoSuchKey {
 				return toJSONError(ctx, errAccessDenied)
 			}
-
-			apiErr := ErrNone
-			if _, ok := globalBucketObjectLockConfig.Get(args.BucketName); ok && (apiErr == ErrNone) {
-				apiErr = enforceRetentionBypassForDeleteWeb(ctx, r, args.BucketName, objectName, getObjectInfo)
-				if apiErr != ErrNone && apiErr != ErrNoSuchKey {
-					return toJSONError(ctx, errAccessDenied)
-				}
-			}
 			if apiErr == ErrNone {
 				if err = deleteObject(ctx, objectAPI, web.CacheAPI(), args.BucketName, objectName, r); err != nil {
 					break next
@@ -1933,7 +1907,7 @@ func (web *webAPIHandlers) SetBucketPolicy(r *http.Request, args *SetBucketPolic
 			return toJSONError(ctx, err, args.BucketName)
 		}
 
-		globalPolicySys.Set(args.BucketName, *bucketPolicy)
+		globalPolicySys.Set(args.BucketName, bucketPolicy)
 		globalNotificationSys.SetBucketPolicy(ctx, args.BucketName, bucketPolicy)
 	}
 
@@ -2058,12 +2032,6 @@ type LoginSTSArgs struct {
 	Token string `json:"token" form:"token"`
 }
 
-// LoginSTSRep - login reply.
-type LoginSTSRep struct {
-	Token     string `json:"token"`
-	UIVersion string `json:"uiVersion"`
-}
-
 // LoginSTS - STS user login handler.
 func (web *webAPIHandlers) LoginSTS(r *http.Request, args *LoginSTSArgs, reply *LoginRep) error {
 	ctx := newWebContext(r, args, "WebLoginSTS")
@@ -2074,6 +2042,9 @@ func (web *webAPIHandlers) LoginSTS(r *http.Request, args *LoginSTSArgs, reply *
 	v.Set("Version", stsAPIVersion)
 
 	scheme := "http"
+	if sourceScheme := handlers.GetSourceScheme(r); sourceScheme != "" {
+		scheme = sourceScheme
+	}
 	if globalIsSSL {
 		scheme = "https"
 	}
@@ -2172,7 +2143,7 @@ func toWebAPIError(ctx context.Context, err error) APIError {
 			Description:    err.Error(),
 		}
 	case errAuthentication, auth.ErrInvalidAccessKeyLength,
-		auth.ErrInvalidSecretKeyLength, errInvalidAccessKeyID:
+		auth.ErrInvalidSecretKeyLength, errInvalidAccessKeyID, errAccessDenied, errLockedObject:
 		return APIError{
 			Code:           "AccessDenied",
 			HTTPStatusCode: http.StatusForbidden,

cmd/web-handlers_test.go

@@ -242,6 +242,7 @@ func testServerInfoWebHandler(obj ObjectLayer, instanceType string, t TestErrHan
 	if serverInfoReply.MinioVersion != Version {
 		t.Fatalf("Cannot get minio version from server info handler")
 	}
+	serverInfoReply.MinioGlobalInfo["domains"] = []string(nil)
 	globalInfo := getGlobalInfo()
 	if !reflect.DeepEqual(serverInfoReply.MinioGlobalInfo, globalInfo) {
 		t.Fatalf("Global info did not match got %#v, expected %#v", serverInfoReply.MinioGlobalInfo, globalInfo)
@@ -547,7 +548,7 @@ func testListObjectsWebHandler(obj ObjectLayer, instanceType string, t TestErrHa
 	if err = obj.SetBucketPolicy(context.Background(), bucketName, bucketPolicy); err != nil {
 		t.Fatalf("unexpected error. %v", err)
 	}
-	globalPolicySys.Set(bucketName, *bucketPolicy)
+	globalPolicySys.Set(bucketName, bucketPolicy)
 	defer globalPolicySys.Remove(bucketName)
 
 	// Unauthenticated ListObjects with READ bucket policy should succeed.
@@ -905,7 +906,7 @@ func testUploadWebHandler(obj ObjectLayer, instanceType string, t TestErrHandler
 	if err := obj.SetBucketPolicy(context.Background(), bucketName, bucketPolicy); err != nil {
 		t.Fatalf("unexpected error. %v", err)
 	}
-	globalPolicySys.Set(bucketName, *bucketPolicy)
+	globalPolicySys.Set(bucketName, bucketPolicy)
 	defer globalPolicySys.Remove(bucketName)
 
 	// Unauthenticated upload with WRITE policy should succeed.
@@ -1024,7 +1025,7 @@ func testDownloadWebHandler(obj ObjectLayer, instanceType string, t TestErrHandl
 	if err := obj.SetBucketPolicy(context.Background(), bucketName, bucketPolicy); err != nil {
 		t.Fatalf("unexpected error. %v", err)
 	}
-	globalPolicySys.Set(bucketName, *bucketPolicy)
+	globalPolicySys.Set(bucketName, bucketPolicy)
 	defer globalPolicySys.Remove(bucketName)
 
 	// Unauthenticated download with READ policy should succeed.

cmd/xl-sets.go

@@ -22,17 +22,18 @@ import (
 	"hash/crc32"
 	"io"
 	"net/http"
+	"sort"
 	"strings"
 	"sync"
 	"time"
 
+	"github.com/minio/minio-go/v6/pkg/tags"
 	"github.com/minio/minio/cmd/config/storageclass"
 	xhttp "github.com/minio/minio/cmd/http"
 	"github.com/minio/minio/cmd/logger"
 	"github.com/minio/minio/pkg/bpool"
 	bucketsse "github.com/minio/minio/pkg/bucket/encryption"
 	"github.com/minio/minio/pkg/bucket/lifecycle"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 	"github.com/minio/minio/pkg/dsync"
 	"github.com/minio/minio/pkg/madmin"
@@ -1689,20 +1690,18 @@ func (s *xlSets) HealObject(ctx context.Context, bucket, object string, opts mad
 
 // Lists all buckets which need healing.
 func (s *xlSets) ListBucketsHeal(ctx context.Context) ([]BucketInfo, error) {
-	listBuckets := []BucketInfo{}
-	var healBuckets = map[string]BucketInfo{}
+	var listBuckets []BucketInfo
+	var healBuckets = make(map[string]VolInfo)
 	for _, set := range s.sets {
-		buckets, _, err := listAllBuckets(set.getDisks())
-		if err != nil {
+		// lists all unique buckets across drives.
+		if err := listAllBuckets(set.getDisks(), healBuckets); err != nil {
 			return nil, err
 		}
-		for _, currBucket := range buckets {
-			healBuckets[currBucket.Name] = BucketInfo(currBucket)
-		}
 	}
-	for _, bucketInfo := range healBuckets {
-		listBuckets = append(listBuckets, bucketInfo)
+	for _, v := range healBuckets {
+		listBuckets = append(listBuckets, BucketInfo(v))
 	}
+	sort.Sort(byBucketName(listBuckets))
 	return listBuckets, nil
 }
 
@@ -1785,7 +1784,7 @@ func (s *xlSets) DeleteObjectTag(ctx context.Context, bucket, object string) err
 }
 
 // GetObjectTag - get object tags from an existing object
-func (s *xlSets) GetObjectTag(ctx context.Context, bucket, object string) (tagging.Tagging, error) {
+func (s *xlSets) GetObjectTag(ctx context.Context, bucket, object string) (*tags.Tags, error) {
 	return s.getHashedSet(object).GetObjectTag(ctx, bucket, object)
 }
 

cmd/xl-v1-healing.go

@@ -173,11 +173,7 @@ func healBucket(ctx context.Context, storageDisks []StorageAPI, bucket string, w
 
 // listAllBuckets lists all buckets from all disks. It also
 // returns the occurrence of each buckets in all disks
-func listAllBuckets(storageDisks []StorageAPI) (buckets map[string]VolInfo,
-	bucketsOcc map[string]int, err error) {
-
-	buckets = make(map[string]VolInfo)
-	bucketsOcc = make(map[string]int)
+func listAllBuckets(storageDisks []StorageAPI, healBuckets map[string]VolInfo) (err error) {
 	for _, disk := range storageDisks {
 		if disk == nil {
 			continue
@@ -188,7 +184,7 @@ func listAllBuckets(storageDisks []StorageAPI) (buckets map[string]VolInfo,
 			if IsErrIgnored(err, bucketMetadataOpIgnoredErrs...) {
 				continue
 			}
-			return nil, nil, err
+			return err
 		}
 		for _, volInfo := range volsInfo {
 			// StorageAPI can send volume names which are
@@ -197,13 +193,14 @@ func listAllBuckets(storageDisks []StorageAPI) (buckets map[string]VolInfo,
 			if isReservedOrInvalidBucket(volInfo.Name, false) {
 				continue
 			}
-			// Increase counter per bucket name
-			bucketsOcc[volInfo.Name]++
-			// Save volume info under bucket name
-			buckets[volInfo.Name] = volInfo
+			// always save unique buckets across drives.
+			if _, ok := healBuckets[volInfo.Name]; !ok {
+				healBuckets[volInfo.Name] = volInfo
+			}
+
 		}
 	}
-	return buckets, bucketsOcc, nil
+	return nil
 }
 
 // Only heal on disks where we are sure that healing is needed. We can expand

cmd/xl-v1-object.go

@@ -25,9 +25,9 @@ import (
 	"path"
 	"sync"
 
+	"github.com/minio/minio-go/v6/pkg/tags"
 	xhttp "github.com/minio/minio/cmd/http"
 	"github.com/minio/minio/cmd/logger"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/mimedb"
 	"github.com/minio/minio/pkg/sync/errgroup"
 )
@@ -1018,7 +1018,7 @@ func (xl xlObjects) PutObjectTag(ctx context.Context, bucket, object string, tag
 
 	_, writeQuorum, err := objectQuorumFromMeta(ctx, xl, metaArr, errs)
 	if err != nil {
-		return err
+		return toObjectErr(err, bucket, object)
 	}
 
 	for i, xlMeta := range metaArr {
@@ -1052,16 +1052,12 @@ func (xl xlObjects) DeleteObjectTag(ctx context.Context, bucket, object string)
 }
 
 // GetObjectTag - get object tags from an existing object
-func (xl xlObjects) GetObjectTag(ctx context.Context, bucket, object string) (tagging.Tagging, error) {
+func (xl xlObjects) GetObjectTag(ctx context.Context, bucket, object string) (*tags.Tags, error) {
 	// GetObjectInfo will return tag value as well
 	oi, err := xl.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
 	if err != nil {
-		return tagging.Tagging{}, err
+		return nil, err
 	}
 
-	tags, err := tagging.FromString(oi.UserTags)
-	if err != nil {
-		return tagging.Tagging{}, err
-	}
-	return tags, nil
+	return tags.ParseObjectTags(oi.UserTags)
 }

cmd/xl-zones.go

@@ -26,11 +26,11 @@ import (
 	"sync"
 	"time"
 
+	"github.com/minio/minio-go/v6/pkg/tags"
 	xhttp "github.com/minio/minio/cmd/http"
 	"github.com/minio/minio/cmd/logger"
 	bucketsse "github.com/minio/minio/pkg/bucket/encryption"
 	"github.com/minio/minio/pkg/bucket/lifecycle"
-	"github.com/minio/minio/pkg/bucket/object/tagging"
 	"github.com/minio/minio/pkg/bucket/policy"
 	"github.com/minio/minio/pkg/madmin"
 	"github.com/minio/minio/pkg/sync/errgroup"
@@ -44,16 +44,6 @@ func (z *xlZones) SingleZone() bool {
 	return len(z.zones) == 1
 }
 
-func (z *xlZones) quickHealBuckets(ctx context.Context) {
-	bucketsInfo, err := z.ListBucketsHeal(ctx)
-	if err != nil {
-		return
-	}
-	for _, bucket := range bucketsInfo {
-		z.MakeBucketWithLocation(ctx, bucket.Name, "")
-	}
-}
-
 // Initialize new zone of erasure sets.
 func newXLZones(ctx context.Context, endpointZones EndpointZones) (ObjectLayer, error) {
 	var (
@@ -88,7 +78,6 @@ func newXLZones(ctx context.Context, endpointZones EndpointZones) (ObjectLayer,
 		}
 	}
 
-	z.quickHealBuckets(ctx)
 	go intDataUpdateTracker.start(GlobalContext, localDrives...)
 	return z, nil
 }
@@ -1588,7 +1577,7 @@ func (z *xlZones) DeleteObjectTag(ctx context.Context, bucket, object string) er
 }
 
 // GetObjectTag - get object tags from an existing object
-func (z *xlZones) GetObjectTag(ctx context.Context, bucket, object string) (tagging.Tagging, error) {
+func (z *xlZones) GetObjectTag(ctx context.Context, bucket, object string) (*tags.Tags, error) {
 	if z.SingleZone() {
 		return z.zones[0].GetObjectTag(ctx, bucket, object)
 	}
@@ -1602,7 +1591,7 @@ func (z *xlZones) GetObjectTag(ctx context.Context, bucket, object string) (tagg
 		}
 		return tags, nil
 	}
-	return tagging.Tagging{}, BucketNotFound{
+	return nil, BucketNotFound{
 		Bucket: bucket,
 	}
 }

docs/deployment/kernel-tuning/README.md

@@ -1,81 +1,98 @@
 # Kernel Tuning for MinIO Production Deployment on Linux Servers [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
 
-## Tuning Network Parameters
+Following are the recommended settings, a copy of this [script](https://github.com/minio/minio/blob/master/docs/deployment/kernel-tuning/sysctl.sh) is available here to be applied on Linux servers.
 
-Following network parameter settings can help ensure optimal MinIO server performance on production workloads.
+> NOTE: Although these settings are generally good on Linux servers, users must be careful on any premature tuning. These tunings are generally considered good to have but not mandatory, these settings do not fix any hardware issues and should not be considered as an alternative to boost performance. Under most circumstances this tuning is to be done after performing baseline performance tests for the hardware with expected results.
 
-- *`tcp_fin_timeout`* : A socket left in memory takes approximately 1.5Kb of memory. It makes sense to close the unused sockets preemptively to ensure no memory leakage. This way, even if a peer doesn't close the socket due to some reason, the system itself closes it after a timeout. `tcp_fin_timeout` variable defines this timeout and tells kernel how long to keep sockets in the state FIN-WAIT-2. We recommend setting it to 30. You can set it as shown below
-
-```sh
-sysctl -w net.ipv4.tcp_fin_timeout=30
 ```
+#!/bin/bash
+cat > sysctl.conf <<EOF
+# maximum number of open files/file descriptors
+fs.file-max = 4194303
 
-- *`tcp_keepalive_probes`* : This variable defines the number of unacknowledged probes to be sent before considering a connection dead. You can set it as shown below
+# use as little swap space as possible
+vm.swappiness = 1
 
-```sh
-sysctl -w net.ipv4.tcp_keepalive_probes=5
-```
+# prioritize application RAM against disk/swap cache
+vm.vfs_cache_pressure = 10
 
-- *`wmem_max`*: This parameter sets the max OS send buffer size for all types of connections.
+# minimum free memory
+vm.min_free_kbytes = 1000000
 
-```sh
-sysctl -w net.core.wmem_max=540000
-```
+# maximum receive socket buffer (bytes)
+net.core.rmem_max = 268435456
 
-- *`rmem_max`*: This parameter sets the max OS receive buffer size for all types of connections.
+# maximum send buffer socket buffer (bytes)
+net.core.wmem_max = 268435456
 
-```sh
-sysctl -w net.core.rmem_max=540000
-```
+# default receive buffer socket size (bytes)
+net.core.rmem_default = 67108864
 
-## Tuning Virtual Memory
+# default send buffer socket size (bytes)
+net.core.wmem_default = 67108864
 
-Recommended virtual memory settings are as follows.
+# maximum number of packets in one poll cycle
+net.core.netdev_budget = 1200
 
-- *`swappiness`* : This parameter controls the relative weight given to swapping out runtime memory, as opposed to dropping pages from the system page cache. It takes values from 0 to 100, both inclusive. We recommend setting it to 10.
+# maximum ancillary buffer size per socket
+net.core.optmem_max = 134217728
 
-```sh
-sysctl -w vm.swappiness=1
-```
+# maximum number of incoming connections
+net.core.somaxconn = 65535
 
-- *`dirty_background_ratio`*: This is the percentage of system memory that can be filled with `dirty` pages, i.e. memory pages that still need to be written to disk. We recommend writing the data to the disk as soon as possible. To do this, set the `dirty_background_ratio` to 1.
+# maximum number of packets queued
+net.core.netdev_max_backlog = 250000
 
-```sh
-sysctl -w vm.dirty_background_ratio=1
-```
+# maximum read buffer space
+net.ipv4.tcp_rmem = 67108864 134217728 268435456
 
-- *`dirty_ratio`*: This defines is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.
+# maximum write buffer space
+net.ipv4.tcp_wmem = 67108864 134217728 268435456
 
-```sh
-sysctl -w vm.dirty_ratio=5
-```
+# enable low latency mode
+net.ipv4.tcp_low_latency = 1
 
-- *`Transparent Hugepage Support`*: This is a Linux kernel feature intended to improve performance by making more efficient use of processor’s memory-mapping hardware. But this may cause [problems](https://blogs.oracle.com/linux/performance-issues-with-transparent-huge-pages-thp) for non-optimized applications. As most Linux distributions set it to `enabled=always` by default, we recommend changing this to `enabled=madvise`. This will allow applications optimized for transparent hugepages to obtain the performance benefits, while preventing the associated problems otherwise.
+# socket buffer portion used for TCP window
+net.ipv4.tcp_adv_win_scale = 1
 
-```sh
+# queue length of completely established sockets waiting for accept
+net.ipv4.tcp_max_syn_backlog = 30000
+
+# maximum number of sockets in TIME_WAIT state
+net.ipv4.tcp_max_tw_buckets = 2000000
+
+# reuse sockets in TIME_WAIT state when safe
+net.ipv4.tcp_tw_reuse = 1
+
+# time to wait (seconds) for FIN packet
+net.ipv4.tcp_fin_timeout = 5
+
+# disable icmp send redirects
+net.ipv4.conf.all.send_redirects = 0
+
+# disable icmp accept redirect
+net.ipv4.conf.all.accept_redirects = 0
+
+# drop packets with LSR or SSR
+net.ipv4.conf.all.accept_source_route = 0
+
+# MTU discovery, only enable when ICMP blackhole detected
+net.ipv4.tcp_mtu_probing = 1
+
+EOF
+
+echo "Enabling system level tuning params"
+sysctl --quiet --load sysctl.conf && rm -f sysctl.conf
+
+# `Transparent Hugepage Support`*: This is a Linux kernel feature intended to improve
+# performance by making more efficient use of processor’s memory-mapping hardware.
+# But this may cause https://blogs.oracle.com/linux/performance-issues-with-transparent-huge-pages-thp
+# for non-optimized applications. As most Linux distributions set it to `enabled=always` by default,
+# we recommend changing this to `enabled=madvise`. This will allow applications optimized
+# for transparent hugepages to obtain the performance benefits, while preventing the
+# associated problems otherwise. Also, set `transparent_hugepage=madvise` on your kernel
+# command line (e.g. in /etc/default/grub) to persistently set this value.
+
+echo "Enabling THP madvise"
 echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
 ```
-
-Also, set `transparent_hugepage=madvise` on your kernel command line (e.g. in /etc/default/grub) to persistently set this value.
-
-All these system level tunings are conveniently packaged in [shell script](https://github.com/minio/minio/blob/master/docs/deployment/kernel-tuning/sysctl.sh). Please review the shell script for our recommendations.
-
-## Tuning Scheduler
-
-Proper scheduler configuration makes sure MinIO process gets adequate CPU time. Here are the recommended scheduler settings
-
-- *`sched_min_granularity_ns`*: This parameter decides the minimum time a task will be be allowed to run on CPU before being pre-empted out. We recommend setting it to 10ms.
-
-```sh
-sysctl -w kernel.sched_min_granularity_ns=10000000
-```
-
-- *`sched_wakeup_granularity_ns`*: Lowering this parameter improves wake-up latency and throughput for latency critical tasks, particularly when a short duty cycle load component must compete with CPU bound components.
-
-```sh
-sysctl -w kernel.sched_wakeup_granularity_ns=15000000
-```
-
-## Tuning Disks
-
-The recommendations for disk tuning are conveniently packaged in a well commented [shell script](https://github.com/minio/minio/blob/master/docs/deployment/kernel-tuning/disk-tuning.sh). Please review the shell script for our recommendations.

docs/deployment/kernel-tuning/disk-tuning.sh

@@ -1,50 +0,0 @@
-#!/bin/bash
-
-## MinIO Cloud Storage, (C) 2017, 2018 MinIO, Inc.
-##
-## Licensed under the Apache License, Version 2.0 (the "License");
-## you may not use this file except in compliance with the License.
-## You may obtain a copy of the License at
-##
-##     http://www.apache.org/licenses/LICENSE-2.0
-##
-## Unless required by applicable law or agreed to in writing, software
-## distributed under the License is distributed on an "AS IS" BASIS,
-## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-## See the License for the specific language governing permissions and
-## limitations under the License.
-
-# This script changes protected files, and must be run as root
-
-for i in $(echo /sys/block/*/queue/iosched 2>/dev/null); do
-    iosched_dir=$(echo "${i}" | awk '/iosched/ {print $1}')
-    [ -z "${iosched_dir}" ] && {
-        continue
-    }
-    ## Change each disk ioscheduler to be "deadline"
-    ## Deadline dispatches I/Os in batches. A batch is a
-    ## sequence of either read or write I/Os which are in
-    ## increasing LBA order (the one-way elevator). After
-    ## processing each batch, the I/O scheduler checks to
-    ## see whether write requests have been starved for too
-    ## long, and then decides whether to start a new batch
-    ## of reads or writes
-    path=$(dirname "${iosched_dir}")
-    [ -f "${path}/scheduler" ] && {
-        echo "deadline" > "${path}/scheduler" 2>/dev/null || true
-    }
-    ## This controls how many requests may be allocated
-    ## in the block layer for read or write requests.
-    ## Note that the total allocated number may be twice
-    ## this amount, since it applies only to reads or
-    ## writes (not the accumulate sum).
-    [ -f "${path}/nr_requests" ] && {
-        echo "256" > "${path}/nr_requests" 2>/dev/null || true
-    }
-    ## This is the maximum number of kilobytes
-    ## supported in a single data transfer at
-    ## block layer.
-    [ -f "${path}/max_sectors_kb" ] && {
-        echo "1024" > "${path}/max_sectors_kb" 2>/dev/null || true
-    }
-done

docs/gateway/README.md

@@ -2,7 +2,7 @@
 MinIO Gateway adds Amazon S3 compatibility to third party cloud storage providers.
 - [NAS](https://github.com/minio/minio/blob/master/docs/gateway/nas.md)
 - [Microsoft Azure Blob Storage](https://github.com/minio/minio/blob/master/docs/gateway/azure.md)
+- [HDFS](https://github.com/minio/minio/blob/master/docs/gateway/hdfs.md)
 - [S3](https://github.com/minio/minio/blob/master/docs/gateway/s3.md)
 - [Google Cloud Storage](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md)
-- [Backblaze B2](https://github.com/minio/minio/blob/master/docs/gateway/b2.md)
 

docs/gateway/b2.md

@@ -1,60 +0,0 @@
-# MinIO B2 Gateway [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
-MinIO Gateway adds Amazon S3 compatibility to Backblaze B2 Cloud Storage.
-
-## Run MinIO Gateway for Backblaze B2 Cloud Storage
-Please follow this [guide](https://www.backblaze.com/b2/docs/quick_account.html) to create an account on backblaze.com to obtain your access credentials for B2 Cloud storage.
-
-### Using Docker
-```
-docker run -p 9000:9000 --name b2-s3 \
- -e "MINIO_ACCESS_KEY=b2_account_id" \
- -e "MINIO_SECRET_KEY=b2_application_key" \
- minio/minio gateway b2
-```
-
-### Using Binary
-```
-export MINIO_ACCESS_KEY=b2_account_id
-export MINIO_SECRET_KEY=b2_application_key
-minio gateway b2
-```
-
-## Test using MinIO Browser
-MinIO Gateway comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 to ensure that your server has started successfully.
-
-![Screenshot](https://raw.githubusercontent.com/minio/minio/master/docs/screenshots/minio-browser-gateway.png)
-
-## Test using MinIO Client `mc`
-`mc` provides a modern alternative to UNIX commands such as ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services.
-
-### Configure `mc`
-```
-mc config host add myb2 http://gateway-ip:9000 b2_account_id b2_application_key
-```
-
-### List buckets on Backblaze B2
-```
-mc ls myb2
-[2017-02-22 01:50:43 PST]     0B ferenginar/
-[2017-02-26 21:43:51 PST]     0B my-bucket/
-[2017-02-26 22:10:11 PST]     0B test-bucket1/
-```
-
-### Known limitations
-Gateway inherits the following B2 limitations:
-
-- No support for CopyObject S3 API (yet).
-- No support for CopyObjectPart S3 API (yet).
-- Only read-only bucket policy supported at bucket level, all other variations will return API Notimplemented error.
-- DeleteObject() might not delete the object right away on Backblaze B2, so you might see the object immediately after a Delete request.
-- Multipart uploads don't return an MD5-hash ETag, not even with the `--compat` flag.
-
-Other limitations:
-
-- Bucket notification APIs are not supported.
-- The PutObject S3 API only returns the md5sum of an uploaded file as it's ETag response when minio is started in `--compat` mode. Don't use this unless you are stuck with a client library where this check cannot be turned off (e.g. boto).
-
-## Explore Further
-- [`mc` command-line interface](https://docs.min.io/docs/minio-client-quickstart-guide)
-- [`aws` command-line interface](https://docs.min.io/docs/aws-cli-with-minio)
-- [`minio-go` Go SDK](https://docs.min.io/docs/golang-client-quickstart-guide)

docs/orchestration/docker-compose/docker-compose.yaml

@@ -5,7 +5,7 @@ version: '3.7'
 # 9001 through 9004.
 services:
   minio1:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     volumes:
       - data1-1:/data1
       - data1-2:/data2
@@ -22,7 +22,7 @@ services:
       retries: 3
 
   minio2:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     volumes:
       - data2-1:/data1
       - data2-2:/data2
@@ -39,7 +39,7 @@ services:
       retries: 3
 
   minio3:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     volumes:
       - data3-1:/data1
       - data3-2:/data2
@@ -56,7 +56,7 @@ services:
       retries: 3
 
   minio4:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     volumes:
       - data4-1:/data1
       - data4-2:/data2

docs/orchestration/docker-swarm/docker-compose-secrets.yaml

@@ -2,7 +2,7 @@ version: '3.7'
 
 services:
   minio1:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio1
     volumes:
       - minio1-data:/export
@@ -29,7 +29,7 @@ services:
       retries: 3
 
   minio2:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio2
     volumes:
       - minio2-data:/export
@@ -56,7 +56,7 @@ services:
       retries: 3
 
   minio3:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio3
     volumes:
       - minio3-data:/export
@@ -83,7 +83,7 @@ services:
       retries: 3
 
   minio4:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio4
     volumes:
       - minio4-data:/export

docs/orchestration/docker-swarm/docker-compose.yaml

@@ -2,7 +2,7 @@ version: '3.7'
 
 services:
   minio1:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio1
     volumes:
       - minio1-data:/export
@@ -33,7 +33,7 @@ services:
       retries: 3
 
   minio2:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio2
     volumes:
       - minio2-data:/export
@@ -64,7 +64,7 @@ services:
       retries: 3
 
   minio3:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio3
     volumes:
       - minio3-data:/export
@@ -95,7 +95,7 @@ services:
       retries: 3
 
   minio4:
-    image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+    image: minio/minio:RELEASE.2020-05-06T23-23-25Z
     hostname: minio4
     volumes:
       - minio4-data:/export

docs/orchestration/kubernetes/minio-distributed-daemonset.yaml

@@ -30,7 +30,7 @@ spec:
           value: "minio"
         - name: MINIO_SECRET_KEY
           value: "minio123"
-        image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+        image: minio/minio:RELEASE.2020-05-06T23-23-25Z
         # Unfortunately you must manually define each server. Perhaps autodiscovery via DNS can be implemented in the future.
         args:
         - server

docs/orchestration/kubernetes/minio-distributed-statefulset.yaml

@@ -22,7 +22,7 @@ spec:
           value: "minio"
         - name: MINIO_SECRET_KEY
           value: "minio123"
-        image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+        image: minio/minio:RELEASE.2020-05-06T23-23-25Z
         args:
         - server
         - http://minio-{0...3}.minio.default.svc.cluster.local/data

docs/orchestration/kubernetes/minio-gcs-gateway-deployment.yaml

@@ -24,7 +24,7 @@ spec:
       containers:
       - name: minio
         # Pulls the default Minio image from Docker Hub
-        image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+        image: minio/minio:RELEASE.2020-05-06T23-23-25Z
         args:
         - gateway
         - gcs

docs/orchestration/kubernetes/minio-standalone-deployment.yaml

@@ -32,7 +32,7 @@ spec:
         - name: data 
           mountPath: "/data"
         # Pulls the lastest Minio image from Docker Hub
-        image: minio/minio:RELEASE.2020-04-28T23-56-56Z
+        image: minio/minio:RELEASE.2020-05-06T23-23-25Z
         args:
         - server
         - /data

docs/sts/ldap.md

@@ -38,7 +38,7 @@ LDAP configuration is designed to be simple for the MinIO administrator. The ful
 
 MinIO can be configured to find the groups of a user from AD/LDAP by specifying the **MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER** and **MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE** environment variables. When a user logs in via the STS API, the MinIO server queries the AD/LDAP server with the given search filter and extracts the given attribute from the search results. These values represent the groups that the user is a member of. On each access MinIO applies the IAM policies attached to these groups in MinIO.
 
-MinIO sends LDAP credentials to LDAP server for validation. So we _strongly recommend_ to use MinIO with AD/LDAP server over TLS _only_. Using plain-text connection between MinIO and LDAP server means _credentials can be compromised_ by anyone listening to network traffic.
+MinIO sends LDAP credentials to LDAP server for validation. So we _strongly recommend_ to use MinIO with AD/LDAP server over TLS or StartTLS _only_. Using plain-text connection between MinIO and LDAP server means _credentials can be compromised_ by anyone listening to network traffic.
 
 LDAP is configured via the following environment variables:
 
@@ -57,11 +57,12 @@ MINIO_IDENTITY_LDAP_USERNAME_SEARCH_BASE_DN  (list)      ";" separated list of u
 MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE     (string)    search attribute for group name e.g. "cn"
 MINIO_IDENTITY_LDAP_STS_EXPIRY               (duration)  temporary credentials validity duration in s,m,h,d. Default is "1h"
 MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY          (on|off)    trust server TLS without verification, defaults to "off" (verify)
+MINIO_IDENTITY_LDAP_SERVER_STARTTLS          (on|off)    use StartTLS instead of TLS
 MINIO_IDENTITY_LDAP_SERVER_INSECURE          (on|off)    allow plain text connection to AD/LDAP server, defaults to "off"
 MINIO_IDENTITY_LDAP_COMMENT                  (sentence)  optionally add a comment to this setting
 ```
 
-MinIO sends LDAP credentials to LDAP server for validation. So we _strongly recommend_ to use MinIO with AD/LDAP server over TLS _only_. Using plain-text connection between MinIO and LDAP server means _credentials can be compromised_ by anyone listening to network traffic.
+MinIO sends LDAP credentials to LDAP server for validation. So we _strongly recommend_ to use MinIO with AD/LDAP server over TLS or StartTLS _only_. Using plain-text connection between MinIO and LDAP server means _credentials can be compromised_ by anyone listening to network traffic.
 
 If a self-signed certificate is being used, the certificate can be added to MinIO's certificates directory, so it can be trusted by the server. An example setup for development or experimentation:
 

go.mod

@@ -55,7 +55,6 @@ require (
 	github.com/klauspost/pgzip v1.2.1
 	github.com/klauspost/readahead v1.3.1
 	github.com/klauspost/reedsolomon v1.9.3
-	github.com/kurin/blazer v0.5.4-0.20200327014341-8f90a40f8af7
 	github.com/lib/pq v1.1.1
 	github.com/mattn/go-colorable v0.1.1
 	github.com/mattn/go-ieproxy v0.0.0-20190805055040-f9202b1cfdeb // indirect
@@ -67,7 +66,7 @@ require (
 	github.com/minio/hdfs/v3 v3.0.1
 	github.com/minio/highwayhash v1.0.0
 	github.com/minio/lsync v1.0.1
-	github.com/minio/minio-go/v6 v6.0.55-0.20200424204115-7506d2996b22
+	github.com/minio/minio-go/v6 v6.0.55-0.20200425081427-89eebdef2af0
 	github.com/minio/parquet-go v0.0.0-20200414234858-838cfa8aae61
 	github.com/minio/sha256-simd v0.1.1
 	github.com/minio/simdjson-go v0.1.5-0.20200303142138-b17fe061ea37
@@ -110,9 +109,7 @@ require (
 	github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c
 	github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
 	go.etcd.io/bbolt v1.3.3 // indirect
-	go.uber.org/atomic v1.3.2
-	go.uber.org/multierr v1.1.0 // indirect
-	go.uber.org/zap v1.10.0 // indirect
+	go.uber.org/zap v1.15.0 // indirect
 	golang.org/x/crypto v0.0.0-20191117063200-497ca9f6d64f
 	golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
 	golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a // indirect

go.sum

@@ -47,7 +47,6 @@ github.com/cheggaaa/pb v1.0.28 h1:kWGpdAcSp3MxMU9CCHOwz/8V0kCHN4+9yQm2MzWuI98=
 github.com/cheggaaa/pb v1.0.28/go.mod h1:pQciLPpbU0oxA0h+VJYYLxO+XeDQb5pZijXscXHm81s=
 github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag=
 github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I=
-github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
 github.com/coredns/coredns v1.4.0 h1:RubBkYmkByUqZWWkjRHvNLnUHgkRVqAWgSMmRFvpE1A=
 github.com/coredns/coredns v1.4.0/go.mod h1:zASH/MVDgR6XZTbxvOnsZfffS+31vg6Ackf/wo1+AM0=
@@ -85,15 +84,12 @@ github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
 github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
-github.com/fortytw2/leaktest v1.2.0 h1:cj6GCiwJDH7l3tMHLjZDo0QqPtrXJiWSI9JgpeQKw+Q=
 github.com/fortytw2/leaktest v1.2.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
 github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
 github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
-github.com/frankban/quicktest v1.4.1 h1:Wv2VwvNn73pAdFIVUQRXYDFp31lXKbqblIXo/Q5GPSg=
 github.com/frankban/quicktest v1.4.1/go.mod h1:36zfPVQyHxymz4cH7wlDmVwDrJuljRB60qkgn7rorfQ=
 github.com/frankban/quicktest v1.7.2 h1:2QxQoC1TS09S7fhCPsrvqYdvP1H5M1P1ih5ABm3BTYk=
 github.com/frankban/quicktest v1.7.2/go.mod h1:jaStnuzAqU1AJdCO0l53JDCJrVDKcS03DbaAcR7Ks/o=
-github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32 h1:Mn26/9ZMNWSw9C9ERFA1PUxfmGpolnw2v0bKOREu5ew=
 github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32/go.mod h1:GIjDIg/heH5DOkXY3YJ/wNhfHsQHoXGjl8G8amsYQ1I=
@@ -117,7 +113,6 @@ github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfb
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/protobuf v1.1.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -127,18 +122,15 @@ github.com/gomodule/redigo v2.0.0+incompatible h1:K/R+8tc58AaqLkqG2Ol3Qk+DR/TlNu
 github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
 github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c h1:964Od4U6p2jUkFxvCydnIczKteheJEzHRToSGK3Bnlw=
 github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
-github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
-github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
-github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
-github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57 h1:eqyIo2HjKhKe/mJzTG8n4VqvLXIOEG+SLdDqX7xGtkY=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
+github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
 github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/googleapis/gax-go/v2 v2.0.4 h1:hU4mGcQI4DaAYW+IbTun+2qEZVFxK0ySjQLTbS0VQKc=
@@ -168,7 +160,6 @@ github.com/hashicorp/go-cleanhttp v0.5.1 h1:dH3aiDG9Jvb5r5+bYHsikaOUIpcM0xvgMXVo
 github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
 github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd/go.mod h1:9bjs9uLqI8l75knNv3lV1kA55veR+WUPSiKIWcQHudI=
 github.com/hashicorp/go-hclog v0.8.0/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
-github.com/hashicorp/go-hclog v0.9.1 h1:9PZfAcVEvez4yhLH2TBU64/h/z4xlFI80cWXRrxuKuM=
 github.com/hashicorp/go-hclog v0.9.1/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
 github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI=
 github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
@@ -179,7 +170,6 @@ github.com/hashicorp/go-msgpack v0.5.5/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iP
 github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o=
 github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
 github.com/hashicorp/go-plugin v1.0.1/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
-github.com/hashicorp/go-retryablehttp v0.5.3 h1:QlWt0KvWT0lq8MFppF9tsJGF+ynG7ztc2KIPhzRGk7s=
 github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
 github.com/hashicorp/go-retryablehttp v0.5.4 h1:1BZvpawXoJCWX6pNtow9+rpEj+3itIlutiqnntI6jOE=
 github.com/hashicorp/go-retryablehttp v0.5.4/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
@@ -223,7 +213,6 @@ github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvW
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.8.2/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
 github.com/klauspost/compress v1.9.4/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
-github.com/klauspost/compress v1.10.1 h1:a/QY0o9S6wCi0XhxaMX/QmusicNUqCqFugR6WKPOSoQ=
 github.com/klauspost/compress v1.10.1/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/klauspost/compress v1.10.3 h1:OP96hzwJVBIHYU52pVTI6CczrxPvrGfgqF9N5eTO0Q8=
 github.com/klauspost/compress v1.10.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
@@ -243,8 +232,6 @@ github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORN
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
-github.com/kurin/blazer v0.5.4-0.20200327014341-8f90a40f8af7 h1:smZXPopqRVVywwzou4WYWvUbJvSAzIDFizfWElpmAqY=
-github.com/kurin/blazer v0.5.4-0.20200327014341-8f90a40f8af7/go.mod h1:4FCXMUWo9DllR2Do4TtBd377ezyAJ51vB5uTBjt0pGU=
 github.com/lib/pq v1.1.1 h1:sJZmqHoEaY7f+NPP8pgLB/WxulyR3fewgCM2qaSlBb4=
 github.com/lib/pq v1.1.1/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
 github.com/mailru/easyjson v0.0.0-20180730094502-03f2033d19d5 h1:0x4qcEHDpruK6ML/m/YSlFUUu0UpRD3I2PHsNCuGnyA=
@@ -275,10 +262,9 @@ github.com/minio/highwayhash v1.0.0 h1:iMSDhgUILCr0TNm8LWlSjF8N0ZIj2qbO8WHp6Q/J2
 github.com/minio/highwayhash v1.0.0/go.mod h1:xQboMTeM9nY9v/LlAOxFctujiv5+Aq2hR5dxBpaMbdc=
 github.com/minio/lsync v1.0.1 h1:AVvILxA976xc27hstd1oR+X9PQG0sPSom1MNb1ImfUs=
 github.com/minio/lsync v1.0.1/go.mod h1:tCFzfo0dlvdGl70IT4IAK/5Wtgb0/BrTmo/jE8pArKA=
-github.com/minio/minio-go/v6 v6.0.53 h1:8jzpwiOzZ5Iz7/goFWqNZRICbyWYShbb5rARjrnSCNI=
 github.com/minio/minio-go/v6 v6.0.53/go.mod h1:DIvC/IApeHX8q1BAMVCXSXwpmrmM+I+iBvhvztQorfI=
-github.com/minio/minio-go/v6 v6.0.55-0.20200424204115-7506d2996b22 h1:nZEve4vdUhwHBoV18zRvPDgjL6NYyDJE5QJvz3l9bRs=
-github.com/minio/minio-go/v6 v6.0.55-0.20200424204115-7506d2996b22/go.mod h1:KQMM+/44DSlSGSQWSfRrAZ12FVMmpWNuX37i2AX0jfI=
+github.com/minio/minio-go/v6 v6.0.55-0.20200425081427-89eebdef2af0 h1:PdHKpM9h2vqCDr1AjJdK8e/6tRdOSjUNzIqeNmxu7ak=
+github.com/minio/minio-go/v6 v6.0.55-0.20200425081427-89eebdef2af0/go.mod h1:KQMM+/44DSlSGSQWSfRrAZ12FVMmpWNuX37i2AX0jfI=
 github.com/minio/parquet-go v0.0.0-20200414234858-838cfa8aae61 h1:pUSI/WKPdd77gcuoJkSzhJ4wdS8OMDOsOu99MtpXEQA=
 github.com/minio/parquet-go v0.0.0-20200414234858-838cfa8aae61/go.mod h1:4trzEJ7N1nBTd5Tt7OCZT5SEin+WiAXpdJ/WgPkESA8=
 github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
@@ -314,8 +300,6 @@ github.com/nats-io/go-nats v1.7.2 h1:cJujlwCYR8iMz5ofZSD/p2WLW8FabhkQ2lIEVbSvNSA
 github.com/nats-io/go-nats v1.7.2/go.mod h1:+t7RHT5ApZebkrQdnn6AhQJmhJJiKAvJUio1PiiCtj0=
 github.com/nats-io/go-nats-streaming v0.4.4 h1:1I3lkZDRdQYXb+holjdqZ2J6xyekrD06o9Fd8rWlgP4=
 github.com/nats-io/go-nats-streaming v0.4.4/go.mod h1:gfq4R3c9sKAINOpelo0gn/b9QDMBZnmrttcsNF+lqyo=
-github.com/nats-io/jwt v0.3.0 h1:xdnzwFETV++jNc4W1mw//qFyJGb2ABOombmZJQS4+Qo=
-github.com/nats-io/jwt v0.3.0/go.mod h1:fRYCDE99xlTsqUzISS1Bi75UBJ6ljOJQOAAu5VglpSg=
 github.com/nats-io/jwt v0.3.0/go.mod h1:fRYCDE99xlTsqUzISS1Bi75UBJ6ljOJQOAAu5VglpSg=
 github.com/nats-io/jwt v0.3.2 h1:+RB5hMpXUUA2dfxuhBTEkMOrYmM+gKIZYS1KjSostMI=
 github.com/nats-io/jwt v0.3.2/go.mod h1:/euKqTS1ZD+zzjYrY7pseZrTtWQSjujC7xjPc8wL6eU=
@@ -327,8 +311,6 @@ github.com/nats-io/nats-streaming-server v0.14.2 h1:WjQMDqVOwsI0Nb0E+XmEs1LY17Cw
 github.com/nats-io/nats-streaming-server v0.14.2/go.mod h1:RyqtDJZvMZO66YmyjIYdIvS69zu/wDAkyNWa8PIUa5c=
 github.com/nats-io/nats.go v1.9.1 h1:ik3HbLhZ0YABLto7iX80pZLPw/6dx3T+++MZJwLnMrQ=
 github.com/nats-io/nats.go v1.9.1/go.mod h1:ZjDU1L/7fJ09jvUSRVBR2e7+RnLiiIQyqyzEE/Zbp4w=
-github.com/nats-io/nkeys v0.1.0 h1:qMd4+pRHgdr1nAClu+2h/2a5F2TmKcCzjCDazVgRoX4=
-github.com/nats-io/nkeys v0.1.0/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxziKVo7w=
 github.com/nats-io/nkeys v0.1.0/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxziKVo7w=
 github.com/nats-io/nkeys v0.1.3 h1:6JrEfig+HzTH85yxzhSVbjHRJv9cn0p6n3IngIcM5/k=
 github.com/nats-io/nkeys v0.1.3/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxziKVo7w=
@@ -346,9 +328,7 @@ github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144T
 github.com/pborman/getopt v0.0.0-20180729010549-6fdd0a2c7117/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
 github.com/philhofer/fwd v1.0.0 h1:UbZqGr5Y38ApvM/V/jEljVxwocdweyH+vmYvRPBnbqQ=
 github.com/philhofer/fwd v1.0.0/go.mod h1:gk3iGcWd9+svBvR0sR+KPcfE+RNWozjowpeBVG3ZVNU=
-github.com/pierrec/lz4 v2.0.5+incompatible h1:2xWsjqPFWcplujydGg4WmhC/6fZqK42wMM8aXeqhl0I=
 github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
-github.com/pierrec/lz4 v2.2.6+incompatible h1:6aCX4/YZ9v8q69hTyiR7dNLnTA3fgtKHVVW5BCd5Znw=
 github.com/pierrec/lz4 v2.2.6+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
 github.com/pierrec/lz4 v2.4.0+incompatible h1:06usnXXDNcPvCHDkmPpkidf4jTc52UKld7UPfqKatY4=
 github.com/pierrec/lz4 v2.4.0+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
@@ -379,6 +359,7 @@ github.com/rcrowley/go-metrics v0.0.0-20190704165056-9c2d0518ed81/go.mod h1:bCqn
 github.com/rjeczalik/notify v0.9.2 h1:MiTWrPj55mNDHEiIX5YUSKefw/+lCQVoAFmD6oQm5w8=
 github.com/rjeczalik/notify v0.9.2/go.mod h1:aErll2f0sUX9PXZnVNyeiObbmTlk5jnMoCa4QEjJeqM=
 github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
+github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/rs/cors v1.6.0 h1:G9tHG9lebljV9mfp9SNPDL36nCDxmo3zTlAf1YgvzmI=
 github.com/rs/cors v1.6.0/go.mod h1:gFx+x8UowdsKA9AchylcLynDq+nNFfI8FkUZdN/jGCU=
 github.com/ryanuber/columnize v2.1.0+incompatible/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
@@ -406,8 +387,9 @@ github.com/streadway/amqp v0.0.0-20190404075320-75d898a42a94/go.mod h1:AZpEONHx3
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
-github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/tidwall/gjson v1.3.5 h1:2oW9FBNu8qt9jy5URgrzsVx/T/KSn3qn/smJQ0crlDQ=
 github.com/tidwall/gjson v1.3.5/go.mod h1:P256ACg0Mn+j1RXIDXoss50DeIABTYK1PULOJHhxOls=
 github.com/tidwall/match v1.0.1 h1:PnKP62LPNxHKTwvHHZZzdOAOCtsJTjo6dZLCwpKm5xc=
@@ -442,18 +424,21 @@ go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk=
 go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
-go.uber.org/atomic v1.3.2 h1:2Oa65PReHzfn29GpvgsYwloV9AVFHPDk8tYxt2c2tr4=
-go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
-go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
-go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
-go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM=
-go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
+go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
+go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
+go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A=
+go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
+go.uber.org/zap v1.15.0 h1:ZZCA22JRF2gQE5FoNmhmrf7jeJJ2uhqDUNRYKm8dvmM=
+go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
 golang.org/x/arch v0.0.0-20190909030613-46d78d1859ac/go.mod h1:flIaEI6LNU6xOCD5PaJvn9wGP0agmIOqjrtsKGRguv4=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181106171534-e4dc69e5b2fd/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190404164418-38d8ce5564a5/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
 golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@@ -463,8 +448,10 @@ golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL
 golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
 golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
-golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0=
 golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
 golang.org/x/mod v0.2.0 h1:KU7oHjnv3XNWfa5COkzUifxZmxp1TyI7ImMXqFxLwvQ=
 golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -483,7 +470,6 @@ golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e h1:3G+cUijn7XD+S4eJFddp53Pv7+slrESplyjG25HgL+k=
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
-golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421 h1:Wo7BWFiOk0QRFMLYMqJGFMd9CgUAcGx7V+qEg/h5IBI=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a h1:tImsplftrFpALCYumobsd0K86vlAs/eXGFms2txfJfA=
 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -491,7 +477,6 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e h1:vcxGaoTs7kV8m5Np9uUNQin4BrLOthgV7252N8V+FwY=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -526,11 +511,12 @@ golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGm
 golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-golang.org/x/tools v0.0.0-20190328211700-ab21143f2384 h1:TFlARGu6Czu1z7q93HTxcP1P+/ZFC/IKythI5RzrnRg=
 golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
-golang.org/x/tools v0.0.0-20190914235951-31e00f45c22e h1:nOOVVcLC+/3MeovP40q5lCiWmP1Z1DaN8yn8ngU63hw=
+golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190914235951-31e00f45c22e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20200428211428-0c9eba77bc32 h1:Xvf3ZQTm5bjXPxhI7g+dwqsCqadK1rcNtwtszuatetk=
 golang.org/x/tools v0.0.0-20200428211428-0c9eba77bc32/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
@@ -538,17 +524,14 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/api v0.4.0 h1:KKgc1aqhV8wDPbDzlDtpvyjZFY3vjz85FP7p4wcQUyI=
 google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/api v0.5.0 h1:lj9SyhMzyoa38fgFF0oO2T6pjs5IzkLPKfVtxpyCRMM=
 google.golang.org/api v0.5.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
-google.golang.org/appengine v1.4.0 h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.6.0 h1:Tfd7cKwKbFRsI8RMAD3oqqw7JPFRrvFlOsfbgVkjOOw=
 google.golang.org/appengine v1.6.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
-google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19 h1:Lj2SnHtxkRGJDqnGaSjo+CCdIieEnwVazbOXILwQemk=
 google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190508193815-b515fa19cec8/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
@@ -556,7 +539,6 @@ google.golang.org/genproto v0.0.0-20190513181449-d00d292a067c h1:m9avZ3wyOWBR0fL
 google.golang.org/genproto v0.0.0-20190513181449-d00d292a067c/go.mod h1:z3L6/3dTEVtUr6QSP8miRzeRqwQOioJ9I66odjN4I7s=
 google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
-google.golang.org/grpc v1.20.1 h1:Hz2g2wirWK7H0qIIhGIqRGTuMwTE8HEKFnDZZ7lm9NU=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.22.0 h1:J0UbZOIrCAl+fpTOf8YLs4dJo8L/owV4LYVtAXQoPkw=
 google.golang.org/grpc v1.22.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
@@ -568,7 +550,7 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/cheggaaa/pb.v1 v1.0.28 h1:n1tBJnnK2r7g9OW2btFH91V92STTUevLXYFb8gy9EMk=
 gopkg.in/cheggaaa/pb.v1 v1.0.28/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw=
-gopkg.in/ini.v1 v1.42.0 h1:7N3gPTt50s8GuLortA00n8AqRTk75qOP98+mTPpgzRk=
+gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/ini.v1 v1.42.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/ini.v1 v1.48.0 h1:URjZc+8ugRY5mL5uUeQH/a63JcHwdX9xZaWvmNWD7z8=
 gopkg.in/ini.v1 v1.48.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
@@ -592,12 +574,12 @@ gopkg.in/square/go-jose.v2 v2.3.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76
 gopkg.in/urfave/cli.v1 v1.20.0/go.mod h1:vuBzUtMdQeixQj8LVd+/98pzhxNGQoyuPBlsXHOQNO0=
 gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
-honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc h1:/hemPrYIhOhy8zYrNj+069zDB68us2sMGsfkFJO0iZs=
 honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
+honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
 rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=

pkg/bucket/lifecycle/and.go

@@ -19,14 +19,14 @@ package lifecycle
 import (
 	"encoding/xml"
 
-	"github.com/minio/minio/pkg/bucket/object/tagging"
+	"github.com/minio/minio-go/v6/pkg/tags"
 )
 
 // And - a tag to combine a prefix and multiple tags for lifecycle configuration rule.
 type And struct {
-	XMLName xml.Name      `xml:"And"`
-	Prefix  string        `xml:"Prefix,omitempty"`
-	Tags    []tagging.Tag `xml:"Tag,omitempty"`
+	XMLName xml.Name   `xml:"And"`
+	Prefix  string     `xml:"Prefix,omitempty"`
+	Tags    []tags.Tag `xml:"Tag,omitempty"`
 }
 
 var errDuplicateTagKey = Errorf("Duplicate Tag Keys are not allowed")

pkg/bucket/lifecycle/filter.go

@@ -19,7 +19,7 @@ package lifecycle
 import (
 	"encoding/xml"
 
-	"github.com/minio/minio/pkg/bucket/object/tagging"
+	"github.com/minio/minio-go/v6/pkg/tags"
 )
 
 var (
@@ -31,7 +31,7 @@ type Filter struct {
 	XMLName xml.Name `xml:"Filter"`
 	Prefix  string
 	And     And
-	Tag     tagging.Tag
+	Tag     tags.Tag
 }
 
 // MarshalXML - produces the xml representation of the Filter struct
@@ -89,5 +89,5 @@ func (f Filter) Validate() error {
 
 // isEmpty - returns true if Filter tag is empty
 func (f Filter) isEmpty() bool {
-	return f.And.isEmpty() && f.Prefix == "" && f.Tag == tagging.Tag{}
+	return f.And.isEmpty() && f.Prefix == "" && f.Tag.IsEmpty()
 }

pkg/bucket/object/lock/lock.go

@@ -161,18 +161,18 @@ func (r Retention) Retain(created time.Time) bool {
 // BucketObjectLockConfig - map of bucket and retention configuration.
 type BucketObjectLockConfig struct {
 	sync.RWMutex
-	retentionMap map[string]Retention
+	retentionMap map[string]*Retention
 }
 
 // Set - set retention configuration.
-func (config *BucketObjectLockConfig) Set(bucketName string, retention Retention) {
+func (config *BucketObjectLockConfig) Set(bucketName string, retention *Retention) {
 	config.Lock()
 	config.retentionMap[bucketName] = retention
 	config.Unlock()
 }
 
 // Get - Get retention configuration.
-func (config *BucketObjectLockConfig) Get(bucketName string) (r Retention, ok bool) {
+func (config *BucketObjectLockConfig) Get(bucketName string) (r *Retention, ok bool) {
 	config.RLock()
 	defer config.RUnlock()
 	r, ok = config.retentionMap[bucketName]
@@ -189,7 +189,7 @@ func (config *BucketObjectLockConfig) Remove(bucketName string) {
 // NewBucketObjectLockConfig returns initialized BucketObjectLockConfig
 func NewBucketObjectLockConfig() *BucketObjectLockConfig {
 	return &BucketObjectLockConfig{
-		retentionMap: map[string]Retention{},
+		retentionMap: make(map[string]*Retention),
 	}
 }
 
@@ -280,7 +280,8 @@ func (config *Config) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error
 }
 
 // ToRetention - convert to Retention type.
-func (config *Config) ToRetention() (r Retention) {
+func (config *Config) ToRetention() (r *Retention) {
+	r = &Retention{}
 	if config.Rule != nil {
 		r.Mode = config.Rule.DefaultRetention.Mode
 

pkg/bucket/object/tagging/error.go

@@ -1,53 +0,0 @@
-/*
- * MinIO Cloud Storage, (C) 2020 MinIO, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package tagging
-
-import (
-	"fmt"
-)
-
-// Error is the generic type for any error happening during tag
-// parsing.
-type Error struct {
-	err  error
-	code string
-}
-
-// Errorf - formats according to a format specifier and returns
-// the string as a value that satisfies error of type tagging.Error
-func Errorf(format, code string, a ...interface{}) error {
-	return Error{err: fmt.Errorf(format, a...), code: code}
-}
-
-// Unwrap the internal error.
-func (e Error) Unwrap() error { return e.err }
-
-// Error 'error' compatible method.
-func (e Error) Error() string {
-	if e.err == nil {
-		return "tagging: cause <nil>"
-	}
-	return e.err.Error()
-}
-
-// Code returns appropriate error code.
-func (e Error) Code() string {
-	if e.code == "" {
-		return "BadRequest"
-	}
-	return e.code
-}

pkg/bucket/object/tagging/tag.go

@@ -1,82 +0,0 @@
-/*
- * MinIO Cloud Storage, (C) 2020 MinIO, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package tagging
-
-import (
-	"encoding/xml"
-	"strings"
-	"unicode/utf8"
-)
-
-// Tag - single tag
-type Tag struct {
-	XMLName xml.Name `xml:"Tag"`
-	Key     string   `xml:"Key,omitempty"`
-	Value   string   `xml:"Value,omitempty"`
-}
-
-// Validate - validates the tag element
-func (t Tag) Validate() error {
-	if err := t.validateKey(); err != nil {
-		return err
-	}
-	if err := t.validateValue(); err != nil {
-		return err
-	}
-	return nil
-}
-
-// validateKey - checks if key is valid or not.
-func (t Tag) validateKey() error {
-	// cannot be longer than maxTagKeyLength characters
-	if utf8.RuneCountInString(t.Key) > maxTagKeyLength {
-		return ErrInvalidTagKey
-	}
-	// cannot be empty
-	if len(t.Key) == 0 {
-		return ErrInvalidTagKey
-	}
-	// Tag key shouldn't have "&"
-	if strings.Contains(t.Key, "&") {
-		return ErrInvalidTagKey
-	}
-	return nil
-}
-
-// validateValue - checks if value is valid or not.
-func (t Tag) validateValue() error {
-	// cannot be longer than maxTagValueLength characters
-	if utf8.RuneCountInString(t.Value) > maxTagValueLength {
-		return ErrInvalidTagValue
-	}
-	// Tag value shouldn't have "&"
-	if strings.Contains(t.Value, "&") {
-		return ErrInvalidTagValue
-	}
-	return nil
-}
-
-// IsEmpty - checks if tag is empty or not
-func (t Tag) IsEmpty() bool {
-	return t.Key == "" && t.Value == ""
-}
-
-// String - returns a string in format "tag1=value1" for the
-// current Tag
-func (t Tag) String() string {
-	return t.Key + "=" + t.Value
-}

pkg/bucket/object/tagging/tagging.go

@@ -1,112 +0,0 @@
-/*
- * MinIO Cloud Storage, (C) 2020 MinIO, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package tagging
-
-import (
-	"bytes"
-	"encoding/xml"
-	"io"
-	"net/url"
-)
-
-// S3 API limits for tags
-// Ref: https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
-const (
-	maxTags           = 10
-	maxTagKeyLength   = 128
-	maxTagValueLength = 256
-)
-
-// errors returned by tagging package
-var (
-	ErrTooManyTags     = Errorf("Object tags cannot be greater than 10", "BadRequest")
-	ErrInvalidTagKey   = Errorf("The TagKey you have provided is invalid", "InvalidTag")
-	ErrInvalidTagValue = Errorf("The TagValue you have provided is invalid", "InvalidTag")
-	ErrInvalidTag      = Errorf("Cannot provide multiple Tags with the same key", "InvalidTag")
-)
-
-// Tagging - object tagging interface
-type Tagging struct {
-	XMLName xml.Name `xml:"Tagging"`
-	TagSet  TagSet   `xml:"TagSet"`
-}
-
-// Validate - validates the tagging configuration
-func (t Tagging) Validate() error {
-	// Tagging can't have more than 10 tags
-	if len(t.TagSet.Tags) > maxTags {
-		return ErrTooManyTags
-	}
-	// Validate all the rules in the tagging config
-	for _, ts := range t.TagSet.Tags {
-		if err := ts.Validate(); err != nil {
-			return err
-		}
-	}
-	if t.TagSet.ContainsDuplicateTag() {
-		return ErrInvalidTag
-	}
-	return nil
-}
-
-// String - returns a string in format "tag1=value1&tag2=value2" with all the
-// tags in this Tagging Struct
-func (t Tagging) String() string {
-	var buf bytes.Buffer
-	for _, tag := range t.TagSet.Tags {
-		if buf.Len() > 0 {
-			buf.WriteString("&")
-		}
-		buf.WriteString(tag.String())
-	}
-	return buf.String()
-}
-
-// FromString - returns a Tagging struct when given a string in format
-// "tag1=value1&tag2=value2"
-func FromString(tagStr string) (Tagging, error) {
-	tags, err := url.ParseQuery(tagStr)
-	if err != nil {
-		return Tagging{}, err
-	}
-	var idx = 0
-	parsedTags := make([]Tag, len(tags))
-	for k := range tags {
-		parsedTags[idx].Key = k
-		parsedTags[idx].Value = tags.Get(k)
-		idx++
-	}
-	return Tagging{
-		TagSet: TagSet{
-			Tags: parsedTags,
-		},
-	}, nil
-}
-
-// ParseTagging - parses incoming xml data in given reader
-// into Tagging interface. After parsing, also validates the
-// parsed fields based on S3 API constraints.
-func ParseTagging(reader io.Reader) (*Tagging, error) {
-	var t Tagging
-	if err := xml.NewDecoder(reader).Decode(&t); err != nil {
-		return nil, err
-	}
-	if err := t.Validate(); err != nil {
-		return nil, err
-	}
-	return &t, nil
-}

pkg/bucket/policy/action.go

@@ -109,6 +109,11 @@ const (
 	// PutBucketObjectLockConfigurationAction - PutObjectLockConfiguration Rest API action
 	PutBucketObjectLockConfigurationAction = "s3:PutBucketObjectLockConfiguration"
 
+	// GetBucketTaggingAction - GetTagging Rest API action
+	GetBucketTaggingAction = "s3:GetBucketTagging"
+	// PutBucketTaggingAction - PutTagging Rest API action
+	PutBucketTaggingAction = "s3:PutBucketTagging"
+
 	// GetObjectTaggingAction - Get Object Tags API action
 	GetObjectTaggingAction = "s3:GetObjectTagging"
 	// PutObjectTaggingAction - Put Object Tags API action
@@ -174,6 +179,8 @@ var supportedActions = map[Action]struct{}{
 	PutObjectLegalHoldAction:               {},
 	PutBucketObjectLockConfigurationAction: {},
 	GetBucketObjectLockConfigurationAction: {},
+	PutBucketTaggingAction:                 {},
+	GetBucketTaggingAction:                 {},
 	BypassGovernanceRetentionAction:        {},
 	GetObjectTaggingAction:                 {},
 	PutObjectTaggingAction:                 {},
@@ -296,6 +303,8 @@ var actionConditionKeyMap = map[Action]condition.KeySet{
 
 	GetBucketObjectLockConfigurationAction: condition.NewKeySet(condition.CommonKeys...),
 	PutBucketObjectLockConfigurationAction: condition.NewKeySet(condition.CommonKeys...),
+	GetBucketTaggingAction:                 condition.NewKeySet(condition.CommonKeys...),
+	PutBucketTaggingAction:                 condition.NewKeySet(condition.CommonKeys...),
 	PutObjectTaggingAction:                 condition.NewKeySet(condition.CommonKeys...),
 	GetObjectTaggingAction:                 condition.NewKeySet(condition.CommonKeys...),
 	DeleteObjectTaggingAction:              condition.NewKeySet(condition.CommonKeys...),

pkg/console/console.go

@@ -34,9 +34,10 @@ var (
 	DebugPrint = false
 
 	// Used by the caller to print multiple lines atomically. Exposed by Lock/Unlock methods.
-	publicMutex = &sync.Mutex{}
+	publicMutex sync.Mutex
+
 	// Used internally by console.
-	privateMutex = &sync.Mutex{}
+	privateMutex sync.Mutex
 
 	stderrColoredOutput = colorable.NewColorableStderr()