Compare commits

..

50 Commits

Author SHA1 Message Date
Andreas Auernhammer 5a16671f72 do not set response header X-Frame-Options for S3 requests (#5838)
This change removes the X-Frame-Options header - It should
not be set for S3 requests since it can break CORS.

Fixes #5813
2018-04-19 11:46:37 -07:00
Ashish Kumar Sinha 9ebb72aa99 Introduce new unauthenticated endpoint /metric (#5723) (#5829)
/metric exposes Promethus compatible data for scraping metrics

Fixes: #5723
2018-04-18 16:01:42 -07:00
wd256 9f31da5d57 Fix PutObject/CopyObject with metadata for GCS gateway (#5828)
Make sure to apply standard headers such as Content-Type,
Content-Disposition and Content-Language to the correct
GCS object attributes during object upload and copy operations.

Fixes: #5800
2018-04-18 21:40:48 +05:30
Harshavardhana 0e9c73cbcf update browser ui-assets (#5831) 2018-04-17 09:31:55 +05:30
Harshavardhana c332b4fb5e fix shadowing issue in logger (#5830) 2018-04-16 13:36:45 -07:00
Rushan ed5f2d899d Browser: Disable input validation styles on Firefox (#5822) 2018-04-16 10:47:03 -07:00
Harshavardhana adf9a9d300 Remove all unused variables and functions (#5823) 2018-04-15 19:26:04 +05:30
Harshavardhana 97a8d856b6 Vendorize new changes from minio-go repo (#5821)
- When policy is empty delete the bucket policy (#966) (04/13/18) <Harshavardhana>
- Add tests to check if ListObjects/V2 returns expected StorageClass (#963) (04/10/18) <Nitish Tiwari>
- Update get/setBucketPolicy methods to use files instead of pkg/policy (#959) (04/10/18) <Nitish Tiwari>
- avoid unnecessary stat call during single copy (#962) (04/06/18) <Andreas Auernhammer>
- avoid sending SSE-S3 header during GET requests. (#965) (04/05/18) <Andreas Auernhammer>
- Fix stream SSE uploads with S3 encrypt type (#960) (04/02/18) <Jesús Espino>
- Fix xml parsing error for RemoveObjects API (#949) (03/29/18) <poornas>
- Allow to upload empty files in stream based uploads (#958) (03/26/18) <Jesús Espino>
- Add missing doneCh in the example for removeobjects (#955) (03/26/18) <Alexandr Korsak>
- tests: Remove partial related tests (#957) (03/26/18) <Anis Elleuch>
- Add transport connection broken error to retry list (#956) (03/19/18) <poornas>
- [refactor]: simplify client encryption examples (#952) (03/19/18) <Andreas Auernhammer>
- Add tests for putObjectContentLanguage (#950) (03/15/18) <Harshavardhana>
- Add putObject/getObject() client side encryption examples (#948) (03/13/18) <Harshavardhana>
2018-04-15 11:39:48 +05:30
Nitish Tiwari 638f01f9e4 Generalize loadConfig method to avoid reading from disk (#5819)
As we move to multiple config backends like local disk and etcd,
config file should not be read from the disk, instead the quick
package should load and verify for duplicate entries.
2018-04-13 15:14:19 -07:00
Harshavardhana 57b8db2088 Cleanup help to bring concise self-help message (#5788) 2018-04-13 11:57:05 -07:00
Harshavardhana bcd54632f5 Fix broken 'minio update' command on Windows (#5816) 2018-04-13 11:51:03 -07:00
Nitish Tiwari 42f775af4f Add cache config for configuration comparison (#5812) 2018-04-13 23:45:19 +05:30
Minio Trusted 75f35b926d Update yaml files to latest version RELEASE.2018-04-12T23-41-09Z 2018-04-12 23:45:16 +00:00
Andreas Auernhammer f60765ac93 add some security HTTP headers (#5814)
This change adds some security headers like Content-Security-Policy.
It does not set the HSTS header because Content-Security-Policy prevents
mixed HTTP and HTTPS content and the server does not use cookies.
However it is a header which could be added later on.

It also moves some header added by #5805 from a vendored file
to a generic handler.

Fixes ##5813
2018-04-12 15:57:41 -07:00
Harshavardhana 1f07545e2a Improve init messages for distributed setup (#5786)
Fixes #5531
2018-04-12 15:43:38 -07:00
rwagner_inf e39de65367 Add security HTTP Headers (#5805)
Some HTTP security headers in Minio.
To avoid problems with XSS and Clickjacking attacks.

X-Frame-Options
X-Frame-Options response header improve the protection
of web applications against Clickjacking. It declares a
policy communicated from a host to the client browser
on whether the browser must not display the transmitted
content in frames of other web pages.

X-XSS-Protection
This header enables the Cross-site scripting (XSS) filter in your browser.
2018-04-12 14:30:06 -07:00
Harshavardhana bd8d6e3c4e Return bucketNameInvalid properly in B2 gateway (#5787)
Fixes #5758
2018-04-12 11:20:21 +05:30
Harshavardhana 4a874dfbc1 Ignore prefix renames when dest directory is not empty (#5798)
Also make sure to not modify the underlying errors from
layers, we should return the error as is and one object
layer should translate the errors.

Fixes #5797
2018-04-11 17:15:42 -07:00
Bala FA 650c6ee8fb event/target/webhook: treat all 2xx as success. (#5792)
Fixes #5769
2018-04-10 17:45:54 -07:00
kannappanr 57a3d9c16c Modify fatalIf, startup and update message logging code (#5780)
Use a common logging framework to log fatalIf, startup, Info and Update
messages.
2018-04-10 09:37:14 -07:00
kannappanr cef992a395 Remove error package and cause functions (#5784) 2018-04-10 09:36:37 -07:00
Harshavardhana 217fb470a7 Add a check to check if disk is writable (#5662)
This check is a pre-emptive check to return
error early before we attempt to use the disk
for any other operations later.

refer #5645
2018-04-10 09:26:09 +05:30
Harshavardhana eb0deabd73 Fail when TLS is configured and HTTP endpoints are provided (#5771) 2018-04-10 07:40:23 +05:30
dingjs a706c21f70 Tranlsate disk-caching/README.md,nas.md,large-bucket/README.md to Chinese. (#5775) 2018-04-09 17:41:16 -07:00
Harshavardhana 41c31d10ca Enable s3,gcs,manta,oss,b2 gateways as production ready (#5795) 2018-04-09 16:33:48 -07:00
Harshavardhana 1d31ad499f Make sure to re-load reference format after HealFormat (#5772)
This PR introduces ReloadFormat API call at objectlayer
to facilitate this. Previously we repurposed HealFormat
but we never ended up updating our reference format on
peers.

Fixes #5700
2018-04-09 22:55:41 +05:30
Krishna Srinivas ae8e863ff4 disk.String() represents just path and not URL when disk is a local disk (#5785) 2018-04-06 16:59:31 -07:00
Andreas Auernhammer da9f0e324e return AWS S3 compatible error for invalid but equal keys during key rotation (#5783)
This change let the server return the S3 error for a key rotation
if the source key is not valid but equal to the destination key.

This change also fixes the SSE-C error messages since AWS returns error messages
ending with a '.'.

Fixes #5625
2018-04-06 14:15:23 -07:00
Harshavardhana 73f7a98590 Enable go1.10.1 version (#5666) 2018-04-06 11:07:39 -07:00
Andreas Auernhammer 2f20d90465 set storage class in FS mode (#5779)
This change sets the storage class of the object-info if a storage
class was specified during PUT. The server now replies with the
storage class which was set during uploading the object in FS mode.

Fixes #5777
2018-04-06 20:50:02 +05:30
Dee Koder c30f75de28 Update screenshots. Use png instead of jpgs. (#5770) 2018-04-06 13:51:25 +05:30
Harshavardhana 861550446a Allow upto 15 minutes of timeskew for RPC calls as well (#5774)
Default installations of cloned VMs on VMware like env
might experience serious problems with time skewing,
allow for a higher value instead of 3 seconds we are
moving to 15 minutes just like API level skew.

Access to internet and configuring ntp might not be possible,
in such situations providing atleast a 15 minute skew could
cater for majority of situations.
2018-04-05 23:55:31 -07:00
kannappanr f8a3fd0c2a Create logger package and rename errorIf to LogIf (#5678)
Removing message from error logging
Replace errors.Trace with LogIf
2018-04-05 15:04:40 -07:00
kannappanr 91fd8ffeb7 Move http package inside cmd (#5776) 2018-04-05 14:19:27 -07:00
Nitish Tiwari 9eb94fe8c8 Fix StorageClass field in ListObject/ListObjectV2 response (#5766)
Fixes: #5754
2018-04-05 10:56:28 -07:00
Harshavardhana 30ab71c9d9 Add missing healthcheck router for gateway (#5764) 2018-04-05 07:37:54 +05:30
Kaan Kabalak 38ce19badd Rename minio-limitations.md to minio-limits.md in Chinese translation (#5753)
Applies the change in #5541 for Chinese docs
2018-04-04 19:00:19 -07:00
Minio Trusted c4a70fdee9 Update yaml files to latest version RELEASE.2018-04-04T05-20-54Z 2018-04-04 05:25:19 +00:00
Harshavardhana 85a57d2021 Make sure to close the disk connections (#5752)
Since we do not re-use storageDisks after moving
the connections to object layer we should close them
appropriately otherwise we have a lot of connection
leaks and these can compound as the time goes by.

This PR also refactors the initialization code to
re-use storageDisks for given set of endpoints until
we have confirmed a valid reference format.
2018-04-04 10:28:48 +05:30
Nitish Tiwari d67e423a32 Add firewall-cmd entry for firewall configuration (#5755) 2018-04-03 17:15:24 -07:00
Edward Vielmetti e76fe435c4 "account_id" for "accound_id" (#5757) 2018-04-03 17:06:59 -07:00
Vladimir Cores 1a71004c91 Multi files selection and upload from Browser (#5711) 2018-04-03 16:44:37 -07:00
Edward Vielmetti 1e50951a4a typo fix (#5756) 2018-04-03 21:00:35 +05:30
Harshavardhana 2c5f2e9669 Stop deleting 'format.json' upon unsuccessful save (#5747)
An issue was reproduced when there a no more inodes
available on an existing setup of 4 disks, now we
took one of the disks and reformatted it to relinquish
inodes. Now we attempt to bring the fresh disk back
into setup and perform a heal - at this point creating
new `format.json` fails on existing disks since they
do not have more inodes available.

At this point due to quorum failure, we end up deleting
existing `format.json` as well, this PR removes the code
which deletes existing `format.json` as there is no need
to delete them.
2018-04-03 10:48:06 +05:30
Harshavardhana b052bfa79e Add necessary safe headers under CORS (#5750)
Fixes #5748
2018-04-02 17:51:32 -07:00
Harshavardhana 8eb838bf91 Extend quorum based listing for prefixes (#5749)
Previous PR 2afd196c83 fixed
the issue of quorum based listing for regular objects, this
PR continues on this idea by extending this support to
object directory prefixes as well.

Fixes #5733
2018-04-02 17:26:34 -07:00
poornas 98315b8e60 Update disk cache documentation (#5743) 2018-04-02 10:25:08 +05:30
kannappanr b87cc3d643 Get proper GOPATH in trim function (#5744)
Set GOPATH string to empty in build-constants.go
Check for both compile time GOPATH and default GOPATH
while trimming the file path in the stack trace.
Fixes #5741
2018-03-30 19:13:25 -07:00
Harshavardhana 4d02f9bccf fix gateway docs comprehensively (#5740) 2018-03-30 14:16:06 +05:30
Minio Trusted cf8d12d169 Update yaml files to latest version RELEASE.2018-03-30T00-38-44Z 2018-03-30 00:41:38 +00:00
301 changed files with 22248 additions and 3428 deletions
+1 -1
View File
@@ -31,4 +31,4 @@ after_success:
- bash <(curl -s https://codecov.io/bash)
go:
- 1.9.4
- '1.10.1'
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.9.4-alpine3.6
FROM golang:1.10.1-alpine3.7
MAINTAINER Minio Inc <dev@minio.io>
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.9.4-alpine3.6
FROM golang:1.10.1-alpine3.7
MAINTAINER Minio Inc <dev@minio.io>
+1 -1
View File
@@ -1,4 +1,4 @@
FROM alpine:3.6
FROM alpine:3.7
MAINTAINER Minio Inc <dev@minio.io>
+22 -2
View File
@@ -101,7 +101,7 @@ iptables -A INPUT -p tcp --dport 9000:9010 -j ACCEPT
service iptables restart
```
### ufw in Debian
### ufw
For hosts with ufw enabled (Debian based distros), you can use `ufw` command to allow traffic to specific ports. Use below command to allow access to port 9000
@@ -115,10 +115,30 @@ Below command enables all incoming traffic to ports ranging from 9000 to 9010.
ufw allow 9000:9010/tcp
```
### firewall-cmd
For hosts with firewall-cmd enabled (CentOS), you can use `firewall-cmd` command to allow traffic to specific ports. Use below commands to allow access to port 9000
```sh
firewall-cmd --get-active-zones
```
This command gets the active zone(s). Now, apply port rules to the relevant zones returned above. For example if the zone is `public`, use
```sh
firewall-cmd --zone=public --add-port=9000/tcp --permanent
```
Note that `permanent` makes sure the rules are persistent across firewall start, restart or reload. Finally reload the firewall for changes to take effect.
```sh
firewall-cmd --reload
```
## Test using Minio Browser
Minio Server comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 ensure your server has started successfully.
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.jpg?raw=true)
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.png?raw=true)
## Test using Minio Client `mc`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the Minio Client [Quickstart Guide](https://docs.minio.io/docs/minio-client-quickstart-guide) for further instructions.
+1 -1
View File
@@ -84,7 +84,7 @@ go get -u github.com/minio/minio
## 使用Minio浏览器进行验证
安装后使用浏览器访问[http://127.0.0.1:9000](http://127.0.0.1:9000),如果可以访问,则表示minio已经安装成功。
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.jpg?raw=true)
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.png?raw=true)
## 使用Minio客户端 `mc`进行验证
`mc` 提供了一些UNIX常用命令的替代品,像ls, cat, cp, mirror, diff这些。 它支持文件系统和亚马逊S3云存储服务。 更多信息请参考 [mc快速入门](https://docs.minio.io/docs/minio-client-quickstart-guide) 。
+1 -1
View File
@@ -12,7 +12,7 @@ clone_folder: c:\gopath\src\github.com\minio\minio
# Environment variables
environment:
GOPATH: c:\gopath
GOROOT: c:\go19
GOROOT: c:\go
# scripts that run after cloning repository
install:
+6 -1
View File
@@ -33,7 +33,11 @@ export const MainActions = ({
)
const onFileUpload = e => {
e.preventDefault()
uploadFile(e.target.files[0])
let files = e.target.files
let filesToUploadCount = files.length
for (let i = 0; i < filesToUploadCount; i++) {
uploadFile(files.item(i))
}
e.target.value = null
}
@@ -55,6 +59,7 @@ export const MainActions = ({
onChange={onFileUpload}
style={{ display: "none" }}
id="file-input"
multiple={true}
/>
<label htmlFor="file-input">
{" "}
@@ -63,11 +63,20 @@ describe("MainActions", () => {
it("should call uploadFile when a file is selected for upload", () => {
const uploadFile = jest.fn()
const wrapper = shallow(<MainActions uploadFile={uploadFile} />)
const file = new Blob(["file content"], { type: "text/plain" })
wrapper.find("#file-input").simulate("change", {
const files = [new Blob(["file content"], { type: "text/plain" })]
const input = wrapper.find("#file-input")
const event = {
preventDefault: jest.fn(),
target: { files: [file] }
})
expect(uploadFile).toHaveBeenCalledWith(file)
target: {
files: {
length: files.length,
item: function(index) {
return files[index]
}
}
}
}
input.simulate("change", event)
expect(uploadFile).toHaveBeenCalledWith(files[0])
})
})
+1
View File
@@ -87,6 +87,7 @@ select.form-control {
z-index: 1;
border-bottom: 1px solid #eee;
color: #32393F;
box-shadow: none;
font-size: 13px;
+24 -24
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -21,7 +21,7 @@ _init() {
## Minimum required versions for build dependencies
GIT_VERSION="1.0"
GO_VERSION="1.9.4"
GO_VERSION="1.10.1"
OSX_VERSION="10.8"
KNAME=$(uname -s)
ARCH=$(uname -m)
+31 -22
View File
@@ -18,7 +18,9 @@ package cmd
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -27,9 +29,11 @@ import (
"time"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/handlers"
"github.com/minio/minio/pkg/madmin"
"github.com/minio/minio/pkg/quick"
)
const (
@@ -71,7 +75,7 @@ func (a adminAPIHandlers) VersionHandler(w http.ResponseWriter, r *http.Request)
jsonBytes, err := json.Marshal(adminAPIVersionInfo)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal Admin API Version to JSON.")
logger.LogIf(context.Background(), err)
return
}
@@ -99,7 +103,7 @@ func (a adminAPIHandlers) ServiceStatusHandler(w http.ResponseWriter, r *http.Re
uptime, err := getPeerUptimes(globalAdminPeers)
if err != nil {
writeErrorResponseJSON(w, toAPIErrorCode(err), r.URL)
errorIf(err, "Possibly failed to get uptime from majority of servers.")
logger.LogIf(context.Background(), err)
return
}
@@ -113,7 +117,7 @@ func (a adminAPIHandlers) ServiceStatusHandler(w http.ResponseWriter, r *http.Re
jsonBytes, err := json.Marshal(serverStatus)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal storage info into json.")
logger.LogIf(context.Background(), err)
return
}
// Reply with storage information (across nodes in a
@@ -136,7 +140,7 @@ func (a adminAPIHandlers) ServiceStopNRestartHandler(w http.ResponseWriter, r *h
var sa madmin.ServiceAction
err := json.NewDecoder(r.Body).Decode(&sa)
if err != nil {
errorIf(err, "Error parsing body JSON")
logger.LogIf(context.Background(), err)
writeErrorResponseJSON(w, ErrRequestBodyParse, r.URL)
return
}
@@ -149,7 +153,7 @@ func (a adminAPIHandlers) ServiceStopNRestartHandler(w http.ResponseWriter, r *h
serviceSig = serviceStop
default:
writeErrorResponseJSON(w, ErrMalformedPOSTRequest, r.URL)
errorIf(err, "Invalid service action received")
logger.LogIf(context.Background(), errors.New("Invalid service action received"))
return
}
@@ -243,7 +247,9 @@ func (a adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Reque
serverInfoData, err := peer.cmdRunner.ServerInfoData()
if err != nil {
errorIf(err, "Unable to get server info from %s.", peer.addr)
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", peer.addr)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
reply[idx].Error = err.Error()
return
}
@@ -258,7 +264,7 @@ func (a adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Reque
jsonBytes, err := json.Marshal(reply)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal storage info into json.")
logger.LogIf(context.Background(), err)
return
}
@@ -292,7 +298,7 @@ func validateLockQueryParams(vars url.Values) (string, string, time.Duration,
}
duration, err := time.ParseDuration(olderThanStr)
if err != nil {
errorIf(err, "Failed to parse duration passed as query value.")
logger.LogIf(context.Background(), err)
return "", "", time.Duration(0), ErrInvalidDuration
}
@@ -325,7 +331,7 @@ func (a adminAPIHandlers) ListLocksHandler(w http.ResponseWriter, r *http.Reques
duration)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to fetch lock information from remote nodes.")
logger.LogIf(context.Background(), err)
return
}
@@ -333,7 +339,7 @@ func (a adminAPIHandlers) ListLocksHandler(w http.ResponseWriter, r *http.Reques
jsonBytes, err := json.Marshal(volLocks)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal lock information into json.")
logger.LogIf(context.Background(), err)
return
}
@@ -369,7 +375,7 @@ func (a adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *http.Reque
duration)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to fetch lock information from remote nodes.")
logger.LogIf(ctx, err)
return
}
@@ -377,7 +383,7 @@ func (a adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *http.Reque
jsonBytes, err := json.Marshal(volLocks)
if err != nil {
writeErrorResponseJSON(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal lock information into json.")
logger.LogIf(ctx, err)
return
}
@@ -425,7 +431,7 @@ func extractHealInitParams(r *http.Request) (bucket, objPrefix string,
if clientToken == "" {
jerr := json.NewDecoder(r.Body).Decode(&hs)
if jerr != nil {
errorIf(jerr, "Error parsing body JSON")
logger.LogIf(context.Background(), jerr)
err = ErrRequestBodyParse
return
}
@@ -583,7 +589,7 @@ func (a adminAPIHandlers) GetConfigHandler(w http.ResponseWriter, r *http.Reques
// occurring on a quorum of the servers is returned.
configBytes, err := getPeerConfig(globalAdminPeers)
if err != nil {
errorIf(err, "Failed to get config from peers")
logger.LogIf(context.Background(), err)
writeErrorResponseJSON(w, toAdminAPIErrCode(err), r.URL)
return
}
@@ -655,6 +661,7 @@ func writeSetConfigResponse(w http.ResponseWriter, peers adminPeers,
// SetConfigHandler - PUT /minio/admin/v1/config
func (a adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
@@ -678,7 +685,7 @@ func (a adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Reques
return
}
if err != io.ErrUnexpectedEOF {
errorIf(err, "Failed to read config from request body.")
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -687,8 +694,8 @@ func (a adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Reques
// Validate JSON provided in the request body: check the
// client has not sent JSON objects with duplicate keys.
if err = checkDupJSONKeys(string(configBytes)); err != nil {
errorIf(err, "config contains duplicate JSON entries.")
if err = quick.CheckDuplicateKeys(string(configBytes)); err != nil {
logger.LogIf(ctx, err)
writeErrorResponse(w, ErrAdminConfigBadJSON, r.URL)
return
}
@@ -696,7 +703,7 @@ func (a adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Reques
var config serverConfig
err = json.Unmarshal(configBytes, &config)
if err != nil {
errorIf(err, "Failed to unmarshal JSON configuration", err)
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -718,7 +725,7 @@ func (a adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Reques
errs := writeTmpConfigPeers(globalAdminPeers, tmpFileName, configBytes)
// Check if the operation succeeded in quorum or more nodes.
rErr := reduceWriteQuorumErrs(errs, nil, len(globalAdminPeers)/2+1)
rErr := reduceWriteQuorumErrs(ctx, errs, nil, len(globalAdminPeers)/2+1)
if rErr != nil {
writeSetConfigResponse(w, globalAdminPeers, errs, false, r.URL)
return
@@ -736,7 +743,7 @@ func (a adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Reques
// Rename the temporary config file to config.json
errs = commitConfigPeers(globalAdminPeers, tmpFileName)
rErr = reduceWriteQuorumErrs(errs, nil, len(globalAdminPeers)/2+1)
rErr = reduceWriteQuorumErrs(ctx, errs, nil, len(globalAdminPeers)/2+1)
if rErr != nil {
writeSetConfigResponse(w, globalAdminPeers, errs, false, r.URL)
return
@@ -777,7 +784,7 @@ func (a adminAPIHandlers) UpdateCredentialsHandler(w http.ResponseWriter,
var req madmin.SetCredsReq
err := json.NewDecoder(r.Body).Decode(&req)
if err != nil {
errorIf(err, "Error parsing body JSON")
logger.LogIf(context.Background(), err)
writeErrorResponseJSON(w, ErrRequestBodyParse, r.URL)
return
}
@@ -804,7 +811,9 @@ func (a adminAPIHandlers) UpdateCredentialsHandler(w http.ResponseWriter,
// Notify all other Minio peers to update credentials
updateErrs := updateCredsOnPeers(creds)
for peer, err := range updateErrs {
errorIf(err, "Unable to update credentials on peer %s.", peer)
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", peer)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
// Update local credentials in memory.
+3 -4
View File
@@ -33,7 +33,6 @@ import (
router "github.com/gorilla/mux"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/madmin"
)
@@ -264,7 +263,7 @@ func initTestXLObjLayer() (ObjectLayer, []string, error) {
return nil, nil, err
}
endpoints := mustGetNewEndpointList(xlDirs...)
format, err := waitForFormatXL(true, endpoints, 1, 16)
format, err := waitForFormatXL(context.Background(), true, endpoints, 1, 16)
if err != nil {
removeRoots(xlDirs)
return nil, nil, err
@@ -762,13 +761,13 @@ func buildAdminRequest(queryVal url.Values, method, path string,
"/minio/admin/v1"+path+"?"+queryVal.Encode(),
contentLength, bodySeeker)
if err != nil {
return nil, errors.Trace(err)
return nil, err
}
cred := globalServerConfig.GetCredential()
err = signRequestV4(req, cred.AccessKey, cred.SecretKey)
if err != nil {
return nil, errors.Trace(err)
return nil, err
}
return req, nil
+13 -5
View File
@@ -222,7 +222,7 @@ func (ahs *allHealState) LaunchNewHealSequence(h *healSequence) (
StartTime: h.startTime,
})
if err != nil {
errorIf(err, "Failed to marshal heal result into json.")
logger.LogIf(context.Background(), err)
return nil, ErrInternalError, ""
}
return b, ErrNone, ""
@@ -270,7 +270,7 @@ func (ahs *allHealState) PopHealStatusJSON(path string,
jbytes, err := json.Marshal(h.currentStatus)
if err != nil {
errorIf(err, "Failed to marshal heal result into json.")
logger.LogIf(context.Background(), err)
return nil, ErrInternalError
}
@@ -321,7 +321,9 @@ type healSequence struct {
func newHealSequence(bucket, objPrefix, clientAddr string,
numDisks int, hs madmin.HealOpts, forceStart bool) *healSequence {
ctx := logger.SetContext(context.Background(), &logger.ReqInfo{clientAddr, "", "", "Heal", bucket, objPrefix, nil})
reqInfo := &logger.ReqInfo{RemoteHost: clientAddr, API: "Heal", BucketName: bucket}
reqInfo.AppendTags("prefix", objPrefix)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
return &healSequence{
bucket: bucket,
@@ -537,11 +539,17 @@ func (h *healSequence) healDiskFormat() error {
}
res, err := objectAPI.HealFormat(h.ctx, h.settings.DryRun)
if err != nil {
// return any error, ignore error returned when disks have
// already healed.
if err != nil && err != errNoHealRequired {
return errFnHealFromAPIErr(err)
}
peersReInitFormat(globalAdminPeers, h.settings.DryRun)
// Healing succeeded notify the peers to reload format and re-initialize disks.
// We will not notify peers only if healing succeeded.
if err == nil {
peersReInitFormat(globalAdminPeers, h.settings.DryRun)
}
// Push format heal result
return h.pushHealResultItem(res)
+14 -10
View File
@@ -29,7 +29,7 @@ import (
"time"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
const (
@@ -84,8 +84,7 @@ func (lc localAdminClient) ReInitFormat(dryRun bool) error {
if objectAPI == nil {
return errServerNotInitialized
}
_, err := objectAPI.HealFormat(context.Background(), dryRun)
return err
return objectAPI.ReloadFormat(context.Background(), dryRun)
}
// ListLocks - Fetches lock information from local lock instrumentation.
@@ -201,7 +200,7 @@ func (rc remoteAdminClient) WriteTmpConfig(tmpFileName string, configBytes []byt
err := rc.Call(writeTmpConfigRPC, &wArgs, &WriteConfigReply{})
if err != nil {
errorIf(err, "Failed to write temporary config file.")
logger.LogIf(context.Background(), err)
return err
}
@@ -215,7 +214,10 @@ func (lc localAdminClient) CommitConfig(tmpFileName string) error {
tmpConfigFile := filepath.Join(getConfigDir(), tmpFileName)
err := os.Rename(tmpConfigFile, configFile)
errorIf(err, fmt.Sprintf("Failed to rename %s to %s", tmpConfigFile, configFile))
reqInfo := (&logger.ReqInfo{}).AppendTags("tmpConfigFile", tmpConfigFile)
reqInfo.AppendTags("configFile", configFile)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return err
}
@@ -228,7 +230,7 @@ func (rc remoteAdminClient) CommitConfig(tmpFileName string) error {
cReply := CommitConfigReply{}
err := rc.Call(commitConfigRPC, &cArgs, &cReply)
if err != nil {
errorIf(err, "Failed to rename config file.")
logger.LogIf(context.Background(), err)
return err
}
@@ -436,7 +438,7 @@ func getPeerUptimes(peers adminPeers) (time.Duration, error) {
latestUptime := time.Duration(0)
for _, uptime := range uptimes {
if uptime.err != nil {
errorIf(uptime.err, "Unable to fetch uptime")
logger.LogIf(context.Background(), uptime.err)
continue
}
@@ -489,15 +491,17 @@ func getPeerConfig(peers adminPeers) ([]byte, error) {
// Unmarshal the received config files.
err := json.Unmarshal(configBytes, &serverConfigs[i])
if err != nil {
errorIf(err, "Failed to unmarshal serverConfig from ", peers[i].addr)
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", peers[i].addr)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return nil, err
}
}
configJSON, err := getValidServerConfig(serverConfigs, errs)
if err != nil {
errorIf(err, "Unable to find a valid server config")
return nil, errors.Trace(err)
logger.LogIf(context.Background(), err)
return nil, err
}
// Return the config.json that was present quorum or more
+11 -6
View File
@@ -26,7 +26,7 @@ import (
"time"
router "github.com/gorilla/mux"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
const adminPath = "/admin"
@@ -94,8 +94,7 @@ func (s *adminCmd) ReInitFormat(args *ReInitFormatArgs, reply *AuthRPCReply) err
if objectAPI == nil {
return errServerNotInitialized
}
_, err := objectAPI.HealFormat(context.Background(), args.DryRun)
return err
return objectAPI.ReloadFormat(context.Background(), args.DryRun)
}
// ListLocks - lists locks held by requests handled by this server instance.
@@ -176,7 +175,9 @@ type WriteConfigReply struct {
func writeTmpConfigCommon(tmpFileName string, configBytes []byte) error {
tmpConfigFile := filepath.Join(getConfigDir(), tmpFileName)
err := ioutil.WriteFile(tmpConfigFile, configBytes, 0666)
errorIf(err, fmt.Sprintf("Failed to write to temporary config file %s", tmpConfigFile))
reqInfo := (&logger.ReqInfo{}).AppendTags("tmpConfigFile", tmpConfigFile)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return err
}
@@ -209,7 +210,10 @@ func (s *adminCmd) CommitConfig(cArgs *CommitConfigArgs, cReply *CommitConfigRep
tmpConfigFile := filepath.Join(getConfigDir(), cArgs.FileName)
err := os.Rename(tmpConfigFile, configFile)
errorIf(err, fmt.Sprintf("Failed to rename %s to %s", tmpConfigFile, configFile))
reqInfo := (&logger.ReqInfo{}).AppendTags("tmpConfigFile", tmpConfigFile)
reqInfo.AppendTags("configFile", configFile)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return err
}
@@ -220,7 +224,8 @@ func registerAdminRPCRouter(mux *router.Router) error {
adminRPCServer := newRPCServer()
err := adminRPCServer.RegisterName("Admin", adminRPCHandler)
if err != nil {
return errors.Trace(err)
logger.LogIf(context.Background(), err)
return err
}
adminRouter := mux.NewRoute().PathPrefix(minioReservedBucketPath).Subrouter()
adminRouter.Path(adminPath).Handler(adminRPCServer)
+16 -10
View File
@@ -22,7 +22,6 @@ import (
"net/http"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/hash"
)
@@ -138,6 +137,7 @@ const (
ErrMissingSSECustomerKey
ErrMissingSSECustomerKeyMD5
ErrSSECustomerKeyMD5Mismatch
ErrInvalidSSECustomerParameters
// Bucket notification related errors.
ErrEventNotification
@@ -629,7 +629,7 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrInsecureSSECustomerRequest: {
Code: "InvalidRequest",
Description: errInsecureSSERequest.Error(),
Description: "Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrSSEMultipartEncrypted: {
@@ -639,7 +639,7 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrSSEEncryptedObject: {
Code: "InvalidRequest",
Description: errEncryptedObject.Error(),
Description: "The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidEncryptionParameters: {
@@ -649,27 +649,32 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrInvalidSSECustomerAlgorithm: {
Code: "InvalidArgument",
Description: errInvalidSSEAlgorithm.Error(),
Description: "Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidSSECustomerKey: {
Code: "InvalidArgument",
Description: errInvalidSSEKey.Error(),
Description: "The secret key was invalid for the specified algorithm.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrMissingSSECustomerKey: {
Code: "InvalidArgument",
Description: errMissingSSEKey.Error(),
Description: "Requests specifying Server Side Encryption with Customer provided keys must provide an appropriate secret key.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrMissingSSECustomerKeyMD5: {
Code: "InvalidArgument",
Description: errMissingSSEKeyMD5.Error(),
Description: "Requests specifying Server Side Encryption with Customer provided keys must provide the client calculated MD5 of the secret key.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrSSECustomerKeyMD5Mismatch: {
Code: "InvalidArgument",
Description: errSSEKeyMD5Mismatch.Error(),
Description: "The calculated MD5 hash of the key did not match the hash that was provided.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidSSECustomerParameters: {
Code: "InvalidArgument",
Description: "The provided encryption parameters did not match the ones used originally.",
HTTPStatusCode: http.StatusBadRequest,
},
@@ -793,7 +798,7 @@ var errorCodeResponse = map[APIErrorCode]APIError{
HTTPStatusCode: http.StatusBadRequest,
},
// Generic Invalid-Request error. Should be used for response errors only for unlikely
// corner case errors for which introducing new APIErrorCode is not worth it. errorIf()
// corner case errors for which introducing new APIErrorCode is not worth it. LogIf()
// should be used to log the error at the source of the error for debugging purposes.
ErrInvalidRequest: {
Code: "InvalidRequest",
@@ -847,7 +852,6 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
return ErrNone
}
err = errors.Cause(err)
// Verify if the underlying error is signature mismatch.
switch err {
case errSignatureMismatch:
@@ -884,6 +888,8 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
return ErrObjectTampered
case errEncryptedObject:
return ErrSSEEncryptedObject
case errInvalidSSEParameters:
return ErrInvalidSSECustomerParameters
case errSSEKeyMismatch:
return ErrAccessDenied // no access without correct key
}
-10
View File
@@ -78,13 +78,3 @@ func getObjectResources(values url.Values) (uploadID string, partNumberMarker, m
encodingType = values.Get("encoding-type")
return
}
// Validates filter values
func validateFilterValues(values []string) (err APIErrorCode) {
for _, value := range values {
if !IsValidObjectPrefix(value) {
return ErrFilterValueInvalid
}
}
return ErrNone
}
-36
View File
@@ -18,7 +18,6 @@ package cmd
import (
"net/url"
"strings"
"testing"
)
@@ -189,38 +188,3 @@ func TestGetObjectsResources(t *testing.T) {
}
}
}
// Validates if filter values are correct
func TestValidateFilterValues(t *testing.T) {
testCases := []struct {
values []string
expectedError APIErrorCode
}{
{
values: []string{""},
expectedError: ErrNone,
},
{
values: []string{"", "prefix"},
expectedError: ErrNone,
},
{
values: []string{strings.Repeat("a", 1025)},
expectedError: ErrFilterValueInvalid,
},
{
values: []string{"a\\b"},
expectedError: ErrFilterValueInvalid,
},
{
values: []string{string([]byte{0xff, 0xfe, 0xfd})},
expectedError: ErrFilterValueInvalid,
},
}
for i, testCase := range testCases {
if actualError := validateFilterValues(testCase.values); actualError != testCase.expectedError {
t.Errorf("Test %d: Expected %d, got %d", i+1, testCase.expectedError, actualError)
}
}
}
+2 -2
View File
@@ -339,7 +339,7 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
content.ETag = "\"" + object.ETag + "\""
}
content.Size = object.Size
content.StorageClass = globalMinioDefaultStorageClass
content.StorageClass = object.StorageClass
content.Owner = owner
contents = append(contents, content)
}
@@ -385,7 +385,7 @@ func generateListObjectsV2Response(bucket, prefix, token, nextToken, startAfter,
content.ETag = "\"" + object.ETag + "\""
}
content.Size = object.Size
content.StorageClass = globalMinioDefaultStorageClass
content.StorageClass = object.StorageClass
content.Owner = owner
contents = append(contents, content)
}
+2 -1
View File
@@ -20,6 +20,7 @@ import (
"net/http"
router "github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
)
// objectAPIHandler implements and provides http handlers for S3 API.
@@ -35,7 +36,7 @@ func registerAPIRouter(mux *router.Router) {
if len(cacheConfig.Drives) > 0 {
// initialize the new disk cache objects.
globalCacheObjectAPI, err = newServerCacheObjects(cacheConfig)
fatalIf(err, "Unable to initialize disk caching")
logger.FatalIf(err, "Unable to initialize disk caching")
}
// Initialize API.
+8 -4
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"context"
"encoding/base64"
"encoding/hex"
"errors"
@@ -25,6 +26,7 @@ import (
"net/http"
"strings"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/handlers"
)
@@ -114,12 +116,14 @@ func checkAdminRequestAuthType(r *http.Request, region string) APIErrorCode {
s3Err = isReqAuthenticated(r, region)
}
if s3Err != ErrNone {
errorIf(errors.New(getAPIError(s3Err).Description), "%s", dumpRequest(r))
reqInfo := (&logger.ReqInfo{}).AppendTags("requestHeaders", dumpRequest(r))
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errors.New(getAPIError(s3Err).Description))
}
return s3Err
}
func checkRequestAuthType(r *http.Request, bucket, policyAction, region string) APIErrorCode {
func checkRequestAuthType(ctx context.Context, r *http.Request, bucket, policyAction, region string) APIErrorCode {
reqAuthType := getRequestAuthType(r)
switch reqAuthType {
@@ -136,7 +140,7 @@ func checkRequestAuthType(r *http.Request, bucket, policyAction, region string)
if err != nil {
return ErrInternalError
}
return enforceBucketPolicy(bucket, policyAction, resource,
return enforceBucketPolicy(ctx, bucket, policyAction, resource,
r.Referer(), handlers.GetSourceIP(r), r.URL.Query())
}
@@ -176,7 +180,7 @@ func isReqAuthenticated(r *http.Request, region string) (s3Error APIErrorCode) {
payload, err := ioutil.ReadAll(r.Body)
if err != nil {
errorIf(err, "Unable to read request body for signature verification")
logger.LogIf(context.Background(), err)
return ErrInternalError
}
+16 -4
View File
@@ -18,9 +18,9 @@ package cmd
import (
"bufio"
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io"
"net"
@@ -29,6 +29,8 @@ import (
"strings"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
)
// Attempt to retry only this many number of times before
@@ -133,6 +135,9 @@ func (authClient *AuthRPCClient) Login() (err error) {
}
if err = rpcClient.Call(loginMethod, &loginArgs, &LoginRPCReply{}); err != nil {
// Closing the connection here.
rpcClient.Close()
// gob doesn't provide any typed errors for us to reflect
// upon, this is the only way to return proper error.
if strings.Contains(err.Error(), "gob: wrong type") {
@@ -192,11 +197,16 @@ func (authClient *AuthRPCClient) Call(serviceMethod string, args interface {
}
}
}
// gob doesn't provide any typed errors for us to reflect
// upon, this is the only way to return proper error.
if err != nil && strings.Contains(err.Error(), "gob: wrong type") {
// Close the rpc client also when the servers have mismatching rpc versions.
authClient.Close()
err = errRPCAPIVersionUnsupported
}
break
}
return err
@@ -248,7 +258,7 @@ func rpcDial(serverAddr, serviceEndpoint string, secureConn bool) (netRPCClient
Op: "dial-http",
Net: serverAddr + serviceEndpoint,
Addr: nil,
Err: fmt.Errorf("Unable to parse server address <%s>: %s", serverAddr, err),
Err: fmt.Errorf("Unable to parse server address <%s>/<%s>: %s", serverAddr, serviceEndpoint, err),
}
}
// ServerName in tls.Config needs to be specified to support SNI certificates.
@@ -264,7 +274,9 @@ func rpcDial(serverAddr, serviceEndpoint string, secureConn bool) (netRPCClient
// Print RPC connection errors that are worthy to display in log.
switch err.(type) {
case x509.HostnameError:
errorIf(err, "Unable to establish secure connection to %s", serverAddr)
reqInfo := (&logger.ReqInfo{}).AppendTags("serverAddr", serverAddr)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
return nil, &net.OpError{
@@ -302,7 +314,7 @@ func rpcDial(serverAddr, serviceEndpoint string, secureConn bool) (netRPCClient
}
if resp.Status != connectSuccessMessage {
conn.Close()
return nil, errors.New("unexpected HTTP response: " + resp.Status)
return nil, fmt.Errorf("Unexpected HTTP response: %s from %s/%s", resp.Status, serverAddr, serviceEndpoint)
}
// Initialize rpc client.
+3 -1
View File
@@ -17,11 +17,13 @@
package cmd
import (
"context"
"fmt"
"path"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
)
@@ -62,7 +64,7 @@ func (br *browserPeerAPIHandlers) SetAuthPeer(args SetAuthPeerArgs, reply *AuthR
// Save the current creds when failed to update.
globalServerConfig.SetCredential(prevCred)
errorIf(err, "Unable to update the config with new credentials sent from browser RPC.")
logger.LogIf(context.Background(), err)
return err
}
+5 -3
View File
@@ -17,9 +17,10 @@
package cmd
import (
router "github.com/gorilla/mux"
"context"
"github.com/minio/minio/pkg/errors"
router "github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
)
// Set up an RPC endpoint that receives browser related calls. The
@@ -42,7 +43,8 @@ func registerBrowserPeerRPCRouter(mux *router.Router) error {
bpRPCServer := newRPCServer()
err := bpRPCServer.RegisterName("BrowserPeer", bpHandlers)
if err != nil {
return errors.Trace(err)
logger.LogIf(context.Background(), err)
return err
}
bpRouter := mux.NewRoute().PathPrefix(minioReservedBucketPath).Subrouter()
+2 -2
View File
@@ -64,7 +64,7 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
return
}
if s3Error := checkRequestAuthType(r, bucket, "s3:ListBucket", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, bucket, "s3:ListBucket", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -134,7 +134,7 @@ func (api objectAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *http
return
}
if s3Error := checkRequestAuthType(r, bucket, "s3:ListBucket", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, bucket, "s3:ListBucket", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
+30 -28
View File
@@ -33,18 +33,17 @@ import (
mux "github.com/gorilla/mux"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/hash"
)
// http://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
// Enforces bucket policies for a bucket for a given tatusaction.
func enforceBucketPolicy(bucket, action, resource, referer, sourceIP string, queryParams url.Values) (s3Error APIErrorCode) {
func enforceBucketPolicy(ctx context.Context, bucket, action, resource, referer, sourceIP string, queryParams url.Values) (s3Error APIErrorCode) {
// Verify if bucket actually exists
objAPI := newObjectLayerFn()
if err := checkBucketExist(bucket, objAPI); err != nil {
err = errors.Cause(err)
if err := checkBucketExist(ctx, bucket, objAPI); err != nil {
switch err.(type) {
case BucketNameInvalid:
// Return error for invalid bucket name.
@@ -53,13 +52,12 @@ func enforceBucketPolicy(bucket, action, resource, referer, sourceIP string, que
// For no bucket found we return NoSuchBucket instead.
return ErrNoSuchBucket
}
errorIf(err, "Unable to read bucket policy.")
// Return internal error for any other errors so that we can investigate.
return ErrInternalError
}
// Fetch bucket policy, if policy is not set return access denied.
p, err := objAPI.GetBucketPolicy(context.Background(), bucket)
p, err := objAPI.GetBucketPolicy(ctx, bucket)
if err != nil {
return ErrAccessDenied
}
@@ -92,7 +90,10 @@ func enforceBucketPolicy(bucket, action, resource, referer, sourceIP string, que
// Check if the action is allowed on the bucket/prefix.
func isBucketActionAllowed(action, bucket, prefix string, objectAPI ObjectLayer) bool {
bp, err := objectAPI.GetBucketPolicy(context.Background(), bucket)
reqInfo := &logger.ReqInfo{BucketName: bucket}
reqInfo.AppendTags("prefix", prefix)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
bp, err := objectAPI.GetBucketPolicy(ctx, bucket)
if err != nil {
return false
}
@@ -120,10 +121,10 @@ func (api objectAPIHandlers) GetBucketLocationHandler(w http.ResponseWriter, r *
return
}
s3Error := checkRequestAuthType(r, bucket, "s3:GetBucketLocation", globalMinioDefaultRegion)
s3Error := checkRequestAuthType(ctx, r, bucket, "s3:GetBucketLocation", globalMinioDefaultRegion)
if s3Error == ErrInvalidRegion {
// Clients like boto3 send getBucketLocation() call signed with region that is configured.
s3Error = checkRequestAuthType(r, "", "s3:GetBucketLocation", globalServerConfig.GetRegion())
s3Error = checkRequestAuthType(ctx, r, "", "s3:GetBucketLocation", globalServerConfig.GetRegion())
}
if s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
@@ -179,7 +180,7 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
return
}
if s3Error := checkRequestAuthType(r, bucket, "s3:ListBucketMultipartUploads", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, bucket, "s3:ListBucketMultipartUploads", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -228,10 +229,10 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
listBuckets = api.CacheAPI().ListBuckets
}
// ListBuckets does not have any bucket action.
s3Error := checkRequestAuthType(r, "", "", globalMinioDefaultRegion)
s3Error := checkRequestAuthType(ctx, r, "", "", globalMinioDefaultRegion)
if s3Error == ErrInvalidRegion {
// Clients like boto3 send listBuckets() call signed with region that is configured.
s3Error = checkRequestAuthType(r, "", "", globalServerConfig.GetRegion())
s3Error = checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion())
}
if s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
@@ -266,7 +267,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
}
var authError APIErrorCode
if authError = checkRequestAuthType(r, bucket, "s3:DeleteObject", globalServerConfig.GetRegion()); authError != ErrNone {
if authError = checkRequestAuthType(ctx, r, bucket, "s3:DeleteObject", globalServerConfig.GetRegion()); authError != ErrNone {
// In the event access is denied, a 200 response should still be returned
// http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
if authError != ErrAccessDenied {
@@ -294,7 +295,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
// Read incoming body XML bytes.
if _, err := io.ReadFull(r.Body, deleteXMLBytes); err != nil {
errorIf(err, "Unable to read HTTP body.")
logger.LogIf(ctx, err)
writeErrorResponse(w, ErrInternalError, r.URL)
return
}
@@ -302,7 +303,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
// Unmarshal list of keys to be deleted.
deleteObjects := &DeleteObjectsRequest{}
if err := xml.Unmarshal(deleteXMLBytes, deleteObjects); err != nil {
errorIf(err, "Unable to unmarshal delete objects request XML.")
logger.LogIf(ctx, err)
writeErrorResponse(w, ErrMalformedXML, r.URL)
return
}
@@ -354,7 +355,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
deletedObjects = append(deletedObjects, object)
continue
}
if _, ok := errors.Cause(err).(ObjectNotFound); ok {
if _, ok := err.(ObjectNotFound); ok {
// If the object is not found it should be
// accounted as deleted as per S3 spec.
deletedObjects = append(deletedObjects, object)
@@ -411,7 +412,7 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
}
// PutBucket does not have any bucket action.
s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion())
s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion())
if s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
@@ -490,7 +491,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
// be loaded in memory, the remaining being put in temporary files.
reader, err := r.MultipartReader()
if err != nil {
errorIf(err, "Unable to initialize multipart reader.")
logger.LogIf(ctx, err)
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
@@ -498,7 +499,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
// Read multipart data and save in memory and in the disk if needed
form, err := reader.ReadForm(maxFormMemory)
if err != nil {
errorIf(err, "Unable to initialize multipart reader.")
logger.LogIf(ctx, err)
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
@@ -507,9 +508,9 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
defer form.RemoveAll()
// Extract all form fields
fileBody, fileName, fileSize, formValues, err := extractPostPolicyFormValues(form)
fileBody, fileName, fileSize, formValues, err := extractPostPolicyFormValues(ctx, form)
if err != nil {
errorIf(err, "Unable to parse form values.")
logger.LogIf(ctx, err)
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
@@ -584,16 +585,15 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
}
// Extract metadata to be saved from received Form.
metadata, err := extractMetadataFromHeader(formValues)
metadata, err := extractMetadataFromHeader(ctx, formValues)
if err != nil {
errorIf(err, "found invalid http request header")
writeErrorResponse(w, ErrInternalError, r.URL)
return
}
hashReader, err := hash.NewReader(fileBody, fileSize, "", "")
if err != nil {
errorIf(err, "Unable to initialize hashReader.")
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -690,7 +690,7 @@ func (api objectAPIHandlers) HeadBucketHandler(w http.ResponseWriter, r *http.Re
return
}
if s3Error := checkRequestAuthType(r, bucket, "s3:ListBucket", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, bucket, "s3:ListBucket", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponseHeadersOnly(w, s3Error)
return
}
@@ -717,7 +717,7 @@ func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.
}
// DeleteBucket does not have any bucket action.
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -736,12 +736,14 @@ func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.
// Notify all peers (including self) to update in-memory state
for addr, err := range globalNotificationSys.UpdateBucketPolicy(bucket) {
errorIf(err, "unable to update policy change in remote peer %v", addr)
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.LogIf(ctx, err)
}
globalNotificationSys.RemoveNotification(bucket)
for addr, err := range globalNotificationSys.DeleteBucket(bucket) {
errorIf(err, "unable to delete bucket in remote peer %v", addr)
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.LogIf(ctx, err)
}
// Write success response.
+17 -15
View File
@@ -23,7 +23,7 @@ import (
"net/http"
"github.com/gorilla/mux"
xerrors "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/event/target"
xnet "github.com/minio/minio/pkg/net"
@@ -53,7 +53,7 @@ func (api objectAPIHandlers) GetBucketNotificationHandler(w http.ResponseWriter,
writeErrorResponse(w, ErrNotImplemented, r.URL)
return
}
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -63,17 +63,15 @@ func (api objectAPIHandlers) GetBucketNotificationHandler(w http.ResponseWriter,
_, err := objAPI.GetBucketInfo(ctx, bucketName)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Attempt to successfully load notification config.
nConfig, err := readNotificationConfig(objAPI, bucketName)
nConfig, err := readNotificationConfig(ctx, objAPI, bucketName)
if err != nil {
// Ignore errNoSuchNotifications to comply with AWS S3.
if xerrors.Cause(err) != errNoSuchNotifications {
errorIf(err, "Unable to read notification configuration.")
if err != errNoSuchNotifications {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -83,7 +81,7 @@ func (api objectAPIHandlers) GetBucketNotificationHandler(w http.ResponseWriter,
notificationBytes, err := xml.Marshal(nConfig)
if err != nil {
errorIf(err, "Unable to marshal notification configuration into XML.", err)
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -106,7 +104,7 @@ func (api objectAPIHandlers) PutBucketNotificationHandler(w http.ResponseWriter,
writeErrorResponse(w, ErrNotImplemented, r.URL)
return
}
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -146,7 +144,8 @@ func (api objectAPIHandlers) PutBucketNotificationHandler(w http.ResponseWriter,
rulesMap := config.ToRulesMap()
globalNotificationSys.AddRulesMap(bucketName, rulesMap)
for addr, err := range globalNotificationSys.PutBucketNotification(bucketName, rulesMap) {
errorIf(err, "unable to put bucket notification to remote peer %v", addr)
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.LogIf(ctx, err)
}
writeSuccessResponseHeadersOnly(w)
@@ -167,7 +166,7 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
writeErrorResponse(w, ErrNotImplemented, r.URL)
return
}
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -217,7 +216,6 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
}
if _, err := objAPI.GetBucketInfo(ctx, bucketName); err != nil {
errorIf(err, "Unable to get bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -227,7 +225,8 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
rulesMap := event.NewRulesMap(eventNames, pattern, target.ID())
if err := globalNotificationSys.AddRemoteTarget(bucketName, target, rulesMap); err != nil {
errorIf(err, "Unable to add httpclient target %v to globalNotificationSys.targetList.", target)
logger.GetReqInfo(ctx).AppendTags("target", target.ID().Name)
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -236,20 +235,23 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
thisAddr := xnet.MustParseHost(GetLocalPeer(globalEndpoints))
if err := SaveListener(objAPI, bucketName, eventNames, pattern, target.ID(), *thisAddr); err != nil {
errorIf(err, "Unable to save HTTP listener %v", target)
logger.GetReqInfo(ctx).AppendTags("target", target.ID().Name)
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
errors := globalNotificationSys.ListenBucketNotification(bucketName, eventNames, pattern, target.ID(), *thisAddr)
for addr, err := range errors {
errorIf(err, "unable to call listen bucket notification to remote peer %v", addr)
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.LogIf(ctx, err)
}
<-target.DoneCh
if err := RemoveListener(objAPI, bucketName, target.ID(), *thisAddr); err != nil {
errorIf(err, "Unable to save HTTP listener %v", target)
logger.GetReqInfo(ctx).AppendTags("target", target.ID().Name)
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
+10 -9
View File
@@ -28,7 +28,7 @@ import (
humanize "github.com/dustin/go-humanize"
mux "github.com/gorilla/mux"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/wildcard"
)
@@ -228,7 +228,7 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
return
}
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -260,7 +260,7 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
// bucket policies are limited to 20KB in size, using a limit reader.
policyBytes, err := ioutil.ReadAll(io.LimitReader(r.Body, maxAccessPolicySize))
if err != nil {
errorIf(err, "Unable to read from client.")
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -276,7 +276,6 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
}
if err = objAPI.SetBucketPolicy(ctx, bucket, policyInfo); err != nil {
err = errors.Cause(err)
switch err.(type) {
case NotImplemented:
// Return error for invalid bucket name.
@@ -288,7 +287,8 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
}
for addr, err := range globalNotificationSys.UpdateBucketPolicy(bucket) {
errorIf(err, "unable to update policy change in remote peer %v", addr)
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.LogIf(ctx, err)
}
// Success.
@@ -308,7 +308,7 @@ func (api objectAPIHandlers) DeleteBucketPolicyHandler(w http.ResponseWriter, r
return
}
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -331,7 +331,8 @@ func (api objectAPIHandlers) DeleteBucketPolicyHandler(w http.ResponseWriter, r
}
for addr, err := range globalNotificationSys.UpdateBucketPolicy(bucket) {
errorIf(err, "unable to update policy change in remote peer %v", addr)
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.LogIf(ctx, err)
}
// Success.
@@ -351,7 +352,7 @@ func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *ht
return
}
if s3Error := checkRequestAuthType(r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, "", "", globalServerConfig.GetRegion()); s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -375,7 +376,7 @@ func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *ht
policyBytes, err := json.Marshal(&policy)
if err != nil {
errorIf(err, "Unable to marshal bucket policy.")
logger.LogIf(ctx, err)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
+19 -20
View File
@@ -25,7 +25,7 @@ import (
"sync"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/hash"
)
@@ -85,7 +85,7 @@ func initBucketPolicies(objAPI ObjectLayer) (*bucketPolicies, error) {
// List buckets to proceed loading all notification configuration.
buckets, err := objAPI.ListBuckets(context.Background())
if err != nil {
return nil, errors.Cause(err)
return nil, err
}
policies := make(map[string]policy.BucketAccessPolicy)
@@ -95,9 +95,9 @@ func initBucketPolicies(objAPI ObjectLayer) (*bucketPolicies, error) {
if pErr != nil {
// net.Dial fails for rpc client or any
// other unexpected errors during net.Dial.
if !errors.IsErrIgnored(pErr, errDiskNotFound) {
if !IsErrIgnored(pErr, errDiskNotFound) {
if !isErrBucketPolicyNotFound(pErr) {
return nil, errors.Cause(pErr)
return nil, pErr
}
}
// Continue to load other bucket policies if possible.
@@ -119,13 +119,13 @@ func readBucketPolicyJSON(bucket string, objAPI ObjectLayer) (bucketPolicyReader
policyPath := pathJoin(bucketConfigPrefix, bucket, bucketPolicyConfig)
var buffer bytes.Buffer
err = objAPI.GetObject(context.Background(), minioMetaBucket, policyPath, 0, -1, &buffer, "")
ctx := logger.SetReqInfo(context.Background(), &logger.ReqInfo{BucketName: bucket})
err = objAPI.GetObject(ctx, minioMetaBucket, policyPath, 0, -1, &buffer, "")
if err != nil {
if isErrObjectNotFound(err) || isErrIncompleteBody(err) {
return nil, PolicyNotFound{Bucket: bucket}
}
errorIf(err, "Unable to load policy for the bucket %s.", bucket)
return nil, errors.Cause(err)
return nil, err
}
return &buffer, nil
@@ -151,11 +151,10 @@ func ReadBucketPolicy(bucket string, objAPI ObjectLayer) (policy.BucketAccessPol
// removeBucketPolicy - removes any previously written bucket policy. Returns BucketPolicyNotFound
// if no policies are found.
func removeBucketPolicy(bucket string, objAPI ObjectLayer) error {
func removeBucketPolicy(ctx context.Context, bucket string, objAPI ObjectLayer) error {
policyPath := pathJoin(bucketConfigPrefix, bucket, bucketPolicyConfig)
err := objAPI.DeleteObject(context.Background(), minioMetaBucket, policyPath)
err := objAPI.DeleteObject(ctx, minioMetaBucket, policyPath)
if err != nil {
err = errors.Cause(err)
if _, ok := err.(ObjectNotFound); ok {
return BucketPolicyNotFound{Bucket: bucket}
}
@@ -165,22 +164,22 @@ func removeBucketPolicy(bucket string, objAPI ObjectLayer) error {
}
// writeBucketPolicy - save a bucket policy that is assumed to be validated.
func writeBucketPolicy(bucket string, objAPI ObjectLayer, bpy policy.BucketAccessPolicy) error {
func writeBucketPolicy(ctx context.Context, bucket string, objAPI ObjectLayer, bpy policy.BucketAccessPolicy) error {
buf, err := json.Marshal(bpy)
if err != nil {
errorIf(err, "Unable to marshal bucket policy '%#v' to JSON", bpy)
logger.LogIf(ctx, err)
return err
}
policyPath := pathJoin(bucketConfigPrefix, bucket, bucketPolicyConfig)
hashReader, err := hash.NewReader(bytes.NewReader(buf), int64(len(buf)), "", getSHA256Hash(buf))
if err != nil {
errorIf(err, "Unable to set policy for the bucket %s", bucket)
return errors.Cause(err)
logger.LogIf(ctx, err)
return err
}
if _, err = objAPI.PutObject(context.Background(), minioMetaBucket, policyPath, hashReader, nil); err != nil {
errorIf(err, "Unable to set policy for the bucket %s", bucket)
return errors.Cause(err)
if _, err = objAPI.PutObject(ctx, minioMetaBucket, policyPath, hashReader, nil); err != nil {
return err
}
return nil
}
@@ -188,9 +187,9 @@ func writeBucketPolicy(bucket string, objAPI ObjectLayer, bpy policy.BucketAcces
// persistAndNotifyBucketPolicyChange - takes a policyChange argument,
// persists it to storage, and notify nodes in the cluster about the
// change. In-memory state is updated in response to the notification.
func persistAndNotifyBucketPolicyChange(bucket string, isRemove bool, bktPolicy policy.BucketAccessPolicy, objAPI ObjectLayer) error {
func persistAndNotifyBucketPolicyChange(ctx context.Context, bucket string, isRemove bool, bktPolicy policy.BucketAccessPolicy, objAPI ObjectLayer) error {
if isRemove {
err := removeBucketPolicy(bucket, objAPI)
err := removeBucketPolicy(ctx, bucket, objAPI)
if err != nil {
return err
}
@@ -198,7 +197,7 @@ func persistAndNotifyBucketPolicyChange(bucket string, isRemove bool, bktPolicy
if reflect.DeepEqual(bktPolicy, emptyBucketPolicy) {
return errInvalidArgument
}
if err := writeBucketPolicy(bucket, objAPI, bktPolicy); err != nil {
if err := writeBucketPolicy(ctx, bucket, objAPI, bktPolicy); err != nil {
return err
}
}
+1 -3
View File
@@ -16,13 +16,11 @@
package cmd
import "go/build"
// DO NOT EDIT THIS FILE DIRECTLY. These are build-time constants
// set through ‘buildscripts/gen-ldflags.go’.
var (
// GOPATH - GOPATH value at the time of build.
GOPATH = build.Default.GOPATH
GOPATH = ""
// Go get development tag.
goGetTag = "DEVELOPMENT.GOGET"
+20 -19
View File
@@ -25,6 +25,7 @@ import (
"time"
"github.com/minio/cli"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
)
@@ -33,9 +34,9 @@ func checkUpdate(mode string) {
// Its OK to ignore any errors during doUpdate() here.
if updateMsg, _, currentReleaseTime, latestReleaseTime, err := getUpdateInfo(2*time.Second, mode); err == nil {
if globalInplaceUpdateDisabled {
log.Println(updateMsg)
logger.StartupMessage(updateMsg)
} else {
log.Println(prepareUpdateMessage("Run `minio update`", latestReleaseTime.Sub(currentReleaseTime)))
logger.StartupMessage(prepareUpdateMessage("Run `minio update`", latestReleaseTime.Sub(currentReleaseTime)))
}
}
}
@@ -43,11 +44,11 @@ func checkUpdate(mode string) {
func initConfig() {
// Config file does not exist, we create it fresh and return upon success.
if isFile(getConfigFile()) {
fatalIf(migrateConfig(), "Config migration failed.")
fatalIf(loadConfig(), "Unable to load config version: '%s'.", serverConfigVersion)
logger.FatalIf(migrateConfig(), "Config migration failed.")
logger.FatalIf(loadConfig(), "Unable to load config version: '%s'.", serverConfigVersion)
} else {
fatalIf(newConfig(), "Unable to initialize minio config for the first time.")
log.Println("Created minio configuration file successfully at " + getConfigDir())
logger.FatalIf(newConfig(), "Unable to initialize minio config for the first time.")
logger.Info("Created minio configuration file successfully at " + getConfigDir())
}
}
@@ -70,17 +71,17 @@ func handleCommonCmdArgs(ctx *cli.Context) {
// default config directory.
configDir = getConfigDir()
if configDir == "" {
fatalIf(errors.New("missing option"), "config-dir option must be provided.")
logger.FatalIf(errors.New("missing option"), "config-dir option must be provided.")
}
}
if configDir == "" {
fatalIf(errors.New("empty directory"), "Configuration directory cannot be empty.")
logger.FatalIf(errors.New("empty directory"), "Configuration directory cannot be empty.")
}
// Disallow relative paths, figure out absolute paths.
configDirAbs, err := filepath.Abs(configDir)
fatalIf(err, "Unable to fetch absolute path for config directory %s", configDir)
logger.FatalIf(err, "Unable to fetch absolute path for config directory %s", configDir)
setConfigDir(configDirAbs)
}
@@ -94,7 +95,7 @@ func handleCommonEnvVars() {
secretKey := os.Getenv("MINIO_SECRET_KEY")
if accessKey != "" && secretKey != "" {
cred, err := auth.CreateCredentials(accessKey, secretKey)
fatalIf(err, "Invalid access/secret Key set in environment.")
logger.FatalIf(err, "Invalid access/secret Key set in environment.")
// credential Envs are set globally.
globalIsEnvCreds = true
@@ -104,7 +105,7 @@ func handleCommonEnvVars() {
if browser := os.Getenv("MINIO_BROWSER"); browser != "" {
browserFlag, err := ParseBrowserFlag(browser)
if err != nil {
fatalIf(errors.New("invalid value"), "Unknown value ‘%s’ in MINIO_BROWSER environment variable.", browser)
logger.FatalIf(errors.New("invalid value"), "Unknown value ‘%s’ in MINIO_BROWSER environment variable.", browser)
}
// browser Envs are set globally, this does not represent
@@ -117,7 +118,7 @@ func handleCommonEnvVars() {
if traceFile != "" {
var err error
globalHTTPTraceFile, err = os.OpenFile(traceFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0660)
fatalIf(err, "error opening file %s", traceFile)
logger.FatalIf(err, "error opening file %s", traceFile)
}
globalDomainName = os.Getenv("MINIO_DOMAIN")
@@ -127,18 +128,18 @@ func handleCommonEnvVars() {
if drives := os.Getenv("MINIO_CACHE_DRIVES"); drives != "" {
driveList, err := parseCacheDrives(strings.Split(drives, cacheEnvDelimiter))
fatalIf(err, "Invalid value set in environment variable MINIO_CACHE_DRIVES %s.", drives)
logger.FatalIf(err, "Invalid value set in environment variable MINIO_CACHE_DRIVES %s.", drives)
globalCacheDrives = driveList
globalIsDiskCacheEnabled = true
}
if excludes := os.Getenv("MINIO_CACHE_EXCLUDE"); excludes != "" {
excludeList, err := parseCacheExcludes(strings.Split(excludes, cacheEnvDelimiter))
fatalIf(err, "Invalid value set in environment variable MINIO_CACHE_EXCLUDE %s.", excludes)
logger.FatalIf(err, "Invalid value set in environment variable MINIO_CACHE_EXCLUDE %s.", excludes)
globalCacheExcludes = excludeList
}
if expiryStr := os.Getenv("MINIO_CACHE_EXPIRY"); expiryStr != "" {
expiry, err := strconv.Atoi(expiryStr)
fatalIf(err, "Invalid value set in environment variable MINIO_CACHE_EXPIRY %s.", expiryStr)
logger.FatalIf(err, "Invalid value set in environment variable MINIO_CACHE_EXPIRY %s.", expiryStr)
globalCacheExpiry = expiry
}
@@ -154,25 +155,25 @@ func handleCommonEnvVars() {
// Check for environment variables and parse into storageClass struct
if ssc := os.Getenv(standardStorageClassEnv); ssc != "" {
globalStandardStorageClass, err = parseStorageClass(ssc)
fatalIf(err, "Invalid value set in environment variable %s.", standardStorageClassEnv)
logger.FatalIf(err, "Invalid value set in environment variable %s.", standardStorageClassEnv)
}
if rrsc := os.Getenv(reducedRedundancyStorageClassEnv); rrsc != "" {
globalRRStorageClass, err = parseStorageClass(rrsc)
fatalIf(err, "Invalid value set in environment variable %s.", reducedRedundancyStorageClassEnv)
logger.FatalIf(err, "Invalid value set in environment variable %s.", reducedRedundancyStorageClassEnv)
}
// Validation is done after parsing both the storage classes. This is needed because we need one
// storage class value to deduce the correct value of the other storage class.
if globalRRStorageClass.Scheme != "" {
err = validateParity(globalStandardStorageClass.Parity, globalRRStorageClass.Parity)
fatalIf(err, "Invalid value set in environment variable %s.", reducedRedundancyStorageClassEnv)
logger.FatalIf(err, "Invalid value set in environment variable %s.", reducedRedundancyStorageClassEnv)
globalIsStorageClass = true
}
if globalStandardStorageClass.Scheme != "" {
err = validateParity(globalStandardStorageClass.Parity, globalRRStorageClass.Parity)
fatalIf(err, "Invalid value set in environment variable %s.", standardStorageClassEnv)
logger.FatalIf(err, "Invalid value set in environment variable %s.", standardStorageClassEnv)
globalIsStorageClass = true
}
}
+4 -66
View File
@@ -19,7 +19,6 @@ package cmd
import (
"errors"
"fmt"
"io/ioutil"
"reflect"
"sync"
@@ -27,7 +26,6 @@ import (
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/event/target"
"github.com/minio/minio/pkg/quick"
"github.com/tidwall/gjson"
)
// Steps to move from version N to version N+1
@@ -139,6 +137,8 @@ func (s *serverConfig) ConfigDiff(t *serverConfig) string {
return "Domain configuration differs"
case s.StorageClass != t.StorageClass:
return "StorageClass configuration differs"
case !reflect.DeepEqual(s.Cache, t.Cache):
return "Cache configuration differs"
case !reflect.DeepEqual(s.Notify.AMQP, t.Notify.AMQP):
return "AMQP Notification configuration differs"
case !reflect.DeepEqual(s.Notify.NATS, t.Notify.NATS):
@@ -252,57 +252,6 @@ func newConfig() error {
return globalServerConfig.Save()
}
// doCheckDupJSONKeys recursively detects duplicate json keys
func doCheckDupJSONKeys(key, value gjson.Result) error {
// Key occurrences map of the current scope to count
// if there is any duplicated json key.
keysOcc := make(map[string]int)
// Holds the found error
var checkErr error
// Iterate over keys in the current json scope
value.ForEach(func(k, v gjson.Result) bool {
// If current key is not null, check if its
// value contains some duplicated keys.
if k.Type != gjson.Null {
keysOcc[k.String()]++
checkErr = doCheckDupJSONKeys(k, v)
}
return checkErr == nil
})
// Check found err
if checkErr != nil {
return errors.New(key.String() + " => " + checkErr.Error())
}
// Check for duplicated keys
for k, v := range keysOcc {
if v > 1 {
return errors.New(key.String() + " => `" + k + "` entry is duplicated")
}
}
return nil
}
// Check recursively if a key is duplicated in the same json scope
// e.g.:
// `{ "key" : { "key" ..` is accepted
// `{ "key" : { "subkey" : "val1", "subkey": "val2" ..` throws subkey duplicated error
func checkDupJSONKeys(json string) error {
// Parse config with gjson library
config := gjson.Parse(json)
// Create a fake rootKey since root json doesn't seem to have representation
// in gjson library.
rootKey := gjson.Result{Type: gjson.String, Str: minioConfigFile}
// Check if loaded json contains any duplicated keys
return doCheckDupJSONKeys(rootKey, config)
}
// getValidConfig - returns valid server configuration
func getValidConfig() (*serverConfig, error) {
srvCfg := &serverConfig{
@@ -310,8 +259,7 @@ func getValidConfig() (*serverConfig, error) {
Browser: true,
}
configFile := getConfigFile()
if _, err := quick.Load(configFile, srvCfg); err != nil {
if _, err := quick.Load(getConfigFile(), srvCfg); err != nil {
return nil, err
}
@@ -319,21 +267,11 @@ func getValidConfig() (*serverConfig, error) {
return nil, fmt.Errorf("configuration version mismatch. Expected: ‘%s’, Got: ‘%s’", serverConfigVersion, srvCfg.Version)
}
// Load config file json and check for duplication json keys
jsonBytes, err := ioutil.ReadFile(configFile)
if err != nil {
return nil, err
}
if err = checkDupJSONKeys(string(jsonBytes)); err != nil {
return nil, err
}
// Validate credential fields only when
// they are not set via the environment
// Error out if global is env credential is not set and config has invalid credential
if !globalIsEnvCreds && !srvCfg.Credential.IsValid() {
return nil, errors.New("invalid credential in config file " + configFile)
return nil, errors.New("invalid credential in config file " + getConfigFile())
}
return srvCfg, nil
-29
View File
@@ -24,7 +24,6 @@ import (
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/event/target"
"github.com/tidwall/gjson"
)
func TestServerConfig(t *testing.T) {
@@ -126,34 +125,6 @@ func TestServerConfigWithEnvs(t *testing.T) {
}
}
func TestCheckDupJSONKeys(t *testing.T) {
testCases := []struct {
json string
shouldPass bool
}{
{`{}`, true},
{`{"version" : "13"}`, true},
{`{"version" : "13", "version": "14"}`, false},
{`{"version" : "13", "credential": {"accessKey": "12345"}}`, true},
{`{"version" : "13", "credential": {"accessKey": "12345", "accessKey":"12345"}}`, false},
{`{"version" : "13", "notify": {"amqp": {"1"}, "webhook":{"3"}}}`, true},
{`{"version" : "13", "notify": {"amqp": {"1"}, "amqp":{"3"}}}`, false},
{`{"version" : "13", "notify": {"amqp": {"1":{}, "2":{}}}}`, true},
{`{"version" : "13", "notify": {"amqp": {"1":{}, "1":{}}}}`, false},
}
for i, testCase := range testCases {
err := doCheckDupJSONKeys(gjson.Result{}, gjson.Parse(testCase.json))
if testCase.shouldPass && err != nil {
t.Errorf("Test %d, should pass but it failed with err = %v", i+1, err)
}
if !testCase.shouldPass && err == nil {
t.Errorf("Test %d, should fail but it succeed.", i+1)
}
}
}
// Tests config validator..
func TestValidateConfig(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
+24 -23
View File
@@ -21,6 +21,7 @@ import (
"os"
"path/filepath"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/event/target"
@@ -29,7 +30,7 @@ import (
)
// DO NOT EDIT following message template, please open a github issue to discuss instead.
var configMigrateMSGTemplate = "Configuration file %s migrated from version '%s' to '%s' successfully.\n"
var configMigrateMSGTemplate = "Configuration file %s migrated from version '%s' to '%s' successfully."
// Migrates all config versions from "1" to "18".
func migrateConfig() error {
@@ -194,7 +195,7 @@ func purgeV1() error {
}
os.RemoveAll(configFile)
log.Println("Removed unsupported config version ‘1’.")
logger.Info("Removed unsupported config version ‘1’.")
return nil
}
@@ -252,7 +253,7 @@ func migrateV2ToV3() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv2.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv2.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv2.Version, srvConfig.Version)
return nil
}
@@ -290,7 +291,7 @@ func migrateV3ToV4() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv3.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv3.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv3.Version, srvConfig.Version)
return nil
}
@@ -331,7 +332,7 @@ func migrateV4ToV5() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv4.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv4.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv4.Version, srvConfig.Version)
return nil
}
@@ -420,7 +421,7 @@ func migrateV5ToV6() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv5.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv5.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv5.Version, srvConfig.Version)
return nil
}
@@ -476,7 +477,7 @@ func migrateV6ToV7() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv6.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv6.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv6.Version, srvConfig.Version)
return nil
}
@@ -539,7 +540,7 @@ func migrateV7ToV8() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv7.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv7.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv7.Version, srvConfig.Version)
return nil
}
@@ -609,7 +610,7 @@ func migrateV8ToV9() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv8.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv8.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv8.Version, srvConfig.Version)
return nil
}
@@ -677,7 +678,7 @@ func migrateV9ToV10() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv9.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv9.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv9.Version, srvConfig.Version)
return nil
}
@@ -748,7 +749,7 @@ func migrateV10ToV11() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv10.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv10.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv10.Version, srvConfig.Version)
return nil
}
@@ -846,7 +847,7 @@ func migrateV11ToV12() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv11.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv11.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv11.Version, srvConfig.Version)
return nil
}
@@ -926,7 +927,7 @@ func migrateV12ToV13() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv12.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv12.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv12.Version, srvConfig.Version)
return nil
}
@@ -1011,7 +1012,7 @@ func migrateV13ToV14() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv13.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv13.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv13.Version, srvConfig.Version)
return nil
}
@@ -1100,7 +1101,7 @@ func migrateV14ToV15() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv14.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv14.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv14.Version, srvConfig.Version)
return nil
}
@@ -1190,7 +1191,7 @@ func migrateV15ToV16() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv15.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv15.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv15.Version, srvConfig.Version)
return nil
}
@@ -1311,7 +1312,7 @@ func migrateV16ToV17() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv16.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv16.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv16.Version, srvConfig.Version)
return nil
}
@@ -1415,7 +1416,7 @@ func migrateV17ToV18() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv17.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv17.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv17.Version, srvConfig.Version)
return nil
}
@@ -1521,7 +1522,7 @@ func migrateV18ToV19() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv18.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv18.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv18.Version, srvConfig.Version)
return nil
}
@@ -1626,7 +1627,7 @@ func migrateV19ToV20() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv19.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv19.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv19.Version, srvConfig.Version)
return nil
}
@@ -1730,7 +1731,7 @@ func migrateV20ToV21() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv20.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv20.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv20.Version, srvConfig.Version)
return nil
}
@@ -1834,7 +1835,7 @@ func migrateV21ToV22() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv21.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv21.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv21.Version, srvConfig.Version)
return nil
}
@@ -1947,6 +1948,6 @@ func migrateV22ToV23() error {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv22.Version, srvConfig.Version, err)
}
log.Printf(configMigrateMSGTemplate, configFile, cv22.Version, srvConfig.Version)
logger.Info(configMigrateMSGTemplate, configFile, cv22.Version, srvConfig.Version)
return nil
}
+4 -1
View File
@@ -580,7 +580,10 @@ type serverConfigV22 struct {
Notify notifier `json:"notify"`
}
// serverConfigV23 is just like version '22' with addition of cache field
// serverConfigV23 is just like version '22' with addition of cache field.
//
// IMPORTANT NOTE: When updating this struct make sure that
// serverConfig.ConfigDiff() is updated as necessary.
type serverConfigV23 struct {
Version string `json:"version"`
+53 -42
View File
@@ -28,8 +28,8 @@ import (
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/disk"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/lock"
)
@@ -37,7 +37,6 @@ import (
const (
// cache.json object metadata for cached objects.
cacheMetaJSONFile = "cache.json"
cacheMetaFormat = "cache"
cacheEnvDelimiter = ";"
)
@@ -93,7 +92,7 @@ func newCacheFSObjects(dir string, expiry int, maxDiskUsagePct int) (*cacheFSObj
appendFileMap: make(map[string]*fsAppendFile),
}
go fsObjects.cleanupStaleMultipartUploads(globalMultipartCleanupInterval, globalMultipartExpiry, globalServiceDoneCh)
go fsObjects.cleanupStaleMultipartUploads(context.Background(), globalMultipartCleanupInterval, globalMultipartExpiry, globalServiceDoneCh)
cacheFS := cacheFSObjects{
FSObjects: fsObjects,
@@ -116,7 +115,9 @@ func (cfs *cacheFSObjects) diskUsageLow() bool {
minUsage := cfs.maxDiskUsagePct * 80 / 100
di, err := disk.GetInfo(cfs.dir)
if err != nil {
errorIf(err, "Error getting disk information on %s", cfs.dir)
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", cfs.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return false
}
usedPercent := (di.Total - di.Free) * 100 / di.Total
@@ -128,7 +129,9 @@ func (cfs *cacheFSObjects) diskUsageLow() bool {
func (cfs *cacheFSObjects) diskUsageHigh() bool {
di, err := disk.GetInfo(cfs.dir)
if err != nil {
errorIf(err, "Error getting disk information on %s", cfs.dir)
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", cfs.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return true
}
usedPercent := (di.Total - di.Free) * 100 / di.Total
@@ -140,7 +143,9 @@ func (cfs *cacheFSObjects) diskUsageHigh() bool {
func (cfs *cacheFSObjects) diskAvailable(size int64) bool {
di, err := disk.GetInfo(cfs.dir)
if err != nil {
errorIf(err, "Error getting disk information on %s", cfs.dir)
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", cfs.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return false
}
usedPercent := (di.Total - (di.Free - uint64(size))) * 100 / di.Total
@@ -163,14 +168,15 @@ func (cfs *cacheFSObjects) purgeTrash() {
return
}
for _, entry := range entries {
fi, err := fsStatVolume(pathJoin(trashPath, entry))
ctx := logger.SetReqInfo(context.Background(), &logger.ReqInfo{})
fi, err := fsStatVolume(ctx, pathJoin(trashPath, entry))
if err != nil {
continue
}
dir := path.Join(trashPath, fi.Name())
// Delete all expired cache content.
fsRemoveAll(dir)
fsRemoveAll(ctx, dir)
}
}
}
@@ -193,7 +199,7 @@ func (cfs *cacheFSObjects) purge() {
deletedCount := 0
buckets, err := cfs.ListBuckets(ctx)
if err != nil {
errorIf(err, "Unable to list buckets.")
logger.LogIf(ctx, err)
}
// Reset cache online status if drive was offline earlier.
if !cfs.IsOnline() {
@@ -221,7 +227,7 @@ func (cfs *cacheFSObjects) purge() {
continue
}
if err = cfs.DeleteObject(ctx, bucket.Name, object.Name); err != nil {
errorIf(err, "Unable to remove cache entry in dir %s/%s", bucket.Name, object.Name)
logger.LogIf(ctx, err)
continue
}
deletedCount++
@@ -271,7 +277,7 @@ func (cfs *cacheFSObjects) Put(ctx context.Context, bucket, object string, data
}
_, err := cfs.PutObject(ctx, bucket, object, data, metadata)
// if err is due to disk being offline , mark cache drive as offline
if errors2.IsErr(err, baseErrs...) {
if IsErr(err, baseErrs...) {
cfs.setOnline(false)
}
return err
@@ -313,7 +319,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
var err error
// Validate if bucket name is valid and exists.
if _, err = fs.statBucketDir(bucket); err != nil {
if _, err = fs.statBucketDir(ctx, bucket); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket)
}
@@ -325,31 +331,32 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
// and return success.
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errors2.Trace(errFileAccessDenied), bucket, object)
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
var fi os.FileInfo
if fi, err = fsStatDir(pathJoin(fs.fsPath, bucket, object)); err != nil {
if fi, err = fsStatDir(ctx, pathJoin(fs.fsPath, bucket, object)); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
if err = checkPutObjectArgs(bucket, object, fs, data.Size()); err != nil {
if err = checkPutObjectArgs(ctx, bucket, object, fs, data.Size()); err != nil {
return ObjectInfo{}, err
}
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errors2.Trace(errFileAccessDenied), bucket, object)
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
}
// Validate input data size and it can never be less than zero.
if data.Size() < 0 {
return ObjectInfo{}, errors2.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return ObjectInfo{}, errInvalidArgument
}
var wlk *lock.LockedFile
@@ -359,7 +366,8 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
wlk, err = fs.rwPool.Create(fsMetaPath)
if err != nil {
return ObjectInfo{}, toObjectErr(errors2.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
// This close will allow for locks to be synchronized on `fs.json`.
defer wlk.Close()
@@ -367,7 +375,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
// Remove meta file when PutObject encounters any error
if retErr != nil {
tmpDir := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID)
fsRemoveMeta(bucketMetaDir, fsMetaPath, tmpDir)
fsRemoveMeta(ctx, bucketMetaDir, fsMetaPath, tmpDir)
}
}()
}
@@ -385,10 +393,9 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
buf := make([]byte, int(bufSize))
fsTmpObjPath := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID, tempObj)
bytesWritten, err := fsCreateFile(fsTmpObjPath, data, buf, data.Size())
bytesWritten, err := fsCreateFile(ctx, fsTmpObjPath, data, buf, data.Size())
if err != nil {
fsRemoveFile(fsTmpObjPath)
errorIf(err, "Failed to create object %s/%s", bucket, object)
fsRemoveFile(ctx, fsTmpObjPath)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
if fsMeta.Meta["etag"] == "" {
@@ -397,18 +404,18 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
// Should return IncompleteBody{} error when reader has fewer
// bytes than specified in request header.
if bytesWritten < data.Size() {
fsRemoveFile(fsTmpObjPath)
return ObjectInfo{}, errors2.Trace(IncompleteBody{})
fsRemoveFile(ctx, fsTmpObjPath)
return ObjectInfo{}, IncompleteBody{}
}
// Delete the temporary object in the case of a
// failure. If PutObject succeeds, then there would be
// nothing to delete.
defer fsRemoveFile(fsTmpObjPath)
defer fsRemoveFile(ctx, fsTmpObjPath)
// Entire object was written to the temp location, now it's safe to rename it to the actual location.
fsNSObjPath := pathJoin(fs.fsPath, bucket, object)
if err = fsRenameFile(fsTmpObjPath, fsNSObjPath); err != nil {
if err = fsRenameFile(ctx, fsTmpObjPath, fsNSObjPath); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
@@ -420,7 +427,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
}
// Stat the file to fetch timestamp, size.
fi, err := fsStatFile(pathJoin(fs.fsPath, bucket, object))
fi, err := fsStatFile(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
@@ -447,11 +454,11 @@ func (cfs *cacheFSObjects) NewMultipartUpload(ctx context.Context, bucket, objec
}
}
fs := cfs.FSObjects
if err := checkNewMultipartArgs(bucket, object, fs); err != nil {
if err := checkNewMultipartArgs(ctx, bucket, object, fs); err != nil {
return "", toObjectErr(err, bucket)
}
if _, err := fs.statBucketDir(bucket); err != nil {
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return "", toObjectErr(err, bucket)
}
@@ -459,7 +466,8 @@ func (cfs *cacheFSObjects) NewMultipartUpload(ctx context.Context, bucket, objec
err := mkdirAll(uploadIDDir, 0755)
if err != nil {
return "", errors2.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
// Initialize fs.json values.
@@ -468,11 +476,13 @@ func (cfs *cacheFSObjects) NewMultipartUpload(ctx context.Context, bucket, objec
fsMetaBytes, err := json.Marshal(fsMeta)
if err != nil {
return "", errors2.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
if err = ioutil.WriteFile(pathJoin(uploadIDDir, fs.metaJSONFile), fsMetaBytes, 0644); err != nil {
return "", errors2.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
return uploadID, nil
}
@@ -485,7 +495,7 @@ func (cfs *cacheFSObjects) moveBucketToTrash(ctx context.Context, bucket string)
return err
}
defer bucketLock.Unlock()
bucketDir, err := fs.getBucketDir(bucket)
bucketDir, err := fs.getBucketDir(ctx, bucket)
if err != nil {
return toObjectErr(err, bucket)
}
@@ -493,12 +503,13 @@ func (cfs *cacheFSObjects) moveBucketToTrash(ctx context.Context, bucket string)
expiredDir := path.Join(trashPath, bucket)
// Attempt to move regular bucket to expired directory.
if err = fsRenameDir(bucketDir, expiredDir); err != nil {
logger.LogIf(ctx, err)
return toObjectErr(err, bucket)
}
// Cleanup all the bucket metadata.
ominioMetadataBucketDir := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, bucket)
nminioMetadataBucketDir := pathJoin(trashPath, MustGetUUID())
_ = fsRenameDir(ominioMetadataBucketDir, nminioMetadataBucketDir)
logger.LogIf(ctx, fsRenameDir(ominioMetadataBucketDir, nminioMetadataBucketDir))
return nil
}
@@ -506,22 +517,22 @@ func (cfs *cacheFSObjects) moveBucketToTrash(ctx context.Context, bucket string)
// paths for windows automatically.
func fsRenameDir(dirPath, newPath string) (err error) {
if dirPath == "" || newPath == "" {
return errors2.Trace(errInvalidArgument)
return errInvalidArgument
}
if err = checkPathLength(dirPath); err != nil {
return errors2.Trace(err)
return err
}
if err = checkPathLength(newPath); err != nil {
return errors2.Trace(err)
return err
}
if err = os.Rename(dirPath, newPath); err != nil {
if os.IsNotExist(err) {
return errors2.Trace(errVolumeNotFound)
return errVolumeNotFound
} else if isSysErrNotEmpty(err) {
return errors2.Trace(errVolumeNotEmpty)
return errVolumeNotEmpty
}
return errors2.Trace(err)
return err
}
return nil
}
+14 -12
View File
@@ -31,7 +31,7 @@ import (
"github.com/djherbis/atime"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/wildcard"
"github.com/minio/minio/pkg/hash"
@@ -103,8 +103,8 @@ type CacheObjectLayer interface {
// backendDownError returns true if err is due to backend failure or faulty disk if in server mode
func backendDownError(err error) bool {
_, backendDown := errors2.Cause(err).(BackendDown)
return backendDown || errors2.IsErr(err, baseErrs...)
_, backendDown := err.(BackendDown)
return backendDown || IsErr(err, baseErrs...)
}
// get cache disk where object is currently cached for a GET operation. If object does not exist at that location,
@@ -191,7 +191,7 @@ func (c cacheObjects) GetObject(ctx context.Context, bucket, object string, star
objInfo, err := GetObjectInfoFn(ctx, bucket, object)
backendDown := backendDownError(err)
if err != nil && !backendDown {
if _, ok := errors2.Cause(err).(ObjectNotFound); ok {
if _, ok := err.(ObjectNotFound); ok {
// Delete the cached entry if backend object was deleted.
dcache.Delete(ctx, bucket, object)
}
@@ -255,7 +255,7 @@ func (c cacheObjects) GetObjectInfo(ctx context.Context, bucket, object string)
}
objInfo, err := getObjectInfoFn(ctx, bucket, object)
if err != nil {
if _, ok := errors2.Cause(err).(ObjectNotFound); ok {
if _, ok := err.(ObjectNotFound); ok {
// Delete the cached entry if backend object was deleted.
dcache.Delete(ctx, bucket, object)
return ObjectInfo{}, err
@@ -345,12 +345,12 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
if err != nil {
return false
}
_, err = fs.getObjectInfo(bucket, object)
_, err = fs.getObjectInfo(ctx, bucket, object)
return err == nil
}
listDir := listDirCacheFactory(isLeaf, cacheTreeWalkIgnoredErrs, c.cache.cfs)
walkResultCh = startTreeWalk(bucket, prefix, marker, recursive, listDir, isLeaf, endWalkCh)
walkResultCh = startTreeWalk(ctx, bucket, prefix, marker, recursive, listDir, isLeaf, endWalkCh)
}
for i := 0; i < maxKeys; {
@@ -378,15 +378,15 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
fs, err := c.cache.getCacheFS(ctx, bucket, entry)
if err != nil {
// Ignore errFileNotFound
if errors2.Cause(err) == errFileNotFound {
if err == errFileNotFound {
continue
}
return result, toObjectErr(err, bucket, prefix)
}
objInfo, err = fs.getObjectInfo(bucket, entry)
objInfo, err = fs.getObjectInfo(ctx, bucket, entry)
if err != nil {
// Ignore errFileNotFound
if errors2.Cause(err) == errFileNotFound {
if err == errFileNotFound {
continue
}
return result, toObjectErr(err, bucket, prefix)
@@ -754,7 +754,8 @@ func (c cacheObjects) StorageInfo(ctx context.Context) (storageInfo StorageInfo)
continue
}
info, err := getDiskInfo((cfs.fsPath))
errorIf(err, "Unable to get disk info %#v", cfs.fsPath)
logger.GetReqInfo(ctx).AppendTags("cachePath", cfs.fsPath)
logger.LogIf(ctx, err)
total += info.Total
free += info.Free
}
@@ -791,7 +792,8 @@ func (c cacheObjects) DeleteBucket(ctx context.Context, bucket string) (err erro
// or the global env overrides.
func newCache(config CacheConfig) (*diskCache, error) {
var cfsObjects []*cacheFSObjects
formats, err := loadAndValidateCacheFormat(config.Drives)
ctx := logger.SetReqInfo(context.Background(), &logger.ReqInfo{})
formats, err := loadAndValidateCacheFormat(ctx, config.Drives)
if err != nil {
return nil, err
}
+17 -15
View File
@@ -35,14 +35,15 @@ import (
var (
// AWS errors for invalid SSE-C requests.
errInsecureSSERequest = errors.New("Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection")
errEncryptedObject = errors.New("The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object")
errInvalidSSEAlgorithm = errors.New("Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm")
errMissingSSEKey = errors.New("Requests specifying Server Side Encryption with Customer provided keys must provide an appropriate secret key")
errInvalidSSEKey = errors.New("The secret key was invalid for the specified algorithm")
errMissingSSEKeyMD5 = errors.New("Requests specifying Server Side Encryption with Customer provided keys must provide the client calculated MD5 of the secret key")
errSSEKeyMD5Mismatch = errors.New("The calculated MD5 hash of the key did not match the hash that was provided")
errSSEKeyMismatch = errors.New("The client provided key does not match the key provided when the object was encrypted") // this msg is not shown to the client
errInsecureSSERequest = errors.New("SSE-C requests require TLS connections")
errEncryptedObject = errors.New("The object was stored using a form of SSE")
errInvalidSSEAlgorithm = errors.New("The SSE-C algorithm is not valid")
errMissingSSEKey = errors.New("The SSE-C request is missing the customer key")
errInvalidSSEKey = errors.New("The SSE-C key is invalid")
errMissingSSEKeyMD5 = errors.New("The SSE-C request is missing the customer key MD5")
errSSEKeyMD5Mismatch = errors.New("The key MD5 does not match the SSE-C key")
errSSEKeyMismatch = errors.New("The SSE-C key is not correct") // access denied
errInvalidSSEParameters = errors.New("The SSE-C key for key-rotation is not correct") // special access denied
// Additional Minio errors for SSE-C requests.
errObjectTampered = errors.New("The requested object was modified and may be compromised")
@@ -247,9 +248,6 @@ func ParseSSECustomerHeader(header http.Header) (key []byte, err error) {
// This function rotates old to new key.
func rotateKey(oldKey []byte, newKey []byte, metadata map[string]string) error {
if subtle.ConstantTimeCompare(oldKey, newKey) == 1 {
return nil
}
delete(metadata, SSECustomerKey) // make sure we do not save the key by accident
if metadata[ServerSideEncryptionSealAlgorithm] != SSESealAlgorithmDareSha256 { // currently DARE-SHA256 is the only option
@@ -273,10 +271,14 @@ func rotateKey(oldKey []byte, newKey []byte, metadata map[string]string) error {
n, err := sio.Decrypt(objectEncryptionKey, bytes.NewReader(sealedKey), sio.Config{
Key: keyEncryptionKey,
})
if n != 32 || err != nil {
// Either the provided key does not match or the object was tampered.
// To provide strict AWS S3 compatibility we return: access denied.
return errSSEKeyMismatch
if n != 32 || err != nil { // Either the provided key does not match or the object was tampered.
if subtle.ConstantTimeCompare(oldKey, newKey) == 1 {
return errInvalidSSEParameters // AWS returns special error for equal but invalid keys.
}
return errSSEKeyMismatch // To provide strict AWS S3 compatibility we return: access denied.
}
if subtle.ConstantTimeCompare(oldKey, newKey) == 1 {
return nil // we don't need to rotate keys if newKey == oldKey
}
nonce := make([]byte, 32) // generate random values for key derivation
+2 -1
View File
@@ -27,6 +27,7 @@ import (
"strings"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/mountinfo"
)
@@ -399,7 +400,7 @@ func CreateEndpoints(serverAddr string, args ...[]string) (string, EndpointList,
}
ipList, err := getHostIP4(host)
fatalIf(err, "unexpected error when resolving host '%s'", host)
logger.FatalIf(err, "unexpected error when resolving host '%s'", host)
// Filter ipList by IPs those start with '127.'.
loopBackIPs := ipList.FuncMatch(func(ip string, matchString string) bool {
+13 -9
View File
@@ -17,18 +17,20 @@
package cmd
import (
"context"
"hash"
"io"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
// CreateFile creates a new bitrot encoded file spread over all available disks. CreateFile will create
// the file at the given volume and path. It will read from src until an io.EOF occurs. The given algorithm will
// be used to protect the erasure encoded file.
func (s *ErasureStorage) CreateFile(src io.Reader, volume, path string, buffer []byte, algorithm BitrotAlgorithm, writeQuorum int) (f ErasureFileInfo, err error) {
func (s *ErasureStorage) CreateFile(ctx context.Context, src io.Reader, volume, path string, buffer []byte, algorithm BitrotAlgorithm, writeQuorum int) (f ErasureFileInfo, err error) {
if !algorithm.Available() {
return f, errors.Trace(errBitrotHashAlgoInvalid)
logger.LogIf(ctx, errBitrotHashAlgoInvalid)
return f, errBitrotHashAlgoInvalid
}
f.Checksums = make([][]byte, len(s.disks))
hashers := make([]hash.Hash, len(s.disks))
@@ -50,21 +52,22 @@ func (s *ErasureStorage) CreateFile(src io.Reader, volume, path string, buffer [
}
blocks = make([][]byte, len(s.disks)) // write empty block
} else if err == nil || (n > 0 && err == io.ErrUnexpectedEOF) {
blocks, err = s.ErasureEncode(buffer[:n])
blocks, err = s.ErasureEncode(ctx, buffer[:n])
if err != nil {
return f, err
}
} else {
return f, errors.Trace(err)
logger.LogIf(ctx, err)
return f, err
}
for i := range errChans { // span workers
go erasureAppendFile(s.disks[i], volume, path, hashers[i], blocks[i], errChans[i])
go erasureAppendFile(ctx, s.disks[i], volume, path, hashers[i], blocks[i], errChans[i])
}
for i := range errChans { // wait until all workers are finished
errs[i] = <-errChans[i]
}
if err = reduceWriteQuorumErrs(errs, objectOpIgnoredErrs, writeQuorum); err != nil {
if err = reduceWriteQuorumErrs(ctx, errs, objectOpIgnoredErrs, writeQuorum); err != nil {
return f, err
}
s.disks = evalDisks(s.disks, errs)
@@ -83,9 +86,10 @@ func (s *ErasureStorage) CreateFile(src io.Reader, volume, path string, buffer [
// erasureAppendFile appends the content of buf to the file on the given disk and updates computes
// the hash of the written data. It sends the write error (or nil) over the error channel.
func erasureAppendFile(disk StorageAPI, volume, path string, hash hash.Hash, buf []byte, errChan chan<- error) {
func erasureAppendFile(ctx context.Context, disk StorageAPI, volume, path string, hash hash.Hash, buf []byte, errChan chan<- error) {
if disk == OfflineDisk {
errChan <- errors.Trace(errDiskNotFound)
logger.LogIf(ctx, errDiskNotFound)
errChan <- errDiskNotFound
return
}
err := disk.AppendFile(volume, path, buf)
+6 -5
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"context"
"crypto/rand"
"io"
"testing"
@@ -70,7 +71,7 @@ func TestErasureCreateFile(t *testing.T) {
if err != nil {
t.Fatalf("Test %d: failed to create test setup: %v", i, err)
}
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks, test.blocksize)
storage, err := NewErasureStorage(context.Background(), setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks, test.blocksize)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create ErasureStorage: %v", i, err)
@@ -82,7 +83,7 @@ func TestErasureCreateFile(t *testing.T) {
setup.Remove()
t.Fatalf("Test %d: failed to generate random test data: %v", i, err)
}
file, err := storage.CreateFile(bytes.NewReader(data[test.offset:]), "testbucket", "object", buffer, test.algorithm, test.dataBlocks+1)
file, err := storage.CreateFile(context.Background(), bytes.NewReader(data[test.offset:]), "testbucket", "object", buffer, test.algorithm, test.dataBlocks+1)
if err != nil && !test.shouldFail {
t.Errorf("Test %d: should pass but failed with: %v", i, err)
}
@@ -100,7 +101,7 @@ func TestErasureCreateFile(t *testing.T) {
if test.offDisks > 0 {
storage.disks[0] = OfflineDisk
}
file, err = storage.CreateFile(bytes.NewReader(data[test.offset:]), "testbucket", "object2", buffer, test.algorithm, test.dataBlocks+1)
file, err = storage.CreateFile(context.Background(), bytes.NewReader(data[test.offset:]), "testbucket", "object2", buffer, test.algorithm, test.dataBlocks+1)
if err != nil && !test.shouldFailQuorum {
t.Errorf("Test %d: should pass but failed with: %v", i, err)
}
@@ -125,7 +126,7 @@ func benchmarkErasureWrite(data, parity, dataDown, parityDown int, size int64, b
b.Fatalf("failed to create test setup: %v", err)
}
defer setup.Remove()
storage, err := NewErasureStorage(setup.disks, data, parity, blockSizeV1)
storage, err := NewErasureStorage(context.Background(), setup.disks, data, parity, blockSizeV1)
if err != nil {
b.Fatalf("failed to create ErasureStorage: %v", err)
}
@@ -143,7 +144,7 @@ func benchmarkErasureWrite(data, parity, dataDown, parityDown int, size int64, b
b.SetBytes(size)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
_, err := storage.CreateFile(bytes.NewReader(content), "testbucket", "object", buffer, DefaultBitrotAlgorithm, data+1)
_, err := storage.CreateFile(context.Background(), bytes.NewReader(content), "testbucket", "object", buffer, DefaultBitrotAlgorithm, data+1)
if err != nil {
panic(err)
}
+10 -6
View File
@@ -17,11 +17,12 @@
package cmd
import (
"context"
"fmt"
"hash"
"strings"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
// HealFile tries to reconstruct an erasure-coded file spread over all
@@ -45,12 +46,13 @@ import (
//
// It returns bitrot checksums for the non-nil staleDisks on which
// healing succeeded.
func (s ErasureStorage) HealFile(staleDisks []StorageAPI, volume, path string, blocksize int64,
func (s ErasureStorage) HealFile(ctx context.Context, staleDisks []StorageAPI, volume, path string, blocksize int64,
dstVol, dstPath string, size int64, alg BitrotAlgorithm, checksums [][]byte) (
f ErasureFileInfo, err error) {
if !alg.Available() {
return f, errors.Trace(errBitrotHashAlgoInvalid)
logger.LogIf(ctx, errBitrotHashAlgoInvalid)
return f, errBitrotHashAlgoInvalid
}
// Initialization
@@ -84,7 +86,7 @@ func (s ErasureStorage) HealFile(staleDisks []StorageAPI, volume, path string, b
}
readLen += lastChunkSize
var buffers [][]byte
buffers, _, err = s.readConcurrent(volume, path, 0, readLen, verifiers)
buffers, _, err = s.readConcurrent(ctx, volume, path, 0, readLen, verifiers)
if err != nil {
return f, err
}
@@ -131,7 +133,7 @@ func (s ErasureStorage) HealFile(staleDisks []StorageAPI, volume, path string, b
}
buffOffset += csize
if err = s.ErasureDecodeDataAndParityBlocks(blocks); err != nil {
if err = s.ErasureDecodeDataAndParityBlocks(ctx, blocks); err != nil {
return f, err
}
@@ -155,7 +157,9 @@ func (s ErasureStorage) HealFile(staleDisks []StorageAPI, volume, path string, b
// If all disks had write errors we quit.
if !writeSucceeded {
// build error from all write errors
return f, errors.Trace(joinWriteErrors(writeErrors))
err := joinWriteErrors(writeErrors)
logger.LogIf(ctx, err)
return f, err
}
}
+4 -3
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"context"
"crypto/rand"
"io"
"reflect"
@@ -74,7 +75,7 @@ func TestErasureHealFile(t *testing.T) {
if err != nil {
t.Fatalf("Test %d: failed to setup XL environment: %v", i, err)
}
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.disks-test.dataBlocks, test.blocksize)
storage, err := NewErasureStorage(context.Background(), setup.disks, test.dataBlocks, test.disks-test.dataBlocks, test.blocksize)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create ErasureStorage: %v", i, err)
@@ -89,7 +90,7 @@ func TestErasureHealFile(t *testing.T) {
algorithm = DefaultBitrotAlgorithm
}
buffer := make([]byte, test.blocksize, 2*test.blocksize)
file, err := storage.CreateFile(bytes.NewReader(data), "testbucket", "testobject", buffer, algorithm, test.dataBlocks+1)
file, err := storage.CreateFile(context.Background(), bytes.NewReader(data), "testbucket", "testobject", buffer, algorithm, test.dataBlocks+1)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create random test data: %v", i, err)
@@ -113,7 +114,7 @@ func TestErasureHealFile(t *testing.T) {
}
// test case setup is complete - now call Healfile()
info, err := storage.HealFile(staleDisks, "testbucket", "testobject", test.blocksize, "testbucket", "healedobject", test.size, test.algorithm, file.Checksums)
info, err := storage.HealFile(context.Background(), staleDisks, "testbucket", "testobject", test.blocksize, "testbucket", "healedobject", test.size, test.algorithm, file.Checksums)
if err != nil && !test.shouldFail {
t.Errorf("Test %d: should pass but it failed with: %v", i, err)
}
+18 -11
View File
@@ -17,9 +17,10 @@
package cmd
import (
"context"
"io"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
type errIdx struct {
@@ -27,7 +28,7 @@ type errIdx struct {
err error
}
func (s ErasureStorage) readConcurrent(volume, path string, offset, length int64,
func (s ErasureStorage) readConcurrent(ctx context.Context, volume, path string, offset, length int64,
verifiers []*BitrotVerifier) (buffers [][]byte, needsReconstruction bool,
err error) {
@@ -39,7 +40,8 @@ func (s ErasureStorage) readConcurrent(volume, path string, offset, length int64
stageBuffers[i] = make([]byte, length)
disk := s.disks[i]
if disk == OfflineDisk {
errChan <- errIdx{i, errors.Trace(errDiskNotFound)}
logger.LogIf(ctx, errDiskNotFound)
errChan <- errIdx{i, errDiskNotFound}
return
}
_, rerr := disk.ReadFile(volume, path, offset, stageBuffers[i], verifiers[i])
@@ -75,7 +77,8 @@ func (s ErasureStorage) readConcurrent(volume, path string, offset, length int64
}
if successCount != s.dataBlocks {
// Not enough disks returns data.
err = errors.Trace(errXLReadQuorum)
err = errXLReadQuorum
logger.LogIf(ctx, err)
}
return
}
@@ -86,18 +89,21 @@ func (s ErasureStorage) readConcurrent(volume, path string, offset, length int64
// integrity of the given file. ReadFile will read data from the given
// offset up to the given length. If parts of the file are corrupted
// ReadFile tries to reconstruct the data.
func (s ErasureStorage) ReadFile(writer io.Writer, volume, path string, offset,
func (s ErasureStorage) ReadFile(ctx context.Context, writer io.Writer, volume, path string, offset,
length, totalLength int64, checksums [][]byte, algorithm BitrotAlgorithm,
blocksize int64) (f ErasureFileInfo, err error) {
if offset < 0 || length < 0 {
return f, errors.Trace(errUnexpected)
logger.LogIf(ctx, errUnexpected)
return f, errUnexpected
}
if offset+length > totalLength {
return f, errors.Trace(errUnexpected)
logger.LogIf(ctx, errUnexpected)
return f, errUnexpected
}
if !algorithm.Available() {
return f, errors.Trace(errBitrotHashAlgoInvalid)
logger.LogIf(ctx, errBitrotHashAlgoInvalid)
return f, errBitrotHashAlgoInvalid
}
f.Checksums = make([][]byte, len(s.disks))
@@ -145,7 +151,7 @@ func (s ErasureStorage) ReadFile(writer io.Writer, volume, path string, offset,
var buffers [][]byte
var needsReconstruction bool
buffers, needsReconstruction, err = s.readConcurrent(volume, path,
buffers, needsReconstruction, err = s.readConcurrent(ctx, volume, path,
partDataStartIndex, partDataLength, verifiers)
if err != nil {
// Could not read enough disks.
@@ -194,7 +200,8 @@ func (s ErasureStorage) ReadFile(writer io.Writer, volume, path string, offset,
if needsReconstruction {
if err = s.ErasureDecodeDataBlocks(blocks); err != nil {
return f, errors.Trace(err)
logger.LogIf(ctx, err)
return f, err
}
}
@@ -210,7 +217,7 @@ func (s ErasureStorage) ReadFile(writer io.Writer, volume, path string, offset,
writeLength = lastBlockLength - writeStart
}
}
n, err := writeDataBlocks(writer, blocks, s.dataBlocks, writeStart, writeLength)
n, err := writeDataBlocks(ctx, writer, blocks, s.dataBlocks, writeStart, writeLength)
if err != nil {
return f, err
}
+11 -10
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"context"
crand "crypto/rand"
"io"
"math/rand"
@@ -86,7 +87,7 @@ func TestErasureReadFile(t *testing.T) {
if err != nil {
t.Fatalf("Test %d: failed to create test setup: %v", i, err)
}
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks, test.blocksize)
storage, err := NewErasureStorage(context.Background(), setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks, test.blocksize)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create ErasureStorage: %v", i, err)
@@ -102,13 +103,13 @@ func TestErasureReadFile(t *testing.T) {
writeAlgorithm = DefaultBitrotAlgorithm
}
buffer := make([]byte, test.blocksize, 2*test.blocksize)
file, err := storage.CreateFile(bytes.NewReader(data[:]), "testbucket", "object", buffer, writeAlgorithm, test.dataBlocks+1)
file, err := storage.CreateFile(context.Background(), bytes.NewReader(data[:]), "testbucket", "object", buffer, writeAlgorithm, test.dataBlocks+1)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create erasure test file: %v", i, err)
}
writer := bytes.NewBuffer(nil)
readInfo, err := storage.ReadFile(writer, "testbucket", "object", test.offset, test.length, test.data, file.Checksums, test.algorithm, test.blocksize)
readInfo, err := storage.ReadFile(context.Background(), writer, "testbucket", "object", test.offset, test.length, test.data, file.Checksums, test.algorithm, test.blocksize)
if err != nil && !test.shouldFail {
t.Errorf("Test %d: should pass but failed with: %v", i, err)
}
@@ -134,7 +135,7 @@ func TestErasureReadFile(t *testing.T) {
if test.offDisks > 0 {
storage.disks[0] = OfflineDisk
}
readInfo, err = storage.ReadFile(writer, "testbucket", "object", test.offset, test.length, test.data, file.Checksums, test.algorithm, test.blocksize)
readInfo, err = storage.ReadFile(context.Background(), writer, "testbucket", "object", test.offset, test.length, test.data, file.Checksums, test.algorithm, test.blocksize)
if err != nil && !test.shouldFailQuorum {
t.Errorf("Test %d: should pass but failed with: %v", i, err)
}
@@ -174,7 +175,7 @@ func TestErasureReadFileRandomOffsetLength(t *testing.T) {
}
defer setup.Remove()
storage, err := NewErasureStorage(setup.disks, dataBlocks, parityBlocks, blockSize)
storage, err := NewErasureStorage(context.Background(), setup.disks, dataBlocks, parityBlocks, blockSize)
if err != nil {
t.Fatalf("failed to create ErasureStorage: %v", err)
}
@@ -191,7 +192,7 @@ func TestErasureReadFileRandomOffsetLength(t *testing.T) {
// Create a test file to read from.
buffer := make([]byte, blockSize, 2*blockSize)
file, err := storage.CreateFile(bytes.NewReader(data), "testbucket", "testobject", buffer, DefaultBitrotAlgorithm, dataBlocks+1)
file, err := storage.CreateFile(context.Background(), bytes.NewReader(data), "testbucket", "testobject", buffer, DefaultBitrotAlgorithm, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
@@ -211,7 +212,7 @@ func TestErasureReadFileRandomOffsetLength(t *testing.T) {
expected := data[offset : offset+readLen]
_, err = storage.ReadFile(buf, "testbucket", "testobject", offset, readLen, length, file.Checksums, DefaultBitrotAlgorithm, blockSize)
_, err = storage.ReadFile(context.Background(), buf, "testbucket", "testobject", offset, readLen, length, file.Checksums, DefaultBitrotAlgorithm, blockSize)
if err != nil {
t.Fatal(err, offset, readLen)
}
@@ -231,14 +232,14 @@ func benchmarkErasureRead(data, parity, dataDown, parityDown int, size int64, b
b.Fatalf("failed to create test setup: %v", err)
}
defer setup.Remove()
storage, err := NewErasureStorage(setup.disks, data, parity, blockSizeV1)
storage, err := NewErasureStorage(context.Background(), setup.disks, data, parity, blockSizeV1)
if err != nil {
b.Fatalf("failed to create ErasureStorage: %v", err)
}
content := make([]byte, size)
buffer := make([]byte, blockSizeV1, 2*blockSizeV1)
file, err := storage.CreateFile(bytes.NewReader(content), "testbucket", "object", buffer, DefaultBitrotAlgorithm, data+1)
file, err := storage.CreateFile(context.Background(), bytes.NewReader(content), "testbucket", "object", buffer, DefaultBitrotAlgorithm, data+1)
if err != nil {
b.Fatalf("failed to create erasure test file: %v", err)
}
@@ -255,7 +256,7 @@ func benchmarkErasureRead(data, parity, dataDown, parityDown int, size int64, b
b.SetBytes(size)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if file, err = storage.ReadFile(bytes.NewBuffer(content[:0]), "testbucket", "object", 0, size, size, checksums, DefaultBitrotAlgorithm, blockSizeV1); err != nil {
if file, err = storage.ReadFile(context.Background(), bytes.NewBuffer(content[:0]), "testbucket", "object", 0, size, size, checksums, DefaultBitrotAlgorithm, blockSizeV1); err != nil {
panic(err)
}
}
+13 -7
View File
@@ -18,10 +18,11 @@ package cmd
import (
"bytes"
"context"
"io"
"github.com/klauspost/reedsolomon"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
// getDataBlockLen - get length of data blocks from encoded blocks.
@@ -36,20 +37,23 @@ func getDataBlockLen(enBlocks [][]byte, dataBlocks int) int {
// Writes all the data blocks from encoded blocks until requested
// outSize length. Provides a way to skip bytes until the offset.
func writeDataBlocks(dst io.Writer, enBlocks [][]byte, dataBlocks int, offset int64, length int64) (int64, error) {
func writeDataBlocks(ctx context.Context, dst io.Writer, enBlocks [][]byte, dataBlocks int, offset int64, length int64) (int64, error) {
// Offset and out size cannot be negative.
if offset < 0 || length < 0 {
return 0, errors.Trace(errUnexpected)
logger.LogIf(ctx, errUnexpected)
return 0, errUnexpected
}
// Do we have enough blocks?
if len(enBlocks) < dataBlocks {
return 0, errors.Trace(reedsolomon.ErrTooFewShards)
logger.LogIf(ctx, reedsolomon.ErrTooFewShards)
return 0, reedsolomon.ErrTooFewShards
}
// Do we have enough data?
if int64(getDataBlockLen(enBlocks, dataBlocks)) < length {
return 0, errors.Trace(reedsolomon.ErrShortData)
logger.LogIf(ctx, reedsolomon.ErrShortData)
return 0, reedsolomon.ErrShortData
}
// Counter to decrement total left to write.
@@ -77,7 +81,8 @@ func writeDataBlocks(dst io.Writer, enBlocks [][]byte, dataBlocks int, offset in
if write < int64(len(block)) {
n, err := io.Copy(dst, bytes.NewReader(block[:write]))
if err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
totalWritten += n
break
@@ -85,7 +90,8 @@ func writeDataBlocks(dst io.Writer, enBlocks [][]byte, dataBlocks int, offset in
// Copy the block.
n, err := io.Copy(dst, bytes.NewReader(block))
if err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
// Decrement output size.
+14 -9
View File
@@ -17,11 +17,12 @@
package cmd
import (
"context"
"crypto/subtle"
"hash"
"github.com/klauspost/reedsolomon"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
// OfflineDisk represents an unavailable disk.
@@ -44,11 +45,12 @@ type ErasureStorage struct {
// NewErasureStorage creates a new ErasureStorage. The storage erasure codes and protects all data written to
// the disks.
func NewErasureStorage(disks []StorageAPI, dataBlocks, parityBlocks int, blockSize int64) (s ErasureStorage, err error) {
func NewErasureStorage(ctx context.Context, disks []StorageAPI, dataBlocks, parityBlocks int, blockSize int64) (s ErasureStorage, err error) {
shardsize := (int(blockSize) + dataBlocks - 1) / dataBlocks
erasure, err := reedsolomon.New(dataBlocks, parityBlocks, reedsolomon.WithAutoGoroutines(shardsize))
if err != nil {
return s, errors.Tracef("failed to create erasure coding: %v", err)
logger.LogIf(ctx, err)
return s, err
}
s = ErasureStorage{
disks: make([]StorageAPI, len(disks)),
@@ -62,13 +64,15 @@ func NewErasureStorage(disks []StorageAPI, dataBlocks, parityBlocks int, blockSi
// ErasureEncode encodes the given data and returns the erasure-coded data.
// It returns an error if the erasure coding failed.
func (s *ErasureStorage) ErasureEncode(data []byte) ([][]byte, error) {
func (s *ErasureStorage) ErasureEncode(ctx context.Context, data []byte) ([][]byte, error) {
encoded, err := s.erasure.Split(data)
if err != nil {
return nil, errors.Tracef("failed to split data: %v", err)
logger.LogIf(ctx, err)
return nil, err
}
if err = s.erasure.Encode(encoded); err != nil {
return nil, errors.Tracef("failed to encode data: %v", err)
logger.LogIf(ctx, err)
return nil, err
}
return encoded, nil
}
@@ -78,16 +82,17 @@ func (s *ErasureStorage) ErasureEncode(data []byte) ([][]byte, error) {
// It returns an error if the decoding failed.
func (s *ErasureStorage) ErasureDecodeDataBlocks(data [][]byte) error {
if err := s.erasure.ReconstructData(data); err != nil {
return errors.Tracef("failed to reconstruct data: %v", err)
return err
}
return nil
}
// ErasureDecodeDataAndParityBlocks decodes the given erasure-coded data and verifies it.
// It returns an error if the decoding failed.
func (s *ErasureStorage) ErasureDecodeDataAndParityBlocks(data [][]byte) error {
func (s *ErasureStorage) ErasureDecodeDataAndParityBlocks(ctx context.Context, data [][]byte) error {
if err := s.erasure.Reconstruct(data); err != nil {
return errors.Tracef("failed to reconstruct data: %v", err)
logger.LogIf(ctx, err)
return err
}
return nil
}
+5 -4
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"context"
"crypto/rand"
"io"
"os"
@@ -52,11 +53,11 @@ func TestErasureDecode(t *testing.T) {
copy(buffer, data)
disks := make([]StorageAPI, test.dataBlocks+test.parityBlocks)
storage, err := NewErasureStorage(disks, test.dataBlocks, test.parityBlocks, blockSizeV1)
storage, err := NewErasureStorage(context.Background(), disks, test.dataBlocks, test.parityBlocks, blockSizeV1)
if err != nil {
t.Fatalf("Test %d: failed to create erasure storage: %v", i, err)
}
encoded, err := storage.ErasureEncode(buffer)
encoded, err := storage.ErasureEncode(context.Background(), buffer)
if err != nil {
t.Fatalf("Test %d: failed to encode data: %v", i, err)
}
@@ -69,7 +70,7 @@ func TestErasureDecode(t *testing.T) {
}
if test.reconstructParity {
err = storage.ErasureDecodeDataAndParityBlocks(encoded)
err = storage.ErasureDecodeDataAndParityBlocks(context.Background(), encoded)
} else {
err = storage.ErasureDecodeDataBlocks(encoded)
}
@@ -98,7 +99,7 @@ func TestErasureDecode(t *testing.T) {
}
decodedData := new(bytes.Buffer)
if _, err = writeDataBlocks(decodedData, decoded, test.dataBlocks, 0, int64(len(data))); err != nil {
if _, err = writeDataBlocks(context.Background(), decodedData, decoded, test.dataBlocks, 0, int64(len(data))); err != nil {
t.Errorf("Test %d: failed to write data blocks: %v", i, err)
}
if !bytes.Equal(decodedData.Bytes(), data) {
+25 -20
View File
@@ -17,13 +17,14 @@
package cmd
import (
"context"
"errors"
"fmt"
"io"
"os"
"reflect"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
const (
@@ -86,28 +87,19 @@ func newFormatCacheV1(drives []string) []*formatCacheV1 {
return formats
}
// Returns format.Cache.Version
func formatCacheGetVersion(r io.ReadSeeker) (string, error) {
format := &formatCacheVersionDetect{}
if err := jsonLoad(r, format); err != nil {
return "", err
}
return format.Cache.Version, nil
}
// Creates a new cache format.json if unformatted.
func createFormatCache(fsFormatPath string, format *formatCacheV1) error {
// open file using READ & WRITE permission
var file, err = os.OpenFile(fsFormatPath, os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
return errors2.Trace(err)
return err
}
// Close the locked file upon return.
defer file.Close()
fi, err := file.Stat()
if err != nil {
return errors2.Trace(err)
return err
}
if fi.Size() != 0 {
// format.json already got created because of another minio process's createFormatCache()
@@ -118,7 +110,7 @@ func createFormatCache(fsFormatPath string, format *formatCacheV1) error {
// This function creates a cache format file on disk and returns a slice
// of format cache config
func initFormatCache(drives []string) (formats []*formatCacheV1, err error) {
func initFormatCache(ctx context.Context, drives []string) (formats []*formatCacheV1, err error) {
nformats := newFormatCacheV1(drives)
for _, drive := range drives {
_, err = os.Stat(drive)
@@ -126,28 +118,36 @@ func initFormatCache(drives []string) (formats []*formatCacheV1, err error) {
continue
}
if !os.IsNotExist(err) {
logger.GetReqInfo(ctx).AppendTags("drive", drive)
logger.LogIf(ctx, err)
return nil, err
}
if err = os.Mkdir(drive, 0777); err != nil {
logger.GetReqInfo(ctx).AppendTags("drive", drive)
logger.LogIf(ctx, err)
return nil, err
}
}
for i, drive := range drives {
if err = os.Mkdir(pathJoin(drive, minioMetaBucket), 0777); err != nil {
if !os.IsExist(err) {
logger.GetReqInfo(ctx).AppendTags("drive", drive)
logger.LogIf(ctx, err)
return nil, err
}
}
cacheFormatPath := pathJoin(drive, minioMetaBucket, formatConfigFile)
// Fresh disk - create format.json for this cfs
if err = createFormatCache(cacheFormatPath, nformats[i]); err != nil {
logger.GetReqInfo(ctx).AppendTags("drive", drive)
logger.LogIf(ctx, err)
return nil, err
}
}
return nformats, nil
}
func loadFormatCache(drives []string) ([]*formatCacheV1, error) {
func loadFormatCache(ctx context.Context, drives []string) ([]*formatCacheV1, error) {
formats := make([]*formatCacheV1, len(drives))
for i, drive := range drives {
cacheFormatPath := pathJoin(drive, minioMetaBucket, formatConfigFile)
@@ -156,6 +156,7 @@ func loadFormatCache(drives []string) ([]*formatCacheV1, error) {
if os.IsNotExist(err) {
continue
}
logger.LogIf(ctx, err)
return nil, err
}
defer f.Close()
@@ -268,7 +269,7 @@ func findCacheDiskIndex(disk string, disks []string) int {
}
// validate whether cache drives order has changed
func validateCacheFormats(formats []*formatCacheV1) error {
func validateCacheFormats(ctx context.Context, formats []*formatCacheV1) error {
count := 0
for _, format := range formats {
if format == nil {
@@ -279,12 +280,16 @@ func validateCacheFormats(formats []*formatCacheV1) error {
return errors.New("Cache format files missing on all drives")
}
if _, err := checkFormatCacheValues(formats); err != nil {
logger.LogIf(ctx, err)
return err
}
if err := checkCacheDisksSliceConsistency(formats); err != nil {
logger.LogIf(ctx, err)
return err
}
return checkCacheDiskConsistency(formats)
err := checkCacheDiskConsistency(formats)
logger.LogIf(ctx, err)
return err
}
// return true if all of the list of cache drives are
@@ -303,16 +308,16 @@ func cacheDrivesUnformatted(drives []string) bool {
// create format.json for each cache drive if fresh disk or load format from disk
// Then validate the format for all drives in the cache to ensure order
// of cache drives has not changed.
func loadAndValidateCacheFormat(drives []string) (formats []*formatCacheV1, err error) {
func loadAndValidateCacheFormat(ctx context.Context, drives []string) (formats []*formatCacheV1, err error) {
if cacheDrivesUnformatted(drives) {
formats, err = initFormatCache(drives)
formats, err = initFormatCache(ctx, drives)
} else {
formats, err = loadFormatCache(drives)
formats, err = loadFormatCache(ctx, drives)
}
if err != nil {
return nil, err
}
if err = validateCacheFormats(formats); err != nil {
if err = validateCacheFormats(ctx, formats); err != nil {
return nil, err
}
return formats, nil
+16 -4
View File
@@ -17,18 +17,30 @@
package cmd
import (
"context"
"io"
"os"
"testing"
)
// Returns format.Cache.Version
func formatCacheGetVersion(r io.ReadSeeker) (string, error) {
format := &formatCacheVersionDetect{}
if err := jsonLoad(r, format); err != nil {
return "", err
}
return format.Cache.Version, nil
}
// TestDiskCacheFormat - tests initFormatCache, formatMetaGetFormatBackendCache, formatCacheGetVersion.
func TestDiskCacheFormat(t *testing.T) {
ctx := context.Background()
fsDirs, err := getRandomDisks(1)
if err != nil {
t.Fatal(err)
}
_, err = initFormatCache(fsDirs)
_, err = initFormatCache(ctx, fsDirs)
if err != nil {
t.Fatal(err)
}
@@ -56,7 +68,7 @@ func TestDiskCacheFormat(t *testing.T) {
t.Fatal(err)
}
if _, err = loadAndValidateCacheFormat(fsDirs); err == nil {
if _, err = loadAndValidateCacheFormat(context.Background(), fsDirs); err == nil {
t.Fatal("expected to fail")
}
@@ -69,7 +81,7 @@ func TestDiskCacheFormat(t *testing.T) {
t.Fatal(err)
}
if _, err = loadAndValidateCacheFormat(fsDirs); err == nil {
if _, err = loadAndValidateCacheFormat(context.Background(), fsDirs); err == nil {
t.Fatal("expected to fail")
}
}
@@ -307,7 +319,7 @@ func TestFormatCache(t *testing.T) {
}
for i, testCase := range testCases {
err := validateCacheFormats(testCase.formatConfigs)
err := validateCacheFormats(context.Background(), testCase.formatConfigs)
if err != nil && testCase.shouldPass {
t.Errorf("Test %d: Expected to pass but failed with %s", i+1, err)
}
+23 -15
View File
@@ -17,13 +17,14 @@
package cmd
import (
"context"
"fmt"
"io"
"os"
"path"
"time"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
)
@@ -97,7 +98,7 @@ func formatFSGetVersion(r io.ReadSeeker) (string, error) {
// Migrate from V1 to V2. V2 implements new backend format for multipart
// uploads. Delete the previous multipart directory.
func formatFSMigrateV1ToV2(wlk *lock.LockedFile, fsPath string) error {
func formatFSMigrateV1ToV2(ctx context.Context, wlk *lock.LockedFile, fsPath string) error {
version, err := formatFSGetVersion(wlk)
if err != nil {
return err
@@ -107,11 +108,12 @@ func formatFSMigrateV1ToV2(wlk *lock.LockedFile, fsPath string) error {
return fmt.Errorf(`format.json version expected %s, found %s`, formatFSVersionV1, version)
}
if err = fsRemoveAll(path.Join(fsPath, minioMetaMultipartBucket)); err != nil {
if err = fsRemoveAll(ctx, path.Join(fsPath, minioMetaMultipartBucket)); err != nil {
return err
}
if err = os.MkdirAll(path.Join(fsPath, minioMetaMultipartBucket), 0755); err != nil {
logger.LogIf(ctx, err)
return err
}
@@ -122,7 +124,7 @@ func formatFSMigrateV1ToV2(wlk *lock.LockedFile, fsPath string) error {
// Migration should happen when formatFSV1.FS.Version changes. This version
// can change when there is a change to the struct formatFSV1.FS or if there
// is any change in the backend file system tree structure.
func formatFSMigrate(wlk *lock.LockedFile, fsPath string) error {
func formatFSMigrate(ctx context.Context, wlk *lock.LockedFile, fsPath string) error {
// Add any migration code here in case we bump format.FS.Version
version, err := formatFSGetVersion(wlk)
if err != nil {
@@ -131,7 +133,7 @@ func formatFSMigrate(wlk *lock.LockedFile, fsPath string) error {
switch version {
case formatFSVersionV1:
if err = formatFSMigrateV1ToV2(wlk, fsPath); err != nil {
if err = formatFSMigrateV1ToV2(ctx, wlk, fsPath); err != nil {
return err
}
fallthrough
@@ -151,19 +153,21 @@ func formatFSMigrate(wlk *lock.LockedFile, fsPath string) error {
}
// Creates a new format.json if unformatted.
func createFormatFS(fsFormatPath string) error {
func createFormatFS(ctx context.Context, fsFormatPath string) error {
// Attempt a write lock on formatConfigFile `format.json`
// file stored in minioMetaBucket(.minio.sys) directory.
lk, err := lock.TryLockedOpenFile(fsFormatPath, os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
return errors2.Trace(err)
logger.LogIf(ctx, err)
return err
}
// Close the locked file upon return.
defer lk.Close()
fi, err := lk.Stat()
if err != nil {
return errors2.Trace(err)
logger.LogIf(ctx, err)
return err
}
if fi.Size() != 0 {
// format.json already got created because of another minio process's createFormatFS()
@@ -177,7 +181,7 @@ func createFormatFS(fsFormatPath string) error {
// The file descriptor should be kept open throughout the life
// of the process so that another minio process does not try to
// migrate the backend when we are actively working on the backend.
func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
func initFormatFS(ctx context.Context, fsPath string) (rlk *lock.RLockedFile, err error) {
fsFormatPath := pathJoin(fsPath, minioMetaBucket, formatConfigFile)
// Any read on format.json should be done with read-lock.
// Any write on format.json should be done with write-lock.
@@ -191,7 +195,8 @@ func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
var fi os.FileInfo
fi, err = rlk.Stat()
if err != nil {
return nil, errors2.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
isEmpty = fi.Size() == 0
}
@@ -200,7 +205,7 @@ func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
rlk.Close()
}
// Fresh disk - create format.json
err = createFormatFS(fsFormatPath)
err = createFormatFS(ctx, fsFormatPath)
if err == lock.ErrAlreadyLocked {
// Lock already present, sleep and attempt again.
// Can happen in a rare situation when a parallel minio process
@@ -209,19 +214,22 @@ func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
continue
}
if err != nil {
return nil, errors2.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
// After successfully creating format.json try to hold a read-lock on
// the file.
continue
}
if err != nil {
return nil, errors2.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
formatBackend, err := formatMetaGetFormatBackendFS(rlk)
if err != nil {
return nil, errors2.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
if formatBackend != formatBackendFS {
return nil, fmt.Errorf(`%s file: expected format-type: %s, found: %s`, formatConfigFile, formatBackendFS, formatBackend)
@@ -244,7 +252,7 @@ func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
if err != nil {
return nil, err
}
err = formatFSMigrate(wlk, fsPath)
err = formatFSMigrate(ctx, wlk, fsPath)
wlk.Close()
if err != nil {
// Migration failed, bail out so that the user can observe what happened.
+4 -3
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"os"
"path/filepath"
"testing"
@@ -38,7 +39,7 @@ func TestFSFormatFS(t *testing.T) {
t.Fatal(err)
}
rlk, err := initFormatFS(disk)
rlk, err := initFormatFS(context.Background(), disk)
if err != nil {
t.Fatal(err)
}
@@ -81,7 +82,7 @@ func TestFSFormatFS(t *testing.T) {
if _, err = formatFSGetVersion(rlk); err == nil {
t.Fatal("expected to fail")
}
if _, err = initFormatFS(disk); err == nil {
if _, err = initFormatFS(context.Background(), disk); err == nil {
t.Fatal("expected to fail")
}
@@ -96,7 +97,7 @@ func TestFSFormatFS(t *testing.T) {
if _, err = formatMetaGetFormatBackendFS(f); err == nil {
t.Fatal("expected to fail")
}
if _, err = initFormatFS(disk); err == nil {
if _, err = initFormatFS(context.Background(), disk); err == nil {
t.Fatal("expected to fail")
}
}
+35 -74
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
@@ -26,7 +27,6 @@ import (
"encoding/hex"
humanize "github.com/dustin/go-humanize"
"github.com/minio/minio/pkg/errors"
sha256 "github.com/minio/sha256-simd"
)
@@ -302,7 +302,7 @@ func hasAnyErrors(errs []error) bool {
func countErrs(errs []error, err error) int {
var i = 0
for _, err1 := range errs {
if errors.Cause(err1) == err {
if err1 == err {
i++
}
}
@@ -314,28 +314,24 @@ func shouldInitXLDisks(errs []error) bool {
return countErrs(errs, errUnformattedDisk) == len(errs)
}
// Check if unformatted disks are equal to write quorum.
func quorumUnformattedDisks(errs []error) bool {
return countErrs(errs, errUnformattedDisk) >= (len(errs)/2)+1
}
// loadFormatXLAll - load all format config from all input disks in parallel.
func loadFormatXLAll(endpoints EndpointList) ([]*formatXLV3, []error) {
func loadFormatXLAll(storageDisks []StorageAPI) ([]*formatXLV3, []error) {
// Initialize sync waitgroup.
var wg = &sync.WaitGroup{}
bootstrapDisks := make([]StorageAPI, len(endpoints))
for i, endpoint := range endpoints {
disk, err := newStorageAPI(endpoint)
if err != nil {
continue
}
bootstrapDisks[i] = disk
}
// Initialize list of errors.
var sErrs = make([]error, len(bootstrapDisks))
var sErrs = make([]error, len(storageDisks))
// Initialize format configs.
var formats = make([]*formatXLV3, len(bootstrapDisks))
var formats = make([]*formatXLV3, len(storageDisks))
// Load format from each disk in parallel
for index, disk := range bootstrapDisks {
for index, disk := range storageDisks {
if disk == nil {
sErrs[index] = errDiskNotFound
continue
@@ -344,10 +340,9 @@ func loadFormatXLAll(endpoints EndpointList) ([]*formatXLV3, []error) {
// Launch go-routine per disk.
go func(index int, disk StorageAPI) {
defer wg.Done()
format, lErr := loadFormatXL(disk)
if lErr != nil {
// close the internal connection, to avoid fd leaks.
disk.Close()
sErrs[index] = lErr
return
}
@@ -362,26 +357,6 @@ func loadFormatXLAll(endpoints EndpointList) ([]*formatXLV3, []error) {
return formats, sErrs
}
func undoSaveFormatXLAll(disks []StorageAPI) {
// Initialize sync waitgroup.
var wg = &sync.WaitGroup{}
// Undo previous save format.json entry from all underlying storage disks.
for index, disk := range disks {
if disk == nil {
continue
}
wg.Add(1)
// Delete a bucket inside a go-routine.
go func(index int, disk StorageAPI) {
defer wg.Done()
_ = disk.DeleteFile(minioMetaBucket, formatConfigFile)
}(index, disk)
}
// Wait for all make vol to finish.
wg.Wait()
}
func saveFormatXL(disk StorageAPI, format interface{}) error {
// Marshal and write to disk.
formatBytes, err := json.Marshal(format)
@@ -390,7 +365,7 @@ func saveFormatXL(disk StorageAPI, format interface{}) error {
}
// Purge any existing temporary file, okay to ignore errors here.
disk.DeleteFile(minioMetaBucket, formatConfigFileTmp)
defer disk.DeleteFile(minioMetaBucket, formatConfigFileTmp)
// Append file `format.json.tmp`.
if err = disk.AppendFile(minioMetaBucket, formatConfigFileTmp, formatBytes); err != nil {
@@ -550,12 +525,7 @@ func formatXLV3Check(reference *formatXLV3, format *formatXLV3) error {
}
// saveFormatXLAll - populates `format.json` on disks in its order.
func saveFormatXLAll(endpoints EndpointList, formats []*formatXLV3) error {
storageDisks, err := initStorageDisks(endpoints)
if err != nil {
return err
}
func saveFormatXLAll(ctx context.Context, storageDisks []StorageAPI, formats []*formatXLV3) error {
var errs = make([]error, len(storageDisks))
var wg = &sync.WaitGroup{}
@@ -576,15 +546,18 @@ func saveFormatXLAll(endpoints EndpointList, formats []*formatXLV3) error {
// Wait for the routines to finish.
wg.Wait()
writeQuorum := len(endpoints)/2 + 1
err = reduceWriteQuorumErrs(errs, nil, writeQuorum)
if errors.Cause(err) == errXLWriteQuorum {
// Purge all successfully created `format.json`
// when we do not have enough quorum.
undoSaveFormatXLAll(storageDisks)
}
writeQuorum := len(storageDisks)/2 + 1
return reduceWriteQuorumErrs(ctx, errs, nil, writeQuorum)
}
return err
// relinquishes the underlying connection for all storage disks.
func closeStorageDisks(storageDisks []StorageAPI) {
for _, disk := range storageDisks {
if disk == nil {
continue
}
disk.Close()
}
}
// Initialize storage disks based on input arguments.
@@ -592,8 +565,6 @@ func initStorageDisks(endpoints EndpointList) ([]StorageAPI, error) {
// Bootstrap disks.
storageDisks := make([]StorageAPI, len(endpoints))
for index, endpoint := range endpoints {
// Intentionally ignore disk not found errors. XL is designed
// to handle these errors internally.
storage, err := newStorageAPI(endpoint)
if err != nil && err != errDiskNotFound {
return nil, err
@@ -625,12 +596,7 @@ func formatXLV3ThisEmpty(formats []*formatXLV3) bool {
}
// fixFormatXLV3 - fix format XL configuration on all disks.
func fixFormatXLV3(endpoints EndpointList, formats []*formatXLV3) error {
storageDisks, err := initStorageDisks(endpoints)
if err != nil {
return err
}
func fixFormatXLV3(storageDisks []StorageAPI, endpoints EndpointList, formats []*formatXLV3) error {
for i, format := range formats {
if format == nil || !endpoints[i].IsLocal {
continue
@@ -644,7 +610,7 @@ func fixFormatXLV3(endpoints EndpointList, formats []*formatXLV3) error {
}
if format.XL.This == "" {
formats[i].XL.This = format.XL.Sets[0][i]
if err = saveFormatXL(storageDisks[i], formats[i]); err != nil {
if err := saveFormatXL(storageDisks[i], formats[i]); err != nil {
return err
}
}
@@ -653,9 +619,9 @@ func fixFormatXLV3(endpoints EndpointList, formats []*formatXLV3) error {
}
// initFormatXL - save XL format configuration on all disks.
func initFormatXL(endpoints EndpointList, setCount, disksPerSet int) (format *formatXLV3, err error) {
func initFormatXL(ctx context.Context, storageDisks []StorageAPI, setCount, disksPerSet int) (format *formatXLV3, err error) {
format = newFormatXLV3(setCount, disksPerSet)
formats := make([]*formatXLV3, len(endpoints))
formats := make([]*formatXLV3, len(storageDisks))
for i := 0; i < setCount; i++ {
for j := 0; j < disksPerSet; j++ {
@@ -666,12 +632,12 @@ func initFormatXL(endpoints EndpointList, setCount, disksPerSet int) (format *fo
}
// Initialize meta volume, if volume already exists ignores it.
if err = initFormatXLMetaVolume(endpoints, formats); err != nil {
if err = initFormatXLMetaVolume(storageDisks, formats); err != nil {
return format, fmt.Errorf("Unable to initialize '.minio.sys' meta volume, %s", err)
}
// Save formats `format.json` across all disks.
if err = saveFormatXLAll(endpoints, formats); err != nil {
if err = saveFormatXLAll(ctx, storageDisks, formats); err != nil {
return nil, err
}
@@ -682,17 +648,17 @@ func initFormatXL(endpoints EndpointList, setCount, disksPerSet int) (format *fo
func makeFormatXLMetaVolumes(disk StorageAPI) error {
// Attempt to create `.minio.sys`.
if err := disk.MakeVol(minioMetaBucket); err != nil {
if !errors.IsErrIgnored(err, initMetaVolIgnoredErrs...) {
if !IsErrIgnored(err, initMetaVolIgnoredErrs...) {
return err
}
}
if err := disk.MakeVol(minioMetaTmpBucket); err != nil {
if !errors.IsErrIgnored(err, initMetaVolIgnoredErrs...) {
if !IsErrIgnored(err, initMetaVolIgnoredErrs...) {
return err
}
}
if err := disk.MakeVol(minioMetaMultipartBucket); err != nil {
if !errors.IsErrIgnored(err, initMetaVolIgnoredErrs...) {
if !IsErrIgnored(err, initMetaVolIgnoredErrs...) {
return err
}
}
@@ -702,12 +668,7 @@ func makeFormatXLMetaVolumes(disk StorageAPI) error {
var initMetaVolIgnoredErrs = append(baseIgnoredErrs, errVolumeExists)
// Initializes meta volume on all input storage disks.
func initFormatXLMetaVolume(endpoints EndpointList, formats []*formatXLV3) error {
storageDisks, err := initStorageDisks(endpoints)
if err != nil {
return err
}
func initFormatXLMetaVolume(storageDisks []StorageAPI, formats []*formatXLV3) error {
// This happens for the first time, but keep this here since this
// is the only place where it can be made expensive optimizing all
// other calls. Create minio meta volume, if it doesn't exist yet.
+10 -6
View File
@@ -22,8 +22,6 @@ import (
"os"
"reflect"
"testing"
"github.com/minio/minio/pkg/errors"
)
// Test get offline/online uuids.
@@ -87,6 +85,11 @@ func TestFixFormatV3(t *testing.T) {
}
endpoints := mustGetNewEndpointList(xlDirs...)
storageDisks, err := initStorageDisks(endpoints)
if err != nil {
t.Fatal(err)
}
format := newFormatXLV3(1, 8)
formats := make([]*formatXLV3, 8)
@@ -96,19 +99,20 @@ func TestFixFormatV3(t *testing.T) {
formats[j] = &newFormat
}
if err = initFormatXLMetaVolume(endpoints, formats); err != nil {
if err = initFormatXLMetaVolume(storageDisks, formats); err != nil {
t.Fatal(err)
}
formats[1] = nil
expThis := formats[2].XL.This
formats[2].XL.This = ""
if err := fixFormatXLV3(endpoints, formats); err != nil {
if err := fixFormatXLV3(storageDisks, endpoints, formats); err != nil {
t.Fatal(err)
}
newFormats, errs := loadFormatXLAll(endpoints)
newFormats, errs := loadFormatXLAll(storageDisks)
for _, err := range errs {
if err != nil && errors.Cause(err) != errUnformattedDisk {
if err != nil && err != errUnformattedDisk {
t.Fatal(err)
}
}
+122 -74
View File
@@ -17,29 +17,36 @@
package cmd
import (
"context"
"io"
"os"
pathutil "path"
"runtime"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
)
// Removes only the file at given path does not remove
// any parent directories, handles long paths for
// windows automatically.
func fsRemoveFile(filePath string) (err error) {
func fsRemoveFile(ctx context.Context, filePath string) (err error) {
if filePath == "" {
return errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return errInvalidArgument
}
if err = checkPathLength(filePath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
if err = os.Remove((filePath)); err != nil {
return osErrToFSFileErr(err)
fsErr := osErrToFSFileErr(err)
if fsErr != errFileNotFound {
logger.LogIf(ctx, err)
}
return fsErr
}
return nil
@@ -47,22 +54,27 @@ func fsRemoveFile(filePath string) (err error) {
// Removes all files and folders at a given path, handles
// long paths for windows automatically.
func fsRemoveAll(dirPath string) (err error) {
func fsRemoveAll(ctx context.Context, dirPath string) (err error) {
if dirPath == "" {
return errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return errInvalidArgument
}
if err = checkPathLength(dirPath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
if err = os.RemoveAll(dirPath); err != nil {
if os.IsPermission(err) {
return errors.Trace(errVolumeAccessDenied)
logger.LogIf(ctx, errVolumeAccessDenied)
return errVolumeAccessDenied
} else if isSysErrNotEmpty(err) {
return errors.Trace(errVolumeNotEmpty)
logger.LogIf(ctx, errVolumeNotEmpty)
return errVolumeNotEmpty
}
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
return nil
@@ -70,22 +82,27 @@ func fsRemoveAll(dirPath string) (err error) {
// Removes a directory only if its empty, handles long
// paths for windows automatically.
func fsRemoveDir(dirPath string) (err error) {
func fsRemoveDir(ctx context.Context, dirPath string) (err error) {
if dirPath == "" {
return errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return errInvalidArgument
}
if err = checkPathLength(dirPath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
if err = os.Remove((dirPath)); err != nil {
if os.IsNotExist(err) {
return errors.Trace(errVolumeNotFound)
logger.LogIf(ctx, errVolumeNotFound)
return errVolumeNotFound
} else if isSysErrNotEmpty(err) {
return errors.Trace(errVolumeNotEmpty)
logger.LogIf(ctx, errVolumeNotEmpty)
return errVolumeNotEmpty
}
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
return nil
@@ -95,29 +112,36 @@ func fsRemoveDir(dirPath string) (err error) {
// otherwise returns an error. If directory already
// exists returns an error. Windows long paths
// are handled automatically.
func fsMkdir(dirPath string) (err error) {
func fsMkdir(ctx context.Context, dirPath string) (err error) {
if dirPath == "" {
return errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return errInvalidArgument
}
if err = checkPathLength(dirPath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
if err = os.Mkdir((dirPath), 0777); err != nil {
if os.IsExist(err) {
return errors.Trace(errVolumeExists)
logger.LogIf(ctx, errVolumeExists)
return errVolumeExists
} else if os.IsPermission(err) {
return errors.Trace(errDiskAccessDenied)
logger.LogIf(ctx, errDiskAccessDenied)
return errDiskAccessDenied
} else if isSysErrNotDir(err) {
// File path cannot be verified since
// one of the parents is a file.
return errors.Trace(errDiskAccessDenied)
logger.LogIf(ctx, errDiskAccessDenied)
return errDiskAccessDenied
} else if isSysErrPathNotFound(err) {
// Add specific case for windows.
return errors.Trace(errDiskAccessDenied)
logger.LogIf(ctx, errDiskAccessDenied)
return errDiskAccessDenied
}
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
return nil
@@ -128,36 +152,39 @@ func fsMkdir(dirPath string) (err error) {
// not perform any higher layer interpretation of files v/s
// directories. For higher level interpretation look at
// fsStatFileDir, fsStatFile, fsStatDir.
func fsStat(statLoc string) (os.FileInfo, error) {
func fsStat(ctx context.Context, statLoc string) (os.FileInfo, error) {
if statLoc == "" {
return nil, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return nil, errInvalidArgument
}
if err := checkPathLength(statLoc); err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
fi, err := os.Stat((statLoc))
if err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
return fi, nil
}
// Lookup if volume exists, returns volume attributes upon success.
func fsStatVolume(volume string) (os.FileInfo, error) {
fi, err := fsStat(volume)
func fsStatVolume(ctx context.Context, volume string) (os.FileInfo, error) {
fi, err := fsStat(ctx, volume)
if err != nil {
err = errors.Cause(err)
if os.IsNotExist(err) {
return nil, errors.Trace(errVolumeNotFound)
return nil, errVolumeNotFound
} else if os.IsPermission(err) {
return nil, errors.Trace(errVolumeAccessDenied)
return nil, errVolumeAccessDenied
}
return nil, errors.Trace(err)
return nil, err
}
if !fi.IsDir() {
return nil, errors.Trace(errVolumeAccessDenied)
logger.LogIf(ctx, errVolumeAccessDenied)
return nil, errVolumeAccessDenied
}
return fi, nil
@@ -171,54 +198,56 @@ func osErrToFSFileErr(err error) error {
if err == nil {
return nil
}
err = errors.Cause(err)
if os.IsNotExist(err) {
return errors.Trace(errFileNotFound)
return errFileNotFound
}
if os.IsPermission(err) {
return errors.Trace(errFileAccessDenied)
return errFileAccessDenied
}
if isSysErrNotDir(err) {
return errors.Trace(errFileAccessDenied)
return errFileAccessDenied
}
if isSysErrPathNotFound(err) {
return errors.Trace(errFileNotFound)
return errFileNotFound
}
return err
}
// Lookup if directory exists, returns directory attributes upon success.
func fsStatDir(statDir string) (os.FileInfo, error) {
fi, err := fsStat(statDir)
func fsStatDir(ctx context.Context, statDir string) (os.FileInfo, error) {
fi, err := fsStat(ctx, statDir)
if err != nil {
return nil, osErrToFSFileErr(err)
}
if !fi.IsDir() {
return nil, errors.Trace(errFileAccessDenied)
return nil, errFileAccessDenied
}
return fi, nil
}
// Lookup if file exists, returns file attributes upon success.
func fsStatFile(statFile string) (os.FileInfo, error) {
fi, err := fsStat(statFile)
func fsStatFile(ctx context.Context, statFile string) (os.FileInfo, error) {
fi, err := fsStat(ctx, statFile)
if err != nil {
return nil, osErrToFSFileErr(err)
}
if fi.IsDir() {
return nil, errors.Trace(errFileAccessDenied)
logger.LogIf(ctx, errFileAccessDenied)
return nil, errFileAccessDenied
}
return fi, nil
}
// Opens the file at given path, optionally from an offset. Upon success returns
// a readable stream and the size of the readable stream.
func fsOpenFile(readPath string, offset int64) (io.ReadCloser, int64, error) {
func fsOpenFile(ctx context.Context, readPath string, offset int64) (io.ReadCloser, int64, error) {
if readPath == "" || offset < 0 {
return nil, 0, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return nil, 0, errInvalidArgument
}
if err := checkPathLength(readPath); err != nil {
return nil, 0, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, 0, err
}
fr, err := os.Open((readPath))
@@ -229,19 +258,22 @@ func fsOpenFile(readPath string, offset int64) (io.ReadCloser, int64, error) {
// Stat to get the size of the file at path.
st, err := os.Stat((readPath))
if err != nil {
return nil, 0, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, 0, err
}
// Verify if its not a regular file, since subsequent Seek is undefined.
if !st.Mode().IsRegular() {
return nil, 0, errors.Trace(errIsNotRegular)
logger.LogIf(ctx, errIsNotRegular)
return nil, 0, errIsNotRegular
}
// Seek to the requested offset.
if offset > 0 {
_, err = fr.Seek(offset, os.SEEK_SET)
if err != nil {
return nil, 0, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, 0, err
}
}
@@ -250,21 +282,25 @@ func fsOpenFile(readPath string, offset int64) (io.ReadCloser, int64, error) {
}
// Creates a file and copies data from incoming reader. Staging buffer is used by io.CopyBuffer.
func fsCreateFile(filePath string, reader io.Reader, buf []byte, fallocSize int64) (int64, error) {
func fsCreateFile(ctx context.Context, filePath string, reader io.Reader, buf []byte, fallocSize int64) (int64, error) {
if filePath == "" || reader == nil {
return 0, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return 0, errInvalidArgument
}
if err := checkPathLength(filePath); err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
if err := mkdirAll(pathutil.Dir(filePath), 0777); err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
if err := checkDiskFree(pathutil.Dir(filePath), fallocSize); err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
writer, err := lock.Open(filePath, os.O_CREATE|os.O_WRONLY, 0666)
@@ -276,7 +312,8 @@ func fsCreateFile(filePath string, reader io.Reader, buf []byte, fallocSize int6
// Fallocate only if the size is final object is known.
if fallocSize > 0 {
if err = fsFAllocate(int(writer.Fd()), 0, fallocSize); err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
}
@@ -284,12 +321,14 @@ func fsCreateFile(filePath string, reader io.Reader, buf []byte, fallocSize int6
if buf != nil {
bytesWritten, err = io.CopyBuffer(writer, reader, buf)
if err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
} else {
bytesWritten, err = io.Copy(writer, reader)
if err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
}
@@ -320,12 +359,14 @@ func fsFAllocate(fd int, offset int64, len int64) (err error) {
// Renames source path to destination path, creates all the
// missing parents if they don't exist.
func fsRenameFile(sourcePath, destPath string) error {
func fsRenameFile(ctx context.Context, sourcePath, destPath string) error {
if err := checkPathLength(sourcePath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
if err := checkPathLength(destPath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
// Verify if source path exists.
@@ -334,27 +375,34 @@ func fsRenameFile(sourcePath, destPath string) error {
}
if err := renameAll(sourcePath, destPath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
return nil
}
// fsDeleteFile is a wrapper for deleteFile(), after checking the path length.
func fsDeleteFile(basePath, deletePath string) error {
func fsDeleteFile(ctx context.Context, basePath, deletePath string) error {
if err := checkPathLength(basePath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
if err := checkPathLength(deletePath); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
return deleteFile(basePath, deletePath)
if err := deleteFile(basePath, deletePath); err != nil {
logger.LogIf(ctx, err)
return err
}
return nil
}
// fsRemoveMeta safely removes a locked file and takes care of Windows special case
func fsRemoveMeta(basePath, deletePath, tmpDir string) error {
func fsRemoveMeta(ctx context.Context, basePath, deletePath, tmpDir string) error {
// Special case for windows please read through.
if runtime.GOOS == globalWindowsOSName {
// Ordinarily windows does not permit deletion or renaming of files still
@@ -388,13 +436,13 @@ func fsRemoveMeta(basePath, deletePath, tmpDir string) error {
tmpPath := pathJoin(tmpDir, mustGetUUID())
fsRenameFile(deletePath, tmpPath)
fsRenameFile(ctx, deletePath, tmpPath)
// Proceed to deleting the directory if empty
fsDeleteFile(basePath, pathutil.Dir(deletePath))
fsDeleteFile(ctx, basePath, pathutil.Dir(deletePath))
// Finally delete the renamed file.
return fsDeleteFile(tmpDir, tmpPath)
return fsDeleteFile(ctx, tmpDir, tmpPath)
}
return fsDeleteFile(basePath, deletePath)
return fsDeleteFile(ctx, basePath, deletePath)
}
+43 -43
View File
@@ -18,13 +18,13 @@ package cmd
import (
"bytes"
"context"
"io"
"io/ioutil"
"os"
"path"
"testing"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/lock"
)
@@ -36,19 +36,19 @@ func TestFSRenameFile(t *testing.T) {
}
defer os.RemoveAll(path)
if err = fsMkdir(pathJoin(path, "testvolume1")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "testvolume1")); err != nil {
t.Fatal(err)
}
if err = fsRenameFile(pathJoin(path, "testvolume1"), pathJoin(path, "testvolume2")); err != nil {
if err = fsRenameFile(context.Background(), pathJoin(path, "testvolume1"), pathJoin(path, "testvolume2")); err != nil {
t.Fatal(err)
}
if err = fsRenameFile(pathJoin(path, "testvolume1"), pathJoin(path, "testvolume2")); errors.Cause(err) != errFileNotFound {
if err = fsRenameFile(context.Background(), pathJoin(path, "testvolume1"), pathJoin(path, "testvolume2")); err != errFileNotFound {
t.Fatal(err)
}
if err = fsRenameFile(pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"), pathJoin(path, "testvolume2")); errors.Cause(err) != errFileNameTooLong {
if err = fsRenameFile(context.Background(), pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"), pathJoin(path, "testvolume2")); err != errFileNameTooLong {
t.Fatal("Unexpected error", err)
}
if err = fsRenameFile(pathJoin(path, "testvolume1"), pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001")); errors.Cause(err) != errFileNameTooLong {
if err = fsRenameFile(context.Background(), pathJoin(path, "testvolume1"), pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001")); err != errFileNameTooLong {
t.Fatal("Unexpected error", err)
}
}
@@ -63,30 +63,30 @@ func TestFSStats(t *testing.T) {
// Setup test environment.
if err = fsMkdir(""); errors.Cause(err) != errInvalidArgument {
if err = fsMkdir(context.Background(), ""); err != errInvalidArgument {
t.Fatal("Unexpected error", err)
}
if err = fsMkdir(pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001")); errors.Cause(err) != errFileNameTooLong {
if err = fsMkdir(context.Background(), pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001")); err != errFileNameTooLong {
t.Fatal("Unexpected error", err)
}
if err = fsMkdir(pathJoin(path, "success-vol")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol")); err != nil {
t.Fatalf("Unable to create volume, %s", err)
}
var reader = bytes.NewReader([]byte("Hello, world"))
if _, err = fsCreateFile(pathJoin(path, "success-vol", "success-file"), reader, nil, 0); err != nil {
if _, err = fsCreateFile(context.Background(), pathJoin(path, "success-vol", "success-file"), reader, nil, 0); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
// Seek back.
reader.Seek(0, 0)
if err = fsMkdir(pathJoin(path, "success-vol", "success-file")); errors.Cause(err) != errVolumeExists {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol", "success-file")); err != errVolumeExists {
t.Fatal("Unexpected error", err)
}
if _, err = fsCreateFile(pathJoin(path, "success-vol", "path/to/success-file"), reader, nil, 0); err != nil {
if _, err = fsCreateFile(context.Background(), pathJoin(path, "success-vol", "path/to/success-file"), reader, nil, 0); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
// Seek back.
@@ -169,12 +169,12 @@ func TestFSStats(t *testing.T) {
for i, testCase := range testCases {
if testCase.srcPath != "" {
if _, err := fsStatFile(pathJoin(testCase.srcFSPath, testCase.srcVol,
testCase.srcPath)); errors.Cause(err) != testCase.expectedErr {
if _, err := fsStatFile(context.Background(), pathJoin(testCase.srcFSPath, testCase.srcVol,
testCase.srcPath)); err != testCase.expectedErr {
t.Fatalf("TestPosix case %d: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, err)
}
} else {
if _, err := fsStatVolume(pathJoin(testCase.srcFSPath, testCase.srcVol)); errors.Cause(err) != testCase.expectedErr {
if _, err := fsStatVolume(context.Background(), pathJoin(testCase.srcFSPath, testCase.srcVol)); err != testCase.expectedErr {
t.Fatalf("TestPosix case %d: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, err)
}
}
@@ -189,20 +189,20 @@ func TestFSCreateAndOpen(t *testing.T) {
}
defer os.RemoveAll(path)
if err = fsMkdir(pathJoin(path, "success-vol")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol")); err != nil {
t.Fatalf("Unable to create directory, %s", err)
}
if _, err = fsCreateFile("", nil, nil, 0); errors.Cause(err) != errInvalidArgument {
if _, err = fsCreateFile(context.Background(), "", nil, nil, 0); err != errInvalidArgument {
t.Fatal("Unexpected error", err)
}
if _, _, err = fsOpenFile("", -1); errors.Cause(err) != errInvalidArgument {
if _, _, err = fsOpenFile(context.Background(), "", -1); err != errInvalidArgument {
t.Fatal("Unexpected error", err)
}
var reader = bytes.NewReader([]byte("Hello, world"))
if _, err = fsCreateFile(pathJoin(path, "success-vol", "success-file"), reader, nil, 0); err != nil {
if _, err = fsCreateFile(context.Background(), pathJoin(path, "success-vol", "success-file"), reader, nil, 0); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
// Seek back.
@@ -230,18 +230,18 @@ func TestFSCreateAndOpen(t *testing.T) {
}
for i, testCase := range testCases {
_, err = fsCreateFile(pathJoin(path, testCase.srcVol, testCase.srcPath), reader, nil, 0)
if errors.Cause(err) != testCase.expectedErr {
_, err = fsCreateFile(context.Background(), pathJoin(path, testCase.srcVol, testCase.srcPath), reader, nil, 0)
if err != testCase.expectedErr {
t.Errorf("Test case %d: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, err)
}
_, _, err = fsOpenFile(pathJoin(path, testCase.srcVol, testCase.srcPath), 0)
if errors.Cause(err) != testCase.expectedErr {
_, _, err = fsOpenFile(context.Background(), pathJoin(path, testCase.srcVol, testCase.srcPath), 0)
if err != testCase.expectedErr {
t.Errorf("Test case %d: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, err)
}
}
// Attempt to open a directory.
if _, _, err = fsOpenFile(pathJoin(path), 0); errors.Cause(err) != errIsNotRegular {
if _, _, err = fsOpenFile(context.Background(), pathJoin(path), 0); err != errIsNotRegular {
t.Fatal("Unexpected error", err)
}
}
@@ -255,20 +255,20 @@ func TestFSDeletes(t *testing.T) {
defer os.RemoveAll(path)
// Setup test environment.
if err = fsMkdir(pathJoin(path, "success-vol")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol")); err != nil {
t.Fatalf("Unable to create directory, %s", err)
}
var buf = make([]byte, 4096)
var reader = bytes.NewReader([]byte("Hello, world"))
if _, err = fsCreateFile(pathJoin(path, "success-vol", "success-file"), reader, buf, reader.Size()); err != nil {
if _, err = fsCreateFile(context.Background(), pathJoin(path, "success-vol", "success-file"), reader, buf, reader.Size()); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
// Seek back.
reader.Seek(0, io.SeekStart)
// folder is not empty
err = fsMkdir(pathJoin(path, "success-vol", "not-empty"))
err = fsMkdir(context.Background(), pathJoin(path, "success-vol", "not-empty"))
if err != nil {
t.Fatal(err)
}
@@ -278,10 +278,10 @@ func TestFSDeletes(t *testing.T) {
}
// recursive
if err = fsMkdir(pathJoin(path, "success-vol", "parent")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol", "parent")); err != nil {
t.Fatal(err)
}
if err = fsMkdir(pathJoin(path, "success-vol", "parent", "dir")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol", "parent", "dir")); err != nil {
t.Fatal(err)
}
@@ -343,7 +343,7 @@ func TestFSDeletes(t *testing.T) {
}
for i, testCase := range testCases {
if err = fsDeleteFile(testCase.basePath, pathJoin(testCase.basePath, testCase.srcVol, testCase.srcPath)); errors.Cause(err) != testCase.expectedErr {
if err = fsDeleteFile(context.Background(), testCase.basePath, pathJoin(testCase.basePath, testCase.srcVol, testCase.srcPath)); err != testCase.expectedErr {
t.Errorf("Test case %d: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, err)
}
}
@@ -358,7 +358,7 @@ func BenchmarkFSDeleteFile(b *testing.B) {
defer os.RemoveAll(path)
// Setup test environment.
if err = fsMkdir(pathJoin(path, "benchmark")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "benchmark")); err != nil {
b.Fatalf("Unable to create directory, %s", err)
}
@@ -375,7 +375,7 @@ func BenchmarkFSDeleteFile(b *testing.B) {
}
b.StartTimer()
err = fsDeleteFile(benchDir, filename)
err = fsDeleteFile(context.Background(), benchDir, filename)
if err != nil {
b.Fatal(err)
}
@@ -392,18 +392,18 @@ func TestFSRemoves(t *testing.T) {
defer os.RemoveAll(path)
// Setup test environment.
if err = fsMkdir(pathJoin(path, "success-vol")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(path, "success-vol")); err != nil {
t.Fatalf("Unable to create directory, %s", err)
}
var reader = bytes.NewReader([]byte("Hello, world"))
if _, err = fsCreateFile(pathJoin(path, "success-vol", "success-file"), reader, nil, 0); err != nil {
if _, err = fsCreateFile(context.Background(), pathJoin(path, "success-vol", "success-file"), reader, nil, 0); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
// Seek back.
reader.Seek(0, 0)
if _, err = fsCreateFile(pathJoin(path, "success-vol", "success-file-new"), reader, nil, 0); err != nil {
if _, err = fsCreateFile(context.Background(), pathJoin(path, "success-vol", "success-file-new"), reader, nil, 0); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
// Seek back.
@@ -477,25 +477,25 @@ func TestFSRemoves(t *testing.T) {
for i, testCase := range testCases {
if testCase.srcPath != "" {
if err = fsRemoveFile(pathJoin(testCase.srcFSPath, testCase.srcVol, testCase.srcPath)); errors.Cause(err) != testCase.expectedErr {
if err = fsRemoveFile(context.Background(), pathJoin(testCase.srcFSPath, testCase.srcVol, testCase.srcPath)); err != testCase.expectedErr {
t.Errorf("Test case %d: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, err)
}
} else {
if err = fsRemoveDir(pathJoin(testCase.srcFSPath, testCase.srcVol, testCase.srcPath)); errors.Cause(err) != testCase.expectedErr {
if err = fsRemoveDir(context.Background(), pathJoin(testCase.srcFSPath, testCase.srcVol, testCase.srcPath)); err != testCase.expectedErr {
t.Error(err)
}
}
}
if err = fsRemoveAll(pathJoin(path, "success-vol")); err != nil {
if err = fsRemoveAll(context.Background(), pathJoin(path, "success-vol")); err != nil {
t.Fatal(err)
}
if err = fsRemoveAll(""); errors.Cause(err) != errInvalidArgument {
if err = fsRemoveAll(context.Background(), ""); err != errInvalidArgument {
t.Fatal(err)
}
if err = fsRemoveAll("my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"); errors.Cause(err) != errFileNameTooLong {
if err = fsRemoveAll(context.Background(), "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"); err != errFileNameTooLong {
t.Fatal(err)
}
}
@@ -509,14 +509,14 @@ func TestFSRemoveMeta(t *testing.T) {
defer os.RemoveAll(fsPath)
// Setup test environment.
if err = fsMkdir(pathJoin(fsPath, "success-vol")); err != nil {
if err = fsMkdir(context.Background(), pathJoin(fsPath, "success-vol")); err != nil {
t.Fatalf("Unable to create directory, %s", err)
}
filePath := pathJoin(fsPath, "success-vol", "success-file")
var reader = bytes.NewReader([]byte("Hello, world"))
if _, err = fsCreateFile(filePath, reader, nil, 0); err != nil {
if _, err = fsCreateFile(context.Background(), filePath, reader, nil, 0); err != nil {
t.Fatalf("Unable to create file, %s", err)
}
@@ -535,7 +535,7 @@ func TestFSRemoveMeta(t *testing.T) {
t.Fatal(tmpErr)
}
if err := fsRemoveMeta(fsPath, filePath, tmpDir); err != nil {
if err := fsRemoveMeta(context.Background(), fsPath, filePath, tmpDir); err != nil {
t.Fatalf("Unable to remove file, %s", err)
}
+17 -15
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"encoding/hex"
"encoding/json"
"io"
@@ -25,7 +26,7 @@ import (
pathutil "path"
"strings"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
"github.com/minio/minio/pkg/mimedb"
"github.com/tidwall/gjson"
@@ -165,10 +166,14 @@ func (m fsMetaV1) ToObjectInfo(bucket, object string, fi os.FileInfo) ObjectInfo
}
}
// Extract etag from metadata.
objInfo.ETag = extractETag(m.Meta)
objInfo.ContentType = m.Meta["content-type"]
objInfo.ContentEncoding = m.Meta["content-encoding"]
if storageClass, ok := m.Meta[amzStorageClass]; ok {
objInfo.StorageClass = storageClass
} else {
objInfo.StorageClass = globalMinioDefaultStorageClass
}
// etag/md5Sum has already been extracted. We need to
// remove to avoid it from appearing as part of
@@ -197,14 +202,6 @@ func parseFSVersion(fsMetaBuf []byte) string {
return gjson.GetBytes(fsMetaBuf, "version").String()
}
func parseFSFormat(fsMetaBuf []byte) string {
return gjson.GetBytes(fsMetaBuf, "format").String()
}
func parseFSRelease(fsMetaBuf []byte) string {
return gjson.GetBytes(fsMetaBuf, "minio.release").String()
}
func parseFSMetaMap(fsMetaBuf []byte) map[string]string {
// Get xlMetaV1.Meta map.
metaMapResult := gjson.GetBytes(fsMetaBuf, "meta").Map()
@@ -237,20 +234,23 @@ func parseFSPartsArray(fsMetaBuf []byte) []objectPartInfo {
return partsArray
}
func (m *fsMetaV1) ReadFrom(lk *lock.LockedFile) (n int64, err error) {
func (m *fsMetaV1) ReadFrom(ctx context.Context, lk *lock.LockedFile) (n int64, err error) {
var fsMetaBuf []byte
fi, err := lk.Stat()
if err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
fsMetaBuf, err = ioutil.ReadAll(io.NewSectionReader(lk, 0, fi.Size()))
if err != nil {
return 0, errors.Trace(err)
logger.LogIf(ctx, err)
return 0, err
}
if len(fsMetaBuf) == 0 {
return 0, errors.Trace(io.EOF)
logger.LogIf(ctx, io.EOF)
return 0, io.EOF
}
// obtain version.
@@ -259,7 +259,9 @@ func (m *fsMetaV1) ReadFrom(lk *lock.LockedFile) (n int64, err error) {
// Verify if the format is valid, return corrupted format
// for unrecognized formats.
if !isFSMetaValid(m.Version) {
return 0, errors.Trace(errCorruptedFormat)
logger.GetReqInfo(ctx).AppendTags("file", lk.Name())
logger.LogIf(ctx, errCorruptedFormat)
return 0, errCorruptedFormat
}
// obtain parts information
+2 -2
View File
@@ -70,7 +70,7 @@ func TestReadFSMetadata(t *testing.T) {
// Regular fs metadata reading, no errors expected
fsMeta := fsMetaV1{}
if _, err = fsMeta.ReadFrom(rlk.LockedFile); err != nil {
if _, err = fsMeta.ReadFrom(context.Background(), rlk.LockedFile); err != nil {
t.Fatal("Unexpected error ", err)
}
}
@@ -105,7 +105,7 @@ func TestWriteFSMetadata(t *testing.T) {
// FS metadata reading, no errors expected (healthy disk)
fsMeta := fsMetaV1{}
_, err = fsMeta.ReadFrom(rlk.LockedFile)
_, err = fsMeta.ReadFrom(context.Background(), rlk.LockedFile)
if err != nil {
t.Fatal("Unexpected error ", err)
}
+115 -93
View File
@@ -29,7 +29,7 @@ import (
"strings"
"time"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
mioutil "github.com/minio/minio/pkg/ioutil"
"github.com/minio/minio/pkg/hash"
@@ -64,8 +64,9 @@ func (fs *FSObjects) decodePartFile(name string) (partNumber int, etag string, e
}
// Appends parts to an appendFile sequentially.
func (fs *FSObjects) backgroundAppend(bucket, object, uploadID string) {
func (fs *FSObjects) backgroundAppend(ctx context.Context, bucket, object, uploadID string) {
fs.appendFileMapMu.Lock()
logger.GetReqInfo(ctx).AppendTags("uploadID", uploadID)
file := fs.appendFileMap[uploadID]
if file == nil {
file = &fsAppendFile{
@@ -84,7 +85,8 @@ func (fs *FSObjects) backgroundAppend(bucket, object, uploadID string) {
entries, err := readDir(uploadIDDir)
if err != nil {
errorIf(err, "error reading directory %s", uploadIDDir)
logger.GetReqInfo(ctx).AppendTags("uploadIDDir", uploadIDDir)
logger.LogIf(ctx, err)
return
}
sort.Strings(entries)
@@ -95,7 +97,8 @@ func (fs *FSObjects) backgroundAppend(bucket, object, uploadID string) {
}
partNumber, etag, err := fs.decodePartFile(entry)
if err != nil {
errorIf(err, "unable to split the file name into partNumber and etag: %s", entry)
logger.GetReqInfo(ctx).AppendTags("entry", entry)
logger.LogIf(ctx, err)
return
}
if partNumber < nextPartNumber {
@@ -110,7 +113,9 @@ func (fs *FSObjects) backgroundAppend(bucket, object, uploadID string) {
partPath := pathJoin(uploadIDDir, entry)
err = mioutil.AppendFile(file.filePath, partPath)
if err != nil {
errorIf(err, "Unable to append %s to %s", partPath, file.filePath)
reqInfo := logger.GetReqInfo(ctx).AppendTags("partPath", partPath)
reqInfo.AppendTags("filepath", file.filePath)
logger.LogIf(ctx, err)
return
}
@@ -122,12 +127,12 @@ func (fs *FSObjects) backgroundAppend(bucket, object, uploadID string) {
// ListMultipartUploads - lists all the uploadIDs for the specified object.
// We do not support prefix based listing.
func (fs *FSObjects) ListMultipartUploads(ctx context.Context, bucket, object, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, e error) {
if err := checkListMultipartArgs(bucket, object, keyMarker, uploadIDMarker, delimiter, fs); err != nil {
return result, toObjectErr(errors.Trace(err))
if err := checkListMultipartArgs(ctx, bucket, object, keyMarker, uploadIDMarker, delimiter, fs); err != nil {
return result, toObjectErr(err)
}
if _, err := fs.statBucketDir(bucket); err != nil {
return result, toObjectErr(errors.Trace(err), bucket)
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return result, toObjectErr(err, bucket)
}
result.MaxUploads = maxUploads
@@ -143,7 +148,8 @@ func (fs *FSObjects) ListMultipartUploads(ctx context.Context, bucket, object, k
result.IsTruncated = false
return result, nil
}
return result, toObjectErr(errors.Trace(err))
logger.LogIf(ctx, err)
return result, toObjectErr(err)
}
// S3 spec says uploaIDs should be sorted based on initiated time. ModTime of fs.json
@@ -151,7 +157,7 @@ func (fs *FSObjects) ListMultipartUploads(ctx context.Context, bucket, object, k
var uploads []MultipartInfo
for _, uploadID := range uploadIDs {
metaFilePath := pathJoin(fs.getMultipartSHADir(bucket, object), uploadID, fs.metaJSONFile)
fi, err := fsStatFile(metaFilePath)
fi, err := fsStatFile(ctx, metaFilePath)
if err != nil {
return result, toObjectErr(err, bucket, object)
}
@@ -204,11 +210,11 @@ func (fs *FSObjects) ListMultipartUploads(ctx context.Context, bucket, object, k
//
// Implements S3 compatible initiate multipart API.
func (fs *FSObjects) NewMultipartUpload(ctx context.Context, bucket, object string, meta map[string]string) (string, error) {
if err := checkNewMultipartArgs(bucket, object, fs); err != nil {
if err := checkNewMultipartArgs(ctx, bucket, object, fs); err != nil {
return "", toObjectErr(err, bucket)
}
if _, err := fs.statBucketDir(bucket); err != nil {
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return "", toObjectErr(err, bucket)
}
@@ -217,7 +223,8 @@ func (fs *FSObjects) NewMultipartUpload(ctx context.Context, bucket, object stri
err := mkdirAll(uploadIDDir, 0755)
if err != nil {
return "", errors.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
// Initialize fs.json values.
@@ -226,11 +233,13 @@ func (fs *FSObjects) NewMultipartUpload(ctx context.Context, bucket, object stri
fsMetaBytes, err := json.Marshal(fsMeta)
if err != nil {
return "", errors.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
if err = ioutil.WriteFile(pathJoin(uploadIDDir, fs.metaJSONFile), fsMetaBytes, 0644); err != nil {
return "", errors.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
return uploadID, nil
@@ -242,22 +251,22 @@ func (fs *FSObjects) NewMultipartUpload(ctx context.Context, bucket, object stri
func (fs *FSObjects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject, uploadID string, partID int,
startOffset int64, length int64, srcInfo ObjectInfo) (pi PartInfo, e error) {
if err := checkNewMultipartArgs(srcBucket, srcObject, fs); err != nil {
return pi, toObjectErr(errors.Trace(err))
if err := checkNewMultipartArgs(ctx, srcBucket, srcObject, fs); err != nil {
return pi, toObjectErr(err)
}
// Initialize pipe.
go func() {
if gerr := fs.GetObject(ctx, srcBucket, srcObject, startOffset, length, srcInfo.Writer, srcInfo.ETag); gerr != nil {
if gerr = srcInfo.Writer.Close(); gerr != nil {
errorIf(gerr, "Unable to read %s/%s.", srcBucket, srcObject)
logger.LogIf(ctx, gerr)
return
}
return
}
// Close writer explicitly signalling we wrote all data.
if gerr := srcInfo.Writer.Close(); gerr != nil {
errorIf(gerr, "Unable to read %s/%s.", srcBucket, srcObject)
logger.LogIf(ctx, gerr)
return
}
}()
@@ -275,26 +284,27 @@ func (fs *FSObjects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, d
// written to '.minio.sys/tmp' location and safely renamed to
// '.minio.sys/multipart' for reach parts.
func (fs *FSObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID string, partID int, data *hash.Reader) (pi PartInfo, e error) {
if err := checkPutObjectPartArgs(bucket, object, fs); err != nil {
return pi, toObjectErr(errors.Trace(err), bucket)
if err := checkPutObjectPartArgs(ctx, bucket, object, fs); err != nil {
return pi, toObjectErr(err, bucket)
}
if _, err := fs.statBucketDir(bucket); err != nil {
return pi, toObjectErr(errors.Trace(err), bucket)
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return pi, toObjectErr(err, bucket)
}
// Validate input data size and it can never be less than zero.
if data.Size() < 0 {
return pi, toObjectErr(errors.Trace(errInvalidArgument))
logger.LogIf(ctx, errInvalidArgument)
return pi, toObjectErr(errInvalidArgument)
}
uploadIDDir := fs.getUploadIDDir(bucket, object, uploadID)
// Just check if the uploadID exists to avoid copy if it doesn't.
_, err := fsStatFile(pathJoin(uploadIDDir, fs.metaJSONFile))
_, err := fsStatFile(ctx, pathJoin(uploadIDDir, fs.metaJSONFile))
if err != nil {
if errors.Cause(err) == errFileNotFound || errors.Cause(err) == errFileAccessDenied {
return pi, errors.Trace(InvalidUploadID{UploadID: uploadID})
if err == errFileNotFound || err == errFileAccessDenied {
return pi, InvalidUploadID{UploadID: uploadID}
}
return pi, toObjectErr(err, bucket, object)
}
@@ -306,23 +316,23 @@ func (fs *FSObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID
buf := make([]byte, bufSize)
tmpPartPath := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID, uploadID+"."+mustGetUUID()+"."+strconv.Itoa(partID))
bytesWritten, err := fsCreateFile(tmpPartPath, data, buf, data.Size())
bytesWritten, err := fsCreateFile(ctx, tmpPartPath, data, buf, data.Size())
if err != nil {
fsRemoveFile(tmpPartPath)
fsRemoveFile(ctx, tmpPartPath)
return pi, toObjectErr(err, minioMetaTmpBucket, tmpPartPath)
}
// Should return IncompleteBody{} error when reader has fewer
// bytes than specified in request header.
if bytesWritten < data.Size() {
fsRemoveFile(tmpPartPath)
return pi, errors.Trace(IncompleteBody{})
fsRemoveFile(ctx, tmpPartPath)
return pi, IncompleteBody{}
}
// Delete temporary part in case of failure. If
// PutObjectPart succeeds then there would be nothing to
// delete in which case we just ignore the error.
defer fsRemoveFile(tmpPartPath)
defer fsRemoveFile(ctx, tmpPartPath)
etag := hex.EncodeToString(data.MD5Current())
if etag == "" {
@@ -330,13 +340,13 @@ func (fs *FSObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID
}
partPath := pathJoin(uploadIDDir, fs.encodePartFile(partID, etag))
if err = fsRenameFile(tmpPartPath, partPath); err != nil {
if err = fsRenameFile(ctx, tmpPartPath, partPath); err != nil {
return pi, toObjectErr(err, minioMetaMultipartBucket, partPath)
}
go fs.backgroundAppend(bucket, object, uploadID)
go fs.backgroundAppend(ctx, bucket, object, uploadID)
fi, err := fsStatFile(partPath)
fi, err := fsStatFile(ctx, partPath)
if err != nil {
return pi, toObjectErr(err, minioMetaMultipartBucket, partPath)
}
@@ -356,8 +366,8 @@ func (fs *FSObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID
// ListPartsInfo structure is unmarshalled directly into XML and
// replied back to the client.
func (fs *FSObjects) ListObjectParts(ctx context.Context, bucket, object, uploadID string, partNumberMarker, maxParts int) (result ListPartsInfo, e error) {
if err := checkListPartsArgs(bucket, object, fs); err != nil {
return result, toObjectErr(errors.Trace(err))
if err := checkListPartsArgs(ctx, bucket, object, fs); err != nil {
return result, toObjectErr(err)
}
result.Bucket = bucket
result.Object = object
@@ -366,22 +376,23 @@ func (fs *FSObjects) ListObjectParts(ctx context.Context, bucket, object, upload
result.PartNumberMarker = partNumberMarker
// Check if bucket exists
if _, err := fs.statBucketDir(bucket); err != nil {
return result, toObjectErr(errors.Trace(err), bucket)
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return result, toObjectErr(err, bucket)
}
uploadIDDir := fs.getUploadIDDir(bucket, object, uploadID)
_, err := fsStatFile(pathJoin(uploadIDDir, fs.metaJSONFile))
_, err := fsStatFile(ctx, pathJoin(uploadIDDir, fs.metaJSONFile))
if err != nil {
if errors.Cause(err) == errFileNotFound || errors.Cause(err) == errFileAccessDenied {
return result, errors.Trace(InvalidUploadID{UploadID: uploadID})
if err == errFileNotFound || err == errFileAccessDenied {
return result, InvalidUploadID{UploadID: uploadID}
}
return result, toObjectErr(errors.Trace(err), bucket, object)
return result, toObjectErr(err, bucket, object)
}
entries, err := readDir(uploadIDDir)
if err != nil {
return result, toObjectErr(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return result, toObjectErr(err, bucket)
}
partsMap := make(map[int]string)
@@ -391,20 +402,21 @@ func (fs *FSObjects) ListObjectParts(ctx context.Context, bucket, object, upload
}
partNumber, etag1, derr := fs.decodePartFile(entry)
if derr != nil {
return result, toObjectErr(errors.Trace(derr))
logger.LogIf(ctx, derr)
return result, toObjectErr(derr)
}
etag2, ok := partsMap[partNumber]
if !ok {
partsMap[partNumber] = etag1
continue
}
stat1, serr := fsStatFile(pathJoin(uploadIDDir, fs.encodePartFile(partNumber, etag1)))
stat1, serr := fsStatFile(ctx, pathJoin(uploadIDDir, fs.encodePartFile(partNumber, etag1)))
if serr != nil {
return result, toObjectErr(errors.Trace(serr))
return result, toObjectErr(serr)
}
stat2, serr := fsStatFile(pathJoin(uploadIDDir, fs.encodePartFile(partNumber, etag2)))
stat2, serr := fsStatFile(ctx, pathJoin(uploadIDDir, fs.encodePartFile(partNumber, etag2)))
if serr != nil {
return result, toObjectErr(errors.Trace(serr))
return result, toObjectErr(serr)
}
if stat1.ModTime().After(stat2.ModTime()) {
partsMap[partNumber] = etag1
@@ -443,9 +455,9 @@ func (fs *FSObjects) ListObjectParts(ctx context.Context, bucket, object, upload
}
for i, part := range result.Parts {
var stat os.FileInfo
stat, err = fsStatFile(pathJoin(uploadIDDir, fs.encodePartFile(part.PartNumber, part.ETag)))
stat, err = fsStatFile(ctx, pathJoin(uploadIDDir, fs.encodePartFile(part.PartNumber, part.ETag)))
if err != nil {
return result, toObjectErr(errors.Trace(err))
return result, toObjectErr(err)
}
result.Parts[i].LastModified = stat.ModTime()
result.Parts[i].Size = stat.Size()
@@ -453,7 +465,8 @@ func (fs *FSObjects) ListObjectParts(ctx context.Context, bucket, object, upload
fsMetaBytes, err := ioutil.ReadFile(pathJoin(uploadIDDir, fs.metaJSONFile))
if err != nil {
return result, errors.Trace(err)
logger.LogIf(ctx, err)
return result, err
}
result.UserDefined = parseFSMetaMap(fsMetaBytes)
@@ -467,31 +480,31 @@ func (fs *FSObjects) ListObjectParts(ctx context.Context, bucket, object, upload
//
// Implements S3 compatible Complete multipart API.
func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string, object string, uploadID string, parts []CompletePart) (oi ObjectInfo, e error) {
if err := checkCompleteMultipartArgs(bucket, object, fs); err != nil {
if err := checkCompleteMultipartArgs(ctx, bucket, object, fs); err != nil {
return oi, toObjectErr(err)
}
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(bucket, pathutil.Dir(object)) {
return oi, toObjectErr(errors.Trace(errFileAccessDenied), bucket, object)
if fs.parentDirIsObject(ctx, bucket, pathutil.Dir(object)) {
return oi, toObjectErr(errFileAccessDenied, bucket, object)
}
if _, err := fs.statBucketDir(bucket); err != nil {
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return oi, toObjectErr(err, bucket)
}
uploadIDDir := fs.getUploadIDDir(bucket, object, uploadID)
// Just check if the uploadID exists to avoid copy if it doesn't.
_, err := fsStatFile(pathJoin(uploadIDDir, fs.metaJSONFile))
_, err := fsStatFile(ctx, pathJoin(uploadIDDir, fs.metaJSONFile))
if err != nil {
if errors.Cause(err) == errFileNotFound || errors.Cause(err) == errFileAccessDenied {
return oi, errors.Trace(InvalidUploadID{UploadID: uploadID})
if err == errFileNotFound || err == errFileAccessDenied {
return oi, InvalidUploadID{UploadID: uploadID}
}
return oi, toObjectErr(err, bucket, object)
}
// Calculate s3 compatible md5sum for complete multipart.
s3MD5, err := getCompleteMultipartMD5(parts)
s3MD5, err := getCompleteMultipartMD5(ctx, parts)
if err != nil {
return oi, err
}
@@ -507,12 +520,12 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
for i, part := range parts {
partPath := pathJoin(uploadIDDir, fs.encodePartFile(part.PartNumber, part.ETag))
var fi os.FileInfo
fi, err = fsStatFile(partPath)
fi, err = fsStatFile(ctx, partPath)
if err != nil {
if errors.Cause(err) == errFileNotFound || errors.Cause(err) == errFileAccessDenied {
return oi, errors.Trace(InvalidPart{})
if err == errFileNotFound || err == errFileAccessDenied {
return oi, InvalidPart{}
}
return oi, errors.Trace(err)
return oi, err
}
if partSize == -1 {
partSize = fi.Size()
@@ -530,11 +543,13 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// All parts except the last part has to be atleast 5MB.
if !isMinAllowedPartSize(fi.Size()) {
return oi, errors.Trace(PartTooSmall{
err = PartTooSmall{
PartNumber: part.PartNumber,
PartSize: fi.Size(),
PartETag: part.ETag,
})
}
logger.LogIf(ctx, err)
return oi, err
}
// TODO: Make necessary changes in future as explained in the below comment.
@@ -545,7 +560,8 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// CompleteMultipartUpload we already have the full file available which can be
// renamed to the main name-space.
if partSize != fi.Size() {
return oi, errors.Trace(PartsSizeUnequal{})
logger.LogIf(ctx, PartsSizeUnequal{})
return oi, PartsSizeUnequal{}
}
}
@@ -557,7 +573,7 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// 1. The last PutObjectPart triggers go-routine fs.backgroundAppend, this go-routine has not started yet.
// 2. Now CompleteMultipartUpload gets called which sees that lastPart is not appended and starts appending
// from the beginning
fs.backgroundAppend(bucket, object, uploadID)
fs.backgroundAppend(ctx, bucket, object, uploadID)
fs.appendFileMapMu.Lock()
file := fs.appendFileMap[uploadID]
@@ -585,12 +601,13 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
}
if appendFallback {
fsRemoveFile(file.filePath)
fsRemoveFile(ctx, file.filePath)
for _, part := range parts {
partPath := pathJoin(uploadIDDir, fs.encodePartFile(part.PartNumber, part.ETag))
err = mioutil.AppendFile(appendFilePath, partPath)
if err != nil {
return oi, toObjectErr(errors.Trace(err))
logger.LogIf(ctx, err)
return oi, toObjectErr(err)
}
}
}
@@ -604,18 +621,21 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
fsMetaPath := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, bucket, object, fs.metaJSONFile)
metaFile, err := fs.rwPool.Create(fsMetaPath)
if err != nil {
return oi, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, bucket, object)
}
defer metaFile.Close()
// Read saved fs metadata for ongoing multipart.
fsMetaBuf, err := ioutil.ReadFile(pathJoin(uploadIDDir, fs.metaJSONFile))
if err != nil {
return oi, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, bucket, object)
}
err = json.Unmarshal(fsMetaBuf, &fsMeta)
if err != nil {
return oi, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, bucket, object)
}
// Save additional metadata.
if len(fsMeta.Meta) == 0 {
@@ -623,24 +643,26 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
}
fsMeta.Meta["etag"] = s3MD5
if _, err = fsMeta.WriteTo(metaFile); err != nil {
return oi, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, bucket, object)
}
// Deny if WORM is enabled
if globalWORMEnabled {
if _, err = fsStatFile(pathJoin(fs.fsPath, bucket, object)); err == nil {
return ObjectInfo{}, errors.Trace(ObjectAlreadyExists{Bucket: bucket, Object: object})
if _, err = fsStatFile(ctx, pathJoin(fs.fsPath, bucket, object)); err == nil {
return ObjectInfo{}, ObjectAlreadyExists{Bucket: bucket, Object: object}
}
}
err = fsRenameFile(appendFilePath, pathJoin(fs.fsPath, bucket, object))
err = fsRenameFile(ctx, appendFilePath, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return oi, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, bucket, object)
}
fsRemoveAll(uploadIDDir)
fi, err := fsStatFile(pathJoin(fs.fsPath, bucket, object))
fsRemoveAll(ctx, uploadIDDir)
fi, err := fsStatFile(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return oi, toObjectErr(errors.Trace(err), bucket, object)
return oi, toObjectErr(err, bucket, object)
}
return fsMeta.ToObjectInfo(bucket, object, fi), nil
@@ -659,12 +681,12 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// no affect and further requests to the same uploadID would not be
// honored.
func (fs *FSObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string) error {
if err := checkAbortMultipartArgs(bucket, object, fs); err != nil {
if err := checkAbortMultipartArgs(ctx, bucket, object, fs); err != nil {
return err
}
if _, err := fs.statBucketDir(bucket); err != nil {
return toObjectErr(errors.Trace(err), bucket)
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return toObjectErr(err, bucket)
}
fs.appendFileMapMu.Lock()
@@ -673,16 +695,16 @@ func (fs *FSObjects) AbortMultipartUpload(ctx context.Context, bucket, object, u
uploadIDDir := fs.getUploadIDDir(bucket, object, uploadID)
// Just check if the uploadID exists to avoid copy if it doesn't.
_, err := fsStatFile(pathJoin(uploadIDDir, fs.metaJSONFile))
_, err := fsStatFile(ctx, pathJoin(uploadIDDir, fs.metaJSONFile))
if err != nil {
if errors.Cause(err) == errFileNotFound || errors.Cause(err) == errFileAccessDenied {
return errors.Trace(InvalidUploadID{UploadID: uploadID})
if err == errFileNotFound || err == errFileAccessDenied {
return InvalidUploadID{UploadID: uploadID}
}
return toObjectErr(errors.Trace(err), bucket, object)
return toObjectErr(err, bucket, object)
}
// Ignore the error returned as Windows fails to remove directory if a file in it
// is Open()ed by the backgroundAppend()
fsRemoveAll(uploadIDDir)
fsRemoveAll(ctx, uploadIDDir)
return nil
}
@@ -690,7 +712,7 @@ func (fs *FSObjects) AbortMultipartUpload(ctx context.Context, bucket, object, u
// Removes multipart uploads if any older than `expiry` duration
// on all buckets for every `cleanupInterval`, this function is
// blocking and should be run in a go-routine.
func (fs *FSObjects) cleanupStaleMultipartUploads(cleanupInterval, expiry time.Duration, doneCh chan struct{}) {
func (fs *FSObjects) cleanupStaleMultipartUploads(ctx context.Context, cleanupInterval, expiry time.Duration, doneCh chan struct{}) {
ticker := time.NewTicker(cleanupInterval)
for {
select {
@@ -710,12 +732,12 @@ func (fs *FSObjects) cleanupStaleMultipartUploads(cleanupInterval, expiry time.D
continue
}
for _, uploadID := range uploadIDs {
fi, err := fsStatDir(pathJoin(fs.fsPath, minioMetaMultipartBucket, entry, uploadID))
fi, err := fsStatDir(ctx, pathJoin(fs.fsPath, minioMetaMultipartBucket, entry, uploadID))
if err != nil {
continue
}
if now.Sub(fi.ModTime()) > expiry {
fsRemoveAll(pathJoin(fs.fsPath, minioMetaMultipartBucket, entry, uploadID))
fsRemoveAll(ctx, pathJoin(fs.fsPath, minioMetaMultipartBucket, entry, uploadID))
}
}
}
+5 -8
View File
@@ -23,8 +23,6 @@ import (
"path/filepath"
"testing"
"time"
"github.com/minio/minio/pkg/errors"
)
// Tests cleanup multipart uploads for filesystem backend.
@@ -49,7 +47,7 @@ func TestFSCleanupMultipartUploadsInRoutine(t *testing.T) {
t.Fatal("Unexpected err: ", err)
}
go fs.cleanupStaleMultipartUploads(20*time.Millisecond, 0, globalServiceDoneCh)
go fs.cleanupStaleMultipartUploads(context.Background(), 20*time.Millisecond, 0, globalServiceDoneCh)
// Wait for 40ms such that - we have given enough time for
// cleanup routine to kick in.
@@ -60,7 +58,6 @@ func TestFSCleanupMultipartUploadsInRoutine(t *testing.T) {
// Check if upload id was already purged.
if err = obj.AbortMultipartUpload(context.Background(), bucketName, objectName, uploadID); err != nil {
err = errors.Cause(err)
if _, ok := err.(InvalidUploadID); !ok {
t.Fatal("Unexpected err: ", err)
}
@@ -85,7 +82,7 @@ func TestNewMultipartUploadFaultyDisk(t *testing.T) {
// Test with disk removed.
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if _, err := fs.NewMultipartUpload(context.Background(), bucketName, objectName, map[string]string{"X-Amz-Meta-xid": "3f"}); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error ", err)
}
}
@@ -123,7 +120,7 @@ func TestPutObjectPartFaultyDisk(t *testing.T) {
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
_, err = fs.PutObjectPart(context.Background(), bucketName, objectName, uploadID, 1, mustGetHashReader(t, bytes.NewReader(data), dataLen, md5Hex, sha256sum))
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error ", err)
}
}
@@ -154,7 +151,7 @@ func TestCompleteMultipartUploadFaultyDisk(t *testing.T) {
parts := []CompletePart{{PartNumber: 1, ETag: md5Hex}}
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if _, err := fs.CompleteMultipartUpload(context.Background(), bucketName, objectName, uploadID, parts); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error ", err)
}
}
@@ -249,7 +246,7 @@ func TestListMultipartUploadsFaultyDisk(t *testing.T) {
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if _, err := fs.ListMultipartUploads(context.Background(), bucketName, objectName, "", "", "", 1000); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error ", err)
}
}
+5 -1
View File
@@ -17,10 +17,12 @@
package cmd
import (
"context"
"os"
pathutil "path"
"sync"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
)
@@ -48,7 +50,9 @@ func (fsi *fsIOPool) lookupToRead(path string) (*lock.RLockedFile, bool) {
// If the file is closed and not removed from map is a bug.
if rlkFile.IsClosed() {
// Log this as an error.
errorIf(errUnexpected, "Unexpected entry found on the map %s", path)
reqInfo := (&logger.ReqInfo{}).AppendTags("path", path)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errUnexpected)
// Purge the cached lock path from map.
delete(fsi.readersMap, path)
+144 -132
View File
@@ -24,15 +24,13 @@ import (
"io/ioutil"
"os"
"path"
"path/filepath"
"reflect"
"sort"
"sync"
"syscall"
"time"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/lock"
"github.com/minio/minio/pkg/madmin"
@@ -97,38 +95,13 @@ func initMetaVolumeFS(fsPath, fsUUID string) error {
// NewFSObjectLayer - initialize new fs object layer.
func NewFSObjectLayer(fsPath string) (ObjectLayer, error) {
ctx := context.Background()
if fsPath == "" {
return nil, errInvalidArgument
}
var err error
// Disallow relative paths, figure out absolute paths.
fsPath, err = filepath.Abs(fsPath)
if err != nil {
return nil, err
}
fi, err := os.Stat((fsPath))
if err == nil {
if !fi.IsDir() {
return nil, syscall.ENOTDIR
}
}
if os.IsNotExist(err) {
// Disk not found create it.
err = os.MkdirAll(fsPath, 0777)
if err != nil {
return nil, err
}
}
di, err := getDiskInfo((fsPath))
if err != nil {
return nil, err
}
// Check if disk has minimum required total space.
if err = checkDiskMinTotal(di); err != nil {
if fsPath, err = checkPathValid(fsPath); err != nil {
return nil, err
}
@@ -142,7 +115,7 @@ func NewFSObjectLayer(fsPath string) (ObjectLayer, error) {
}
// Initialize `format.json`, this function also returns.
rlk, err := initFormatFS(fsPath)
rlk, err := initFormatFS(ctx, fsPath)
if err != nil {
return nil, err
}
@@ -177,7 +150,7 @@ func NewFSObjectLayer(fsPath string) (ObjectLayer, error) {
return nil, fmt.Errorf("Unable to initialize event notification. %s", err)
}
go fs.cleanupStaleMultipartUploads(globalMultipartCleanupInterval, globalMultipartExpiry, globalServiceDoneCh)
go fs.cleanupStaleMultipartUploads(ctx, globalMultipartCleanupInterval, globalMultipartExpiry, globalServiceDoneCh)
// Return successfully initialized object layer.
return fs, nil
@@ -188,13 +161,14 @@ func (fs *FSObjects) Shutdown(ctx context.Context) error {
fs.fsFormatRlk.Close()
// Cleanup and delete tmp uuid.
return fsRemoveAll(pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID))
return fsRemoveAll(ctx, pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID))
}
// StorageInfo - returns underlying storage statistics.
func (fs *FSObjects) StorageInfo(ctx context.Context) StorageInfo {
info, err := getDiskInfo((fs.fsPath))
errorIf(err, "Unable to get disk info %#v", fs.fsPath)
logger.GetReqInfo(ctx).AppendTags("path", fs.fsPath)
logger.LogIf(ctx, err)
storageInfo := StorageInfo{
Total: info.Total,
Free: info.Free,
@@ -220,22 +194,24 @@ func (fs *FSObjects) ClearLocks(ctx context.Context, info []VolumeLockInfo) erro
// getBucketDir - will convert incoming bucket names to
// corresponding valid bucket names on the backend in a platform
// compatible way for all operating systems.
func (fs *FSObjects) getBucketDir(bucket string) (string, error) {
func (fs *FSObjects) getBucketDir(ctx context.Context, bucket string) (string, error) {
// Verify if bucket is valid.
if !IsValidBucketName(bucket) {
return "", errors.Trace(BucketNameInvalid{Bucket: bucket})
err := BucketNameInvalid{Bucket: bucket}
logger.LogIf(ctx, err)
return "", err
}
bucketDir := pathJoin(fs.fsPath, bucket)
return bucketDir, nil
}
func (fs *FSObjects) statBucketDir(bucket string) (os.FileInfo, error) {
bucketDir, err := fs.getBucketDir(bucket)
func (fs *FSObjects) statBucketDir(ctx context.Context, bucket string) (os.FileInfo, error) {
bucketDir, err := fs.getBucketDir(ctx, bucket)
if err != nil {
return nil, err
}
st, err := fsStatVolume(bucketDir)
st, err := fsStatVolume(ctx, bucketDir)
if err != nil {
return nil, err
}
@@ -250,12 +226,12 @@ func (fs *FSObjects) MakeBucketWithLocation(ctx context.Context, bucket, locatio
return err
}
defer bucketLock.Unlock()
bucketDir, err := fs.getBucketDir(bucket)
bucketDir, err := fs.getBucketDir(ctx, bucket)
if err != nil {
return toObjectErr(err, bucket)
}
if err = fsMkdir(bucketDir); err != nil {
if err = fsMkdir(ctx, bucketDir); err != nil {
return toObjectErr(err, bucket)
}
@@ -269,7 +245,7 @@ func (fs *FSObjects) GetBucketInfo(ctx context.Context, bucket string) (bi Bucke
return bi, e
}
defer bucketLock.RUnlock()
st, err := fs.statBucketDir(bucket)
st, err := fs.statBucketDir(ctx, bucket)
if err != nil {
return bi, toObjectErr(err, bucket)
}
@@ -285,12 +261,14 @@ func (fs *FSObjects) GetBucketInfo(ctx context.Context, bucket string) (bi Bucke
// ListBuckets - list all s3 compatible buckets (directories) at fsPath.
func (fs *FSObjects) ListBuckets(ctx context.Context) ([]BucketInfo, error) {
if err := checkPathLength(fs.fsPath); err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
var bucketInfos []BucketInfo
entries, err := readDir((fs.fsPath))
if err != nil {
return nil, toObjectErr(errors.Trace(errDiskNotFound))
logger.LogIf(ctx, errDiskNotFound)
return nil, toObjectErr(errDiskNotFound)
}
for _, entry := range entries {
@@ -299,7 +277,7 @@ func (fs *FSObjects) ListBuckets(ctx context.Context) ([]BucketInfo, error) {
continue
}
var fi os.FileInfo
fi, err = fsStatVolume(pathJoin(fs.fsPath, entry))
fi, err = fsStatVolume(ctx, pathJoin(fs.fsPath, entry))
// There seems like no practical reason to check for errors
// at this point, if there are indeed errors we can simply
// just ignore such buckets and list only those which
@@ -327,27 +305,28 @@ func (fs *FSObjects) ListBuckets(ctx context.Context) ([]BucketInfo, error) {
func (fs *FSObjects) DeleteBucket(ctx context.Context, bucket string) error {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
if err := bucketLock.GetLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return err
}
defer bucketLock.Unlock()
bucketDir, err := fs.getBucketDir(bucket)
bucketDir, err := fs.getBucketDir(ctx, bucket)
if err != nil {
return toObjectErr(err, bucket)
}
// Attempt to delete regular bucket.
if err = fsRemoveDir(bucketDir); err != nil {
if err = fsRemoveDir(ctx, bucketDir); err != nil {
return toObjectErr(err, bucket)
}
// Cleanup all the bucket metadata.
minioMetadataBucketDir := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, bucket)
if err = fsRemoveAll(minioMetadataBucketDir); err != nil {
if err = fsRemoveAll(ctx, minioMetadataBucketDir); err != nil {
return toObjectErr(err, bucket)
}
// Delete all bucket metadata.
deleteBucketMetadata(bucket, fs)
deleteBucketMetadata(ctx, bucket, fs)
return nil
}
@@ -380,7 +359,7 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
}
defer objectSRLock.RUnlock()
}
if _, err := fs.statBucketDir(srcBucket); err != nil {
if _, err := fs.statBucketDir(ctx, srcBucket); err != nil {
return oi, toObjectErr(err, srcBucket)
}
@@ -391,14 +370,15 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
fsMetaPath := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, srcBucket, srcObject, fs.metaJSONFile)
wlk, err := fs.rwPool.Write(fsMetaPath)
if err != nil {
return oi, toObjectErr(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, srcBucket, srcObject)
}
// This close will allow for locks to be synchronized on `fs.json`.
defer wlk.Close()
// Save objects' metadata in `fs.json`.
fsMeta := newFSMetaV1()
if _, err = fsMeta.ReadFrom(wlk); err != nil {
if _, err = fsMeta.ReadFrom(ctx, wlk); err != nil {
return oi, toObjectErr(err, srcBucket, srcObject)
}
@@ -409,7 +389,7 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
}
// Stat the file to get file size.
fi, err := fsStatFile(pathJoin(fs.fsPath, srcBucket, srcObject))
fi, err := fsStatFile(ctx, pathJoin(fs.fsPath, srcBucket, srcObject))
if err != nil {
return oi, toObjectErr(err, srcBucket, srcObject)
}
@@ -419,20 +399,20 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
}
go func() {
if gerr := fs.getObject(srcBucket, srcObject, 0, srcInfo.Size, srcInfo.Writer, srcInfo.ETag, !cpSrcDstSame); gerr != nil {
if gerr := fs.getObject(ctx, srcBucket, srcObject, 0, srcInfo.Size, srcInfo.Writer, srcInfo.ETag, !cpSrcDstSame); gerr != nil {
if gerr = srcInfo.Writer.Close(); gerr != nil {
errorIf(gerr, "Unable to read the object %s/%s.", srcBucket, srcObject)
logger.LogIf(ctx, gerr)
}
return
}
// Close writer explicitly signalling we wrote all data.
if gerr := srcInfo.Writer.Close(); gerr != nil {
errorIf(gerr, "Unable to read the object %s/%s.", srcBucket, srcObject)
logger.LogIf(ctx, gerr)
return
}
}()
objInfo, err := fs.putObject(dstBucket, dstObject, srcInfo.Reader, srcInfo.UserDefined)
objInfo, err := fs.putObject(ctx, dstBucket, dstObject, srcInfo.Reader, srcInfo.UserDefined)
if err != nil {
return oi, toObjectErr(err, dstBucket, dstObject)
}
@@ -447,39 +427,43 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (fs *FSObjects) GetObject(ctx context.Context, bucket, object string, offset int64, length int64, writer io.Writer, etag string) (err error) {
if err = checkGetObjArgs(bucket, object); err != nil {
if err = checkGetObjArgs(ctx, bucket, object); err != nil {
return err
}
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return err
}
defer objectLock.RUnlock()
return fs.getObject(bucket, object, offset, length, writer, etag, true)
return fs.getObject(ctx, bucket, object, offset, length, writer, etag, true)
}
// getObject - wrapper for GetObject
func (fs *FSObjects) getObject(bucket, object string, offset int64, length int64, writer io.Writer, etag string, lock bool) (err error) {
if _, err = fs.statBucketDir(bucket); err != nil {
func (fs *FSObjects) getObject(ctx context.Context, bucket, object string, offset int64, length int64, writer io.Writer, etag string, lock bool) (err error) {
if _, err = fs.statBucketDir(ctx, bucket); err != nil {
return toObjectErr(err, bucket)
}
// Offset cannot be negative.
if offset < 0 {
return toObjectErr(errors.Trace(errUnexpected), bucket, object)
logger.LogIf(ctx, errUnexpected)
return toObjectErr(errUnexpected, bucket, object)
}
// Writer cannot be nil.
if writer == nil {
return toObjectErr(errors.Trace(errUnexpected), bucket, object)
logger.LogIf(ctx, errUnexpected)
return toObjectErr(errUnexpected, bucket, object)
}
// If its a directory request, we return an empty body.
if hasSuffix(object, slashSeparator) {
_, err = writer.Write([]byte(""))
return toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return toObjectErr(err, bucket, object)
}
if bucket != minioMetaBucket {
@@ -487,25 +471,27 @@ func (fs *FSObjects) getObject(bucket, object string, offset int64, length int64
if lock {
_, err = fs.rwPool.Open(fsMetaPath)
if err != nil && err != errFileNotFound {
return toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return toObjectErr(err, bucket, object)
}
defer fs.rwPool.Close(fsMetaPath)
}
}
if etag != "" {
objEtag, perr := fs.getObjectETag(bucket, object, lock)
objEtag, perr := fs.getObjectETag(ctx, bucket, object, lock)
if perr != nil {
return toObjectErr(errors.Trace(perr), bucket, object)
return toObjectErr(perr, bucket, object)
}
if objEtag != etag {
return toObjectErr(errors.Trace(InvalidETag{}), bucket, object)
logger.LogIf(ctx, InvalidETag{})
return toObjectErr(InvalidETag{}, bucket, object)
}
}
// Read the object, doesn't exist returns an s3 compatible error.
fsObjPath := pathJoin(fs.fsPath, bucket, object)
reader, size, err := fsOpenFile(fsObjPath, offset)
reader, size, err := fsOpenFile(ctx, fsObjPath, offset)
if err != nil {
return toObjectErr(err, bucket, object)
}
@@ -523,22 +509,24 @@ func (fs *FSObjects) getObject(bucket, object string, offset int64, length int64
// Reply back invalid range if the input offset and length fall out of range.
if offset > size || offset+length > size {
return errors.Trace(InvalidRange{offset, length, size})
err = InvalidRange{offset, length, size}
logger.LogIf(ctx, err)
return err
}
// Allocate a staging buffer.
buf := make([]byte, int(bufSize))
_, err = io.CopyBuffer(writer, io.LimitReader(reader, length), buf)
return toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return toObjectErr(err, bucket, object)
}
// getObjectInfo - wrapper for reading object metadata and constructs ObjectInfo.
func (fs *FSObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e error) {
func (fs *FSObjects) getObjectInfo(ctx context.Context, bucket, object string) (oi ObjectInfo, e error) {
fsMeta := fsMetaV1{}
fi, err := fsStatDir(pathJoin(fs.fsPath, bucket, object))
if err != nil && errors.Cause(err) != errFileAccessDenied {
fi, err := fsStatDir(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil && err != errFileAccessDenied {
return oi, toObjectErr(err, bucket, object)
}
if fi != nil {
@@ -547,6 +535,7 @@ func (fs *FSObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e erro
if hasSuffix(object, slashSeparator) {
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
logger.LogIf(ctx, errFileNotFound)
return oi, toObjectErr(errFileNotFound, bucket, object)
}
@@ -558,11 +547,11 @@ func (fs *FSObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e erro
if err == nil {
// Read from fs metadata only if it exists.
defer fs.rwPool.Close(fsMetaPath)
if _, rerr := fsMeta.ReadFrom(rlk.LockedFile); rerr != nil {
if _, rerr := fsMeta.ReadFrom(ctx, rlk.LockedFile); rerr != nil {
// `fs.json` can be empty due to previously failed
// PutObject() transaction, if we arrive at such
// a situation we just ignore and continue.
if errors.Cause(rerr) != io.EOF {
if rerr != io.EOF {
return oi, toObjectErr(rerr, bucket, object)
}
}
@@ -570,11 +559,12 @@ func (fs *FSObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e erro
// Ignore if `fs.json` is not available, this is true for pre-existing data.
if err != nil && err != errFileNotFound {
return oi, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, toObjectErr(err, bucket, object)
}
// Stat the file to get file size.
fi, err = fsStatFile(pathJoin(fs.fsPath, bucket, object))
fi, err = fsStatFile(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return oi, toObjectErr(err, bucket, object)
}
@@ -591,27 +581,27 @@ func (fs *FSObjects) GetObjectInfo(ctx context.Context, bucket, object string) (
}
defer objectLock.RUnlock()
if err := checkGetObjArgs(bucket, object); err != nil {
if err := checkGetObjArgs(ctx, bucket, object); err != nil {
return oi, err
}
if _, err := fs.statBucketDir(bucket); err != nil {
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return oi, toObjectErr(err, bucket)
}
return fs.getObjectInfo(bucket, object)
return fs.getObjectInfo(ctx, bucket, object)
}
// This function does the following check, suppose
// object is "a/b/c/d", stat makes sure that objects ""a/b/c""
// "a/b" and "a" do not exist.
func (fs *FSObjects) parentDirIsObject(bucket, parent string) bool {
func (fs *FSObjects) parentDirIsObject(ctx context.Context, bucket, parent string) bool {
var isParentDirObject func(string) bool
isParentDirObject = func(p string) bool {
if p == "." || p == "/" {
return false
}
if _, err := fsStatFile(pathJoin(fs.fsPath, bucket, p)); err == nil {
if _, err := fsStatFile(ctx, pathJoin(fs.fsPath, bucket, p)); err == nil {
// If there is already a file at prefix "p", return true.
return true
}
@@ -627,20 +617,21 @@ func (fs *FSObjects) parentDirIsObject(bucket, parent string) bool {
// Additionally writes `fs.json` which carries the necessary metadata
// for future object operations.
func (fs *FSObjects) PutObject(ctx context.Context, bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo ObjectInfo, retErr error) {
if err := checkPutObjectArgs(bucket, object, fs, data.Size()); err != nil {
if err := checkPutObjectArgs(ctx, bucket, object, fs, data.Size()); err != nil {
return ObjectInfo{}, err
}
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return objInfo, err
}
defer objectLock.Unlock()
return fs.putObject(bucket, object, data, metadata)
return fs.putObject(ctx, bucket, object, data, metadata)
}
// putObject - wrapper for PutObject
func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo ObjectInfo, retErr error) {
func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo ObjectInfo, retErr error) {
// No metadata is set, allocate a new one.
meta := make(map[string]string)
for k, v := range metadata {
@@ -649,7 +640,7 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
var err error
// Validate if bucket name is valid and exists.
if _, err = fs.statBucketDir(bucket); err != nil {
if _, err = fs.statBucketDir(ctx, bucket); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket)
}
@@ -661,31 +652,35 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
// and return success.
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errors.Trace(errFileAccessDenied), bucket, object)
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
logger.LogIf(ctx, errFileAccessDenied)
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
logger.LogIf(ctx, err)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
var fi os.FileInfo
if fi, err = fsStatDir(pathJoin(fs.fsPath, bucket, object)); err != nil {
if fi, err = fsStatDir(ctx, pathJoin(fs.fsPath, bucket, object)); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
if err = checkPutObjectArgs(bucket, object, fs, data.Size()); err != nil {
if err = checkPutObjectArgs(ctx, bucket, object, fs, data.Size()); err != nil {
return ObjectInfo{}, err
}
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errors.Trace(errFileAccessDenied), bucket, object)
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
logger.LogIf(ctx, errFileAccessDenied)
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
}
// Validate input data size and it can never be less than zero.
if data.Size() < 0 {
return ObjectInfo{}, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return ObjectInfo{}, errInvalidArgument
}
var wlk *lock.LockedFile
@@ -695,7 +690,8 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
fsMetaPath := pathJoin(bucketMetaDir, bucket, object, fs.metaJSONFile)
wlk, err = fs.rwPool.Create(fsMetaPath)
if err != nil {
return ObjectInfo{}, toObjectErr(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
// This close will allow for locks to be synchronized on `fs.json`.
defer wlk.Close()
@@ -703,7 +699,7 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
// Remove meta file when PutObject encounters any error
if retErr != nil {
tmpDir := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID)
fsRemoveMeta(bucketMetaDir, fsMetaPath, tmpDir)
fsRemoveMeta(ctx, bucketMetaDir, fsMetaPath, tmpDir)
}
}()
}
@@ -721,9 +717,9 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
buf := make([]byte, int(bufSize))
fsTmpObjPath := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID, tempObj)
bytesWritten, err := fsCreateFile(fsTmpObjPath, data, buf, data.Size())
bytesWritten, err := fsCreateFile(ctx, fsTmpObjPath, data, buf, data.Size())
if err != nil {
fsRemoveFile(fsTmpObjPath)
fsRemoveFile(ctx, fsTmpObjPath)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
@@ -732,24 +728,24 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
// Should return IncompleteBody{} error when reader has fewer
// bytes than specified in request header.
if bytesWritten < data.Size() {
fsRemoveFile(fsTmpObjPath)
return ObjectInfo{}, errors.Trace(IncompleteBody{})
fsRemoveFile(ctx, fsTmpObjPath)
return ObjectInfo{}, IncompleteBody{}
}
// Delete the temporary object in the case of a
// failure. If PutObject succeeds, then there would be
// nothing to delete.
defer fsRemoveFile(fsTmpObjPath)
defer fsRemoveFile(ctx, fsTmpObjPath)
// Entire object was written to the temp location, now it's safe to rename it to the actual location.
fsNSObjPath := pathJoin(fs.fsPath, bucket, object)
// Deny if WORM is enabled
if globalWORMEnabled {
if _, err = fsStatFile(fsNSObjPath); err == nil {
return ObjectInfo{}, errors.Trace(ObjectAlreadyExists{Bucket: bucket, Object: object})
if _, err = fsStatFile(ctx, fsNSObjPath); err == nil {
return ObjectInfo{}, ObjectAlreadyExists{Bucket: bucket, Object: object}
}
}
if err = fsRenameFile(fsTmpObjPath, fsNSObjPath); err != nil {
if err = fsRenameFile(ctx, fsTmpObjPath, fsNSObjPath); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
@@ -761,7 +757,7 @@ func (fs *FSObjects) putObject(bucket string, object string, data *hash.Reader,
}
// Stat the file to fetch timestamp, size.
fi, err := fsStatFile(pathJoin(fs.fsPath, bucket, object))
fi, err := fsStatFile(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
@@ -780,11 +776,11 @@ func (fs *FSObjects) DeleteObject(ctx context.Context, bucket, object string) er
}
defer objectLock.Unlock()
if err := checkDelObjArgs(bucket, object); err != nil {
if err := checkDelObjArgs(ctx, bucket, object); err != nil {
return err
}
if _, err := fs.statBucketDir(bucket); err != nil {
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return toObjectErr(err, bucket)
}
@@ -797,19 +793,20 @@ func (fs *FSObjects) DeleteObject(ctx context.Context, bucket, object string) er
defer rwlk.Close()
}
if lerr != nil && lerr != errFileNotFound {
return toObjectErr(errors.Trace(lerr), bucket, object)
logger.LogIf(ctx, lerr)
return toObjectErr(lerr, bucket, object)
}
}
// Delete the object.
if err := fsDeleteFile(pathJoin(fs.fsPath, bucket), pathJoin(fs.fsPath, bucket, object)); err != nil {
if err := fsDeleteFile(ctx, pathJoin(fs.fsPath, bucket), pathJoin(fs.fsPath, bucket, object)); err != nil {
return toObjectErr(err, bucket, object)
}
if bucket != minioMetaBucket {
// Delete the metadata object.
err := fsDeleteFile(minioMetaBucketDir, fsMetaPath)
if err != nil && errors.Cause(err) != errFileNotFound {
err := fsDeleteFile(ctx, minioMetaBucketDir, fsMetaPath)
if err != nil && err != errFileNotFound {
return toObjectErr(err, bucket, object)
}
}
@@ -836,7 +833,7 @@ func (fs *FSObjects) listDirFactory(isLeaf isLeafFunc) listDirFunc {
// getObjectETag is a helper function, which returns only the md5sum
// of the file on the disk.
func (fs *FSObjects) getObjectETag(bucket, entry string, lock bool) (string, error) {
func (fs *FSObjects) getObjectETag(ctx context.Context, bucket, entry string, lock bool) (string, error) {
fsMetaPath := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, bucket, entry, fs.metaJSONFile)
var reader io.Reader
@@ -848,7 +845,8 @@ func (fs *FSObjects) getObjectETag(bucket, entry string, lock bool) (string, err
rlk, err := fs.rwPool.Open(fsMetaPath)
// Ignore if `fs.json` is not available, this is true for pre-existing data.
if err != nil && err != errFileNotFound {
return "", toObjectErr(errors.Trace(err), bucket, entry)
logger.LogIf(ctx, err)
return "", toObjectErr(err, bucket, entry)
}
// If file is not found, we don't need to proceed forward.
@@ -862,16 +860,17 @@ func (fs *FSObjects) getObjectETag(bucket, entry string, lock bool) (string, err
// Fetch the size of the underlying file.
fi, err = rlk.LockedFile.Stat()
if err != nil {
return "", toObjectErr(errors.Trace(err), bucket, entry)
logger.LogIf(ctx, err)
return "", toObjectErr(err, bucket, entry)
}
size = fi.Size()
reader = io.NewSectionReader(rlk.LockedFile, 0, fi.Size())
} else {
var err error
reader, size, err = fsOpenFile(fsMetaPath, 0)
reader, size, err = fsOpenFile(ctx, fsMetaPath, 0)
if err != nil {
return "", toObjectErr(errors.Trace(err), bucket, entry)
return "", toObjectErr(err, bucket, entry)
}
}
@@ -884,12 +883,14 @@ func (fs *FSObjects) getObjectETag(bucket, entry string, lock bool) (string, err
fsMetaBuf, err := ioutil.ReadAll(reader)
if err != nil {
return "", toObjectErr(errors.Trace(err), bucket, entry)
logger.LogIf(ctx, err)
return "", toObjectErr(err, bucket, entry)
}
// Check if FS metadata is valid, if not return error.
if !isFSMetaValid(parseFSVersion(fsMetaBuf)) {
return "", toObjectErr(errors.Trace(errCorruptedFormat), bucket, entry)
logger.LogIf(ctx, errCorruptedFormat)
return "", toObjectErr(errCorruptedFormat, bucket, entry)
}
return extractETag(parseFSMetaMap(fsMetaBuf)), nil
@@ -898,7 +899,7 @@ func (fs *FSObjects) getObjectETag(bucket, entry string, lock bool) (string, err
// ListObjects - list all objects at prefix upto maxKeys., optionally delimited by '/'. Maintains the list pool
// state for future re-entrant list requests.
func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
if err := checkListObjsArgs(bucket, prefix, marker, delimiter, fs); err != nil {
if err := checkListObjsArgs(ctx, bucket, prefix, marker, delimiter, fs); err != nil {
return loi, err
}
// Marker is set validate pre-condition.
@@ -908,7 +909,7 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
return ListObjectsInfo{}, e
}
}
if _, err := fs.statBucketDir(bucket); err != nil {
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return loi, err
}
@@ -942,10 +943,11 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
// Protect the entry from concurrent deletes, or renames.
objectLock := fs.nsMutex.NewNSLock(bucket, entry)
if err = objectLock.GetRLock(globalListingTimeout); err != nil {
logger.LogIf(ctx, err)
return ObjectInfo{}, err
}
defer objectLock.RUnlock()
return fs.getObjectInfo(bucket, entry)
return fs.getObjectInfo(ctx, bucket, entry)
}
heal := false // true only for xl.ListObjectsHeal()
@@ -959,7 +961,7 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
return !hasSuffix(object, slashSeparator)
}
listDir := fs.listDirFactory(isLeaf)
walkResultCh = startTreeWalk(bucket, prefix, marker, recursive, listDir, isLeaf, endWalkCh)
walkResultCh = startTreeWalk(ctx, bucket, prefix, marker, recursive, listDir, isLeaf, endWalkCh)
}
var objInfos []ObjectInfo
@@ -977,14 +979,13 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
// For any walk error return right away.
if walkResult.err != nil {
// File not found is a valid case.
if errors.Cause(walkResult.err) == errFileNotFound {
if walkResult.err == errFileNotFound {
return loi, nil
}
return loi, toObjectErr(walkResult.err, bucket, prefix)
}
objInfo, err := entryToObjectInfo(walkResult.entry)
if err != nil {
errorIf(err, "Unable to fetch object info for %s", walkResult.entry)
return loi, nil
}
nextMarker = objInfo.Name
@@ -1016,36 +1017,47 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
return result, nil
}
// ReloadFormat - no-op for fs, Valid only for XL.
func (fs *FSObjects) ReloadFormat(ctx context.Context, dryRun bool) error {
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// HealFormat - no-op for fs, Valid only for XL.
func (fs *FSObjects) HealFormat(ctx context.Context, dryRun bool) (madmin.HealResultItem, error) {
return madmin.HealResultItem{}, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return madmin.HealResultItem{}, NotImplemented{}
}
// HealObject - no-op for fs. Valid only for XL.
func (fs *FSObjects) HealObject(ctx context.Context, bucket, object string, dryRun bool) (
res madmin.HealResultItem, err error) {
return res, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return res, NotImplemented{}
}
// HealBucket - no-op for fs, Valid only for XL.
func (fs *FSObjects) HealBucket(ctx context.Context, bucket string, dryRun bool) ([]madmin.HealResultItem,
error) {
return nil, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return nil, NotImplemented{}
}
// ListObjectsHeal - list all objects to be healed. Valid only for XL
func (fs *FSObjects) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
return loi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return loi, NotImplemented{}
}
// ListBucketsHeal - list all buckets to be healed. Valid only for XL
func (fs *FSObjects) ListBucketsHeal(ctx context.Context) ([]BucketInfo, error) {
return []BucketInfo{}, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return []BucketInfo{}, NotImplemented{}
}
// SetBucketPolicy sets policy on bucket
func (fs *FSObjects) SetBucketPolicy(ctx context.Context, bucket string, policy policy.BucketAccessPolicy) error {
return persistAndNotifyBucketPolicyChange(bucket, false, policy, fs)
return persistAndNotifyBucketPolicyChange(ctx, bucket, false, policy, fs)
}
// GetBucketPolicy will get policy on bucket
@@ -1059,7 +1071,7 @@ func (fs *FSObjects) GetBucketPolicy(ctx context.Context, bucket string) (policy
// DeleteBucketPolicy deletes all policies on bucket
func (fs *FSObjects) DeleteBucketPolicy(ctx context.Context, bucket string) error {
return persistAndNotifyBucketPolicyChange(bucket, true, emptyBucketPolicy, fs)
return persistAndNotifyBucketPolicyChange(ctx, bucket, true, emptyBucketPolicy, fs)
}
// ListObjectsV2 lists all blobs in bucket filtered by prefix
+19 -21
View File
@@ -23,8 +23,6 @@ import (
"os"
"path/filepath"
"testing"
"github.com/minio/minio/pkg/errors"
)
// Tests for if parent directory is object
@@ -91,7 +89,7 @@ func TestFSParentDirIsObject(t *testing.T) {
},
}
for i, testCase := range testCases {
gotValue := fs.parentDirIsObject(bucketName, testCase.objectName)
gotValue := fs.parentDirIsObject(context.Background(), bucketName, testCase.objectName)
if testCase.parentIsObject != gotValue {
t.Errorf("Test %d: Unexpected value returned got %t, expected %t", i+1, gotValue, testCase.parentIsObject)
}
@@ -181,7 +179,7 @@ func TestFSGetBucketInfo(t *testing.T) {
// Test with inexistant bucket
_, err = fs.GetBucketInfo(context.Background(), "a")
if !isSameType(errors.Cause(err), BucketNameInvalid{}) {
if !isSameType(err, BucketNameInvalid{}) {
t.Fatal("BucketNameInvalid error not returned")
}
@@ -189,7 +187,7 @@ func TestFSGetBucketInfo(t *testing.T) {
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
_, err = fs.GetBucketInfo(context.Background(), bucketName)
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("BucketNotFound error not returned")
}
}
@@ -212,7 +210,7 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, bucket doesn't exist")
}
if _, ok := errors.Cause(err).(BucketNotFound); !ok {
if _, ok := err.(BucketNotFound); !ok {
t.Fatalf("Expected error type BucketNotFound, got %#v", err)
}
@@ -221,7 +219,7 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, bucket doesn't exist")
}
if _, ok := errors.Cause(err).(BucketNotFound); !ok {
if _, ok := err.(BucketNotFound); !ok {
t.Fatalf("Expected error type BucketNotFound, got %#v", err)
}
@@ -233,7 +231,7 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, backend corruption occurred")
}
if nerr, ok := errors.Cause(err).(PrefixAccessDenied); !ok {
if nerr, ok := err.(PrefixAccessDenied); !ok {
t.Fatalf("Expected PrefixAccessDenied, got %#v", err)
} else {
if nerr.Bucket != "bucket" {
@@ -248,7 +246,7 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, backned corruption occurred")
}
if nerr, ok := errors.Cause(err).(PrefixAccessDenied); !ok {
if nerr, ok := err.(PrefixAccessDenied); !ok {
t.Fatalf("Expected PrefixAccessDenied, got %#v", err)
} else {
if nerr.Bucket != "bucket" {
@@ -275,19 +273,19 @@ func TestFSDeleteObject(t *testing.T) {
obj.PutObject(context.Background(), bucketName, objectName, mustGetHashReader(t, bytes.NewReader([]byte("abcd")), int64(len("abcd")), "", ""), nil)
// Test with invalid bucket name
if err := fs.DeleteObject(context.Background(), "fo", objectName); !isSameType(errors.Cause(err), BucketNameInvalid{}) {
if err := fs.DeleteObject(context.Background(), "fo", objectName); !isSameType(err, BucketNameInvalid{}) {
t.Fatal("Unexpected error: ", err)
}
// Test with bucket does not exist
if err := fs.DeleteObject(context.Background(), "foobucket", "fooobject"); !isSameType(errors.Cause(err), BucketNotFound{}) {
if err := fs.DeleteObject(context.Background(), "foobucket", "fooobject"); !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error: ", err)
}
// Test with invalid object name
if err := fs.DeleteObject(context.Background(), bucketName, "\\"); !isSameType(errors.Cause(err), ObjectNameInvalid{}) {
if err := fs.DeleteObject(context.Background(), bucketName, "\\"); !isSameType(err, ObjectNameInvalid{}) {
t.Fatal("Unexpected error: ", err)
}
// Test with object does not exist.
if err := fs.DeleteObject(context.Background(), bucketName, "foooobject"); !isSameType(errors.Cause(err), ObjectNotFound{}) {
if err := fs.DeleteObject(context.Background(), bucketName, "foooobject"); !isSameType(err, ObjectNotFound{}) {
t.Fatal("Unexpected error: ", err)
}
// Test with valid condition
@@ -298,7 +296,7 @@ func TestFSDeleteObject(t *testing.T) {
// Delete object should err disk not found.
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if err := fs.DeleteObject(context.Background(), bucketName, objectName); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error: ", err)
}
}
@@ -321,11 +319,11 @@ func TestFSDeleteBucket(t *testing.T) {
}
// Test with an invalid bucket name
if err = fs.DeleteBucket(context.Background(), "fo"); !isSameType(errors.Cause(err), BucketNameInvalid{}) {
if err = fs.DeleteBucket(context.Background(), "fo"); !isSameType(err, BucketNameInvalid{}) {
t.Fatal("Unexpected error: ", err)
}
// Test with an inexistant bucket
if err = fs.DeleteBucket(context.Background(), "foobucket"); !isSameType(errors.Cause(err), BucketNotFound{}) {
if err = fs.DeleteBucket(context.Background(), "foobucket"); !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error: ", err)
}
// Test with a valid case
@@ -338,7 +336,7 @@ func TestFSDeleteBucket(t *testing.T) {
// Delete bucket should get error disk not found.
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if err = fs.DeleteBucket(context.Background(), bucketName); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
if !isSameType(err, BucketNotFound{}) {
t.Fatal("Unexpected error: ", err)
}
}
@@ -381,7 +379,7 @@ func TestFSListBuckets(t *testing.T) {
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if _, err := fs.ListBuckets(context.Background()); err != nil {
if errors.Cause(err) != errDiskNotFound {
if err != errDiskNotFound {
t.Fatal("Unexpected error: ", err)
}
}
@@ -389,7 +387,7 @@ func TestFSListBuckets(t *testing.T) {
longPath := fmt.Sprintf("%0256d", 1)
fs.fsPath = longPath
if _, err := fs.ListBuckets(context.Background()); err != nil {
if errors.Cause(err) != errFileNameTooLong {
if err != errFileNameTooLong {
t.Fatal("Unexpected error: ", err)
}
}
@@ -402,7 +400,7 @@ func TestFSHealObject(t *testing.T) {
obj := initFSObjects(disk, t)
_, err := obj.HealObject(context.Background(), "bucket", "object", false)
if err == nil || !isSameType(errors.Cause(err), NotImplemented{}) {
if err == nil || !isSameType(err, NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
}
@@ -414,7 +412,7 @@ func TestFSListObjectsHeal(t *testing.T) {
obj := initFSObjects(disk, t)
_, err := obj.ListObjectsHeal(context.Background(), "bucket", "prefix", "marker", "delimiter", 1000)
if err == nil || !isSameType(errors.Cause(err), NotImplemented{}) {
if err == nil || !isSameType(err, NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
}
+3 -22
View File
@@ -19,7 +19,6 @@ package cmd
import (
"net/http"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
minio "github.com/minio/minio-go"
@@ -31,12 +30,6 @@ var (
// MustGetUUID function alias.
MustGetUUID = mustGetUUID
// ErrorIf provides errorIf function alias.
ErrorIf = errorIf
// FatalIf provides fatalIf function alias.
FatalIf = fatalIf
)
// AnonErrToObjectErr - converts standard http codes into meaningful object layer errors.
@@ -262,16 +255,6 @@ func ErrorRespToObjectError(err error, params ...string) error {
return nil
}
e, ok := err.(*errors.Error)
if !ok {
// Code should be fixed if this function is called without doing traceError()
// Else handling different situations in this function makes this function complicated.
errorIf(err, "Expected type *Error")
return err
}
err = e.Cause
bucket := ""
object := ""
if len(params) >= 1 {
@@ -282,15 +265,14 @@ func ErrorRespToObjectError(err error, params ...string) error {
}
if isNetworkOrHostDown(err) {
e.Cause = BackendDown{}
return e
return BackendDown{}
}
minioErr, ok := err.(minio.ErrorResponse)
if !ok {
// We don't interpret non Minio errors. As minio errors will
// have StatusCode to help to convert to object errors.
return e
return err
}
switch minioErr.Code {
@@ -325,6 +307,5 @@ func ErrorRespToObjectError(err error, params ...string) error {
err = PartTooSmall{}
}
e.Cause = err
return e
return err
}
+27 -45
View File
@@ -17,6 +17,8 @@
package cmd
import (
"context"
"errors"
"fmt"
"net/url"
"os"
@@ -27,8 +29,8 @@ import (
"github.com/gorilla/mux"
"github.com/minio/cli"
"github.com/minio/minio/pkg/errors"
miniohttp "github.com/minio/minio/pkg/http"
miniohttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
)
var (
@@ -100,10 +102,14 @@ func ValidateGatewayArguments(serverAddr, endpointAddr string) error {
return nil
}
func init() {
logger.Init(GOPATH)
}
// StartGateway - handler for 'minio gateway <name>'.
func StartGateway(ctx *cli.Context, gw Gateway) {
if gw == nil {
fatalIf(errUnexpected, "Gateway implementation not initialized, exiting.")
logger.FatalIf(errUnexpected, "Gateway implementation not initialized, exiting.")
}
// Validate if we have access, secret set through environment.
@@ -116,13 +122,13 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// enable json and quite modes if jason flag is turned on.
jsonFlag := ctx.IsSet("json") || ctx.GlobalIsSet("json")
if jsonFlag {
log.EnableJSON()
logger.EnableJSON()
}
// Get quiet flag from command line argument.
quietFlag := ctx.IsSet("quiet") || ctx.GlobalIsSet("quiet")
if quietFlag {
log.EnableQuiet()
logger.EnableQuiet()
}
// Fetch address option
@@ -139,73 +145,49 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// Validate if we have access, secret set through environment.
if !globalIsEnvCreds {
errorIf(fmt.Errorf("Access and secret keys not set"), "Access and Secret keys should be set through ENVs for backend [%s]", gatewayName)
reqInfo := (&logger.ReqInfo{}).AppendTags("gatewayName", gatewayName)
contxt := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(contxt, errors.New("Access and Secret keys should be set through ENVs for backend"))
cli.ShowCommandHelpAndExit(ctx, gatewayName, 1)
}
// Create certs path.
fatalIf(createConfigDir(), "Unable to create configuration directories.")
logger.FatalIf(createConfigDir(), "Unable to create configuration directories.")
// Initialize gateway config.
initConfig()
// Init the error tracing module.
errors.Init(GOPATH, "github.com/minio/minio")
// Check and load SSL certificates.
var err error
globalPublicCerts, globalRootCAs, globalTLSCertificate, globalIsSSL, err = getSSLConfig()
fatalIf(err, "Invalid SSL certificate file")
logger.FatalIf(err, "Invalid SSL certificate file")
// Set system resources to maximum.
errorIf(setMaxResources(), "Unable to change resource limit")
logger.LogIf(context.Background(), setMaxResources())
initNSLock(false) // Enable local namespace lock.
// Initialize notification system.
globalNotificationSys, err = NewNotificationSys(globalServerConfig, EndpointList{})
fatalIf(err, "Unable to initialize notification system.")
logger.FatalIf(err, "Unable to initialize notification system.")
newObject, err := gw.NewGatewayLayer(globalServerConfig.GetCredential())
fatalIf(err, "Unable to initialize gateway layer")
logger.FatalIf(err, "Unable to initialize gateway layer")
router := mux.NewRouter().SkipClean(true)
// Add healthcheck router
registerHealthCheckRouter(router)
// Register web router when its enabled.
if globalIsBrowserEnabled {
fatalIf(registerWebRouter(router), "Unable to configure web browser")
logger.FatalIf(registerWebRouter(router), "Unable to configure web browser")
}
// Add API router.
registerAPIRouter(router)
var handlerFns = []HandlerFunc{
// Validate all the incoming paths.
setPathValidityHandler,
// Limits all requests size to a maximum fixed limit
setRequestSizeLimitHandler,
// Adds 'crossdomain.xml' policy handler to serve legacy flash clients.
setCrossDomainPolicy,
// Validates all incoming requests to have a valid date header.
// Redirect some pre-defined browser request paths to a static location prefix.
setBrowserRedirectHandler,
// Validates if incoming request is for restricted buckets.
setReservedBucketHandler,
// Adds cache control for all browser requests.
setBrowserCacheControlHandler,
// Validates all incoming requests to have a valid date header.
setTimeValidityHandler,
// CORS setting for all browser API requests.
setCorsHandler,
// Validates all incoming URL resources, for invalid/unsupported
// resources client receives a HTTP error.
setIgnoreResourcesHandler,
// Auth handler verifies incoming authorization headers and
// routes them accordingly. Client receives a HTTP error for
// invalid/unsupported signatures.
setAuthHandler,
// Add new handlers here.
}
globalHTTPServer = miniohttp.NewServer([]string{gatewayAddr}, registerHandlers(router, handlerFns...), globalTLSCertificate)
globalHTTPServer = miniohttp.NewServer([]string{gatewayAddr}, registerHandlers(router, globalHandlers...), globalTLSCertificate)
// Start server, automatically configures TLS if certs are available.
go func() {
@@ -227,7 +209,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// Print a warning message if gateway is not ready for production before the startup banner.
if !gw.Production() {
log.Println(colorYellow("\n *** Warning: Not Ready for Production ***"))
logger.StartupMessage(colorYellow(" *** Warning: Not Ready for Production ***"))
}
// Print gateway startup message.
+7 -5
View File
@@ -20,6 +20,8 @@ import (
"context"
"fmt"
"strings"
"github.com/minio/minio/cmd/logger"
)
// Prints the formatted startup message.
@@ -54,12 +56,12 @@ func printGatewayCommonMsg(apiEndpoints []string) {
apiEndpointStr := strings.Join(apiEndpoints, " ")
// Colorize the message and print.
log.Println(colorBlue("\nEndpoint: ") + colorBold(fmt.Sprintf(getFormatStr(len(apiEndpointStr), 1), apiEndpointStr)))
log.Println(colorBlue("AccessKey: ") + colorBold(fmt.Sprintf("%s ", cred.AccessKey)))
log.Println(colorBlue("SecretKey: ") + colorBold(fmt.Sprintf("%s ", cred.SecretKey)))
logger.StartupMessage(colorBlue("\nEndpoint: ") + colorBold(fmt.Sprintf(getFormatStr(len(apiEndpointStr), 1), apiEndpointStr)))
logger.StartupMessage(colorBlue("AccessKey: ") + colorBold(fmt.Sprintf("%s ", cred.AccessKey)))
logger.StartupMessage(colorBlue("SecretKey: ") + colorBold(fmt.Sprintf("%s ", cred.SecretKey)))
if globalIsBrowserEnabled {
log.Println(colorBlue("\nBrowser Access:"))
log.Println(fmt.Sprintf(getFormatStr(len(apiEndpointStr), 3), apiEndpointStr))
logger.StartupMessage(colorBlue("\nBrowser Access:"))
logger.StartupMessage(fmt.Sprintf(getFormatStr(len(apiEndpointStr), 3), apiEndpointStr))
}
}
+46 -21
View File
@@ -21,7 +21,7 @@ import (
"time"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/madmin"
)
@@ -31,105 +31,130 @@ type GatewayUnsupported struct{}
// ListMultipartUploads lists all multipart uploads.
func (a GatewayUnsupported) ListMultipartUploads(ctx context.Context, bucket string, prefix string, keyMarker string, uploadIDMarker string, delimiter string, maxUploads int) (lmi ListMultipartsInfo, err error) {
return lmi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return lmi, NotImplemented{}
}
// NewMultipartUpload upload object in multiple parts
func (a GatewayUnsupported) NewMultipartUpload(ctx context.Context, bucket string, object string, metadata map[string]string) (uploadID string, err error) {
return "", errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return "", NotImplemented{}
}
// CopyObjectPart copy part of object to uploadID for another object
func (a GatewayUnsupported) CopyObjectPart(ctx context.Context, srcBucket, srcObject, destBucket, destObject, uploadID string, partID int, startOffset, length int64, srcInfo ObjectInfo) (pi PartInfo, err error) {
return pi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return pi, NotImplemented{}
}
// PutObjectPart puts a part of object in bucket
func (a GatewayUnsupported) PutObjectPart(ctx context.Context, bucket string, object string, uploadID string, partID int, data *hash.Reader) (pi PartInfo, err error) {
return pi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return pi, NotImplemented{}
}
// ListObjectParts returns all object parts for specified object in specified bucket
func (a GatewayUnsupported) ListObjectParts(ctx context.Context, bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (lpi ListPartsInfo, err error) {
return lpi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return lpi, NotImplemented{}
}
// AbortMultipartUpload aborts a ongoing multipart upload
func (a GatewayUnsupported) AbortMultipartUpload(ctx context.Context, bucket string, object string, uploadID string) error {
return errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// CompleteMultipartUpload completes ongoing multipart upload and finalizes object
func (a GatewayUnsupported) CompleteMultipartUpload(ctx context.Context, bucket string, object string, uploadID string, uploadedParts []CompletePart) (oi ObjectInfo, err error) {
return oi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return oi, NotImplemented{}
}
// SetBucketPolicy sets policy on bucket
func (a GatewayUnsupported) SetBucketPolicy(ctx context.Context, bucket string, policyInfo policy.BucketAccessPolicy) error {
return errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// GetBucketPolicy will get policy on bucket
func (a GatewayUnsupported) GetBucketPolicy(ctx context.Context, bucket string) (bal policy.BucketAccessPolicy, err error) {
return bal, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return bal, NotImplemented{}
}
// DeleteBucketPolicy deletes all policies on bucket
func (a GatewayUnsupported) DeleteBucketPolicy(ctx context.Context, bucket string) error {
return errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// ReloadFormat - Not implemented stub.
func (a GatewayUnsupported) ReloadFormat(ctx context.Context, dryRun bool) error {
return NotImplemented{}
}
// HealFormat - Not implemented stub
func (a GatewayUnsupported) HealFormat(ctx context.Context, dryRun bool) (madmin.HealResultItem, error) {
return madmin.HealResultItem{}, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return madmin.HealResultItem{}, NotImplemented{}
}
// HealBucket - Not implemented stub
func (a GatewayUnsupported) HealBucket(ctx context.Context, bucket string, dryRun bool) ([]madmin.HealResultItem, error) {
return nil, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return nil, NotImplemented{}
}
// ListBucketsHeal - Not implemented stub
func (a GatewayUnsupported) ListBucketsHeal(ctx context.Context) (buckets []BucketInfo, err error) {
return nil, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return nil, NotImplemented{}
}
// HealObject - Not implemented stub
func (a GatewayUnsupported) HealObject(ctx context.Context, bucket, object string, dryRun bool) (h madmin.HealResultItem, e error) {
return h, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return h, NotImplemented{}
}
// ListObjectsV2 - Not implemented stub
func (a GatewayUnsupported) ListObjectsV2(ctx context.Context, bucket, prefix, continuationToken, delimiter string, maxKeys int, fetchOwner bool, startAfter string) (result ListObjectsV2Info, err error) {
return result, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return result, NotImplemented{}
}
// ListObjectsHeal - Not implemented stub
func (a GatewayUnsupported) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
return loi, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return loi, NotImplemented{}
}
// CopyObject copies a blob from source container to destination container.
func (a GatewayUnsupported) CopyObject(ctx context.Context, srcBucket string, srcObject string, destBucket string, destObject string,
srcInfo ObjectInfo) (objInfo ObjectInfo, err error) {
return objInfo, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return objInfo, NotImplemented{}
}
// Locking operations
// ListLocks lists namespace locks held in object layer
func (a GatewayUnsupported) ListLocks(ctx context.Context, bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
return []VolumeLockInfo{}, errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return []VolumeLockInfo{}, NotImplemented{}
}
// ClearLocks clears namespace locks held in object layer
func (a GatewayUnsupported) ClearLocks(ctx context.Context, info []VolumeLockInfo) error {
return errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// RefreshBucketPolicy refreshes cache policy with what's on disk.
func (a GatewayUnsupported) RefreshBucketPolicy(ctx context.Context, bucket string) error {
return errors.Trace(NotImplemented{})
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// IsNotificationSupported returns whether bucket notification is applicable for this layer.
+117 -74
View File
@@ -35,8 +35,8 @@ import (
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
sha256 "github.com/minio/sha256-simd"
@@ -73,7 +73,7 @@ ENVIRONMENT VARIABLES:
MINIO_BROWSER: To disable web browser access, set this value to "off".
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -97,7 +97,7 @@ EXAMPLES:
3. Start minio gateway server for Azure Blob Storage backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=azureaccountname
$ export MINIO_SECRET_KEY=azureaccountkey
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}}
@@ -118,7 +118,7 @@ func azureGatewayMain(ctx *cli.Context) {
// Validate gateway arguments.
host := ctx.Args().First()
// Validate gateway arguments.
minio.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
logger.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
minio.StartGateway(ctx, &Azure{host})
}
@@ -181,11 +181,12 @@ func (g *Azure) Production() bool {
// copied into BlobProperties.
//
// Header names are canonicalized as in http.Header.
func s3MetaToAzureProperties(s3Metadata map[string]string) (storage.BlobMetadata,
func s3MetaToAzureProperties(ctx context.Context, s3Metadata map[string]string) (storage.BlobMetadata,
storage.BlobProperties, error) {
for k := range s3Metadata {
if strings.Contains(k, "--") {
return storage.BlobMetadata{}, storage.BlobProperties{}, errors.Trace(minio.UnsupportedMetadata{})
logger.LogIf(ctx, minio.UnsupportedMetadata{})
return storage.BlobMetadata{}, storage.BlobProperties{}, minio.UnsupportedMetadata{}
}
}
@@ -300,15 +301,6 @@ func azureToObjectError(err error, params ...string) error {
return nil
}
e, ok := err.(*errors.Error)
if !ok {
// Code should be fixed if this function is called without doing errors.Trace()
// Else handling different situations in this function makes this function complicated.
minio.ErrorIf(err, "Expected type *Error")
return err
}
err = e.Cause
bucket := ""
object := ""
if len(params) >= 1 {
@@ -322,7 +314,7 @@ func azureToObjectError(err error, params ...string) error {
if !ok {
// We don't interpret non Azure errors. As azure errors will
// have StatusCode to help to convert to object errors.
return e
return err
}
switch azureErr.Code {
@@ -349,8 +341,7 @@ func azureToObjectError(err error, params ...string) error {
err = minio.BucketNameInvalid{Bucket: bucket}
}
}
e.Cause = err
return e
return err
}
// mustGetAzureUploadID - returns new upload ID which is hex encoded 8 bytes random value.
@@ -371,17 +362,23 @@ func mustGetAzureUploadID() string {
}
// checkAzureUploadID - returns error in case of given string is upload ID.
func checkAzureUploadID(uploadID string) (err error) {
func checkAzureUploadID(ctx context.Context, uploadID string) (err error) {
if len(uploadID) != 16 {
return errors.Trace(minio.MalformedUploadID{
logger.LogIf(ctx, minio.MalformedUploadID{
UploadID: uploadID,
})
return minio.MalformedUploadID{
UploadID: uploadID,
}
}
if _, err = hex.DecodeString(uploadID); err != nil {
return errors.Trace(minio.MalformedUploadID{
logger.LogIf(ctx, minio.MalformedUploadID{
UploadID: uploadID,
})
return minio.MalformedUploadID{
UploadID: uploadID,
}
}
return nil
@@ -438,7 +435,8 @@ func (a *azureObjects) MakeBucketWithLocation(ctx context.Context, bucket, locat
err := container.Create(&storage.CreateContainerOptions{
Access: storage.ContainerAccessTypePrivate,
})
return azureToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return azureToObjectError(err, bucket)
}
// GetBucketInfo - Get bucket metadata..
@@ -448,7 +446,8 @@ func (a *azureObjects) GetBucketInfo(ctx context.Context, bucket string) (bi min
// in azure documentation, so we will simply use the same function here.
// Ref - https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata
if !minio.IsValidBucketName(bucket) {
return bi, errors.Trace(minio.BucketNameInvalid{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNameInvalid{Bucket: bucket})
return bi, minio.BucketNameInvalid{Bucket: bucket}
}
// Azure does not have an equivalent call, hence use
@@ -457,7 +456,8 @@ func (a *azureObjects) GetBucketInfo(ctx context.Context, bucket string) (bi min
Prefix: bucket,
})
if err != nil {
return bi, azureToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return bi, azureToObjectError(err, bucket)
}
for _, container := range resp.Containers {
if container.Name == bucket {
@@ -470,19 +470,22 @@ func (a *azureObjects) GetBucketInfo(ctx context.Context, bucket string) (bi min
} // else continue
}
}
return bi, errors.Trace(minio.BucketNotFound{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNotFound{Bucket: bucket})
return bi, minio.BucketNotFound{Bucket: bucket}
}
// ListBuckets - Lists all azure containers, uses Azure equivalent ListContainers.
func (a *azureObjects) ListBuckets(ctx context.Context) (buckets []minio.BucketInfo, err error) {
resp, err := a.client.ListContainers(storage.ListContainersParameters{})
if err != nil {
return nil, azureToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return nil, azureToObjectError(err)
}
for _, container := range resp.Containers {
t, e := time.Parse(time.RFC1123, container.Properties.LastModified)
if e != nil {
return nil, errors.Trace(e)
logger.LogIf(ctx, e)
return nil, e
}
buckets = append(buckets, minio.BucketInfo{
Name: container.Name,
@@ -495,7 +498,9 @@ func (a *azureObjects) ListBuckets(ctx context.Context) (buckets []minio.BucketI
// DeleteBucket - delete a container on azure, uses Azure equivalent DeleteContainer.
func (a *azureObjects) DeleteBucket(ctx context.Context, bucket string) error {
container := a.client.GetContainerReference(bucket)
return azureToObjectError(errors.Trace(container.Delete(nil)), bucket)
err := container.Delete(nil)
logger.LogIf(ctx, err)
return azureToObjectError(err, bucket)
}
// ListObjects - lists all blobs on azure with in a container filtered by prefix
@@ -512,7 +517,8 @@ func (a *azureObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
MaxResults: uint(maxKeys),
})
if err != nil {
return result, azureToObjectError(errors.Trace(err), bucket, prefix)
logger.LogIf(ctx, err)
return result, azureToObjectError(err, bucket, prefix)
}
for _, object := range resp.Blobs {
@@ -580,7 +586,8 @@ func (a *azureObjects) ListObjectsV2(ctx context.Context, bucket, prefix, contin
func (a *azureObjects) GetObject(ctx context.Context, bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
// startOffset cannot be negative.
if startOffset < 0 {
return azureToObjectError(errors.Trace(minio.InvalidRange{}), bucket, object)
logger.LogIf(ctx, minio.InvalidRange{})
return azureToObjectError(minio.InvalidRange{}, bucket, object)
}
blobRange := &storage.BlobRange{Start: uint64(startOffset)}
@@ -599,11 +606,13 @@ func (a *azureObjects) GetObject(ctx context.Context, bucket, object string, sta
})
}
if err != nil {
return azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return azureToObjectError(err, bucket, object)
}
_, err = io.Copy(writer, rc)
rc.Close()
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
// GetObjectInfo - reads blob metadata properties and replies back minio.ObjectInfo,
@@ -612,7 +621,8 @@ func (a *azureObjects) GetObjectInfo(ctx context.Context, bucket, object string)
blob := a.client.GetContainerReference(bucket).GetBlobReference(object)
err = blob.GetProperties(nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
return minio.ObjectInfo{
@@ -631,13 +641,14 @@ func (a *azureObjects) GetObjectInfo(ctx context.Context, bucket, object string)
// uses Azure equivalent CreateBlockBlobFromReader.
func (a *azureObjects) PutObject(ctx context.Context, bucket, object string, data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
blob := a.client.GetContainerReference(bucket).GetBlobReference(object)
blob.Metadata, blob.Properties, err = s3MetaToAzureProperties(metadata)
blob.Metadata, blob.Properties, err = s3MetaToAzureProperties(ctx, metadata)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
err = blob.CreateBlockBlobFromReader(data, nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
return a.GetObjectInfo(ctx, bucket, object)
}
@@ -647,19 +658,21 @@ func (a *azureObjects) PutObject(ctx context.Context, bucket, object string, dat
func (a *azureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, destBucket, destObject string, srcInfo minio.ObjectInfo) (objInfo minio.ObjectInfo, err error) {
srcBlobURL := a.client.GetContainerReference(srcBucket).GetBlobReference(srcObject).GetURL()
destBlob := a.client.GetContainerReference(destBucket).GetBlobReference(destObject)
azureMeta, props, err := s3MetaToAzureProperties(srcInfo.UserDefined)
azureMeta, props, err := s3MetaToAzureProperties(ctx, srcInfo.UserDefined)
if err != nil {
return objInfo, azureToObjectError(err, srcBucket, srcObject)
}
destBlob.Metadata = azureMeta
err = destBlob.Copy(srcBlobURL, nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, srcBucket, srcObject)
}
destBlob.Properties = props
err = destBlob.SetProperties(nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, srcBucket, srcObject)
}
return a.GetObjectInfo(ctx, destBucket, destObject)
}
@@ -670,7 +683,7 @@ func (a *azureObjects) DeleteObject(ctx context.Context, bucket, object string)
blob := a.client.GetContainerReference(bucket).GetBlobReference(object)
err := blob.Delete(nil)
if err != nil {
return azureToObjectError(errors.Trace(err), bucket, object)
return azureToObjectError(err, bucket, object)
}
return nil
}
@@ -690,19 +703,21 @@ func getAzureMetadataObjectName(objectName, uploadID string) string {
return fmt.Sprintf(metadataObjectNameTemplate, uploadID, sha256.Sum256([]byte(objectName)))
}
func (a *azureObjects) checkUploadIDExists(bucketName, objectName, uploadID string) (err error) {
func (a *azureObjects) checkUploadIDExists(ctx context.Context, bucketName, objectName, uploadID string) (err error) {
blob := a.client.GetContainerReference(bucketName).GetBlobReference(
getAzureMetadataObjectName(objectName, uploadID))
err = blob.GetMetadata(nil)
err = azureToObjectError(errors.Trace(err), bucketName, objectName)
logger.LogIf(ctx, err)
err = azureToObjectError(err, bucketName, objectName)
oerr := minio.ObjectNotFound{
Bucket: bucketName,
Object: objectName,
}
if errors.Cause(err) == oerr {
err = errors.Trace(minio.InvalidUploadID{
if err == oerr {
logger.LogIf(ctx, minio.InvalidUploadID{UploadID: uploadID})
err = minio.InvalidUploadID{
UploadID: uploadID,
})
}
}
return err
}
@@ -714,13 +729,15 @@ func (a *azureObjects) NewMultipartUpload(ctx context.Context, bucket, object st
var jsonData []byte
if jsonData, err = json.Marshal(azureMultipartMetadata{Name: object, Metadata: metadata}); err != nil {
return "", errors.Trace(err)
logger.LogIf(ctx, err)
return "", err
}
blob := a.client.GetContainerReference(bucket).GetBlobReference(metadataObject)
err = blob.CreateBlockBlobFromReader(bytes.NewBuffer(jsonData), nil)
if err != nil {
return "", azureToObjectError(errors.Trace(err), bucket, metadataObject)
logger.LogIf(ctx, err)
return "", azureToObjectError(err, bucket, metadataObject)
}
return uploadID, nil
@@ -728,11 +745,11 @@ func (a *azureObjects) NewMultipartUpload(ctx context.Context, bucket, object st
// PutObjectPart - Use Azure equivalent PutBlockWithLength.
func (a *azureObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID string, partID int, data *hash.Reader) (info minio.PartInfo, err error) {
if err = a.checkUploadIDExists(bucket, object, uploadID); err != nil {
if err = a.checkUploadIDExists(ctx, bucket, object, uploadID); err != nil {
return info, err
}
if err = checkAzureUploadID(uploadID); err != nil {
if err = checkAzureUploadID(ctx, uploadID); err != nil {
return info, err
}
@@ -756,7 +773,8 @@ func (a *azureObjects) PutObjectPart(ctx context.Context, bucket, object, upload
blob := a.client.GetContainerReference(bucket).GetBlobReference(object)
err = blob.PutBlockWithLength(id, uint64(subPartSize), io.LimitReader(data, subPartSize), nil)
if err != nil {
return info, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return info, azureToObjectError(err, bucket, object)
}
subPartNumber++
}
@@ -770,7 +788,7 @@ func (a *azureObjects) PutObjectPart(ctx context.Context, bucket, object, upload
// ListObjectParts - Use Azure equivalent GetBlockList.
func (a *azureObjects) ListObjectParts(ctx context.Context, bucket, object, uploadID string, partNumberMarker int, maxParts int) (result minio.ListPartsInfo, err error) {
if err = a.checkUploadIDExists(bucket, object, uploadID); err != nil {
if err = a.checkUploadIDExists(ctx, bucket, object, uploadID); err != nil {
return result, err
}
@@ -787,7 +805,8 @@ func (a *azureObjects) ListObjectParts(ctx context.Context, bucket, object, uplo
return result, nil
}
if err != nil {
return result, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return result, azureToObjectError(err, bucket, object)
}
// Build a sorted list of parts and return the requested entries.
partsMap := make(map[int]minio.PartInfo)
@@ -796,7 +815,8 @@ func (a *azureObjects) ListObjectParts(ctx context.Context, bucket, object, uplo
var parsedUploadID string
var md5Hex string
if partNumber, _, parsedUploadID, md5Hex, err = azureParseBlockID(block.Name); err != nil {
return result, azureToObjectError(errors.Trace(fmt.Errorf("Unexpected error")), bucket, object)
logger.LogIf(ctx, fmt.Errorf("Unexpected error"))
return result, azureToObjectError(fmt.Errorf("Unexpected error"), bucket, object)
}
if parsedUploadID != uploadID {
continue
@@ -813,7 +833,8 @@ func (a *azureObjects) ListObjectParts(ctx context.Context, bucket, object, uplo
if part.ETag != md5Hex {
// If two parts of same partNumber were uploaded with different contents
// return error as we won't be able to decide which the latest part is.
return result, azureToObjectError(errors.Trace(fmt.Errorf("Unexpected error")), bucket, object)
logger.LogIf(ctx, fmt.Errorf("Unexpected error"))
return result, azureToObjectError(fmt.Errorf("Unexpected error"), bucket, object)
}
part.Size += block.Size
partsMap[partNumber] = part
@@ -856,7 +877,7 @@ func (a *azureObjects) ListObjectParts(ctx context.Context, bucket, object, uplo
// There is no corresponding API in azure to abort an incomplete upload. The uncommmitted blocks
// gets deleted after one week.
func (a *azureObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string) (err error) {
if err = a.checkUploadIDExists(bucket, object, uploadID); err != nil {
if err = a.checkUploadIDExists(ctx, bucket, object, uploadID); err != nil {
return err
}
@@ -868,23 +889,25 @@ func (a *azureObjects) AbortMultipartUpload(ctx context.Context, bucket, object,
// CompleteMultipartUpload - Use Azure equivalent PutBlockList.
func (a *azureObjects) CompleteMultipartUpload(ctx context.Context, bucket, object, uploadID string, uploadedParts []minio.CompletePart) (objInfo minio.ObjectInfo, err error) {
metadataObject := getAzureMetadataObjectName(object, uploadID)
if err = a.checkUploadIDExists(bucket, object, uploadID); err != nil {
if err = a.checkUploadIDExists(ctx, bucket, object, uploadID); err != nil {
return objInfo, err
}
if err = checkAzureUploadID(uploadID); err != nil {
if err = checkAzureUploadID(ctx, uploadID); err != nil {
return objInfo, err
}
var metadataReader io.Reader
blob := a.client.GetContainerReference(bucket).GetBlobReference(metadataObject)
if metadataReader, err = blob.Get(nil); err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, metadataObject)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, metadataObject)
}
var metadata azureMultipartMetadata
if err = json.NewDecoder(metadataReader).Decode(&metadata); err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, metadataObject)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, metadataObject)
}
defer func() {
@@ -894,13 +917,15 @@ func (a *azureObjects) CompleteMultipartUpload(ctx context.Context, bucket, obje
blob := a.client.GetContainerReference(bucket).GetBlobReference(metadataObject)
derr := blob.Delete(nil)
minio.ErrorIf(derr, "unable to remove meta data object for upload ID %s", uploadID)
logger.GetReqInfo(ctx).AppendTags("uploadID", uploadID)
logger.LogIf(ctx, derr)
}()
objBlob := a.client.GetContainerReference(bucket).GetBlobReference(object)
resp, err := objBlob.GetBlockList(storage.BlockListTypeUncommitted, nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
getBlocks := func(partNumber int, etag string) (blocks []storage.Block, size int64, err error) {
@@ -936,7 +961,8 @@ func (a *azureObjects) CompleteMultipartUpload(ctx context.Context, bucket, obje
var size int64
blocks, size, err = getBlocks(part.PartNumber, part.ETag)
if err != nil {
return objInfo, errors.Trace(err)
logger.LogIf(ctx, err)
return objInfo, err
}
allBlocks = append(allBlocks, blocks...)
@@ -946,30 +972,39 @@ func (a *azureObjects) CompleteMultipartUpload(ctx context.Context, bucket, obje
// Error out if parts except last part sizing < 5MiB.
for i, size := range partSizes[:len(partSizes)-1] {
if size < azureS3MinPartSize {
return objInfo, errors.Trace(minio.PartTooSmall{
logger.LogIf(ctx, minio.PartTooSmall{
PartNumber: uploadedParts[i].PartNumber,
PartSize: size,
PartETag: uploadedParts[i].ETag,
})
return objInfo, minio.PartTooSmall{
PartNumber: uploadedParts[i].PartNumber,
PartSize: size,
PartETag: uploadedParts[i].ETag,
}
}
}
err = objBlob.PutBlockList(allBlocks, nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
if len(metadata.Metadata) > 0 {
objBlob.Metadata, objBlob.Properties, err = s3MetaToAzureProperties(metadata.Metadata)
objBlob.Metadata, objBlob.Properties, err = s3MetaToAzureProperties(ctx, metadata.Metadata)
if err != nil {
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
err = objBlob.SetProperties(nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
err = objBlob.SetMetadata(nil)
if err != nil {
return objInfo, azureToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, azureToObjectError(err, bucket, object)
}
}
return a.GetObjectInfo(ctx, bucket, object)
@@ -992,13 +1027,16 @@ func (a *azureObjects) SetBucketPolicy(ctx context.Context, bucket string, polic
}
prefix := bucket + "/*" // For all objects inside the bucket.
if len(policies) != 1 {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
if policies[0].Prefix != prefix {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
if policies[0].Policy != policy.BucketPolicyReadOnly {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
perm := storage.ContainerPermissions{
AccessType: storage.ContainerAccessTypeContainer,
@@ -1006,7 +1044,8 @@ func (a *azureObjects) SetBucketPolicy(ctx context.Context, bucket string, polic
}
container := a.client.GetContainerReference(bucket)
err := container.SetPermissions(perm, nil)
return azureToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return azureToObjectError(err, bucket)
}
// GetBucketPolicy - Get the container ACL and convert it to canonical []bucketAccessPolicy
@@ -1015,15 +1054,18 @@ func (a *azureObjects) GetBucketPolicy(ctx context.Context, bucket string) (poli
container := a.client.GetContainerReference(bucket)
perm, err := container.GetPermissions(nil)
if err != nil {
return policy.BucketAccessPolicy{}, azureToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return policy.BucketAccessPolicy{}, azureToObjectError(err, bucket)
}
switch perm.AccessType {
case storage.ContainerAccessTypePrivate:
return policy.BucketAccessPolicy{}, errors.Trace(minio.PolicyNotFound{Bucket: bucket})
logger.LogIf(ctx, minio.PolicyNotFound{Bucket: bucket})
return policy.BucketAccessPolicy{}, minio.PolicyNotFound{Bucket: bucket}
case storage.ContainerAccessTypeContainer:
policyInfo.Statements = policy.SetPolicy(policyInfo.Statements, policy.BucketPolicyReadOnly, bucket, "")
default:
return policy.BucketAccessPolicy{}, azureToObjectError(errors.Trace(minio.NotImplemented{}))
logger.LogIf(ctx, minio.NotImplemented{})
return policy.BucketAccessPolicy{}, azureToObjectError(minio.NotImplemented{})
}
return policyInfo, nil
}
@@ -1036,5 +1078,6 @@ func (a *azureObjects) DeleteBucketPolicy(ctx context.Context, bucket string) er
}
container := a.client.GetContainerReference(bucket)
err := container.SetPermissions(perm, nil)
return azureToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return azureToObjectError(err)
}
+21 -28
View File
@@ -17,6 +17,7 @@
package azure
import (
"context"
"fmt"
"net/http"
"reflect"
@@ -24,7 +25,6 @@ import (
"github.com/Azure/azure-sdk-for-go/storage"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/pkg/errors"
)
// Test canonical metadata.
@@ -55,7 +55,7 @@ func TestS3MetaToAzureProperties(t *testing.T) {
"X_Amz_Matdesc": "{}",
"X_Amz_Iv": "eWmyryl8kq+EVnnsE7jpOg==",
}
meta, _, err := s3MetaToAzureProperties(headers)
meta, _, err := s3MetaToAzureProperties(context.Background(), headers)
if err != nil {
t.Fatalf("Test failed, with %s", err)
}
@@ -65,8 +65,8 @@ func TestS3MetaToAzureProperties(t *testing.T) {
headers = map[string]string{
"invalid--meta": "value",
}
_, _, err = s3MetaToAzureProperties(headers)
if err = errors.Cause(err); err != nil {
_, _, err = s3MetaToAzureProperties(context.Background(), headers)
if err != nil {
if _, ok := err.(minio.UnsupportedMetadata); !ok {
t.Fatalf("Test failed with unexpected error %s, expected UnsupportedMetadata", err)
}
@@ -75,7 +75,7 @@ func TestS3MetaToAzureProperties(t *testing.T) {
headers = map[string]string{
"content-md5": "Dce7bmCX61zvxzP5QmfelQ==",
}
_, props, err := s3MetaToAzureProperties(headers)
_, props, err := s3MetaToAzureProperties(context.Background(), headers)
if err != nil {
t.Fatalf("Test failed, with %s", err)
}
@@ -137,53 +137,46 @@ func TestAzureToObjectError(t *testing.T) {
nil, nil, "", "",
},
{
errors.Trace(fmt.Errorf("Non azure error")),
fmt.Errorf("Non azure error"),
fmt.Errorf("Non azure error"), "", "",
},
{
storage.AzureStorageServiceError{
Code: "ContainerAlreadyExists",
}, storage.AzureStorageServiceError{
Code: "ContainerAlreadyExists",
}, "bucket", "",
}, minio.BucketExists{Bucket: "bucket"}, "bucket", "",
},
{
errors.Trace(storage.AzureStorageServiceError{
Code: "ContainerAlreadyExists",
}), minio.BucketExists{Bucket: "bucket"}, "bucket", "",
},
{
errors.Trace(storage.AzureStorageServiceError{
storage.AzureStorageServiceError{
Code: "InvalidResourceName",
}), minio.BucketNameInvalid{Bucket: "bucket."}, "bucket.", "",
}, minio.BucketNameInvalid{Bucket: "bucket."}, "bucket.", "",
},
{
errors.Trace(storage.AzureStorageServiceError{
storage.AzureStorageServiceError{
Code: "RequestBodyTooLarge",
}), minio.PartTooBig{}, "", "",
}, minio.PartTooBig{}, "", "",
},
{
errors.Trace(storage.AzureStorageServiceError{
storage.AzureStorageServiceError{
Code: "InvalidMetadata",
}), minio.UnsupportedMetadata{}, "", "",
}, minio.UnsupportedMetadata{}, "", "",
},
{
errors.Trace(storage.AzureStorageServiceError{
storage.AzureStorageServiceError{
StatusCode: http.StatusNotFound,
}), minio.ObjectNotFound{
}, minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
}, "bucket", "object",
},
{
errors.Trace(storage.AzureStorageServiceError{
storage.AzureStorageServiceError{
StatusCode: http.StatusNotFound,
}), minio.BucketNotFound{Bucket: "bucket"}, "bucket", "",
}, minio.BucketNotFound{Bucket: "bucket"}, "bucket", "",
},
{
errors.Trace(storage.AzureStorageServiceError{
storage.AzureStorageServiceError{
StatusCode: http.StatusBadRequest,
}), minio.BucketNameInvalid{Bucket: "bucket."}, "bucket.", "",
}, minio.BucketNameInvalid{Bucket: "bucket."}, "bucket.", "",
},
}
for i, testCase := range testCases {
@@ -307,7 +300,7 @@ func TestCheckAzureUploadID(t *testing.T) {
}
for _, uploadID := range invalidUploadIDs {
if err := checkAzureUploadID(uploadID); err == nil {
if err := checkAzureUploadID(context.Background(), uploadID); err == nil {
t.Fatalf("%s: expected: <error>, got: <nil>", uploadID)
}
}
@@ -318,7 +311,7 @@ func TestCheckAzureUploadID(t *testing.T) {
}
for _, uploadID := range validUploadIDs {
if err := checkAzureUploadID(uploadID); err != nil {
if err := checkAzureUploadID(context.Background(), uploadID); err != nil {
t.Fatalf("%s: expected: <nil>, got: %s", uploadID, err)
}
}
+74 -48
View File
@@ -31,8 +31,8 @@ import (
b2 "github.com/minio/blazer/base"
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
h2 "github.com/minio/minio/pkg/hash"
minio "github.com/minio/minio/cmd"
@@ -64,7 +64,7 @@ ENVIRONMENT VARIABLES:
MINIO_BROWSER: To disable web browser access, set this value to "off".
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -83,7 +83,7 @@ EXAMPLES:
2. Start minio gateway server for B2 backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=accountID
$ export MINIO_SECRET_KEY=applicationKey
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}}
@@ -126,10 +126,9 @@ func (g *B2) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error)
}, nil
}
// Production - Ready for production use?
// Production - Ready for production use.
func (g *B2) Production() bool {
// Not ready for production use just yet.
return false
return true
}
// b2Object implements gateway for Minio and BackBlaze B2 compatible object storage servers.
@@ -146,16 +145,6 @@ func b2ToObjectError(err error, params ...string) error {
if err == nil {
return nil
}
e, ok := err.(*errors.Error)
if !ok {
// Code should be fixed if this function is called without doing errors.Trace()
// Else handling different situations in this function makes this function complicated.
minio.ErrorIf(err, "Expected type *Error")
return err
}
err = e.Cause
bucket := ""
object := ""
uploadID := ""
@@ -177,7 +166,7 @@ func b2ToObjectError(err error, params ...string) error {
if statusCode == 0 {
// We don't interpret non B2 errors. B2 errors have statusCode
// to help us convert them to S3 object errors.
return e
return err
}
switch code {
@@ -192,6 +181,15 @@ func b2ToObjectError(err error, params ...string) error {
} else if bucket != "" {
err = minio.BucketNotFound{Bucket: bucket}
}
case "bad_json":
if object != "" {
err = minio.ObjectNameInvalid{
Bucket: bucket,
Object: object,
}
} else if bucket != "" {
err = minio.BucketNameInvalid{Bucket: bucket}
}
case "bad_bucket_id":
err = minio.BucketNotFound{Bucket: bucket}
case "file_not_present", "not_found":
@@ -208,8 +206,7 @@ func b2ToObjectError(err error, params ...string) error {
err = minio.InvalidUploadID{UploadID: uploadID}
}
e.Cause = err
return e
return err
}
// Shutdown saves any gateway metadata to disk
@@ -230,7 +227,8 @@ func (l *b2Objects) MakeBucketWithLocation(ctx context.Context, bucket, location
// All buckets are set to private by default.
_, err := l.b2Client.CreateBucket(l.ctx, bucket, bucketTypePrivate, nil, nil)
return b2ToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket)
}
func (l *b2Objects) reAuthorizeAccount(ctx context.Context) error {
@@ -271,14 +269,16 @@ func (l *b2Objects) listBuckets(ctx context.Context, err error) ([]*b2.Bucket, e
func (l *b2Objects) Bucket(ctx context.Context, bucket string) (*b2.Bucket, error) {
bktList, err := l.listBuckets(ctx, nil)
if err != nil {
return nil, b2ToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return nil, b2ToObjectError(err, bucket)
}
for _, bkt := range bktList {
if bkt.Name == bucket {
return bkt, nil
}
}
return nil, errors.Trace(minio.BucketNotFound{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNotFound{Bucket: bucket})
return nil, minio.BucketNotFound{Bucket: bucket}
}
// GetBucketInfo gets bucket metadata..
@@ -315,7 +315,8 @@ func (l *b2Objects) DeleteBucket(ctx context.Context, bucket string) error {
return err
}
err = bkt.DeleteBucket(l.ctx)
return b2ToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket)
}
// ListObjects lists all objects in B2 bucket filtered by prefix, returns upto at max 1000 entries at a time.
@@ -326,7 +327,8 @@ func (l *b2Objects) ListObjects(ctx context.Context, bucket string, prefix strin
}
files, next, lerr := bkt.ListFileNames(l.ctx, maxKeys, marker, prefix, delimiter)
if lerr != nil {
return loi, b2ToObjectError(errors.Trace(lerr), bucket)
logger.LogIf(ctx, lerr)
return loi, b2ToObjectError(lerr, bucket)
}
loi.IsTruncated = next != ""
loi.NextMarker = next
@@ -359,7 +361,8 @@ func (l *b2Objects) ListObjectsV2(ctx context.Context, bucket, prefix, continuat
}
files, next, lerr := bkt.ListFileNames(l.ctx, maxKeys, continuationToken, prefix, delimiter)
if lerr != nil {
return loi, b2ToObjectError(errors.Trace(lerr), bucket)
logger.LogIf(ctx, lerr)
return loi, b2ToObjectError(lerr, bucket)
}
loi.IsTruncated = next != ""
loi.ContinuationToken = continuationToken
@@ -396,11 +399,13 @@ func (l *b2Objects) GetObject(ctx context.Context, bucket string, object string,
}
reader, err := bkt.DownloadFileByName(l.ctx, object, startOffset, length)
if err != nil {
return b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket, object)
}
defer reader.Close()
_, err = io.Copy(writer, reader)
return b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket, object)
}
// GetObjectInfo reads object info and replies back ObjectInfo
@@ -411,12 +416,14 @@ func (l *b2Objects) GetObjectInfo(ctx context.Context, bucket string, object str
}
f, err := bkt.DownloadFileByName(l.ctx, object, 0, 1)
if err != nil {
return objInfo, b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, b2ToObjectError(err, bucket, object)
}
f.Close()
fi, err := bkt.File(f.ID, object).GetFileInfo(l.ctx)
if err != nil {
return objInfo, b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, b2ToObjectError(err, bucket, object)
}
return minio.ObjectInfo{
Bucket: bucket,
@@ -504,20 +511,23 @@ func (l *b2Objects) PutObject(ctx context.Context, bucket string, object string,
var u *b2.URL
u, err = bkt.GetUploadURL(l.ctx)
if err != nil {
return objInfo, b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, b2ToObjectError(err, bucket, object)
}
hr := newB2Reader(data, data.Size())
var f *b2.File
f, err = u.UploadFile(l.ctx, hr, int(hr.Size()), object, contentType, sha1AtEOF, metadata)
if err != nil {
return objInfo, b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, b2ToObjectError(err, bucket, object)
}
var fi *b2.FileInfo
fi, err = f.GetFileInfo(l.ctx)
if err != nil {
return objInfo, b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, b2ToObjectError(err, bucket, object)
}
return minio.ObjectInfo{
@@ -539,12 +549,14 @@ func (l *b2Objects) DeleteObject(ctx context.Context, bucket string, object stri
}
reader, err := bkt.DownloadFileByName(l.ctx, object, 0, 1)
if err != nil {
return b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket, object)
}
io.Copy(ioutil.Discard, reader)
reader.Close()
err = bkt.File(reader.ID, object).DeleteFileVersion(l.ctx)
return b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket, object)
}
// ListMultipartUploads lists all multipart uploads.
@@ -563,7 +575,8 @@ func (l *b2Objects) ListMultipartUploads(ctx context.Context, bucket string, pre
}
largeFiles, nextMarker, err := bkt.ListUnfinishedLargeFiles(l.ctx, uploadIDMarker, maxUploads)
if err != nil {
return lmi, b2ToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return lmi, b2ToObjectError(err, bucket)
}
lmi = minio.ListMultipartsInfo{
MaxUploads: maxUploads,
@@ -598,7 +611,8 @@ func (l *b2Objects) NewMultipartUpload(ctx context.Context, bucket string, objec
delete(metadata, "content-type")
lf, err := bkt.StartLargeFile(l.ctx, object, contentType, metadata)
if err != nil {
return uploadID, b2ToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return uploadID, b2ToObjectError(err, bucket, object)
}
return lf.ID, nil
@@ -613,13 +627,15 @@ func (l *b2Objects) PutObjectPart(ctx context.Context, bucket string, object str
fc, err := bkt.File(uploadID, object).CompileParts(0, nil).GetUploadPartURL(l.ctx)
if err != nil {
return pi, b2ToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return pi, b2ToObjectError(err, bucket, object, uploadID)
}
hr := newB2Reader(data, data.Size())
sha1, err := fc.UploadPart(l.ctx, hr, sha1AtEOF, int(hr.Size()), partID)
if err != nil {
return pi, b2ToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return pi, b2ToObjectError(err, bucket, object, uploadID)
}
return minio.PartInfo{
@@ -647,7 +663,8 @@ func (l *b2Objects) ListObjectParts(ctx context.Context, bucket string, object s
partNumberMarker++
partsList, next, err := bkt.File(uploadID, object).ListParts(l.ctx, partNumberMarker, maxParts)
if err != nil {
return lpi, b2ToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return lpi, b2ToObjectError(err, bucket, object, uploadID)
}
if next != 0 {
lpi.IsTruncated = true
@@ -670,7 +687,8 @@ func (l *b2Objects) AbortMultipartUpload(ctx context.Context, bucket string, obj
return err
}
err = bkt.File(uploadID, object).CompileParts(0, nil).CancelLargeFile(l.ctx)
return b2ToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return b2ToObjectError(err, bucket, object, uploadID)
}
// CompleteMultipartUpload completes ongoing multipart upload and finalizes object, uses B2's LargeFile upload API.
@@ -684,7 +702,8 @@ func (l *b2Objects) CompleteMultipartUpload(ctx context.Context, bucket string,
// B2 requires contigous part numbers starting with 1, they do not support
// hand picking part numbers, we return an S3 compatible error instead.
if i+1 != uploadedPart.PartNumber {
return oi, b2ToObjectError(errors.Trace(minio.InvalidPart{}), bucket, object, uploadID)
logger.LogIf(ctx, minio.InvalidPart{})
return oi, b2ToObjectError(minio.InvalidPart{}, bucket, object, uploadID)
}
// Trim "-1" suffix in ETag as PutObjectPart() treats B2 returned SHA1 as ETag.
@@ -692,7 +711,8 @@ func (l *b2Objects) CompleteMultipartUpload(ctx context.Context, bucket string,
}
if _, err = bkt.File(uploadID, object).CompileParts(0, hashes).FinishLargeFile(l.ctx); err != nil {
return oi, b2ToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return oi, b2ToObjectError(err, bucket, object, uploadID)
}
return l.GetObjectInfo(ctx, bucket, object)
@@ -713,13 +733,16 @@ func (l *b2Objects) SetBucketPolicy(ctx context.Context, bucket string, policyIn
}
prefix := bucket + "/*" // For all objects inside the bucket.
if len(policies) != 1 {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
if policies[0].Prefix != prefix {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
if policies[0].Policy != policy.BucketPolicyReadOnly {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
bkt, err := l.Bucket(ctx, bucket)
if err != nil {
@@ -727,7 +750,8 @@ func (l *b2Objects) SetBucketPolicy(ctx context.Context, bucket string, policyIn
}
bkt.Type = bucketTypeReadOnly
_, err = bkt.Update(l.ctx)
return b2ToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return b2ToObjectError(err)
}
// GetBucketPolicy, returns the current bucketType from B2 backend and convert
@@ -745,7 +769,8 @@ func (l *b2Objects) GetBucketPolicy(ctx context.Context, bucket string) (policy.
// bkt.Type can also be snapshot, but it is only allowed through B2 browser console,
// just return back as policy not found for all cases.
// CreateBucket always sets the value to allPrivate by default.
return policy.BucketAccessPolicy{}, errors.Trace(minio.PolicyNotFound{Bucket: bucket})
logger.LogIf(ctx, minio.PolicyNotFound{Bucket: bucket})
return policy.BucketAccessPolicy{}, minio.PolicyNotFound{Bucket: bucket}
}
// DeleteBucketPolicy - resets the bucketType of bucket on B2 to 'allPrivate'.
@@ -756,5 +781,6 @@ func (l *b2Objects) DeleteBucketPolicy(ctx context.Context, bucket string) error
}
bkt.Type = bucketTypePrivate
_, err = bkt.Update(l.ctx)
return b2ToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return b2ToObjectError(err)
}
+17 -18
View File
@@ -21,7 +21,6 @@ import (
"testing"
b2 "github.com/minio/blazer/base"
"github.com/minio/minio/pkg/errors"
minio "github.com/minio/minio/cmd"
)
@@ -40,70 +39,70 @@ func TestB2ObjectError(t *testing.T) {
[]string{}, fmt.Errorf("Not *Error"), fmt.Errorf("Not *Error"),
},
{
[]string{}, errors.Trace(fmt.Errorf("Non B2 Error")), fmt.Errorf("Non B2 Error"),
[]string{}, fmt.Errorf("Non B2 Error"), fmt.Errorf("Non B2 Error"),
},
{
[]string{"bucket"}, errors.Trace(b2.Error{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "duplicate_bucket_name",
}), minio.BucketAlreadyOwnedByYou{
}, minio.BucketAlreadyOwnedByYou{
Bucket: "bucket",
},
},
{
[]string{"bucket"}, errors.Trace(b2.Error{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "bad_request",
}), minio.BucketNotFound{
}, minio.BucketNotFound{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"}, errors.Trace(b2.Error{
[]string{"bucket", "object"}, b2.Error{
StatusCode: 1,
Code: "bad_request",
}), minio.ObjectNameInvalid{
}, minio.ObjectNameInvalid{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket"}, errors.Trace(b2.Error{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "bad_bucket_id",
}), minio.BucketNotFound{Bucket: "bucket"},
}, minio.BucketNotFound{Bucket: "bucket"},
},
{
[]string{"bucket", "object"}, errors.Trace(b2.Error{
[]string{"bucket", "object"}, b2.Error{
StatusCode: 1,
Code: "file_not_present",
}), minio.ObjectNotFound{
}, minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket", "object"}, errors.Trace(b2.Error{
[]string{"bucket", "object"}, b2.Error{
StatusCode: 1,
Code: "not_found",
}), minio.ObjectNotFound{
}, minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket"}, errors.Trace(b2.Error{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "cannot_delete_non_empty_bucket",
}), minio.BucketNotEmpty{
}, minio.BucketNotEmpty{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object", "uploadID"}, errors.Trace(b2.Error{
[]string{"bucket", "object", "uploadID"}, b2.Error{
StatusCode: 1,
Message: "No active upload for",
}), minio.InvalidUploadID{
}, minio.InvalidUploadID{
UploadID: "uploadID",
},
},
+157 -84
View File
@@ -34,8 +34,8 @@ import (
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
"google.golang.org/api/googleapi"
@@ -110,7 +110,7 @@ ENVIRONMENT VARIABLES:
MINIO_BROWSER: To disable web browser access, set this value to "off".
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -135,7 +135,7 @@ EXAMPLES:
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}} mygcsprojectid
@@ -154,11 +154,13 @@ EXAMPLES:
func gcsGatewayMain(ctx *cli.Context) {
projectID := ctx.Args().First()
if projectID == "" && os.Getenv("GOOGLE_APPLICATION_CREDENTIALS") == "" {
minio.ErrorIf(errGCSProjectIDNotFound, "project-id should be provided as argument or GOOGLE_APPLICATION_CREDENTIALS should be set with path to credentials.json")
logger.LogIf(context.Background(), errGCSProjectIDNotFound)
cli.ShowCommandHelpAndExit(ctx, "gcs", 1)
}
if projectID != "" && !isValidGCSProjectIDFormat(projectID) {
minio.ErrorIf(errGCSInvalidProjectID, "Unable to start GCS gateway with %s", ctx.Args().First())
reqInfo := (&logger.ReqInfo{}).AppendTags("projectID", ctx.Args().First())
contxt := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(contxt, errGCSInvalidProjectID)
cli.ShowCommandHelpAndExit(ctx, "gcs", 1)
}
@@ -208,9 +210,9 @@ func (g *GCS) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error)
return gcs, nil
}
// Production - FIXME: GCS is not production ready yet.
// Production - GCS gateway is production ready.
func (g *GCS) Production() bool {
return false
return true
}
// Stored in gcs.json - Contents of this file is not used anywhere. It can be
@@ -237,16 +239,6 @@ func gcsToObjectError(err error, params ...string) error {
return nil
}
e, ok := err.(*errors.Error)
if !ok {
// Code should be fixed if this function is called without doing errors.Trace()
// Else handling different situations in this function makes this function complicated.
minio.ErrorIf(err, "Expected type *Error")
return err
}
err = e.Cause
bucket := ""
object := ""
uploadID := ""
@@ -266,8 +258,7 @@ func gcsToObjectError(err error, params ...string) error {
err = minio.BucketNotFound{
Bucket: bucket,
}
e.Cause = err
return e
return err
case "storage: object doesn't exist":
if uploadID != "" {
err = minio.InvalidUploadID{
@@ -279,21 +270,18 @@ func gcsToObjectError(err error, params ...string) error {
Object: object,
}
}
e.Cause = err
return e
return err
}
googleAPIErr, ok := err.(*googleapi.Error)
if !ok {
// We don't interpret non Minio errors. As minio errors will
// have StatusCode to help to convert to object errors.
e.Cause = err
return e
return err
}
if len(googleAPIErr.Errors) == 0 {
e.Cause = err
return e
return err
}
reason := googleAPIErr.Errors[0].Reason
@@ -337,8 +325,7 @@ func gcsToObjectError(err error, params ...string) error {
err = fmt.Errorf("Unsupported error reason: %s", reason)
}
e.Cause = err
return e
return err
}
// gcsProjectIDRegex defines a valid gcs project id format
@@ -381,7 +368,9 @@ func (l *gcsGateway) CleanupGCSMinioSysTmpBucket(bucket string) {
attrs, err := it.Next()
if err != nil {
if err != iterator.Done {
minio.ErrorIf(err, "Object listing error on bucket %s during purging of old files in minio.sys.tmp", bucket)
reqInfo := &logger.ReqInfo{BucketName: bucket}
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
return
}
@@ -389,7 +378,9 @@ func (l *gcsGateway) CleanupGCSMinioSysTmpBucket(bucket string) {
// Delete files older than 2 weeks.
err := l.client.Bucket(bucket).Object(attrs.Name).Delete(l.ctx)
if err != nil {
minio.ErrorIf(err, "Unable to delete %s/%s during purging of old files in minio.sys.tmp", bucket, attrs.Name)
reqInfo := &logger.ReqInfo{BucketName: bucket, ObjectName: attrs.Name}
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return
}
}
@@ -404,7 +395,8 @@ func (l *gcsGateway) CleanupGCSMinioSysTmp() {
attrs, err := it.Next()
if err != nil {
if err != iterator.Done {
minio.ErrorIf(err, "Bucket listing error during purging of old files in minio.sys.tmp")
ctx := logger.SetReqInfo(context.Background(), &logger.ReqInfo{})
logger.LogIf(ctx, err)
}
break
}
@@ -438,15 +430,16 @@ func (l *gcsGateway) MakeBucketWithLocation(ctx context.Context, bucket, locatio
err := bkt.Create(l.ctx, l.projectID, &storage.BucketAttrs{
Location: location,
})
return gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket)
}
// GetBucketInfo - Get bucket metadata..
func (l *gcsGateway) GetBucketInfo(ctx context.Context, bucket string) (minio.BucketInfo, error) {
attrs, err := l.client.Bucket(bucket).Attrs(l.ctx)
if err != nil {
return minio.BucketInfo{}, gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return minio.BucketInfo{}, gcsToObjectError(err, bucket)
}
return minio.BucketInfo{
@@ -467,7 +460,8 @@ func (l *gcsGateway) ListBuckets(ctx context.Context) (buckets []minio.BucketInf
}
if ierr != nil {
return buckets, gcsToObjectError(errors.Trace(ierr))
logger.LogIf(ctx, ierr)
return buckets, gcsToObjectError(ierr)
}
buckets = append(buckets, minio.BucketInfo{
@@ -495,7 +489,8 @@ func (l *gcsGateway) DeleteBucket(ctx context.Context, bucket string) error {
break
}
if err != nil {
return gcsToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return gcsToObjectError(err)
}
if objAttrs.Prefix == minio.GatewayMinioSysTmp {
gcsMinioPathFound = true
@@ -505,7 +500,8 @@ func (l *gcsGateway) DeleteBucket(ctx context.Context, bucket string) error {
break
}
if nonGCSMinioPathFound {
return gcsToObjectError(errors.Trace(minio.BucketNotEmpty{}))
logger.LogIf(ctx, minio.BucketNotEmpty{})
return gcsToObjectError(minio.BucketNotEmpty{})
}
if gcsMinioPathFound {
// Remove minio.sys.tmp before deleting the bucket.
@@ -516,16 +512,19 @@ func (l *gcsGateway) DeleteBucket(ctx context.Context, bucket string) error {
break
}
if err != nil {
return gcsToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return gcsToObjectError(err)
}
err = l.client.Bucket(bucket).Object(objAttrs.Name).Delete(l.ctx)
if err != nil {
return gcsToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return gcsToObjectError(err)
}
}
}
err := l.client.Bucket(bucket).Delete(l.ctx)
return gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket)
}
func toGCSPageToken(name string) string {
@@ -607,7 +606,8 @@ func (l *gcsGateway) ListObjects(ctx context.Context, bucket string, prefix stri
break
}
if err != nil {
return minio.ListObjectsInfo{}, gcsToObjectError(errors.Trace(err), bucket, prefix)
logger.LogIf(ctx, err)
return minio.ListObjectsInfo{}, gcsToObjectError(err, bucket, prefix)
}
nextMarker = toGCSPageToken(attrs.Name)
@@ -689,7 +689,8 @@ func (l *gcsGateway) ListObjectsV2(ctx context.Context, bucket, prefix, continua
}
if err != nil {
return minio.ListObjectsV2Info{}, gcsToObjectError(errors.Trace(err), bucket, prefix)
logger.LogIf(ctx, err)
return minio.ListObjectsV2Info{}, gcsToObjectError(err, bucket, prefix)
}
if attrs.Prefix == minio.GatewayMinioSysTmp {
@@ -733,18 +734,21 @@ func (l *gcsGateway) GetObject(ctx context.Context, bucket string, key string, s
// if we want to mimic S3 behavior exactly, we need to verify if bucket exists first,
// otherwise gcs will just return object not exist in case of non-existing bucket
if _, err := l.client.Bucket(bucket).Attrs(l.ctx); err != nil {
return gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket)
}
object := l.client.Bucket(bucket).Object(key)
r, err := object.NewRangeReader(l.ctx, startOffset, length)
if err != nil {
return gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket, key)
}
defer r.Close()
if _, err := io.Copy(writer, r); err != nil {
return gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket, key)
}
return nil
@@ -754,29 +758,70 @@ func (l *gcsGateway) GetObject(ctx context.Context, bucket string, key string, s
func fromGCSAttrsToObjectInfo(attrs *storage.ObjectAttrs) minio.ObjectInfo {
// All google cloud storage objects have a CRC32c hash, whereas composite objects may not have a MD5 hash
// Refer https://cloud.google.com/storage/docs/hashes-etags. Use CRC32C for ETag
metadata := make(map[string]string)
for k, v := range attrs.Metadata {
metadata[k] = v
}
if attrs.ContentType != "" {
metadata["content-type"] = attrs.ContentType
}
if attrs.ContentEncoding != "" {
metadata["content-encoding"] = attrs.ContentEncoding
}
if attrs.CacheControl != "" {
metadata["cache-control"] = attrs.CacheControl
}
if attrs.ContentDisposition != "" {
metadata["content-disposition"] = attrs.ContentDisposition
}
if attrs.ContentLanguage != "" {
metadata["content-language"] = attrs.ContentLanguage
}
return minio.ObjectInfo{
Name: attrs.Name,
Bucket: attrs.Bucket,
ModTime: attrs.Updated,
Size: attrs.Size,
ETag: minio.ToS3ETag(fmt.Sprintf("%d", attrs.CRC32C)),
UserDefined: attrs.Metadata,
UserDefined: metadata,
ContentType: attrs.ContentType,
ContentEncoding: attrs.ContentEncoding,
}
}
// applyMetadataToGCSAttrs applies metadata to a GCS ObjectAttrs instance
func applyMetadataToGCSAttrs(metadata map[string]string, attrs *storage.ObjectAttrs) {
attrs.ContentType = metadata["content-type"]
attrs.ContentEncoding = metadata["content-encoding"]
attrs.CacheControl = metadata["cache-control"]
attrs.ContentDisposition = metadata["content-disposition"]
attrs.ContentLanguage = metadata["content-language"]
attrs.Metadata = make(map[string]string)
for k, v := range metadata {
attrs.Metadata[k] = v
}
// Filter metadata which is stored as a unique attribute
for _, key := range []string{
"content-type", "content-encoding", "cache-control", "content-disposition", "content-language",
} {
delete(attrs.Metadata, key)
}
}
// GetObjectInfo - reads object info and replies back ObjectInfo
func (l *gcsGateway) GetObjectInfo(ctx context.Context, bucket string, object string) (minio.ObjectInfo, error) {
// if we want to mimic S3 behavior exactly, we need to verify if bucket exists first,
// otherwise gcs will just return object not exist in case of non-existing bucket
if _, err := l.client.Bucket(bucket).Attrs(l.ctx); err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket)
}
attrs, err := l.client.Bucket(bucket).Object(object).Attrs(l.ctx)
if err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, object)
}
return fromGCSAttrsToObjectInfo(attrs), nil
@@ -787,21 +832,20 @@ func (l *gcsGateway) PutObject(ctx context.Context, bucket string, key string, d
// if we want to mimic S3 behavior exactly, we need to verify if bucket exists first,
// otherwise gcs will just return object not exist in case of non-existing bucket
if _, err := l.client.Bucket(bucket).Attrs(l.ctx); err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket)
}
object := l.client.Bucket(bucket).Object(key)
w := object.NewWriter(l.ctx)
w.ContentType = metadata["content-type"]
w.ContentEncoding = metadata["content-encoding"]
w.Metadata = metadata
applyMetadataToGCSAttrs(metadata, &w.ObjectAttrs)
if _, err := io.Copy(w, data); err != nil {
// Close the object writer upon error.
w.CloseWithError(err)
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
// Close the object writer upon success.
@@ -809,7 +853,8 @@ func (l *gcsGateway) PutObject(ctx context.Context, bucket string, key string, d
attrs, err := object.Attrs(l.ctx)
if err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
return fromGCSAttrsToObjectInfo(attrs), nil
@@ -823,11 +868,12 @@ func (l *gcsGateway) CopyObject(ctx context.Context, srcBucket string, srcObject
dst := l.client.Bucket(destBucket).Object(destObject)
copier := dst.CopierFrom(src)
copier.ObjectAttrs.Metadata = srcInfo.UserDefined
applyMetadataToGCSAttrs(srcInfo.UserDefined, &copier.ObjectAttrs)
attrs, err := copier.Run(l.ctx)
if err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), destBucket, destObject)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, destBucket, destObject)
}
return fromGCSAttrsToObjectInfo(attrs), nil
@@ -837,7 +883,8 @@ func (l *gcsGateway) CopyObject(ctx context.Context, srcBucket string, srcObject
func (l *gcsGateway) DeleteObject(ctx context.Context, bucket string, object string) error {
err := l.client.Bucket(bucket).Object(object).Delete(l.ctx)
if err != nil {
return gcsToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket, object)
}
return nil
@@ -854,16 +901,15 @@ func (l *gcsGateway) NewMultipartUpload(ctx context.Context, bucket string, key
w := l.client.Bucket(bucket).Object(meta).NewWriter(l.ctx)
defer w.Close()
w.ContentType = metadata["content-type"]
w.ContentEncoding = metadata["content-encoding"]
w.Metadata = metadata
applyMetadataToGCSAttrs(metadata, &w.ObjectAttrs)
if err = json.NewEncoder(w).Encode(gcsMultipartMetaV1{
gcsMinioMultipartMetaCurrentVersion,
bucket,
key,
}); err != nil {
return "", gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return "", gcsToObjectError(err, bucket, key)
}
return uploadID, nil
}
@@ -883,7 +929,8 @@ func (l *gcsGateway) ListMultipartUploads(ctx context.Context, bucket string, pr
// an object layer compatible error upon any error.
func (l *gcsGateway) checkUploadIDExists(ctx context.Context, bucket string, key string, uploadID string) error {
_, err := l.client.Bucket(bucket).Object(gcsMultipartMetaName(uploadID)).Attrs(l.ctx)
return gcsToObjectError(errors.Trace(err), bucket, key, uploadID)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket, key, uploadID)
}
// PutObjectPart puts a part of object in bucket
@@ -904,7 +951,8 @@ func (l *gcsGateway) PutObjectPart(ctx context.Context, bucket string, key strin
if _, err := io.Copy(w, data); err != nil {
// Make sure to close object writer upon error.
w.Close()
return minio.PartInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.PartInfo{}, gcsToObjectError(err, bucket, key)
}
// Make sure to close the object writer upon success.
w.Close()
@@ -923,7 +971,7 @@ func (l *gcsGateway) ListObjectParts(ctx context.Context, bucket string, key str
}
// Called by AbortMultipartUpload and CompleteMultipartUpload for cleaning up.
func (l *gcsGateway) cleanupMultipartUpload(bucket, key, uploadID string) error {
func (l *gcsGateway) cleanupMultipartUpload(ctx context.Context, bucket, key, uploadID string) error {
prefix := fmt.Sprintf("%s/%s/", gcsMinioMultipartPathV1, uploadID)
// iterate through all parts and delete them
@@ -935,7 +983,8 @@ func (l *gcsGateway) cleanupMultipartUpload(bucket, key, uploadID string) error
break
}
if err != nil {
return gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket, key)
}
object := l.client.Bucket(bucket).Object(attrs.Name)
@@ -951,7 +1000,7 @@ func (l *gcsGateway) AbortMultipartUpload(ctx context.Context, bucket string, ke
if err := l.checkUploadIDExists(ctx, bucket, key, uploadID); err != nil {
return err
}
return l.cleanupMultipartUpload(bucket, key, uploadID)
return l.cleanupMultipartUpload(ctx, bucket, key, uploadID)
}
// CompleteMultipartUpload completes ongoing multipart upload and finalizes object
@@ -968,23 +1017,27 @@ func (l *gcsGateway) CompleteMultipartUpload(ctx context.Context, bucket string,
partZeroAttrs, err := object.Attrs(l.ctx)
if err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key, uploadID)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key, uploadID)
}
r, err := object.NewReader(l.ctx)
if err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
defer r.Close()
// Check version compatibility of the meta file before compose()
multipartMeta := gcsMultipartMetaV1{}
if err = json.NewDecoder(r).Decode(&multipartMeta); err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
if multipartMeta.Version != gcsMinioMultipartMetaCurrentVersion {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(errGCSFormat), bucket, key)
logger.LogIf(ctx, errGCSFormat)
return minio.ObjectInfo{}, gcsToObjectError(errGCSFormat, bucket, key)
}
// Validate if the gcs.json stores valid entries for the bucket and key.
@@ -1001,7 +1054,8 @@ func (l *gcsGateway) CompleteMultipartUpload(ctx context.Context, bucket string,
uploadedPart.PartNumber, uploadedPart.ETag)))
partAttr, pErr := l.client.Bucket(bucket).Object(gcsMultipartDataName(uploadID, uploadedPart.PartNumber, uploadedPart.ETag)).Attrs(l.ctx)
if pErr != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(pErr), bucket, key, uploadID)
logger.LogIf(ctx, pErr)
return minio.ObjectInfo{}, gcsToObjectError(pErr, bucket, key, uploadID)
}
partSizes[i] = partAttr.Size
}
@@ -1009,11 +1063,16 @@ func (l *gcsGateway) CompleteMultipartUpload(ctx context.Context, bucket string,
// Error out if parts except last part sizing < 5MiB.
for i, size := range partSizes[:len(partSizes)-1] {
if size < 5*humanize.MiByte {
return minio.ObjectInfo{}, errors.Trace(minio.PartTooSmall{
logger.LogIf(ctx, minio.PartTooSmall{
PartNumber: uploadedParts[i].PartNumber,
PartSize: size,
PartETag: uploadedParts[i].ETag,
})
return minio.ObjectInfo{}, minio.PartTooSmall{
PartNumber: uploadedParts[i].PartNumber,
PartSize: size,
PartETag: uploadedParts[i].ETag,
}
}
}
@@ -1040,7 +1099,8 @@ func (l *gcsGateway) CompleteMultipartUpload(ctx context.Context, bucket string,
composer.Metadata = partZeroAttrs.Metadata
if _, err = composer.Run(l.ctx); err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
}
@@ -1050,13 +1110,18 @@ func (l *gcsGateway) CompleteMultipartUpload(ctx context.Context, bucket string,
composer := l.client.Bucket(bucket).Object(key).ComposerFrom(parts...)
composer.ContentType = partZeroAttrs.ContentType
composer.ContentEncoding = partZeroAttrs.ContentEncoding
composer.CacheControl = partZeroAttrs.CacheControl
composer.ContentDisposition = partZeroAttrs.ContentDisposition
composer.ContentLanguage = partZeroAttrs.ContentLanguage
composer.Metadata = partZeroAttrs.Metadata
attrs, err := composer.Run(l.ctx)
if err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
if err = l.cleanupMultipartUpload(bucket, key, uploadID); err != nil {
return minio.ObjectInfo{}, gcsToObjectError(errors.Trace(err), bucket, key)
if err = l.cleanupMultipartUpload(ctx, bucket, key, uploadID); err != nil {
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
return fromGCSAttrsToObjectInfo(attrs), nil
}
@@ -1075,16 +1140,19 @@ func (l *gcsGateway) SetBucketPolicy(ctx context.Context, bucket string, policyI
prefix := bucket + "/*" // For all objects inside the bucket.
if len(policies) != 1 {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
if policies[0].Prefix != prefix {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
acl := l.client.Bucket(bucket).ACL()
if policies[0].Policy == policy.BucketPolicyNone {
if err := acl.Delete(l.ctx, storage.AllUsers); err != nil {
return gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket)
}
return nil
}
@@ -1096,11 +1164,13 @@ func (l *gcsGateway) SetBucketPolicy(ctx context.Context, bucket string, policyI
case policy.BucketPolicyWriteOnly:
role = storage.RoleWriter
default:
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
if err := acl.Set(l.ctx, storage.AllUsers, role); err != nil {
return gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket)
}
return nil
@@ -1110,7 +1180,8 @@ func (l *gcsGateway) SetBucketPolicy(ctx context.Context, bucket string, policyI
func (l *gcsGateway) GetBucketPolicy(ctx context.Context, bucket string) (policy.BucketAccessPolicy, error) {
rules, err := l.client.Bucket(bucket).ACL().List(l.ctx)
if err != nil {
return policy.BucketAccessPolicy{}, gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return policy.BucketAccessPolicy{}, gcsToObjectError(err, bucket)
}
policyInfo := policy.BucketAccessPolicy{Version: "2012-10-17"}
for _, r := range rules {
@@ -1126,7 +1197,8 @@ func (l *gcsGateway) GetBucketPolicy(ctx context.Context, bucket string) (policy
}
// Return NoSuchBucketPolicy error, when policy is not set
if len(policyInfo.Statements) == 0 {
return policy.BucketAccessPolicy{}, gcsToObjectError(errors.Trace(minio.PolicyNotFound{}), bucket)
logger.LogIf(ctx, minio.PolicyNotFound{})
return policy.BucketAccessPolicy{}, gcsToObjectError(minio.PolicyNotFound{}, bucket)
}
return policyInfo, nil
}
@@ -1135,7 +1207,8 @@ func (l *gcsGateway) GetBucketPolicy(ctx context.Context, bucket string) (policy
func (l *gcsGateway) DeleteBucketPolicy(ctx context.Context, bucket string) error {
// This only removes the storage.AllUsers policies
if err := l.client.Bucket(bucket).ACL().Delete(l.ctx, storage.AllUsers); err != nil {
return gcsToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return gcsToObjectError(err, bucket)
}
return nil
+26 -27
View File
@@ -24,7 +24,6 @@ import (
"reflect"
"testing"
"github.com/minio/minio/pkg/errors"
"google.golang.org/api/googleapi"
miniogo "github.com/minio/minio-go"
@@ -237,14 +236,14 @@ func TestGCSToObjectError(t *testing.T) {
},
{
[]string{"bucket"},
errors.Trace(fmt.Errorf("storage: bucket doesn't exist")),
fmt.Errorf("storage: bucket doesn't exist"),
minio.BucketNotFound{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"},
errors.Trace(fmt.Errorf("storage: object doesn't exist")),
fmt.Errorf("storage: object doesn't exist"),
minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
@@ -252,76 +251,76 @@ func TestGCSToObjectError(t *testing.T) {
},
{
[]string{"bucket", "object", "uploadID"},
errors.Trace(fmt.Errorf("storage: object doesn't exist")),
fmt.Errorf("storage: object doesn't exist"),
minio.InvalidUploadID{
UploadID: "uploadID",
},
},
{
[]string{},
errors.Trace(fmt.Errorf("Unknown error")),
fmt.Errorf("Unknown error"),
fmt.Errorf("Unknown error"),
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Message: "No list of errors",
}),
},
&googleapi.Error{
Message: "No list of errors",
},
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "conflict",
Message: "You already own this bucket. Please select another name.",
}},
}),
},
minio.BucketAlreadyOwnedByYou{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "conflict",
Message: "Sorry, that name is not available. Please try a different one.",
}},
}),
},
minio.BucketAlreadyExists{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "conflict",
}},
}),
},
minio.BucketNotEmpty{Bucket: "bucket"},
},
{
[]string{"bucket"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "notFound",
}},
}),
},
minio.BucketNotFound{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "notFound",
}},
}),
},
minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
@@ -329,22 +328,22 @@ func TestGCSToObjectError(t *testing.T) {
},
{
[]string{"bucket"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "invalid",
}},
}),
},
minio.BucketNameInvalid{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "forbidden",
}},
}),
},
minio.PrefixAccessDenied{
Bucket: "bucket",
Object: "object",
@@ -352,11 +351,11 @@ func TestGCSToObjectError(t *testing.T) {
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "keyInvalid",
}},
}),
},
minio.PrefixAccessDenied{
Bucket: "bucket",
Object: "object",
@@ -364,11 +363,11 @@ func TestGCSToObjectError(t *testing.T) {
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "required",
}},
}),
},
minio.PrefixAccessDenied{
Bucket: "bucket",
Object: "object",
@@ -376,11 +375,11 @@ func TestGCSToObjectError(t *testing.T) {
},
{
[]string{"bucket", "object"},
errors.Trace(&googleapi.Error{
&googleapi.Error{
Errors: []googleapi.ErrorItem{{
Reason: "unknown",
}},
}),
},
fmt.Errorf("Unsupported error reason: unknown"),
},
}
+29 -17
View File
@@ -33,8 +33,8 @@ import (
"github.com/joyent/triton-go/storage"
"github.com/minio/cli"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
)
@@ -75,7 +75,7 @@ ENVIRONMENT VARIABLES:
MINIO_DOMAIN: To enable virtual-host-style requests, set this value to Minio host domain name.
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -99,7 +99,7 @@ EXAMPLES:
4. Start minio gateway server for Manta Object Storage backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=manta_account_name
$ export MINIO_SECRET_KEY=manta_key_id
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}}
@@ -118,7 +118,7 @@ func mantaGatewayMain(ctx *cli.Context) {
// Validate gateway arguments.
host := ctx.Args().First()
// Validate gateway arguments.
minio.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
logger.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
minio.StartGateway(ctx, &Manta{host})
}
@@ -139,6 +139,7 @@ func (g *Manta) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, erro
var err error
var signer authentication.Signer
var endpoint = defaultMantaURL
ctx := context.Background()
if g.host != "" {
endpoint, _, err = minio.ParseGatewayEndpoint(g.host)
@@ -163,7 +164,8 @@ func (g *Manta) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, erro
}
signer, err = authentication.NewSSHAgentSigner(input)
if err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
} else {
var keyBytes []byte
@@ -200,7 +202,8 @@ func (g *Manta) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, erro
signer, err = authentication.NewPrivateKeySigner(input)
if err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
}
@@ -222,9 +225,9 @@ func (g *Manta) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, erro
}, nil
}
// Production - Manta is not production ready.
// Production - Manta is production ready.
func (g *Manta) Production() bool {
return false
return true
}
// tritonObjects - Implements Object layer for Triton Manta storage
@@ -352,7 +355,8 @@ func (t *tritonObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
if terrors.IsResourceNotFoundError(err) {
return result, nil
}
return result, errors.Trace(err)
logger.LogIf(ctx, err)
return result, err
}
for _, obj := range objs.Entries {
@@ -362,7 +366,8 @@ func (t *tritonObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
input.DirectoryName = path.Join(mantaRoot, bucket, prefix)
objs, err = t.client.Dir().List(ctx, input)
if err != nil {
return result, errors.Trace(err)
logger.LogIf(ctx, err)
return result, err
}
break
}
@@ -428,7 +433,8 @@ func (t *tritonObjects) ListObjectsV2(ctx context.Context, bucket, prefix, conti
if terrors.IsResourceNotFoundError(err) {
return result, nil
}
return result, errors.Trace(err)
logger.LogIf(ctx, err)
return result, err
}
for _, obj := range objs.Entries {
@@ -436,7 +442,8 @@ func (t *tritonObjects) ListObjectsV2(ctx context.Context, bucket, prefix, conti
input.DirectoryName = path.Join(mantaRoot, bucket, prefix)
objs, err = t.client.Dir().List(ctx, input)
if err != nil {
return result, errors.Trace(err)
logger.LogIf(ctx, err)
return result, err
}
break
}
@@ -479,7 +486,8 @@ func (t *tritonObjects) ListObjectsV2(ctx context.Context, bucket, prefix, conti
func (t *tritonObjects) GetObject(ctx context.Context, bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
// Start offset cannot be negative.
if startOffset < 0 {
return errors.Trace(fmt.Errorf("Unexpected error"))
logger.LogIf(ctx, fmt.Errorf("Unexpected error"))
return fmt.Errorf("Unexpected error")
}
output, err := t.client.Objects().Get(ctx, &storage.GetObjectInput{
@@ -555,11 +563,13 @@ func (t *tritonObjects) PutObject(ctx context.Context, bucket, object string, da
ObjectReader: dummySeeker{data},
ForceInsert: true,
}); err != nil {
return objInfo, errors.Trace(err)
logger.LogIf(ctx, err)
return objInfo, err
}
if err = data.Verify(); err != nil {
t.DeleteObject(ctx, bucket, object)
return objInfo, errors.Trace(err)
logger.LogIf(ctx, err)
return objInfo, err
}
return t.GetObjectInfo(ctx, bucket, object)
@@ -574,7 +584,8 @@ func (t *tritonObjects) CopyObject(ctx context.Context, srcBucket, srcObject, de
SourcePath: path.Join(mantaRoot, srcBucket, srcObject),
LinkPath: path.Join(mantaRoot, destBucket, destObject),
}); err != nil {
return objInfo, errors.Trace(err)
logger.LogIf(ctx, err)
return objInfo, err
}
return t.GetObjectInfo(ctx, destBucket, destObject)
@@ -587,7 +598,8 @@ func (t *tritonObjects) DeleteObject(ctx context.Context, bucket, object string)
if err := t.client.Objects().Delete(ctx, &storage.DeleteObjectInput{
ObjectPath: path.Join(mantaRoot, bucket, object),
}); err != nil {
return errors.Trace(err)
logger.LogIf(ctx, err)
return err
}
return nil
+2 -2
View File
@@ -51,7 +51,7 @@ ENVIRONMENT VARIABLES:
MINIO_BROWSER: To disable web browser access, set this value to "off".
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -70,7 +70,7 @@ EXAMPLES:
2. Start minio gateway server for NAS with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}} /shared/nasvol
+119 -81
View File
@@ -32,8 +32,8 @@ import (
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/policy"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
)
@@ -72,7 +72,7 @@ ENVIRONMENT VARIABLES:
MINIO_DOMAIN: To enable virtual-host-style requests, set this value to Minio host domain name.
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -90,7 +90,7 @@ EXAMPLES:
3. Start minio gateway server for Aliyun OSS backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}}
@@ -113,7 +113,7 @@ func ossGatewayMain(ctx *cli.Context) {
// Validate gateway arguments.
host := ctx.Args().First()
minio.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
logger.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
minio.StartGateway(ctx, &OSS{host})
}
@@ -149,9 +149,9 @@ func (g *OSS) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error)
}, nil
}
// Production - oss is not production ready yet.
// Production - oss is production ready.
func (g *OSS) Production() bool {
return false
return true
}
// appendS3MetaToOSSOptions converts metadata meant for S3 PUT/COPY
@@ -161,7 +161,7 @@ func (g *OSS) Production() bool {
// `X-Amz-Meta-` prefix and converted into `X-Oss-Meta-`.
//
// Header names are canonicalized as in http.Header.
func appendS3MetaToOSSOptions(opts []oss.Option, s3Metadata map[string]string) ([]oss.Option, error) {
func appendS3MetaToOSSOptions(ctx context.Context, opts []oss.Option, s3Metadata map[string]string) ([]oss.Option, error) {
if opts == nil {
opts = make([]oss.Option, 0, len(s3Metadata))
}
@@ -173,7 +173,8 @@ func appendS3MetaToOSSOptions(opts []oss.Option, s3Metadata map[string]string) (
metaKey := k[len("X-Amz-Meta-"):]
// NOTE(timonwong): OSS won't allow headers with underscore(_).
if strings.Contains(metaKey, "_") {
return nil, errors.Trace(minio.UnsupportedMetadata{})
logger.LogIf(ctx, minio.UnsupportedMetadata{})
return nil, minio.UnsupportedMetadata{}
}
opts = append(opts, oss.Meta(metaKey, v))
case k == "X-Amz-Acl":
@@ -271,15 +272,6 @@ func ossToObjectError(err error, params ...string) error {
return nil
}
e, ok := err.(*errors.Error)
if !ok {
// Code should be fixed if this function is called without doing errors.Trace()
// Else handling different situations in this function makes this function complicated.
minio.ErrorIf(err, "Expected type *Error")
return err
}
err = e.Cause
bucket := ""
object := ""
uploadID := ""
@@ -298,7 +290,7 @@ func ossToObjectError(err error, params ...string) error {
if !ok {
// We don't interpret non OSS errors. As oss errors will
// have StatusCode to help to convert to object errors.
return e
return err
}
switch ossErr.Code {
@@ -330,8 +322,7 @@ func ossToObjectError(err error, params ...string) error {
err = minio.InvalidPart{}
}
e.Cause = err
return e
return err
}
// ossObjects implements gateway for Aliyun Object Storage Service.
@@ -366,22 +357,26 @@ func ossIsValidBucketName(bucket string) bool {
// MakeBucketWithLocation creates a new container on OSS backend.
func (l *ossObjects) MakeBucketWithLocation(ctx context.Context, bucket, location string) error {
if !ossIsValidBucketName(bucket) {
return errors.Trace(minio.BucketNameInvalid{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNameInvalid{Bucket: bucket})
return minio.BucketNameInvalid{Bucket: bucket}
}
err := l.Client.CreateBucket(bucket)
return ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket)
}
// ossGeBucketInfo gets bucket metadata.
func ossGeBucketInfo(client *oss.Client, bucket string) (bi minio.BucketInfo, err error) {
func ossGeBucketInfo(ctx context.Context, client *oss.Client, bucket string) (bi minio.BucketInfo, err error) {
if !ossIsValidBucketName(bucket) {
return bi, errors.Trace(minio.BucketNameInvalid{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNameInvalid{Bucket: bucket})
return bi, minio.BucketNameInvalid{Bucket: bucket}
}
bgir, err := client.GetBucketInfo(bucket)
if err != nil {
return bi, ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return bi, ossToObjectError(err, bucket)
}
return minio.BucketInfo{
@@ -392,7 +387,7 @@ func ossGeBucketInfo(client *oss.Client, bucket string) (bi minio.BucketInfo, er
// GetBucketInfo gets bucket metadata.
func (l *ossObjects) GetBucketInfo(ctx context.Context, bucket string) (bi minio.BucketInfo, err error) {
return ossGeBucketInfo(l.Client, bucket)
return ossGeBucketInfo(ctx, l.Client, bucket)
}
// ListBuckets lists all OSS buckets.
@@ -401,7 +396,8 @@ func (l *ossObjects) ListBuckets(ctx context.Context) (buckets []minio.BucketInf
for {
lbr, err := l.Client.ListBuckets(marker)
if err != nil {
return nil, ossToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return nil, ossToObjectError(err)
}
for _, bi := range lbr.Buckets {
@@ -424,7 +420,8 @@ func (l *ossObjects) ListBuckets(ctx context.Context) (buckets []minio.BucketInf
func (l *ossObjects) DeleteBucket(ctx context.Context, bucket string) error {
err := l.Client.DeleteBucket(bucket)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket)
}
return nil
}
@@ -462,10 +459,11 @@ func fromOSSClientListObjectsResult(bucket string, lor oss.ListObjectsResult) mi
}
// ossListObjects lists all blobs in OSS bucket filtered by prefix.
func ossListObjects(client *oss.Client, bucket, prefix, marker, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, err error) {
func ossListObjects(ctx context.Context, client *oss.Client, bucket, prefix, marker, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, err error) {
buck, err := client.Bucket(bucket)
if err != nil {
return loi, ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return loi, ossToObjectError(err, bucket)
}
// maxKeys should default to 1000 or less.
@@ -475,19 +473,20 @@ func ossListObjects(client *oss.Client, bucket, prefix, marker, delimiter string
lor, err := buck.ListObjects(oss.Prefix(prefix), oss.Marker(marker), oss.Delimiter(delimiter), oss.MaxKeys(maxKeys))
if err != nil {
return loi, ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return loi, ossToObjectError(err, bucket)
}
return fromOSSClientListObjectsResult(bucket, lor), nil
}
// ossListObjectsV2 lists all blobs in OSS bucket filtered by prefix.
func ossListObjectsV2(client *oss.Client, bucket, prefix, continuationToken, delimiter string, maxKeys int,
func ossListObjectsV2(ctx context.Context, client *oss.Client, bucket, prefix, continuationToken, delimiter string, maxKeys int,
fetchOwner bool, startAfter string) (loi minio.ListObjectsV2Info, err error) {
// fetchOwner and startAfter are not supported and unused.
marker := continuationToken
resultV1, err := ossListObjects(client, bucket, prefix, marker, delimiter, maxKeys)
resultV1, err := ossListObjects(ctx, client, bucket, prefix, marker, delimiter, maxKeys)
if err != nil {
return loi, err
}
@@ -503,13 +502,13 @@ func ossListObjectsV2(client *oss.Client, bucket, prefix, continuationToken, del
// ListObjects lists all blobs in OSS bucket filtered by prefix.
func (l *ossObjects) ListObjects(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, err error) {
return ossListObjects(l.Client, bucket, prefix, marker, delimiter, maxKeys)
return ossListObjects(ctx, l.Client, bucket, prefix, marker, delimiter, maxKeys)
}
// ListObjectsV2 lists all blobs in OSS bucket filtered by prefix
func (l *ossObjects) ListObjectsV2(ctx context.Context, bucket, prefix, continuationToken, delimiter string, maxKeys int,
fetchOwner bool, startAfter string) (loi minio.ListObjectsV2Info, err error) {
return ossListObjectsV2(l.Client, bucket, prefix, continuationToken, delimiter, maxKeys, fetchOwner, startAfter)
return ossListObjectsV2(ctx, l.Client, bucket, prefix, continuationToken, delimiter, maxKeys, fetchOwner, startAfter)
}
// ossGetObject reads an object on OSS. Supports additional
@@ -518,14 +517,16 @@ func (l *ossObjects) ListObjectsV2(ctx context.Context, bucket, prefix, continua
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func ossGetObject(client *oss.Client, bucket, key string, startOffset, length int64, writer io.Writer, etag string) error {
func ossGetObject(ctx context.Context, client *oss.Client, bucket, key string, startOffset, length int64, writer io.Writer, etag string) error {
if length < 0 && length != -1 {
return ossToObjectError(errors.Trace(fmt.Errorf("Invalid argument")), bucket, key)
logger.LogIf(ctx, fmt.Errorf("Invalid argument"))
return ossToObjectError(fmt.Errorf("Invalid argument"), bucket, key)
}
bkt, err := client.Bucket(bucket)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, key)
}
var opts []oss.Option
@@ -535,12 +536,14 @@ func ossGetObject(client *oss.Client, bucket, key string, startOffset, length in
object, err := bkt.GetObject(key, opts...)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, key)
}
defer object.Close()
if _, err := io.Copy(writer, object); err != nil {
return ossToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, key)
}
return nil
}
@@ -552,7 +555,7 @@ func ossGetObject(client *oss.Client, bucket, key string, startOffset, length in
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (l *ossObjects) GetObject(ctx context.Context, bucket, key string, startOffset, length int64, writer io.Writer, etag string) error {
return ossGetObject(l.Client, bucket, key, startOffset, length, writer, etag)
return ossGetObject(ctx, l.Client, bucket, key, startOffset, length, writer, etag)
}
func translatePlainError(err error) error {
@@ -569,15 +572,17 @@ func translatePlainError(err error) error {
}
// ossGetObjectInfo reads object info and replies back ObjectInfo.
func ossGetObjectInfo(client *oss.Client, bucket, object string) (objInfo minio.ObjectInfo, err error) {
func ossGetObjectInfo(ctx context.Context, client *oss.Client, bucket, object string) (objInfo minio.ObjectInfo, err error) {
bkt, err := client.Bucket(bucket)
if err != nil {
return objInfo, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, ossToObjectError(err, bucket, object)
}
header, err := bkt.GetObjectDetailedMeta(object)
if err != nil {
return objInfo, ossToObjectError(errors.Trace(translatePlainError(err)), bucket, object)
logger.LogIf(ctx, translatePlainError(err))
return objInfo, ossToObjectError(translatePlainError(err), bucket, object)
}
// Build S3 metadata from OSS metadata
@@ -600,40 +605,43 @@ func ossGetObjectInfo(client *oss.Client, bucket, object string) (objInfo minio.
// GetObjectInfo reads object info and replies back ObjectInfo.
func (l *ossObjects) GetObjectInfo(ctx context.Context, bucket, object string) (objInfo minio.ObjectInfo, err error) {
return ossGetObjectInfo(l.Client, bucket, object)
return ossGetObjectInfo(ctx, l.Client, bucket, object)
}
// ossPutObject creates a new object with the incoming data.
func ossPutObject(client *oss.Client, bucket, object string, data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
func ossPutObject(ctx context.Context, client *oss.Client, bucket, object string, data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
bkt, err := client.Bucket(bucket)
if err != nil {
return objInfo, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, ossToObjectError(err, bucket, object)
}
// Build OSS metadata
opts, err := appendS3MetaToOSSOptions(nil, metadata)
opts, err := appendS3MetaToOSSOptions(ctx, nil, metadata)
if err != nil {
return objInfo, ossToObjectError(err, bucket, object)
}
err = bkt.PutObject(object, data, opts...)
if err != nil {
return objInfo, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, ossToObjectError(err, bucket, object)
}
return ossGetObjectInfo(client, bucket, object)
return ossGetObjectInfo(ctx, client, bucket, object)
}
// PutObject creates a new object with the incoming data.
func (l *ossObjects) PutObject(ctx context.Context, bucket, object string, data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
return ossPutObject(l.Client, bucket, object, data, metadata)
return ossPutObject(ctx, l.Client, bucket, object, data, metadata)
}
// CopyObject copies an object from source bucket to a destination bucket.
func (l *ossObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject string, srcInfo minio.ObjectInfo) (objInfo minio.ObjectInfo, err error) {
bkt, err := l.Client.Bucket(srcBucket)
if err != nil {
return objInfo, ossToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return objInfo, ossToObjectError(err, srcBucket, srcObject)
}
opts := make([]oss.Option, 0, len(srcInfo.UserDefined)+1)
@@ -644,13 +652,14 @@ func (l *ossObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
opts = append(opts, oss.MetadataDirective(oss.MetaReplace))
// Build OSS metadata
opts, err = appendS3MetaToOSSOptions(opts, srcInfo.UserDefined)
opts, err = appendS3MetaToOSSOptions(ctx, opts, srcInfo.UserDefined)
if err != nil {
return objInfo, ossToObjectError(err, srcBucket, srcObject)
}
if _, err = bkt.CopyObjectTo(dstBucket, dstObject, srcObject, opts...); err != nil {
return objInfo, ossToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return objInfo, ossToObjectError(err, srcBucket, srcObject)
}
return l.GetObjectInfo(ctx, dstBucket, dstObject)
}
@@ -659,12 +668,14 @@ func (l *ossObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
func (l *ossObjects) DeleteObject(ctx context.Context, bucket, object string) error {
bkt, err := l.Client.Bucket(bucket)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, object)
}
err = bkt.DeleteObject(object)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, object)
}
return nil
}
@@ -701,13 +712,15 @@ func fromOSSClientListMultipartsInfo(lmur oss.ListMultipartUploadResult) minio.L
func (l *ossObjects) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (lmi minio.ListMultipartsInfo, err error) {
bkt, err := l.Client.Bucket(bucket)
if err != nil {
return lmi, ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return lmi, ossToObjectError(err, bucket)
}
lmur, err := bkt.ListMultipartUploads(oss.Prefix(prefix), oss.KeyMarker(keyMarker), oss.UploadIDMarker(uploadIDMarker),
oss.Delimiter(delimiter), oss.MaxUploads(maxUploads))
if err != nil {
return lmi, ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return lmi, ossToObjectError(err, bucket)
}
return fromOSSClientListMultipartsInfo(lmur), nil
@@ -717,18 +730,20 @@ func (l *ossObjects) ListMultipartUploads(ctx context.Context, bucket, prefix, k
func (l *ossObjects) NewMultipartUpload(ctx context.Context, bucket, object string, metadata map[string]string) (uploadID string, err error) {
bkt, err := l.Client.Bucket(bucket)
if err != nil {
return uploadID, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return uploadID, ossToObjectError(err, bucket, object)
}
// Build OSS metadata
opts, err := appendS3MetaToOSSOptions(nil, metadata)
opts, err := appendS3MetaToOSSOptions(ctx, nil, metadata)
if err != nil {
return uploadID, ossToObjectError(err, bucket, object)
}
lmur, err := bkt.InitiateMultipartUpload(object, opts...)
if err != nil {
return uploadID, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return uploadID, ossToObjectError(err, bucket, object)
}
return lmur.UploadID, nil
@@ -738,7 +753,8 @@ func (l *ossObjects) NewMultipartUpload(ctx context.Context, bucket, object stri
func (l *ossObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID string, partID int, data *hash.Reader) (pi minio.PartInfo, err error) {
bkt, err := l.Client.Bucket(bucket)
if err != nil {
return pi, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return pi, ossToObjectError(err, bucket, object)
}
imur := oss.InitiateMultipartUploadResult{
@@ -749,7 +765,8 @@ func (l *ossObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID
size := data.Size()
up, err := bkt.UploadPart(imur, data, size, partID)
if err != nil {
return pi, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return pi, ossToObjectError(err, bucket, object)
}
return minio.PartInfo{
@@ -820,11 +837,12 @@ func (l *ossObjects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, d
bkt, err := l.Client.Bucket(destBucket)
if err != nil {
return p, ossToObjectError(errors.Trace(err), destBucket)
logger.LogIf(ctx, err)
return p, ossToObjectError(err, destBucket)
}
// Build OSS metadata
opts, err := appendS3MetaToOSSOptions(nil, srcInfo.UserDefined)
opts, err := appendS3MetaToOSSOptions(ctx, nil, srcInfo.UserDefined)
if err != nil {
return p, ossToObjectError(err, srcBucket, srcObject)
}
@@ -835,7 +853,8 @@ func (l *ossObjects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, d
}, srcBucket, srcObject, startOffset, length, partID, opts...)
if err != nil {
return p, ossToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return p, ossToObjectError(err, srcBucket, srcObject)
}
p.PartNumber = completePart.PartNumber
@@ -847,7 +866,8 @@ func (l *ossObjects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, d
func (l *ossObjects) ListObjectParts(ctx context.Context, bucket, object, uploadID string, partNumberMarker, maxParts int) (lpi minio.ListPartsInfo, err error) {
lupr, err := ossListObjectParts(l.Client, bucket, object, uploadID, partNumberMarker, maxParts)
if err != nil {
return lpi, ossToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return lpi, ossToObjectError(err, bucket, object, uploadID)
}
return fromOSSClientListPartsInfo(lupr, partNumberMarker), nil
@@ -857,7 +877,8 @@ func (l *ossObjects) ListObjectParts(ctx context.Context, bucket, object, upload
func (l *ossObjects) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string) error {
bkt, err := l.Client.Bucket(bucket)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, object)
}
err = bkt.AbortMultipartUpload(oss.InitiateMultipartUploadResult{
@@ -866,7 +887,8 @@ func (l *ossObjects) AbortMultipartUpload(ctx context.Context, bucket, object, u
UploadID: uploadID,
})
if err != nil {
return ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket, object)
}
return nil
}
@@ -876,7 +898,8 @@ func (l *ossObjects) CompleteMultipartUpload(ctx context.Context, bucket, object
client := l.Client
bkt, err := client.Bucket(bucket)
if err != nil {
return oi, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, ossToObjectError(err, bucket, object)
}
// Error out if uploadedParts except last part sizing < 5MiB.
@@ -886,7 +909,8 @@ func (l *ossObjects) CompleteMultipartUpload(ctx context.Context, bucket, object
for lupr.IsTruncated {
lupr, err = ossListObjectParts(client, bucket, object, uploadID, partNumberMarker, ossMaxParts)
if err != nil {
return oi, ossToObjectError(errors.Trace(err), bucket, object, uploadID)
logger.LogIf(ctx, err)
return oi, ossToObjectError(err, bucket, object, uploadID)
}
uploadedParts := lupr.UploadedParts
@@ -900,11 +924,16 @@ func (l *ossObjects) CompleteMultipartUpload(ctx context.Context, bucket, object
for _, part := range uploadedParts {
if part.Size < ossS3MinPartSize {
return oi, errors.Trace(minio.PartTooSmall{
logger.LogIf(ctx, minio.PartTooSmall{
PartNumber: part.PartNumber,
PartSize: int64(part.Size),
PartETag: minio.ToS3ETag(part.ETag),
})
return oi, minio.PartTooSmall{
PartNumber: part.PartNumber,
PartSize: int64(part.Size),
PartETag: minio.ToS3ETag(part.ETag),
}
}
}
@@ -926,7 +955,8 @@ func (l *ossObjects) CompleteMultipartUpload(ctx context.Context, bucket, object
_, err = bkt.CompleteMultipartUpload(imur, parts)
if err != nil {
return oi, ossToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, ossToObjectError(err, bucket, object)
}
return l.GetObjectInfo(ctx, bucket, object)
@@ -940,13 +970,15 @@ func (l *ossObjects) CompleteMultipartUpload(ctx context.Context, bucket, object
func (l *ossObjects) SetBucketPolicy(ctx context.Context, bucket string, policyInfo policy.BucketAccessPolicy) error {
bucketPolicies := policy.GetPolicies(policyInfo.Statements, bucket, "")
if len(bucketPolicies) != 1 {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
prefix := bucket + "/*" // For all objects inside the bucket.
for policyPrefix, bucketPolicy := range bucketPolicies {
if policyPrefix != prefix {
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
var acl oss.ACLType
@@ -958,12 +990,14 @@ func (l *ossObjects) SetBucketPolicy(ctx context.Context, bucket string, policyI
case policy.BucketPolicyReadWrite:
acl = oss.ACLPublicReadWrite
default:
return errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return minio.NotImplemented{}
}
err := l.Client.SetBucketACL(bucket, acl)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket)
}
}
@@ -974,20 +1008,23 @@ func (l *ossObjects) SetBucketPolicy(ctx context.Context, bucket string, policyI
func (l *ossObjects) GetBucketPolicy(ctx context.Context, bucket string) (policy.BucketAccessPolicy, error) {
result, err := l.Client.GetBucketACL(bucket)
if err != nil {
return policy.BucketAccessPolicy{}, ossToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return policy.BucketAccessPolicy{}, ossToObjectError(err)
}
policyInfo := policy.BucketAccessPolicy{Version: "2012-10-17"}
switch result.ACL {
case string(oss.ACLPrivate):
// By default, all buckets starts with a "private" policy.
return policy.BucketAccessPolicy{}, ossToObjectError(errors.Trace(minio.PolicyNotFound{}), bucket)
logger.LogIf(ctx, minio.PolicyNotFound{})
return policy.BucketAccessPolicy{}, ossToObjectError(minio.PolicyNotFound{}, bucket)
case string(oss.ACLPublicRead):
policyInfo.Statements = policy.SetPolicy(policyInfo.Statements, policy.BucketPolicyReadOnly, bucket, "")
case string(oss.ACLPublicReadWrite):
policyInfo.Statements = policy.SetPolicy(policyInfo.Statements, policy.BucketPolicyReadWrite, bucket, "")
default:
return policy.BucketAccessPolicy{}, errors.Trace(minio.NotImplemented{})
logger.LogIf(ctx, minio.NotImplemented{})
return policy.BucketAccessPolicy{}, minio.NotImplemented{}
}
return policyInfo, nil
@@ -997,7 +1034,8 @@ func (l *ossObjects) GetBucketPolicy(ctx context.Context, bucket string) (policy
func (l *ossObjects) DeleteBucketPolicy(ctx context.Context, bucket string) error {
err := l.Client.SetBucketACL(bucket, oss.ACLPrivate)
if err != nil {
return ossToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return ossToObjectError(err, bucket)
}
return nil
}
+8 -8
View File
@@ -17,6 +17,7 @@
package oss
import (
"context"
"fmt"
"net/http"
"reflect"
@@ -25,13 +26,12 @@ import (
"github.com/aliyun/aliyun-oss-go-sdk/oss"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/pkg/errors"
)
func ossErrResponse(code string) error {
return errors.Trace(oss.ServiceError{
return oss.ServiceError{
Code: code,
})
}
}
func TestOSSToObjectError(t *testing.T) {
@@ -103,8 +103,8 @@ func TestOSSToObjectError(t *testing.T) {
for i, tc := range testCases {
actualErr := ossToObjectError(tc.inputErr, tc.bucket, tc.object)
if e, ok := actualErr.(*errors.Error); ok && e.Cause != tc.expectedErr {
t.Errorf("Test case %d: Expected error '%v' but received error '%v'", i+1, tc.expectedErr, e.Cause)
if actualErr != nil && tc.expectedErr != nil && actualErr.Error() != tc.expectedErr.Error() {
t.Errorf("Test case %d: Expected error '%v' but received error '%v'", i+1, tc.expectedErr, actualErr)
}
}
}
@@ -116,8 +116,8 @@ func TestS3MetaToOSSOptions(t *testing.T) {
headers = map[string]string{
"x-amz-meta-invalid_meta": "value",
}
_, err = appendS3MetaToOSSOptions(nil, headers)
if err = errors.Cause(err); err != nil {
_, err = appendS3MetaToOSSOptions(context.Background(), nil, headers)
if err != nil {
if _, ok := err.(minio.UnsupportedMetadata); !ok {
t.Fatalf("Test failed with unexpected error %s, expected UnsupportedMetadata", err)
}
@@ -133,7 +133,7 @@ func TestS3MetaToOSSOptions(t *testing.T) {
"X-Amz-Meta-X-Amz-Matdesc": "{}",
"X-Amz-Meta-X-Amz-Iv": "eWmyryl8kq+EVnnsE7jpOg==",
}
opts, err := appendS3MetaToOSSOptions(nil, headers)
opts, err := appendS3MetaToOSSOptions(context.Background(), nil, headers)
if err != nil {
t.Fatalf("Test failed, with %s", err)
}
+66 -33
View File
@@ -18,14 +18,15 @@ package s3
import (
"context"
"encoding/json"
"io"
"github.com/minio/cli"
miniogo "github.com/minio/minio-go"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
minio "github.com/minio/minio/cmd"
@@ -57,7 +58,7 @@ ENVIRONMENT VARIABLES:
MINIO_BROWSER: To disable web browser access, set this value to "off".
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -81,7 +82,7 @@ EXAMPLES:
3. Start minio gateway server for AWS S3 backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}}
@@ -101,7 +102,7 @@ func s3GatewayMain(ctx *cli.Context) {
// Validate gateway arguments.
host := ctx.Args().First()
// Validate gateway arguments.
minio.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
logger.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
minio.StartGateway(ctx, &S3{host})
}
@@ -147,9 +148,9 @@ func (g *S3) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error)
}, nil
}
// Production - s3 gateway is not production ready.
// Production - s3 gateway is production ready.
func (g *S3) Production() bool {
return false
return true
}
// s3Objects implements gateway for Minio and S3 compatible object storage servers.
@@ -173,7 +174,8 @@ func (l *s3Objects) StorageInfo(ctx context.Context) (si minio.StorageInfo) {
func (l *s3Objects) MakeBucketWithLocation(ctx context.Context, bucket, location string) error {
err := l.Client.MakeBucket(bucket, location)
if err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket)
}
return err
}
@@ -188,12 +190,14 @@ func (l *s3Objects) GetBucketInfo(ctx context.Context, bucket string) (bi minio.
// access to these buckets.
// Ref - http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
if s3utils.CheckValidBucketName(bucket) != nil {
return bi, errors.Trace(minio.BucketNameInvalid{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNameInvalid{Bucket: bucket})
return bi, minio.BucketNameInvalid{Bucket: bucket}
}
buckets, err := l.Client.ListBuckets()
if err != nil {
return bi, minio.ErrorRespToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return bi, minio.ErrorRespToObjectError(err, bucket)
}
for _, bi := range buckets {
@@ -207,14 +211,16 @@ func (l *s3Objects) GetBucketInfo(ctx context.Context, bucket string) (bi minio.
}, nil
}
return bi, errors.Trace(minio.BucketNotFound{Bucket: bucket})
logger.LogIf(ctx, minio.BucketNotFound{Bucket: bucket})
return bi, minio.BucketNotFound{Bucket: bucket}
}
// ListBuckets lists all S3 buckets
func (l *s3Objects) ListBuckets(ctx context.Context) ([]minio.BucketInfo, error) {
buckets, err := l.Client.ListBuckets()
if err != nil {
return nil, minio.ErrorRespToObjectError(errors.Trace(err))
logger.LogIf(ctx, err)
return nil, minio.ErrorRespToObjectError(err)
}
b := make([]minio.BucketInfo, len(buckets))
@@ -232,7 +238,8 @@ func (l *s3Objects) ListBuckets(ctx context.Context) ([]minio.BucketInfo, error)
func (l *s3Objects) DeleteBucket(ctx context.Context, bucket string) error {
err := l.Client.RemoveBucket(bucket)
if err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket)
}
return nil
}
@@ -241,7 +248,8 @@ func (l *s3Objects) DeleteBucket(ctx context.Context, bucket string) error {
func (l *s3Objects) ListObjects(ctx context.Context, bucket string, prefix string, marker string, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, e error) {
result, err := l.Client.ListObjects(bucket, prefix, marker, delimiter, maxKeys)
if err != nil {
return loi, minio.ErrorRespToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return loi, minio.ErrorRespToObjectError(err, bucket)
}
return minio.FromMinioClientListBucketResult(bucket, result), nil
@@ -251,7 +259,8 @@ func (l *s3Objects) ListObjects(ctx context.Context, bucket string, prefix strin
func (l *s3Objects) ListObjectsV2(ctx context.Context, bucket, prefix, continuationToken, delimiter string, maxKeys int, fetchOwner bool, startAfter string) (loi minio.ListObjectsV2Info, e error) {
result, err := l.Client.ListObjectsV2(bucket, prefix, continuationToken, fetchOwner, delimiter, maxKeys)
if err != nil {
return loi, minio.ErrorRespToObjectError(errors.Trace(err), bucket)
logger.LogIf(ctx, err)
return loi, minio.ErrorRespToObjectError(err, bucket)
}
return minio.FromMinioClientListBucketV2Result(bucket, result), nil
@@ -265,23 +274,27 @@ func (l *s3Objects) ListObjectsV2(ctx context.Context, bucket, prefix, continuat
// length indicates the total length of the object.
func (l *s3Objects) GetObject(ctx context.Context, bucket string, key string, startOffset int64, length int64, writer io.Writer, etag string) error {
if length < 0 && length != -1 {
return minio.ErrorRespToObjectError(errors.Trace(minio.InvalidRange{}), bucket, key)
logger.LogIf(ctx, minio.InvalidRange{})
return minio.ErrorRespToObjectError(minio.InvalidRange{}, bucket, key)
}
opts := miniogo.GetObjectOptions{}
if startOffset >= 0 && length >= 0 {
if err := opts.SetRange(startOffset, startOffset+length-1); err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, key)
}
}
object, _, err := l.Client.GetObject(bucket, key, opts)
if err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, key)
}
defer object.Close()
if _, err := io.Copy(writer, object); err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, key)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, key)
}
return nil
}
@@ -290,7 +303,8 @@ func (l *s3Objects) GetObject(ctx context.Context, bucket string, key string, st
func (l *s3Objects) GetObjectInfo(ctx context.Context, bucket string, object string) (objInfo minio.ObjectInfo, err error) {
oi, err := l.Client.StatObject(bucket, object, miniogo.StatObjectOptions{})
if err != nil {
return minio.ObjectInfo{}, minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, minio.ErrorRespToObjectError(err, bucket, object)
}
return minio.FromMinioClientObjectInfo(bucket, oi), nil
@@ -300,7 +314,8 @@ func (l *s3Objects) GetObjectInfo(ctx context.Context, bucket string, object str
func (l *s3Objects) PutObject(ctx context.Context, bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
oi, err := l.Client.PutObject(bucket, object, data, data.Size(), data.MD5Base64String(), data.SHA256HexString(), minio.ToMinioClientMetadata(metadata))
if err != nil {
return objInfo, minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return objInfo, minio.ErrorRespToObjectError(err, bucket, object)
}
return minio.FromMinioClientObjectInfo(bucket, oi), nil
@@ -315,7 +330,8 @@ func (l *s3Objects) CopyObject(ctx context.Context, srcBucket string, srcObject
srcInfo.UserDefined["x-amz-metadata-directive"] = "REPLACE"
srcInfo.UserDefined["x-amz-copy-source-if-match"] = srcInfo.ETag
if _, err = l.Client.CopyObject(srcBucket, srcObject, dstBucket, dstObject, srcInfo.UserDefined); err != nil {
return objInfo, minio.ErrorRespToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return objInfo, minio.ErrorRespToObjectError(err, srcBucket, srcObject)
}
return l.GetObjectInfo(ctx, dstBucket, dstObject)
}
@@ -324,7 +340,8 @@ func (l *s3Objects) CopyObject(ctx context.Context, srcBucket string, srcObject
func (l *s3Objects) DeleteObject(ctx context.Context, bucket string, object string) error {
err := l.Client.RemoveObject(bucket, object)
if err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, object)
}
return nil
@@ -346,7 +363,8 @@ func (l *s3Objects) NewMultipartUpload(ctx context.Context, bucket string, objec
opts := miniogo.PutObjectOptions{UserMetadata: metadata}
uploadID, err = l.Client.NewMultipartUpload(bucket, object, opts)
if err != nil {
return uploadID, minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return uploadID, minio.ErrorRespToObjectError(err, bucket, object)
}
return uploadID, nil
}
@@ -355,7 +373,8 @@ func (l *s3Objects) NewMultipartUpload(ctx context.Context, bucket string, objec
func (l *s3Objects) PutObjectPart(ctx context.Context, bucket string, object string, uploadID string, partID int, data *hash.Reader) (pi minio.PartInfo, e error) {
info, err := l.Client.PutObjectPart(bucket, object, uploadID, partID, data, data.Size(), data.MD5Base64String(), data.SHA256HexString())
if err != nil {
return pi, minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return pi, minio.ErrorRespToObjectError(err, bucket, object)
}
return minio.FromMinioClientObjectPart(info), nil
@@ -372,7 +391,8 @@ func (l *s3Objects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, de
completePart, err := l.Client.CopyObjectPart(srcBucket, srcObject, destBucket, destObject,
uploadID, partID, startOffset, length, srcInfo.UserDefined)
if err != nil {
return p, minio.ErrorRespToObjectError(errors.Trace(err), srcBucket, srcObject)
logger.LogIf(ctx, err)
return p, minio.ErrorRespToObjectError(err, srcBucket, srcObject)
}
p.PartNumber = completePart.PartNumber
p.ETag = completePart.ETag
@@ -392,14 +412,16 @@ func (l *s3Objects) ListObjectParts(ctx context.Context, bucket string, object s
// AbortMultipartUpload aborts a ongoing multipart upload
func (l *s3Objects) AbortMultipartUpload(ctx context.Context, bucket string, object string, uploadID string) error {
err := l.Client.AbortMultipartUpload(bucket, object, uploadID)
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, object)
}
// CompleteMultipartUpload completes ongoing multipart upload and finalizes object
func (l *s3Objects) CompleteMultipartUpload(ctx context.Context, bucket string, object string, uploadID string, uploadedParts []minio.CompletePart) (oi minio.ObjectInfo, e error) {
err := l.Client.CompleteMultipartUpload(bucket, object, uploadID, minio.ToMinioClientCompleteParts(uploadedParts))
if err != nil {
return oi, minio.ErrorRespToObjectError(errors.Trace(err), bucket, object)
logger.LogIf(ctx, err)
return oi, minio.ErrorRespToObjectError(err, bucket, object)
}
return l.GetObjectInfo(ctx, bucket, object)
@@ -407,8 +429,13 @@ func (l *s3Objects) CompleteMultipartUpload(ctx context.Context, bucket string,
// SetBucketPolicy sets policy on bucket
func (l *s3Objects) SetBucketPolicy(ctx context.Context, bucket string, policyInfo policy.BucketAccessPolicy) error {
if err := l.Client.PutBucketPolicy(bucket, policyInfo); err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, "")
data, err := json.Marshal(&policyInfo)
if err != nil {
return err
}
if err := l.Client.SetBucketPolicy(bucket, string(data)); err != nil {
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, "")
}
return nil
@@ -416,17 +443,23 @@ func (l *s3Objects) SetBucketPolicy(ctx context.Context, bucket string, policyIn
// GetBucketPolicy will get policy on bucket
func (l *s3Objects) GetBucketPolicy(ctx context.Context, bucket string) (policy.BucketAccessPolicy, error) {
policyInfo, err := l.Client.GetBucketPolicy(bucket)
data, err := l.Client.GetBucketPolicy(bucket)
if err != nil {
return policy.BucketAccessPolicy{}, minio.ErrorRespToObjectError(errors.Trace(err), bucket, "")
logger.LogIf(ctx, err)
return policy.BucketAccessPolicy{}, minio.ErrorRespToObjectError(err, bucket, "")
}
var policyInfo policy.BucketAccessPolicy
if err = json.Unmarshal([]byte(data), &policyInfo); err != nil {
return policyInfo, err
}
return policyInfo, nil
}
// DeleteBucketPolicy deletes all policies on bucket
func (l *s3Objects) DeleteBucketPolicy(ctx context.Context, bucket string) error {
if err := l.Client.PutBucketPolicy(bucket, policy.BucketAccessPolicy{}); err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, "")
if err := l.Client.SetBucketPolicy(bucket, ""); err != nil {
logger.LogIf(ctx, err)
return minio.ErrorRespToObjectError(err, bucket, "")
}
return nil
}
+3 -10
View File
@@ -21,7 +21,6 @@ import (
"testing"
miniogo "github.com/minio/minio-go"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
minio "github.com/minio/minio/cmd"
@@ -110,18 +109,12 @@ func TestS3ToObjectError(t *testing.T) {
inputErr: fmt.Errorf("not a minio.ErrorResponse"),
expectedErr: fmt.Errorf("not a minio.ErrorResponse"),
},
// Special test case for error value that is not of
// type (*Error)
{
inputErr: fmt.Errorf("not a *Error"),
expectedErr: fmt.Errorf("not a *Error"),
},
}
for i, tc := range testCases {
actualErr := minio.ErrorRespToObjectError(errors.Trace(tc.inputErr), tc.bucket, tc.object)
if e, ok := actualErr.(*errors.Error); ok && e.Cause.Error() != tc.expectedErr.Error() {
t.Errorf("Test case %d: Expected error %v but received error %v", i+1, tc.expectedErr, e)
actualErr := minio.ErrorRespToObjectError(tc.inputErr, tc.bucket, tc.object)
if actualErr != nil && tc.expectedErr != nil && actualErr.Error() != tc.expectedErr.Error() {
t.Errorf("Test case %d: Expected error %v but received error %v", i+1, tc.expectedErr, actualErr)
}
}
}
+56 -41
View File
@@ -24,7 +24,6 @@ import (
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
@@ -38,8 +37,8 @@ import (
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/set"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
)
@@ -74,7 +73,7 @@ ENVIRONMENT VARIABLES: (Default values in parenthesis)
MINIO_BROWSER: To disable web browser access, set this value to "off".
CACHE:
MINIO_CACHE_DRIVES: List of cache drives delimited by ";".
MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
MINIO_CACHE_EXPIRY: Cache expiry duration in days.
@@ -92,7 +91,7 @@ EXAMPLES:
$ {{.HelpName}}
2. Start minio gateway server for Sia backend with edge caching enabled.
$ export MINIO_CACHE_DRIVES="/home/drive1;/home/drive2;/home/drive3;/home/drive4"
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ {{.HelpName}}
@@ -112,7 +111,7 @@ func siaGatewayMain(ctx *cli.Context) {
// Validate gateway arguments.
host := ctx.Args().First()
// Validate gateway arguments.
minio.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
logger.FatalIf(minio.ValidateGatewayArguments(ctx.GlobalString("address"), host), "Invalid argument")
minio.StartGateway(ctx, &Sia{host})
}
@@ -164,9 +163,11 @@ func (g *Sia) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error)
colorBlue := color.New(color.FgBlue).SprintfFunc()
colorBold := color.New(color.Bold).SprintFunc()
log.Println(colorBlue("\nSia Gateway Configuration:"))
log.Println(colorBlue(" Sia Daemon API Address:") + colorBold(fmt.Sprintf(" %s\n", sia.Address)))
log.Println(colorBlue(" Sia Temp Directory:") + colorBold(fmt.Sprintf(" %s\n", sia.TempDir)))
formatStr := "%" + fmt.Sprintf("%ds", len(sia.Address)+7)
logger.StartupMessage(colorBlue("\nSia Configuration:"))
logger.StartupMessage(colorBlue(" API Address:") + colorBold(fmt.Sprintf(formatStr, sia.Address)))
logger.StartupMessage(colorBlue(" Staging Directory:") + colorBold(fmt.Sprintf(" %s", sia.TempDir)))
return sia, nil
}
@@ -217,10 +218,11 @@ func (s MethodNotSupported) Error() string {
// apiGet wraps a GET request with a status code check, such that if the GET does
// not return 2xx, the error will be read and returned. The response body is
// not closed.
func apiGet(addr, call, apiPassword string) (*http.Response, error) {
func apiGet(ctx context.Context, addr, call, apiPassword string) (*http.Response, error) {
req, err := http.NewRequest("GET", "http://"+addr+call, nil)
if err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
req.Header.Set("User-Agent", "Sia-Agent")
if apiPassword != "" {
@@ -228,15 +230,18 @@ func apiGet(addr, call, apiPassword string) (*http.Response, error) {
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
if resp.StatusCode == http.StatusNotFound {
resp.Body.Close()
logger.LogIf(ctx, MethodNotSupported{call})
return nil, MethodNotSupported{call}
}
if non2xx(resp.StatusCode) {
err := decodeError(resp)
resp.Body.Close()
logger.LogIf(ctx, err)
return nil, err
}
return resp, nil
@@ -245,7 +250,7 @@ func apiGet(addr, call, apiPassword string) (*http.Response, error) {
// apiPost wraps a POST request with a status code check, such that if the POST
// does not return 2xx, the error will be read and returned. The response body
// is not closed.
func apiPost(addr, call, vals, apiPassword string) (*http.Response, error) {
func apiPost(ctx context.Context, addr, call, vals, apiPassword string) (*http.Response, error) {
req, err := http.NewRequest("POST", "http://"+addr+call, strings.NewReader(vals))
if err != nil {
return nil, err
@@ -257,7 +262,8 @@ func apiPost(addr, call, vals, apiPassword string) (*http.Response, error) {
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, err
}
if resp.StatusCode == http.StatusNotFound {
@@ -275,8 +281,8 @@ func apiPost(addr, call, vals, apiPassword string) (*http.Response, error) {
// post makes an API call and discards the response. An error is returned if
// the response status is not 2xx.
func post(addr, call, vals, apiPassword string) error {
resp, err := apiPost(addr, call, vals, apiPassword)
func post(ctx context.Context, addr, call, vals, apiPassword string) error {
resp, err := apiPost(ctx, addr, call, vals, apiPassword)
if err != nil {
return err
}
@@ -285,24 +291,26 @@ func post(addr, call, vals, apiPassword string) error {
}
// list makes a lists all the uploaded files, decodes the json response.
func list(addr string, apiPassword string, obj *renterFiles) error {
resp, err := apiGet(addr, "/renter/files", apiPassword)
func list(ctx context.Context, addr string, apiPassword string, obj *renterFiles) error {
resp, err := apiGet(ctx, addr, "/renter/files", apiPassword)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusNoContent {
logger.LogIf(ctx, fmt.Errorf("Expecting a response, but API returned %s", resp.Status))
return fmt.Errorf("Expecting a response, but API returned %s", resp.Status)
}
return json.NewDecoder(resp.Body).Decode(obj)
err = json.NewDecoder(resp.Body).Decode(obj)
logger.LogIf(ctx, err)
return err
}
// get makes an API call and discards the response. An error is returned if the
// responsee status is not 2xx.
func get(addr, call, apiPassword string) error {
resp, err := apiGet(addr, call, apiPassword)
func get(ctx context.Context, addr, call, apiPassword string) error {
resp, err := apiGet(ctx, addr, call, apiPassword)
if err != nil {
return err
}
@@ -336,7 +344,7 @@ func (s *siaObjects) MakeBucketWithLocation(ctx context.Context, bucket, locatio
sha256sum := sha256.Sum256([]byte(bucket))
var siaObj = path.Join(s.RootDir, bucket, hex.EncodeToString(sha256sum[:]))
return post(s.Address, "/renter/upload/"+siaObj, "source="+srcFile, s.password)
return post(ctx, s.Address, "/renter/upload/"+siaObj, "source="+srcFile, s.password)
}
// GetBucketInfo gets bucket metadata.
@@ -347,7 +355,7 @@ func (s *siaObjects) GetBucketInfo(ctx context.Context, bucket string) (bi minio
dstFile := path.Join(s.TempDir, minio.MustGetUUID())
defer os.Remove(dstFile)
if err := get(s.Address, "/renter/download/"+siaObj+"?destination="+url.QueryEscape(dstFile), s.password); err != nil {
if err := get(ctx, s.Address, "/renter/download/"+siaObj+"?destination="+url.QueryEscape(dstFile), s.password); err != nil {
return bi, err
}
return minio.BucketInfo{Name: bucket}, nil
@@ -355,7 +363,7 @@ func (s *siaObjects) GetBucketInfo(ctx context.Context, bucket string) (bi minio
// ListBuckets will detect and return existing buckets on Sia.
func (s *siaObjects) ListBuckets(ctx context.Context) (buckets []minio.BucketInfo, err error) {
sObjs, serr := s.listRenterFiles("")
sObjs, serr := s.listRenterFiles(ctx, "")
if serr != nil {
return buckets, serr
}
@@ -388,11 +396,11 @@ func (s *siaObjects) DeleteBucket(ctx context.Context, bucket string) error {
sha256sum := sha256.Sum256([]byte(bucket))
var siaObj = path.Join(s.RootDir, bucket, hex.EncodeToString(sha256sum[:]))
return post(s.Address, "/renter/delete/"+siaObj, "", s.password)
return post(ctx, s.Address, "/renter/delete/"+siaObj, "", s.password)
}
func (s *siaObjects) ListObjects(ctx context.Context, bucket string, prefix string, marker string, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, err error) {
siaObjs, siaErr := s.listRenterFiles(bucket)
siaObjs, siaErr := s.listRenterFiles(ctx, bucket)
if siaErr != nil {
return loi, siaErr
}
@@ -429,7 +437,7 @@ func (s *siaObjects) GetObject(ctx context.Context, bucket string, object string
defer os.Remove(dstFile)
var siaObj = path.Join(s.RootDir, bucket, object)
if err := get(s.Address, "/renter/download/"+siaObj+"?destination="+url.QueryEscape(dstFile), s.password); err != nil {
if err := get(ctx, s.Address, "/renter/download/"+siaObj+"?destination="+url.QueryEscape(dstFile), s.password); err != nil {
return err
}
@@ -459,11 +467,16 @@ func (s *siaObjects) GetObject(ctx context.Context, bucket string, object string
// Reply back invalid range if the input offset and length fall out of range.
if startOffset > size || startOffset+length > size {
return errors.Trace(minio.InvalidRange{
logger.LogIf(ctx, minio.InvalidRange{
OffsetBegin: startOffset,
OffsetEnd: length,
ResourceSize: size,
})
return minio.InvalidRange{
OffsetBegin: startOffset,
OffsetEnd: length,
ResourceSize: size,
}
}
// Allocate a staging buffer.
@@ -476,10 +489,10 @@ func (s *siaObjects) GetObject(ctx context.Context, bucket string, object string
// findSiaObject retrieves the siaObjectInfo for the Sia object with the given
// Sia path name.
func (s *siaObjects) findSiaObject(bucket, object string) (siaObjectInfo, error) {
func (s *siaObjects) findSiaObject(ctx context.Context, bucket, object string) (siaObjectInfo, error) {
siaPath := path.Join(s.RootDir, bucket, object)
sObjs, err := s.listRenterFiles("")
sObjs, err := s.listRenterFiles(ctx, "")
if err != nil {
return siaObjectInfo{}, err
}
@@ -489,16 +502,19 @@ func (s *siaObjects) findSiaObject(bucket, object string) (siaObjectInfo, error)
return sObj, nil
}
}
return siaObjectInfo{}, errors.Trace(minio.ObjectNotFound{
logger.LogIf(ctx, minio.ObjectNotFound{
Bucket: bucket,
Object: object,
})
return siaObjectInfo{}, minio.ObjectNotFound{
Bucket: bucket,
Object: object,
}
}
// GetObjectInfo reads object info and replies back ObjectInfo
func (s *siaObjects) GetObjectInfo(ctx context.Context, bucket string, object string) (minio.ObjectInfo, error) {
so, err := s.findSiaObject(bucket, object)
so, err := s.findSiaObject(ctx, bucket, object)
if err != nil {
return minio.ObjectInfo{}, err
}
@@ -527,11 +543,11 @@ func (s *siaObjects) PutObject(ctx context.Context, bucket string, object string
return objInfo, err
}
if err = post(s.Address, "/renter/upload/"+path.Join(s.RootDir, bucket, object), "source="+srcFile, s.password); err != nil {
if err = post(ctx, s.Address, "/renter/upload/"+path.Join(s.RootDir, bucket, object), "source="+srcFile, s.password); err != nil {
os.Remove(srcFile)
return objInfo, err
}
defer s.deleteTempFileWhenUploadCompletes(srcFile, bucket, object)
defer s.deleteTempFileWhenUploadCompletes(ctx, srcFile, bucket, object)
return minio.ObjectInfo{
Name: object,
@@ -546,7 +562,7 @@ func (s *siaObjects) PutObject(ctx context.Context, bucket string, object string
func (s *siaObjects) DeleteObject(ctx context.Context, bucket string, object string) error {
// Tell Sia daemon to delete the object
var siaObj = path.Join(s.RootDir, bucket, object)
return post(s.Address, "/renter/delete/"+siaObj, "", s.password)
return post(ctx, s.Address, "/renter/delete/"+siaObj, "", s.password)
}
// siaObjectInfo represents object info stored on Sia
@@ -565,10 +581,10 @@ type renterFiles struct {
}
// listRenterFiles will return a list of existing objects in the bucket provided
func (s *siaObjects) listRenterFiles(bucket string) (siaObjs []siaObjectInfo, err error) {
func (s *siaObjects) listRenterFiles(ctx context.Context, bucket string) (siaObjs []siaObjectInfo, err error) {
// Get list of all renter files
var rf renterFiles
if err = list(s.Address, s.password, &rf); err != nil {
if err = list(ctx, s.Address, s.password, &rf); err != nil {
return siaObjs, err
}
@@ -592,16 +608,15 @@ func (s *siaObjects) listRenterFiles(bucket string) (siaObjs []siaObjectInfo, er
// deleteTempFileWhenUploadCompletes checks the status of a Sia file upload
// until it reaches 100% upload progress, then deletes the local temp copy from
// the filesystem.
func (s *siaObjects) deleteTempFileWhenUploadCompletes(tempFile string, bucket, object string) {
func (s *siaObjects) deleteTempFileWhenUploadCompletes(ctx context.Context, tempFile string, bucket, object string) {
var soi siaObjectInfo
// Wait until 100% upload instead of 1x redundancy because if we delete
// after 1x redundancy, the user has to pay the cost of other hosts
// redistributing the file.
for soi.UploadProgress < 100.0 {
var err error
soi, err = s.findSiaObject(bucket, object)
soi, err = s.findSiaObject(ctx, bucket, object)
if err != nil {
minio.ErrorIf(err, "Unable to find file uploaded to Sia path %s/%s", bucket, object)
break
}
+39 -4
View File
@@ -205,6 +205,17 @@ func guessIsHealthCheckReq(req *http.Request) bool {
req.URL.Path == healthCheckPathPrefix+healthCheckReadinessPath)
}
// guessIsMetricsReq - returns true if incoming request looks
// like metrics request
func guessIsMetricsReq(req *http.Request) bool {
if req == nil {
return false
}
aType := getRequestAuthType(req)
return aType == authTypeAnonymous &&
req.URL.Path == minioReservedBucketPath+prometheusMetricsPath
}
// guessIsRPCReq - returns true if the request is for an RPC endpoint.
func guessIsRPCReq(req *http.Request) bool {
if req == nil {
@@ -275,7 +286,7 @@ func setReservedBucketHandler(h http.Handler) http.Handler {
func (h minioReservedBucketHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch {
case guessIsRPCReq(r), guessIsBrowserReq(r), guessIsHealthCheckReq(r), isAdminReq(r):
case guessIsRPCReq(r), guessIsBrowserReq(r), guessIsHealthCheckReq(r), guessIsMetricsReq(r), isAdminReq(r):
// Allow access to reserved buckets
default:
// For all other requests reject access to reserved
@@ -375,9 +386,18 @@ var defaultAllowableHTTPMethods = []string{
// setCorsHandler handler for CORS (Cross Origin Resource Sharing)
func setCorsHandler(h http.Handler) http.Handler {
commonS3Headers := []string{"Content-Length", "Content-Type", "Connection",
"Date", "ETag", "Server", "x-amz-delete-marker", "x-amz-id-2",
"x-amz-request-id", "x-amz-version-id"}
commonS3Headers := []string{
"Date",
"ETag",
"Server",
"Connection",
"Accept-Ranges",
"Content-Range",
"Content-Encoding",
"Content-Length",
"Content-Type",
"x-amz-request-id",
}
c := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowedMethods: defaultAllowableHTTPMethods,
@@ -616,3 +636,18 @@ func (l rateLimit) ServeHTTP(w http.ResponseWriter, r *http.Request) {
l.handler.ServeHTTP(w, r)
}
type securityHeaderHandler struct {
handler http.Handler
}
func addSecurityHeaders(h http.Handler) http.Handler {
return securityHeaderHandler{handler: h}
}
func (s securityHeaderHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
header := w.Header()
header.Set("X-XSS-Protection", "\"1; mode=block\"") // Prevents against XSS attacks
header.Set("Content-Security-Policy", "block-all-mixed-content") // prevent mixed (HTTP / HTTPS content)
s.handler.ServeHTTP(w, r)
}
+1 -2
View File
@@ -25,8 +25,8 @@ import (
humanize "github.com/dustin/go-humanize"
"github.com/fatih/color"
miniohttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/pkg/auth"
miniohttp "github.com/minio/minio/pkg/http"
)
// minio configuration related constants.
@@ -191,7 +191,6 @@ var (
colorBold = color.New(color.Bold).SprintFunc()
colorBlue = color.New(color.FgBlue).SprintfFunc()
colorYellow = color.New(color.FgYellow).SprintfFunc()
colorRed = color.New(color.FgRed).SprintfFunc()
)
// Returns minio global information, as a key value map.
+25 -14
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"io"
"mime/multipart"
"net"
@@ -24,7 +25,7 @@ import (
"net/url"
"strings"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
httptracer "github.com/minio/minio/pkg/handlers"
)
@@ -36,7 +37,7 @@ func parseLocationConstraint(r *http.Request) (location string, s3Error APIError
locationConstraint := createBucketLocationConfiguration{}
err := xmlDecoder(r.Body, &locationConstraint, r.ContentLength)
if err != nil && err != io.EOF {
errorIf(err, "Unable to xml decode location constraint")
logger.LogIf(context.Background(), err)
// Treat all other failures as XML parsing errors.
return "", ErrMalformedXML
} // else for both err as nil or io.EOF
@@ -113,9 +114,10 @@ var userMetadataKeyPrefixes = []string{
}
// extractMetadataFromHeader extracts metadata from HTTP header.
func extractMetadataFromHeader(header http.Header) (map[string]string, error) {
func extractMetadataFromHeader(ctx context.Context, header http.Header) (map[string]string, error) {
if header == nil {
return nil, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return nil, errInvalidArgument
}
metadata := make(map[string]string)
@@ -134,7 +136,8 @@ func extractMetadataFromHeader(header http.Header) (map[string]string, error) {
// Go through all other headers for any additional headers that needs to be saved.
for key := range header {
if key != http.CanonicalHeaderKey(key) {
return nil, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return nil, errInvalidArgument
}
for _, prefix := range userMetadataKeyPrefixes {
if strings.HasPrefix(key, prefix) {
@@ -187,12 +190,13 @@ func trimAwsChunkedContentEncoding(contentEnc string) (trimmedContentEnc string)
}
// Validate form field size for s3 specification requirement.
func validateFormFieldSize(formValues http.Header) error {
func validateFormFieldSize(ctx context.Context, formValues http.Header) error {
// Iterate over form values
for k := range formValues {
// Check if value's field exceeds S3 limit
if int64(len(formValues.Get(k))) > maxFormFieldSize {
return errors.Trace(errSizeUnexpected)
logger.LogIf(ctx, errSizeUnexpected)
return errSizeUnexpected
}
}
@@ -201,7 +205,7 @@ func validateFormFieldSize(formValues http.Header) error {
}
// Extract form fields and file data from a HTTP POST Policy
func extractPostPolicyFormValues(form *multipart.Form) (filePart io.ReadCloser, fileName string, fileSize int64, formValues http.Header, err error) {
func extractPostPolicyFormValues(ctx context.Context, form *multipart.Form) (filePart io.ReadCloser, fileName string, fileSize int64, formValues http.Header, err error) {
/// HTML Form values
fileName = ""
@@ -212,7 +216,7 @@ func extractPostPolicyFormValues(form *multipart.Form) (filePart io.ReadCloser,
}
// Validate form values.
if err = validateFormFieldSize(formValues); err != nil {
if err = validateFormFieldSize(ctx, formValues); err != nil {
return nil, "", 0, nil, err
}
@@ -221,7 +225,8 @@ func extractPostPolicyFormValues(form *multipart.Form) (filePart io.ReadCloser,
canonicalFormName := http.CanonicalHeaderKey(k)
if canonicalFormName == "File" {
if len(v) == 0 {
return nil, "", 0, nil, errors.Trace(errInvalidArgument)
logger.LogIf(ctx, errInvalidArgument)
return nil, "", 0, nil, errInvalidArgument
}
// Fetch fileHeader which has the uploaded file information
fileHeader := v[0]
@@ -230,17 +235,20 @@ func extractPostPolicyFormValues(form *multipart.Form) (filePart io.ReadCloser,
// Open the uploaded part
filePart, err = fileHeader.Open()
if err != nil {
return nil, "", 0, nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, "", 0, nil, err
}
// Compute file size
fileSize, err = filePart.(io.Seeker).Seek(0, 2)
if err != nil {
return nil, "", 0, nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, "", 0, nil, err
}
// Reset Seek to the beginning
_, err = filePart.(io.Seeker).Seek(0, 0)
if err != nil {
return nil, "", 0, nil, errors.Trace(err)
logger.LogIf(ctx, err)
return nil, "", 0, nil, err
}
// File found and ready for reading
break
@@ -276,7 +284,10 @@ func getResource(path string, host string, domain string) (string, error) {
// In bucket.mydomain.com:9000, strip out :9000
var err error
if host, _, err = net.SplitHostPort(host); err != nil {
errorIf(err, "Unable to split %s", host)
reqInfo := (&logger.ReqInfo{}).AppendTags("host", host)
reqInfo.AppendTags("path", path)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return "", err
}
}
+4 -5
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"context"
"encoding/xml"
"io/ioutil"
"net/http"
@@ -25,8 +26,6 @@ import (
"reflect"
"strings"
"testing"
"github.com/minio/minio/pkg/errors"
)
// Tests validate bucket LocationConstraint.
@@ -114,9 +113,9 @@ func TestValidateFormFieldSize(t *testing.T) {
// Run validate form field size check under all test cases.
for i, testCase := range testCases {
err := validateFormFieldSize(testCase.header)
err := validateFormFieldSize(context.Background(), testCase.header)
if err != nil {
if errors.Cause(err).Error() != testCase.err.Error() {
if err.Error() != testCase.err.Error() {
t.Errorf("Test %d: Expected error %s, got %s", i+1, testCase.err, err)
}
}
@@ -180,7 +179,7 @@ func TestExtractMetadataHeaders(t *testing.T) {
// Validate if the extracting headers.
for i, testCase := range testCases {
metadata, err := extractMetadataFromHeader(testCase.header)
metadata, err := extractMetadataFromHeader(context.Background(), testCase.header)
if err != nil && !testCase.shouldFail {
t.Fatalf("Test %d failed to extract metadata: %v", i+1, err)
}
-6
View File
@@ -18,7 +18,6 @@ package cmd
import (
"crypto/md5"
"encoding/base64"
"encoding/hex"
"github.com/minio/sha256-simd"
@@ -47,8 +46,3 @@ func getMD5Sum(data []byte) []byte {
func getMD5Hash(data []byte) string {
return hex.EncodeToString(getMD5Sum(data))
}
// getMD5HashBase64 returns MD5 hash in base64 encoding of given data.
func getMD5HashBase64(data []byte) string {
return base64.StdEncoding.EncodeToString(getMD5Sum(data))
}
+6
View File
@@ -21,6 +21,8 @@ import (
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus"
"go.uber.org/atomic"
)
@@ -188,6 +190,10 @@ func (st *HTTPStats) updateStats(r *http.Request, w *httpResponseRecorder, durat
st.successDELETEs.Duration.Add(durationSecs)
}
}
// Increment the prometheus http request count with appropriate label
httpRequests.With(prometheus.Labels{"request_type": r.Method}).Inc()
// Increment the prometheus http request response histogram with appropriate label
httpRequestsDuration.With(prometheus.Labels{"request_type": r.Method}).Observe(durationSecs)
}
// Prepare new HTTPStats structure
+31 -35
View File
@@ -17,8 +17,8 @@
package http
import (
"context"
"crypto/tls"
"errors"
"fmt"
"io"
"net"
@@ -28,6 +28,8 @@ import (
"sync"
"syscall"
"time"
"github.com/minio/minio/cmd/logger"
)
var sslRequiredErrMsg = []byte("HTTP/1.0 403 Forbidden\r\n\r\nSSL required")
@@ -85,9 +87,8 @@ type httpListener struct {
tcpKeepAliveTimeout time.Duration
readTimeout time.Duration
writeTimeout time.Duration
updateBytesReadFunc func(int) // function to be called to update bytes read in BufConn.
updateBytesWrittenFunc func(int) // function to be called to update bytes written in BufConn.
errorLogFunc func(error, string, ...interface{}) // function to be called on errors.
updateBytesReadFunc func(int) // function to be called to update bytes read in BufConn.
updateBytesWrittenFunc func(int) // function to be called to update bytes written in BufConn.
}
// isRoutineNetErr returns true if error is due to a network timeout,
@@ -139,17 +140,16 @@ func (listener *httpListener) start() {
// Peek bytes of maximum length of all HTTP methods.
data, err := bufconn.Peek(methodMaxLen)
if err != nil {
if listener.errorLogFunc != nil {
// Peek could fail legitimately when clients abruptly close
// connection. E.g. Chrome browser opens connections speculatively to
// speed up loading of a web page. Peek may also fail due to network
// saturation on a transport with read timeout set. All other kind of
// errors should be logged for further investigation. Thanks @brendanashworth.
if !isRoutineNetErr(err) {
listener.errorLogFunc(err,
"Error in reading from new connection %s at server %s",
bufconn.RemoteAddr(), bufconn.LocalAddr())
}
// Peek could fail legitimately when clients abruptly close
// connection. E.g. Chrome browser opens connections speculatively to
// speed up loading of a web page. Peek may also fail due to network
// saturation on a transport with read timeout set. All other kind of
// errors should be logged for further investigation. Thanks @brendanashworth.
if !isRoutineNetErr(err) {
reqInfo := (&logger.ReqInfo{}).AppendTags("remoteAddr", bufconn.RemoteAddr().String())
reqInfo.AppendTags("localAddr", bufconn.LocalAddr().String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
bufconn.Close()
return
@@ -172,12 +172,11 @@ func (listener *httpListener) start() {
if listener.tlsConfig != nil {
// As the listener is configured with TLS, try to do TLS handshake, drop the connection if it fails.
tlsConn := tls.Server(bufconn, listener.tlsConfig)
if err := tlsConn.Handshake(); err != nil {
if listener.errorLogFunc != nil {
listener.errorLogFunc(err,
"TLS handshake failed with new connection %s at server %s",
bufconn.RemoteAddr(), bufconn.LocalAddr())
}
if err = tlsConn.Handshake(); err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remoteAddr", bufconn.RemoteAddr().String())
reqInfo.AppendTags("localAddr", bufconn.LocalAddr().String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
bufconn.Close()
return
}
@@ -187,12 +186,13 @@ func (listener *httpListener) start() {
listener.updateBytesReadFunc, listener.updateBytesWrittenFunc)
// Peek bytes of maximum length of all HTTP methods.
data, err := bufconn.Peek(methodMaxLen)
data, err = bufconn.Peek(methodMaxLen)
if err != nil {
if !isRoutineNetErr(err) && listener.errorLogFunc != nil {
listener.errorLogFunc(err,
"Error in reading from new TLS connection %s at server %s",
bufconn.RemoteAddr(), bufconn.LocalAddr())
if !isRoutineNetErr(err) {
reqInfo := (&logger.ReqInfo{}).AppendTags("remoteAddr", bufconn.RemoteAddr().String())
reqInfo.AppendTags("localAddr", bufconn.LocalAddr().String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
bufconn.Close()
return
@@ -205,12 +205,10 @@ func (listener *httpListener) start() {
return
}
}
if listener.errorLogFunc != nil {
listener.errorLogFunc(errors.New("junk message"),
"Received non-HTTP message from new connection %s at server %s",
bufconn.RemoteAddr(), bufconn.LocalAddr())
}
reqInfo := (&logger.ReqInfo{}).AppendTags("remoteAddr", bufconn.RemoteAddr().String())
reqInfo.AppendTags("localAddr", bufconn.LocalAddr().String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
bufconn.Close()
return
@@ -299,8 +297,7 @@ func newHTTPListener(serverAddrs []string,
readTimeout time.Duration,
writeTimeout time.Duration,
updateBytesReadFunc func(int),
updateBytesWrittenFunc func(int),
errorLogFunc func(error, string, ...interface{})) (listener *httpListener, err error) {
updateBytesWrittenFunc func(int)) (listener *httpListener, err error) {
var tcpListeners []*net.TCPListener
// Close all opened listeners on error
@@ -337,7 +334,6 @@ func newHTTPListener(serverAddrs []string,
writeTimeout: writeTimeout,
updateBytesReadFunc: updateBytesReadFunc,
updateBytesWrittenFunc: updateBytesWrittenFunc,
errorLogFunc: errorLogFunc,
}
listener.start()
@@ -19,6 +19,7 @@ package http
import (
"bufio"
"bytes"
"context"
"crypto/tls"
"errors"
"fmt"
@@ -205,7 +206,7 @@ func TestNewHTTPListener(t *testing.T) {
writeTimeout time.Duration
updateBytesReadFunc func(int)
updateBytesWrittenFunc func(int)
errorLogFunc func(error, string, ...interface{})
errorLogFunc func(context.Context, error)
expectedErr error
}{
{[]string{"93.184.216.34:65432"}, nil, time.Duration(0), time.Duration(0), time.Duration(0), nil, nil, nil, errors.New(remoteAddrErrMsg)},
@@ -227,7 +228,6 @@ func TestNewHTTPListener(t *testing.T) {
testCase.writeTimeout,
testCase.updateBytesReadFunc,
testCase.updateBytesWrittenFunc,
testCase.errorLogFunc,
)
if testCase.expectedErr == nil {
@@ -279,7 +279,6 @@ func TestHTTPListenerStartClose(t *testing.T) {
time.Duration(0),
nil,
nil,
nil,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -327,7 +326,6 @@ func TestHTTPListenerAddr(t *testing.T) {
time.Duration(0),
nil,
nil,
nil,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -372,7 +370,6 @@ func TestHTTPListenerAddrs(t *testing.T) {
time.Duration(0),
nil,
nil,
nil,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -419,7 +416,6 @@ func TestHTTPListenerAccept(t *testing.T) {
time.Duration(0),
nil,
nil,
nil,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -480,11 +476,6 @@ func TestHTTPListenerAccept(t *testing.T) {
func TestHTTPListenerAcceptPeekError(t *testing.T) {
tlsConfig := getTLSConfig(t)
nonLoopBackIP := getNonLoopBackIP(t)
errorFunc := func(err error, template string, args ...interface{}) {
msg := fmt.Sprintf("error: %v. ", err)
msg += fmt.Sprintf(template, args...)
fmt.Println(msg)
}
testCases := []struct {
serverAddrs []string
@@ -504,7 +495,6 @@ func TestHTTPListenerAcceptPeekError(t *testing.T) {
time.Duration(0),
nil,
nil,
errorFunc,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -540,11 +530,6 @@ func TestHTTPListenerAcceptPeekError(t *testing.T) {
func TestHTTPListenerAcceptTLSError(t *testing.T) {
tlsConfig := getTLSConfig(t)
nonLoopBackIP := getNonLoopBackIP(t)
errorFunc := func(err error, template string, args ...interface{}) {
msg := fmt.Sprintf("error: %v. ", err)
msg += fmt.Sprintf(template, args...)
fmt.Println(msg)
}
testCases := []struct {
serverAddrs []string
@@ -563,7 +548,6 @@ func TestHTTPListenerAcceptTLSError(t *testing.T) {
time.Duration(0),
nil,
nil,
errorFunc,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -609,11 +593,6 @@ func TestHTTPListenerAcceptTLSError(t *testing.T) {
func TestHTTPListenerAcceptError(t *testing.T) {
tlsConfig := getTLSConfig(t)
nonLoopBackIP := getNonLoopBackIP(t)
errorFunc := func(err error, template string, args ...interface{}) {
msg := fmt.Sprintf("error: %v. ", err)
msg += fmt.Sprintf(template, args...)
fmt.Println(msg)
}
testCases := []struct {
serverAddrs []string
@@ -635,7 +614,6 @@ func TestHTTPListenerAcceptError(t *testing.T) {
time.Duration(0),
nil,
nil,
errorFunc,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
@@ -761,7 +739,6 @@ func TestHTTPListenerAcceptParallel(t *testing.T) {
time.Duration(0),
nil,
nil,
nil,
)
if err != nil {
t.Fatalf("Test %d: error: expected = <nil>, got = %v", i+1, err)
+9 -12
View File
@@ -50,16 +50,15 @@ const (
// Server - extended http.Server supports multiple addresses to serve and enhanced connection handling.
type Server struct {
http.Server
Addrs []string // addresses on which the server listens for new connection.
ShutdownTimeout time.Duration // timeout used for graceful server shutdown.
TCPKeepAliveTimeout time.Duration // timeout used for underneath TCP connection.
UpdateBytesReadFunc func(int) // function to be called to update bytes read in bufConn.
UpdateBytesWrittenFunc func(int) // function to be called to update bytes written in bufConn.
ErrorLogFunc func(error, string, ...interface{}) // function to be called on errors.
listenerMutex *sync.Mutex // to guard 'listener' field.
listener *httpListener // HTTP listener for all 'Addrs' field.
inShutdown uint32 // indicates whether the server is in shutdown or not
requestCount int32 // counter holds no. of request in process.
Addrs []string // addresses on which the server listens for new connection.
ShutdownTimeout time.Duration // timeout used for graceful server shutdown.
TCPKeepAliveTimeout time.Duration // timeout used for underneath TCP connection.
UpdateBytesReadFunc func(int) // function to be called to update bytes read in bufConn.
UpdateBytesWrittenFunc func(int) // function to be called to update bytes written in bufConn.
listenerMutex *sync.Mutex // to guard 'listener' field.
listener *httpListener // HTTP listener for all 'Addrs' field.
inShutdown uint32 // indicates whether the server is in shutdown or not
requestCount int32 // counter holds no. of request in process.
}
// Start - start HTTP server
@@ -77,7 +76,6 @@ func (srv *Server) Start() (err error) {
tcpKeepAliveTimeout := srv.TCPKeepAliveTimeout
updateBytesReadFunc := srv.UpdateBytesReadFunc
updateBytesWrittenFunc := srv.UpdateBytesWrittenFunc
errorLogFunc := srv.ErrorLogFunc // if srv.ErrorLogFunc holds non-synced state -> possible data race
// Create new HTTP listener.
var listener *httpListener
@@ -89,7 +87,6 @@ func (srv *Server) Start() (err error) {
writeTimeout,
updateBytesReadFunc,
updateBytesWrittenFunc,
errorLogFunc,
)
if err != nil {
return err
+4 -2
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"errors"
"fmt"
"net/http"
@@ -24,6 +25,7 @@ import (
jwtgo "github.com/dgrijalva/jwt-go"
jwtreq "github.com/dgrijalva/jwt-go/request"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
)
@@ -97,11 +99,11 @@ func isAuthTokenValid(tokenString string) bool {
var claims jwtgo.StandardClaims
jwtToken, err := jwtgo.ParseWithClaims(tokenString, &claims, keyFuncCallback)
if err != nil {
errorIf(err, "Unable to parse JWT token string")
logger.LogIf(context.Background(), err)
return false
}
if err = claims.Valid(); err != nil {
errorIf(err, "Invalid claims in JWT token string")
logger.LogIf(context.Background(), err)
return false
}
return jwtToken.Valid && claims.Subject == globalServerConfig.GetCredential().AccessKey
+27 -12
View File
@@ -17,10 +17,11 @@
package cmd
import (
"context"
"fmt"
"time"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
type statusType string
@@ -112,29 +113,34 @@ func (n *nsLockMap) initLockInfoForVolumePath(param nsParam) {
// Change the state of the lock from Blocked to Running.
func (n *nsLockMap) statusBlockedToRunning(param nsParam, lockSource, opsID string, readLock bool) error {
// This function is called outside nsLockMap.mutex.Lock(), so must be held explicitly.
ctx := context.Background()
n.lockMapMutex.Lock()
defer n.lockMapMutex.Unlock()
// Check whether the lock info entry for <volume, path> pair already exists.
_, ok := n.debugLockMap[param]
if !ok {
return errors.Trace(LockInfoVolPathMissing{param.volume, param.path})
logger.LogIf(ctx, LockInfoVolPathMissing{param.volume, param.path})
return LockInfoVolPathMissing{param.volume, param.path}
}
// Check whether lock info entry for the given `opsID` exists.
lockInfo, ok := n.debugLockMap[param].lockInfo[opsID]
if !ok {
return errors.Trace(LockInfoOpsIDNotFound{param.volume, param.path, opsID})
logger.LogIf(ctx, LockInfoOpsIDNotFound{param.volume, param.path, opsID})
return LockInfoOpsIDNotFound{param.volume, param.path, opsID}
}
// Check whether lockSource is same.
if lockInfo.lockSource != lockSource {
return errors.Trace(LockInfoOriginMismatch{param.volume, param.path, opsID, lockSource})
logger.LogIf(ctx, LockInfoOriginMismatch{param.volume, param.path, opsID, lockSource})
return LockInfoOriginMismatch{param.volume, param.path, opsID, lockSource}
}
// Status of the lock should be set to "Blocked".
if lockInfo.status != blockedStatus {
return errors.Trace(LockInfoStateNotBlocked{param.volume, param.path, opsID})
logger.LogIf(ctx, LockInfoStateNotBlocked{param.volume, param.path, opsID})
return LockInfoStateNotBlocked{param.volume, param.path, opsID}
}
// Change lock status to running and update the time.
n.debugLockMap[param].lockInfo[opsID] = newDebugLockInfo(lockSource, runningStatus, readLock)
@@ -182,24 +188,29 @@ func (n *nsLockMap) statusNoneToBlocked(param nsParam, lockSource, opsID string,
// Change the state of the lock from Blocked to none.
func (n *nsLockMap) statusBlockedToNone(param nsParam, lockSource, opsID string, readLock bool) error {
_, ok := n.debugLockMap[param]
ctx := context.Background()
if !ok {
return errors.Trace(LockInfoVolPathMissing{param.volume, param.path})
logger.LogIf(ctx, LockInfoVolPathMissing{param.volume, param.path})
return LockInfoVolPathMissing{param.volume, param.path}
}
// Check whether lock info entry for the given `opsID` exists.
lockInfo, ok := n.debugLockMap[param].lockInfo[opsID]
if !ok {
return errors.Trace(LockInfoOpsIDNotFound{param.volume, param.path, opsID})
logger.LogIf(ctx, LockInfoOpsIDNotFound{param.volume, param.path, opsID})
return LockInfoOpsIDNotFound{param.volume, param.path, opsID}
}
// Check whether lockSource is same.
if lockInfo.lockSource != lockSource {
return errors.Trace(LockInfoOriginMismatch{param.volume, param.path, opsID, lockSource})
logger.LogIf(ctx, LockInfoOriginMismatch{param.volume, param.path, opsID, lockSource})
return LockInfoOriginMismatch{param.volume, param.path, opsID, lockSource}
}
// Status of the lock should be set to "Blocked".
if lockInfo.status != blockedStatus {
return errors.Trace(LockInfoStateNotBlocked{param.volume, param.path, opsID})
logger.LogIf(ctx, LockInfoStateNotBlocked{param.volume, param.path, opsID})
return LockInfoStateNotBlocked{param.volume, param.path, opsID}
}
// Update global lock stats.
@@ -214,7 +225,8 @@ func (n *nsLockMap) statusBlockedToNone(param nsParam, lockSource, opsID string,
func (n *nsLockMap) deleteLockInfoEntryForVolumePath(param nsParam) error {
// delete the lock info for the given operation.
if _, found := n.debugLockMap[param]; !found {
return errors.Trace(LockInfoVolPathMissing{param.volume, param.path})
logger.LogIf(context.Background(), LockInfoVolPathMissing{param.volume, param.path})
return LockInfoVolPathMissing{param.volume, param.path}
}
// The following stats update is relevant only in case of a
@@ -235,17 +247,20 @@ func (n *nsLockMap) deleteLockInfoEntryForVolumePath(param nsParam) error {
// Called when the nsLk ref count for the given (volume, path) is
// not 0.
func (n *nsLockMap) deleteLockInfoEntryForOps(param nsParam, opsID string) error {
ctx := context.Background()
// delete the lock info for the given operation.
infoMap, found := n.debugLockMap[param]
if !found {
return errors.Trace(LockInfoVolPathMissing{param.volume, param.path})
logger.LogIf(ctx, LockInfoVolPathMissing{param.volume, param.path})
return LockInfoVolPathMissing{param.volume, param.path}
}
// The operation finished holding the lock on the resource, remove
// the entry for the given operation with the operation ID.
opsIDLock, foundInfo := infoMap.lockInfo[opsID]
if !foundInfo {
// Unlock request with invalid operation ID not accepted.
return errors.Trace(LockInfoOpsIDNotFound{param.volume, param.path, opsID})
logger.LogIf(ctx, LockInfoOpsIDNotFound{param.volume, param.path, opsID})
return LockInfoOpsIDNotFound{param.volume, param.path, opsID}
}
// Update global and (volume, path) lock status.
granted := opsIDLock.status == runningStatus
+8 -10
View File
@@ -18,8 +18,6 @@ package cmd
import (
"testing"
"github.com/minio/minio/pkg/errors"
)
type lockStateCase struct {
@@ -282,7 +280,7 @@ func TestNsLockMapStatusBlockedToRunning(t *testing.T) {
testCases[0].opsID, testCases[0].readLock)
expectedErr := LockInfoVolPathMissing{testCases[0].volume, testCases[0].path}
if errors.Cause(actualErr) != expectedErr {
if actualErr != expectedErr {
t.Fatalf("Errors mismatch: Expected \"%s\", got \"%s\"", expectedErr, actualErr)
}
@@ -302,7 +300,7 @@ func TestNsLockMapStatusBlockedToRunning(t *testing.T) {
testCases[0].opsID, testCases[0].readLock)
expectedOpsErr := LockInfoOpsIDNotFound{testCases[0].volume, testCases[0].path, testCases[0].opsID}
if errors.Cause(actualErr) != expectedOpsErr {
if actualErr != expectedOpsErr {
t.Fatalf("Errors mismatch: Expected \"%s\", got \"%s\"", expectedOpsErr, actualErr)
}
@@ -325,7 +323,7 @@ func TestNsLockMapStatusBlockedToRunning(t *testing.T) {
testCases[0].opsID, testCases[0].readLock)
expectedBlockErr := LockInfoStateNotBlocked{testCases[0].volume, testCases[0].path, testCases[0].opsID}
if errors.Cause(actualErr) != expectedBlockErr {
if actualErr != expectedBlockErr {
t.Fatalf("Errors mismatch: Expected: \"%s\", got: \"%s\"", expectedBlockErr, actualErr)
}
@@ -346,7 +344,7 @@ func TestNsLockMapStatusBlockedToRunning(t *testing.T) {
}
// invoking the method under test.
actualErr = globalNSMutex.statusBlockedToRunning(param, testCase.lockSource, testCase.opsID, testCase.readLock)
if errors.Cause(actualErr) != testCase.expectedErr {
if actualErr != testCase.expectedErr {
t.Fatalf("Test %d: Errors mismatch: Expected: \"%s\", got: \"%s\"", i+1, testCase.expectedErr, actualErr)
}
// In case of no error proceed with validating the lock state information.
@@ -465,7 +463,7 @@ func TestNsLockMapStatusNoneToBlocked(t *testing.T) {
testCases[0].opsID, testCases[0].readLock)
expectedErr := LockInfoVolPathMissing{testCases[0].volume, testCases[0].path}
if errors.Cause(actualErr) != expectedErr {
if actualErr != expectedErr {
t.Fatalf("Errors mismatch: Expected \"%s\", got \"%s\"", expectedErr, actualErr)
}
@@ -509,7 +507,7 @@ func TestNsLockMapDeleteLockInfoEntryForOps(t *testing.T) {
actualErr := globalNSMutex.deleteLockInfoEntryForOps(param, testCases[0].opsID)
expectedErr := LockInfoVolPathMissing{testCases[0].volume, testCases[0].path}
if errors.Cause(actualErr) != expectedErr {
if actualErr != expectedErr {
t.Fatalf("Errors mismatch: Expected \"%s\", got \"%s\"", expectedErr, actualErr)
}
@@ -528,7 +526,7 @@ func TestNsLockMapDeleteLockInfoEntryForOps(t *testing.T) {
actualErr = globalNSMutex.deleteLockInfoEntryForOps(param, "non-existent-OpsID")
expectedOpsIDErr := LockInfoOpsIDNotFound{param.volume, param.path, "non-existent-OpsID"}
if errors.Cause(actualErr) != expectedOpsIDErr {
if actualErr != expectedOpsIDErr {
t.Fatalf("Errors mismatch: Expected \"%s\", got \"%s\"", expectedOpsIDErr, actualErr)
}
// case - 4.
@@ -592,7 +590,7 @@ func TestNsLockMapDeleteLockInfoEntryForVolumePath(t *testing.T) {
param := nsParam{testCases[0].volume, testCases[0].path}
actualErr := globalNSMutex.deleteLockInfoEntryForVolumePath(param)
expectedNilErr := LockInfoVolPathMissing{param.volume, param.path}
if errors.Cause(actualErr) != expectedNilErr {
if actualErr != expectedNilErr {
t.Fatalf("Errors mismatch: Expected \"%s\", got \"%s\"", expectedNilErr, actualErr)
}
+7 -1
View File
@@ -17,8 +17,11 @@
package cmd
import (
"context"
"errors"
"time"
"github.com/minio/minio/cmd/logger"
)
// Similar to removeEntry but only removes an entry only if the lock entry exists in map.
@@ -29,7 +32,10 @@ func (l *localLocker) removeEntryIfExists(nlrip nameLockRequesterInfoPair) {
// Remove failed, in case it is a:
if nlrip.lri.writer {
// Writer: this should never happen as the whole (mapped) entry should have been deleted
errorIf(errors.New(""), "Lock maintenance failed to remove entry for write lock (should never happen) %#v %#v %#v", nlrip.name, nlrip.lri.uid, lri)
reqInfo := (&logger.ReqInfo{}).AppendTags("name", nlrip.name)
reqInfo.AppendTags("uid", nlrip.lri.uid)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errors.New("Lock maintenance failed to remove entry for write lock (should never happen)"))
} // Reader: this can happen if multiple read locks were active and
// the one we are looking for has been released concurrently (so it is fine).
} // Removal went okay, all is fine.
+4 -2
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"fmt"
"math/rand"
"sync"
@@ -24,7 +25,7 @@ import (
router "github.com/gorilla/mux"
"github.com/minio/dsync"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/cmd/logger"
)
const (
@@ -98,7 +99,8 @@ func registerDistNSLockRouter(mux *router.Router, endpoints EndpointList) error
func registerStorageLockers(mux *router.Router, lkSrv *lockServer) error {
lockRPCServer := newRPCServer()
if err := lockRPCServer.RegisterName(lockServiceName, lkSrv); err != nil {
return errors.Trace(err)
logger.LogIf(context.Background(), err)
return err
}
lockRouter := mux.PathPrefix(minioReservedBucketPath).Subrouter()
lockRouter.Path(lockServicePath).Handler(lockRPCServer)

Some files were not shown because too many files have changed in this diff Show More