Compare commits

..

72 Commits

Author SHA1 Message Date
Harshavardhana 6df05e489d Set Read/Write timeouts only for net.Conn not http.Server (#7431)
Fixes #7425
2019-03-27 22:10:06 +05:30
Harshavardhana 4a698c731b HealObjects should remove objects without quorum (#7407)
This PR adds a way to list objects without quorum
such that they can purged by `mc admin heal --remove`
2019-03-26 14:57:44 -07:00
Harshavardhana 9629de8230 Add proper context based logging when bitrot stream calls fail (#7415) 2019-03-26 13:59:33 -07:00
Harshavardhana 0250f7de67 Cleanup stale multipart uploads older than 3 days (#7424)
Fixes #6627
2019-03-25 13:41:05 -07:00
kannappanr 7154b8a568 Error log: Correct error type in anonymous mode (#7414)
Currently message is set to error type value.
Message field is not used in error logs. it is used only in the case of info logs.

This PR sets error message field to store error type correctly.
2019-03-25 13:40:08 -07:00
Anis Elleuch 8689ec258b Don't decrypt ETag in validation when source is SSEC multipart (#7423)
Copying an encrypted SSEC object when this latter is uploaded using
multipart mechanism was failing because ETag in case of encrypted
multipart upload is not encrypted.

This PR fixes the behavior.
2019-03-25 12:17:31 -07:00
Krishnan Parthasarathi aac9e2a7dd Return deploymentID in ServerInfo REST call (#7422)
Makes deploymentID information uniform in distributed setup
2019-03-25 11:55:28 -07:00
Harshavardhana 719d21efd8 Generate coverage across all sub-dirs (#7416) 2019-03-25 11:54:14 -07:00
Harshavardhana e0a87e96de Populate host value from GetSourceIP directly (#7417) 2019-03-25 11:45:42 -07:00
Harshavardhana 0a44e70177 Change findEndpoint to return errors for Scanlines (#7390) 2019-03-25 13:31:02 +05:30
Rushan 22b4fe0a51 Update browser UI with new logo and colors (#7408) 2019-03-23 09:57:09 -07:00
Praveen raj Mani 89e45d0695 Restart process should use the current process' pid (#7373)
This fixes varying pids for server-respawns. And avoids duplicate process
creating multiple pids when the server restart signal is triggered with
service restart enabled.

Fixes #7350
2019-03-20 22:20:30 -07:00
poornas 8e1e701d35 Azure:ETag returned by ListObjects to be consistent with GetObjectInfo (#7301) 2019-03-20 18:11:46 -07:00
kannappanr 87cf51d5ab unused code: Remove LoadCredentials function (#7369)
It is required to set the environment variable in the case of distributed
minio. LoadCredentials is used to notify peers of the change and will not work if
environment variable is set. so, this function will never be called.
2019-03-20 18:09:57 -07:00
Minio Trusted 8af45f0135 Update yaml files to latest version RELEASE.2019-03-20T22-38-47Z 2019-03-20 22:49:06 +00:00
Harshavardhana a9032b52b8 Change storageRESTTimeout to 1minute (#7398) 2019-03-20 13:20:09 -07:00
Harshavardhana c184038b6a Add proper custom errors object creations (#7387)
In scenario 1

```
- bucket/object-prefix
- bucket/object-prefix/object
```

Server responds with `XMinioParentIsObject`

In scenario 2

```
- bucket/object-prefix/object
- bucket/object-prefix
```

Server responds with `XMinioObjectExistsAsDirectory`

Fixes #6566
2019-03-20 13:06:53 -07:00
poornas 12b79d9f3b Remove duplicate error in switch case. (#7381)
Fixes: #7380

crypto.ErrInvalidCustomerKey was being handled twice in toAPIErrorCode()
2019-03-19 17:21:05 -07:00
Sidhartha Mani 6bc0de2a75 add go modules file and start running go 1.11 style builds (#7354) 2019-03-19 13:50:58 -07:00
Krishnan Parthasarathi 8a77a298f2 Add deploymentID to ServerInfo (#7372) 2019-03-19 16:12:24 +05:30
Harshavardhana 328eb74cbb Fix regression in peer clients in TLS setups (#7391)
Regression was introduced in eb69c4f946
2019-03-19 09:44:49 +05:30
zy 73be3ed0ca format import style (#7383) 2019-03-18 13:07:58 -07:00
Kirill Motkov 3d29ab4059 Rewrite if-else chains to switch statements (#7382) 2019-03-18 07:46:20 -07:00
Harshavardhana 6702d23d52 Simplify ReadFileStream closer, make sure to flush all HTTP responses (#7374) 2019-03-18 10:50:26 +05:30
poornas 1011d21416 Fix credential parsing in signature v4 (#7377)
Fixes #7376
2019-03-16 22:45:42 -07:00
Kirill Motkov 85c5acc088 fix staticcheck warning (#7378) 2019-03-16 22:44:43 -07:00
kannappanr eb69c4f946 Use REST api for inter node communication (#7205) 2019-03-14 16:27:31 -07:00
Anis Elleuch facbd653ba Add normal/deep type of heal scanning (#7251)
Healing scan used to read all objects parts to check for bitrot
checksum. This commit will add a quicker way of healing scan
by only checking if parts are actually present in disks or not.
2019-03-14 13:08:51 -07:00
Harshavardhana 233824bf92 Configure http2 with higher maxconcurrent streams (#7363)
This value is needed for Minio's internode communication,
read the meaning of this value as per the HTTP 2.0 spec

https://http2.github.io/http2-spec/#rfc.section.5.1.2
2019-03-14 11:57:35 -07:00
poornas a0ee7be050 update link on README (#7361) 2019-03-14 15:18:12 +05:30
Harshavardhana 91d85a0d53 Fix stale locks held by SelectParquet API (#7364)
Vendorize upstream parquet-go to fix this issue.
2019-03-13 20:33:18 -07:00
Harshavardhana 7079abc931 Implement HealObjects API to simplify healing (#7351) 2019-03-13 17:35:09 -07:00
Minio Trusted 8377d00574 Update yaml files to latest version RELEASE.2019-03-13T21-59-47Z 2019-03-13 22:06:26 +00:00
Harshavardhana 285c09fe6b Support buckets with '.' with etcd+coreDNS (#7353)
Fixes #7340
2019-03-12 17:57:08 -07:00
kannappanr ce4563370c Distributed: Allow healing if all disks are on root partitions (#7358)
If all the disks are on root partitions in distributed mode, consider it
to be a test setup and allow healing to proceed.

Fixes #7346
2019-03-12 16:47:06 -07:00
Aditya Manthramurthy e463386921 Add JSON Path expression evaluation support (#7315)
- Includes support for FROM clause JSON path
2019-03-09 08:13:37 -08:00
Aditya Manthramurthy b296b3cf8b Skip non-bucket dns entry in federated bucket list (#7339) 2019-03-08 14:22:58 -08:00
Harshavardhana 0b96ad4fdc http2 throws custom error Content-Length shorter handle it (#7334)
We should internally handle when http2 input stream has smaller
content than its content-length header

Upstream issue reported https://github.com/golang/go/issues/30648

This a change which we need to handle internally until Go fixes it
correctly, till now our code doesn't expect a custom error to be returned.
2019-03-07 16:11:28 -08:00
Aditya Manthramurthy f4879ed96d Use jstream to serialize records to JSON format in S3Select (#7318)
- Also, switch to jstream to generate internal record representation
  from CSV/JSON readers

- This fixes a bug in which JSON output objects have their keys
  reversed from the order they are specified in the Select columns.

- Also includes a fix for tests.
2019-03-07 00:20:10 -08:00
Aditya Manthramurthy f97a33a63f Downgrade node temporarily (#7333)
To fix jest related build failure as recommended at https://github.com/facebook/jest/issues/8069#issuecomment-470307590
2019-03-06 16:05:49 -08:00
Minio Trusted 6387407e79 Update yaml files to latest version RELEASE.2019-03-06T22-47-10Z 2019-03-06 22:56:46 +00:00
Anis Elleuch b05825ffe8 s3: Fix precondition failed in CopyObjectPart when src is encrypted (#7276)
CopyObject precondition checks into GetObjectReader
in order to perform SSE-C pre-condition checks using the
last 32 bytes of encrypted ETag rather than the decrypted
ETag

This also necessitates moving precondition checks for
gateways to gateway layer rather than object handler check
2019-03-06 12:38:41 -08:00
kannappanr 39ddb78c75 CORS: Expose all headers on response (#7331)
Fixes #7289
2019-03-06 11:58:53 -08:00
Harshavardhana 12eb71828b Fix posix tests for SimpleCI (#7328) 2019-03-05 19:53:01 -08:00
Praveen raj Mani c0a1369b73 Construct dynamic XML error responses for postpolicyform validation (#7321)
Fixes #7314
2019-03-05 12:10:47 -08:00
kannappanr c57159a0fe fs mode: List already existing buckets with capital letters (#7244)
if a bucket with `Captialized letters` is created, `InvalidBucketName` error
will be returned. 
In the case of pre-existing buckets, it will be listed.

Fixes #6938
2019-03-05 10:42:32 -08:00
Kale Blankenship ef132c5714 Replace snappy.Writer/io.Pipe with snappyCompressReader. (#7316)
Prevents deferred close functions from being called while still
  attempting to copy reader to snappyWriter.
 Reduces code duplication when compressing objects.
2019-03-05 08:35:37 -08:00
Aditya Manthramurthy c54b0c0ca1 Fix a race in tests (#7326) 2019-03-05 21:34:17 +05:30
Aditya Manthramurthy e8e9cd3e74 Close GlobalServiceDoneCh when quitting (#7322)
This change allows indefinitely running go-routines to cleanup
gracefully.

This channel is now closed at the beginning of each test so that
long-running go-routines quit and a new one is assigned.
2019-03-04 14:33:14 -08:00
Sidhartha Mani f7611bcd48 update simple-ci scripts (#7309) 2019-03-03 21:59:14 +05:30
Aditya Manthramurthy 19c10cb4d0 Update vendored sjson (includes a bug fix) (#7317) 2019-03-01 15:04:38 -08:00
poornas 6b4c6f69af Remove minio-go validator from vendor dir (#7312) 2019-02-28 12:16:26 -08:00
poornas 2564147ab4 Filter Expires header from user metadata (#7269)
Instead save it as a struct field in ObjectInfo as it is
a standard HTTP header - Fixes minio/mc#2690
2019-02-28 11:01:25 -08:00
Harshavardhana c3ca954684 Implement AssumeRole API for Minio users (#7267)
For actual API reference read here

https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html

Documentation is added and updated as well at docs/sts/assume-role.md

Fixes #6381
2019-02-27 17:46:55 -08:00
Harshavardhana ce588d1489 Improve ListObjects performance by listing in parallel (#7270)
The side affect of this change memory
increase, but this is a trade-off between
performance and actual memory usage.

For all practical scenarios this should be
an adequate change.
2019-02-27 14:39:22 -08:00
Sidhartha Mani b983da957d run gateway mint test in full mode (#7296) 2019-02-27 10:03:23 -08:00
Harshavardhana 5e69a107d8 Support detecting mountpoints correctly (#7288)
Currently windows support was relying on Symlink as
a way to detect a drive, this doesn't work in latest
Windows 2016, fix this to use a proper mechanism by
using win32 APIs.

Additionally also add support for detecting bind mounts
on Linux.
2019-02-26 18:04:53 -08:00
Dee Koder 5085bef397 Share button icon is changed (#7300) 2019-02-26 16:34:29 -08:00
poornas 8022a6efd9 Return ETag for 0-byte object prefixes (#7291)
Fixes: #7290
2019-02-26 15:09:14 -08:00
Minio Trusted d3125857a8 Update yaml files to latest version RELEASE.2019-02-26T19-51-46Z 2019-02-26 19:58:09 +00:00
Praveen raj Mani 78d116c487 Event persistence for MQTT (#7268)
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.

The events are replayed back when the mqtt broker is back online.
2019-02-25 18:01:13 -08:00
dcharbonnier 2fc341394d fix json syntax (#7285) 2019-02-25 13:02:59 +05:30
Anis Elleuch 6584c7ea2b s3: Encode StartAfter when encoding type is passed (#7281)
In ObjectList V2, StartAfter needs to be encoded when encoding-type
is specified.
2019-02-24 18:50:28 -08:00
Harshavardhana 2520e535a0 Allow lazyQuotes for certain types of CSV (#7278)
Set lazyQuotes to true, to allow a quote to appear
in an unquote field and a non-doubled quote may
appear in a quoted field.
2019-02-24 06:51:02 -08:00
Anis Elleuch 5efbe8a1b3 s3: Add support of encodingType parameter (#7265)
This commit honors encoding-type parameter in object listing,
parts listing and multipart uploads listing.
2019-02-24 11:44:24 +05:30
Nitish Tiwari dab314900d Fix healthcheck script to wait for netstat command output (#7275)
Fixes #7272
2019-02-23 09:32:53 -08:00
Harshavardhana 7923b83953 Support multiple-domains in MINIO_DOMAIN (#7274)
Fixes #7173
2019-02-23 08:48:01 +05:30
Aditya Manthramurthy 80a351633f Update vendorized bcicen/jstream (#7257)
- Includes an error handling fix that is waiting to be merged upstream
- Uses order-preserving (un)marshalling for JSON objects.
2019-02-20 23:59:23 -08:00
Harshavardhana bedcb7442a Write xml.Header first instead of spaces to handle XML parsers (#7253)
Clients like AWS SDK Java and AWS cli XML parsers are
unable to handle on `\r\n` characters to avoid these
errors send XML header first and write white space characters
instead.

Also handle cases to avoid double WriteHeader calls
2019-02-21 11:50:15 +05:30
Harshavardhana 2232b0b55f Fix mac build failure for healthcheck binary (#7263) 2019-02-21 08:11:16 +05:30
Harshavardhana 91576d416d Fix GetLocalPeer usage in perf handlers (#7249)
GetLocalPeer usage should be fixed and used only
once per call for not all local endpoints.
2019-02-20 16:04:55 -08:00
Minio Trusted 3aabe45fd9 Update yaml files to latest version RELEASE.2019-02-20T22-44-29Z 2019-02-20 22:49:40 +00:00
255 changed files with 16649 additions and 4044 deletions
+2 -1
View File
@@ -17,6 +17,7 @@ matrix:
env:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
go: 1.11.4
script:
- make
@@ -44,7 +45,7 @@ before_script:
- if [[ "${TRAVIS_OS_NAME}" == "linux" ]]; then sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'; fi
before_install:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then nvm install stable ; fi
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then nvm install 11.10.1 ; fi
after_success:
- bash <(curl -s https://codecov.io/bash)
+28 -4
View File
@@ -7,12 +7,39 @@ COPY . /go/src/github.com/minio/minio
WORKDIR /go/src/github.com/minio/minio
RUN apt-get update && apt-get install -y jq
ENV GO111MODULE=on
RUN git config --global http.cookiefile /gitcookie/.gitcookie
RUN make
RUN bash -c 'diff -au <(gofmt -s -d cmd) <(printf "")'
RUN bash -c 'diff -au <(gofmt -s -d pkg) <(printf "")'
RUN apt-get update && \
apt-get -y install sudo
RUN touch /etc/sudoers
RUN echo "root ALL=(ALL) ALL" >> /etc/sudoers
RUN echo "ci ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN echo "Defaults env_reset" >> /etc/sudoers
RUN echo 'Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go:/usr/local/go/bin"' >> /etc/sudoers
RUN mkdir -p /home/ci/.cache
RUN groupadd -g 999 ci && \
useradd -r -u 999 -g ci ci
RUN chown -R ci:ci /go
RUN chown -R ci:ci /home/ci
RUN chmod -R a+rw /go
USER ci
RUN for d in $(go list ./... | grep -v browser); do go test -v -race --timeout 15m "$d"; done
RUN make verifiers
RUN make crosscompile
RUN make coverage
RUN make verify
RUN make coverage || true
#-------------------------------------------------------------
# Stage 2: Test Frontend
@@ -35,9 +62,6 @@ COPY --from=0 /go/src/github.com/minio/minio/minio ./minio
COPY buildscripts/gateway-tests.sh ./gateway-tests.sh
RUN apt-get update && apt-get install -y git wget jq curl dnsmasq
RUN wget https://dl.google.com/go/go1.11.4.linux-amd64.tar.gz && \
tar -C /usr/local -xzf go1.11.4.linux-amd64.tar.gz
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
ENV GOROOT /usr/local/go
+5 -4
View File
@@ -13,9 +13,10 @@ checks:
@(env bash $(PWD)/buildscripts/checkgopath.sh)
getdeps:
@echo "Installing golint" && go get -u golang.org/x/lint/golint
@echo "Installing staticcheck" && go get -u honnef.co/go/tools/...
@echo "Installing misspell" && go get -u github.com/client9/misspell/cmd/misspell
@mkdir -p $(GOPATH)/bin
@echo "Installing golint" && which golint || go get -u golang.org/x/lint/golint
@echo "Installing staticcheck" && which staticcheck || wget --quiet -O $(GOPATH)/bin/staticcheck https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_linux_amd64 && chmod +x $(GOPATH)/bin/staticcheck
@echo "Installing misspell" && which misspell || wget --quiet https://github.com/client9/misspell/releases/download/v0.3.4/misspell_0.3.4_linux_64bit.tar.gz && tar xvf misspell_0.3.4_linux_64bit.tar.gz && mv misspell $(GOPATH)/bin/misspell && chmod +x $(GOPATH)/bin/misspell && rm -r misspell_0.3.4_linux_64bit.tar.gz
crosscompile:
@(env bash $(PWD)/buildscripts/cross-compile.sh)
@@ -66,7 +67,7 @@ coverage: build
build: checks
@echo "Building minio binary to './minio'"
@GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio
@GOFLAGS="" CGO_ENABLED=0 go build --ldflags="-s -w" -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go
@GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags="-s -w" -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go
docker: build
@docker build -t $(TAG) . -f Dockerfile.dev
+1 -1
View File
@@ -4,7 +4,7 @@
height: 100%;
top: 0;
left: 0;
background: #32393F;
background: #002a37;
z-index: 100;
transition: opacity 200ms;
-webkit-transition: opacity 200ms;
Binary file not shown.

Before

Width:  |  Height:  |  Size: 501 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

+50
View File
@@ -0,0 +1,50 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="469px" height="60px" viewBox="0 0 469 60" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 48.2 (47327) - http://www.bohemiancoding.com/sketch -->
<title>Untitled</title>
<desc>Created with Sketch.</desc>
<defs></defs>
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<path d="M27.9171869,0.94921875 L5.01068171,0.94921875 C2.24925796,0.94921875 0.0106817104,3.187795 0.0106817104,5.94921875 L0.0106817104,54.9836742 C0.0106817104,57.7450979 2.24925796,59.9836742 5.01068171,59.9836742 L43.0384854,59.9836742 C45.7999092,59.9836742 48.0384854,57.7450979 48.0384854,54.9836742 L48.0384854,21.4784647 C48.0384854,20.1754261 47.5298023,18.9238644 46.6207587,17.9902963 L31.4994602,2.46105043 C30.558243,1.49444073 29.2663438,0.94921875 27.9171869,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M87.9171869,0.94921875 L65.0106817,0.94921875 C62.249258,0.94921875 60.0106817,3.187795 60.0106817,5.94921875 L60.0106817,54.9836742 C60.0106817,57.7450979 62.249258,59.9836742 65.0106817,59.9836742 L103.038485,59.9836742 C105.799909,59.9836742 108.038485,57.7450979 108.038485,54.9836742 L108.038485,21.4784647 C108.038485,20.1754261 107.529802,18.9238644 106.620759,17.9902963 L91.4994602,2.46105043 C90.558243,1.49444073 89.2663438,0.94921875 87.9171869,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M147.917187,0.94921875 L125.010682,0.94921875 C122.249258,0.94921875 120.010682,3.187795 120.010682,5.94921875 L120.010682,54.9836742 C120.010682,57.7450979 122.249258,59.9836742 125.010682,59.9836742 L163.038485,59.9836742 C165.799909,59.9836742 168.038485,57.7450979 168.038485,54.9836742 L168.038485,21.4784647 C168.038485,20.1754261 167.529802,18.9238644 166.620759,17.9902963 L151.49946,2.46105043 C150.558243,1.49444073 149.266344,0.94921875 147.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M207.917187,0.94921875 L185.010682,0.94921875 C182.249258,0.94921875 180.010682,3.187795 180.010682,5.94921875 L180.010682,54.9836742 C180.010682,57.7450979 182.249258,59.9836742 185.010682,59.9836742 L223.038485,59.9836742 C225.799909,59.9836742 228.038485,57.7450979 228.038485,54.9836742 L228.038485,21.4784647 C228.038485,20.1754261 227.529802,18.9238644 226.620759,17.9902963 L211.49946,2.46105043 C210.558243,1.49444073 209.266344,0.94921875 207.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M267.917187,0.94921875 L245.010682,0.94921875 C242.249258,0.94921875 240.010682,3.187795 240.010682,5.94921875 L240.010682,54.9836742 C240.010682,57.7450979 242.249258,59.9836742 245.010682,59.9836742 L283.038485,59.9836742 C285.799909,59.9836742 288.038485,57.7450979 288.038485,54.9836742 L288.038485,21.4784647 C288.038485,20.1754261 287.529802,18.9238644 286.620759,17.9902963 L271.49946,2.46105043 C270.558243,1.49444073 269.266344,0.94921875 267.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M327.917187,0.94921875 L305.010682,0.94921875 C302.249258,0.94921875 300.010682,3.187795 300.010682,5.94921875 L300.010682,54.9836742 C300.010682,57.7450979 302.249258,59.9836742 305.010682,59.9836742 L343.038485,59.9836742 C345.799909,59.9836742 348.038485,57.7450979 348.038485,54.9836742 L348.038485,21.4784647 C348.038485,20.1754261 347.529802,18.9238644 346.620759,17.9902963 L331.49946,2.46105043 C330.558243,1.49444073 329.266344,0.94921875 327.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M387.917187,0.94921875 L365.010682,0.94921875 C362.249258,0.94921875 360.010682,3.187795 360.010682,5.94921875 L360.010682,54.9836742 C360.010682,57.7450979 362.249258,59.9836742 365.010682,59.9836742 L403.038485,59.9836742 C405.799909,59.9836742 408.038485,57.7450979 408.038485,54.9836742 L408.038485,21.4784647 C408.038485,20.1754261 407.529802,18.9238644 406.620759,17.9902963 L391.49946,2.46105043 C390.558243,1.49444073 389.266344,0.94921875 387.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M447.917187,0.94921875 L425.010682,0.94921875 C422.249258,0.94921875 420.010682,3.187795 420.010682,5.94921875 L420.010682,54.9836742 C420.010682,57.7450979 422.249258,59.9836742 425.010682,59.9836742 L463.038485,59.9836742 C465.799909,59.9836742 468.038485,57.7450979 468.038485,54.9836742 L468.038485,21.4784647 C468.038485,20.1754261 467.529802,18.9238644 466.620759,17.9902963 L451.49946,2.46105043 C450.558243,1.49444073 449.266344,0.94921875 447.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<g id="excel" transform="translate(434.000000, 28.000000)" fill="#617A8B" fill-rule="nonzero">
<rect id="Rectangle-2" x="0" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="0" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="0" y="10.3086611" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="10.3086611" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="10.3086611" width="5" height="3.09259835"></rect>
</g>
<g id="folder" transform="translate(14.000000, 25.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M7.99999996,0 L2,0 C0.900000087,0 0.00999998996,0.900000093 0.00999998996,2.00000001 L0,14 C0,15.1000003 0.900000087,16 2,16 L18,16 C19.1000002,16 20,15.1000003 20,14 L20,4.00000001 C20,2.90000009 19.1000002,2.00000001 18,2.00000001 L9.99999996,2.00000001 L7.99999996,0 Z" id="Shape"></path>
</g>
<g id="image" transform="translate(72.000000, 24.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M13.590909,-3.76363638e-07 L9.6704545,5.22727238 L12.6500002,9.19999976 L10.9772726,10.4545451 C9.21045494,8.10227239 6.27272722,4.18181782 6.27272722,4.18181782 L-6.27273216e-08,12.5454541 L22.9999999,12.5454541 L13.590909,-3.76363638e-07 Z" id="Shape"></path>
</g>
<g id="pdf" transform="translate(135.000000, 22.500000)" fill="#617A8B" fill-rule="nonzero">
<path d="M12.9434813,11.7078193 C11.5243518,10.2886898 9.92783089,8.15999533 8.50870126,6.2086921 C8.86348367,4.61217127 9.04087488,3.37043286 9.04087488,2.66086804 C9.04087488,-0.886956014 4.78348601,-0.886956014 4.78348601,2.66086804 C4.78348601,3.37043286 5.49305082,4.61217127 6.73478924,6.5634745 C6.20261563,9.04695135 5.49305082,12.0626018 4.6060948,14.5460787 C2.83218278,15.2556434 1.59044436,16.1425995 0.880879547,16.6747731 C-0.00607646688,17.5617291 -0.183467669,18.6260763 0.171314736,19.5130323 C0.526097141,20.3999883 1.41305316,20.932162 2.30000917,20.932162 C2.83218278,20.932162 3.36435639,20.7547707 3.71913879,20.3999883 C3.89652999,20.2225971 4.6060948,19.5130323 5.84783322,15.6104258 C8.15391886,14.7234698 10.6373957,13.8365138 12.4113077,13.3043402 C14.0078286,14.7234698 15.249567,15.4330347 16.4913054,15.4330347 C17.7330438,15.4330347 18.6199998,14.5460787 18.6199998,13.4817314 C18.6199998,12.7721666 18.2652175,12.239993 17.5556526,11.7078193 C16.8460878,11.353037 15.9591318,11.1756458 14.7173934,11.1756458 C14.362611,11.353037 13.8304373,11.353037 12.9434813,11.7078193 Z M2.47740038,19.1582499 C2.30000917,19.3356411 2.30000917,19.3356411 2.30000917,19.3356411 C2.12261797,19.3356411 1.94522676,19.1582499 1.76783556,18.8034675 C1.59044436,18.6260763 1.59044436,18.0939027 2.12261797,17.7391203 C2.30000917,17.5617291 2.83218278,17.2069466 3.89652999,16.6747731 C3.18696518,18.2712939 2.65479157,18.9808587 2.47740038,19.1582499 Z M6.38000684,2.48347683 C6.38000684,1.77391203 6.73478924,1.41912963 6.73478924,1.41912963 C6.73478924,1.41912963 7.08957165,1.77391203 7.08957165,2.48347683 C7.08957165,2.83825924 7.08957165,3.37043286 6.91218044,4.07999767 C6.55739803,3.19304165 6.38000684,2.66086804 6.38000684,2.48347683 Z M6.55739803,13.6591226 C7.08957165,11.8852106 7.62174525,10.1112986 7.97652766,8.33738654 C9.04087488,9.75651615 10.1052221,10.9982546 11.1695693,12.0626018 C9.75043969,12.5947754 8.15391886,12.9495579 6.55739803,13.6591226 Z M17.023479,13.6591226 C17.023479,13.8365138 16.8460878,14.013905 16.4913054,14.013905 C16.3139142,14.013905 15.6043494,13.8365138 14.5400021,12.9495579 C14.7173934,12.9495579 14.7173934,12.9495579 14.8947846,12.9495579 C15.9591318,12.9495579 16.6686966,13.126949 16.8460878,13.3043402 C16.8460878,13.4817314 17.023479,13.6591226 17.023479,13.6591226 Z" id="Shape"></path>
</g>
<g id="video" transform="translate(196.000000, 24.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M13.1249998,-1.68750002e-07 L13.1249998,1.87499983 L11.2499998,1.87499983 L11.2499998,-1.68750002e-07 L3.74999978,-1.68750002e-07 L3.74999978,1.87499983 L1.87499978,1.87499983 L1.87499978,-1.68750002e-07 L-2.25000019e-07,-1.68750002e-07 L-2.25000019e-07,16.8749998 L1.87499978,16.8749998 L1.87499978,14.9999998 L3.74999978,14.9999998 L3.74999978,16.8749998 L11.2499998,16.8749998 L11.2499998,14.9999998 L13.1249998,14.9999998 L13.1249998,16.8749998 L14.9999998,16.8749998 L14.9999998,-1.68750002e-07 L13.1249998,-1.68750002e-07 Z M3.74999955,13.1249991 L1.87499967,13.1249991 L1.87499967,11.2499992 L3.74999955,11.2499992 L3.74999955,13.1249991 Z M3.74999955,9.37499927 L1.87499967,9.37499927 L1.87499967,7.49999938 L3.74999955,7.49999938 L3.74999955,9.37499927 Z M3.74999955,5.62499949 L1.87499967,5.62499949 L1.87499967,3.74999961 L3.74999955,3.74999961 L3.74999955,5.62499949 Z M13.124999,13.1249991 L11.2499991,13.1249991 L11.2499991,11.2499992 L13.124999,11.2499992 L13.124999,13.1249991 Z M13.124999,9.37499927 L11.2499991,9.37499927 L11.2499991,7.49999938 L13.124999,7.49999938 L13.124999,9.37499927 Z M13.124999,5.62499949 L11.2499991,5.62499949 L11.2499991,3.74999961 L13.124999,3.74999961 L13.124999,5.62499949 Z" id="Shape"></path>
</g>
<g id="audio" transform="translate(253.000000, 25.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M16.6899956,0.302885892 C16.4835716,0.100936704 16.2326065,0 15.9375267,0 C15.8194481,0 15.7159842,0.0145115457 15.6276386,0.0433073022 L6.41930364,2.81258763 C6.19803256,2.88469069 6.01719345,3.01271813 5.87698015,3.1965942 C5.73684439,3.38058392 5.66671835,3.58427601 5.66671835,3.80789779 L5.66671835,14.2686004 C5.02469108,13.9872357 4.31641429,13.8465532 3.54169413,13.8465532 C3.20970997,13.8465532 2.8536527,13.8844045 2.47371613,13.9599933 C2.09358575,14.0355821 1.71171093,14.1509926 1.32813045,14.3060728 C0.944356146,14.4611532 0.627141486,14.6793946 0.376215114,14.9604564 C0.125366273,15.2417075 0,15.5628561 0,15.9233335 C0,16.2840761 0.125366273,16.6048457 0.376215114,16.8862105 C0.627141486,17.1673102 0.944317382,17.3855517 1.32813045,17.540594 C1.71186599,17.6956744 2.09374081,17.810971 2.47371613,17.8866736 C2.8536527,17.9622625 3.20970997,18 3.54169413,18 C3.87367828,18 4.22961926,17.9622625 4.60967212,17.8866736 C4.9898025,17.810971 5.37152225,17.6956744 5.7552578,17.540594 C6.13887704,17.3855137 6.4560917,17.1672722 6.70694055,16.8862105 C6.95802198,16.6048457 7.08334949,16.2840381 7.08334949,15.9233335 L7.08334949,8.25381104 L15.5834076,5.69004149 L15.5834076,11.4990929 C14.9412253,11.2178795 14.2330648,11.0770455 13.4583446,11.0770455 C13.1263217,11.0770455 12.7702257,11.1147831 12.3902891,11.1904856 C12.0101975,11.2660745 11.6285553,11.3814848 11.2447422,11.5365651 C10.861123,11.6916454 10.543792,11.909887 10.2927493,12.1912139 C10.0420168,12.4722 9.91626286,12.7932726 9.91626286,13.1538259 C9.91626286,13.5145685 10.0420168,13.8353381 10.2927493,14.1167028 C10.543792,14.3979541 10.8610842,14.6161956 11.2447422,14.7710864 C11.6284002,14.9261668 12.0101975,15.041577 12.3902891,15.1173175 C12.7702257,15.1930201 13.126283,15.230606 13.4583446,15.230606 C13.7902513,15.230606 14.1463085,15.1930201 14.5262451,15.1173175 C14.9062204,15.041577 15.2879788,14.9261668 15.6719083,14.7710864 C16.0555662,14.6161576 16.3726259,14.3979161 16.6236297,14.1167028 C16.8747888,13.8353381 17,13.5145305 17,13.1538259 L17,1.03820069 C16.9998837,0.749788454 16.8969236,0.504721411 16.6899956,0.302885892 Z" id="Shape"></path>
</g>
<g id="code" transform="translate(313.000000, 27.000000)" fill="#617A8B" fill-rule="nonzero">
<polygon id="Shape" transform="translate(17.091428, 6.857142) scale(-1, 1) translate(-17.091428, -6.857142) " points="21.3257133 1.61142788 19.7142851 -4.11428573e-07 12.8571425 6.85714204 19.7142851 13.7142845 21.3257133 12.1028561 16.091428 6.85714204"></polygon>
<polygon id="Shape" points="8.46857015 1.61142788 6.8571419 -4.11428573e-07 -5.48571475e-07 6.85714204 6.8571419 13.7142845 8.46857015 12.1028561 3.23428485 6.85714204"></polygon>
</g>
<g id="presentation" transform="translate(374.464844, 23.500000)" fill="#617A8B" fill-rule="nonzero">
<path d="M8.16304337,0.383534587 L8.16304337,17.4406771 C3.83905758,17.0142486 0.48732923,13.3554907 0.48732923,8.91210585 C0.48732923,4.468721 3.83905799,0.809963151 8.16304337,0.383534587 Z M9.89434308,0.383534587 L9.89434308,8.05071991 L17.5444718,8.05071991 C17.1436294,4.00817738 13.9283578,0.784376849 9.89434308,0.383534587 Z M9.89434308,9.77349173 L9.89434308,17.4406771 C13.9368865,17.0398348 17.1436278,13.8160343 17.5444718,9.77349173 L9.89434308,9.77349173 Z" id="Shape"></path>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 14 KiB

+4 -8
View File
@@ -1,16 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="302px" height="579px" viewBox="0 0 302 579" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<svg width="93px" height="187px" viewBox="0 0 93 187" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 48.2 (47327) - http://www.bohemiancoding.com/sketch -->
<title>Untitled 2</title>
<title>logo</title>
<desc>Created with Sketch.</desc>
<defs></defs>
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g id="Minio_Logo_White" fill-rule="nonzero">
<g id="Group">
<path d="M225.835,28.882 C225.835,28.882 282.569,120.259 301.531,152.735 C301.837,153.264 301.707,153.963 301.251,154.419 C300.676,154.994 299.81,155.056 299.24,154.481 L201.484,52.243 L225.835,28.882 Z" id="Shape" fill="#F15A29"></path>
<path d="M58.706,361.854 C75.299,326.75 97.681,294.989 125.38,267.29 C135.915,256.76 147.036,247.002 158.701,238.037 L158.701,310.385 L58.706,361.854 Z M0.971,428.672 L158.7,348.375 L158.7,531.932 L194.244,578.623 L194.244,330.092 L215.833,318.987 C225.778,313.991 235.105,307.343 243.402,299.041 C284.517,257.931 285.17,191.574 245.226,149.749 L163.742,64.855 C160.493,61.383 158.788,56.838 158.938,52.06 C159.088,47.251 161.099,42.784 164.597,39.488 C171.811,32.704 183.305,33.161 190.088,40.369 L201.484,52.247 L225.835,28.881 C197.064,-8.431 161.861,-3.746 141.47,14.92 C131.417,24.383 125.644,37.198 125.208,51.003 C124.773,64.829 129.753,77.981 139.221,88.04 L139.33,88.159 L221.239,173.483 C248.176,202.151 247.637,247.235 219.617,275.255 C215.248,279.623 210.46,283.324 205.387,286.36 L194.245,292.091 L194.245,176.126 C160.167,193.688 129.09,216.251 101.692,243.655 C67.568,277.774 40.776,317.516 22.058,361.772 C12.875,383.479 5.837,405.829 0.971,428.672 Z" id="Shape" fill="#FFFFFF"></path>
<polygon id="Shape" fill="#BCBEC0" points="194.246 330.09 194.246 367.609 158.701 385.648 158.701 348.373"></polygon>
</g>
<g id="logo" transform="translate(0.187500, -0.683594)" fill="#FFFFFF" fill-rule="nonzero">
<path d="M91.49,46.551 C86.7827023,38.7699609 82.062696,30.9966172 77.33,23.231 C74.87,19.231 72.33,15.231 69.88,11.231 C69.57,10.731 69.18,10.291 68.88,9.831 C64.35,2.931 55.44,-1.679 46.73,2.701 C42.9729806,4.51194908 40.0995718,7.75449451 38.7536428,11.7020516 C37.4077139,15.6496086 37.701799,19.9721186 39.57,23.701 C41.08,26.641 43.57,29.121 45.91,31.581 C53.03,39.141 60.38,46.491 67.45,54.111 C72.4175495,59.4492221 74.4526451,66.8835066 72.8965704,74.0075359 C71.3404956,81.1315653 66.390952,87.0402215 59.65,89.821 C59.4938176,89.83842 59.3361824,89.83842 59.18,89.821 L59.18,54.591 C46.6388051,61.0478363 35.3944735,69.759905 26.01,80.291 C11.32,96.671 2.64,117.141 0.01,132.071 L23.96,119.821 C31.96,115.771 39.86,111.821 48.14,107.581 L48.14,175.921 L59.14,187.131 L59.14,101.831 C59.14,101.831 59.39,101.711 60.22,101.261 C63.5480598,99.6738911 66.7772674,97.8873078 69.89,95.911 C77.7130888,90.4306687 82.7479457,81.8029342 83.6709542,72.295947 C84.5939627,62.7889599 81.3127806,53.3538429 74.69,46.471 C66.49,37.891 58.24,29.351 50.05,20.761 C47.67,18.261 47.72,15.101 50.05,12.881 C52.38,10.661 55.56,10.881 57.96,13.331 L61.38,16.781 C64.1,19.681 66.79,22.611 69.53,25.481 C76.4547149,32.7389629 83.3947303,39.9823123 90.35,47.211 C90.7,47.571 91.12,47.871 91.5,48.211 L91.93,47.951 C91.8351945,47.4695902 91.6876376,47.0000911 91.49,46.551 Z M48.11,94.931 C47.9883217,95.5022568 47.6230065,95.9917791 47.11,96.271 C42.72,98.601 38.29,100.871 33.87,103.141 L17.76,111.401 C24.771203,96.7435071 35.1132853,83.9289138 47.96,73.981 C48.08,74.221 48.16,74.301 48.16,74.381 C48.15,81.231 48.17,88.081 48.11,94.931 Z" id="Shape"></path>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 1.9 KiB

After

Width:  |  Height:  |  Size: 2.2 KiB

+6 -3
View File
@@ -1,10 +1,14 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Minio Browser</title>
<title>MinIO Browser</title>
<link rel="icon" type="image/png" sizes="32x32" href="/minio/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="/minio/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="/minio/favicon-16x16.png">
<link rel="stylesheet" href="/minio/loader.css" type="text/css">
</head>
@@ -23,7 +27,6 @@
You are using Internet Explorer version 12.0 or lower. Due to security issues and lack of support for Web Standards it is highly recommended that you upgrade to a modern browser
<ul>
<li>
<a href="http://www.google.com/chrome/">
+1 -1
View File
@@ -38,7 +38,7 @@ export const SideBar = ({ sidebarOpen, clickOutside }) => {
>
<div className="fes-header clearfix hidden-sm hidden-xs">
<img src={logo} alt="" />
<h2>Minio Browser</h2>
<h2>MinIO Browser</h2>
</div>
<div className="fes-list">
{web.LoggedIn() && <BucketSearch />}
+1 -1
View File
@@ -67,7 +67,7 @@ export class ObjectActions extends React.Component {
className="fiad-action"
onClick={this.shareObject.bind(this)}
>
<i className="fa fa-copy" />
<i className="fa fa-share-alt" />
</a>
<a
href=""
+2
View File
@@ -170,10 +170,12 @@ select.form-control {
position: absolute;
left: 2px;
top: 8px;
color: rgba(255, 255, 255, 0.5);
}
.ig-text {
padding-left: 25px;
.placeholder(rgba(255, 255, 255, 0.5))
}
}
+3 -2
View File
@@ -1,7 +1,7 @@
.login {
height: 100vh;
min-height: 500px;
background: @dark-gray;
background: #002a37;
text-align: center;
&:before {
@@ -99,6 +99,7 @@
Chrome autofill fix
-------------------------------*/
input:-webkit-autofill {
-webkit-box-shadow:0 0 0 50px @dark-gray inset !important;
-webkit-box-shadow:0 0 0 50px #002a37 inset !important;
-webkit-text-fill-color: @white !important;
caret-color: white;
}
+9
View File
@@ -50,3 +50,12 @@
-webkit-animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
}
/*----------------------------------------
Placeholder
-----------------------------------------*/
.placeholder(@color) {
::-webkit-input-placeholder { color: @color }
::-moz-placeholder { color: @color }
:-ms-input-placeholder { color: @color }
}
+3 -3
View File
@@ -32,7 +32,7 @@
}
.modal-content {
background-color: @dark-gray;
background-color: #00303f;
}
}
@@ -139,8 +139,8 @@
top: 0;
left: 0;
position: absolute;
border-radius: 3px 0 0px 3px;
background-color: #23282C;
border-radius: 3px 0 0 3px;
background-color: #022631;
}
}
}
+9 -8
View File
@@ -3,7 +3,7 @@
----------------------------*/
.fe-sidebar {
width: @fe-sidebar-width;
background-color: @dark-gray;
background-color: @fe-sidebar-bg;
position: fixed;
height: 100%;
overflow: hidden;
@@ -27,7 +27,7 @@
}
a {
color: rgba(255, 255, 255, 0.58);
color: rgba(255, 255, 255, 0.75);
&:hover {
color: @white;
@@ -47,12 +47,14 @@
}
h2 {
margin: 13px 0 0 10px;
margin: 6px 0 0 13px;
font-weight: normal;
color: @white;
font-size: 20px;
}
img {
width: 32px;
width: 20px;
}
}
@@ -95,7 +97,7 @@
}
&.active {
background-color: #282e32;
background-color: rgba(0,0,0,0.1);
& > a {
color: @white;
@@ -103,7 +105,7 @@
}
&:not(.active):hover {
background-color: rgba(0, 0, 0, 0.1);
background-color: rgba(0, 0, 0, 0.05);
& > a {
color: @white;
@@ -192,8 +194,7 @@
left: 0;
bottom: 0;
z-index: 1;
background: @dark-gray;
color: rgba(255, 255, 255, 0.4);
background-color: rgba(0,0,0,0.1);
font-size: 15px;
font-weight: 400;
width: @fe-sidebar-width;
+2
View File
@@ -14,9 +14,11 @@
File Explorer
----------------------------*/
@fe-sidebar-width : 320px;
@fe-sidebar-bg : #00303f;
@text-muted-color : #BDBDBD;
@text-strong-color : #333;
/*--------------------------
Colors
----------------------------*/
+84 -56
View File
File diff suppressed because one or more lines are too long
+3 -1
View File
@@ -93,11 +93,13 @@ var exports = {
plugins: [
new CopyWebpackPlugin([
{from: 'app/css/loader.css'},
{from: 'app/img/favicon.ico'},
{from: 'app/img/browsers/chrome.png'},
{from: 'app/img/browsers/firefox.png'},
{from: 'app/img/browsers/safari.png'},
{from: 'app/img/logo.svg'},
{from: 'app/img/favicon/favicon-16x16.png'},
{from: 'app/img/favicon/favicon-32x32.png'},
{from: 'app/img/favicon/favicon-96x96.png'},
{from: 'app/index.html'}
]),
new webpack.ContextReplacementPlugin(/moment[\\\/]locale$/, /^\.\/(en)$/),
+3 -1
View File
@@ -69,11 +69,13 @@ var exports = {
plugins: [
new CopyWebpackPlugin([
{from: 'app/css/loader.css'},
{from: 'app/img/favicon.ico'},
{from: 'app/img/browsers/chrome.png'},
{from: 'app/img/browsers/firefox.png'},
{from: 'app/img/browsers/safari.png'},
{from: 'app/img/logo.svg'},
{from: 'app/img/favicon/favicon-16x16.png'},
{from: 'app/img/favicon/favicon-32x32.png'},
{from: 'app/img/favicon/favicon-96x96.png'},
{from: 'app/index.html'}
]),
new webpack.ContextReplacementPlugin(/moment[\\\/]locale$/, /^\.\/(en)$/),
+1 -2
View File
@@ -42,10 +42,9 @@ cd /mint
export MINT_ROOT_DIR=${MINT_ROOT_DIR:-/mint}
export MINT_RUN_CORE_DIR="$MINT_ROOT_DIR/run/core"
export MINT_RUN_SECURITY_DIR="$MINT_ROOT_DIR/run/security"
export MINT_MODE="full"
export WGET="wget --quiet --no-check-certificate"
go get github.com/go-ini/ini
./create-data-files.sh
./preinstall.sh
+2 -8
View File
@@ -1,12 +1,6 @@
#!/usr/bin/env bash
set -e
echo "" > coverage.txt
for d in $(go list ./... | grep -v browser); do
CGO_ENABLED=0 go test -v -coverprofile=profile.out -covermode=atomic "$d"
if [ -f profile.out ]; then
cat profile.out >> coverage.txt
rm profile.out
fi
done
CGO_ENABLED=0 go test -v -coverprofile=coverage.txt -covermode=atomic ./...
+4 -5
View File
@@ -319,11 +319,10 @@ function __init__()
mkdir -p "$MINIO_CONFIG_DIR"
mkdir -p "$MINT_DATA_DIR"
if ! go get -u github.com/minio/mc; then
echo "failed to download https://github.com/minio/mc"
exit 1
fi
/bin/cp -a "$(go env GOPATH)"/bin/mc "$WORK_DIR/mc"
git clone https://github.com/minio/mc
cd mc
go build
/bin/cp -a ./mc "$WORK_DIR/mc"
chmod a+x "$WORK_DIR/mc"
+17 -24
View File
@@ -177,11 +177,12 @@ func (a adminAPIHandlers) ServiceStopNRestartHandler(w http.ResponseWriter, r *h
// ServerProperties holds some server information such as, version, region
// uptime, etc..
type ServerProperties struct {
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
DeploymentID string `json:"deploymentID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
}
// ServerConnStats holds transferred bytes from/to the server
@@ -256,11 +257,12 @@ func (a adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Reque
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
DeploymentID: globalDeploymentID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
},
},
})
@@ -319,9 +321,8 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
}
vars := mux.Vars(r)
perfType := vars["perfType"]
if perfType == "drive" {
switch perfType := vars["perfType"]; perfType {
case "drive":
info := objectAPI.StorageInfo(ctx)
if !(info.Backend.Type == BackendFS || info.Backend.Type == BackendErasure) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrMethodNotAllowed), r.URL)
@@ -344,7 +345,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
// Reply with performance information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
} else if perfType == "cpu" {
case "cpu":
// Get CPU load details from local server's cpu(s)
cpu := localEndpointsCPULoad(globalEndpoints)
// Notify all other Minio peers to report cpu load numbers
@@ -361,7 +362,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
// Reply with cpu load information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
} else if perfType == "mem" {
case "mem":
// Get mem usage details from local server(s)
m := localEndpointsMemUsage(globalEndpoints)
// Notify all other Minio peers to report mem usage numbers
@@ -378,7 +379,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
// Reply with mem usage information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
} else {
default:
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrMethodNotAllowed), r.URL)
}
}
@@ -1479,14 +1480,6 @@ func (a adminAPIHandlers) UpdateAdminCredentialsHandler(w http.ResponseWriter,
return
}
// Notify all other Minio peers to update credentials
for _, nerr := range globalNotificationSys.LoadCredentials() {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
}
}
// Reply to the client before restarting minio server.
writeSuccessResponseHeadersOnly(w)
}
+2 -1
View File
@@ -122,7 +122,8 @@ var (
"password": "",
"reconnectInterval": 0,
"keepAliveInterval": 0,
"queueDir": ""
"queueDir": "",
"queueLimit": 0
}
},
"mysql": {
+27 -85
View File
@@ -21,14 +21,12 @@ import (
"encoding/json"
"fmt"
"net/http"
"runtime"
"strings"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/madmin"
"github.com/minio/minio/pkg/sync/errgroup"
)
// healStatusSummary - overall short summary of a healing sequence
@@ -573,51 +571,22 @@ func (h *healSequence) healMinioSysMeta(metaPrefix string) func() error {
// NOTE: Healing on meta is run regardless
// of any bucket being selected, this is to ensure that
// meta are always upto date and correct.
marker := ""
isTruncated := true
for isTruncated {
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
return objectAPI.HealObjects(h.ctx, minioMetaBucket, metaPrefix, func(bucket string, object string) error {
if h.isQuitting() {
return errHealStopSignalled
}
// Lists all objects under `config` prefix.
objectInfos, err := objectAPI.ListObjectsHeal(h.ctx, minioMetaBucket, metaPrefix,
marker, "", 1000)
if err != nil {
return errFnHealFromAPIErr(h.ctx, err)
res, herr := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove, h.settings.ScanMode)
// Object might have been deleted, by the time heal
// was attempted we ignore this object an move on.
if isErrObjectNotFound(herr) {
return nil
}
for index := range objectInfos.Objects {
if h.isQuitting() {
return errHealStopSignalled
}
o := objectInfos.Objects[index]
res, herr := objectAPI.HealObject(h.ctx, o.Bucket, o.Name, h.settings.DryRun, h.settings.Remove)
// Object might have been deleted, by the time heal
// was attempted we ignore this file an move on.
if isErrObjectNotFound(herr) {
continue
}
if herr != nil {
return herr
}
res.Type = madmin.HealItemBucketMetadata
if err = h.pushHealResultItem(res); err != nil {
return err
}
if herr != nil {
return herr
}
isTruncated = objectInfos.IsTruncated
marker = objectInfos.NextMarker
}
return nil
res.Type = madmin.HealItemBucketMetadata
return h.pushHealResultItem(res)
})
}
}
@@ -720,46 +689,9 @@ func (h *healSequence) healBucket(bucket string) error {
return nil
}
entries := runtime.NumCPU()
marker := ""
isTruncated := true
for isTruncated {
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
}
// Heal numCPU * nodes objects at a time.
objectInfos, err := objectAPI.ListObjectsHeal(h.ctx, bucket,
h.objPrefix, marker, "", entries)
if err != nil {
return errFnHealFromAPIErr(h.ctx, err)
}
g := errgroup.WithNErrs(len(objectInfos.Objects))
for index := range objectInfos.Objects {
index := index
g.Go(func() error {
o := objectInfos.Objects[index]
return h.healObject(o.Bucket, o.Name)
}, index)
}
for _, err := range g.Wait() {
if err != nil {
return err
}
}
isTruncated = objectInfos.IsTruncated
marker = objectInfos.NextMarker
if err = objectAPI.HealObjects(h.ctx, bucket,
h.objPrefix, h.healObject); err != nil {
return errFnHealFromAPIErr(h.ctx, err)
}
return nil
}
@@ -770,13 +702,23 @@ func (h *healSequence) healObject(bucket, object string) error {
return errHealStopSignalled
}
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
}
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
hri, err := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove)
hri, err := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove, h.settings.ScanMode)
if isErrObjectNotFound(err) {
return nil
}
+34 -13
View File
@@ -21,6 +21,7 @@ import (
"encoding/xml"
"fmt"
"net/http"
"strings"
"github.com/Azure/azure-sdk-for-go/storage"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
@@ -73,6 +74,7 @@ const (
ErrInvalidCopyPartRange
ErrInvalidCopyPartRangeSource
ErrInvalidMaxKeys
ErrInvalidEncodingMethod
ErrInvalidMaxUploads
ErrInvalidMaxParts
ErrInvalidPartNumberMarker
@@ -113,7 +115,6 @@ const (
ErrInvalidRequestVersion
ErrMissingSignTag
ErrMissingSignHeadersTag
ErrPolicyAlreadyExpired
ErrMalformedDate
ErrMalformedPresignedDate
ErrMalformedCredentialDate
@@ -179,6 +180,7 @@ const (
// Minio extended errors.
ErrReadQuorum
ErrWriteQuorum
ErrParentIsObject
ErrStorageFull
ErrRequestBodyParse
ErrObjectExistsAsDirectory
@@ -357,6 +359,11 @@ var errorCodes = errorCodeMap{
Description: "Argument maxKeys must be an integer between 0 and 2147483647",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidEncodingMethod: {
Code: "InvalidArgument",
Description: "Invalid Encoding Method specified in Request",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidMaxParts: {
Code: "InvalidArgument",
Description: "Argument max-parts must be an integer between 0 and 2147483647",
@@ -619,11 +626,6 @@ var errorCodes = errorCodeMap{
Description: "Signature header missing SignedHeaders field.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrPolicyAlreadyExpired: {
Code: "AccessDenied",
Description: "Invalid according to Policy: Policy expired.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrMalformedExpires: {
Code: "AuthorizationQueryParametersError",
Description: "X-Amz-Expires should be a number",
@@ -857,6 +859,11 @@ var errorCodes = errorCodeMap{
Description: "Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed.",
HTTPStatusCode: http.StatusInsufficientStorage,
},
ErrParentIsObject: {
Code: "XMinioParentIsObject",
Description: "Object-prefix is already an object, please choose a different object-prefix name.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrRequestBodyParse: {
Code: "XMinioRequestBodyParse",
Description: "The request body failed to parse.",
@@ -1506,8 +1513,6 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrInvalidEncryptionMethod
case crypto.ErrInvalidCustomerAlgorithm:
apiErr = ErrInvalidSSECustomerAlgorithm
case crypto.ErrInvalidCustomerKey:
apiErr = ErrInvalidSSECustomerKey
case crypto.ErrMissingCustomerKey:
apiErr = ErrMissingSSECustomerKey
case crypto.ErrMissingCustomerKeyMD5:
@@ -1562,6 +1567,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrObjectExistsAsDirectory
case PrefixAccessDenied:
apiErr = ErrAccessDenied
case ParentIsObject:
apiErr = ErrParentIsObject
case BucketNameInvalid:
apiErr = ErrInvalidBucketName
case BucketNotFound:
@@ -1641,11 +1648,25 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
case crypto.Error:
apiErr = ErrObjectTampered
default:
apiErr = ErrInternalError
// Make sure to log the errors which we cannot translate
// to a meaningful S3 API errors. This is added to aid in
// debugging unexpected/unhandled errors.
logger.LogIf(ctx, err)
var ie, iw int
// This work-around is to handle the issue golang/go#30648
if _, ferr := fmt.Fscanf(strings.NewReader(err.Error()),
"request declared a Content-Length of %d but only wrote %d bytes",
&ie, &iw); ferr != nil {
apiErr = ErrInternalError
// Make sure to log the errors which we cannot translate
// to a meaningful S3 API errors. This is added to aid in
// debugging unexpected/unhandled errors.
logger.LogIf(ctx, err)
} else if ie > iw {
apiErr = ErrIncompleteBody
} else {
apiErr = ErrInternalError
// Make sure to log the errors which we cannot translate
// to a meaningful S3 API errors. This is added to aid in
// debugging unexpected/unhandled errors.
logger.LogIf(ctx, err)
}
}
return apiErr
+1 -1
View File
@@ -55,7 +55,7 @@ var toAPIErrorTests = []struct {
// SSE-C errors
{err: crypto.ErrInvalidCustomerAlgorithm, errCode: ErrInvalidSSECustomerAlgorithm},
{err: crypto.ErrMissingCustomerKey, errCode: ErrMissingSSECustomerKey},
{err: crypto.ErrInvalidCustomerKey, errCode: ErrInvalidSSECustomerKey},
{err: crypto.ErrInvalidCustomerKey, errCode: ErrAccessDenied},
{err: crypto.ErrMissingCustomerKeyMD5, errCode: ErrMissingSSECustomerKeyMD5},
{err: crypto.ErrCustomerKeyMD5Mismatch, errCode: ErrSSECustomerKeyMD5Mismatch},
{err: errObjectTampered, errCode: ErrObjectTampered},
+3
View File
@@ -87,6 +87,9 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, rs *HTTPRangeSp
w.Header().Set("Content-Encoding", objInfo.ContentEncoding)
}
if !objInfo.Expires.IsZero() {
w.Header().Set("Expires", objInfo.Expires.UTC().Format(http.TimeFormat))
}
// Set all other user defined metadata.
for k, v := range objInfo.UserDefined {
if hasPrefix(k, ReservedMetadataPrefix) {
-60
View File
@@ -1,60 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"net/http"
)
// Represents additional fields necessary for ErrPartTooSmall S3 error.
type completeMultipartAPIError struct {
// Proposed size represents uploaded size of the part.
ProposedSize int64
// Minimum size allowed epresents the minimum size allowed per
// part. Defaults to 5MB.
MinSizeAllowed int64
// Part number of the part which is incorrect.
PartNumber int
// ETag of the part which is incorrect.
PartETag string
// Other default XML error responses.
APIErrorResponse
}
// writeErrorResponsePartTooSmall - function is used specifically to
// construct a proper error response during CompleteMultipartUpload
// when one of the parts is < 5MB.
// The requirement comes due to the fact that generic ErrorResponse
// XML doesn't carry the additional fields required to send this
// error. So we construct a new type which lies well within the scope
// of this function.
func writePartSmallErrorResponse(ctx context.Context, w http.ResponseWriter, r *http.Request, err PartTooSmall) {
apiError := toAPIError(ctx, err)
// Generate complete multipart error response.
errorResponse := getAPIErrorResponse(ctx, apiError, r.URL.Path,
w.Header().Get(responseRequestIDKey),
w.Header().Get(responseDeploymentIDKey))
cmpErrResp := completeMultipartAPIError{err.PartSize, int64(5242880), err.PartNumber, err.PartETag, errorResponse}
encodedErrorResponse := encodeResponse(cmpErrResp)
// respond with 400 bad request.
w.WriteHeader(apiError.HTTPStatusCode)
// Write error body.
w.Write(encodedErrorResponse)
w.(http.Flusher).Flush()
}
+80 -29
View File
@@ -280,7 +280,7 @@ func getURLScheme(tls bool) string {
}
// getObjectLocation gets the fully qualified URL of an object.
func getObjectLocation(r *http.Request, domain, bucket, object string) string {
func getObjectLocation(r *http.Request, domains []string, bucket, object string) string {
// unit tests do not have host set.
if r.Host == "" {
return path.Clean(r.URL.Path)
@@ -295,15 +295,31 @@ func getObjectLocation(r *http.Request, domain, bucket, object string) string {
Scheme: proto,
}
// If domain is set then we need to use bucket DNS style.
if domain != "" {
for _, domain := range domains {
if strings.Contains(r.Host, domain) {
u.Host = bucket + "." + r.Host
u.Path = path.Join(slashSeparator, object)
break
}
}
return u.String()
}
// s3EncodeName encodes string in response when encodingType
// is specified in AWS S3 requests.
func s3EncodeName(name string, encodingType string) (result string) {
// Quick path to exit
if encodingType == "" {
return name
}
encodingType = strings.ToLower(encodingType)
switch encodingType {
case "url":
return url.QueryEscape(name)
}
return name
}
// generates ListBucketsResponse from array of BucketInfo which can be
// serialized to match XML and JSON API spec output.
func generateListBucketsResponse(buckets []BucketInfo) ListBucketsResponse {
@@ -326,7 +342,7 @@ func generateListBucketsResponse(buckets []BucketInfo) ListBucketsResponse {
}
// generates an ListObjectsV1 response for the said bucket with other enumerated options.
func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, maxKeys int, resp ListObjectsInfo) ListObjectsResponse {
func generateListObjectsV1Response(bucket, prefix, marker, delimiter, encodingType string, maxKeys int, resp ListObjectsInfo) ListObjectsResponse {
var contents []Object
var prefixes []CommonPrefix
var owner = Owner{}
@@ -338,7 +354,7 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
if object.Name == "" {
continue
}
content.Key = object.Name
content.Key = s3EncodeName(object.Name, encodingType)
content.LastModified = object.ModTime.UTC().Format(timeFormatAMZLong)
if object.ETag != "" {
content.ETag = "\"" + object.ETag + "\""
@@ -348,20 +364,20 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
content.Owner = owner
contents = append(contents, content)
}
// TODO - support EncodingType in xml decoding
data.Name = bucket
data.Contents = contents
data.Prefix = prefix
data.Marker = marker
data.Delimiter = delimiter
data.EncodingType = encodingType
data.Prefix = s3EncodeName(prefix, encodingType)
data.Marker = s3EncodeName(marker, encodingType)
data.Delimiter = s3EncodeName(delimiter, encodingType)
data.MaxKeys = maxKeys
data.NextMarker = resp.NextMarker
data.NextMarker = s3EncodeName(resp.NextMarker, encodingType)
data.IsTruncated = resp.IsTruncated
for _, prefix := range resp.Prefixes {
var prefixItem = CommonPrefix{}
prefixItem.Prefix = prefix
prefixItem.Prefix = s3EncodeName(prefix, encodingType)
prefixes = append(prefixes, prefixItem)
}
data.CommonPrefixes = prefixes
@@ -369,7 +385,7 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
}
// generates an ListObjectsV2 response for the said bucket with other enumerated options.
func generateListObjectsV2Response(bucket, prefix, token, nextToken, startAfter, delimiter string, fetchOwner, isTruncated bool, maxKeys int, objects []ObjectInfo, prefixes []string) ListObjectsV2Response {
func generateListObjectsV2Response(bucket, prefix, token, nextToken, startAfter, delimiter, encodingType string, fetchOwner, isTruncated bool, maxKeys int, objects []ObjectInfo, prefixes []string) ListObjectsV2Response {
var contents []Object
var commonPrefixes []CommonPrefix
var owner = Owner{}
@@ -384,7 +400,7 @@ func generateListObjectsV2Response(bucket, prefix, token, nextToken, startAfter,
if object.Name == "" {
continue
}
content.Key = object.Name
content.Key = s3EncodeName(object.Name, encodingType)
content.LastModified = object.ModTime.UTC().Format(timeFormatAMZLong)
if object.ETag != "" {
content.ETag = "\"" + object.ETag + "\""
@@ -394,20 +410,20 @@ func generateListObjectsV2Response(bucket, prefix, token, nextToken, startAfter,
content.Owner = owner
contents = append(contents, content)
}
// TODO - support EncodingType in xml decoding
data.Name = bucket
data.Contents = contents
data.StartAfter = startAfter
data.Delimiter = delimiter
data.Prefix = prefix
data.EncodingType = encodingType
data.StartAfter = s3EncodeName(startAfter, encodingType)
data.Delimiter = s3EncodeName(delimiter, encodingType)
data.Prefix = s3EncodeName(prefix, encodingType)
data.MaxKeys = maxKeys
data.ContinuationToken = token
data.NextContinuationToken = nextToken
data.IsTruncated = isTruncated
for _, prefix := range prefixes {
var prefixItem = CommonPrefix{}
prefixItem.Prefix = prefix
prefixItem.Prefix = s3EncodeName(prefix, encodingType)
commonPrefixes = append(commonPrefixes, prefixItem)
}
data.CommonPrefixes = commonPrefixes
@@ -451,11 +467,10 @@ func generateCompleteMultpartUploadResponse(bucket, key, location, etag string)
}
// generates ListPartsResponse from ListPartsInfo.
func generateListPartsResponse(partsInfo ListPartsInfo) ListPartsResponse {
// TODO - support EncodingType in xml decoding
func generateListPartsResponse(partsInfo ListPartsInfo, encodingType string) ListPartsResponse {
listPartsResponse := ListPartsResponse{}
listPartsResponse.Bucket = partsInfo.Bucket
listPartsResponse.Key = partsInfo.Object
listPartsResponse.Key = s3EncodeName(partsInfo.Object, encodingType)
listPartsResponse.UploadID = partsInfo.UploadID
listPartsResponse.StorageClass = globalMinioDefaultStorageClass
listPartsResponse.Initiator.ID = globalMinioDefaultOwnerID
@@ -479,29 +494,29 @@ func generateListPartsResponse(partsInfo ListPartsInfo) ListPartsResponse {
}
// generates ListMultipartUploadsResponse for given bucket and ListMultipartsInfo.
func generateListMultipartUploadsResponse(bucket string, multipartsInfo ListMultipartsInfo) ListMultipartUploadsResponse {
func generateListMultipartUploadsResponse(bucket string, multipartsInfo ListMultipartsInfo, encodingType string) ListMultipartUploadsResponse {
listMultipartUploadsResponse := ListMultipartUploadsResponse{}
listMultipartUploadsResponse.Bucket = bucket
listMultipartUploadsResponse.Delimiter = multipartsInfo.Delimiter
listMultipartUploadsResponse.Delimiter = s3EncodeName(multipartsInfo.Delimiter, encodingType)
listMultipartUploadsResponse.IsTruncated = multipartsInfo.IsTruncated
listMultipartUploadsResponse.EncodingType = multipartsInfo.EncodingType
listMultipartUploadsResponse.Prefix = multipartsInfo.Prefix
listMultipartUploadsResponse.KeyMarker = multipartsInfo.KeyMarker
listMultipartUploadsResponse.NextKeyMarker = multipartsInfo.NextKeyMarker
listMultipartUploadsResponse.EncodingType = encodingType
listMultipartUploadsResponse.Prefix = s3EncodeName(multipartsInfo.Prefix, encodingType)
listMultipartUploadsResponse.KeyMarker = s3EncodeName(multipartsInfo.KeyMarker, encodingType)
listMultipartUploadsResponse.NextKeyMarker = s3EncodeName(multipartsInfo.NextKeyMarker, encodingType)
listMultipartUploadsResponse.MaxUploads = multipartsInfo.MaxUploads
listMultipartUploadsResponse.NextUploadIDMarker = multipartsInfo.NextUploadIDMarker
listMultipartUploadsResponse.UploadIDMarker = multipartsInfo.UploadIDMarker
listMultipartUploadsResponse.CommonPrefixes = make([]CommonPrefix, len(multipartsInfo.CommonPrefixes))
for index, commonPrefix := range multipartsInfo.CommonPrefixes {
listMultipartUploadsResponse.CommonPrefixes[index] = CommonPrefix{
Prefix: commonPrefix,
Prefix: s3EncodeName(commonPrefix, encodingType),
}
}
listMultipartUploadsResponse.Uploads = make([]Upload, len(multipartsInfo.Uploads))
for index, upload := range multipartsInfo.Uploads {
newUpload := Upload{}
newUpload.UploadID = upload.UploadID
newUpload.Key = upload.Object
newUpload.Key = s3EncodeName(upload.Object, encodingType)
newUpload.Initiated = upload.Initiated.UTC().Format(timeFormatAMZLong)
listMultipartUploadsResponse.Uploads[index] = newUpload
}
@@ -587,7 +602,8 @@ func writeErrorResponse(ctx context.Context, w http.ResponseWriter, err APIError
}
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path, w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path,
w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
encodedErrorResponse := encodeResponse(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeXML)
}
@@ -624,3 +640,38 @@ func writeCustomErrorResponseJSON(ctx context.Context, w http.ResponseWriter, er
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeJSON)
}
// writeCustomErrorResponseXML - similar to writeErrorResponse,
// but accepts the error message directly (this allows messages to be
// dynamically generated.)
func writeCustomErrorResponseXML(ctx context.Context, w http.ResponseWriter, err APIError, errBody string, reqURL *url.URL, browser bool) {
switch err.Code {
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
case "AccessDenied":
// The request is from browser and also if browser
// is enabled we need to redirect.
if browser && globalIsBrowserEnabled {
w.Header().Set("Location", minioReservedBucketPath+reqURL.Path)
w.WriteHeader(http.StatusTemporaryRedirect)
return
}
}
reqInfo := logger.GetReqInfo(ctx)
errorResponse := APIErrorResponse{
Code: err.Code,
Message: errBody,
Resource: reqURL.Path,
BucketName: reqInfo.BucketName,
Key: reqInfo.ObjectName,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
}
encodedErrorResponse := encodeResponse(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeXML)
}
+4 -4
View File
@@ -26,7 +26,7 @@ func TestObjectLocation(t *testing.T) {
testCases := []struct {
request *http.Request
bucket, object string
domain string
domains []string
expectedLocation string
}{
// Server binding to localhost IP with https.
@@ -80,7 +80,7 @@ func TestObjectLocation(t *testing.T) {
Host: "mys3.bucket.org",
Header: map[string][]string{},
},
domain: "mys3.bucket.org",
domains: []string{"mys3.bucket.org"},
bucket: "mybucket",
object: "test/1.txt",
expectedLocation: "http://mybucket.mys3.bucket.org/test/1.txt",
@@ -92,14 +92,14 @@ func TestObjectLocation(t *testing.T) {
"X-Forwarded-Scheme": {httpsScheme},
},
},
domain: "mys3.bucket.org",
domains: []string{"mys3.bucket.org"},
bucket: "mybucket",
object: "test/1.txt",
expectedLocation: "https://mybucket.mys3.bucket.org/test/1.txt",
},
}
for i, testCase := range testCases {
gotLocation := getObjectLocation(testCase.request, testCase.domain, testCase.bucket, testCase.object)
gotLocation := getObjectLocation(testCase.request, testCase.domains, testCase.bucket, testCase.object)
if testCase.expectedLocation != gotLocation {
t.Errorf("Test %d: expected %s, got %s", i+1, testCase.expectedLocation, gotLocation)
}
+2 -2
View File
@@ -44,8 +44,8 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled bool) {
// API Router
apiRouter := router.PathPrefix("/").Subrouter()
var routers []*mux.Router
if globalDomainName != "" {
routers = append(routers, apiRouter.Host("{bucket:.+}."+globalDomainName).Subrouter())
for _, domainName := range globalDomainNames {
routers = append(routers, apiRouter.Host("{bucket:.+}."+domainName).Subrouter())
}
routers = append(routers, apiRouter.PathPrefix("/{bucket}").Subrouter())
+12 -12
View File
@@ -32,7 +32,7 @@ import (
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/iam/policy"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/policy"
)
@@ -126,7 +126,7 @@ func checkAdminRequestAuthType(ctx context.Context, r *http.Request, region stri
// We only support admin credentials to access admin APIs.
var owner bool
_, owner, s3Err = getReqAccessKeyV4(r, region)
_, owner, s3Err = getReqAccessKeyV4(r, region, serviceS3)
if s3Err != ErrNone {
return s3Err
}
@@ -136,7 +136,7 @@ func checkAdminRequestAuthType(ctx context.Context, r *http.Request, region stri
}
// we only support V4 (no presign) with auth body
s3Err = isReqAuthenticated(ctx, r, region)
s3Err = isReqAuthenticated(ctx, r, region, serviceS3)
}
if s3Err != ErrNone {
reqInfo := (&logger.ReqInfo{}).AppendTags("requestHeaders", dumpRequest(r))
@@ -241,10 +241,10 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
case policy.GetBucketLocationAction, policy.ListAllMyBucketsAction:
region = ""
}
if s3Err = isReqAuthenticated(ctx, r, region); s3Err != ErrNone {
if s3Err = isReqAuthenticated(ctx, r, region, serviceS3); s3Err != ErrNone {
return s3Err
}
cred, owner, s3Err = getReqAccessKeyV4(r, region)
cred, owner, s3Err = getReqAccessKeyV4(r, region, serviceS3)
}
if s3Err != ErrNone {
return s3Err
@@ -314,21 +314,21 @@ func isReqAuthenticatedV2(r *http.Request) (s3Error APIErrorCode) {
return doesPresignV2SignatureMatch(r)
}
func reqSignatureV4Verify(r *http.Request, region string) (s3Error APIErrorCode) {
sha256sum := getContentSha256Cksum(r)
func reqSignatureV4Verify(r *http.Request, region string, stype serviceType) (s3Error APIErrorCode) {
sha256sum := getContentSha256Cksum(r, stype)
switch {
case isRequestSignatureV4(r):
return doesSignatureMatch(sha256sum, r, region)
return doesSignatureMatch(sha256sum, r, region, stype)
case isRequestPresignedSignatureV4(r):
return doesPresignedSignatureMatch(sha256sum, r, region)
return doesPresignedSignatureMatch(sha256sum, r, region, stype)
default:
return ErrAccessDenied
}
}
// Verify if request has valid AWS Signature Version '4'.
func isReqAuthenticated(ctx context.Context, r *http.Request, region string) (s3Error APIErrorCode) {
if errCode := reqSignatureV4Verify(r, region); errCode != ErrNone {
func isReqAuthenticated(ctx context.Context, r *http.Request, region string, stype serviceType) (s3Error APIErrorCode) {
if errCode := reqSignatureV4Verify(r, region, stype); errCode != ErrNone {
return errCode
}
@@ -432,7 +432,7 @@ func isPutAllowed(atype authType, bucketName, objectName string, r *http.Request
cred, owner, s3Err = getReqAccessKeyV2(r)
case authTypeStreamingSigned, authTypePresigned, authTypeSigned:
region := globalServerConfig.GetRegion()
cred, owner, s3Err = getReqAccessKeyV4(r, region)
cred, owner, s3Err = getReqAccessKeyV4(r, region, serviceS3)
}
if s3Err != ErrNone {
return s3Err
+2 -1
View File
@@ -381,7 +381,8 @@ func TestIsReqAuthenticated(t *testing.T) {
ctx := context.Background()
// Validates all testcases.
for i, testCase := range testCases {
if s3Error := isReqAuthenticated(ctx, testCase.req, globalServerConfig.GetRegion()); s3Error != testCase.s3Error {
s3Error := isReqAuthenticated(ctx, testCase.req, globalServerConfig.GetRegion(), serviceS3)
if s3Error != testCase.s3Error {
if _, err := ioutil.ReadAll(testCase.req.Body); toAPIErrorCode(ctx, err) != testCase.s3Error {
t.Fatalf("Test %d: Unexpected S3 error: want %d - got %d (got after reading request %s)", i, testCase.s3Error, s3Error, toAPIError(ctx, err).Code)
}
+7 -3
View File
@@ -84,9 +84,11 @@ func newStreamingBitrotWriter(disk StorageAPI, volume, filePath string, length i
totalFileSize := bitrotSumsTotalSize + length
err := disk.CreateFile(volume, filePath, totalFileSize, r)
if err != nil {
logger.LogIf(context.Background(), err)
r.CloseWithError(err)
reqInfo := (&logger.ReqInfo{}).AppendTags("storageDisk", disk.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
r.CloseWithError(err)
close(bw.canClose)
}()
return bw
@@ -125,7 +127,9 @@ func (b *streamingBitrotReader) ReadAt(buf []byte, offset int64) (int, error) {
streamOffset := (offset/b.shardSize)*int64(b.h.Size()) + offset
b.rc, err = b.disk.ReadFileStream(b.volume, b.filePath, streamOffset, b.tillOffset-streamOffset)
if err != nil {
logger.LogIf(context.Background(), err)
reqInfo := (&logger.ReqInfo{}).AppendTags("storageDisk", b.disk.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return 0, err
}
}
+2 -2
View File
@@ -133,7 +133,7 @@ func newBitrotReader(disk StorageAPI, bucket string, filePath string, tillOffset
// Close all the readers.
func closeBitrotReaders(rs []io.ReaderAt) {
for _, r := range rs {
if br, ok := r.(*streamingBitrotReader); ok {
if br, ok := r.(io.Closer); ok {
br.Close()
}
}
@@ -142,7 +142,7 @@ func closeBitrotReaders(rs []io.ReaderAt) {
// Close all the writers.
func closeBitrotWriters(ws []io.Writer) {
for _, w := range ws {
if bw, ok := w.(*streamingBitrotWriter); ok {
if bw, ok := w.(io.Closer); ok {
bw.Close()
}
}
+15 -7
View File
@@ -18,6 +18,7 @@ package cmd
import (
"net/http"
"strings"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/crypto"
@@ -32,12 +33,19 @@ import (
// - delimiter if set should be equal to '/', otherwise the request is rejected.
// - marker if set should have a common prefix with 'prefix' param, otherwise
// the request is rejected.
func validateListObjectsArgs(prefix, marker, delimiter string, maxKeys int) APIErrorCode {
func validateListObjectsArgs(prefix, marker, delimiter, encodingType string, maxKeys int) APIErrorCode {
// Max keys cannot be negative.
if maxKeys < 0 {
return ErrInvalidMaxKeys
}
if encodingType != "" {
// Only url encoding type is supported
if strings.ToLower(encodingType) != "url" {
return ErrInvalidEncodingMethod
}
}
/// Minio special conditions for ListObjects.
// Verify if delimiter is anything other than '/', which we do not support.
@@ -78,7 +86,7 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
urlValues := r.URL.Query()
// Extract all the listObjectsV2 query params to their native values.
prefix, token, startAfter, delimiter, fetchOwner, maxKeys, _, errCode := getListObjectsV2Args(urlValues)
prefix, token, startAfter, delimiter, fetchOwner, maxKeys, encodingType, errCode := getListObjectsV2Args(urlValues)
if errCode != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(errCode), r.URL, guessIsBrowserReq(r))
return
@@ -86,7 +94,7 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
// Validate the query params before beginning to serve the request.
// fetch-owner is not validated since it is a boolean
if s3Error := validateListObjectsArgs(prefix, token, delimiter, maxKeys); s3Error != ErrNone {
if s3Error := validateListObjectsArgs(prefix, token, delimiter, encodingType, maxKeys); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
@@ -126,7 +134,7 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
}
response := generateListObjectsV2Response(bucket, prefix, token, listObjectsV2Info.NextContinuationToken, startAfter,
delimiter, fetchOwner, listObjectsV2Info.IsTruncated, maxKeys, listObjectsV2Info.Objects, listObjectsV2Info.Prefixes)
delimiter, encodingType, fetchOwner, listObjectsV2Info.IsTruncated, maxKeys, listObjectsV2Info.Objects, listObjectsV2Info.Prefixes)
// Write success response.
writeSuccessResponseXML(w, encodeResponse(response))
@@ -158,14 +166,14 @@ func (api objectAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *http
}
// Extract all the litsObjectsV1 query params to their native values.
prefix, marker, delimiter, maxKeys, _, s3Error := getListObjectsV1Args(r.URL.Query())
prefix, marker, delimiter, maxKeys, encodingType, s3Error := getListObjectsV1Args(r.URL.Query())
if s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
// Validate all the query params before beginning to serve the request.
if s3Error := validateListObjectsArgs(prefix, marker, delimiter, maxKeys); s3Error != ErrNone {
if s3Error := validateListObjectsArgs(prefix, marker, delimiter, encodingType, maxKeys); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
@@ -204,7 +212,7 @@ func (api objectAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *http
}
}
}
response := generateListObjectsV1Response(bucket, prefix, marker, delimiter, maxKeys, listObjectsInfo)
response := generateListObjectsV1Response(bucket, prefix, marker, delimiter, encodingType, maxKeys, listObjectsInfo)
// Write success response.
writeSuccessResponseXML(w, encodeResponse(response))
+10 -26
View File
@@ -22,7 +22,6 @@ import (
"encoding/xml"
"fmt"
"io"
"net"
"net/http"
"net/url"
"path"
@@ -155,7 +154,7 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
return
}
prefix, keyMarker, uploadIDMarker, delimiter, maxUploads, _, errCode := getBucketMultipartResources(r.URL.Query())
prefix, keyMarker, uploadIDMarker, delimiter, maxUploads, encodingType, errCode := getBucketMultipartResources(r.URL.Query())
if errCode != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(errCode), r.URL, guessIsBrowserReq(r))
return
@@ -180,7 +179,7 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
return
}
// generate response
response := generateListMultipartUploadsResponse(bucket, listMultipartsInfo)
response := generateListMultipartUploadsResponse(bucket, listMultipartsInfo, encodingType)
encodedSuccessResponse := encodeResponse(response)
// write success response.
@@ -226,7 +225,7 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
continue
}
bucketsInfo = append(bucketsInfo, BucketInfo{
Name: strings.Trim(dnsRecord.Key, slashSeparator),
Name: dnsRecord.Key,
Created: dnsRecord.CreationDate,
})
bucketSet.Add(dnsRecord.Key)
@@ -371,13 +370,6 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr failing which
// fill them with empty strings.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify deleted event for objects.
for _, dobj := range deletedObjects {
sendEvent(eventArgs{
@@ -389,8 +381,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
}
@@ -446,7 +437,7 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
}
// Make sure to add Location information here only for bucket
w.Header().Set("Location", getObjectLocation(r, globalDomainName, bucket, ""))
w.Header().Set("Location", getObjectLocation(r, globalDomainNames, bucket, ""))
writeSuccessResponseHeadersOnly(w)
return
@@ -505,7 +496,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentLength), r.URL, guessIsBrowserReq(r))
return
}
resource, err := getResource(r.URL.Path, r.Host, globalDomainName)
resource, err := getResource(r.URL.Path, r.Host, globalDomainNames)
if err != nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidRequest), r.URL, guessIsBrowserReq(r))
return
@@ -595,8 +586,8 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
}
// Make sure formValues adhere to policy restrictions.
if errCode = checkPostPolicy(formValues, postPolicyForm); errCode != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(errCode), r.URL, guessIsBrowserReq(r))
if err = checkPostPolicy(formValues, postPolicyForm); err != nil {
writeCustomErrorResponseXML(ctx, w, errorCodes.ToAPIErr(ErrAccessDenied), err.Error(), r.URL, guessIsBrowserReq(r))
return
}
@@ -677,16 +668,10 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
location := getObjectLocation(r, globalDomainName, bucket, object)
location := getObjectLocation(r, globalDomainNames, bucket, object)
w.Header().Set("ETag", `"`+objInfo.ETag+`"`)
w.Header().Set("Location", location)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object created event.
defer sendEvent(eventArgs{
EventName: event.ObjectCreatedPost,
@@ -695,8 +680,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
if successRedirect != "" {
+11 -7
View File
@@ -262,10 +262,14 @@ func handleCommonEnvVars() {
logger.FatalIf(err, "Unable to initialize etcd with %s", etcdEndpoints)
}
globalDomainName, globalIsEnvDomainName = os.LookupEnv("MINIO_DOMAIN")
if globalDomainName != "" {
if _, ok = dns2.IsDomainName(globalDomainName); !ok {
logger.Fatal(uiErrInvalidDomainValue(nil).Msg("Unknown value `%s`", globalDomainName), "Invalid MINIO_DOMAIN value in environment variable")
v, ok := os.LookupEnv("MINIO_DOMAIN")
if ok {
for _, domainName := range strings.Split(v, ",") {
if _, ok = dns2.IsDomainName(domainName); !ok {
logger.Fatal(uiErrInvalidDomainValue(nil).Msg("Unknown value `%s`", domainName),
"Invalid MINIO_DOMAIN value in environment variable")
}
globalDomainNames = append(globalDomainNames, domainName)
}
}
@@ -294,10 +298,10 @@ func handleCommonEnvVars() {
updateDomainIPs(localIP4)
}
if globalDomainName != "" && !globalDomainIPs.IsEmpty() && globalEtcdClient != nil {
if len(globalDomainNames) != 0 && !globalDomainIPs.IsEmpty() && globalEtcdClient != nil {
var err error
globalDNSConfig, err = dns.NewCoreDNS(globalDomainName, globalDomainIPs, globalMinioPort, globalEtcdClient)
logger.FatalIf(err, "Unable to initialize DNS config for %s.", globalDomainName)
globalDNSConfig, err = dns.NewCoreDNS(globalDomainNames, globalDomainIPs, globalMinioPort, globalEtcdClient)
logger.FatalIf(err, "Unable to initialize DNS config for %s.", globalDomainNames)
}
if drives := os.Getenv("MINIO_CACHE_DRIVES"); drives != "" {
+3 -3
View File
@@ -130,8 +130,8 @@ func TestServerConfigWithEnvs(t *testing.T) {
}
// Check if serverConfig has the correct domain
if globalDomainName != "domain.com" {
t.Errorf("Expected Domain to be `domain.com`, found " + globalDomainName)
if globalDomainNames[0] != "domain.com" {
t.Errorf("Expected Domain to be `domain.com`, found " + globalDomainNames[0])
}
}
@@ -233,7 +233,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "redis": { "1": { "enable": true, "format": "namespace", "address": "example.com:80", "password": "xxx", "key": "key1" } }}}`, true},
// Test 27 - Test MQTT
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mqtt": { "1": { "enable": true, "broker": "", "topic": "", "qos": 0, "username": "", "password": "", "queueDir": ""}}}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mqtt": { "1": { "enable": true, "broker": "", "topic": "", "qos": 0, "username": "", "password": "", "queueDir": "", "queueLimit": 0}}}}`, false},
// Test 28 - Test NSQ
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nsq": { "1": { "enable": true, "nsqdAddress": "", "topic": ""} }}}`, false},
+1 -1
View File
@@ -884,7 +884,7 @@ type serverConfigV32 struct {
} `json:"policy"`
}
// serverConfigV33 is just like version '32', removes clientID from NATS and MQTT, and adds queueDir with MQTT.
// serverConfigV33 is just like version '32', removes clientID from NATS and MQTT, and adds queueDir, queueLimit with MQTT.
type serverConfigV33 struct {
quick.Config `json:"-"` // ignore interfaces
+2 -2
View File
@@ -332,7 +332,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
@@ -350,7 +350,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
// Validate input data size and it can never be less than zero.
+3 -4
View File
@@ -254,8 +254,7 @@ func (c cacheObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
cleanupBackend := func() { bkReader.Close() }
cleanupPipe := func() { pipeReader.Close() }
gr = NewGetObjectReaderFromReader(teeReader, bkReader.ObjInfo, cleanupBackend, cleanupPipe)
return gr, nil
return NewGetObjectReaderFromReader(teeReader, bkReader.ObjInfo, opts.CheckCopyPrecondFn, cleanupBackend, cleanupPipe)
}
// Uses cached-object to serve the request. If object is not cached it serves the request from the backend and also
@@ -424,7 +423,7 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
if delimiter == slashSeparator {
recursive = false
}
walkResultCh, endWalkCh := c.listPool.Release(listParams{bucket, recursive, marker, prefix, false})
walkResultCh, endWalkCh := c.listPool.Release(listParams{bucket, recursive, marker, prefix})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, object string) bool {
@@ -496,7 +495,7 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
}
}
params := listParams{bucket, recursive, nextMarker, prefix, false}
params := listParams{bucket, recursive, nextMarker, prefix}
if !eof {
c.listPool.Set(params, walkResultCh, endWalkCh)
}
+8 -20
View File
@@ -205,7 +205,6 @@ func (endpoints EndpointList) GetString(i int) string {
func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
var memUsages []mem.Usage
var historicUsages []mem.Usage
var addr string
scratchSpace := map[string]bool{}
for _, endpoint := range endpoints {
// Only proceed for local endpoints
@@ -213,16 +212,13 @@ func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
if _, ok := scratchSpace[endpoint.Host]; ok {
continue
}
addr = GetLocalPeer(endpoints)
memUsage := mem.GetUsage()
memUsages = append(memUsages, memUsage)
historicUsage := mem.GetHistoricUsage()
historicUsages = append(historicUsages, historicUsage)
memUsages = append(memUsages, mem.GetUsage())
historicUsages = append(historicUsages, mem.GetHistoricUsage())
scratchSpace[endpoint.Host] = true
}
}
return ServerMemUsageInfo{
Addr: addr,
Addr: GetLocalPeer(endpoints),
Usage: memUsages,
HistoricUsage: historicUsages,
}
@@ -233,7 +229,6 @@ func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
func localEndpointsCPULoad(endpoints EndpointList) ServerCPULoadInfo {
var cpuLoads []cpu.Load
var historicLoads []cpu.Load
var addr string
scratchSpace := map[string]bool{}
for _, endpoint := range endpoints {
// Only proceed for local endpoints
@@ -241,16 +236,13 @@ func localEndpointsCPULoad(endpoints EndpointList) ServerCPULoadInfo {
if _, ok := scratchSpace[endpoint.Host]; ok {
continue
}
addr = GetLocalPeer(endpoints)
cpuLoad := cpu.GetLoad()
cpuLoads = append(cpuLoads, cpuLoad)
historicLoad := cpu.GetHistoricLoad()
historicLoads = append(historicLoads, historicLoad)
cpuLoads = append(cpuLoads, cpu.GetLoad())
historicLoads = append(historicLoads, cpu.GetHistoricLoad())
scratchSpace[endpoint.Host] = true
}
}
return ServerCPULoadInfo{
Addr: addr,
Addr: GetLocalPeer(endpoints),
Load: cpuLoads,
HistoricLoad: historicLoads,
}
@@ -260,26 +252,22 @@ func localEndpointsCPULoad(endpoints EndpointList) ServerCPULoadInfo {
// local endpoints from given list of endpoints
func localEndpointsDrivePerf(endpoints EndpointList) ServerDrivesPerfInfo {
var dps []disk.Performance
var addr string
for _, endpoint := range endpoints {
// Only proceed for local endpoints
if endpoint.IsLocal {
addr = GetLocalPeer(endpoints)
if _, err := os.Stat(endpoint.Path); err != nil {
// Since this drive is not available, add relevant details and proceed
dps = append(dps, disk.Performance{Path: endpoint.Path, Error: err.Error()})
continue
}
tempObj := mustGetUUID()
fsPath := pathJoin(endpoint.Path, minioMetaTmpBucket, tempObj)
dp := disk.GetPerformance(fsPath)
dp := disk.GetPerformance(pathJoin(endpoint.Path, minioMetaTmpBucket, mustGetUUID()))
dp.Path = endpoint.Path
dps = append(dps, dp)
}
}
return ServerDrivesPerfInfo{
Addr: addr,
Addr: GetLocalPeer(endpoints),
Perf: dps,
}
}
+11 -1
View File
@@ -22,8 +22,10 @@ import (
"encoding/json"
"io"
"io/ioutil"
"net/http"
"os"
pathutil "path"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
@@ -169,7 +171,15 @@ func (m fsMetaV1) ToObjectInfo(bucket, object string, fi os.FileInfo) ObjectInfo
} else {
objInfo.StorageClass = globalMinioDefaultStorageClass
}
var (
t time.Time
e error
)
if exp, ok := m.Meta["expires"]; ok {
if t, e = time.Parse(http.TimeFormat, exp); e == nil {
objInfo.Expires = t.UTC()
}
}
// etag/md5Sum has already been extracted. We need to
// remove to avoid it from appearing as part of
// response headers. e.g, X-Minio-* or X-Amz-*.
+3
View File
@@ -38,6 +38,9 @@ func TestFSV1MetadataObjInfo(t *testing.T) {
if objInfo.IsDir {
t.Fatal("Unexpected object info value for IsDir", objInfo.IsDir)
}
if !objInfo.Expires.IsZero() {
t.Fatal("Unexpected object info value for Expires ", objInfo.Expires)
}
}
// TestReadFSMetadata - readFSMetadata testing with a healthy and faulty disk
+1 -1
View File
@@ -487,7 +487,7 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, pathutil.Dir(object)) {
return oi, toObjectErr(errFileAccessDenied, bucket, object)
return oi, toObjectErr(errFileParentIsFile, bucket, object)
}
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
+20 -26
View File
@@ -30,6 +30,7 @@ import (
"sync/atomic"
"time"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
"github.com/minio/minio/pkg/madmin"
@@ -289,10 +290,8 @@ func (fs *FSObjects) MakeBucketWithLocation(ctx context.Context, bucket, locatio
}
defer bucketLock.Unlock()
// Verify if bucket is valid.
if !IsValidBucketName(bucket) {
err := BucketNameInvalid{Bucket: bucket}
logger.LogIf(ctx, err)
return err
if s3utils.CheckValidBucketNameStrict(bucket) != nil {
return BucketNameInvalid{Bucket: bucket}
}
bucketDir, err := fs.getBucketDir(ctx, bucket)
if err != nil {
@@ -341,7 +340,7 @@ func (fs *FSObjects) ListBuckets(ctx context.Context) ([]BucketInfo, error) {
for _, entry := range entries {
// Ignore all reserved bucket names and invalid bucket names.
if isReservedOrInvalidBucket(entry) {
if isReservedOrInvalidBucket(entry, false) {
continue
}
var fi os.FileInfo
@@ -490,20 +489,18 @@ func (fs *FSObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
}
}
// For a directory, we need to send an reader that returns no bytes.
if hasSuffix(object, slashSeparator) {
// The lock taken above is released when
// objReader.Close() is called by the caller.
return NewGetObjectReaderFromReader(bytes.NewBuffer(nil), ObjectInfo{}, nsUnlocker), nil
}
// Otherwise we get the object info
var objInfo ObjectInfo
if objInfo, err = fs.getObjectInfo(ctx, bucket, object); err != nil {
nsUnlocker()
return nil, toObjectErr(err, bucket, object)
}
// For a directory, we need to send an reader that returns no bytes.
if hasSuffix(object, slashSeparator) {
// The lock taken above is released when
// objReader.Close() is called by the caller.
return NewGetObjectReaderFromReader(bytes.NewBuffer(nil), objInfo, opts.CheckCopyPrecondFn, nsUnlocker)
}
// Take a rwPool lock for NFS gateway type deployment
rwPoolUnlocker := func() {}
if bucket != minioMetaBucket && lockType != noLock {
@@ -519,7 +516,7 @@ func (fs *FSObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
rwPoolUnlocker = func() { fs.rwPool.Close(fsMetaPath) }
}
objReaderFn, off, length, rErr := NewGetObjectReader(rs, objInfo, nsUnlocker, rwPoolUnlocker)
objReaderFn, off, length, rErr := NewGetObjectReader(rs, objInfo, opts.CheckCopyPrecondFn, nsUnlocker, rwPoolUnlocker)
if rErr != nil {
return nil, rErr
}
@@ -547,7 +544,7 @@ func (fs *FSObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
return nil, err
}
return objReaderFn(reader, h, closeFn)
return objReaderFn(reader, h, opts.CheckCopyPrecondFn, closeFn)
}
// GetObject - reads an object from the disk.
@@ -842,8 +839,7 @@ func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
logger.LogIf(ctx, errFileAccessDenied)
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
logger.LogIf(ctx, err)
@@ -862,8 +858,7 @@ func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
logger.LogIf(ctx, errFileAccessDenied)
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
// Validate input data size and it can never be less than zero.
@@ -1151,8 +1146,7 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
return fs.getObjectInfo(ctx, bucket, entry)
}
heal := false // true only for xl.ListObjectsHeal()
walkResultCh, endWalkCh := fs.listPool.Release(listParams{bucket, recursive, marker, prefix, heal})
walkResultCh, endWalkCh := fs.listPool.Release(listParams{bucket, recursive, marker, prefix})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, object string) bool {
@@ -1206,7 +1200,7 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
}
// Save list routine for the next marker if we haven't reached EOF.
params := listParams{bucket, recursive, nextMarker, prefix, heal}
params := listParams{bucket, recursive, nextMarker, prefix}
if !eof {
fs.listPool.Set(params, walkResultCh, endWalkCh)
}
@@ -1244,7 +1238,7 @@ func (fs *FSObjects) HealFormat(ctx context.Context, dryRun bool) (madmin.HealRe
}
// HealObject - no-op for fs. Valid only for XL.
func (fs *FSObjects) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (
func (fs *FSObjects) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (
res madmin.HealResultItem, err error) {
logger.LogIf(ctx, NotImplemented{})
return res, NotImplemented{}
@@ -1257,10 +1251,10 @@ func (fs *FSObjects) HealBucket(ctx context.Context, bucket string, dryRun, remo
return madmin.HealResultItem{}, NotImplemented{}
}
// ListObjectsHeal - list all objects to be healed. Valid only for XL
func (fs *FSObjects) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
// HealObjects - no-op for fs. Valid only for XL.
func (fs *FSObjects) HealObjects(ctx context.Context, bucket, prefix string, fn func(string, string) error) (e error) {
logger.LogIf(ctx, NotImplemented{})
return loi, NotImplemented{}
return NotImplemented{}
}
// ListBucketsHeal - list all buckets to be healed. Valid only for XL
+10 -8
View File
@@ -22,6 +22,8 @@ import (
"os"
"path/filepath"
"testing"
"github.com/minio/minio/pkg/madmin"
)
// Tests for if parent directory is object
@@ -228,8 +230,8 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, backend corruption occurred")
}
if nerr, ok := err.(PrefixAccessDenied); !ok {
t.Fatalf("Expected PrefixAccessDenied, got %#v", err)
if nerr, ok := err.(ParentIsObject); !ok {
t.Fatalf("Expected ParentIsObject, got %#v", err)
} else {
if nerr.Bucket != "bucket" {
t.Fatalf("Expected 'bucket', got %s", nerr.Bucket)
@@ -243,8 +245,8 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, backned corruption occurred")
}
if nerr, ok := err.(PrefixAccessDenied); !ok {
t.Fatalf("Expected PrefixAccessDenied, got %#v", err)
if nerr, ok := err.(ParentIsObject); !ok {
t.Fatalf("Expected ParentIsObject, got %#v", err)
} else {
if nerr.Bucket != "bucket" {
t.Fatalf("Expected 'bucket', got %s", nerr.Bucket)
@@ -390,19 +392,19 @@ func TestFSHealObject(t *testing.T) {
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
_, err := obj.HealObject(context.Background(), "bucket", "object", false, false)
_, err := obj.HealObject(context.Background(), "bucket", "object", false, false, madmin.HealDeepScan)
if err == nil || !isSameType(err, NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
}
// TestFSListObjectHeal - tests for fs ListObjectHeals
func TestFSListObjectsHeal(t *testing.T) {
// TestFSHealObjects - tests for fs HealObjects to return not implemented.
func TestFSHealObjects(t *testing.T) {
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
_, err := obj.ListObjectsHeal(context.Background(), "bucket", "prefix", "marker", "delimiter", 1000)
err := obj.HealObjects(context.Background(), "bucket", "prefix", nil)
if err == nil || !isSameType(err, NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
+1
View File
@@ -166,6 +166,7 @@ func FromMinioClientObjectInfo(bucket string, oi minio.ObjectInfo) ObjectInfo {
ContentType: oi.ContentType,
ContentEncoding: oi.Metadata.Get("Content-Encoding"),
StorageClass: oi.StorageClass,
Expires: oi.Expires,
}
}
+4 -4
View File
@@ -102,7 +102,7 @@ func (a GatewayUnsupported) ListBucketsHeal(ctx context.Context) (buckets []Buck
}
// HealObject - Not implemented stub
func (a GatewayUnsupported) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (h madmin.HealResultItem, e error) {
func (a GatewayUnsupported) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (h madmin.HealResultItem, e error) {
return h, NotImplemented{}
}
@@ -111,9 +111,9 @@ func (a GatewayUnsupported) ListObjectsV2(ctx context.Context, bucket, prefix, c
return result, NotImplemented{}
}
// ListObjectsHeal - Not implemented stub
func (a GatewayUnsupported) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
return loi, NotImplemented{}
// HealObjects - Not implemented stub
func (a GatewayUnsupported) HealObjects(ctx context.Context, bucket, prefix string, fn func(string, string) error) (e error) {
return NotImplemented{}
}
// CopyObject copies a blob from source container to destination container.
+28 -2
View File
@@ -564,12 +564,35 @@ func (a *azureObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
// skip all the entries till we reach the marker.
continue
}
// Populate correct ETag's if possible, this code primarily exists
// because AWS S3 indicates that
//
// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
//
// Objects created by the PUT Object, POST Object, or Copy operation,
// or through the AWS Management Console, and are encrypted by SSE-S3
// or plaintext, have ETags that are an MD5 digest of their object data.
//
// Some applications depend on this behavior refer https://github.com/minio/minio/issues/6550
// So we handle it here and make this consistent.
etag := minio.ToS3ETag(blob.Properties.Etag)
switch {
case blob.Properties.ContentMD5 != "":
b, err := base64.StdEncoding.DecodeString(blob.Properties.ContentMD5)
if err == nil {
etag = hex.EncodeToString(b)
}
case blob.Metadata["md5sum"] != "":
etag = blob.Metadata["md5sum"]
delete(blob.Metadata, "md5sum")
}
objects = append(objects, minio.ObjectInfo{
Bucket: bucket,
Name: blob.Name,
ModTime: time.Time(blob.Properties.LastModified),
Size: blob.Properties.ContentLength,
ETag: minio.ToS3ETag(blob.Properties.Etag),
ETag: etag,
ContentType: blob.Properties.ContentType,
ContentEncoding: blob.Properties.ContentEncoding,
})
@@ -651,7 +674,7 @@ func (a *azureObjects) GetObjectNInfo(ctx context.Context, bucket, object string
// Setup cleanup function to cause the above go-routine to
// exit in case of partial read
pipeCloser := func() { pr.Close() }
return minio.NewGetObjectReaderFromReader(pr, objInfo, pipeCloser), nil
return minio.NewGetObjectReaderFromReader(pr, objInfo, opts.CheckCopyPrecondFn, pipeCloser)
}
// GetObject - reads an object from azure. Supports additional
@@ -829,6 +852,9 @@ func (a *azureObjects) PutObject(ctx context.Context, bucket, object string, r *
// CopyObject - Copies a blob from source container to destination container.
// Uses Azure equivalent CopyBlob API.
func (a *azureObjects) CopyObject(ctx context.Context, srcBucket, srcObject, destBucket, destObject string, srcInfo minio.ObjectInfo, srcOpts, dstOpts minio.ObjectOptions) (objInfo minio.ObjectInfo, err error) {
if srcOpts.CheckCopyPrecondFn != nil && srcOpts.CheckCopyPrecondFn(srcInfo, "") {
return minio.ObjectInfo{}, minio.PreConditionFailed{}
}
srcBlobURL := a.client.GetContainerReference(srcBucket).GetBlobReference(srcObject).GetURL()
destBlob := a.client.GetContainerReference(destBucket).GetBlobReference(destObject)
azureMeta, props, err := s3MetaToAzureProperties(ctx, srcInfo.UserDefined)
+1 -1
View File
@@ -417,7 +417,7 @@ func (l *b2Objects) GetObjectNInfo(ctx context.Context, bucket, object string, r
// Setup cleanup function to cause the above go-routine to
// exit in case of partial read
pipeCloser := func() { pr.Close() }
return minio.NewGetObjectReaderFromReader(pr, objInfo, pipeCloser), nil
return minio.NewGetObjectReaderFromReader(pr, objInfo, opts.CheckCopyPrecondFn, pipeCloser)
}
// GetObject reads an object from B2. Supports additional
+15 -2
View File
@@ -753,7 +753,7 @@ func (l *gcsGateway) GetObjectNInfo(ctx context.Context, bucket, object string,
// Setup cleanup function to cause the above go-routine to
// exit in case of partial read
pipeCloser := func() { pr.Close() }
return minio.NewGetObjectReaderFromReader(pr, objInfo, pipeCloser), nil
return minio.NewGetObjectReaderFromReader(pr, objInfo, opts.CheckCopyPrecondFn, pipeCloser)
}
// GetObject - reads an object from GCS. Supports additional
@@ -797,12 +797,22 @@ func fromGCSAttrsToObjectInfo(attrs *storage.ObjectAttrs) minio.ObjectInfo {
// All google cloud storage objects have a CRC32c hash, whereas composite objects may not have a MD5 hash
// Refer https://cloud.google.com/storage/docs/hashes-etags. Use CRC32C for ETag
metadata := make(map[string]string)
var (
expiry time.Time
e error
)
for k, v := range attrs.Metadata {
k = http.CanonicalHeaderKey(k)
// Translate the GCS custom metadata prefix
if strings.HasPrefix(k, "X-Goog-Meta-") {
k = strings.Replace(k, "X-Goog-Meta-", "X-Amz-Meta-", 1)
}
if k == "Expires" {
if expiry, e = time.Parse(http.TimeFormat, v); e == nil {
expiry = expiry.UTC()
}
continue
}
metadata[k] = v
}
if attrs.ContentType != "" {
@@ -829,6 +839,7 @@ func fromGCSAttrsToObjectInfo(attrs *storage.ObjectAttrs) minio.ObjectInfo {
UserDefined: metadata,
ContentType: attrs.ContentType,
ContentEncoding: attrs.ContentEncoding,
Expires: expiry,
}
}
@@ -918,7 +929,9 @@ func (l *gcsGateway) PutObject(ctx context.Context, bucket string, key string, r
// CopyObject - Copies a blob from source container to destination container.
func (l *gcsGateway) CopyObject(ctx context.Context, srcBucket string, srcObject string, destBucket string, destObject string,
srcInfo minio.ObjectInfo, srcOpts, dstOpts minio.ObjectOptions) (minio.ObjectInfo, error) {
if srcOpts.CheckCopyPrecondFn != nil && srcOpts.CheckCopyPrecondFn(srcInfo, "") {
return minio.ObjectInfo{}, minio.PreConditionFailed{}
}
src := l.client.Bucket(srcBucket).Object(srcObject)
dst := l.client.Bucket(destBucket).Object(destObject)
+8 -4
View File
@@ -26,10 +26,11 @@ import (
"strings"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"github.com/dustin/go-humanize"
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
miniogopolicy "github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio-go/pkg/s3utils"
minio "github.com/minio/minio/cmd"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
@@ -181,7 +182,7 @@ func appendS3MetaToOSSOptions(ctx context.Context, opts []oss.Option, s3Metadata
case k == "X-Amz-Acl":
// Valid values: public-read, private, and public-read-write
opts = append(opts, oss.ObjectACL(oss.ACLType(v)))
case k == "X-Amz-Server-Side​-Encryption":
case k == "X-Amz-Server-Side-Encryption":
opts = append(opts, oss.ServerSideEncryption(v))
case k == "X-Amz-Copy-Source-If-Match":
opts = append(opts, oss.CopySourceIfMatch(v))
@@ -349,7 +350,7 @@ func ossIsValidBucketName(bucket string) bool {
if strings.Contains(bucket, ".") {
return false
}
if !minio.IsValidBucketName(bucket) {
if s3utils.CheckValidBucketNameStrict(bucket) != nil {
return false
}
return true
@@ -569,7 +570,7 @@ func (l *ossObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
// Setup cleanup function to cause the above go-routine to
// exit in case of partial read
pipeCloser := func() { pr.Close() }
return minio.NewGetObjectReaderFromReader(pr, objInfo, pipeCloser), nil
return minio.NewGetObjectReaderFromReader(pr, objInfo, opts.CheckCopyPrecondFn, pipeCloser)
}
// GetObject reads an object on OSS. Supports additional
@@ -664,6 +665,9 @@ func (l *ossObjects) PutObject(ctx context.Context, bucket, object string, r *mi
// CopyObject copies an object from source bucket to a destination bucket.
func (l *ossObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject string, srcInfo minio.ObjectInfo, srcOpts, dstOpts minio.ObjectOptions) (objInfo minio.ObjectInfo, err error) {
if srcOpts.CheckCopyPrecondFn != nil && srcOpts.CheckCopyPrecondFn(srcInfo, "") {
return minio.ObjectInfo{}, minio.PreConditionFailed{}
}
bkt, err := l.Client.Bucket(srcBucket)
if err != nil {
logger.LogIf(ctx, err)
+11
View File
@@ -21,6 +21,7 @@ import (
"context"
"encoding/json"
"errors"
"net/http"
"time"
minio "github.com/minio/minio/cmd"
@@ -83,6 +84,7 @@ func (m gwMetaV1) ToObjectInfo(bucket, object string) minio.ObjectInfo {
"Content-Length",
"Last-Modified",
"Content-Type",
"Expires",
}, defaultFilterKeys...)
objInfo := minio.ObjectInfo{
IsDir: false,
@@ -100,6 +102,15 @@ func (m gwMetaV1) ToObjectInfo(bucket, object string) minio.ObjectInfo {
if sc, ok := m.Meta["x-amz-storage-class"]; ok {
objInfo.StorageClass = sc
}
var (
t time.Time
e error
)
if exp, ok := m.Meta["expires"]; ok {
if t, e = time.Parse(http.TimeFormat, exp); e == nil {
objInfo.Expires = t.UTC()
}
}
// Success.
return objInfo
}
+2 -2
View File
@@ -313,7 +313,7 @@ func (l *s3EncObjects) GetObjectNInfo(ctx context.Context, bucket, object string
return l.s3Objects.GetObjectNInfo(ctx, bucket, object, rs, h, lockType, opts)
}
objInfo.UserDefined = minio.CleanMinioInternalMetadataKeys(objInfo.UserDefined)
fn, off, length, err := minio.NewGetObjectReader(rs, objInfo)
fn, off, length, err := minio.NewGetObjectReader(rs, objInfo, o.CheckCopyPrecondFn)
if err != nil {
return nil, minio.ErrorRespToObjectError(err)
}
@@ -329,7 +329,7 @@ func (l *s3EncObjects) GetObjectNInfo(ctx context.Context, bucket, object string
// Setup cleanup function to cause the above go-routine to
// exit in case of partial read
pipeCloser := func() { pr.Close() }
return fn(pr, h, pipeCloser)
return fn(pr, h, o.CheckCopyPrecondFn, pipeCloser)
}
// GetObjectInfo reads object info and replies back ObjectInfo
+7 -1
View File
@@ -400,7 +400,7 @@ func (l *s3Objects) GetObjectNInfo(ctx context.Context, bucket, object string, r
// Setup cleanup function to cause the above go-routine to
// exit in case of partial read
pipeCloser := func() { pr.Close() }
return minio.NewGetObjectReaderFromReader(pr, objInfo, pipeCloser), nil
return minio.NewGetObjectReaderFromReader(pr, objInfo, opts.CheckCopyPrecondFn, pipeCloser)
}
// GetObject reads an object from S3. Supports additional
@@ -463,6 +463,9 @@ func (l *s3Objects) PutObject(ctx context.Context, bucket string, object string,
// CopyObject copies an object from source bucket to a destination bucket.
func (l *s3Objects) CopyObject(ctx context.Context, srcBucket string, srcObject string, dstBucket string, dstObject string, srcInfo minio.ObjectInfo, srcOpts, dstOpts minio.ObjectOptions) (objInfo minio.ObjectInfo, err error) {
if srcOpts.CheckCopyPrecondFn != nil && srcOpts.CheckCopyPrecondFn(srcInfo, "") {
return minio.ObjectInfo{}, minio.PreConditionFailed{}
}
// Set this header such that following CopyObject() always sets the right metadata on the destination.
// metadata input is already a trickled down value from interpreting x-amz-metadata-directive at
// handler layer. So what we have right now is supposed to be applied on the destination object anyways.
@@ -533,6 +536,9 @@ func (l *s3Objects) PutObjectPart(ctx context.Context, bucket string, object str
// existing object or a part of it.
func (l *s3Objects) CopyObjectPart(ctx context.Context, srcBucket, srcObject, destBucket, destObject, uploadID string,
partID int, startOffset, length int64, srcInfo minio.ObjectInfo, srcOpts, dstOpts minio.ObjectOptions) (p minio.PartInfo, err error) {
if srcOpts.CheckCopyPrecondFn != nil && srcOpts.CheckCopyPrecondFn(srcInfo, "") {
return minio.PartInfo{}, minio.PreConditionFailed{}
}
srcInfo.UserDefined = map[string]string{
"x-amz-copy-source-if-match": srcInfo.ETag,
}
+19 -3
View File
@@ -383,7 +383,23 @@ type resourceHandler struct {
// setCorsHandler handler for CORS (Cross Origin Resource Sharing)
func setCorsHandler(h http.Handler) http.Handler {
c := cors.AllowAll()
c := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowedMethods: []string{
http.MethodGet,
http.MethodPut,
http.MethodHead,
http.MethodPost,
http.MethodDelete,
http.MethodOptions,
http.MethodPatch,
},
AllowedHeaders: []string{"*"},
ExposedHeaders: []string{"*"},
AllowCredentials: true,
})
return c.Handler(h)
}
@@ -623,7 +639,7 @@ type bucketForwardingHandler struct {
}
func (f bucketForwardingHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if globalDNSConfig == nil || globalDomainName == "" ||
if globalDNSConfig == nil || len(globalDomainNames) == 0 ||
guessIsHealthCheckReq(r) || guessIsMetricsReq(r) ||
guessIsRPCReq(r) || isAdminReq(r) {
f.handler.ServeHTTP(w, r)
@@ -632,7 +648,7 @@ func (f bucketForwardingHandler) ServeHTTP(w http.ResponseWriter, r *http.Reques
// For browser requests, when federation is setup we need to
// specifically handle download and upload for browser requests.
if guessIsBrowserReq(r) && globalDNSConfig != nil && globalDomainName != "" {
if guessIsBrowserReq(r) && globalDNSConfig != nil && len(globalDomainNames) > 0 {
var bucket, _ string
switch r.Method {
case http.MethodPut:
+4 -6
View File
@@ -33,7 +33,7 @@ import (
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/certs"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/iam/policy"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/iam/validator"
)
@@ -57,7 +57,6 @@ const (
globalMinioDefaultStorageClass = "STANDARD"
globalWindowsOSName = "windows"
globalNetBSDOSName = "netbsd"
globalSolarisOSName = "solaris"
globalMinioModeFS = "mode-server-fs"
globalMinioModeXL = "mode-server-xl"
globalMinioModeDistXL = "mode-server-distributed-xl"
@@ -79,7 +78,7 @@ const (
globalMaxSkewTime = 15 * time.Minute // 15 minutes skew allowed.
// GlobalMultipartExpiry - Expiry duration after which the multipart uploads are deemed stale.
GlobalMultipartExpiry = time.Hour * 24 * 14 // 2 weeks.
GlobalMultipartExpiry = time.Hour * 24 * 3 // 3 days.
// GlobalMultipartCleanupInterval - Cleanup interval when the stale multipart cleanup is initiated.
GlobalMultipartCleanupInterval = time.Hour * 24 // 24 hrs.
@@ -175,9 +174,8 @@ var (
globalActiveCred auth.Credentials
globalPublicCerts []*x509.Certificate
globalIsEnvDomainName bool
globalDomainName string // Root domain for virtual host style requests
globalDomainIPs set.StringSet // Root domain IP address(s) for a distributed Minio deployment
globalDomainNames []string // Root domains for virtual host style requests
globalDomainIPs set.StringSet // Root domain IP address(s) for a distributed Minio deployment
globalListingTimeout = newDynamicTimeout( /*30*/ 600*time.Second /*5*/, 600*time.Second) // timeout for listing related ops
globalObjectTimeout = newDynamicTimeout( /*1*/ 10*time.Minute /*10*/, 600*time.Second) // timeout for Object API related ops
+10 -7
View File
@@ -184,7 +184,7 @@ func getRedirectPostRawQuery(objInfo ObjectInfo) string {
// Returns access credentials in the request Authorization header.
func getReqAccessCred(r *http.Request, region string) (cred auth.Credentials) {
cred, _, _ = getReqAccessKeyV4(r, region)
cred, _, _ = getReqAccessKeyV4(r, region, serviceS3)
if cred.AccessKey == "" {
cred, _, _ = getReqAccessKeyV2(r)
}
@@ -341,8 +341,8 @@ func httpTraceHdrs(f http.HandlerFunc) http.HandlerFunc {
}
// Returns "/bucketName/objectName" for path-style or virtual-host-style requests.
func getResource(path string, host string, domain string) (string, error) {
if domain == "" {
func getResource(path string, host string, domains []string) (string, error) {
if len(domains) == 0 {
return path, nil
}
// If virtual-host-style is enabled construct the "resource" properly.
@@ -357,11 +357,14 @@ func getResource(path string, host string, domain string) (string, error) {
return "", err
}
}
if !strings.HasSuffix(host, "."+domain) {
return path, nil
for _, domain := range domains {
if !strings.HasSuffix(host, "."+domain) {
continue
}
bucket := strings.TrimSuffix(host, "."+domain)
return slashSeparator + pathJoin(bucket, path), nil
}
bucket := strings.TrimSuffix(host, "."+domain)
return slashSeparator + pathJoin(bucket, path), nil
return path, nil
}
// If none of the http routes match respond with MethodNotAllowed, in JSON
+5 -5
View File
@@ -213,15 +213,15 @@ func TestGetResource(t *testing.T) {
testCases := []struct {
p string
host string
domain string
domains []string
expectedResource string
}{
{"/a/b/c", "test.mydomain.com", "mydomain.com", "/test/a/b/c"},
{"/a/b/c", "test.mydomain.com", "notmydomain.com", "/a/b/c"},
{"/a/b/c", "test.mydomain.com", "", "/a/b/c"},
{"/a/b/c", "test.mydomain.com", []string{"mydomain.com"}, "/test/a/b/c"},
{"/a/b/c", "test.mydomain.com", []string{"notmydomain.com"}, "/a/b/c"},
{"/a/b/c", "test.mydomain.com", nil, "/a/b/c"},
}
for i, test := range testCases {
gotResource, err := getResource(test.p, test.host, test.domain)
gotResource, err := getResource(test.p, test.host, test.domains)
if err != nil {
t.Fatal(err)
}
+13 -2
View File
@@ -25,6 +25,8 @@ import (
"time"
humanize "github.com/dustin/go-humanize"
"golang.org/x/net/http2"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/pkg/certs"
)
@@ -46,17 +48,22 @@ const (
// DefaultMaxHeaderBytes - default maximum HTTP header size in bytes.
DefaultMaxHeaderBytes = 1 * humanize.MiByte
// DefaultHTTP2MaxConcurrentStreams - default value for HTTP 2.0 maximum concurrent streams allowed.
DefaultHTTP2MaxConcurrentStreams = 1024
)
// Server - extended http.Server supports multiple addresses to serve and enhanced connection handling.
type Server struct {
http.Server
Addrs []string // addresses on which the server listens for new connection.
ReadTimeout time.Duration // timeout used for net.Conn.Read() deadlines.
WriteTimeout time.Duration // timeout used for net.Conn.Write() deadlines.
ShutdownTimeout time.Duration // timeout used for graceful server shutdown.
TCPKeepAliveTimeout time.Duration // timeout used for underneath TCP connection.
UpdateBytesReadFunc func(int) // function to be called to update bytes read in bufConn.
UpdateBytesWrittenFunc func(int) // function to be called to update bytes written in bufConn.
listenerMutex *sync.Mutex // to guard 'listener' field.
listenerMutex sync.Mutex // to guard 'listener' field.
listener *httpListener // HTTP listener for all 'Addrs' field.
inShutdown uint32 // indicates whether the server is in shutdown or not
requestCount int32 // counter holds no. of request in progress.
@@ -120,6 +127,11 @@ func (srv *Server) Start() (err error) {
// Start servicing with listener.
if tlsConfig != nil {
if err = http2.ConfigureServer(&srv.Server, &http2.Server{
MaxConcurrentStreams: DefaultHTTP2MaxConcurrentStreams,
}); err != nil {
return err
}
return srv.Server.Serve(tls.NewListener(listener, tlsConfig))
}
return srv.Server.Serve(listener)
@@ -202,7 +214,6 @@ func NewServer(addrs []string, handler http.Handler, getCert certs.GetCertificat
Addrs: addrs,
ShutdownTimeout: DefaultShutdownTimeout,
TCPKeepAliveTimeout: DefaultTCPKeepAliveTimeout,
listenerMutex: &sync.Mutex{},
}
httpServer.Handler = handler
httpServer.TLSConfig = tlsConfig
-4
View File
@@ -77,10 +77,6 @@ func TestNewServer(t *testing.T) {
t.Fatalf("Case %v: server.TCPKeepAliveTimeout: expected: %v, got: %v", (i + 1), DefaultTCPKeepAliveTimeout, server.TCPKeepAliveTimeout)
}
if server.listenerMutex == nil {
t.Fatalf("Case %v: server.listenerMutex: expected: <non-nil>, got: <nil>", (i + 1))
}
if server.ReadTimeout != DefaultReadTimeout {
t.Fatalf("Case %v: server.ReadTimeout: expected: %v, got: %v", (i + 1), DefaultReadTimeout, server.ReadTimeout)
}
+22 -1
View File
@@ -28,7 +28,7 @@ import (
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/iam/policy"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/madmin"
)
@@ -326,6 +326,27 @@ func (sys *IAMSys) SetTempUser(accessKey string, cred auth.Credentials, policyNa
return nil
}
// GetUserPolicy - returns canned policy name associated with a user.
func (sys *IAMSys) GetUserPolicy(accessKey string) (policyName string, err error) {
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return "", errServerNotInitialized
}
sys.RLock()
defer sys.RUnlock()
if _, ok := sys.iamUsersMap[accessKey]; !ok {
return "", errNoSuchUser
}
if _, ok := sys.iamPolicyMap[accessKey]; !ok {
return "", errNoSuchUser
}
return sys.iamPolicyMap[accessKey], nil
}
// ListUsers - list all users.
func (sys *IAMSys) ListUsers() (map[string]madmin.UserInfo, error) {
objectAPI := newObjectLayerFn()
+1 -1
View File
@@ -343,7 +343,7 @@ func logIf(ctx context.Context, err error) {
entry.API.Args.Bucket = hashString(entry.API.Args.Bucket)
entry.API.Args.Object = hashString(entry.API.Args.Object)
entry.RemoteHost = hashString(entry.RemoteHost)
entry.Message = reflect.TypeOf(err).String()
entry.Trace.Message = reflect.TypeOf(err).String()
entry.Trace.Variables = make(map[string]string)
}
+139 -117
View File
@@ -43,7 +43,7 @@ type NotificationSys struct {
targetList *event.TargetList
bucketRulesMap map[string]event.RulesMap
bucketRemoteTargetRulesMap map[string]map[event.TargetID]event.RulesMap
peerRPCClientMap map[xnet.Host]*PeerRPCClient
peerClients []*peerRESTClient
}
// GetARNList - returns available ARNs.
@@ -63,11 +63,6 @@ func (sys *NotificationSys) GetARNList() []string {
return arns
}
// GetPeerRPCClient - returns PeerRPCClient of addr.
func (sys *NotificationSys) GetPeerRPCClient(addr xnet.Host) *PeerRPCClient {
return sys.peerRPCClientMap[addr]
}
// NotificationPeerErr returns error associated for a remote peer.
type NotificationPeerErr struct {
Host xnet.Host // Remote host on which the rpc call was initiated
@@ -78,15 +73,15 @@ type NotificationPeerErr struct {
func (sys *NotificationSys) DeleteBucket(ctx context.Context, bucketName string) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.DeleteBucket(bucketName); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -146,52 +141,45 @@ func (g *NotificationGroup) Go(ctx context.Context, f func() error, index int, a
}()
}
// ReloadFormat - calls ReloadFormat RPC call on all peers.
// ReloadFormat - calls ReloadFormat REST call on all peers.
func (sys *NotificationSys) ReloadFormat(dryRun bool) []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), func() error {
return client.ReloadFormat(dryRun)
}, idx, addr)
idx++
}, idx, *client.host)
}
return ng.Wait()
}
// LoadUsers - calls LoadUsers RPC call on all peers.
func (sys *NotificationSys) LoadUsers() []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng.Go(context.Background(), client.LoadUsers, idx, addr)
idx++
}
return ng.Wait()
}
// LoadCredentials - calls LoadCredentials RPC call on all peers.
func (sys *NotificationSys) LoadCredentials() []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng.Go(context.Background(), client.LoadCredentials, idx, addr)
idx++
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), client.LoadUsers, idx, *client.host)
}
return ng.Wait()
}
// StartProfiling - start profiling on remote peers, by initiating a remote RPC.
func (sys *NotificationSys) StartProfiling(profiler string) []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), func() error {
return client.StartProfiling(profiler)
}, idx, addr)
idx++
}, idx, *client.host)
}
return ng.Wait()
}
@@ -205,10 +193,13 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
zipWriter := zip.NewWriter(writer)
defer zipWriter.Close()
for addr, client := range sys.peerRPCClientMap {
data, err := client.DownloadProfilingData()
for _, client := range sys.peerClients {
if client == nil {
continue
}
data, err := client.DownloadProfileData()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, err)
continue
@@ -218,7 +209,7 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
// Send profiling data to zip as file
header, zerr := zip.FileInfoHeader(dummyFileInfo{
name: fmt.Sprintf("profiling-%s.pprof", addr),
name: fmt.Sprintf("profiling-%s.pprof", client.host.String()),
size: int64(len(data)),
mode: 0600,
modTime: UTCNow(),
@@ -226,20 +217,20 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
sys: nil,
})
if zerr != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, zerr)
continue
}
zwriter, zerr := zipWriter.CreateHeader(header)
if zerr != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, zerr)
continue
}
if _, err = io.Copy(zwriter, bytes.NewBuffer(data)); err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, err)
continue
@@ -289,45 +280,48 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
// SignalService - calls signal service RPC call on all peers.
func (sys *NotificationSys) SignalService(sig serviceSignal) []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), func() error {
return client.SignalService(sig)
}, idx, addr)
idx++
}, idx, *client.host)
}
return ng.Wait()
}
// ServerInfo - calls ServerInfo RPC call on all peers.
func (sys *NotificationSys) ServerInfo(ctx context.Context) []ServerInfo {
var idx = 0
serverInfo := make([]ServerInfo, len(sys.peerRPCClientMap))
serverInfo := make([]ServerInfo, len(sys.peerClients))
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for index, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(idx int, addr xnet.Host, client *PeerRPCClient) {
go func(idx int, client *peerRESTClient) {
defer wg.Done()
// Try to fetch serverInfo remotely in three attempts.
for i := 0; i < 3; i++ {
info, err := client.ServerInfo()
if err == nil {
serverInfo[idx] = ServerInfo{
Addr: addr.String(),
Addr: client.host.String(),
Data: &info,
}
return
}
serverInfo[idx] = ServerInfo{
Addr: addr.String(),
Addr: client.host.String(),
Data: &info,
Error: err.Error(),
}
// Last iteration log the error.
if i == 2 {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, err)
}
@@ -336,8 +330,7 @@ func (sys *NotificationSys) ServerInfo(ctx context.Context) []ServerInfo {
time.Sleep(1 * time.Second)
}
}
}(idx, addr, client)
idx++
}(index, client)
}
wg.Wait()
return serverInfo
@@ -345,19 +338,22 @@ func (sys *NotificationSys) ServerInfo(ctx context.Context) []ServerInfo {
// GetLocks - makes GetLocks RPC call on all peers.
func (sys *NotificationSys) GetLocks(ctx context.Context) []*PeerLocks {
var idx = 0
locksResp := make([]*PeerLocks, len(sys.peerRPCClientMap))
locksResp := make([]*PeerLocks, len(sys.peerClients))
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for index, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(idx int, addr xnet.Host, client *PeerRPCClient) {
go func(idx int, client *peerRESTClient) {
defer wg.Done()
// Try to fetch serverInfo remotely in three attempts.
for i := 0; i < 3; i++ {
serverLocksResp, err := client.GetLocks()
if err == nil {
locksResp[idx] = &PeerLocks{
Addr: addr.String(),
Addr: client.host.String(),
Locks: serverLocksResp,
}
return
@@ -365,17 +361,16 @@ func (sys *NotificationSys) GetLocks(ctx context.Context) []*PeerLocks {
// Last iteration log the error.
if i == 2 {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogOnceIf(ctx, err, addr.String())
logger.LogOnceIf(ctx, err, client.host.String())
}
// Wait for one second and no need wait after last attempt.
if i < 2 {
time.Sleep(1 * time.Second)
}
}
}(idx, addr, client)
idx++
}(index, client)
}
wg.Wait()
return locksResp
@@ -385,15 +380,18 @@ func (sys *NotificationSys) GetLocks(ctx context.Context) []*PeerLocks {
func (sys *NotificationSys) SetBucketPolicy(ctx context.Context, bucketName string, bucketPolicy *policy.Policy) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.SetBucketPolicy(bucketName, bucketPolicy); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -403,15 +401,18 @@ func (sys *NotificationSys) SetBucketPolicy(ctx context.Context, bucketName stri
func (sys *NotificationSys) RemoveBucketPolicy(ctx context.Context, bucketName string) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.RemoveBucketPolicy(bucketName); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -421,15 +422,18 @@ func (sys *NotificationSys) RemoveBucketPolicy(ctx context.Context, bucketName s
func (sys *NotificationSys) PutBucketNotification(ctx context.Context, bucketName string, rulesMap event.RulesMap) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, rulesMap event.RulesMap) {
go func(client *peerRESTClient, rulesMap event.RulesMap) {
defer wg.Done()
if err := client.PutBucketNotification(bucketName, rulesMap); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client, rulesMap.Clone())
}(client, rulesMap.Clone())
}
wg.Wait()
}()
@@ -440,15 +444,18 @@ func (sys *NotificationSys) ListenBucketNotification(ctx context.Context, bucket
targetID event.TargetID, localPeer xnet.Host) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.ListenBucketNotification(bucketName, eventNames, pattern, targetID, localPeer); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -492,7 +499,17 @@ func (sys *NotificationSys) RemoteTargetExist(bucketName string, targetID event.
return ok
}
// initListeners - initializes PeerRPC clients available in listener.json.
// ListenBucketNotificationArgs - listen bucket notification RPC arguments.
type ListenBucketNotificationArgs struct {
AuthArgs `json:"-"`
BucketName string `json:"-"`
EventNames []event.Name `json:"eventNames"`
Pattern string `json:"pattern"`
TargetID event.TargetID `json:"targetId"`
Addr xnet.Host `json:"addr"`
}
// initListeners - initializes PeerREST clients available in listener.json.
func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLayer, bucketName string) error {
// listener.json is available/applicable only in DistXL mode.
if !globalIsDistXL {
@@ -542,12 +559,12 @@ func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLaye
continue
}
rpcClient := sys.GetPeerRPCClient(args.Addr)
if rpcClient == nil {
return fmt.Errorf("unable to find PeerRPCClient by address %v in listener.json for bucket %v", args.Addr, bucketName)
client, err := newPeerRESTClient(&args.Addr)
if err != nil {
return fmt.Errorf("unable to find PeerHost by address %v in listener.json for bucket %v", args.Addr, bucketName)
}
exist, err := rpcClient.RemoteTargetExist(bucketName, args.TargetID)
exist, err := client.RemoteTargetExist(bucketName, args.TargetID)
if err != nil {
logger.GetReqInfo(ctx).AppendTags("targetID", args.TargetID.Name)
logger.LogIf(ctx, err)
@@ -558,7 +575,7 @@ func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLaye
continue
}
target := NewPeerRPCClientTarget(bucketName, args.TargetID, rpcClient)
target := NewPeerRESTClientTarget(bucketName, args.TargetID, client)
rulesMap := event.NewRulesMap(args.EventNames, args.Pattern, target.ID())
if err = sys.AddRemoteTarget(bucketName, target, rulesMap); err != nil {
logger.GetReqInfo(ctx).AppendTags("targetName", target.id.Name)
@@ -721,24 +738,25 @@ func (sys *NotificationSys) Send(args eventArgs) []event.TargetIDErr {
// DrivePerfInfo - Drive speed (read and write) information
func (sys *NotificationSys) DrivePerfInfo() []ServerDrivesPerfInfo {
reply := make([]ServerDrivesPerfInfo, len(sys.peerRPCClientMap))
reply := make([]ServerDrivesPerfInfo, len(sys.peerClients))
var wg sync.WaitGroup
var i int
for addr, client := range sys.peerRPCClientMap {
for i, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, idx int) {
go func(client *peerRESTClient, idx int) {
defer wg.Done()
di, err := client.DrivePerfInfo()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", client.host.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
di.Addr = addr.String()
di.Addr = client.host.String()
di.Error = err.Error()
}
reply[idx] = di
}(addr, client, i)
i++
}(client, i)
}
wg.Wait()
return reply
@@ -746,24 +764,25 @@ func (sys *NotificationSys) DrivePerfInfo() []ServerDrivesPerfInfo {
// MemUsageInfo - Mem utilization information
func (sys *NotificationSys) MemUsageInfo() []ServerMemUsageInfo {
reply := make([]ServerMemUsageInfo, len(sys.peerRPCClientMap))
reply := make([]ServerMemUsageInfo, len(sys.peerClients))
var wg sync.WaitGroup
var i int
for addr, client := range sys.peerRPCClientMap {
for i, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, idx int) {
go func(client *peerRESTClient, idx int) {
defer wg.Done()
memi, err := client.MemUsageInfo()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", client.host.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
memi.Addr = addr.String()
memi.Addr = client.host.String()
memi.Error = err.Error()
}
reply[idx] = memi
}(addr, client, i)
i++
}(client, i)
}
wg.Wait()
return reply
@@ -771,24 +790,25 @@ func (sys *NotificationSys) MemUsageInfo() []ServerMemUsageInfo {
// CPULoadInfo - CPU utilization information
func (sys *NotificationSys) CPULoadInfo() []ServerCPULoadInfo {
reply := make([]ServerCPULoadInfo, len(sys.peerRPCClientMap))
reply := make([]ServerCPULoadInfo, len(sys.peerClients))
var wg sync.WaitGroup
var i int
for addr, client := range sys.peerRPCClientMap {
for i, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, idx int) {
go func(client *peerRESTClient, idx int) {
defer wg.Done()
cpui, err := client.CPULoadInfo()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", client.host.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
cpui.Addr = addr.String()
cpui.Addr = client.host.String()
cpui.Error = err.Error()
}
reply[idx] = cpui
}(addr, client, i)
i++
}(client, i)
}
wg.Wait()
return reply
@@ -797,14 +817,18 @@ func (sys *NotificationSys) CPULoadInfo() []ServerCPULoadInfo {
// NewNotificationSys - creates new notification system object.
func NewNotificationSys(config *serverConfig, endpoints EndpointList) *NotificationSys {
targetList := getNotificationTargets(config)
peerRPCClientMap := makeRemoteRPCClients(endpoints)
remoteHosts := getRemoteHosts(endpoints)
remoteClients, err := getRestClients(remoteHosts)
if err != nil {
logger.FatalIf(err, "Unable to start notification sub system")
}
// bucketRulesMap/bucketRemoteTargetRulesMap are initialized by NotificationSys.Init()
return &NotificationSys{
targetList: targetList,
bucketRulesMap: make(map[string]event.RulesMap),
bucketRemoteTargetRulesMap: make(map[string]map[event.TargetID]event.RulesMap),
peerRPCClientMap: peerRPCClientMap,
peerClients: remoteClients,
}
}
@@ -815,7 +839,6 @@ type eventArgs struct {
ReqParams map[string]string
RespElements map[string]string
Host string
Port string
UserAgent string
}
@@ -870,7 +893,6 @@ func (args eventArgs) ToEvent() event.Event {
},
Source: event.Source{
Host: args.Host,
Port: args.Port,
UserAgent: args.UserAgent,
},
}
+3
View File
@@ -96,6 +96,9 @@ type ObjectInfo struct {
// by the Content-Type header field.
ContentEncoding string
// Date and time at which the object is no longer able to be cached
Expires time.Time
// Specify object storage class
StorageClass string
+27
View File
@@ -19,6 +19,7 @@ package cmd
import (
"fmt"
"io"
"path"
)
// Converts underlying storage error. Convenience function written to
@@ -47,6 +48,13 @@ func toObjectErr(err error, params ...string) error {
Object: params[1],
}
}
case errFileParentIsFile:
if len(params) >= 2 {
err = ParentIsObject{
Bucket: params[0],
Object: params[1],
}
}
case errIsNotRegular:
if len(params) >= 2 {
err = ObjectExistsAsDirectory{
@@ -182,6 +190,13 @@ func (e PrefixAccessDenied) Error() string {
return "Prefix access is denied: " + e.Bucket + "/" + e.Object
}
// ParentIsObject object access is denied.
type ParentIsObject GenericError
func (e ParentIsObject) Error() string {
return "Parent is object " + e.Bucket + "/" + path.Dir(e.Object)
}
// BucketExists bucket exists.
type BucketExists GenericError
@@ -383,3 +398,15 @@ func isErrObjectNotFound(err error) bool {
_, ok := err.(ObjectNotFound)
return ok
}
// PreConditionFailed - Check if copy precondition failed
type PreConditionFailed struct{}
func (e PreConditionFailed) Error() string {
return "At least one of the pre-conditions you specified did not hold"
}
func isErrPreconditionFailed(err error) bool {
_, ok := err.(PreConditionFailed)
return ok
}
+6 -2
View File
@@ -26,10 +26,14 @@ import (
"github.com/minio/minio/pkg/policy"
)
// CheckCopyPreconditionFn returns true if copy precondition check failed.
type CheckCopyPreconditionFn func(o ObjectInfo, encETag string) bool
// ObjectOptions represents object options for ObjectLayer operations
type ObjectOptions struct {
ServerSideEncryption encrypt.ServerSide
UserDefined map[string]string
CheckCopyPrecondFn CheckCopyPreconditionFn
}
// LockType represents required locking for ObjectLayer operations
@@ -84,9 +88,9 @@ type ObjectLayer interface {
ReloadFormat(ctx context.Context, dryRun bool) error
HealFormat(ctx context.Context, dryRun bool) (madmin.HealResultItem, error)
HealBucket(ctx context.Context, bucket string, dryRun, remove bool) (madmin.HealResultItem, error)
HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (madmin.HealResultItem, error)
HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (madmin.HealResultItem, error)
ListBucketsHeal(ctx context.Context) (buckets []BucketInfo, err error)
ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, error)
HealObjects(ctx context.Context, bucket, prefix string, healObjectFn func(string, string) error) error
// Policy operations
SetBucketPolicy(context.Context, string, *policy.Policy) error
+117 -16
View File
@@ -34,6 +34,7 @@ import (
"unicode/utf8"
snappy "github.com/golang/snappy"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
@@ -207,8 +208,8 @@ func getCompleteMultipartMD5(ctx context.Context, parts []CompletePart) (string,
func cleanMetadata(metadata map[string]string) map[string]string {
// Remove STANDARD StorageClass
metadata = removeStandardStorageClass(metadata)
// Clean meta etag keys 'md5Sum', 'etag'.
return cleanMetadataKeys(metadata, "md5Sum", "etag")
// Clean meta etag keys 'md5Sum', 'etag', "expires".
return cleanMetadataKeys(metadata, "md5Sum", "etag", "expires")
}
// Filter X-Amz-Storage-Class field only if it is set to STANDARD.
@@ -273,10 +274,16 @@ func isStringEqual(s1 string, s2 string) bool {
}
// Ignores all reserved bucket names or invalid bucket names.
func isReservedOrInvalidBucket(bucketEntry string) bool {
func isReservedOrInvalidBucket(bucketEntry string, strict bool) bool {
bucketEntry = strings.TrimSuffix(bucketEntry, slashSeparator)
if !IsValidBucketName(bucketEntry) {
return true
if strict {
if err := s3utils.CheckValidBucketNameStrict(bucketEntry); err != nil {
return true
}
} else {
if err := s3utils.CheckValidBucketName(bucketEntry); err != nil {
return true
}
}
return isMinioMetaBucket(bucketEntry) || isMinioReservedBucket(bucketEntry)
}
@@ -422,30 +429,41 @@ type GetObjectReader struct {
pReader io.Reader
cleanUpFns []func()
precondFn func(ObjectInfo, string) bool
once sync.Once
}
// NewGetObjectReaderFromReader sets up a GetObjectReader with a given
// reader. This ignores any object properties.
func NewGetObjectReaderFromReader(r io.Reader, oi ObjectInfo, cleanupFns ...func()) *GetObjectReader {
func NewGetObjectReaderFromReader(r io.Reader, oi ObjectInfo, pcfn CheckCopyPreconditionFn, cleanupFns ...func()) (*GetObjectReader, error) {
if pcfn != nil {
if ok := pcfn(oi, ""); ok {
// Call the cleanup funcs
for i := len(cleanupFns) - 1; i >= 0; i-- {
cleanupFns[i]()
}
return nil, PreConditionFailed{}
}
}
return &GetObjectReader{
ObjInfo: oi,
pReader: r,
cleanUpFns: cleanupFns,
}
precondFn: pcfn,
}, nil
}
// ObjReaderFn is a function type that takes a reader and returns
// GetObjectReader and an error. Request headers are passed to provide
// encryption parameters. cleanupFns allow cleanup funcs to be
// registered for calling after usage of the reader.
type ObjReaderFn func(inputReader io.Reader, h http.Header, cleanupFns ...func()) (r *GetObjectReader, err error)
type ObjReaderFn func(inputReader io.Reader, h http.Header, pcfn CheckCopyPreconditionFn, cleanupFns ...func()) (r *GetObjectReader, err error)
// NewGetObjectReader creates a new GetObjectReader. The cleanUpFns
// are called on Close() in reverse order as passed here. NOTE: It is
// assumed that clean up functions do not panic (otherwise, they may
// not all run!).
func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func()) (
func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, pcfn CheckCopyPreconditionFn, cleanUpFns ...func()) (
fn ObjReaderFn, off, length int64, err error) {
// Call the clean-up functions immediately in case of exit
@@ -486,7 +504,7 @@ func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func())
// a reader that returns the desired range of
// encrypted bytes. The header parameter is used to
// provide encryption parameters.
fn = func(inputReader io.Reader, h http.Header, cFns ...func()) (r *GetObjectReader, err error) {
fn = func(inputReader io.Reader, h http.Header, pcfn CheckCopyPreconditionFn, cFns ...func()) (r *GetObjectReader, err error) {
copySource := h.Get(crypto.SSECopyAlgorithm) != ""
cFns = append(cleanUpFns, cFns...)
@@ -501,9 +519,18 @@ func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func())
}
return nil, err
}
encETag := oi.ETag
oi.ETag = getDecryptedETag(h, oi, copySource) // Decrypt the ETag before top layer consumes this value.
// Decrypt the ETag before top layer consumes this value.
oi.ETag = getDecryptedETag(h, oi, copySource)
if pcfn != nil {
if ok := pcfn(oi, encETag); ok {
// Call the cleanup funcs
for i := len(cFns) - 1; i >= 0; i-- {
cFns[i]()
}
return nil, PreConditionFailed{}
}
}
// Apply the skipLen and limit on the
// decrypted stream
@@ -514,6 +541,7 @@ func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func())
ObjInfo: oi,
pReader: decReader,
cleanUpFns: cFns,
precondFn: pcfn,
}
return r, nil
}
@@ -545,7 +573,17 @@ func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func())
return nil, 0, 0, errInvalidRange
}
}
fn = func(inputReader io.Reader, _ http.Header, cFns ...func()) (r *GetObjectReader, err error) {
fn = func(inputReader io.Reader, _ http.Header, pcfn CheckCopyPreconditionFn, cFns ...func()) (r *GetObjectReader, err error) {
cFns = append(cleanUpFns, cFns...)
if pcfn != nil {
if ok := pcfn(oi, ""); ok {
// Call the cleanup funcs
for i := len(cFns) - 1; i >= 0; i-- {
cFns[i]()
}
return nil, PreConditionFailed{}
}
}
// Decompression reader.
snappyReader := snappy.NewReader(inputReader)
// Apply the skipLen and limit on the
@@ -557,7 +595,8 @@ func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func())
r = &GetObjectReader{
ObjInfo: oi,
pReader: decReader,
cleanUpFns: append(cleanUpFns, cFns...),
cleanUpFns: cFns,
precondFn: pcfn,
}
return r, nil
}
@@ -567,11 +606,22 @@ func NewGetObjectReader(rs *HTTPRangeSpec, oi ObjectInfo, cleanUpFns ...func())
if err != nil {
return nil, 0, 0, err
}
fn = func(inputReader io.Reader, _ http.Header, cFns ...func()) (r *GetObjectReader, err error) {
fn = func(inputReader io.Reader, _ http.Header, pcfn CheckCopyPreconditionFn, cFns ...func()) (r *GetObjectReader, err error) {
cFns = append(cleanUpFns, cFns...)
if pcfn != nil {
if ok := pcfn(oi, ""); ok {
// Call the cleanup funcs
for i := len(cFns) - 1; i >= 0; i-- {
cFns[i]()
}
return nil, PreConditionFailed{}
}
}
r = &GetObjectReader{
ObjInfo: oi,
pReader: inputReader,
cleanUpFns: append(cleanUpFns, cFns...),
cleanUpFns: cFns,
precondFn: pcfn,
}
return r, nil
}
@@ -675,3 +725,54 @@ func CleanMinioInternalMetadataKeys(metadata map[string]string) map[string]strin
}
return newMeta
}
// snappyCompressReader compresses data as it reads
// from the underlying io.Reader.
type snappyCompressReader struct {
r io.Reader
w *snappy.Writer
closed bool
buf bytes.Buffer
}
func newSnappyCompressReader(r io.Reader) *snappyCompressReader {
cr := &snappyCompressReader{r: r}
cr.w = snappy.NewBufferedWriter(&cr.buf)
return cr
}
func (cr *snappyCompressReader) Read(p []byte) (int, error) {
if cr.closed {
// if snappy writer is closed r has been completely read,
// return any remaining data in buf.
return cr.buf.Read(p)
}
// read from original using p as buffer
nr, readErr := cr.r.Read(p)
// write read bytes to snappy writer
nw, err := cr.w.Write(p[:nr])
if err != nil {
return 0, err
}
if nw != nr {
return 0, io.ErrShortWrite
}
// if last of data from reader, close snappy writer to flush
if readErr == io.EOF {
err := cr.w.Close()
cr.closed = true
if err != nil {
return 0, err
}
}
// read compressed bytes out of buf
n, err := cr.buf.Read(p)
if readErr != io.EOF && (err == nil || err == io.EOF) {
err = readErr
}
return n, err
}
+59
View File
@@ -17,10 +17,14 @@
package cmd
import (
"bytes"
"context"
"io"
"net/http"
"reflect"
"testing"
"github.com/golang/snappy"
)
// Tests validate bucket name.
@@ -544,3 +548,58 @@ func TestGetCompressedOffsets(t *testing.T) {
}
}
}
func TestSnappyCompressReader(t *testing.T) {
tests := []struct {
name string
data []byte
}{
{name: "empty", data: nil},
{name: "small", data: []byte("hello, world")},
{name: "large", data: bytes.Repeat([]byte("hello, world"), 1000)},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
buf := make([]byte, 100) // make small buffer to ensure multiple reads are required for large case
r := newSnappyCompressReader(bytes.NewReader(tt.data))
var rdrBuf bytes.Buffer
_, err := io.CopyBuffer(&rdrBuf, r, buf)
if err != nil {
t.Fatal(err)
}
var stdBuf bytes.Buffer
w := snappy.NewBufferedWriter(&stdBuf)
_, err = io.CopyBuffer(w, bytes.NewReader(tt.data), buf)
if err != nil {
t.Fatal(err)
}
err = w.Close()
if err != nil {
t.Fatal(err)
}
var (
got = rdrBuf.Bytes()
want = stdBuf.Bytes()
)
if !bytes.Equal(got, want) {
t.Errorf("encoded data does not match\n\t%q\n\t%q", got, want)
}
var decBuf bytes.Buffer
decRdr := snappy.NewReader(&rdrBuf)
_, err = io.Copy(&decBuf, decRdr)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(tt.data, decBuf.Bytes()) {
t.Errorf("roundtrip failed\n\t%q\n\t%q", tt.data, decBuf.Bytes())
}
})
}
}
+20 -11
View File
@@ -18,11 +18,11 @@ package cmd
import (
"context"
"net"
"net/http"
"strings"
"time"
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/handlers"
)
@@ -33,8 +33,8 @@ import (
// x-amz-copy-source-if-unmodified-since
// x-amz-copy-source-if-match
// x-amz-copy-source-if-none-match
func checkCopyObjectPartPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Request, objInfo ObjectInfo) bool {
return checkCopyObjectPreconditions(ctx, w, r, objInfo)
func checkCopyObjectPartPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Request, objInfo ObjectInfo, encETag string) bool {
return checkCopyObjectPreconditions(ctx, w, r, objInfo, encETag)
}
// Validates the preconditions for CopyObject, returns true if CopyObject operation should not proceed.
@@ -43,11 +43,14 @@ func checkCopyObjectPartPreconditions(ctx context.Context, w http.ResponseWriter
// x-amz-copy-source-if-unmodified-since
// x-amz-copy-source-if-match
// x-amz-copy-source-if-none-match
func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Request, objInfo ObjectInfo) bool {
func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Request, objInfo ObjectInfo, encETag string) bool {
// Return false for methods other than GET and HEAD.
if r.Method != "PUT" {
return false
}
if encETag == "" {
encETag = objInfo.ETag
}
// If the object doesn't have a modtime (IsZero), or the modtime
// is obviously garbage (Unix time == 0), then ignore modtimes
// and don't process the If-Modified-Since header.
@@ -95,11 +98,17 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
}
}
shouldDecryptEtag := crypto.SSECopy.IsRequested(r.Header) && !crypto.IsMultiPart(objInfo.UserDefined)
// x-amz-copy-source-if-match : Return the object only if its entity tag (ETag) is the
// same as the one specified; otherwise return a 412 (precondition failed).
ifMatchETagHeader := r.Header.Get("x-amz-copy-source-if-match")
if ifMatchETagHeader != "" {
if objInfo.ETag != "" && !isETagEqual(objInfo.ETag, ifMatchETagHeader) {
etag := objInfo.ETag
if shouldDecryptEtag {
etag = encETag[len(encETag)-32:]
}
if objInfo.ETag != "" && !isETagEqual(etag, ifMatchETagHeader) {
// If the object ETag does not match with the specified ETag.
writeHeaders()
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrPreconditionFailed), r.URL, guessIsBrowserReq(r))
@@ -111,7 +120,11 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
// one specified otherwise, return a 304 (not modified).
ifNoneMatchETagHeader := r.Header.Get("x-amz-copy-source-if-none-match")
if ifNoneMatchETagHeader != "" {
if objInfo.ETag != "" && isETagEqual(objInfo.ETag, ifNoneMatchETagHeader) {
etag := objInfo.ETag
if shouldDecryptEtag {
etag = encETag[len(encETag)-32:]
}
if objInfo.ETag != "" && isETagEqual(etag, ifNoneMatchETagHeader) {
// If the object ETag matches with the specified ETag.
writeHeaders()
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrPreconditionFailed), r.URL, guessIsBrowserReq(r))
@@ -240,9 +253,6 @@ func deleteObject(ctx context.Context, obj ObjectLayer, cache CacheObjectLayer,
return err
}
// Get host and port from Request.RemoteAddr.
host, port, _ := net.SplitHostPort(handlers.GetSourceIP(r))
// Notify object deleted event.
sendEvent(eventArgs{
EventName: event.ObjectRemovedDelete,
@@ -252,8 +262,7 @@ func deleteObject(ctx context.Context, obj ObjectLayer, cache CacheObjectLayer,
},
ReqParams: extractReqParams(r),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
return nil
+110 -148
View File
@@ -24,7 +24,6 @@ import (
"encoding/xml"
"io"
goioutil "io/ioutil"
"net"
"net/http"
"net/url"
"sort"
@@ -33,7 +32,6 @@ import (
"time"
snappy "github.com/golang/snappy"
"github.com/gorilla/mux"
miniogo "github.com/minio/minio-go"
"github.com/minio/minio-go/pkg/encrypt"
@@ -222,12 +220,6 @@ func (api objectAPIHandlers) SelectObjectContentHandler(w http.ResponseWriter, r
s3Select.Evaluate(w)
s3Select.Close()
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a GET request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedGet,
@@ -236,8 +228,7 @@ func (api objectAPIHandlers) SelectObjectContentHandler(w http.ResponseWriter, r
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -403,12 +394,6 @@ func (api objectAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Req
}
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a GET request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedGet,
@@ -417,8 +402,7 @@ func (api objectAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Req
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -563,11 +547,6 @@ func (api objectAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.Re
w.WriteHeader(http.StatusOK)
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a HEAD request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedHead,
@@ -576,8 +555,7 @@ func (api objectAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.Re
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -761,21 +739,23 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
if !cpSrcDstSame {
lock = readLock
}
checkCopyPrecondFn := func(o ObjectInfo, encETag string) bool {
return checkCopyObjectPreconditions(ctx, w, r, o, encETag)
}
getOpts.CheckCopyPrecondFn = checkCopyPrecondFn
srcOpts.CheckCopyPrecondFn = checkCopyPrecondFn
var rs *HTTPRangeSpec
gr, err := getObjectNInfo(ctx, srcBucket, srcObject, rs, r.Header, lock, getOpts)
if err != nil {
if isErrPreconditionFailed(err) {
return
}
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
defer gr.Close()
srcInfo := gr.ObjInfo
// Verify before x-amz-copy-source preconditions before continuing with CopyObject.
if checkCopyObjectPreconditions(ctx, w, r, srcInfo) {
return
}
/// maximum Upload size for object in a single CopyObject operation.
if isMaxObjectSize(srcInfo.Size) {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrEntityTooLarge), r.URL, guessIsBrowserReq(r))
@@ -831,21 +811,9 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
// Remove all source encrypted related metadata to
// avoid copying them in target object.
crypto.RemoveInternalEntries(srcInfo.UserDefined)
// Open a pipe for compression.
// Where pipeWriter is piped to srcInfo.Reader.
// gr writes to pipeWriter.
pipeReader, pipeWriter := io.Pipe()
reader = pipeReader
reader = newSnappyCompressReader(gr)
length = -1
snappyWriter := snappy.NewBufferedWriter(pipeWriter)
go func() {
// Compress the decompressed source object.
_, cerr := io.Copy(snappyWriter, gr)
snappyWriter.Close()
pipeWriter.CloseWithError(cerr)
}()
} else {
// Remove the metadata for remote calls.
delete(srcInfo.UserDefined, ReservedMetadataPrefix+"compression")
@@ -1043,12 +1011,6 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
if objInfo.IsCompressed() {
objInfo.Size = actualSize
}
@@ -1061,8 +1023,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -1200,12 +1161,12 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
}
case authTypePresigned, authTypeSigned:
if s3Err = reqSignatureV4Verify(r, globalServerConfig.GetRegion()); s3Err != ErrNone {
if s3Err = reqSignatureV4Verify(r, globalServerConfig.GetRegion(), serviceS3); s3Err != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Err), r.URL, guessIsBrowserReq(r))
return
}
if !skipContentSha256Cksum(r) {
sha256hex = getContentSha256Cksum(r)
sha256hex = getContentSha256Cksum(r, serviceS3)
}
}
@@ -1216,28 +1177,17 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
metadata[ReservedMetadataPrefix+"compression"] = compressionAlgorithmV1
metadata[ReservedMetadataPrefix+"actual-size"] = strconv.FormatInt(size, 10)
pipeReader, pipeWriter := io.Pipe()
snappyWriter := snappy.NewBufferedWriter(pipeWriter)
var actualReader *hash.Reader
actualReader, err = hash.NewReader(reader, size, md5hex, sha256hex, actualSize)
actualReader, err := hash.NewReader(reader, size, md5hex, sha256hex, actualSize)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
go func() {
// Writing to the compressed writer.
_, cerr := io.CopyN(snappyWriter, actualReader, actualSize)
snappyWriter.Close()
pipeWriter.CloseWithError(cerr)
}()
// Set compression metrics.
reader = newSnappyCompressReader(actualReader)
size = -1 // Since compressed size is un-predictable.
md5hex = "" // Do not try to verify the content.
sha256hex = ""
reader = pipeReader
}
hashReader, err := hash.NewReader(reader, size, md5hex, sha256hex, actualSize)
@@ -1325,12 +1275,6 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
writeSuccessResponseHeadersOnly(w)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object created event.
sendEvent(eventArgs{
EventName: event.ObjectCreatedPut,
@@ -1339,8 +1283,7 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -1597,9 +1540,17 @@ func (api objectAPIHandlers) CopyObjectPartHandler(w http.ResponseWriter, r *htt
}
}
checkCopyPartPrecondFn := func(o ObjectInfo, encETag string) bool {
return checkCopyObjectPartPreconditions(ctx, w, r, o, encETag)
}
getOpts.CheckCopyPrecondFn = checkCopyPartPrecondFn
srcOpts.CheckCopyPrecondFn = checkCopyPartPrecondFn
gr, err := getObjectNInfo(ctx, srcBucket, srcObject, rs, r.Header, readLock, getOpts)
if err != nil {
if isErrPreconditionFailed(err) {
return
}
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
@@ -1621,11 +1572,6 @@ func (api objectAPIHandlers) CopyObjectPartHandler(w http.ResponseWriter, r *htt
return
}
// Verify before x-amz-copy-source preconditions before continuing with CopyObject.
if checkCopyObjectPartPreconditions(ctx, w, r, srcInfo) {
return
}
// Get the object offset & length
startOffset, length, err := rs.GetOffsetLength(actualPartSize)
if err != nil {
@@ -1654,21 +1600,8 @@ func (api objectAPIHandlers) CopyObjectPartHandler(w http.ResponseWriter, r *htt
isCompressed := compressPart
// Compress only if the compression is enabled during initial multipart.
if isCompressed {
// Open a pipe for compression.
// Where pipeWriter is piped to srcInfo.Reader.
// gr writes to pipeWriter.
pipeReader, pipeWriter := io.Pipe()
reader = pipeReader
reader = newSnappyCompressReader(gr)
length = -1
snappyWriter := snappy.NewBufferedWriter(pipeWriter)
go func() {
// Compress the decompressed source object.
_, cerr := io.Copy(snappyWriter, gr)
snappyWriter.Close()
pipeWriter.CloseWithError(cerr)
}()
} else {
reader = gr
}
@@ -1868,19 +1801,17 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
return
}
case authTypePresigned, authTypeSigned:
if s3Error = reqSignatureV4Verify(r, globalServerConfig.GetRegion()); s3Error != ErrNone {
if s3Error = reqSignatureV4Verify(r, globalServerConfig.GetRegion(), serviceS3); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
if !skipContentSha256Cksum(r) {
sha256hex = getContentSha256Cksum(r)
sha256hex = getContentSha256Cksum(r, serviceS3)
}
}
actualSize := size
var pipeReader *io.PipeReader
var pipeWriter *io.PipeWriter
// get encryption options
var opts ObjectOptions
@@ -1902,28 +1833,17 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
isCompressed := false
if objectAPI.IsCompressionSupported() && compressPart {
pipeReader, pipeWriter = io.Pipe()
snappyWriter := snappy.NewBufferedWriter(pipeWriter)
var actualReader *hash.Reader
actualReader, err = hash.NewReader(reader, size, md5hex, sha256hex, actualSize)
actualReader, err := hash.NewReader(reader, size, md5hex, sha256hex, actualSize)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
go func() {
// Writing to the compressed writer.
_, cerr := io.CopyN(snappyWriter, actualReader, actualSize)
snappyWriter.Close()
pipeWriter.CloseWithError(cerr)
}()
// Set compression metrics.
reader = newSnappyCompressReader(actualReader)
size = -1 // Since compressed size is un-predictable.
md5hex = "" // Do not try to verify the content.
sha256hex = ""
reader = pipeReader
isCompressed = true
}
@@ -2016,7 +1936,6 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
etag := partInfo.ETag
if isCompressed {
pipeWriter.Close()
// Suppress compressed ETag.
etag = partInfo.ETag + "-1"
} else if isEncrypted {
@@ -2094,7 +2013,7 @@ func (api objectAPIHandlers) ListObjectPartsHandler(w http.ResponseWriter, r *ht
return
}
uploadID, partNumberMarker, maxParts, _, s3Error := getObjectResources(r.URL.Query())
uploadID, partNumberMarker, maxParts, encodingType, s3Error := getObjectResources(r.URL.Query())
if s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
@@ -2145,34 +2064,59 @@ func (api objectAPIHandlers) ListObjectPartsHandler(w http.ResponseWriter, r *ht
}
}
response := generateListPartsResponse(listPartsInfo)
response := generateListPartsResponse(listPartsInfo, encodingType)
encodedSuccessResponse := encodeResponse(response)
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
}
// Send white spaces every 10 seconds to the client till completeMultiPartUpload() is done so that the client does not time out.
// Downside is we might send 200OK and then send error XML. But accoording to S3 spec
// the client is supposed to check for error XML even if it received 200OK. But for erasure this is not a
// problem as completeMultiPartUpload() is quick. Even For FS, it would not be an issue
// as we do background append as and when the parts arrive and completeMultiPartUpload is quick.
// Only in a rare case where parts would be out of order will FS' completeMultiPartUpload() take a longer time.
func sendWhiteSpace(ctx context.Context, w http.ResponseWriter) <-chan struct{} {
doneCh := make(chan struct{})
type whiteSpaceWriter struct {
http.ResponseWriter
http.Flusher
written bool
}
func (w *whiteSpaceWriter) Write(b []byte) (n int, err error) {
n, err = w.ResponseWriter.Write(b)
w.written = true
return
}
func (w *whiteSpaceWriter) WriteHeader(statusCode int) {
if !w.written {
w.ResponseWriter.WriteHeader(statusCode)
}
}
// Send empty whitespaces every 10 seconds to the client till completeMultiPartUpload() is
// done so that the client does not time out. Downside is we might send 200 OK and
// then send error XML. But accoording to S3 spec the client is supposed to check
// for error XML even if it received 200 OK. But for erasure this is not a problem
// as completeMultiPartUpload() is quick. Even For FS, it would not be an issue as
// we do background append as and when the parts arrive and completeMultiPartUpload
// is quick. Only in a rare case where parts would be out of order will
// FS:completeMultiPartUpload() take a longer time.
func sendWhiteSpace(ctx context.Context, w http.ResponseWriter) <-chan bool {
doneCh := make(chan bool)
go func() {
ticker := time.NewTicker(time.Second * 10)
headerWritten := false
for {
select {
case <-ticker.C:
// Write a white space char to the client to prevent client timeouts
// when the server takes a long time to completeMultiPartUpload()
if _, err := w.Write([]byte("\n\r")); err != nil {
logger.LogIf(ctx, err)
return
// Write header if not written yet.
if !headerWritten {
w.Write([]byte(xml.Header))
headerWritten = true
}
// Once header is written keep writing empty spaces
// which are ignored by client SDK XML parsers.
// This occurs when server takes long time to completeMultiPartUpload()
w.Write([]byte(" "))
w.(http.Flusher).Flush()
case doneCh <- struct{}{}:
case doneCh <- headerWritten:
ticker.Stop()
return
}
@@ -2309,31 +2253,54 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
if api.CacheAPI() != nil {
completeMultiPartUpload = api.CacheAPI().CompleteMultipartUpload
}
// This code is specifically to handle the requirements for slow
// complete multipart upload operations on FS mode.
writeErrorResponseWithoutXMLHeader := func(ctx context.Context, w http.ResponseWriter, err APIError, reqURL *url.URL) {
switch err.Code {
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
}
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path,
w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
encodedErrorResponse, _ := xml.Marshal(errorResponse)
setCommonHeaders(w)
w.Header().Set("Content-Type", string(mimeXML))
w.Write(encodedErrorResponse)
w.(http.Flusher).Flush()
}
w = &whiteSpaceWriter{ResponseWriter: w, Flusher: w.(http.Flusher)}
completeDoneCh := sendWhiteSpace(ctx, w)
objInfo, err := completeMultiPartUpload(ctx, bucket, object, uploadID, completeParts, opts)
// Stop writing white spaces to the client. Note that close(doneCh) style is not used as it
// can cause white space to be written after we send XML response in a race condition.
<-completeDoneCh
headerWritten := <-completeDoneCh
if err != nil {
switch oErr := err.(type) {
case PartTooSmall:
// Write part too small error.
writePartSmallErrorResponse(ctx, w, r, oErr)
default:
// Handle all other generic issues.
if headerWritten {
writeErrorResponseWithoutXMLHeader(ctx, w, toAPIError(ctx, err), r.URL)
} else {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
}
return
}
// Get object location.
location := getObjectLocation(r, globalDomainName, bucket, object)
location := getObjectLocation(r, globalDomainNames, bucket, object)
// Generate complete multipart response.
response := generateCompleteMultpartUploadResponse(bucket, object, location, objInfo.ETag)
encodedSuccessResponse := encodeResponse(response)
if err != nil {
writeErrorResponse(ctx, w, toAdminAPIErr(ctx, err), r.URL, guessIsBrowserReq(r))
return
var encodedSuccessResponse []byte
if !headerWritten {
encodedSuccessResponse = encodeResponse(response)
} else {
encodedSuccessResponse, err = xml.Marshal(response)
if err != nil {
writeErrorResponseWithoutXMLHeader(ctx, w, toAPIError(ctx, err), r.URL)
return
}
}
// Set etag.
@@ -2343,16 +2310,12 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
if objectAPI.IsEncryptionSupported() {
if crypto.IsEncrypted(objInfo.UserDefined) {
objInfo.Size, _ = objInfo.DecryptedSize()
}
}
// Notify object created event.
sendEvent(eventArgs{
EventName: event.ObjectCreatedCompleteMultipartUpload,
@@ -2361,8 +2324,7 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
+3 -3
View File
@@ -2740,9 +2740,9 @@ func testAPICompleteMultipartHandler(obj ObjectLayer, instanceType, bucketName s
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
expectedContent: encodeResponse(completeMultipartAPIError{int64(4), int64(5242880), 1, "e2fc714c4727ee9395f324cd2e7f331f",
getAPIErrorResponse(ctx, toAPIError(ctx, PartTooSmall{PartNumber: 1}),
getGetObjectURL("", bucketName, objectName), "", "")}),
expectedContent: encodeResponse(getAPIErrorResponse(ctx,
toAPIError(ctx, PartTooSmall{PartNumber: 1}),
getGetObjectURL("", bucketName, objectName), "", "")),
expectedRespStatus: http.StatusBadRequest,
},
// Test case - 5.
+4
View File
@@ -142,6 +142,10 @@ func renameAll(srcFilePath, dstFilePath string) (err error) {
return fmt.Errorf("%s (%s)->(%s)", errCrossDeviceLink, srcFilePath, dstFilePath)
case os.IsNotExist(err):
return errFileNotFound
case os.IsExist(err):
// This is returned only when destination is a directory and we
// are attempting a rename from file to directory.
return errIsNotRegular
default:
return err
}
@@ -18,35 +18,35 @@ package cmd
import "github.com/minio/minio/pkg/event"
// PeerRPCClientTarget - RPCClient is an event.Target which sends event to target of remote peer.
type PeerRPCClientTarget struct {
// PeerRESTClientTarget - RPCClient is an event.Target which sends event to target of remote peer.
type PeerRESTClientTarget struct {
id event.TargetID
remoteTargetID event.TargetID
rpcClient *PeerRPCClient
restClient *peerRESTClient
bucketName string
}
// ID - returns target ID.
func (target *PeerRPCClientTarget) ID() event.TargetID {
func (target *PeerRESTClientTarget) ID() event.TargetID {
return target.id
}
// Send - sends event to remote peer by making RPC call.
func (target *PeerRPCClientTarget) Send(eventData event.Event) error {
return target.rpcClient.SendEvent(target.bucketName, target.id, target.remoteTargetID, eventData)
func (target *PeerRESTClientTarget) Send(eventData event.Event) error {
return target.restClient.SendEvent(target.bucketName, target.id, target.remoteTargetID, eventData)
}
// Close - does nothing and available for interface compatibility.
func (target *PeerRPCClientTarget) Close() error {
func (target *PeerRESTClientTarget) Close() error {
return nil
}
// NewPeerRPCClientTarget - creates RPCClient target with given target ID available in remote peer.
func NewPeerRPCClientTarget(bucketName string, targetID event.TargetID, rpcClient *PeerRPCClient) *PeerRPCClientTarget {
return &PeerRPCClientTarget{
// NewPeerRESTClientTarget - creates RPCClient target with given target ID available in remote peer.
func NewPeerRESTClientTarget(bucketName string, targetID event.TargetID, restClient *peerRESTClient) *PeerRESTClientTarget {
return &PeerRESTClientTarget{
id: event.TargetID{ID: targetID.ID, Name: targetID.Name + "+" + mustGetUUID()},
remoteTargetID: targetID,
bucketName: bucketName,
rpcClient: rpcClient,
restClient: restClient,
}
}
+415
View File
@@ -0,0 +1,415 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"bytes"
"context"
"crypto/tls"
"encoding/gob"
"io"
"net/url"
"github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/cmd/rest"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
// client to talk to peer Nodes.
type peerRESTClient struct {
host *xnet.Host
restClient *rest.Client
connected bool
}
// Reconnect to a peer rest server.
func (client *peerRESTClient) reConnect() error {
// correct (intelligent) retry logic will be
// implemented in subsequent PRs.
client.connected = true
return nil
}
// Wrapper to restClient.Call to handle network errors, in case of network error the connection is marked disconnected
// permanently. The only way to restore the connection is at the xl-sets layer by xlsets.monitorAndConnectEndpoints()
// after verifying format.json
func (client *peerRESTClient) call(method string, values url.Values, body io.Reader, length int64) (respBody io.ReadCloser, err error) {
if !client.connected {
err := client.reConnect()
logger.LogIf(context.Background(), err)
if err != nil {
return nil, err
}
}
if values == nil {
values = make(url.Values)
}
respBody, err = client.restClient.Call(method, values, body, length)
if err == nil {
return respBody, nil
}
if isNetworkError(err) {
client.connected = false
}
return nil, err
}
// Stringer provides a canonicalized representation of node.
func (client *peerRESTClient) String() string {
return client.host.String()
}
// IsOnline - returns whether RPC client failed to connect or not.
func (client *peerRESTClient) IsOnline() bool {
return client.connected
}
// Close - marks the client as closed.
func (client *peerRESTClient) Close() error {
client.connected = false
client.restClient.Close()
return nil
}
// GetLocksResp stores various info from the client for each lock that is requested.
type GetLocksResp map[string][]lockRequesterInfo
// GetLocks - fetch older locks for a remote node.
func (client *peerRESTClient) GetLocks() (locks GetLocksResp, err error) {
respBody, err := client.call(peerRESTMethodGetLocks, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&locks)
return locks, err
}
// ServerInfo - fetch server information for a remote node.
func (client *peerRESTClient) ServerInfo() (info ServerInfoData, err error) {
respBody, err := client.call(peerRESTMethodServerInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// CPULoadInfo - fetch CPU information for a remote node.
func (client *peerRESTClient) CPULoadInfo() (info ServerCPULoadInfo, err error) {
respBody, err := client.call(peerRESTMethodCPULoadInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// DrivePerfInfo - fetch Drive performance information for a remote node.
func (client *peerRESTClient) DrivePerfInfo() (info ServerDrivesPerfInfo, err error) {
respBody, err := client.call(peerRESTMethodDrivePerfInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// MemUsageInfo - fetch memory usage information for a remote node.
func (client *peerRESTClient) MemUsageInfo() (info ServerMemUsageInfo, err error) {
respBody, err := client.call(peerRESTMethodMemUsageInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// StartProfiling - Issues profiling command on the peer node.
func (client *peerRESTClient) StartProfiling(profiler string) error {
values := make(url.Values)
values.Set(peerRESTProfiler, profiler)
respBody, err := client.call(peerRESTMethodStartProfiling, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// DownloadProfileData - download profiled data from a remote node.
func (client *peerRESTClient) DownloadProfileData() (data []byte, err error) {
respBody, err := client.call(peerRESTMethodDownloadProfilingData, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&data)
return data, err
}
// DeleteBucket - Delete notification and policies related to the bucket.
func (client *peerRESTClient) DeleteBucket(bucket string) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
respBody, err := client.call(peerRESTMethodDeleteBucket, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// ReloadFormat - reload format on the peer node.
func (client *peerRESTClient) ReloadFormat(dryRun bool) error {
values := make(url.Values)
if dryRun {
values.Set(peerRESTDryRun, "true")
} else {
values.Set(peerRESTDryRun, "false")
}
respBody, err := client.call(peerRESTMethodReloadFormat, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// ListenBucketNotification - send listent bucket notification to peer nodes.
func (client *peerRESTClient) ListenBucketNotification(bucket string, eventNames []event.Name,
pattern string, targetID event.TargetID, addr xnet.Host) error {
args := listenBucketNotificationReq{
EventNames: eventNames,
Pattern: pattern,
TargetID: targetID,
Addr: addr,
}
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(args)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketNotificationListen, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// SendEvent - calls send event RPC.
func (client *peerRESTClient) SendEvent(bucket string, targetID, remoteTargetID event.TargetID, eventData event.Event) error {
args := sendEventRequest{
TargetID: remoteTargetID,
Event: eventData,
}
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(args)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodSendEvent, values, &reader, -1)
if err != nil {
return err
}
var eventResp sendEventResp
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&eventResp)
if err != nil || !eventResp.Success {
reqInfo := &logger.ReqInfo{BucketName: bucket}
reqInfo.AppendTags("targetID", targetID.Name)
reqInfo.AppendTags("event", eventData.EventName.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
globalNotificationSys.RemoveRemoteTarget(bucket, targetID)
}
return err
}
// RemoteTargetExist - calls remote target ID exist REST API.
func (client *peerRESTClient) RemoteTargetExist(bucket string, targetID event.TargetID) (bool, error) {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(targetID)
if err != nil {
return false, err
}
respBody, err := client.call(peerRESTMethodTargetExists, values, &reader, -1)
if err != nil {
return false, err
}
defer http.DrainBody(respBody)
var targetExists remoteTargetExistsResp
err = gob.NewDecoder(respBody).Decode(&targetExists)
return targetExists.Exists, err
}
// RemoveBucketPolicy - Remove bucket policy on the peer node.
func (client *peerRESTClient) RemoveBucketPolicy(bucket string) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
respBody, err := client.call(peerRESTMethodBucketPolicyRemove, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// SetBucketPolicy - Set bucket policy on the peer node.
func (client *peerRESTClient) SetBucketPolicy(bucket string, bucketPolicy *policy.Policy) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(bucketPolicy)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketPolicySet, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// PutBucketNotification - Put bucket notification on the peer node.
func (client *peerRESTClient) PutBucketNotification(bucket string, rulesMap event.RulesMap) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(&rulesMap)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketNotificationPut, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// LoadUsers - send load users command to peer nodes.
func (client *peerRESTClient) LoadUsers() (err error) {
respBody, err := client.call(peerRESTMethodLoadUsers, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
return nil
}
// SignalService - sends signal to peer nodes.
func (client *peerRESTClient) SignalService(sig serviceSignal) error {
values := make(url.Values)
values.Set(peerRESTSignal, string(sig))
respBody, err := client.call(peerRESTMethodSignalService, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
func getRemoteHosts(endpoints EndpointList) []*xnet.Host {
var remoteHosts []*xnet.Host
for _, hostStr := range GetRemotePeers(endpoints) {
host, err := xnet.ParseHost(hostStr)
logger.FatalIf(err, "Unable to parse peer Host")
remoteHosts = append(remoteHosts, host)
}
return remoteHosts
}
func getRestClients(peerHosts []*xnet.Host) ([]*peerRESTClient, error) {
restClients := make([]*peerRESTClient, len(peerHosts))
for i, host := range peerHosts {
client, err := newPeerRESTClient(host)
if err != nil {
logger.LogIf(context.Background(), err)
}
restClients[i] = client
}
return restClients, nil
}
// Returns a peer rest client.
func newPeerRESTClient(peer *xnet.Host) (*peerRESTClient, error) {
scheme := "http"
if globalIsSSL {
scheme = "https"
}
serverURL := &url.URL{
Scheme: scheme,
Host: peer.String(),
Path: peerRESTPath,
}
var tlsConfig *tls.Config
if globalIsSSL {
tlsConfig = &tls.Config{
ServerName: peer.Name,
RootCAs: globalRootCAs,
}
}
restClient, err := rest.NewClient(serverURL, tlsConfig, rest.DefaultRESTTimeout, newAuthToken)
if err != nil {
return &peerRESTClient{host: peer, restClient: restClient, connected: false}, err
}
return &peerRESTClient{host: peer, restClient: restClient, connected: true}, nil
}
+47
View File
@@ -0,0 +1,47 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
const peerRESTVersion = "v1"
const peerRESTPath = minioReservedBucketPath + "/peer/" + peerRESTVersion
const (
peerRESTMethodServerInfo = "serverinfo"
peerRESTMethodCPULoadInfo = "cpuloadinfo"
peerRESTMethodMemUsageInfo = "memusageinfo"
peerRESTMethodDrivePerfInfo = "driveperfinfo"
peerRESTMethodDeleteBucket = "deletebucket"
peerRESTMethodSignalService = "signalservice"
peerRESTMethodGetLocks = "getlocks"
peerRESTMethodBucketPolicyRemove = "removebucketpolicy"
peerRESTMethodLoadUsers = "loadusers"
peerRESTMethodStartProfiling = "startprofiling"
peerRESTMethodDownloadProfilingData = "downloadprofilingdata"
peerRESTMethodBucketPolicySet = "setbucketpolicy"
peerRESTMethodBucketNotificationPut = "putbucketnotification"
peerRESTMethodBucketNotificationListen = "listenbucketnotification"
peerRESTMethodReloadFormat = "reloadformat"
peerRESTMethodTargetExists = "targetexists"
peerRESTMethodSendEvent = "sendevent"
)
const (
peerRESTBucket = "bucket"
peerRESTSignal = "signal"
peerRESTProfiler = "profiler"
peerRESTDryRun = "dry-run"
)
+592
View File
@@ -0,0 +1,592 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"encoding/gob"
"errors"
"fmt"
"net/http"
"sort"
"strings"
"time"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
// To abstract a node over network.
type peerRESTServer struct {
}
func getServerInfo() (*ServerInfoData, error) {
if globalBootTime.IsZero() {
return nil, errServerNotInitialized
}
objLayer := newObjectLayerFn()
if objLayer == nil {
return nil, errServerNotInitialized
}
// Server info data.
return &ServerInfoData{
StorageInfo: objLayer.StorageInfo(context.Background()),
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
DeploymentID: globalDeploymentID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
},
}, nil
}
// uptimes - used to sort uptimes in chronological order.
type uptimes []time.Duration
func (ts uptimes) Len() int {
return len(ts)
}
func (ts uptimes) Less(i, j int) bool {
return ts[i] < ts[j]
}
func (ts uptimes) Swap(i, j int) {
ts[i], ts[j] = ts[j], ts[i]
}
// getPeerUptimes - returns the uptime.
func getPeerUptimes(serverInfo []ServerInfo) time.Duration {
// In a single node Erasure or FS backend setup the uptime of
// the setup is the uptime of the single minio server
// instance.
if !globalIsDistXL {
return UTCNow().Sub(globalBootTime)
}
var times []time.Duration
for _, info := range serverInfo {
if info.Error != "" {
continue
}
times = append(times, info.Data.Properties.Uptime)
}
// Sort uptimes in chronological order.
sort.Sort(uptimes(times))
// Return the latest time as the uptime.
return times[0]
}
// GetLocksHandler - returns list of older lock from the server.
func (s *peerRESTServer) GetLocksHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "GetLocks")
locks := globalLockServer.ll.DupLockMap()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(locks))
w.(http.Flusher).Flush()
}
// LoadUsersHandler - returns server info.
func (s *peerRESTServer) LoadUsersHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
objAPI := newObjectLayerFn()
if objAPI == nil {
s.writeErrorResponse(w, errServerNotInitialized)
return
}
err := globalIAMSys.Load(objAPI)
if err != nil {
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
// StartProfilingHandler - Issues the start profiling command.
func (s *peerRESTServer) StartProfilingHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
profiler := vars[peerRESTProfiler]
if profiler == "" {
s.writeErrorResponse(w, errors.New("profiler name is missing"))
return
}
if globalProfiler != nil {
globalProfiler.Stop()
}
var err error
globalProfiler, err = startProfiler(profiler, "")
if err != nil {
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
// ServerInfoHandler - returns server info.
func (s *peerRESTServer) ServerInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "ServerInfo")
info, err := getServerInfo()
if err != nil {
s.writeErrorResponse(w, err)
return
}
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// DownloadProflingDataHandler - returns proflied data.
func (s *peerRESTServer) DownloadProflingDataHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "DownloadProfiling")
profileData, err := getProfileData()
if err != nil {
s.writeErrorResponse(w, err)
return
}
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(profileData))
}
// CPULoadInfoHandler - returns CPU Load info.
func (s *peerRESTServer) CPULoadInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "CPULoadInfo")
info := localEndpointsCPULoad(globalEndpoints)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// DrivePerfInfoHandler - returns Drive Performance info.
func (s *peerRESTServer) DrivePerfInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "DrivePerfInfo")
info := localEndpointsDrivePerf(globalEndpoints)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// MemUsageInfoHandler - returns Memory Usage info.
func (s *peerRESTServer) MemUsageInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "MemUsageInfo")
info := localEndpointsMemUsage(globalEndpoints)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// DeleteBucketHandler - Delete notification and policies related to the bucket.
func (s *peerRESTServer) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
globalNotificationSys.RemoveNotification(bucketName)
globalPolicySys.Remove(bucketName)
w.(http.Flusher).Flush()
}
// ReloadFormatHandler - Reload Format.
func (s *peerRESTServer) ReloadFormatHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
dryRunString := vars[peerRESTDryRun]
if dryRunString == "" {
s.writeErrorResponse(w, errors.New("dry run parameter is missing"))
return
}
var dryRun bool
switch strings.ToLower(dryRunString) {
case "true":
dryRun = true
case "false":
dryRun = false
default:
s.writeErrorResponse(w, errInvalidArgument)
return
}
objAPI := newObjectLayerFn()
if objAPI == nil {
s.writeErrorResponse(w, errServerNotInitialized)
return
}
err := objAPI.ReloadFormat(context.Background(), dryRun)
if err != nil {
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
// RemoveBucketPolicyHandler - Remove bucket policy.
func (s *peerRESTServer) RemoveBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
globalPolicySys.Remove(bucketName)
w.(http.Flusher).Flush()
}
// SetBucketPolicyHandler - Set bucket policy.
func (s *peerRESTServer) SetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var policyData policy.Policy
if r.ContentLength < 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&policyData)
if err != nil {
s.writeErrorResponse(w, err)
return
}
globalPolicySys.Set(bucketName, policyData)
w.(http.Flusher).Flush()
}
type remoteTargetExistsResp struct {
Exists bool
}
// TargetExistsHandler - Check if Target exists.
func (s *peerRESTServer) TargetExistsHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "TargetExists")
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var targetID event.TargetID
if r.ContentLength <= 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&targetID)
if err != nil {
s.writeErrorResponse(w, err)
return
}
var targetExists remoteTargetExistsResp
targetExists.Exists = globalNotificationSys.RemoteTargetExist(bucketName, targetID)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(&targetExists))
}
type sendEventRequest struct {
Event event.Event
TargetID event.TargetID
}
type sendEventResp struct {
Success bool
}
// SendEventHandler - Send Event.
func (s *peerRESTServer) SendEventHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "SendEvent")
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var eventReq sendEventRequest
if r.ContentLength <= 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&eventReq)
if err != nil {
s.writeErrorResponse(w, err)
return
}
var eventResp sendEventResp
eventResp.Success = true
errs := globalNotificationSys.send(bucketName, eventReq.Event, eventReq.TargetID)
for i := range errs {
reqInfo := (&logger.ReqInfo{}).AppendTags("Event", eventReq.Event.EventName.String())
reqInfo.AppendTags("targetName", eventReq.TargetID.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errs[i].Err)
eventResp.Success = false
s.writeErrorResponse(w, errs[i].Err)
return
}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(&eventResp))
w.(http.Flusher).Flush()
}
// PutBucketNotificationHandler - Set bucket policy.
func (s *peerRESTServer) PutBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var rulesMap event.RulesMap
if r.ContentLength < 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&rulesMap)
if err != nil {
s.writeErrorResponse(w, err)
return
}
globalNotificationSys.AddRulesMap(bucketName, rulesMap)
w.(http.Flusher).Flush()
}
type listenBucketNotificationReq struct {
EventNames []event.Name `json:"eventNames"`
Pattern string `json:"pattern"`
TargetID event.TargetID `json:"targetId"`
Addr xnet.Host `json:"addr"`
}
// ListenBucketNotificationHandler - Listen bucket notification handler.
func (s *peerRESTServer) ListenBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var args listenBucketNotificationReq
if r.ContentLength <= 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&args)
if err != nil {
s.writeErrorResponse(w, err)
return
}
restClient, err := newPeerRESTClient(&args.Addr)
if err != nil {
s.writeErrorResponse(w, fmt.Errorf("unable to find PeerRESTClient for provided address %v. This happens only if remote and this minio run with different set of endpoints", args.Addr))
return
}
target := NewPeerRESTClientTarget(bucketName, args.TargetID, restClient)
rulesMap := event.NewRulesMap(args.EventNames, args.Pattern, target.ID())
if err := globalNotificationSys.AddRemoteTarget(bucketName, target, rulesMap); err != nil {
reqInfo := &logger.ReqInfo{BucketName: target.bucketName}
reqInfo.AppendTags("target", target.id.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
var errUnsupportedSignal = fmt.Errorf("unsupported signal: only restart and stop signals are supported")
// SignalServiceHandler - signal service handler.
func (s *peerRESTServer) SignalServiceHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
signalString := vars[peerRESTSignal]
if signalString == "" {
s.writeErrorResponse(w, errors.New("signal name is missing"))
return
}
signal := serviceSignal(signalString)
defer w.(http.Flusher).Flush()
switch signal {
case serviceRestart, serviceStop:
globalServiceSignalCh <- signal
default:
s.writeErrorResponse(w, errUnsupportedSignal)
return
}
}
func (s *peerRESTServer) writeErrorResponse(w http.ResponseWriter, err error) {
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(err.Error()))
}
// IsValid - To authenticate and verify the time difference.
func (s *peerRESTServer) IsValid(w http.ResponseWriter, r *http.Request) bool {
if err := storageServerRequestValidate(r); err != nil {
s.writeErrorResponse(w, err)
return false
}
return true
}
// registerPeerRESTHandlers - register peer rest router.
func registerPeerRESTHandlers(router *mux.Router) {
server := &peerRESTServer{}
subrouter := router.PathPrefix(peerRESTPath).Subrouter()
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodGetLocks).HandlerFunc(httpTraceHdrs(server.GetLocksHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodServerInfo).HandlerFunc(httpTraceHdrs(server.ServerInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodCPULoadInfo).HandlerFunc(httpTraceHdrs(server.CPULoadInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodMemUsageInfo).HandlerFunc(httpTraceHdrs(server.MemUsageInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodDrivePerfInfo).HandlerFunc(httpTraceHdrs(server.DrivePerfInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodDeleteBucket).HandlerFunc(httpTraceHdrs(server.DeleteBucketHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodSignalService).HandlerFunc(httpTraceHdrs(server.SignalServiceHandler)).Queries(restQueries(peerRESTSignal)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketPolicyRemove).HandlerFunc(httpTraceAll(server.RemoveBucketPolicyHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketPolicySet).HandlerFunc(httpTraceHdrs(server.SetBucketPolicyHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodLoadUsers).HandlerFunc(httpTraceAll(server.LoadUsersHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodStartProfiling).HandlerFunc(httpTraceAll(server.StartProfilingHandler)).Queries(restQueries(peerRESTProfiler)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodDownloadProfilingData).HandlerFunc(httpTraceHdrs(server.DownloadProflingDataHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodTargetExists).HandlerFunc(httpTraceHdrs(server.TargetExistsHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodSendEvent).HandlerFunc(httpTraceHdrs(server.SendEventHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketNotificationPut).HandlerFunc(httpTraceHdrs(server.PutBucketNotificationHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketNotificationListen).HandlerFunc(httpTraceHdrs(server.ListenBucketNotificationHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodReloadFormat).HandlerFunc(httpTraceHdrs(server.ReloadFormatHandler)).Queries(restQueries(peerRESTDryRun)...)
router.NotFoundHandler = http.HandlerFunc(httpTraceAll(notFoundHandler))
}
-261
View File
@@ -1,261 +0,0 @@
/*
* Minio Cloud Storage, (C) 2018, 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"crypto/tls"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
// PeerRPCClient - peer RPC client talks to peer RPC server.
type PeerRPCClient struct {
*RPCClient
}
// DeleteBucket - calls delete bucket RPC.
func (rpcClient *PeerRPCClient) DeleteBucket(bucketName string) error {
args := DeleteBucketArgs{BucketName: bucketName}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".DeleteBucket", &args, &reply)
}
// SetBucketPolicy - calls set bucket policy RPC.
func (rpcClient *PeerRPCClient) SetBucketPolicy(bucketName string, bucketPolicy *policy.Policy) error {
args := SetBucketPolicyArgs{
BucketName: bucketName,
Policy: *bucketPolicy,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".SetBucketPolicy", &args, &reply)
}
// RemoveBucketPolicy - calls remove bucket policy RPC.
func (rpcClient *PeerRPCClient) RemoveBucketPolicy(bucketName string) error {
args := RemoveBucketPolicyArgs{
BucketName: bucketName,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".RemoveBucketPolicy", &args, &reply)
}
// PutBucketNotification - calls put bukcet notification RPC.
func (rpcClient *PeerRPCClient) PutBucketNotification(bucketName string, rulesMap event.RulesMap) error {
args := PutBucketNotificationArgs{
BucketName: bucketName,
RulesMap: rulesMap,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".PutBucketNotification", &args, &reply)
}
// ListenBucketNotification - calls listen bucket notification RPC.
func (rpcClient *PeerRPCClient) ListenBucketNotification(bucketName string, eventNames []event.Name,
pattern string, targetID event.TargetID, addr xnet.Host) error {
args := ListenBucketNotificationArgs{
BucketName: bucketName,
EventNames: eventNames,
Pattern: pattern,
TargetID: targetID,
Addr: addr,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".ListenBucketNotification", &args, &reply)
}
// RemoteTargetExist - calls remote target ID exist RPC.
func (rpcClient *PeerRPCClient) RemoteTargetExist(bucketName string, targetID event.TargetID) (bool, error) {
args := RemoteTargetExistArgs{
BucketName: bucketName,
TargetID: targetID,
}
var reply bool
err := rpcClient.Call(peerServiceName+".RemoteTargetExist", &args, &reply)
return reply, err
}
// SendEvent - calls send event RPC.
func (rpcClient *PeerRPCClient) SendEvent(bucketName string, targetID, remoteTargetID event.TargetID, eventData event.Event) error {
args := SendEventArgs{
BucketName: bucketName,
TargetID: remoteTargetID,
Event: eventData,
}
var reply bool
err := rpcClient.Call(peerServiceName+".SendEvent", &args, &reply)
if err != nil && !reply {
reqInfo := &logger.ReqInfo{BucketName: bucketName}
reqInfo.AppendTags("targetID", targetID.Name)
reqInfo.AppendTags("event", eventData.EventName.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
globalNotificationSys.RemoveRemoteTarget(bucketName, targetID)
}
return err
}
// ReloadFormat - calls reload format RPC.
func (rpcClient *PeerRPCClient) ReloadFormat(dryRun bool) error {
args := ReloadFormatArgs{
DryRun: dryRun,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".ReloadFormat", &args, &reply)
}
// LoadUsers - calls load users RPC.
func (rpcClient *PeerRPCClient) LoadUsers() error {
args := AuthArgs{}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".LoadUsers", &args, &reply)
}
// LoadCredentials - calls load credentials RPC.
func (rpcClient *PeerRPCClient) LoadCredentials() error {
args := AuthArgs{}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".LoadCredentials", &args, &reply)
}
// DrivePerfInfo - returns drive performance info for remote server.
func (rpcClient *PeerRPCClient) DrivePerfInfo() (ServerDrivesPerfInfo, error) {
args := AuthArgs{}
var reply ServerDrivesPerfInfo
err := rpcClient.Call(peerServiceName+".DrivePerfInfo", &args, &reply)
return reply, err
}
// MemUsageInfo - returns mem utilization info for remote server
func (rpcClient *PeerRPCClient) MemUsageInfo() (ServerMemUsageInfo, error) {
args := AuthArgs{}
var reply ServerMemUsageInfo
err := rpcClient.Call(peerServiceName+".MemUsageInfo", &args, &reply)
return reply, err
}
// CPULoadInfo - returns cpu performance info for remote server
func (rpcClient *PeerRPCClient) CPULoadInfo() (ServerCPULoadInfo, error) {
args := AuthArgs{}
var reply ServerCPULoadInfo
err := rpcClient.Call(peerServiceName+".CPULoadInfo", &args, &reply)
return reply, err
}
// StartProfiling - starts profiling on the remote server.
func (rpcClient *PeerRPCClient) StartProfiling(profiler string) error {
args := StartProfilingArgs{Profiler: profiler}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".StartProfiling", &args, &reply)
}
// DownloadProfilingData - download already running profiling on the remote server.
func (rpcClient *PeerRPCClient) DownloadProfilingData() ([]byte, error) {
args := AuthArgs{}
var reply []byte
err := rpcClient.Call(peerServiceName+".DownloadProfilingData", &args, &reply)
return reply, err
}
// SignalService - calls load server info RPC.
func (rpcClient *PeerRPCClient) SignalService(sig serviceSignal) error {
args := SignalServiceArgs{Sig: sig}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".SignalService", &args, &reply)
}
// ServerInfo - calls load server info RPC.
func (rpcClient *PeerRPCClient) ServerInfo() (ServerInfoData, error) {
args := AuthArgs{}
reply := ServerInfoData{}
err := rpcClient.Call(peerServiceName+".ServerInfo", &args, &reply)
return reply, err
}
// GetLocksResp stores various info from the client for each lock that is requested.
type GetLocksResp map[string][]lockRequesterInfo
// GetLocks - returns the lock information on the server to which the RPC call is made.
func (rpcClient *PeerRPCClient) GetLocks() (resp GetLocksResp, err error) {
err = rpcClient.Call(peerServiceName+".GetLocks", &AuthArgs{}, &resp)
return resp, err
}
// NewPeerRPCClient - returns new peer RPC client.
func NewPeerRPCClient(host *xnet.Host) (*PeerRPCClient, error) {
scheme := "http"
if globalIsSSL {
scheme = "https"
}
serviceURL := &xnet.URL{
Scheme: scheme,
Host: host.String(),
Path: peerServicePath,
}
var tlsConfig *tls.Config
if globalIsSSL {
tlsConfig = &tls.Config{
ServerName: host.Name,
RootCAs: globalRootCAs,
}
}
rpcClient, err := NewRPCClient(
RPCClientArgs{
NewAuthTokenFunc: newAuthToken,
RPCVersion: globalRPCAPIVersion,
ServiceName: peerServiceName,
ServiceURL: serviceURL,
TLSConfig: tlsConfig,
},
)
if err != nil {
return nil, err
}
return &PeerRPCClient{rpcClient}, nil
}
// makeRemoteRPCClients - creates Peer RPCClients for given endpoint list.
func makeRemoteRPCClients(endpoints EndpointList) map[xnet.Host]*PeerRPCClient {
peerRPCClientMap := make(map[xnet.Host]*PeerRPCClient)
for _, hostStr := range GetRemotePeers(endpoints) {
host, err := xnet.ParseHost(hostStr)
logger.FatalIf(err, "Unable to parse peer RPC Host")
rpcClient, err := NewPeerRPCClient(host)
logger.FatalIf(err, "Unable to parse peer RPC Client")
peerRPCClientMap[*host] = rpcClient
}
return peerRPCClientMap
}
-419
View File
@@ -1,419 +0,0 @@
/*
* Minio Cloud Storage, (C) 2018, 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"fmt"
"path"
"sort"
"time"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
xrpc "github.com/minio/minio/cmd/rpc"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
const peerServiceName = "Peer"
const peerServiceSubPath = "/peer/remote"
var peerServicePath = path.Join(minioReservedBucketPath, peerServiceSubPath)
// peerRPCReceiver - Peer RPC receiver for peer RPC server.
type peerRPCReceiver struct{}
// DeleteBucketArgs - delete bucket RPC arguments.
type DeleteBucketArgs struct {
AuthArgs
BucketName string
}
// DeleteBucket - handles delete bucket RPC call which removes all values of given bucket in global NotificationSys object.
func (receiver *peerRPCReceiver) DeleteBucket(args *DeleteBucketArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalNotificationSys.RemoveNotification(args.BucketName)
globalPolicySys.Remove(args.BucketName)
return nil
}
// SetBucketPolicyArgs - set bucket policy RPC arguments.
type SetBucketPolicyArgs struct {
AuthArgs
BucketName string
Policy policy.Policy
}
// SetBucketPolicy - handles set bucket policy RPC call which adds bucket policy to globalPolicySys.
func (receiver *peerRPCReceiver) SetBucketPolicy(args *SetBucketPolicyArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalPolicySys.Set(args.BucketName, args.Policy)
return nil
}
// RemoveBucketPolicyArgs - delete bucket policy RPC arguments.
type RemoveBucketPolicyArgs struct {
AuthArgs
BucketName string
}
// RemoveBucketPolicy - handles delete bucket policy RPC call which removes bucket policy to globalPolicySys.
func (receiver *peerRPCReceiver) RemoveBucketPolicy(args *RemoveBucketPolicyArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalPolicySys.Remove(args.BucketName)
return nil
}
// PutBucketNotificationArgs - put bucket notification RPC arguments.
type PutBucketNotificationArgs struct {
AuthArgs
BucketName string
RulesMap event.RulesMap
}
// PutBucketNotification - handles put bucket notification RPC call which adds rules to given bucket to global NotificationSys object.
func (receiver *peerRPCReceiver) PutBucketNotification(args *PutBucketNotificationArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalNotificationSys.AddRulesMap(args.BucketName, args.RulesMap)
return nil
}
// ListenBucketNotificationArgs - listen bucket notification RPC arguments.
type ListenBucketNotificationArgs struct {
AuthArgs `json:"-"`
BucketName string `json:"-"`
EventNames []event.Name `json:"eventNames"`
Pattern string `json:"pattern"`
TargetID event.TargetID `json:"targetId"`
Addr xnet.Host `json:"addr"`
}
// ListenBucketNotification - handles listen bucket notification RPC call.
// It creates PeerRPCClient target which pushes requested events to target in remote peer.
func (receiver *peerRPCReceiver) ListenBucketNotification(args *ListenBucketNotificationArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
rpcClient := globalNotificationSys.GetPeerRPCClient(args.Addr)
if rpcClient == nil {
return fmt.Errorf("unable to find PeerRPCClient for provided address %v. This happens only if remote and this minio run with different set of endpoints", args.Addr)
}
target := NewPeerRPCClientTarget(args.BucketName, args.TargetID, rpcClient)
rulesMap := event.NewRulesMap(args.EventNames, args.Pattern, target.ID())
if err := globalNotificationSys.AddRemoteTarget(args.BucketName, target, rulesMap); err != nil {
reqInfo := &logger.ReqInfo{BucketName: target.bucketName}
reqInfo.AppendTags("target", target.id.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return err
}
return nil
}
// RemoteTargetExistArgs - remote target ID exist RPC arguments.
type RemoteTargetExistArgs struct {
AuthArgs
BucketName string
TargetID event.TargetID
}
// RemoteTargetExist - handles target ID exist RPC call which checks whether given target ID is a HTTP client target or not.
func (receiver *peerRPCReceiver) RemoteTargetExist(args *RemoteTargetExistArgs, reply *bool) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = globalNotificationSys.RemoteTargetExist(args.BucketName, args.TargetID)
return nil
}
// SendEventArgs - send event RPC arguments.
type SendEventArgs struct {
AuthArgs
Event event.Event
TargetID event.TargetID
BucketName string
}
// SendEvent - handles send event RPC call which sends given event to target by given target ID.
func (receiver *peerRPCReceiver) SendEvent(args *SendEventArgs, reply *bool) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
// Set default to true to keep the target.
*reply = true
errs := globalNotificationSys.send(args.BucketName, args.Event, args.TargetID)
for i := range errs {
reqInfo := (&logger.ReqInfo{}).AppendTags("Event", args.Event.EventName.String())
reqInfo.AppendTags("targetName", args.TargetID.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errs[i].Err)
*reply = false // send failed i.e. do not keep the target.
return errs[i].Err
}
return nil
}
// ReloadFormatArgs - send event RPC arguments.
type ReloadFormatArgs struct {
AuthArgs
DryRun bool
}
// ReloadFormat - handles reload format RPC call, reloads latest `format.json`
func (receiver *peerRPCReceiver) ReloadFormat(args *ReloadFormatArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
return objAPI.ReloadFormat(context.Background(), args.DryRun)
}
// LoadUsers - handles load users RPC call.
func (receiver *peerRPCReceiver) LoadUsers(args *AuthArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
return globalIAMSys.Load(objAPI)
}
// LoadCredentials - handles load credentials RPC call.
func (receiver *peerRPCReceiver) LoadCredentials(args *AuthArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
// Construct path to config.json for the given bucket.
configFile := path.Join(bucketConfigPrefix, minioConfigFile)
transactionConfigFile := configFile + ".transaction"
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid race.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
if err := objLock.GetRLock(globalOperationTimeout); err != nil {
return err
}
objLock.RUnlock()
return globalConfigSys.Load(newObjectLayerFn())
}
// DrivePerfInfo - handles drive performance RPC call
func (receiver *peerRPCReceiver) DrivePerfInfo(args *AuthArgs, reply *ServerDrivesPerfInfo) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = localEndpointsDrivePerf(globalEndpoints)
return nil
}
// CPULoadInfo - handles cpu performance RPC call
func (receiver *peerRPCReceiver) CPULoadInfo(args *AuthArgs, reply *ServerCPULoadInfo) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = localEndpointsCPULoad(globalEndpoints)
return nil
}
// MemUsageInfo - handles mem utilization RPC call
func (receiver *peerRPCReceiver) MemUsageInfo(args *AuthArgs, reply *ServerMemUsageInfo) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = localEndpointsMemUsage(globalEndpoints)
return nil
}
// uptimes - used to sort uptimes in chronological order.
type uptimes []time.Duration
func (ts uptimes) Len() int {
return len(ts)
}
func (ts uptimes) Less(i, j int) bool {
return ts[i] < ts[j]
}
func (ts uptimes) Swap(i, j int) {
ts[i], ts[j] = ts[j], ts[i]
}
// getPeerUptimes - returns the uptime.
func getPeerUptimes(serverInfo []ServerInfo) time.Duration {
// In a single node Erasure or FS backend setup the uptime of
// the setup is the uptime of the single minio server
// instance.
if !globalIsDistXL {
return UTCNow().Sub(globalBootTime)
}
var times []time.Duration
for _, info := range serverInfo {
if info.Error != "" {
continue
}
times = append(times, info.Data.Properties.Uptime)
}
// Sort uptimes in chronological order.
sort.Sort(uptimes(times))
// Return the latest time as the uptime.
return times[0]
}
// StartProfilingArgs - holds the RPC argument for StartingProfiling RPC call
type StartProfilingArgs struct {
AuthArgs
Profiler string
}
// StartProfiling - profiling server receiver.
func (receiver *peerRPCReceiver) StartProfiling(args *StartProfilingArgs, reply *VoidReply) error {
if globalProfiler != nil {
globalProfiler.Stop()
}
var err error
globalProfiler, err = startProfiler(args.Profiler, "")
return err
}
// DownloadProfilingData - download profiling data.
func (receiver *peerRPCReceiver) DownloadProfilingData(args *AuthArgs, reply *[]byte) error {
var err error
*reply, err = getProfileData()
return err
}
var errUnsupportedSignal = fmt.Errorf("unsupported signal: only restart and stop signals are supported")
// SignalServiceArgs - send event RPC arguments.
type SignalServiceArgs struct {
AuthArgs
Sig serviceSignal
}
// SignalService - signal service receiver.
func (receiver *peerRPCReceiver) SignalService(args *SignalServiceArgs, reply *VoidReply) error {
switch args.Sig {
case serviceRestart, serviceStop:
globalServiceSignalCh <- args.Sig
default:
return errUnsupportedSignal
}
return nil
}
// ServerInfo - server info receiver.
func (receiver *peerRPCReceiver) ServerInfo(args *AuthArgs, reply *ServerInfoData) error {
if globalBootTime.IsZero() {
return errServerNotInitialized
}
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
return errServerNotInitialized
}
// Server info data.
*reply = ServerInfoData{
StorageInfo: objLayer.StorageInfo(context.Background()),
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
},
}
return nil
}
// GetLocks - Get Locks receiver.
func (receiver *peerRPCReceiver) GetLocks(args *AuthArgs, reply *GetLocksResp) error {
if globalBootTime.IsZero() {
return errServerNotInitialized
}
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
return errServerNotInitialized
}
// Locks data.
*reply = globalLockServer.ll.DupLockMap()
return nil
}
// NewPeerRPCServer - returns new peer RPC server.
func NewPeerRPCServer() (*xrpc.Server, error) {
rpcServer := xrpc.NewServer()
if err := rpcServer.RegisterName(peerServiceName, &peerRPCReceiver{}); err != nil {
return nil, err
}
return rpcServer, nil
}
// registerPeerRPCRouter - creates and registers Peer RPC server and its router.
func registerPeerRPCRouter(router *mux.Router) {
rpcServer, err := NewPeerRPCServer()
logger.FatalIf(err, "Unable to initialize peer RPC Server")
subrouter := router.PathPrefix(minioReservedBucketPath).Subrouter()
subrouter.Path(peerServiceSubPath).HandlerFunc(httpTraceHdrs(rpcServer.ServeHTTP))
}
+15 -2
View File
@@ -20,6 +20,7 @@ import (
"fmt"
"io/ioutil"
"os"
"path"
"path/filepath"
"runtime"
"sort"
@@ -33,15 +34,27 @@ func TestReadDirFail(t *testing.T) {
t.Fatalf("expected = %s, got: %s", errFileNotFound, err)
}
file := path.Join(os.TempDir(), "issue")
if err := ioutil.WriteFile(file, []byte(""), 0644); err != nil {
t.Fatal(err)
}
defer os.RemoveAll(file)
// Check if file is given.
if _, err := readDir("/etc/issue/mydir"); err != errFileNotFound {
if _, err := readDir(path.Join(file, "mydir")); err != errFileNotFound {
t.Fatalf("expected = %s, got: %s", errFileNotFound, err)
}
// Only valid for linux.
if runtime.GOOS == "linux" {
permDir := path.Join(os.TempDir(), "perm-dir")
if err := os.MkdirAll(permDir, os.FileMode(0200)); err != nil {
t.Fatal(err)
}
defer os.RemoveAll(permDir)
// Check if permission denied.
if _, err := readDir("/proc/1/fd"); err == nil {
if _, err := readDir(permDir); err == nil {
t.Fatalf("expected = an error, got: nil")
}
}
+32 -34
View File
@@ -232,7 +232,6 @@ func getDiskInfo(diskPath string) (di disk.Info, err error) {
var ignoreDiskFreeOS = []string{
globalWindowsOSName,
globalNetBSDOSName,
globalSolarisOSName,
}
// check if disk total has minimum required size.
@@ -943,20 +942,6 @@ func (s *posix) openFile(volume, path string, mode int) (f *os.File, err error)
return w, nil
}
// Just like io.LimitedReader but supports Close() to be compatible with io.ReadCloser that is
// returned by posix.ReadFileStream()
type posixLimitedReader struct {
io.LimitedReader
}
func (l *posixLimitedReader) Close() error {
c, ok := l.R.(io.Closer)
if !ok {
return errUnexpected
}
return c.Close()
}
// ReadFileStream - Returns the read stream of the file.
func (s *posix) ReadFileStream(volume, path string, offset, length int64) (io.ReadCloser, error) {
var err error
@@ -1030,7 +1015,10 @@ func (s *posix) ReadFileStream(volume, path string, offset, length int64) (io.Re
if _, err = file.Seek(offset, io.SeekStart); err != nil {
return nil, err
}
return &posixLimitedReader{io.LimitedReader{R: file, N: length}}, nil
return struct {
io.Reader
io.Closer
}{Reader: io.LimitReader(file, length), Closer: file}, nil
}
// CreateFile - creates the file.
@@ -1195,18 +1183,19 @@ func (s *posix) StatFile(volume, path string) (file FileInfo, err error) {
}
st, err := os.Stat((filePath))
if err != nil {
// File is really not found.
if os.IsNotExist(err) {
switch {
case os.IsNotExist(err):
// File is really not found.
return FileInfo{}, errFileNotFound
} else if isSysErrIO(err) {
case isSysErrIO(err):
return FileInfo{}, errFaultyDisk
} else if isSysErrNotDir(err) {
case isSysErrNotDir(err):
// File path cannot be verified since one of the parents is a file.
return FileInfo{}, errFileNotFound
default:
// Return all errors here.
return FileInfo{}, err
}
// Return all errors here.
return FileInfo{}, err
}
// If its a directory its not a regular file.
if st.Mode().IsDir() {
@@ -1231,21 +1220,21 @@ func deleteFile(basePath, deletePath string) error {
// Attempt to remove path.
if err := os.Remove((deletePath)); err != nil {
// Ignore errors if the directory is not empty. The server relies on
// this functionality, and sometimes uses recursion that should not
// error on parent directories.
if isSysErrNotEmpty(err) {
switch {
case isSysErrNotEmpty(err):
// Ignore errors if the directory is not empty. The server relies on
// this functionality, and sometimes uses recursion that should not
// error on parent directories.
return nil
}
if os.IsNotExist(err) {
case os.IsNotExist(err):
return errFileNotFound
} else if os.IsPermission(err) {
case os.IsPermission(err):
return errFileAccessDenied
} else if isSysErrIO(err) {
case isSysErrIO(err):
return errFaultyDisk
default:
return err
}
return err
}
// Trailing slash is removed when found to ensure
@@ -1369,9 +1358,18 @@ func (s *posix) RenameFile(srcVolume, srcPath, dstVolume, dstPath string) (err e
if err == nil && !isDirEmpty(dstFilePath) {
return errFileAccessDenied
}
if !os.IsNotExist(err) {
if err != nil && !os.IsNotExist(err) {
return err
}
// Empty destination remove it before rename.
if isDirEmpty(dstFilePath) {
if err = os.Remove(dstFilePath); err != nil {
if isSysErrNotEmpty(err) {
return errFileAccessDenied
}
return err
}
}
}
if err = renameAll(srcFilePath, dstFilePath); err != nil {
+12 -41
View File
@@ -1069,70 +1069,53 @@ func TestPosixReadFile(t *testing.T) {
volume, "path/to/my/object", 0, 5,
[]byte("hello"), nil,
},
// One path segment length is 255 chars long. - 3
{
volume, "path/to/my/object000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001",
0, 5, []byte("hello"), nil},
// Whole path is 1024 characters long, success case. - 4
{
volume, "level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003/object000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001",
0, 5, []byte("hello"),
func() error {
// On darwin HFS does not support > 1024 characters.
if runtime.GOOS == "darwin" {
return errFileNameTooLong
}
// On all other platforms return success.
return nil
}(),
},
// Object is a directory. - 5
// Object is a directory. - 3
{
volume, "object-as-dir",
0, 5, nil, errIsNotRegular},
// One path segment length is > 255 chars long. - 6
// One path segment length is > 255 chars long. - 4
{
volume, "path/to/my/object0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001",
0, 5, nil, errFileNameTooLong},
// Path length is > 1024 chars long. - 7
// Path length is > 1024 chars long. - 5
{
volume, "level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003/object000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001",
0, 5, nil, errFileNameTooLong},
// Buffer size greater than object size. - 8
// Buffer size greater than object size. - 6
{
volume, "myobject", 0, 16,
[]byte("hello, world"),
io.ErrUnexpectedEOF,
},
// Reading from an offset success. - 9
// Reading from an offset success. - 7
{
volume, "myobject", 7, 5,
[]byte("world"), nil,
},
// Reading from an object but buffer size greater. - 10
// Reading from an object but buffer size greater. - 8
{
volume, "myobject",
7, 8,
[]byte("world"),
io.ErrUnexpectedEOF,
},
// Seeking ahead returns io.EOF. - 11
// Seeking ahead returns io.EOF. - 9
{
volume, "myobject", 14, 1, nil, io.EOF,
},
// Empty volume name. - 12
// Empty volume name. - 10
{
"", "myobject", 14, 1, nil, errVolumeNotFound,
},
// Empty filename name. - 13
// Empty filename name. - 11
{
volume, "", 14, 1, nil, errIsNotRegular,
},
// Non existent volume name - 14.
// Non existent volume name - 12
{
"abcd", "", 14, 1, nil, errVolumeNotFound,
},
// Non existent filename - 15.
// Non existent filename - 13
{
volume, "abcd", 14, 1, nil, errFileNotFound,
},
@@ -1361,8 +1344,7 @@ func TestPosixAppendFile(t *testing.T) {
{"myobject", nil},
// TestPosix to use same path of previously created file.
{"path/to/my/testobject", nil},
// One path segment length is 255 chars long.
{"path/to/my/object000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001", nil},
// TestPosix to use object is a directory now.
{"object-as-dir", errIsNotRegular},
// path segment uses previously uploaded object.
{"myobject/testobject", errFileAccessDenied},
@@ -1372,17 +1354,6 @@ func TestPosixAppendFile(t *testing.T) {
{"level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003/object000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001", errFileNameTooLong},
}
// Add path length > 1024 test specially as OS X system does not support 1024 long path.
err = errFileNameTooLong
if runtime.GOOS != "darwin" {
err = nil
}
// path length is 1024 chars long.
testCases = append(testCases, struct {
fileName string
expectedErr error
}{"level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002/level0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003/object000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001", err})
for i, testCase := range testCases {
if err = posixStorage.AppendFile("success-vol", testCase.fileName, []byte("hello, world")); err != testCase.expectedErr {
t.Errorf("Case: %d, expected: %s, got: %s", i+1, testCase.expectedErr, err)
+2 -2
View File
@@ -300,7 +300,7 @@ func testPostPolicyBucketHandler(obj ObjectLayer, instanceType string, t TestErr
{
objectName: "test",
data: []byte("Hello, World"),
expectedRespStatus: http.StatusBadRequest,
expectedRespStatus: http.StatusForbidden,
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
dates: []interface{}{curTime.Add(-1 * time.Minute * 5).Format(expirationDateFormat), curTime.Format(iso8601DateFormat), curTime.Format(yyyymmdd)},
@@ -499,7 +499,7 @@ func testPostPolicyBucketHandlerRedirect(obj ObjectLayer, instanceType string, t
// postPresignSignatureV4 - presigned signature for PostPolicy requests.
func postPresignSignatureV4(policyBase64 string, t time.Time, secretAccessKey, location string) string {
// Get signining key.
signingkey := getSigningKey(secretAccessKey, t, location)
signingkey := getSigningKey(secretAccessKey, t, location, "s3")
// Calculate signature.
signature := getSignature(signingkey, policyBase64)
return signature
+10 -9
View File
@@ -217,10 +217,10 @@ func checkPolicyCond(op string, input1, input2 string) bool {
// checkPostPolicy - apply policy conditions and validate input values.
// (http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html)
func checkPostPolicy(formValues http.Header, postPolicyForm PostPolicyForm) APIErrorCode {
func checkPostPolicy(formValues http.Header, postPolicyForm PostPolicyForm) error {
// Check if policy document expiry date is still not reached
if !postPolicyForm.Expiration.After(UTCNow()) {
return ErrPolicyAlreadyExpired
return fmt.Errorf("Invalid according to Policy: Policy expired")
}
// Flag to indicate if all policies conditions are satisfied
@@ -237,22 +237,23 @@ func checkPostPolicy(formValues http.Header, postPolicyForm PostPolicyForm) APIE
if startsWithSupported, condFound := startsWithConds[cond]; condFound {
// Check if the current condition supports starts-with operator
if op == policyCondStartsWith && !startsWithSupported {
return ErrAccessDenied
return fmt.Errorf("Invalid according to Policy: Policy Condition failed")
}
// Check if current policy condition is satisfied
condPassed = checkPolicyCond(op, formValues.Get(formCanonicalName), v.Value)
if !condPassed {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed")
}
} else {
// This covers all conditions X-Amz-Meta-* and X-Amz-*
if strings.HasPrefix(cond, "$x-amz-meta-") || strings.HasPrefix(cond, "$x-amz-") {
// Check if policy condition is satisfied
condPassed = checkPolicyCond(op, formValues.Get(formCanonicalName), v.Value)
if !condPassed {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed: [%s, %s, %s]", op, cond, v.Value)
}
}
}
// Check if current policy condition is satisfied, quit immediately otherwise
if !condPassed {
return ErrAccessDenied
}
}
return ErrNone
return nil
}
+15 -11
View File
@@ -18,6 +18,7 @@ package cmd
import (
"encoding/base64"
"fmt"
"net/http"
"testing"
@@ -45,26 +46,28 @@ func TestPostPolicyForm(t *testing.T) {
SuccessActionStatus string
Policy string
Expired bool
ErrCode APIErrorCode
expectedErr error
}
testCases := []testCase{
// Everything is fine with this test
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", ErrCode: ErrNone},
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: nil},
// Expired policy document
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", Expired: true, ErrCode: ErrPolicyAlreadyExpired},
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", Expired: true, expectedErr: fmt.Errorf("Invalid according to Policy: Policy expired")},
// Different AMZ date
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", ErrCode: ErrAccessDenied},
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "2017T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Key which doesn't start with user/user1/filename
{Bucket: "testbucket", Key: "myfile.txt", XAmzDate: "20160727T000000Z", XAmzMetaUUID: "14365123651274", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", ErrCode: ErrAccessDenied},
{Bucket: "testbucket", Key: "myfile.txt", XAmzDate: "20160727T000000Z", XAmzMetaUUID: "14365123651274", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect bucket name.
{Bucket: "incorrect", Key: "user/user1/filename/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", ErrCode: ErrAccessDenied},
{Bucket: "incorrect", Key: "user/user1/filename/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect key name
{Bucket: "testbucket", Key: "incorrect", XAmzDate: "20160727T000000Z", XAmzMetaUUID: "14365123651274", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", ErrCode: ErrAccessDenied},
{Bucket: "testbucket", Key: "incorrect", XAmzDate: "20160727T000000Z", XAmzMetaUUID: "14365123651274", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect date
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "incorrect", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", ErrCode: ErrAccessDenied},
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "incorrect", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect ContentType
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "incorrect", ErrCode: ErrAccessDenied},
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "14365123651274", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "incorrect", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed")},
// Incorrect Metadata
{Bucket: "testbucket", Key: "user/user1/filename/${filename}/myfile.txt", XAmzMetaUUID: "151274", SuccessActionStatus: "201", XAmzCredential: "KVGKMDUQ23TCZXTLTHLP/20160727/us-east-1/s3/aws4_request", XAmzDate: "20160727T000000Z", XAmzAlgorithm: "AWS4-HMAC-SHA256", ContentType: "image/jpeg", expectedErr: fmt.Errorf("Invalid according to Policy: Policy Condition failed: [eq, $x-amz-meta-uuid, 14365123651274]")},
}
// Validate all the test cases.
for i, tt := range testCases {
@@ -96,8 +99,9 @@ func TestPostPolicyForm(t *testing.T) {
t.Fatal(err)
}
if errCode := checkPostPolicy(formValues, postPolicyForm); tt.ErrCode != errCode {
t.Fatalf("Test %d:, Expected %d, got %d", i+1, tt.ErrCode, errCode)
err = checkPostPolicy(formValues, postPolicyForm)
if err != nil && tt.expectedErr != nil && err.Error() != tt.expectedErr.Error() {
t.Fatalf("Test %d:, Expected %s, got %s", i+1, tt.expectedErr.Error(), err.Error())
}
}
}
+3 -2
View File
@@ -38,11 +38,12 @@ func registerDistXLRouters(router *mux.Router, endpoints EndpointList) {
// Register storage rpc router only if its a distributed setup.
registerStorageRESTHandlers(router, endpoints)
// Register peer REST router only if its a distributed setup.
registerPeerRESTHandlers(router)
// Register distributed namespace lock.
registerDistNSLockRouter(router)
// Register peer communication router.
registerPeerRPCRouter(router)
}
// List of some generic handlers which are applied for all incoming requests.
+49 -53
View File
@@ -1658,62 +1658,58 @@ func (s *TestSuiteCommon) TestListObjectsHandler(c *check) {
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
buffer1 := bytes.NewReader([]byte("Hello World"))
request, err = newTestSignedRequest("PUT", getPutObjectURL(s.endPoint, bucketName, "bar"),
int64(buffer1.Len()), buffer1, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
for _, objectName := range []string{"foo bar 1", "foo bar 2"} {
buffer := bytes.NewReader([]byte("Hello World"))
request, err = newTestSignedRequest("PUT", getPutObjectURL(s.endPoint, bucketName, objectName),
int64(buffer.Len()), buffer, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
client = http.Client{Transport: s.transport}
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
}
// create listObjectsV1 request with valid parameters
request, err = newTestSignedRequest("GET", getListObjectsV1URL(s.endPoint, bucketName, "1000"),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
// execute the HTTP request.
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
var testCases = []struct {
getURL string
expectedStrings []string
}{
{getListObjectsV1URL(s.endPoint, bucketName, "", "1000", ""), []string{"<Key>foo bar 1</Key>", "<Key>foo bar 2</Key>"}},
{getListObjectsV1URL(s.endPoint, bucketName, "", "1000", "url"), []string{"<Key>foo+bar+1</Key>", "<Key>foo+bar+2</Key>"}},
{getListObjectsV2URL(s.endPoint, bucketName, "", "1000", "", ""),
[]string{
"<Key>foo bar 1</Key>",
"<Key>foo bar 2</Key>",
"<Owner><ID></ID><DisplayName></DisplayName></Owner>",
},
},
{getListObjectsV2URL(s.endPoint, bucketName, "", "1000", "true", ""),
[]string{
"<Key>foo bar 1</Key>",
"<Key>foo bar 2</Key>",
fmt.Sprintf("<Owner><ID>%s</ID><DisplayName></DisplayName></Owner>", globalMinioDefaultOwnerID),
},
},
{getListObjectsV2URL(s.endPoint, bucketName, "", "1000", "", "url"), []string{"<Key>foo+bar+1</Key>", "<Key>foo+bar+2</Key>"}},
}
getContent, err := ioutil.ReadAll(response.Body)
c.Assert(err, nil)
c.Assert(strings.Contains(string(getContent), "<Key>bar</Key>"), true)
for _, testCase := range testCases {
// create listObjectsV1 request with valid parameters
request, err = newTestSignedRequest("GET", testCase.getURL, 0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
// execute the HTTP request.
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
// create listObjectsV2 request with valid parameters
request, err = newTestSignedRequest("GET", getListObjectsV2URL(s.endPoint, bucketName, "1000", ""),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
// execute the HTTP request.
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
getContent, err = ioutil.ReadAll(response.Body)
c.Assert(err, nil)
c.Assert(strings.Contains(string(getContent), "<Key>bar</Key>"), true)
c.Assert(strings.Contains(string(getContent), "<Owner><ID></ID><DisplayName></DisplayName></Owner>"), true)
// create listObjectsV2 request with valid parameters and fetch-owner activated
request, err = newTestSignedRequest("GET", getListObjectsV2URL(s.endPoint, bucketName, "1000", "true"),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
// execute the HTTP request.
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
getContent, err = ioutil.ReadAll(response.Body)
c.Assert(err, nil)
c.Assert(strings.Contains(string(getContent), "<Key>bar</Key>"), true)
c.Assert(strings.Contains(string(getContent), fmt.Sprintf("<Owner><ID>%s</ID><DisplayName></DisplayName></Owner>",
globalMinioDefaultOwnerID)), true)
getContent, err := ioutil.ReadAll(response.Body)
c.Assert(err, nil)
for _, expectedStr := range testCase.expectedStrings {
c.Assert(strings.Contains(string(getContent), expectedStr), true)
}
}
}
// TestListObjectsHandlerErrors - Setting invalid parameters to List Objects
@@ -1733,7 +1729,7 @@ func (s *TestSuiteCommon) TestListObjectsHandlerErrors(c *check) {
c.Assert(response.StatusCode, http.StatusOK)
// create listObjectsV1 request with invalid value of max-keys parameter. max-keys is set to -2.
request, err = newTestSignedRequest("GET", getListObjectsV1URL(s.endPoint, bucketName, "-2"),
request, err = newTestSignedRequest("GET", getListObjectsV1URL(s.endPoint, bucketName, "", "-2", ""),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
@@ -1744,7 +1740,7 @@ func (s *TestSuiteCommon) TestListObjectsHandlerErrors(c *check) {
verifyError(c, response, "InvalidArgument", "Argument maxKeys must be an integer between 0 and 2147483647", http.StatusBadRequest)
// create listObjectsV2 request with invalid value of max-keys parameter. max-keys is set to -2.
request, err = newTestSignedRequest("GET", getListObjectsV2URL(s.endPoint, bucketName, "-2", ""),
request, err = newTestSignedRequest("GET", getListObjectsV2URL(s.endPoint, bucketName, "", "-2", "", ""),
0, nil, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
client = http.Client{Transport: s.transport}
+9 -10
View File
@@ -19,15 +19,16 @@ package cmd
import (
"os"
"os/exec"
"syscall"
)
// Type of service signals currently supported.
type serviceSignal int
type serviceSignal string
const (
serviceStatus = iota // Gets status about the service.
serviceRestart // Restarts the service.
serviceStop // Stops the server.
serviceStatus serviceSignal = "serviceStatus" // Gets status about the service.
serviceRestart = "serviceRestart" // Restarts the service.
serviceStop = "serviceStop" // Stops the server.
// Add new service requests here.
)
@@ -39,7 +40,7 @@ var GlobalServiceDoneCh chan struct{}
// Initialize service mutex once.
func init() {
GlobalServiceDoneCh = make(chan struct{}, 1)
GlobalServiceDoneCh = make(chan struct{})
globalServiceSignalCh = make(chan serviceSignal)
}
@@ -56,9 +57,7 @@ func restartProcess() error {
return err
}
// Pass on the environment and replace the old count key with the new one.
cmd := exec.Command(argv0, os.Args[1:]...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Start()
// Invokes the execve system call.
// Re-uses the same pid. This preserves the pid over multiple server-respawns.
return syscall.Exec(argv0, os.Args, os.Environ())
}
+3
View File
@@ -52,6 +52,9 @@ func handleSignals() {
err = globalHTTPServer.Shutdown()
logger.LogIf(context.Background(), err)
// send signal to various go-routines that they need to quit.
close(GlobalServiceDoneCh)
if objAPI := newObjectLayerFn(); objAPI != nil {
oerr = objAPI.Shutdown(context.Background())
logger.LogIf(context.Background(), oerr)
+2 -2
View File
@@ -163,7 +163,7 @@ func doesPresignV2SignatureMatch(r *http.Request) APIErrorCode {
return ErrExpiredPresignRequest
}
encodedResource, err = getResource(encodedResource, r.Host, globalDomainName)
encodedResource, err = getResource(encodedResource, r.Host, globalDomainNames)
if err != nil {
return ErrInvalidRequest
}
@@ -257,7 +257,7 @@ func doesSignV2Match(r *http.Request) APIErrorCode {
return ErrInvalidQueryParams
}
encodedResource, err = getResource(encodedResource, r.Host, globalDomainName)
encodedResource, err = getResource(encodedResource, r.Host, globalDomainNames)
if err != nil {
return ErrInvalidRequest
}
+10 -10
View File
@@ -47,8 +47,8 @@ func (c credentialHeader) getScope() string {
}, "/")
}
func getReqAccessKeyV4(r *http.Request, region string) (auth.Credentials, bool, APIErrorCode) {
ch, err := parseCredentialHeader("Credential="+r.URL.Query().Get("X-Amz-Credential"), region)
func getReqAccessKeyV4(r *http.Request, region string, stype serviceType) (auth.Credentials, bool, APIErrorCode) {
ch, err := parseCredentialHeader("Credential="+r.URL.Query().Get("X-Amz-Credential"), region, stype)
if err != ErrNone {
// Strip off the Algorithm prefix.
v4Auth := strings.TrimPrefix(r.Header.Get("Authorization"), signV4Algorithm)
@@ -56,7 +56,7 @@ func getReqAccessKeyV4(r *http.Request, region string) (auth.Credentials, bool,
if len(authFields) != 3 {
return auth.Credentials{}, false, ErrMissingFields
}
ch, err = parseCredentialHeader(authFields[0], region)
ch, err = parseCredentialHeader(authFields[0], region, stype)
if err != ErrNone {
return auth.Credentials{}, false, err
}
@@ -65,8 +65,8 @@ func getReqAccessKeyV4(r *http.Request, region string) (auth.Credentials, bool,
}
// parse credentialHeader string into its structured form.
func parseCredentialHeader(credElement string, region string) (ch credentialHeader, aec APIErrorCode) {
creds := strings.Split(strings.TrimSpace(credElement), "=")
func parseCredentialHeader(credElement string, region string, stype serviceType) (ch credentialHeader, aec APIErrorCode) {
creds := strings.SplitN(strings.TrimSpace(credElement), "=", 2)
if len(creds) != 2 {
return ch, ErrMissingFields
}
@@ -107,7 +107,7 @@ func parseCredentialHeader(credElement string, region string) (ch credentialHead
return ch, ErrAuthorizationHeaderMalformed
}
if credElements[2] != "s3" {
if credElements[2] != string(stype) {
return ch, ErrInvalidService
}
cred.scope.service = credElements[2]
@@ -185,7 +185,7 @@ func doesV4PresignParamsExist(query url.Values) APIErrorCode {
}
// Parses all the presigned signature values into separate elements.
func parsePreSignV4(query url.Values, region string) (psv preSignValues, aec APIErrorCode) {
func parsePreSignV4(query url.Values, region string, stype serviceType) (psv preSignValues, aec APIErrorCode) {
// verify whether the required query params exist.
err := doesV4PresignParamsExist(query)
if err != ErrNone {
@@ -201,7 +201,7 @@ func parsePreSignV4(query url.Values, region string) (psv preSignValues, aec API
preSignV4Values := preSignValues{}
// Save credential.
preSignV4Values.Credential, err = parseCredentialHeader("Credential="+query.Get("X-Amz-Credential"), region)
preSignV4Values.Credential, err = parseCredentialHeader("Credential="+query.Get("X-Amz-Credential"), region, stype)
if err != ErrNone {
return psv, err
}
@@ -249,7 +249,7 @@ func parsePreSignV4(query url.Values, region string) (psv preSignValues, aec API
// Authorization: algorithm Credential=accessKeyID/credScope, \
// SignedHeaders=signedHeaders, Signature=signature
//
func parseSignV4(v4Auth string, region string) (sv signValues, aec APIErrorCode) {
func parseSignV4(v4Auth string, region string, stype serviceType) (sv signValues, aec APIErrorCode) {
// Replace all spaced strings, some clients can send spaced
// parameters and some won't. So we pro-actively remove any spaces
// to make parsing easier.
@@ -275,7 +275,7 @@ func parseSignV4(v4Auth string, region string) (sv signValues, aec APIErrorCode)
var err APIErrorCode
// Save credentail values.
signV4Values.Credential, err = parseCredentialHeader(authFields[0], region)
signV4Values.Credential, err = parseCredentialHeader(authFields[0], region, stype)
if err != ErrNone {
return sv, err
}
+22 -3
View File
@@ -216,10 +216,29 @@ func TestParseCredentialHeader(t *testing.T) {
"aws4_request"),
expectedErrCode: ErrNone,
},
// Test Case - 11.
// Test case with right inputs -> AccessKey contains `=`. See minio/#7376
// "aws4_request" is the valid request version.
{
inputCredentialStr: generateCredentialStr(
"LOCALKEY/DEV/1=",
sampleTimeStr,
"us-west-1",
"s3",
"aws4_request"),
expectedCredentials: generateCredentials(
t,
"LOCALKEY/DEV/1=",
sampleTimeStr,
"us-west-1",
"s3",
"aws4_request"),
expectedErrCode: ErrNone,
},
}
for i, testCase := range testCases {
actualCredential, actualErrCode := parseCredentialHeader(testCase.inputCredentialStr, "us-west-1")
actualCredential, actualErrCode := parseCredentialHeader(testCase.inputCredentialStr, "us-west-1", "s3")
// validating the credential fields.
if testCase.expectedErrCode != actualErrCode {
t.Fatalf("Test %d: Expected the APIErrCode to be %s, got %s", i+1, errorCodes[testCase.expectedErrCode].Code, errorCodes[actualErrCode].Code)
@@ -446,7 +465,7 @@ func TestParseSignV4(t *testing.T) {
}
for i, testCase := range testCases {
parsedAuthField, actualErrCode := parseSignV4(testCase.inputV4AuthStr, "")
parsedAuthField, actualErrCode := parseSignV4(testCase.inputV4AuthStr, "", "s3")
if testCase.expectedErrCode != actualErrCode {
t.Fatalf("Test %d: Expected the APIErrCode to be %d, got %d", i+1, testCase.expectedErrCode, actualErrCode)
@@ -813,7 +832,7 @@ func TestParsePreSignV4(t *testing.T) {
inputQuery.Set(testCase.inputQueryKeyVals[j], testCase.inputQueryKeyVals[j+1])
}
// call the function under test.
parsedPreSign, actualErrCode := parsePreSignV4(inputQuery, "")
parsedPreSign, actualErrCode := parsePreSignV4(inputQuery, "", serviceS3)
if testCase.expectedErrCode != actualErrCode {
t.Fatalf("Test %d: Expected the APIErrCode to be %d, got %d", i+1, testCase.expectedErrCode, actualErrCode)
}
+17 -1
View File
@@ -17,11 +17,16 @@
package cmd
import (
"bytes"
"context"
"crypto/hmac"
"encoding/hex"
"io/ioutil"
"net/http"
"strconv"
"strings"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/sha256-simd"
)
@@ -53,7 +58,18 @@ func skipContentSha256Cksum(r *http.Request) bool {
}
// Returns SHA256 for calculating canonical-request.
func getContentSha256Cksum(r *http.Request) string {
func getContentSha256Cksum(r *http.Request, stype serviceType) string {
if stype == serviceSTS {
payload, err := ioutil.ReadAll(r.Body)
if err != nil {
logger.CriticalIf(context.Background(), err)
}
sum256 := sha256.New()
sum256.Write(payload)
r.Body = ioutil.NopCloser(bytes.NewReader(payload))
return hex.EncodeToString(sum256.Sum(nil))
}
var (
defaultSha256Cksum string
v []string
+1 -1
View File
@@ -248,7 +248,7 @@ func TestGetContentSha256Cksum(t *testing.T) {
if testCase.h != "" {
r.Header.Set("x-amz-content-sha256", testCase.h)
}
got := getContentSha256Cksum(r)
got := getContentSha256Cksum(r, serviceS3)
if got != testCase.expected {
t.Errorf("Test %d: got:%s expected:%s", i+1, got, testCase.expected)
}

Some files were not shown because too many files have changed in this diff Show More