Compare commits

...

61 Commits

Author SHA1 Message Date
kannappanr 3e3fbdf8e6 Remove file added inadvertently (#7968) 2019-07-23 18:51:54 -07:00
Praveen raj Mani c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani 2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Daryl Finlay 9389a55e5d Cancel PutObjectPart on upload abort (#7940)
Calling ListMultipartUploads fails if an upload is aborted while a
part is being uploaded because the directory for the upload exists
(since fsRenameFile ends up calling os.MkdirAll) but the meta JSON file
doesn't. To fix this we make sure an upload hasn't been aborted during
PutObjectPart by checking the existence of the directory for the upload
while moving the temporary part file into it.
2019-07-22 22:36:15 -07:00
Harshavardhana 87e6533cf3 Add some design docs for distributed setup (#7950) 2019-07-23 07:48:10 +05:30
Christian Muehlhaeuser 38bc3a45db Fixed tautological conditions (#7959)
We already check for err being equal to nil above, no need
to check again.
2019-07-22 17:06:08 -07:00
Christian Muehlhaeuser c5faba55c1 Comment: Typo Fix (#7958) 2019-07-21 05:55:09 +01:00
Harshavardhana 8b5e6e338c Fix: Only add SRV records that match the bucket name exactly (#7957)
Problem: MinIO incorrectly appends DNS SRV records of buckets that have a prefix match with a given bucket. E.g bucket1 would incorrectly get bucket's DNS records too.
Solution: This fix ensures that we only add SRV records that match the key exactly
2019-07-20 11:29:05 +01:00
poornas 0373a1699b Add error filter to admin trace API (#7923)
This allows MinIO to have the ability to send back only error trace
2019-07-20 01:38:26 +01:00
Krishnan Parthasarathi 559a59220e Add initial support for bucket lifecycle (#7563)
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)

N B the code for actual expiration of objects will be included in a
subsequent PR.
2019-07-19 21:20:33 +01:00
Yao Zongyou 59e1763816 doc: use make instead of go command to test changes (#7951) 2019-07-19 14:40:39 +01:00
poornas 041a812ba0 trace api: add call stats to trace (#7915)
Stats such as call latency, bytes received and sent have been added
2019-07-18 23:29:17 +01:00
Krishnan Parthasarathi fbfc9a61ec Add node address information to logs (#7941) 2019-07-18 09:58:37 -07:00
Philipp Dallig be9baa1464 Fix startup without MINIO_USERNAME and MINIO_GROUPNAME (#7944) 2019-07-18 21:49:49 +05:30
Minio Trusted b058e32348 Update yaml files to latest version RELEASE.2019-07-17T22-54-12Z 2019-07-17 22:59:33 +00:00
Lucas ea66a52ed1 Add KMS master key from Docker secret (#7825) 2019-07-17 20:55:26 +01:00
Harshavardhana 55dd017e62 Deprecate auto detection of container user (#7930)
There is no reliable way to handle fallbacks for
MinIO deployments, due to various command line
options and multiple locations which require
access inside container.

Parsing command line options is tricky to figure
out which is the backend disk etc, we did try
to fix this in implementations of check-user.go
but it wasn't complete and introduced more bugs.

This PR simplifies the entire approach to rather
than running Docker container as non-root by default
always, it allows users to opt-in. Such that they
are aware that that is what they are planning to do.

In-fact there are other ways docker containers can
be run as regular users, without modifying our
internal behavior and adding more complexities.
2019-07-17 19:20:55 +01:00
Kanagaraj M 12353caf35 Fix: Support Unicode delimiters in s3 select (#7931) 2019-07-17 19:10:17 +01:00
Harshavardhana a57c747667 Document vault in prod mode instead of dev mode (#7928) 2019-07-16 01:32:15 +01:00
Anis Elleuch 28661c0413 heal: Trigger auto-heal once each month instead of 24 hours (#7934) 2019-07-16 00:03:42 +01:00
Harshavardhana 04a152be12 Redirect to browser only if browser is enabled (#7914) 2019-07-15 20:01:17 +01:00
Harshavardhana bce3f8237d Allow users to give anonymous access (#7926)
Current code already allows users to GetPolicy/SetPolicy
there was a missing code in ListAllBucketPolicies to allow
access, this fixes this behavior.

Fixes #7913
2019-07-15 20:00:41 +01:00
Harshavardhana 16a45e5aff Fix dynamic help vars for sub-commands (#7925)
The fix in #7646 introduced a regression which
was left unnoticed, the fix didn't work for
sub-commands unfortunately. This fixes it
by moving v1.21.0 version of the minio/cli
package.

Fixes #7924
2019-07-12 23:32:27 -07:00
Anis Elleuch 000a60f238 xl: Heal empty parts (#7860)
posix.VerifyFile() doesn't know how to check if a file
is corrupted if that file is empty. We do have the part
size in xl.json so we pass it to VerifyFile to return
an error so healing empty parts can work properly.
2019-07-13 00:29:44 +01:00
Praveen raj Mani bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Ashish Kumar Sinha 97f2bc26b9 Add validations for object name length and prefix (#7746)
fixes #7717
2019-07-12 10:08:12 +05:30
Praveen raj Mani bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
dependabot[bot] 2337e5f803 Bump lodash from 4.17.4 to 4.17.14 in /browser (#7912)
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.4 to 4.17.14.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.4...4.17.14)

Signed-off-by: dependabot[bot] <support@github.com>
2019-07-11 16:44:47 -07:00
Krishnan Parthasarathi ffd7b7059c Pass on web-handler arguments properly to log entries (#7894) 2019-07-11 22:37:13 +01:00
Harshavardhana 5c0acbc6fc Add text/event-stream for long running http connections (#7909)
When MinIO is behind a proxy, proxies end up killing
clients when no data is seen on the connection, adding
the right content-type ensures that proxies do not come
in the way.
2019-07-11 13:19:25 -07:00
Harshavardhana 5a52bc7ff6 Fix mint hub.docker.com builds (#7908) 2019-07-11 11:45:57 -07:00
poornas 045e1fed2b Fix dotnet tests build to be project structure agnostic (#7906) 2019-07-11 12:51:30 +05:30
dependabot[bot] a861d38532 Bump lodash-es from 4.17.4 to 4.17.14 in /browser (#7910)
Bumps [lodash-es](https://github.com/lodash/lodash) from 4.17.4 to 4.17.14.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.4...4.17.14)

Signed-off-by: dependabot[bot] <support@github.com>
2019-07-11 10:55:13 +05:30
poornas 20a15567b8 Fix atime support check for disk cache (#7891)
- add a sleep between Stat operations to
accurately detect atime
2019-07-10 23:41:11 +01:00
Krishnan Parthasarathi 94f67ad224 Log error response even if a handler doesn't logBody (#7867) 2019-07-10 11:49:02 -07:00
ebozduman 36ee110563 Regression fix to bring back checkPolicyCond function call (#7897)
Fixes #7895
2019-07-10 10:48:43 +05:30
Harshavardhana 1dc25bcf5f Add mint tests into MinIO repo (#7886) 2019-07-09 18:32:39 -07:00
Minio Trusted 2d96745156 Update yaml files to latest version RELEASE.2019-07-10T00-34-56Z 2019-07-10 00:39:45 +00:00
mzukowski-reef 9d49688c87 Switch to kurin/blazer from minio/blazer fork for b2 gateway (#7879) 2019-07-09 08:14:02 -07:00
Anis Elleuch 8e09374cb8 Avoid go-prompt to show colored prompt properly in Windows (#7890)
Update prompt shows some weird characters under Windows, the reason
is that go-prompt is used to show a yes/no prompt, since go-prompt
does not seem to have a way to support color/fatih, this PR will
implements its own yes/no prompt with the correct text coloration.
2019-07-09 01:46:04 +01:00
Krishna Srinivas 58d90ed73c Avoid network transfer for bitrot verification during healing (#7375) 2019-07-08 13:51:18 -07:00
Anis Elleuch e857b6741d Add one log in health checker liveness code (#7861) 2019-07-06 16:38:39 -07:00
poornas 0505ef83b5 Fix host address returned in admin API calls (#7846) 2019-07-05 20:41:35 -07:00
Minio Trusted 22bc15d89b Update yaml files to latest version RELEASE.2019-07-05T21-20-21Z 2019-07-05 21:24:43 +00:00
Krishna Srinivas a2e904b966 Support any string as delimiter for listing (#7882) 2019-07-05 14:06:12 -07:00
Kaan Kabalak cc7dc61eb4 Allow folders inside buckets to be opened in a new tab (#7840)
Fixes #7836
2019-07-05 13:21:06 -07:00
Yao Zongyou c4f480a839 fix csv read bug (#7885) 2019-07-05 12:08:56 -07:00
Yao Zongyou 60831e3299 aggregation functions' argument may already has been cast to numeric (#7876) 2019-07-05 10:38:38 -07:00
Yao Zongyou 037319066f fix unicode support related bugs in s3select (#7877) 2019-07-05 09:43:10 -07:00
Praveen raj Mani bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
Harshavardhana 0ebbd3caef Avoid chown instead fallback to rootpath for user perms (#7874)
Fixes #7864
2019-07-03 18:57:34 -07:00
Ryan Tam bd56f80250 Fix ignored alias for aggregate result in S3 Select (#7849)
The SQL parser as it stands right now ignores alias for aggregate
result, e.g. `SELECT COUNT(*) AS thing FROM s3object` doesn't actually
return record like `{"thing": 42}`, it returns a record like `{"_1": 42}`.
Column alias for aggregate result is supported in AWS's S3 Select, so
this commit fixes that by respecting the `expr.As` in the expression.

Also improve test for S3 select

On top of testing a simple `SELECT` query, we want to test a few more
"advanced" queries (e.g. aggregation).

Convert existing tests into table driven tests[1], and add the new test
cases with "advanced" queries into them.

[1] - https://github.com/golang/go/wiki/TableDrivenTests
2019-07-03 16:34:54 -07:00
iliul a39e810965 docs: Fix dead link of HighwayHash (#7847)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-07-03 14:32:58 -07:00
mizuno-keyence 09103991ea [Bugfix] duplicating flag registration (#7853) 2019-07-03 14:31:19 -07:00
Harshavardhana c43f745449 Ensure that we use constants everywhere (#7845)
This allows for canonicalization of the strings
throughout our code and provides a common space
for all these constants to reside.

This list is rather non-exhaustive but captures
all the headers used in AWS S3 API operations
2019-07-02 22:34:32 -07:00
Anis Elleuch 9610a74c19 auto-heal: Use fast scan instead of the deep one (#7868) 2019-07-02 18:53:08 -07:00
Matthew Wegner 0bcd8abc5c doc: "admin user policy" command typo (#7865)
Under "change user policy", the `mc admin set-policy` command is wrong.  It should be `mc admin user set-policy`.
2019-07-02 11:48:26 -07:00
kannappanr 70b350c383 Remove DeploymentID from response headers (#7815)
Response headers need not contain deployment ID.
2019-07-01 12:22:01 -07:00
Krishna Srinivas 338e9a9be9 Put object client disconnect (#7824)
Fail putObject  and postpolicy in case client prematurely disconnects
Use request's context to cancel lock requests on client disconnects
2019-06-28 22:09:17 -07:00
Krishnan Parthasarathi edbd8709ec Simplify PR template to ease new contributors' workflow (#7844) 2019-06-27 15:47:46 -07:00
Minio Trusted 5db60a6c59 Update yaml files to latest version RELEASE.2019-06-27T21-13-50Z 2019-06-27 21:18:52 +00:00
245 changed files with 13010 additions and 1379 deletions
+7 -21
View File
@@ -1,33 +1,19 @@
<!--- Provide a general summary of your changes in the Title above -->
## Description
<!--- Describe your changes in detail -->
## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
## Regression
<!-- Is this PR fixing a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
## How to test this PR?
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have added unit tests to cover my changes.
- [ ] I have added/updated functional tests in [mint](https://github.com/minio/mint). (If yes, add `mint` PR # here: )
- [ ] All new and existing tests passed.
- [ ] Fixes a regression (If yes, please add `commit-id` or `PR #` here)
- [ ] Documentation needed
- [ ] Unit tests needed
- [ ] Functional tests needed (If yes, add [mint](https://github.com/minio/mint) PR # here: )
+1 -3
View File
@@ -9,7 +9,6 @@ site/
/.idea/
/Minio.iml
**/access.log
build/
vendor/**/*.js
vendor/**/*.json
release
@@ -23,5 +22,4 @@ prime/
stage/
.sia_temp/
config.json
healthcheck
check-user
dockerscripts/healthcheck
+12 -2
View File
@@ -2,6 +2,14 @@ go_import_path: github.com/minio/minio
language: go
addons:
apt:
packages:
- shellcheck
services:
- docker
# this ensures PRs based on a local branch are not built twice
# the downside is that a PR targeting a different branch is not built
# but as a workaround you can add the branch to this list
@@ -26,7 +34,7 @@ matrix:
- $HOME/gopath/pkg/mod
- $HOME/go/pkg/mod
go: 1.12.1
go: 1.12.5
script:
- make
- diff -au <(gofmt -s -d cmd) <(printf "")
@@ -37,12 +45,14 @@ matrix:
- make verify
- make coverage
- cd browser && yarn && yarn test && cd ..
- bash -c 'shopt -s globstar; shellcheck mint/**/*.sh'
- os: windows
env:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
go: 1.12.1
go: 1.12.5
script:
- go build --ldflags="$(go run buildscripts/gen-ldflags.go)" -o %GOPATH%\bin\minio.exe
- bash buildscripts/go-coverage.sh
+1 -1
View File
@@ -37,7 +37,7 @@ After your code changes, make sure
- To add test cases for the new code. If you have questions about how to do it, please ask on our [Slack](https://slack.min.io) channel.
- To run `make verifiers`
- To squash your commits into a single commit. `git rebase -i`. It's okay to force update your pull request.
- To run `go test -race ./...` and `go build` completes.
- To run `make test` and `make build` completes.
### Commit changes
After verification, commit your changes. This is a [great post](https://chris.beams.io/posts/git-commit/) on how to write useful commit messages
+3 -4
View File
@@ -10,20 +10,19 @@ RUN \
apk add --no-cache git && \
git clone https://github.com/minio/minio && cd minio && \
go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
cd dockerscripts; go build -tags kqueue -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go && \
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/check-user check-user.go
cd dockerscripts; go build -tags kqueue -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go
FROM alpine:3.9
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
MINIO_SECRET_KEY_FILE=secret_key \
MINIO_SSE_MASTER_KEY_FILE=sse_master_key
EXPOSE 9000
COPY --from=0 /go/bin/minio /usr/bin/minio
COPY --from=0 /usr/bin/healthcheck /usr/bin/healthcheck
COPY --from=0 /usr/bin/check-user /usr/bin/check-user
COPY dockerscripts/docker-entrypoint.sh /usr/bin/
RUN \
+4 -4
View File
@@ -2,20 +2,20 @@ FROM alpine:3.9
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY dockerscripts/docker-entrypoint.sh dockerscripts/healthcheck dockerscripts/check-user /usr/bin/
COPY dockerscripts/docker-entrypoint.sh dockerscripts/healthcheck /usr/bin/
COPY minio /usr/bin/
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
MINIO_SECRET_KEY_FILE=secret_key \
MINIO_SSE_MASTER_KEY_FILE=sse_master_key
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' 'su-exec>=0.2' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck && \
chmod +x /usr/bin/check-user
chmod +x /usr/bin/healthcheck
EXPOSE 9000
+23
View File
@@ -0,0 +1,23 @@
FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
ENV GOROOT /usr/local/go
ENV GOPATH /usr/local
ENV PATH $GOPATH/bin:$GOROOT/bin:$PATH
ENV MINT_ROOT_DIR /mint
COPY mint /mint
RUN apt-get --yes update && apt-get --yes upgrade && \
apt-get --yes --quiet install wget jq curl git dnsmasq && \
cd /mint && /mint/release.sh
WORKDIR /mint
ENTRYPOINT ["/mint/entrypoint.sh"]
+4 -6
View File
@@ -7,20 +7,19 @@ ENV GO111MODULE on
RUN \
apk add --no-cache git && \
git clone https://github.com/minio/minio && cd minio/dockerscripts && \
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go && \
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/check-user check-user.go
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go
FROM alpine:3.9
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY --from=0 /usr/bin/healthcheck /usr/bin/healthcheck
COPY --from=0 /usr/bin/check-user /usr/bin/check-user
COPY dockerscripts/docker-entrypoint.sh /usr/bin/
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
MINIO_SECRET_KEY_FILE=secret_key \
MINIO_SSE_MASTER_KEY_FILE=sse_master_key
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' 'su-exec>=0.2' && \
@@ -28,8 +27,7 @@ RUN \
curl https://dl.min.io/server/minio/release/linux-amd64/minio > /usr/bin/minio && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck && \
chmod +x /usr/bin/check-user
chmod +x /usr/bin/healthcheck
EXPOSE 9000
+5 -3
View File
@@ -33,7 +33,7 @@ USER ci
RUN make
RUN bash -c 'diff -au <(gofmt -s -d cmd) <(printf "")'
RUN bash -c 'diff -au <(gofmt -s -d pkg) <(printf "")'
RUN for d in $(go list ./... | grep -v browser); do go test -v -race --timeout 15m "$d"; done
RUN for d in $(go list ./... | grep -v browser); do go test -v -race --timeout 20m "$d"; done
RUN make verifiers
RUN make crosscompile
RUN make verify
@@ -56,15 +56,17 @@ FROM ubuntu:16.04
COPY --from=0 /go/src/github.com/minio/minio/minio /usr/bin/minio
COPY buildscripts/gateway-tests.sh /usr/bin/gateway-tests.sh
COPY mint /mint
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
ENV GOROOT /usr/local/go
ENV GOPATH /usr/local
ENV PATH $GOPATH/bin:$GOROOT/bin:$PATH
ENV MINT_ROOT_DIR /mint
RUN apt-get --yes update && apt-get --yes upgrade && apt-get --yes --quiet install wget jq curl git dnsmasq && \
git clone https://github.com/minio/mint.git /mint && \
RUN apt-get --yes update && apt-get --yes upgrade && \
apt-get --yes --quiet install wget jq curl git dnsmasq && \
cd /mint && /mint/release.sh
WORKDIR /mint
-1
View File
@@ -73,7 +73,6 @@ build: checks
@echo "Building minio binary to './minio'"
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio 1>/dev/null
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go 1>/dev/null
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/check-user $(PWD)/dockerscripts/check-user.go 1>/dev/null
docker: build
@docker build -t $(TAG) . -f Dockerfile.dev
+1 -1
View File
@@ -51,7 +51,7 @@ export const ObjectItem = ({
</div>
<div className="fesl-item fesl-item-name">
<a
href="#"
href={getDataType(name, contentType) === "folder" ? name : "#"}
onClick={e => {
e.preventDefault()
if (onClick) {
+6 -26
View File
@@ -5817,12 +5817,6 @@
"integrity": "sha512-s5kLOcnH0XqDO+FvuaLX8DDjZ18CGFk7VygH40QoKPUQhW4e2rvM0rwUq0t8IQDOwYSeLK01U90OjzBTme2QqA==",
"dev": true
},
"lodash": {
"version": "4.17.11",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz",
"integrity": "sha512-cQKh8igo5QUhZ7lg38DYWAxMvjSAKG0A8wGSVimP07SIUEK2UO+arSRKbRZWtelMtN5V0Hkwh5ryOto/SshYIg==",
"dev": true
},
"micromatch": {
"version": "3.1.10",
"resolved": "https://registry.npmjs.org/micromatch/-/micromatch-3.1.10.tgz",
@@ -6408,12 +6402,6 @@
"requires": {
"lodash": "^4.17.11"
}
},
"lodash": {
"version": "4.17.11",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz",
"integrity": "sha512-cQKh8igo5QUhZ7lg38DYWAxMvjSAKG0A8wGSVimP07SIUEK2UO+arSRKbRZWtelMtN5V0Hkwh5ryOto/SshYIg==",
"dev": true
}
}
},
@@ -7645,14 +7633,14 @@
}
},
"lodash": {
"version": "4.17.11",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz",
"integrity": "sha512-cQKh8igo5QUhZ7lg38DYWAxMvjSAKG0A8wGSVimP07SIUEK2UO+arSRKbRZWtelMtN5V0Hkwh5ryOto/SshYIg=="
"version": "4.17.14",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.14.tgz",
"integrity": "sha512-mmKYbW3GLuJeX+iGP+Y7Gp1AiGHGbXHCOh/jZmrawMmsE7MS4znI3RL2FsjbqOyMayHInjOeykW7PEajUk1/xw=="
},
"lodash-es": {
"version": "4.17.11",
"resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.17.11.tgz",
"integrity": "sha512-DHb1ub+rMjjrxqlB3H56/6MXtm1lSksDp2rA2cNWjG8mlDUYFhUj3Di2Zn5IwSU87xLv8tNIQ7sSwE/YOX/D/Q=="
"version": "4.17.14",
"resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.17.14.tgz",
"integrity": "sha512-7zchRrGa8UZXjD/4ivUWP1867jDkhzTG2c/uj739utSd7O/pFFdxspCemIFKEEjErbcqRzn8nKnGsi7mvTgRPA=="
},
"lodash._baseisequal": {
"version": "3.0.7",
@@ -11078,14 +11066,6 @@
"dev": true,
"requires": {
"lodash": "^4.17.11"
},
"dependencies": {
"lodash": {
"version": "4.17.11",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz",
"integrity": "sha512-cQKh8igo5QUhZ7lg38DYWAxMvjSAKG0A8wGSVimP07SIUEK2UO+arSRKbRZWtelMtN5V0Hkwh5ryOto/SshYIg==",
"dev": true
}
}
},
"request-promise-native": {
+25 -25
View File
File diff suppressed because one or more lines are too long
+7 -10
View File
@@ -5075,8 +5075,9 @@ locate-path@^3.0.0:
path-exists "^3.0.0"
lodash-es@^4.2.0, lodash-es@^4.2.1:
version "4.17.4"
resolved "https://registry.yarnpkg.com/lodash-es/-/lodash-es-4.17.4.tgz#dcc1d7552e150a0640073ba9cb31d70f032950e7"
version "4.17.14"
resolved "https://registry.yarnpkg.com/lodash-es/-/lodash-es-4.17.14.tgz#12a95a963cc5955683cee3b74e85458954f37ecc"
integrity sha512-7zchRrGa8UZXjD/4ivUWP1867jDkhzTG2c/uj739utSd7O/pFFdxspCemIFKEEjErbcqRzn8nKnGsi7mvTgRPA==
lodash._baseisequal@^3.0.0:
version "3.0.7"
@@ -5185,14 +5186,10 @@ lodash.words@^3.0.0:
dependencies:
lodash._root "^3.0.0"
lodash@^4.13.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.17.4, lodash@^4.2.0, lodash@^4.2.1:
version "4.17.4"
resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.4.tgz#78203a4d1c328ae1d86dca6460e369b57f4055ae"
lodash@^4.17.3, lodash@^4.17.5, lodash@^4.3.0:
version "4.17.11"
resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.11.tgz#b39ea6229ef607ecd89e2c8df12536891cac9b8d"
integrity sha512-cQKh8igo5QUhZ7lg38DYWAxMvjSAKG0A8wGSVimP07SIUEK2UO+arSRKbRZWtelMtN5V0Hkwh5ryOto/SshYIg==
lodash@^4.13.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.17.3, lodash@^4.17.4, lodash@^4.17.5, lodash@^4.2.0, lodash@^4.2.1, lodash@^4.3.0:
version "4.17.14"
resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.14.tgz#9ce487ae66c96254fe20b599f21b6816028078ba"
integrity sha512-mmKYbW3GLuJeX+iGP+Y7Gp1AiGHGbXHCOh/jZmrawMmsE7MS4znI3RL2FsjbqOyMayHInjOeykW7PEajUk1/xw==
loglevel@^1.4.1:
version "1.6.1"
+22 -27
View File
@@ -34,6 +34,7 @@ import (
"github.com/tidwall/gjson"
"github.com/tidwall/sjson"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/cpu"
"github.com/minio/minio/pkg/disk"
@@ -241,17 +242,11 @@ func (a adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Reque
return
}
thisAddr, err := xnet.ParseHost(GetLocalPeer(globalEndpoints))
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
serverInfo := globalNotificationSys.ServerInfo(ctx)
// Once we have received all the ServerInfo from peers
// add the local peer server info as well.
serverInfo = append(serverInfo, ServerInfo{
Addr: thisAddr.String(),
Addr: getHostName(r),
Data: &ServerInfoData{
StorageInfo: objectAPI.StorageInfo(ctx),
ConnStats: globalConnStats.toServerConnStats(),
@@ -329,7 +324,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
return
}
// Get drive performance details from local server's drive(s)
dp := localEndpointsDrivePerf(globalEndpoints)
dp := localEndpointsDrivePerf(globalEndpoints, r)
// Notify all other MinIO peers to report drive performance numbers
dps := globalNotificationSys.DrivePerfInfo()
@@ -347,7 +342,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
writeSuccessResponseJSON(w, jsonBytes)
case "cpu":
// Get CPU load details from local server's cpu(s)
cpu := localEndpointsCPULoad(globalEndpoints)
cpu := localEndpointsCPULoad(globalEndpoints, r)
// Notify all other MinIO peers to report cpu load numbers
cpus := globalNotificationSys.CPULoadInfo()
cpus = append(cpus, cpu)
@@ -364,7 +359,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
writeSuccessResponseJSON(w, jsonBytes)
case "mem":
// Get mem usage details from local server(s)
m := localEndpointsMemUsage(globalEndpoints)
m := localEndpointsMemUsage(globalEndpoints, r)
// Notify all other MinIO peers to report mem usage numbers
mems := globalNotificationSys.MemUsageInfo()
mems = append(mems, m)
@@ -443,18 +438,12 @@ func (a adminAPIHandlers) TopLocksHandler(w http.ResponseWriter, r *http.Request
return
}
thisAddr, err := xnet.ParseHost(GetLocalPeer(globalEndpoints))
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
peerLocks := globalNotificationSys.GetLocks(ctx)
// Once we have received all the locks currently used from peers
// add the local peer locks list as well.
localLocks := globalLockServer.ll.DupLockMap()
peerLocks = append(peerLocks, &PeerLocks{
Addr: thisAddr.String(),
Addr: getHostName(r),
Locks: localLocks,
})
@@ -682,12 +671,12 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
// Start writing response to client
started = true
setCommonHeaders(w)
w.Header().Set("Content-Type", string(mimeJSON))
w.Header().Set(xhttp.ContentType, "text/event-stream")
// Set 200 OK status
w.WriteHeader(200)
}
// Send whitespace and keep connection open
w.Write([]byte("\n\r"))
w.Write([]byte(" "))
w.(http.Flusher).Flush()
case hr := <-respCh:
switch hr.apiErr {
@@ -702,20 +691,20 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
var errorRespJSON []byte
if hr.errBody == "" {
errorRespJSON = encodeResponseJSON(getAPIErrorResponse(ctx, hr.apiErr,
r.URL.Path, w.Header().Get(responseRequestIDKey),
w.Header().Get(responseDeploymentIDKey)))
r.URL.Path, w.Header().Get(xhttp.AmzRequestID),
globalDeploymentID))
} else {
errorRespJSON = encodeResponseJSON(APIErrorResponse{
Code: hr.apiErr.Code,
Message: hr.errBody,
Resource: r.URL.Path,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
})
}
if !started {
setCommonHeaders(w)
w.Header().Set("Content-Type", string(mimeJSON))
w.Header().Set(xhttp.ContentType, string(mimeJSON))
w.WriteHeader(hr.apiErr.HTTPStatusCode)
}
w.Write(errorRespJSON)
@@ -1476,6 +1465,7 @@ func (a adminAPIHandlers) SetConfigKeysHandler(w http.ResponseWriter, r *http.Re
func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "HTTPTrace")
trcAll := r.URL.Query().Get("all") == "true"
trcErr := r.URL.Query().Get("err") == "true"
// Validate request signature.
adminAPIErr := checkAdminRequestAuthType(ctx, r, "")
@@ -1487,7 +1477,8 @@ func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
// Avoid reusing tcp connection if read timeout is hit
// This is needed to make r.Context().Done() work as
// expected in case of read timeout
w.Header().Add("Connection", "close")
w.Header().Set(xhttp.Connection, "close")
w.Header().Set(xhttp.ContentType, "text/event-stream")
doneCh := make(chan struct{})
defer close(doneCh)
@@ -1497,11 +1488,15 @@ func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
traceCh := make(chan interface{}, 4000)
filter := func(entry interface{}) bool {
trcInfo := entry.(trace.Info)
if trcErr && isHTTPStatusOK(trcInfo.RespInfo.StatusCode) {
return false
}
if trcAll {
return true
}
trcInfo := entry.(trace.Info)
return !strings.HasPrefix(trcInfo.ReqInfo.Path, minioReservedBucketPath)
}
remoteHosts := getRemoteHosts(globalEndpoints)
peers, err := getRestClients(remoteHosts)
@@ -1511,7 +1506,7 @@ func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
globalHTTPTrace.Subscribe(traceCh, doneCh, filter)
for _, peer := range peers {
peer.Trace(traceCh, doneCh, trcAll)
peer.Trace(traceCh, doneCh, trcAll, trcErr)
}
enc := json.NewEncoder(w)
+14 -7
View File
@@ -91,7 +91,9 @@ var (
"enable": false,
"format": "namespace",
"url": "",
"index": ""
"index": "",
"queueDir": "",
"queueLimit": 0
}
},
"kafka": {
@@ -150,6 +152,8 @@ var (
"token": "",
"secure": false,
"pingInterval": 0,
"queueDir": "",
"queueLimit": 0,
"streaming": {
"enable": false,
"clusterID": "",
@@ -166,7 +170,9 @@ var (
"tls": {
"enable": false,
"skipVerify": false
}
},
"queueDir": "",
"queueLimit": 0
}
},
"postgresql": {
@@ -188,13 +194,17 @@ var (
"format": "namespace",
"address": "",
"password": "",
"key": ""
"key": "",
"queueDir": "",
"queueLimit": 0
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
"endpoint": "",
"queueDir": "",
"queueLimit": 0
}
}
},
@@ -704,9 +714,6 @@ func TestAdminServerInfo(t *testing.T) {
}
for _, serverInfo := range results {
if len(serverInfo.Addr) == 0 {
t.Error("Expected server address to be non empty")
}
if serverInfo.Error != "" {
t.Errorf("Unexpected error = %v\n", serverInfo.Error)
}
-7
View File
@@ -20,13 +20,6 @@ import (
"encoding/xml"
)
const (
// Response request id.
responseRequestIDKey = "x-amz-request-id"
// Deployment id.
responseDeploymentIDKey = "x-minio-deployment-id"
)
// ObjectIdentifier carries key name for the object to delete.
type ObjectIdentifier struct {
ObjectName string `xml:"Key"`
+24
View File
@@ -92,6 +92,7 @@ const (
ErrMissingRequestBodyError
ErrNoSuchBucket
ErrNoSuchBucketPolicy
ErrNoSuchBucketLifecycle
ErrNoSuchKey
ErrNoSuchUpload
ErrNoSuchVersion
@@ -139,6 +140,7 @@ const (
ErrSlowDown
ErrInvalidPrefixMarker
ErrBadRequest
ErrKeyTooLongError
// Add new error codes here.
// SSE-S3 related API errors
@@ -187,6 +189,7 @@ const (
ErrRequestBodyParse
ErrObjectExistsAsDirectory
ErrInvalidObjectName
ErrInvalidObjectNamePrefixSlash
ErrInvalidResourceName
ErrServerNotInitialized
ErrOperationTimedOut
@@ -465,6 +468,11 @@ var errorCodes = errorCodeMap{
Description: "The bucket policy does not exist",
HTTPStatusCode: http.StatusNotFound,
},
ErrNoSuchBucketLifecycle: {
Code: "NoSuchBucketLifecycle",
Description: "The bucket lifecycle configuration does not exist",
HTTPStatusCode: http.StatusNotFound,
},
ErrNoSuchKey: {
Code: "NoSuchKey",
Description: "The specified key does not exist.",
@@ -682,6 +690,11 @@ var errorCodes = errorCodeMap{
Description: "400 BadRequest",
HTTPStatusCode: http.StatusBadRequest,
},
ErrKeyTooLongError: {
Code: "KeyTooLongError",
Description: "Your key is too long",
HTTPStatusCode: http.StatusBadRequest,
},
// FIXME: Actual XML error response also contains the header which missed in list of signed header parameters.
ErrUnsignedHeaders: {
@@ -885,6 +898,11 @@ var errorCodes = errorCodeMap{
Description: "Object name contains unsupported characters.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidObjectNamePrefixSlash: {
Code: "XMinioInvalidObjectName",
Description: "Object name contains a leading slash.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidResourceName: {
Code: "XMinioInvalidResourceName",
Description: "Resource name contains bad components such as \"..\" or \".\".",
@@ -1579,6 +1597,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrMethodNotAllowed
case ObjectNameInvalid:
apiErr = ErrInvalidObjectName
case ObjectNamePrefixAsSlash:
apiErr = ErrInvalidObjectNamePrefixSlash
case InvalidUploadID:
apiErr = ErrNoSuchUpload
case InvalidPart:
@@ -1613,6 +1633,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrUnsupportedMetadata
case BucketPolicyNotFound:
apiErr = ErrNoSuchBucketPolicy
case BucketLifecycleNotFound:
apiErr = ErrNoSuchBucketLifecycle
case *event.ErrInvalidEventName:
apiErr = ErrEventNotification
case *event.ErrInvalidARN:
@@ -1639,6 +1661,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrBackendDown
case crypto.Error:
apiErr = ErrObjectTampered
case ObjectNameTooLong:
apiErr = ErrKeyTooLongError
default:
var ie, iw int
// This work-around is to handle the issue golang/go#30648
+11 -10
View File
@@ -26,6 +26,7 @@ import (
"time"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
)
// Returns a hexadecimal representation of time at the
@@ -36,13 +37,13 @@ func mustGetRequestID(t time.Time) string {
// Write http common headers
func setCommonHeaders(w http.ResponseWriter) {
w.Header().Set("Server", "MinIO/"+ReleaseTag)
w.Header().Set(xhttp.ServerInfo, "MinIO/"+ReleaseTag)
// Set `x-amz-bucket-region` only if region is set on the server
// by default minio uses an empty region.
if region := globalServerConfig.GetRegion(); region != "" {
w.Header().Set("X-Amz-Bucket-Region", region)
w.Header().Set(xhttp.AmzBucketRegion, region)
}
w.Header().Set("Accept-Ranges", "bytes")
w.Header().Set(xhttp.AcceptRanges, "bytes")
// Remove sensitive information
crypto.RemoveSensitiveHeaders(w.Header())
@@ -72,23 +73,23 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, rs *HTTPRangeSp
// Set last modified time.
lastModified := objInfo.ModTime.UTC().Format(http.TimeFormat)
w.Header().Set("Last-Modified", lastModified)
w.Header().Set(xhttp.LastModified, lastModified)
// Set Etag if available.
if objInfo.ETag != "" {
w.Header()["ETag"] = []string{"\"" + objInfo.ETag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + objInfo.ETag + "\""}
}
if objInfo.ContentType != "" {
w.Header().Set("Content-Type", objInfo.ContentType)
w.Header().Set(xhttp.ContentType, objInfo.ContentType)
}
if objInfo.ContentEncoding != "" {
w.Header().Set("Content-Encoding", objInfo.ContentEncoding)
w.Header().Set(xhttp.ContentEncoding, objInfo.ContentEncoding)
}
if !objInfo.Expires.IsZero() {
w.Header().Set("Expires", objInfo.Expires.UTC().Format(http.TimeFormat))
w.Header().Set(xhttp.Expires, objInfo.Expires.UTC().Format(http.TimeFormat))
}
// Set all other user defined metadata.
@@ -124,10 +125,10 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, rs *HTTPRangeSp
}
// Set content length.
w.Header().Set("Content-Length", strconv.FormatInt(rangeLen, 10))
w.Header().Set(xhttp.ContentLength, strconv.FormatInt(rangeLen, 10))
if rs != nil {
contentRange := fmt.Sprintf("bytes %d-%d/%d", start, start+rangeLen-1, totalObjectSize)
w.Header().Set("Content-Range", contentRange)
w.Header().Set(xhttp.ContentRange, contentRange)
}
return nil
+15 -14
View File
@@ -26,6 +26,7 @@ import (
"strings"
"time"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/handlers"
)
@@ -522,9 +523,9 @@ func generateMultiDeleteResponse(quiet bool, deletedObjects []ObjectIdentifier,
func writeResponse(w http.ResponseWriter, statusCode int, response []byte, mType mimeType) {
setCommonHeaders(w)
if mType != mimeNone {
w.Header().Set("Content-Type", string(mType))
w.Header().Set(xhttp.ContentType, string(mType))
}
w.Header().Set("Content-Length", strconv.Itoa(len(response)))
w.Header().Set(xhttp.ContentLength, strconv.Itoa(len(response)))
w.WriteHeader(statusCode)
if response != nil {
w.Write(response)
@@ -563,7 +564,7 @@ func writeSuccessNoContent(w http.ResponseWriter) {
// writeRedirectSeeOther writes Location header with http status 303
func writeRedirectSeeOther(w http.ResponseWriter, location string) {
w.Header().Set("Location", location)
w.Header().Set(xhttp.Location, location)
writeResponse(w, http.StatusSeeOther, nil, mimeNone)
}
@@ -577,12 +578,12 @@ func writeErrorResponse(ctx context.Context, w http.ResponseWriter, err APIError
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
w.Header().Set(xhttp.RetryAfter, "120")
case "AccessDenied":
// The request is from browser and also if browser
// is enabled we need to redirect.
if browser {
w.Header().Set("Location", minioReservedBucketPath+reqURL.Path)
if browser && globalIsBrowserEnabled {
w.Header().Set(xhttp.Location, minioReservedBucketPath+reqURL.Path)
w.WriteHeader(http.StatusTemporaryRedirect)
return
}
@@ -590,7 +591,7 @@ func writeErrorResponse(ctx context.Context, w http.ResponseWriter, err APIError
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path,
w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
w.Header().Get(xhttp.AmzRequestID), globalDeploymentID)
encodedErrorResponse := encodeResponse(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeXML)
}
@@ -603,7 +604,7 @@ func writeErrorResponseHeadersOnly(w http.ResponseWriter, err APIError) {
// useful for admin APIs.
func writeErrorResponseJSON(ctx context.Context, w http.ResponseWriter, err APIError, reqURL *url.URL) {
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path, w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path, w.Header().Get(xhttp.AmzRequestID), globalDeploymentID)
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeJSON)
}
@@ -621,8 +622,8 @@ func writeCustomErrorResponseJSON(ctx context.Context, w http.ResponseWriter, er
Resource: reqURL.Path,
BucketName: reqInfo.BucketName,
Key: reqInfo.ObjectName,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
}
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeJSON)
@@ -637,12 +638,12 @@ func writeCustomErrorResponseXML(ctx context.Context, w http.ResponseWriter, err
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
w.Header().Set(xhttp.RetryAfter, "120")
case "AccessDenied":
// The request is from browser and also if browser
// is enabled we need to redirect.
if browser && globalIsBrowserEnabled {
w.Header().Set("Location", minioReservedBucketPath+reqURL.Path)
w.Header().Set(xhttp.Location, minioReservedBucketPath+reqURL.Path)
w.WriteHeader(http.StatusTemporaryRedirect)
return
}
@@ -655,8 +656,8 @@ func writeCustomErrorResponseXML(ctx context.Context, w http.ResponseWriter, err
Resource: reqURL.Path,
BucketName: reqInfo.BucketName,
Key: reqInfo.ObjectName,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
}
encodedErrorResponse := encodeResponse(errorResponse)
+46 -38
View File
@@ -20,6 +20,7 @@ import (
"net/http"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
)
// objectAPIHandler implements and provides http handlers for S3 API.
@@ -58,98 +59,105 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled, allowSSEKMS bool)
for _, bucket := range routers {
// Object operations
// HeadObject
bucket.Methods("HEAD").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.HeadObjectHandler))
bucket.Methods(http.MethodHead).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.HeadObjectHandler))
// CopyObjectPart
bucket.Methods("PUT").Path("/{object:.+}").HeadersRegexp("X-Amz-Copy-Source", ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodPut).Path("/{object:.+}").HeadersRegexp(xhttp.AmzCopySource, ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
// PutObjectPart
bucket.Methods("PUT").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
// ListObjectPxarts
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.ListObjectPartsHandler)).Queries("uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.ListObjectPartsHandler)).Queries("uploadId", "{uploadId:.*}")
// CompleteMultipartUpload
bucket.Methods("POST").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.CompleteMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.CompleteMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
// NewMultipartUpload
bucket.Methods("POST").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.NewMultipartUploadHandler)).Queries("uploads", "")
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.NewMultipartUploadHandler)).Queries("uploads", "")
// AbortMultipartUpload
bucket.Methods("DELETE").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.AbortMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.AbortMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
// GetObjectACL - this is a dummy call.
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectACLHandler)).Queries("acl", "")
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectACLHandler)).Queries("acl", "")
// GetObjectTagging - this is a dummy call.
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectTaggingHandler)).Queries("tagging", "")
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectTaggingHandler)).Queries("tagging", "")
// SelectObjectContent
bucket.Methods("POST").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.SelectObjectContentHandler)).Queries("select", "").Queries("select-type", "2")
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.SelectObjectContentHandler)).Queries("select", "").Queries("select-type", "2")
// GetObject
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectHandler))
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectHandler))
// CopyObject
bucket.Methods("PUT").Path("/{object:.+}").HeadersRegexp("X-Amz-Copy-Source", ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectHandler))
bucket.Methods(http.MethodPut).Path("/{object:.+}").HeadersRegexp(xhttp.AmzCopySource, ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectHandler))
// PutObject
bucket.Methods("PUT").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectHandler))
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectHandler))
// DeleteObject
bucket.Methods("DELETE").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.DeleteObjectHandler))
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.DeleteObjectHandler))
/// Bucket operations
// GetBucketLocation
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLocationHandler)).Queries("location", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketLocationHandler)).Queries("location", "")
// GetBucketPolicy
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketPolicyHandler)).Queries("policy", "")
// GetBucketLifecycle
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLifecycleHandler)).Queries("lifecycle", "")
// Dummy Bucket Calls
// GetBucketACL -- this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketACLHandler)).Queries("acl", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketACLHandler)).Queries("acl", "")
// GetBucketCors - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketCorsHandler)).Queries("cors", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketCorsHandler)).Queries("cors", "")
// GetBucketWebsiteHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketWebsiteHandler)).Queries("website", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketWebsiteHandler)).Queries("website", "")
// GetBucketVersioningHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketVersioningHandler)).Queries("versioning", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketVersioningHandler)).Queries("versioning", "")
// GetBucketAccelerateHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketAccelerateHandler)).Queries("accelerate", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketAccelerateHandler)).Queries("accelerate", "")
// GetBucketRequestPaymentHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketRequestPaymentHandler)).Queries("requestPayment", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketRequestPaymentHandler)).Queries("requestPayment", "")
// GetBucketLoggingHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLoggingHandler)).Queries("logging", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketLoggingHandler)).Queries("logging", "")
// GetBucketLifecycleHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLifecycleHandler)).Queries("lifecycle", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketLifecycleHandler)).Queries("lifecycle", "")
// GetBucketReplicationHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketReplicationHandler)).Queries("replication", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketReplicationHandler)).Queries("replication", "")
// GetBucketTaggingHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketTaggingHandler)).Queries("tagging", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketTaggingHandler)).Queries("tagging", "")
//DeleteBucketWebsiteHandler
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketWebsiteHandler)).Queries("website", "")
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketWebsiteHandler)).Queries("website", "")
// DeleteBucketTaggingHandler
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketTaggingHandler)).Queries("tagging", "")
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketTaggingHandler)).Queries("tagging", "")
// GetBucketNotification
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketNotificationHandler)).Queries("notification", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketNotificationHandler)).Queries("notification", "")
// ListenBucketNotification
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListenBucketNotificationHandler)).Queries("events", "{events:.*}")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListenBucketNotificationHandler)).Queries("events", "{events:.*}")
// ListMultipartUploads
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListMultipartUploadsHandler)).Queries("uploads", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListMultipartUploadsHandler)).Queries("uploads", "")
// ListObjectsV2
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListObjectsV2Handler)).Queries("list-type", "2")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListObjectsV2Handler)).Queries("list-type", "2")
// ListObjectsV1 (Legacy)
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListObjectsV1Handler))
// PutBucketLifecycle
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketLifecycleHandler)).Queries("lifecycle", "")
// PutBucketPolicy
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketPolicyHandler)).Queries("policy", "")
// PutBucketNotification
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketNotificationHandler)).Queries("notification", "")
bucket.Methods(http.MethodPut).HandlerFunc(httpTraceAll(api.PutBucketNotificationHandler)).Queries("notification", "")
// PutBucket
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketHandler))
bucket.Methods(http.MethodPut).HandlerFunc(httpTraceAll(api.PutBucketHandler))
// HeadBucket
bucket.Methods("HEAD").HandlerFunc(httpTraceAll(api.HeadBucketHandler))
bucket.Methods(http.MethodHead).HandlerFunc(httpTraceAll(api.HeadBucketHandler))
// PostPolicy
bucket.Methods("POST").HeadersRegexp("Content-Type", "multipart/form-data*").HandlerFunc(httpTraceHdrs(api.PostPolicyBucketHandler))
bucket.Methods(http.MethodPost).HeadersRegexp(xhttp.ContentType, "multipart/form-data*").HandlerFunc(httpTraceHdrs(api.PostPolicyBucketHandler))
// DeleteMultipleObjects
bucket.Methods("POST").HandlerFunc(httpTraceAll(api.DeleteMultipleObjectsHandler)).Queries("delete", "")
bucket.Methods(http.MethodPost).HandlerFunc(httpTraceAll(api.DeleteMultipleObjectsHandler)).Queries("delete", "")
// DeleteBucketPolicy
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketPolicyHandler)).Queries("policy", "")
// DeleteBucketLifecycle
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketLifecycleHandler)).Queries("lifecycle", "")
// DeleteBucket
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketHandler))
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketHandler))
}
/// Root operation
// ListBuckets
apiRouter.Methods("GET").Path("/").HandlerFunc(httpTraceAll(api.ListBucketsHandler))
apiRouter.Methods(http.MethodGet).Path("/").HandlerFunc(httpTraceAll(api.ListBucketsHandler))
// If none of the routes match.
apiRouter.NotFoundHandler = http.HandlerFunc(httpTraceAll(notFoundHandler))
+21 -19
View File
@@ -29,6 +29,7 @@ import (
"strings"
jwtgo "github.com/dgrijalva/jwt-go"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/hash"
@@ -38,41 +39,41 @@ import (
// Verify if request has JWT.
func isRequestJWT(r *http.Request) bool {
return strings.HasPrefix(r.Header.Get("Authorization"), jwtAlgorithm)
return strings.HasPrefix(r.Header.Get(xhttp.Authorization), jwtAlgorithm)
}
// Verify if request has AWS Signature Version '4'.
func isRequestSignatureV4(r *http.Request) bool {
return strings.HasPrefix(r.Header.Get("Authorization"), signV4Algorithm)
return strings.HasPrefix(r.Header.Get(xhttp.Authorization), signV4Algorithm)
}
// Verify if request has AWS Signature Version '2'.
func isRequestSignatureV2(r *http.Request) bool {
return (!strings.HasPrefix(r.Header.Get("Authorization"), signV4Algorithm) &&
strings.HasPrefix(r.Header.Get("Authorization"), signV2Algorithm))
return (!strings.HasPrefix(r.Header.Get(xhttp.Authorization), signV4Algorithm) &&
strings.HasPrefix(r.Header.Get(xhttp.Authorization), signV2Algorithm))
}
// Verify if request has AWS PreSign Version '4'.
func isRequestPresignedSignatureV4(r *http.Request) bool {
_, ok := r.URL.Query()["X-Amz-Credential"]
_, ok := r.URL.Query()[xhttp.AmzCredential]
return ok
}
// Verify request has AWS PreSign Version '2'.
func isRequestPresignedSignatureV2(r *http.Request) bool {
_, ok := r.URL.Query()["AWSAccessKeyId"]
_, ok := r.URL.Query()[xhttp.AmzAccessKeyID]
return ok
}
// Verify if request has AWS Post policy Signature Version '4'.
func isRequestPostPolicySignatureV4(r *http.Request) bool {
return strings.Contains(r.Header.Get("Content-Type"), "multipart/form-data") &&
return strings.Contains(r.Header.Get(xhttp.ContentType), "multipart/form-data") &&
r.Method == http.MethodPost
}
// Verify if the request has AWS Streaming Signature Version '4'. This is only valid for 'PUT' operation.
func isRequestSignStreamingV4(r *http.Request) bool {
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 &&
return r.Header.Get(xhttp.AmzContentSha256) == streamingContentSHA256 &&
r.Method == http.MethodPut
}
@@ -109,9 +110,9 @@ func getRequestAuthType(r *http.Request) authType {
return authTypeJWT
} else if isRequestPostPolicySignatureV4(r) {
return authTypePostPolicy
} else if _, ok := r.URL.Query()["Action"]; ok {
} else if _, ok := r.URL.Query()[xhttp.Action]; ok {
return authTypeSTS
} else if _, ok := r.Header["Authorization"]; !ok {
} else if _, ok := r.Header[xhttp.Authorization]; !ok {
return authTypeAnonymous
}
return authTypeUnknown
@@ -121,7 +122,7 @@ func getRequestAuthType(r *http.Request) authType {
// It does not accept presigned or JWT or anonymous requests.
func checkAdminRequestAuthType(ctx context.Context, r *http.Request, region string) APIErrorCode {
s3Err := ErrAccessDenied
if _, ok := r.Header["X-Amz-Content-Sha256"]; ok &&
if _, ok := r.Header[xhttp.AmzContentSha256]; ok &&
getRequestAuthType(r) == authTypeSigned && !skipContentSha256Cksum(r) {
// We only support admin credentials to access admin APIs.
@@ -148,11 +149,11 @@ func checkAdminRequestAuthType(ctx context.Context, r *http.Request, region stri
// Fetch the security token set by the client.
func getSessionToken(r *http.Request) (token string) {
token = r.Header.Get("X-Amz-Security-Token")
token = r.Header.Get(xhttp.AmzSecurityToken)
if token != "" {
return token
}
return r.URL.Query().Get("X-Amz-Security-Token")
return r.URL.Query().Get(xhttp.AmzSecurityToken)
}
// Fetch claims in the security token returned by the client, doesn't return
@@ -370,8 +371,8 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
contentMD5, contentSHA256 []byte
)
// Extract 'Content-Md5' if present.
if _, ok := r.Header["Content-Md5"]; ok {
contentMD5, err = base64.StdEncoding.Strict().DecodeString(r.Header.Get("Content-Md5"))
if _, ok := r.Header[xhttp.ContentMD5]; ok {
contentMD5, err = base64.StdEncoding.Strict().DecodeString(r.Header.Get(xhttp.ContentMD5))
if err != nil || len(contentMD5) == 0 {
return ErrInvalidDigest
}
@@ -380,14 +381,14 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
// Extract either 'X-Amz-Content-Sha256' header or 'X-Amz-Content-Sha256' query parameter (if V4 presigned)
// Do not verify 'X-Amz-Content-Sha256' if skipSHA256.
if skipSHA256 := skipContentSha256Cksum(r); !skipSHA256 && isRequestPresignedSignatureV4(r) {
if sha256Sum, ok := r.URL.Query()["X-Amz-Content-Sha256"]; ok && len(sha256Sum) > 0 {
if sha256Sum, ok := r.URL.Query()[xhttp.AmzContentSha256]; ok && len(sha256Sum) > 0 {
contentSHA256, err = hex.DecodeString(sha256Sum[0])
if err != nil {
return ErrContentSHA256Mismatch
}
}
} else if _, ok := r.Header["X-Amz-Content-Sha256"]; !skipSHA256 && ok {
contentSHA256, err = hex.DecodeString(r.Header.Get("X-Amz-Content-Sha256"))
} else if _, ok := r.Header[xhttp.AmzContentSha256]; !skipSHA256 && ok {
contentSHA256, err = hex.DecodeString(r.Header.Get(xhttp.AmzContentSha256))
if err != nil || len(contentSHA256) == 0 {
return ErrContentSHA256Mismatch
}
@@ -395,7 +396,8 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
// Verify 'Content-Md5' and/or 'X-Amz-Content-Sha256' if present.
// The verification happens implicit during reading.
reader, err := hash.NewReader(r.Body, -1, hex.EncodeToString(contentMD5), hex.EncodeToString(contentSHA256), -1, globalCLIContext.StrictS3Compat)
reader, err := hash.NewReader(r.Body, -1, hex.EncodeToString(contentMD5),
hex.EncodeToString(contentSHA256), -1, globalCLIContext.StrictS3Compat)
if err != nil {
return toAPIErrorCode(ctx, err)
}
+1 -1
View File
@@ -131,7 +131,7 @@ func (b *streamingBitrotReader) ReadAt(buf []byte, offset int64) (int, error) {
b.h.Write(buf)
if !bytes.Equal(b.h.Sum(nil), b.hashBytes) {
err = hashMismatchError{hex.EncodeToString(b.hashBytes), hex.EncodeToString(b.h.Sum(nil))}
err = HashMismatchError{hex.EncodeToString(b.hashBytes), hex.EncodeToString(b.h.Sum(nil))}
logger.LogIf(context.Background(), err)
return 0, err
}
-30
View File
@@ -155,33 +155,3 @@ func bitrotWriterSum(w io.Writer) []byte {
}
return nil
}
// Verify if a file has bitrot error.
func bitrotCheckFile(disk StorageAPI, volume string, filePath string, tillOffset int64, algo BitrotAlgorithm, sum []byte, shardSize int64) (err error) {
if algo != HighwayHash256S {
buf := []byte{}
// For whole-file bitrot we don't need to read the entire file as the bitrot verify happens on the server side even if we read 0-bytes.
_, err = disk.ReadFile(volume, filePath, 0, buf, NewBitrotVerifier(algo, sum))
return err
}
buf := make([]byte, shardSize)
r := newStreamingBitrotReader(disk, volume, filePath, tillOffset, algo, shardSize)
defer closeBitrotReaders([]io.ReaderAt{r})
var offset int64
for {
if offset == tillOffset {
break
}
var n int
tmpBuf := buf
if int64(len(tmpBuf)) > (tillOffset - offset) {
tmpBuf = tmpBuf[:(tillOffset - offset)]
}
n, err = r.ReadAt(tmpBuf, offset)
if err != nil {
return err
}
offset += int64(n)
}
return nil
}
-7
View File
@@ -46,13 +46,6 @@ func validateListObjectsArgs(prefix, marker, delimiter, encodingType string, max
}
}
/// MinIO special conditions for ListObjects.
// Verify if delimiter is anything other than '/', which we do not support.
if delimiter != "" && delimiter != "/" {
return ErrNotImplemented
}
// Success.
return ErrNone
}
+12 -5
View File
@@ -32,6 +32,7 @@ import (
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/event"
@@ -445,7 +446,8 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
}
// Make sure to add Location information here only for bucket
w.Header().Set("Location", getObjectLocation(r, globalDomainNames, bucket, ""))
w.Header().Set(xhttp.Location,
getObjectLocation(r, globalDomainNames, bucket, ""))
writeSuccessResponseHeadersOnly(w)
return
@@ -466,7 +468,7 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
}
// Make sure to add Location information here only for bucket
w.Header().Set("Location", path.Clean(r.URL.Path)) // Clean any trailing slashes.
w.Header().Set(xhttp.Location, path.Clean(r.URL.Path)) // Clean any trailing slashes.
writeSuccessResponseHeadersOnly(w)
}
@@ -498,6 +500,9 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
bucket := mux.Vars(r)["bucket"]
// To detect if the client has disconnected.
r.Body = &detectDisconnect{r.Body, r.Context().Done()}
// Require Content-Length to be set in the request
size := r.ContentLength
if size < 0 {
@@ -532,7 +537,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
// Remove all tmp files creating during multipart upload
// Remove all tmp files created during multipart upload
defer form.RemoveAll()
// Extract all form fields
@@ -678,8 +683,8 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
}
location := getObjectLocation(r, globalDomainNames, bucket, object)
w.Header()["ETag"] = []string{`"` + objInfo.ETag + `"`}
w.Header().Set("Location", location)
w.Header()[xhttp.ETag] = []string{`"` + objInfo.ETag + `"`}
w.Header().Set(xhttp.Location, location)
// Notify object created event.
defer sendEvent(eventArgs{
@@ -795,6 +800,8 @@ func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.
globalNotificationSys.RemoveNotification(bucket)
globalPolicySys.Remove(bucket)
globalNotificationSys.DeleteBucket(ctx, bucket)
globalLifecycleSys.Remove(bucket)
globalNotificationSys.RemoveBucketLifecycle(ctx, bucket)
// Write success response.
writeSuccessNoContent(w)
+164
View File
@@ -0,0 +1,164 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/xml"
"io"
"net/http"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lifecycle"
)
const (
// Lifecycle configuration file.
bucketLifecycleConfig = "lifecycle.xml"
)
// PutBucketLifecycleHandler - This HTTP handler stores given bucket lifecycle configuration as per
// https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html
func (api objectAPIHandlers) PutBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "PutBucketLifecycle")
defer logger.AuditLog(w, r, "PutBucketLifecycle", mustGetClaimsFromToken(r))
objAPI := api.ObjectAPI()
if objAPI == nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
return
}
vars := mux.Vars(r)
bucket := vars["bucket"]
if s3Error := checkRequestAuthType(ctx, r, lifecycle.PutBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
// Check if bucket exists.
if _, err := objAPI.GetBucketInfo(ctx, bucket); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
// PutBucketLifecycle always needs a Content-Md5
if _, ok := r.Header["Content-Md5"]; !ok {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentMD5), r.URL, guessIsBrowserReq(r))
return
}
bucketLifecycle, err := lifecycle.ParseLifecycleConfig(io.LimitReader(r.Body, r.ContentLength))
if err != nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMalformedXML), r.URL, guessIsBrowserReq(r))
return
}
if err = objAPI.SetBucketLifecycle(ctx, bucket, bucketLifecycle); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
globalLifecycleSys.Set(bucket, *bucketLifecycle)
globalNotificationSys.SetBucketLifecycle(ctx, bucket, bucketLifecycle)
// Success.
writeSuccessNoContent(w)
}
// GetBucketLifecycleHandler - This HTTP handler returns bucket policy configuration.
func (api objectAPIHandlers) GetBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "GetBucketLifecycle")
defer logger.AuditLog(w, r, "GetBucketLifecycle", mustGetClaimsFromToken(r))
objAPI := api.ObjectAPI()
if objAPI == nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
return
}
vars := mux.Vars(r)
bucket := vars["bucket"]
if s3Error := checkRequestAuthType(ctx, r, lifecycle.GetBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
// Check if bucket exists.
if _, err := objAPI.GetBucketInfo(ctx, bucket); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
// Read bucket access lifecycle.
bucketLifecycle, err := objAPI.GetBucketLifecycle(ctx, bucket)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
lifecycleData, err := xml.Marshal(bucketLifecycle)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
// Write lifecycle configuration to client.
writeSuccessResponseXML(w, lifecycleData)
}
// DeleteBucketLifecycleHandler - This HTTP handler removes bucket lifecycle configuration.
func (api objectAPIHandlers) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "DeleteBucketLifecycle")
defer logger.AuditLog(w, r, "DeleteBucketLifecycle", mustGetClaimsFromToken(r))
objAPI := api.ObjectAPI()
if objAPI == nil {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrServerNotInitialized), r.URL, guessIsBrowserReq(r))
return
}
vars := mux.Vars(r)
bucket := vars["bucket"]
if s3Error := checkRequestAuthType(ctx, r, lifecycle.DeleteBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
// Check if bucket exists.
if _, err := objAPI.GetBucketInfo(ctx, bucket); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
if err := objAPI.DeleteBucketLifecycle(ctx, bucket); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
globalLifecycleSys.Remove(bucket)
globalNotificationSys.RemoveBucketLifecycle(ctx, bucket)
// Success.
writeSuccessNoContent(w)
}
+3
View File
@@ -23,6 +23,7 @@ import (
"net/http"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/event/target"
@@ -246,6 +247,8 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
return
}
w.Header().Set(xhttp.ContentType, "text/event-stream")
target, err := target.NewHTTPClientTarget(*host, w)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
+9 -9
View File
@@ -314,7 +314,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewElasticsearchTarget(k, v)
t, err := target.NewElasticsearchTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("elasticsearch(%s): %s", k, err.Error())
}
@@ -358,7 +358,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewNATSTarget(k, v)
t, err := target.NewNATSTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("nats(%s): %s", k, err.Error())
}
@@ -369,7 +369,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewNSQTarget(k, v)
t, err := target.NewNSQTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("nsq(%s): %s", k, err.Error())
}
@@ -391,7 +391,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewRedisTarget(k, v)
t, err := target.NewRedisTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("redis(%s): %s", k, err.Error())
}
@@ -651,7 +651,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.Elasticsearch {
if args.Enable {
newTarget, err := target.NewElasticsearchTarget(id, args)
newTarget, err := target.NewElasticsearchTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -710,7 +710,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.NATS {
if args.Enable {
newTarget, err := target.NewNATSTarget(id, args)
newTarget, err := target.NewNATSTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -724,7 +724,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.NSQ {
if args.Enable {
newTarget, err := target.NewNSQTarget(id, args)
newTarget, err := target.NewNSQTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -752,7 +752,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.Redis {
if args.Enable {
newTarget, err := target.NewRedisTarget(id, args)
newTarget, err := target.NewRedisTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -767,7 +767,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.Webhook {
if args.Enable {
args.RootCAs = globalRootCAs
newTarget := target.NewWebhookTarget(id, args)
newTarget := target.NewWebhookTarget(id, args, GlobalServiceDoneCh)
if err := targetList.Add(newTarget); err != nil {
logger.LogIf(context.Background(), err)
continue
+5 -5
View File
@@ -188,7 +188,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "amqp": { "1": { "enable": true, "url": "", "exchange": "", "routingKey": "", "exchangeType": "", "mandatory": false, "immediate": false, "durable": false, "internal": false, "noWait": false, "autoDeleted": false }}}}`, false},
// Test 12 - Test NATS
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nats": { "1": { "enable": true, "address": "", "subject": "", "username": "", "password": "", "token": "", "secure": false, "pingInterval": 0, "streaming": { "enable": false, "clusterID": "", "async": false, "maxPubAcksInflight": 0 } } }}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nats": { "1": { "enable": true, "address": "", "subject": "", "username": "", "password": "", "token": "", "secure": false, "pingInterval": 0, "queueDir": "", "queueLimit": 0, "streaming": { "enable": false, "clusterID": "", "async": false, "maxPubAcksInflight": 0 } } }}}`, false},
// Test 13 - Test ElasticSearch
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "elasticsearch": { "1": { "enable": true, "url": "", "index": "" } }}}`, false},
@@ -203,7 +203,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "kafka": { "1": { "enable": true, "brokers": null, "topic": "", "queueDir": "", "queueLimit": 0 } }}}`, false},
// Test 17 - Test Webhook
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "webhook": { "1": { "enable": true, "endpoint": "" } }}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "webhook": { "1": { "enable": true, "endpoint": "", "queueDir": "", "queueLimit": 0} }}}`, false},
// Test 18 - Test MySQL
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mysql": { "1": { "enable": true, "dsnString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "" }}}}`, false},
@@ -224,10 +224,10 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "elasticsearch": { "1": { "enable": true, "format": "invalid", "url": "example.com", "index": "myindex" } }}}`, false},
// Test 24 - Test valid Format for ElasticSearch
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "elasticsearch": { "1": { "enable": true, "format": "namespace", "url": "example.com", "index": "myindex" } }}}`, true},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "elasticsearch": { "1": { "enable": true, "format": "namespace", "url": "example.com", "index": "myindex", "queueDir": "", "queueLimit": 0 } }}}`, true},
// Test 25 - Test Format for Redis
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "redis": { "1": { "enable": true, "format": "invalid", "address": "example.com:80", "password": "xxx", "key": "key1" } }}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "redis": { "1": { "enable": true, "format": "invalid", "address": "example.com:80", "password": "xxx", "key": "key1", "queueDir": "", "queueLimit": 0 } }}}`, false},
// Test 26 - Test valid Format for Redis
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "redis": { "1": { "enable": true, "format": "namespace", "address": "example.com:80", "password": "xxx", "key": "key1" } }}}`, true},
@@ -236,7 +236,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mqtt": { "1": { "enable": true, "broker": "", "topic": "", "qos": 0, "username": "", "password": "", "queueDir": "", "queueLimit": 0}}}}`, false},
// Test 28 - Test NSQ
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nsq": { "1": { "enable": true, "nsqdAddress": "", "topic": ""} }}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nsq": { "1": { "enable": true, "nsqdAddress": "", "topic": "", "queueDir": "", "queueLimit": 0} }}}`, false},
}
for i, testCase := range testCases {
+2 -2
View File
@@ -2440,7 +2440,7 @@ func migrateConfigToMinioSys(objAPI ObjectLayer) (err error) {
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(context.Background(), minioMetaBucket, transactionConfigFile)
if err = objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
@@ -2492,7 +2492,7 @@ func migrateMinioSysConfig(objAPI ObjectLayer) error {
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(context.Background(), minioMetaBucket, transactionConfigFile)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
+2 -2
View File
@@ -22,7 +22,7 @@ import (
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/event/target"
"github.com/minio/minio/pkg/iam/policy"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/iam/validator"
"github.com/minio/minio/pkg/quick"
)
@@ -884,7 +884,7 @@ type serverConfigV32 struct {
} `json:"policy"`
}
// serverConfigV33 is just like version '32', removes clientID from NATS and MQTT, and adds queueDir, queueLimit in MQTT and kafka.
// serverConfigV33 is just like version '32', removes clientID from NATS and MQTT, and adds queueDir, queueLimit in all notification targets.
type serverConfigV33 struct {
quick.Config `json:"-"` // ignore interfaces
+1 -1
View File
@@ -39,7 +39,7 @@ func newBgHealSequence(numDisks int) *healSequence {
hs := madmin.HealOpts{
// Remove objects that do not have read-quorum
Remove: true,
ScanMode: madmin.HealDeepScan,
ScanMode: madmin.HealNormalScan,
}
return &healSequence{
+3 -3
View File
@@ -55,7 +55,7 @@ func sweepRound(ctx context.Context, objAPI ObjectLayer) error {
zeroDynamicTimeout := newDynamicTimeout(zeroDuration, zeroDuration)
// General lock so we avoid parallel daily sweep by different instances.
sweepLock := globalNSMutex.NewNSLock("system", "daily-sweep")
sweepLock := globalNSMutex.NewNSLock(ctx, "system", "daily-sweep")
if err := sweepLock.GetLock(zeroDynamicTimeout); err != nil {
return err
}
@@ -119,9 +119,9 @@ func dailySweeper() {
break
}
// Perform a sweep round each 24 hours
// Perform a sweep round each month
for {
if time.Since(lastSweepTime) < 24*time.Hour {
if time.Since(lastSweepTime) < 30*24*time.Hour {
time.Sleep(time.Hour)
continue
}
+2 -2
View File
@@ -303,7 +303,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
data := r.Reader
fs := cfs.FSObjects
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
return objInfo, err
}
@@ -489,7 +489,7 @@ func (cfs *cacheFSObjects) NewMultipartUpload(ctx context.Context, bucket, objec
// moveBucketToTrash clears cacheFSObjects of bucket contents and moves it to trash folder.
func (cfs *cacheFSObjects) moveBucketToTrash(ctx context.Context, bucket string) (err error) {
fs := cfs.FSObjects
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err = bucketLock.GetLock(globalObjectTimeout); err != nil {
return err
}
+4 -1
View File
@@ -945,7 +945,10 @@ func checkAtimeSupport(dir string) (err error) {
if err != nil {
return
}
if _, err = io.Copy(ioutil.Discard, file); err != io.EOF {
// add a sleep to ensure atime change is detected
time.Sleep(10 * time.Millisecond)
if _, err = io.Copy(ioutil.Discard, file); err != nil {
return
}
+5 -12
View File
@@ -124,17 +124,10 @@ func TestGetCacheFSMaxUse(t *testing.T) {
// test wildcard patterns for excluding entries from cache
func TestCacheExclusion(t *testing.T) {
fsDirs, err := getRandomDisks(1)
if err != nil {
t.Fatal(err)
cobjects := &cacheObjects{
cache: nil,
}
cconfig := CacheConfig{Expiry: 30, Drives: fsDirs}
cobjects, err := newServerCacheObjects(cconfig)
if err != nil {
t.Fatal(err)
}
cobj := cobjects.(*cacheObjects)
GlobalServiceDoneCh <- struct{}{}
testCases := []struct {
bucketName string
objectName string
@@ -155,8 +148,8 @@ func TestCacheExclusion(t *testing.T) {
}
for i, testCase := range testCases {
cobj.exclude = []string{testCase.excludePattern}
if cobj.isCacheExclude(testCase.bucketName, testCase.objectName) != testCase.expectedResult {
cobjects.exclude = []string{testCase.excludePattern}
if cobjects.isCacheExclude(testCase.bucketName, testCase.objectName) != testCase.expectedResult {
t.Fatal("Cache exclusion test failed for case ", i)
}
}
-6
View File
@@ -72,12 +72,6 @@ func (api objectAPIHandlers) GetBucketLoggingHandler(w http.ResponseWriter, r *h
w.(http.Flusher).Flush()
}
// GetBucketLifecycleHandler - GET bucket lifecycle, a dummy api
func (api objectAPIHandlers) GetBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
writeSuccessResponseHeadersOnly(w)
w.(http.Flusher).Flush()
}
// GetBucketReplicationHandler - GET bucket replication, a dummy api
func (api objectAPIHandlers) GetBucketReplicationHandler(w http.ResponseWriter, r *http.Request) {
writeSuccessResponseHeadersOnly(w)
+19 -7
View File
@@ -20,6 +20,7 @@ import (
"context"
"fmt"
"net"
"net/http"
"net/url"
"os"
"path"
@@ -288,7 +289,7 @@ func (endpoints EndpointList) UpdateIsLocal() error {
// localEndpointsMemUsage - returns ServerMemUsageInfo for only the
// local endpoints from given list of endpoints
func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
func localEndpointsMemUsage(endpoints EndpointList, r *http.Request) ServerMemUsageInfo {
var memUsages []mem.Usage
var historicUsages []mem.Usage
scratchSpace := map[string]bool{}
@@ -303,8 +304,12 @@ func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
scratchSpace[endpoint.Host] = true
}
}
addr := r.Host
if globalIsDistXL {
addr = GetLocalPeer(endpoints)
}
return ServerMemUsageInfo{
Addr: GetLocalPeer(endpoints),
Addr: addr,
Usage: memUsages,
HistoricUsage: historicUsages,
}
@@ -312,7 +317,7 @@ func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
// localEndpointsCPULoad - returns ServerCPULoadInfo for only the
// local endpoints from given list of endpoints
func localEndpointsCPULoad(endpoints EndpointList) ServerCPULoadInfo {
func localEndpointsCPULoad(endpoints EndpointList, r *http.Request) ServerCPULoadInfo {
var cpuLoads []cpu.Load
var historicLoads []cpu.Load
scratchSpace := map[string]bool{}
@@ -327,8 +332,12 @@ func localEndpointsCPULoad(endpoints EndpointList) ServerCPULoadInfo {
scratchSpace[endpoint.Host] = true
}
}
addr := r.Host
if globalIsDistXL {
addr = GetLocalPeer(endpoints)
}
return ServerCPULoadInfo{
Addr: GetLocalPeer(endpoints),
Addr: addr,
Load: cpuLoads,
HistoricLoad: historicLoads,
}
@@ -336,7 +345,7 @@ func localEndpointsCPULoad(endpoints EndpointList) ServerCPULoadInfo {
// localEndpointsDrivePerf - returns ServerDrivesPerfInfo for only the
// local endpoints from given list of endpoints
func localEndpointsDrivePerf(endpoints EndpointList) ServerDrivesPerfInfo {
func localEndpointsDrivePerf(endpoints EndpointList, r *http.Request) ServerDrivesPerfInfo {
var dps []disk.Performance
for _, endpoint := range endpoints {
// Only proceed for local endpoints
@@ -351,9 +360,12 @@ func localEndpointsDrivePerf(endpoints EndpointList) ServerDrivesPerfInfo {
dps = append(dps, dp)
}
}
addr := r.Host
if globalIsDistXL {
addr = GetLocalPeer(endpoints)
}
return ServerDrivesPerfInfo{
Addr: GetLocalPeer(endpoints),
Addr: addr,
Perf: dps,
}
}
+1 -1
View File
@@ -491,7 +491,7 @@ func formatXLGetDeploymentID(refFormat *formatXLV3, formats []*formatXLV3) (stri
func formatXLFixDeploymentID(ctx context.Context, endpoints EndpointList, storageDisks []StorageAPI, refFormat *formatXLV3) (err error) {
// Acquire lock on format.json
mutex := newNSLock(globalIsDistXL)
formatLock := mutex.NewNSLock(minioMetaBucket, formatConfigFile)
formatLock := mutex.NewNSLock(ctx, minioMetaBucket, formatConfigFile)
if err = formatLock.GetLock(globalHealingTimeout); err != nil {
return err
}
+21 -1
View File
@@ -280,7 +280,7 @@ func fsOpenFile(ctx context.Context, readPath string, offset int64) (io.ReadClos
}
// Stat to get the size of the file at path.
st, err := os.Stat(readPath)
st, err := fr.Stat()
if err != nil {
err = osErrToFSFileErr(err)
if err != errFileNotFound {
@@ -386,6 +386,26 @@ func fsFAllocate(fd int, offset int64, len int64) (err error) {
return nil
}
// Renames source path to destination path, fails if the destination path
// parents are not already created.
func fsSimpleRenameFile(ctx context.Context, sourcePath, destPath string) error {
if err := checkPathLength(sourcePath); err != nil {
logger.LogIf(ctx, err)
return err
}
if err := checkPathLength(destPath); err != nil {
logger.LogIf(ctx, err)
return err
}
if err := os.Rename(sourcePath, destPath); err != nil {
logger.LogIf(ctx, err)
return osErrToFSFileErr(err)
}
return nil
}
// Renames source path to destination path, creates all the
// missing parents if they don't exist.
func fsRenameFile(ctx context.Context, sourcePath, destPath string) error {
+7 -3
View File
@@ -152,7 +152,7 @@ func (fs *FSObjects) ListMultipartUploads(ctx context.Context, bucket, object, k
return result, toObjectErr(err)
}
// S3 spec says uploaIDs should be sorted based on initiated time. ModTime of fs.json
// S3 spec says uploadIDs should be sorted based on initiated time. ModTime of fs.json
// is the creation time of the uploadID, hence we will use that.
var uploads []MultipartInfo
for _, uploadID := range uploadIDs {
@@ -326,7 +326,11 @@ func (fs *FSObjects) PutObjectPart(ctx context.Context, bucket, object, uploadID
partPath := pathJoin(uploadIDDir, fs.encodePartFile(partID, etag, data.ActualSize()))
if err = fsRenameFile(ctx, tmpPartPath, partPath); err != nil {
// Make sure not to create parent directories if they don't exist - the upload might have been aborted.
if err = fsSimpleRenameFile(ctx, tmpPartPath, partPath); err != nil {
if err == errFileNotFound || err == errFileAccessDenied {
return pi, InvalidUploadID{UploadID: uploadID}
}
return pi, toObjectErr(err, minioMetaMultipartBucket, partPath)
}
@@ -634,7 +638,7 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
}
// Hold write lock on the object.
destLock := fs.nsMutex.NewNSLock(bucket, object)
destLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err = destLock.GetLock(globalObjectTimeout); err != nil {
return oi, err
}
+26 -10
View File
@@ -32,6 +32,7 @@ import (
"github.com/minio/minio-go/v6/pkg/s3utils"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lifecycle"
"github.com/minio/minio/pkg/lock"
"github.com/minio/minio/pkg/madmin"
"github.com/minio/minio/pkg/mimedb"
@@ -286,7 +287,7 @@ func (fs *FSObjects) statBucketDir(ctx context.Context, bucket string) (os.FileI
// MakeBucketWithLocation - create a new bucket, returns if it
// already exists.
func (fs *FSObjects) MakeBucketWithLocation(ctx context.Context, bucket, location string) error {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err := bucketLock.GetLock(globalObjectTimeout); err != nil {
return err
}
@@ -309,7 +310,7 @@ func (fs *FSObjects) MakeBucketWithLocation(ctx context.Context, bucket, locatio
// GetBucketInfo - fetch bucket metadata info.
func (fs *FSObjects) GetBucketInfo(ctx context.Context, bucket string) (bi BucketInfo, e error) {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if e := bucketLock.GetRLock(globalObjectTimeout); e != nil {
return bi, e
}
@@ -372,7 +373,7 @@ func (fs *FSObjects) ListBuckets(ctx context.Context) ([]BucketInfo, error) {
// DeleteBucket - delete a bucket and all the metadata associated
// with the bucket including pending multipart, object metadata.
func (fs *FSObjects) DeleteBucket(ctx context.Context, bucket string) error {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err := bucketLock.GetLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return err
@@ -408,7 +409,7 @@ func (fs *FSObjects) DeleteBucket(ctx context.Context, bucket string) error {
func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject string, srcInfo ObjectInfo, srcOpts, dstOpts ObjectOptions) (oi ObjectInfo, e error) {
cpSrcDstSame := isStringEqual(pathJoin(srcBucket, srcObject), pathJoin(dstBucket, dstObject))
if !cpSrcDstSame {
objectDWLock := fs.nsMutex.NewNSLock(dstBucket, dstObject)
objectDWLock := fs.nsMutex.NewNSLock(ctx, dstBucket, dstObject)
if err := objectDWLock.GetLock(globalObjectTimeout); err != nil {
return oi, err
}
@@ -480,7 +481,7 @@ func (fs *FSObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
if lockType != noLock {
// Lock the object before reading.
lock := fs.nsMutex.NewNSLock(bucket, object)
lock := fs.nsMutex.NewNSLock(ctx, bucket, object)
switch lockType {
case writeLock:
if err = lock.GetLock(globalObjectTimeout); err != nil {
@@ -567,7 +568,7 @@ func (fs *FSObjects) GetObject(ctx context.Context, bucket, object string, offse
}
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return err
@@ -739,7 +740,7 @@ func (fs *FSObjects) getObjectInfo(ctx context.Context, bucket, object string) (
// getObjectInfoWithLock - reads object metadata and replies back ObjectInfo.
func (fs *FSObjects) getObjectInfoWithLock(ctx context.Context, bucket, object string) (oi ObjectInfo, e error) {
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
return oi, err
}
@@ -764,7 +765,7 @@ func (fs *FSObjects) getObjectInfoWithLock(ctx context.Context, bucket, object s
func (fs *FSObjects) GetObjectInfo(ctx context.Context, bucket, object string, opts ObjectOptions) (oi ObjectInfo, e error) {
oi, err := fs.getObjectInfoWithLock(ctx, bucket, object)
if err == errCorruptedFormat || err == io.EOF {
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err = objectLock.GetLock(globalObjectTimeout); err != nil {
return oi, toObjectErr(err, bucket, object)
}
@@ -810,7 +811,7 @@ func (fs *FSObjects) PutObject(ctx context.Context, bucket string, object string
return ObjectInfo{}, err
}
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return objInfo, err
@@ -965,7 +966,7 @@ func (fs *FSObjects) DeleteObjects(ctx context.Context, bucket string, objects [
// and there are no rollbacks supported.
func (fs *FSObjects) DeleteObject(ctx context.Context, bucket, object string) error {
// Acquire a write lock before deleting the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalOperationTimeout); err != nil {
return err
}
@@ -1170,6 +1171,21 @@ func (fs *FSObjects) DeleteBucketPolicy(ctx context.Context, bucket string) erro
return removePolicyConfig(ctx, fs, bucket)
}
// SetBucketLifecycle sets lifecycle on bucket
func (fs *FSObjects) SetBucketLifecycle(ctx context.Context, bucket string, lifecycle *lifecycle.Lifecycle) error {
return saveLifecycleConfig(ctx, fs, bucket, lifecycle)
}
// GetBucketLifecycle will get lifecycle on bucket
func (fs *FSObjects) GetBucketLifecycle(ctx context.Context, bucket string) (*lifecycle.Lifecycle, error) {
return getLifecycleConfig(fs, bucket)
}
// DeleteBucketLifecycle deletes all lifecycle on bucket
func (fs *FSObjects) DeleteBucketLifecycle(ctx context.Context, bucket string) error {
return removeLifecycleConfig(ctx, fs, bucket)
}
// ListObjectsV2 lists all blobs in bucket filtered by prefix
func (fs *FSObjects) ListObjectsV2(ctx context.Context, bucket, prefix, continuationToken, delimiter string, maxKeys int, fetchOwner bool, startAfter string) (result ListObjectsV2Info, err error) {
marker := continuationToken
+5 -2
View File
@@ -21,6 +21,7 @@ import (
"os"
"strings"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/hash"
@@ -166,7 +167,7 @@ func FromMinioClientListMultipartsInfo(lmur minio.ListMultipartUploadsResult) Li
// FromMinioClientObjectInfo converts minio ObjectInfo to gateway ObjectInfo
func FromMinioClientObjectInfo(bucket string, oi minio.ObjectInfo) ObjectInfo {
userDefined := FromMinioClientMetadata(oi.Metadata)
userDefined["Content-Type"] = oi.ContentType
userDefined[xhttp.ContentType] = oi.ContentType
return ObjectInfo{
Bucket: bucket,
@@ -176,7 +177,7 @@ func FromMinioClientObjectInfo(bucket string, oi minio.ObjectInfo) ObjectInfo {
ETag: canonicalizeETag(oi.ETag),
UserDefined: userDefined,
ContentType: oi.ContentType,
ContentEncoding: oi.Metadata.Get("Content-Encoding"),
ContentEncoding: oi.Metadata.Get(xhttp.ContentEncoding),
StorageClass: oi.StorageClass,
Expires: oi.Expires,
}
@@ -316,6 +317,8 @@ func ErrorRespToObjectError(err error, params ...string) error {
err = BucketNotEmpty{}
case "NoSuchBucketPolicy":
err = BucketPolicyNotFound{}
case "NoSuchBucketLifecycle":
err = BucketLifecycleNotFound{}
case "InvalidBucketName":
err = BucketNameInvalid{Bucket: bucket}
case "InvalidPart":
+5 -1
View File
@@ -48,7 +48,7 @@ var (
// RegisterGatewayCommand registers a new command for gateway.
func RegisterGatewayCommand(cmd cli.Command) error {
cmd.Flags = append(append(cmd.Flags, append(cmd.Flags, ServerFlags...)...), GlobalFlags...)
cmd.Flags = append(append(cmd.Flags, ServerFlags...), GlobalFlags...)
gatewayCmd.Subcommands = append(gatewayCmd.Subcommands, cmd)
return nil
}
@@ -243,6 +243,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// This is only to uniquely identify each gateway deployments.
globalDeploymentID = os.Getenv("MINIO_GATEWAY_DEPLOYMENT_ID")
logger.SetDeploymentID(globalDeploymentID)
var cacheConfig = globalServerConfig.GetCacheConfig()
if len(cacheConfig.Drives) > 0 {
@@ -268,6 +269,9 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// Initialize policy system.
go globalPolicySys.Init(newObject)
// Create new lifecycle system
globalLifecycleSys = NewLifecycleSys()
// Create new notification system.
globalNotificationSys = NewNotificationSys(globalServerConfig, globalEndpoints)
if globalEtcdClient != nil && newObject.IsNotificationSupported() {
+31
View File
@@ -17,6 +17,7 @@
package cmd
import (
"fmt"
"strings"
"testing"
@@ -34,6 +35,36 @@ func TestRegisterGatewayCommand(t *testing.T) {
}
}
// Test running a registered gateway command with a flag
func TestRunRegisteredGatewayCommand(t *testing.T) {
var err error
flagName := "test-flag"
flagValue := "foo"
cmd := cli.Command{
Name: "test-run-with-flag",
Flags: []cli.Flag{
cli.StringFlag{Name: flagName},
},
Action: func(ctx *cli.Context) {
if actual := ctx.String(flagName); actual != flagValue {
t.Errorf("value of %s expects %s, but got %s", flagName, flagValue, actual)
}
},
}
err = RegisterGatewayCommand(cmd)
if err != nil {
t.Errorf("RegisterGatewayCommand got unexpected error: %s", err)
}
if err = newApp("minio").Run(
[]string{"minio", "gateway", cmd.Name, fmt.Sprintf("--%s", flagName), flagValue}); err != nil {
t.Errorf("running registered gateway command got unexpected error: %s", err)
}
}
// Test parseGatewayEndpoint
func TestParseGatewayEndpoint(t *testing.T) {
testCases := []struct {
+17
View File
@@ -20,6 +20,7 @@ import (
"context"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lifecycle"
"github.com/minio/minio/pkg/madmin"
"github.com/minio/minio/pkg/policy"
)
@@ -81,6 +82,22 @@ func (a GatewayUnsupported) DeleteBucketPolicy(ctx context.Context, bucket strin
return NotImplemented{}
}
// SetBucketLifecycle sets lifecycle on bucket
func (a GatewayUnsupported) SetBucketLifecycle(ctx context.Context, bucket string, lifecycle *lifecycle.Lifecycle) error {
logger.LogIf(ctx, NotImplemented{})
return NotImplemented{}
}
// GetBucketLifecycle will get lifecycle on bucket
func (a GatewayUnsupported) GetBucketLifecycle(ctx context.Context, bucket string) (*lifecycle.Lifecycle, error) {
return nil, NotImplemented{}
}
// DeleteBucketLifecycle deletes all lifecycle on bucket
func (a GatewayUnsupported) DeleteBucketLifecycle(ctx context.Context, bucket string) error {
return NotImplemented{}
}
// ReloadFormat - Not implemented stub.
func (a GatewayUnsupported) ReloadFormat(ctx context.Context, dryRun bool) error {
return NotImplemented{}
+1 -1
View File
@@ -111,7 +111,7 @@ EXAMPLES:
minio.RegisterGatewayCommand(cli.Command{
Name: azureBackend,
Usage: "Microsoft Azure Blob Storage.",
Usage: "Microsoft Azure Blob Storage",
Action: azureGatewayMain,
CustomHelpTemplate: azureGatewayTemplate,
HideHelpCommand: true,
+32 -20
View File
@@ -29,7 +29,7 @@ import (
"sync"
"time"
b2 "github.com/minio/blazer/base"
b2 "github.com/kurin/blazer/base"
"github.com/minio/cli"
miniogopolicy "github.com/minio/minio-go/v6/pkg/policy"
"github.com/minio/minio/cmd/logger"
@@ -147,6 +147,29 @@ func b2ToObjectError(err error, params ...string) error {
if err == nil {
return nil
}
code, msgCode, msg := b2.MsgCode(err)
if code == 0 {
// We don't interpret non B2 errors. B2 errors have statusCode
// to help us convert them to S3 object errors.
return err
}
objErr := b2MsgCodeToObjectError(code, msgCode, msg, params...)
if objErr == nil {
return err
}
return objErr
}
func b2MsgCodeToObjectError(code int, msgCode string, msg string, params ...string) error {
// Following code is a non-exhaustive check to convert
// B2 errors into S3 compatible errors.
//
// For a more complete information - https://www.backblaze.com/b2/docs/
var err error
bucket := ""
object := ""
uploadID := ""
@@ -160,18 +183,7 @@ func b2ToObjectError(err error, params ...string) error {
uploadID = params[2]
}
// Following code is a non-exhaustive check to convert
// B2 errors into S3 compatible errors.
//
// For a more complete information - https://www.backblaze.com/b2/docs/
statusCode, code, msg := b2.Code(err)
if statusCode == 0 {
// We don't interpret non B2 errors. B2 errors have statusCode
// to help us convert them to S3 object errors.
return err
}
switch code {
switch msgCode {
case "duplicate_bucket_name":
err = minio.BucketAlreadyOwnedByYou{Bucket: bucket}
case "bad_request":
@@ -343,7 +355,7 @@ func (l *b2Objects) ListObjects(ctx context.Context, bucket string, prefix strin
Name: file.Name,
ModTime: file.Timestamp,
Size: file.Size,
ETag: minio.ToS3ETag(file.Info.ID),
ETag: minio.ToS3ETag(file.ID),
ContentType: file.Info.ContentType,
UserDefined: file.Info.Info,
})
@@ -386,7 +398,7 @@ func (l *b2Objects) ListObjectsV2(ctx context.Context, bucket, prefix, continuat
Name: file.Name,
ModTime: file.Timestamp,
Size: file.Size,
ETag: minio.ToS3ETag(file.Info.ID),
ETag: minio.ToS3ETag(file.ID),
ContentType: file.Info.ContentType,
UserDefined: file.Info.Info,
})
@@ -462,7 +474,7 @@ func (l *b2Objects) GetObjectInfo(ctx context.Context, bucket string, object str
return minio.ObjectInfo{
Bucket: bucket,
Name: object,
ETag: minio.ToS3ETag(fi.ID),
ETag: minio.ToS3ETag(f.ID),
Size: fi.Size,
ModTime: fi.Timestamp,
ContentType: fi.ContentType,
@@ -569,7 +581,7 @@ func (l *b2Objects) PutObject(ctx context.Context, bucket string, object string,
return minio.ObjectInfo{
Bucket: bucket,
Name: object,
ETag: minio.ToS3ETag(fi.ID),
ETag: minio.ToS3ETag(f.ID),
Size: fi.Size,
ModTime: fi.Timestamp,
ContentType: fi.ContentType,
@@ -617,7 +629,7 @@ func (l *b2Objects) ListMultipartUploads(ctx context.Context, bucket string, pre
if maxUploads > 100 {
maxUploads = 100
}
largeFiles, nextMarker, err := bkt.ListUnfinishedLargeFiles(l.ctx, uploadIDMarker, maxUploads)
largeFiles, nextMarker, err := bkt.ListUnfinishedLargeFiles(l.ctx, maxUploads, uploadIDMarker)
if err != nil {
logger.LogIf(ctx, err)
return lmi, b2ToObjectError(err, bucket)
@@ -677,7 +689,7 @@ func (l *b2Objects) PutObjectPart(ctx context.Context, bucket string, object str
}
hr := newB2Reader(data, data.Size())
sha1, err := fc.UploadPart(l.ctx, hr, sha1AtEOF, int(hr.Size()), partID)
_, err = fc.UploadPart(l.ctx, hr, sha1AtEOF, int(hr.Size()), partID)
if err != nil {
logger.LogIf(ctx, err)
return pi, b2ToObjectError(err, bucket, object, uploadID)
@@ -686,7 +698,7 @@ func (l *b2Objects) PutObjectPart(ctx context.Context, bucket string, object str
return minio.PartInfo{
PartNumber: partID,
LastModified: minio.UTCNow(),
ETag: minio.ToS3ETag(sha1),
ETag: minio.ToS3ETag(fmt.Sprintf("%x", hr.sha1Hash.Sum(nil))),
Size: data.Size(),
}, nil
}
+94 -67
View File
@@ -20,8 +20,6 @@ import (
"fmt"
"testing"
b2 "github.com/minio/blazer/base"
minio "github.com/minio/minio/cmd"
)
@@ -41,71 +39,6 @@ func TestB2ObjectError(t *testing.T) {
{
[]string{}, fmt.Errorf("Non B2 Error"), fmt.Errorf("Non B2 Error"),
},
{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "duplicate_bucket_name",
}, minio.BucketAlreadyOwnedByYou{
Bucket: "bucket",
},
},
{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "bad_request",
}, minio.BucketNotFound{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"}, b2.Error{
StatusCode: 1,
Code: "bad_request",
}, minio.ObjectNameInvalid{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "bad_bucket_id",
}, minio.BucketNotFound{Bucket: "bucket"},
},
{
[]string{"bucket", "object"}, b2.Error{
StatusCode: 1,
Code: "file_not_present",
}, minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket", "object"}, b2.Error{
StatusCode: 1,
Code: "not_found",
}, minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket"}, b2.Error{
StatusCode: 1,
Code: "cannot_delete_non_empty_bucket",
}, minio.BucketNotEmpty{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object", "uploadID"}, b2.Error{
StatusCode: 1,
Message: "No active upload for",
}, minio.InvalidUploadID{
UploadID: "uploadID",
},
},
}
for i, testCase := range testCases {
@@ -117,3 +50,97 @@ func TestB2ObjectError(t *testing.T) {
}
}
}
// Test b2 msg code to object error.
func TestB2MsgCodeToObjectError(t *testing.T) {
testCases := []struct {
params []string
code int
msgCode string
msg string
expectedErr error
}{
{
[]string{"bucket"},
1,
"duplicate_bucket_name",
"",
minio.BucketAlreadyOwnedByYou{
Bucket: "bucket",
},
},
{
[]string{"bucket"},
1,
"bad_request",
"",
minio.BucketNotFound{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object"},
1,
"bad_request",
"",
minio.ObjectNameInvalid{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket"},
1,
"bad_bucket_id",
"",
minio.BucketNotFound{Bucket: "bucket"},
},
{
[]string{"bucket", "object"},
1,
"file_not_present",
"",
minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket", "object"},
1,
"not_found",
"",
minio.ObjectNotFound{
Bucket: "bucket",
Object: "object",
},
},
{
[]string{"bucket"},
1,
"cannot_delete_non_empty_bucket",
"",
minio.BucketNotEmpty{
Bucket: "bucket",
},
},
{
[]string{"bucket", "object", "uploadID"},
1,
"",
"No active upload for",
minio.InvalidUploadID{
UploadID: "uploadID",
},
},
}
for i, testCase := range testCases {
actualErr := b2MsgCodeToObjectError(testCase.code, testCase.msgCode, testCase.msg, testCase.params...)
if actualErr != nil {
if actualErr.Error() != testCase.expectedErr.Error() {
t.Errorf("Test %d: Expected %s, got %s", i+1, testCase.expectedErr, actualErr)
}
}
}
}
+17 -17
View File
@@ -77,37 +77,37 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for AWS S3 backend.
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}}
2. Start minio gateway server for S3 backend on custom endpoint.
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.HelpName}} https://play.min.io:9000
3. Start minio gateway server for AWS S3 backend logging all requests to http endpoint.
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}"http://localhost:8000/"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_LOGGER_HTTP_ENDPOINT{{.AssignmentOperator}}"http://localhost:8000/"
{{.Prompt}} {{.HelpName}} https://play.min.io:9000
4. Start minio gateway server for AWS S3 backend with edge caching enabled.
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}80
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}}
4. Start minio gateway server for AWS S3 backend using AWS environment variables.
NOTE: The access and secret key in this case will authenticate with MinIO instead
of AWS and AWS envs will be used to authenticate to AWS S3.
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}aws_access_key
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}aws_secret_key
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} AWS_ACCESS_KEY_ID{{.AssignmentOperator}}aws_access_key
{{.Prompt}} {{.EnvVarSetCommand}} AWS_SECRET_ACCESS_KEY{{.AssignmentOperator}}aws_secret_key
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}}
`
+15 -19
View File
@@ -28,6 +28,7 @@ import (
humanize "github.com/dustin/go-humanize"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/handlers"
@@ -261,10 +262,10 @@ func (h cacheControlHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if hasSuffix(r.URL.Path, ".js") || r.URL.Path == minioReservedBucketPath+"/favicon.ico" {
// For assets set cache expiry of one year. For each release, the name
// of the asset name will change and hence it can not be served from cache.
w.Header().Set("Cache-Control", "max-age=31536000")
w.Header().Set(xhttp.CacheControl, "max-age=31536000")
} else {
// For non asset requests we serve index.html which will never be cached.
w.Header().Set("Cache-Control", "no-store")
w.Header().Set(xhttp.CacheControl, "no-store")
}
}
}
@@ -380,15 +381,15 @@ type resourceHandler struct {
// setCorsHandler handler for CORS (Cross Origin Resource Sharing)
func setCorsHandler(h http.Handler) http.Handler {
commonS3Headers := []string{
"Date",
"ETag",
"Server",
"Connection",
"Accept-Ranges",
"Content-Range",
"Content-Encoding",
"Content-Length",
"Content-Type",
xhttp.Date,
xhttp.ETag,
xhttp.ServerInfo,
xhttp.Connection,
xhttp.AcceptRanges,
xhttp.ContentRange,
xhttp.ContentEncoding,
xhttp.ContentLength,
xhttp.ContentType,
"X-Amz*",
"x-amz*",
"*",
@@ -472,7 +473,6 @@ var notimplementedBucketResourceNames = map[string]bool{
"acl": true,
"cors": true,
"inventory": true,
"lifecycle": true,
"logging": true,
"metrics": true,
"replication": true,
@@ -750,7 +750,7 @@ func setBucketForwardingHandler(h http.Handler) http.Handler {
return bucketForwardingHandler{fwd, h}
}
// customHeaderHandler sets x-amz-request-id, x-minio-deployment-id header.
// customHeaderHandler sets x-amz-request-id header.
// Previously, this value was set right before a response was sent to
// the client. So, logger and Error response XML were not using this
// value. This is set here so that this header can be logged as
@@ -764,12 +764,8 @@ func addCustomHeaders(h http.Handler) http.Handler {
}
func (s customHeaderHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Set custom headers such as x-amz-request-id and x-minio-deployment-id
// for each request.
w.Header().Set(responseRequestIDKey, mustGetRequestID(UTCNow()))
if globalDeploymentID != "" {
w.Header().Set(responseDeploymentIDKey, globalDeploymentID)
}
// Set custom headers such as x-amz-request-id for each request.
w.Header().Set(xhttp.AmzRequestID, mustGetRequestID(UTCNow()))
s.handler.ServeHTTP(logger.NewResponseWriter(w), r)
}
+7
View File
@@ -83,6 +83,11 @@ const (
// GlobalMultipartCleanupInterval - Cleanup interval when the stale multipart cleanup is initiated.
GlobalMultipartCleanupInterval = time.Hour * 24 // 24 hrs.
// GlobalServiceExecutionInterval - Executes the Lifecycle events.
GlobalServiceExecutionInterval = time.Hour * 24 // 24 hrs.
// Refresh interval to update in-memory bucket lifecycle cache.
globalRefreshBucketLifecycleInterval = 5 * time.Minute
// Refresh interval to update in-memory iam config cache.
globalRefreshIAMInterval = 5 * time.Minute
@@ -148,6 +153,8 @@ var (
globalPolicySys *PolicySys
globalIAMSys *IAMSys
globalLifecycleSys *LifecycleSys
// CA root certificates, a nil value means system certs pool will be used
globalRootCAs *x509.CertPool
+30 -2
View File
@@ -27,6 +27,7 @@ import (
"net/url"
"strings"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/handlers"
@@ -219,8 +220,8 @@ func extractReqParams(r *http.Request) map[string]string {
func extractRespElements(w http.ResponseWriter) map[string]string {
return map[string]string{
"requestId": w.Header().Get(responseRequestIDKey),
"content-length": w.Header().Get("Content-Length"),
"requestId": w.Header().Get(xhttp.AmzRequestID),
"content-length": w.Header().Get(xhttp.ContentLength),
// Add more fields here.
}
}
@@ -383,3 +384,30 @@ func notFoundHandlerJSON(w http.ResponseWriter, r *http.Request) {
func notFoundHandler(w http.ResponseWriter, r *http.Request) {
writeErrorResponse(context.Background(), w, errorCodes.ToAPIErr(ErrMethodNotAllowed), r.URL, guessIsBrowserReq(r))
}
// gets host name for current node
func getHostName(r *http.Request) (hostName string) {
if globalIsDistXL {
hostName = GetLocalPeer(globalEndpoints)
} else {
hostName = r.Host
}
return
}
func isHTTPStatusOK(statusCode int) bool {
// List of success status.
var successStatus = []int{
http.StatusOK,
http.StatusCreated,
http.StatusAccepted,
http.StatusNoContent,
http.StatusPartialContent,
}
for _, okstatus := range successStatus {
if statusCode == okstatus {
return true
}
}
return false
}
+1
View File
@@ -62,6 +62,7 @@ func LivenessCheckHandler(w http.ResponseWriter, r *http.Request) {
if s.Backend.Type == Unknown {
// ListBuckets to confirm gateway backend is up
if _, err := objLayer.ListBuckets(ctx); err != nil {
logger.LogOnceIf(ctx, err, struct{}{})
writeResponse(w, http.StatusServiceUnavailable, nil, mimeNone)
return
}
+80 -43
View File
@@ -21,6 +21,7 @@ import (
"fmt"
"io"
"io/ioutil"
"net"
"net/http"
"reflect"
"runtime"
@@ -28,10 +29,11 @@ import (
"strings"
"time"
xnet "github.com/minio/minio/pkg/net"
trace "github.com/minio/minio/pkg/trace"
)
var traceBodyPlaceHolder = []byte("<BODY>")
// recordRequest - records the first recLen bytes
// of a given io.Reader
type recordRequest struct {
@@ -41,10 +43,16 @@ type recordRequest struct {
logBody bool
// Internal recording buffer
buf bytes.Buffer
// request headers
headers http.Header
// total bytes read including header size
bytesRead int
}
func (r *recordRequest) Read(p []byte) (n int, err error) {
n, err = r.Reader.Read(p)
r.bytesRead += n
if r.logBody {
r.buf.Write(p[:n])
}
@@ -53,10 +61,22 @@ func (r *recordRequest) Read(p []byte) (n int, err error) {
}
return n, err
}
func (r *recordRequest) Size() int {
sz := r.bytesRead
for k, v := range r.headers {
sz += len(k) + len(v)
}
return sz
}
// Return the bytes that were recorded.
func (r *recordRequest) Data() []byte {
return r.buf.Bytes()
// If body logging is enabled then we return the actual body
if r.logBody {
return r.buf.Bytes()
}
// ... otherwise we return <BODY> placeholder
return traceBodyPlaceHolder
}
// recordResponseWriter - records the first recLen bytes
@@ -73,13 +93,17 @@ type recordResponseWriter struct {
statusCode int
// Indicate if headers are written in the log
headersLogged bool
// number of bytes written
bytesWritten int
}
// Write the headers into the given buffer
func writeHeaders(w io.Writer, statusCode int, headers http.Header) {
fmt.Fprintf(w, "%d %s\n", statusCode, http.StatusText(statusCode))
func (r *recordResponseWriter) writeHeaders(w io.Writer, statusCode int, headers http.Header) {
n, _ := fmt.Fprintf(w, "%d %s\n", statusCode, http.StatusText(statusCode))
r.bytesWritten += n
for k, v := range headers {
fmt.Fprintf(w, "%s: %s\n", k, v[0])
n, _ := fmt.Fprintf(w, "%s: %s\n", k, v[0])
r.bytesWritten += n
}
}
@@ -87,7 +111,7 @@ func writeHeaders(w io.Writer, statusCode int, headers http.Header) {
func (r *recordResponseWriter) WriteHeader(i int) {
r.statusCode = i
if !r.headersLogged {
writeHeaders(&r.headers, i, r.ResponseWriter.Header())
r.writeHeaders(&r.headers, i, r.ResponseWriter.Header())
r.headersLogged = true
}
r.ResponseWriter.WriteHeader(i)
@@ -95,10 +119,11 @@ func (r *recordResponseWriter) WriteHeader(i int) {
func (r *recordResponseWriter) Write(p []byte) (n int, err error) {
n, err = r.ResponseWriter.Write(p)
r.bytesWritten += n
if !r.headersLogged {
// We assume the response code to be '200 OK' when WriteHeader() is not called,
// that way following Golang HTTP response behavior.
writeHeaders(&r.headers, http.StatusOK, r.ResponseWriter.Header())
r.writeHeaders(&r.headers, http.StatusOK, r.ResponseWriter.Header())
r.headersLogged = true
}
if (r.statusCode != http.StatusOK && r.statusCode != http.StatusPartialContent && r.statusCode != 0) || r.logBody {
@@ -108,6 +133,10 @@ func (r *recordResponseWriter) Write(p []byte) (n int, err error) {
return n, err
}
func (r *recordResponseWriter) Size() int {
return r.bytesWritten
}
// Calls the underlying Flush.
func (r *recordResponseWriter) Flush() {
r.ResponseWriter.(http.Flusher).Flush()
@@ -115,7 +144,13 @@ func (r *recordResponseWriter) Flush() {
// Return response body.
func (r *recordResponseWriter) Body() []byte {
return r.body.Bytes()
// If there was an error response or body logging is enabled
// then we return the body contents
if r.statusCode >= 400 || r.logBody {
return r.body.Bytes()
}
// ... otherwise we return the <BODY> place holder
return traceBodyPlaceHolder
}
// getOpName sanitizes the operation name for mc
@@ -136,55 +171,57 @@ func getOpName(name string) (op string) {
// Trace gets trace of http request
func Trace(f http.HandlerFunc, logBody bool, w http.ResponseWriter, r *http.Request) trace.Info {
name := getOpName(runtime.FuncForPC(reflect.ValueOf(f).Pointer()).Name())
bodyPlaceHolder := []byte("<BODY>")
var reqBodyRecorder *recordRequest
t := trace.Info{FuncName: name}
reqBodyRecorder = &recordRequest{Reader: r.Body, logBody: logBody}
r.Body = ioutil.NopCloser(reqBodyRecorder)
host, err := xnet.ParseHost(GetLocalPeer(globalEndpoints))
if err == nil {
t.NodeName = host.Name
}
rq := trace.RequestInfo{Time: time.Now().UTC(), Method: r.Method, Path: r.URL.Path, RawQuery: r.URL.RawQuery, Client: r.RemoteAddr}
rq.Headers = cloneHeader(r.Header)
rq.Headers.Set("Content-Length", strconv.Itoa(int(r.ContentLength)))
rq.Headers.Set("Host", r.Host)
// Setup a http request body recorder
reqHeaders := cloneHeader(r.Header)
reqHeaders.Set("Content-Length", strconv.Itoa(int(r.ContentLength)))
reqHeaders.Set("Host", r.Host)
for _, enc := range r.TransferEncoding {
rq.Headers.Add("Transfer-Encoding", enc)
reqHeaders.Add("Transfer-Encoding", enc)
}
if logBody {
// If body logging is disabled then we print <BODY> as a placeholder
// for the actual body.
rq.Body = reqBodyRecorder.Data()
} else {
rq.Body = bodyPlaceHolder
var reqBodyRecorder *recordRequest
t := trace.Info{FuncName: name}
reqBodyRecorder = &recordRequest{Reader: r.Body, logBody: logBody, headers: reqHeaders}
r.Body = ioutil.NopCloser(reqBodyRecorder)
t.NodeName = r.Host
if globalIsDistXL {
t.NodeName = GetLocalPeer(globalEndpoints)
}
// strip port from the host address
if host, _, err := net.SplitHostPort(t.NodeName); err == nil {
t.NodeName = host
}
rq := trace.RequestInfo{
Time: time.Now().UTC(),
Method: r.Method,
Path: r.URL.Path,
RawQuery: r.URL.RawQuery,
Client: r.RemoteAddr,
Headers: reqHeaders,
Body: reqBodyRecorder.Data(),
}
// Setup a http response body recorder
respBodyRecorder := &recordResponseWriter{ResponseWriter: w, logBody: logBody}
f(respBodyRecorder, r)
rs := trace.ResponseInfo{Time: time.Now().UTC()}
rs.Headers = cloneHeader(respBodyRecorder.Header())
rs.StatusCode = respBodyRecorder.statusCode
rs := trace.ResponseInfo{
Time: time.Now().UTC(),
Headers: cloneHeader(respBodyRecorder.Header()),
StatusCode: respBodyRecorder.statusCode,
Body: respBodyRecorder.Body(),
}
if rs.StatusCode == 0 {
rs.StatusCode = http.StatusOK
}
bodyContents := respBodyRecorder.Body()
if bodyContents != nil {
rs.Body = bodyContents
}
if !logBody {
// If there was no error response and body logging is disabled
// then we print <BODY> as a placeholder for the actual body.
rs.Body = bodyPlaceHolder
}
t.ReqInfo = rq
t.RespInfo = rs
t.CallStats = trace.CallStats{Latency: rs.Time.Sub(rq.Time), InputBytes: reqBodyRecorder.Size(), OutputBytes: respBodyRecorder.Size()}
return t
}
+81
View File
@@ -0,0 +1,81 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package http
// Standard S3 HTTP response constants
const (
LastModified = "Last-Modified"
Date = "Date"
ETag = "ETag"
ContentType = "Content-Type"
ContentMD5 = "Content-Md5"
ContentEncoding = "Content-Encoding"
Expires = "Expires"
ContentLength = "Content-Length"
ContentLanguage = "Content-Language"
ContentRange = "Content-Range"
Connection = "Connection"
AcceptRanges = "Accept-Ranges"
AmzBucketRegion = "X-Amz-Bucket-Region"
ServerInfo = "Server"
RetryAfter = "Retry-After"
Location = "Location"
CacheControl = "Cache-Control"
ContentDisposition = "Content-Disposition"
Authorization = "Authorization"
Action = "Action"
)
// Standard S3 HTTP request constants
const (
IfModifiedSince = "If-Modified-Since"
IfUnmodifiedSince = "If-Unmodified-Since"
IfMatch = "If-Match"
IfNoneMatch = "If-None-Match"
// S3 extensions
AmzCopySourceIfModifiedSince = "x-amz-copy-source-if-modified-since"
AmzCopySourceIfUnmodifiedSince = "x-amz-copy-source-if-unmodified-since"
AmzCopySourceIfNoneMatch = "x-amz-copy-source-if-none-match"
AmzCopySourceIfMatch = "x-amz-copy-source-if-match"
AmzCopySource = "X-Amz-Copy-Source"
AmzCopySourceVersionID = "X-Amz-Copy-Source-Version-Id"
AmzCopySourceRange = "X-Amz-Copy-Source-Range"
// Signature V4 related contants.
AmzContentSha256 = "X-Amz-Content-Sha256"
AmzDate = "X-Amz-Date"
AmzAlgorithm = "X-Amz-Algorithm"
AmzExpires = "X-Amz-Expires"
AmzSignedHeaders = "X-Amz-SignedHeaders"
AmzSignature = "X-Amz-Signature"
AmzCredential = "X-Amz-Credential"
AmzSecurityToken = "X-Amz-Security-Token"
AmzDecodedContentLength = "X-Amz-Decoded-Content-Length"
// Signature v2 related constants
AmzSignatureV2 = "Signature"
AmzAccessKeyID = "AWSAccessKeyId"
// Response request id.
AmzRequestID = "x-amz-request-id"
// Deployment id.
MinioDeploymentID = "x-minio-deployment-id"
)
+191
View File
@@ -0,0 +1,191 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"bytes"
"context"
"encoding/xml"
"path"
"strings"
"sync"
"time"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lifecycle"
)
const (
// Disabled means the lifecycle rule is inactive
Disabled = "Disabled"
)
// LifecycleSys - Bucket lifecycle subsystem.
type LifecycleSys struct {
sync.RWMutex
bucketLifecycleMap map[string]lifecycle.Lifecycle
}
// Set - sets lifecycle config to given bucket name.
func (sys *LifecycleSys) Set(bucketName string, lifecycle lifecycle.Lifecycle) {
sys.Lock()
defer sys.Unlock()
sys.bucketLifecycleMap[bucketName] = lifecycle
}
func saveLifecycleConfig(ctx context.Context, objAPI ObjectLayer, bucketName string, bucketLifecycle *lifecycle.Lifecycle) error {
data, err := xml.Marshal(bucketLifecycle)
if err != nil {
return err
}
// Construct path to lifecycle.xml for the given bucket.
configFile := path.Join(bucketConfigPrefix, bucketName, bucketLifecycleConfig)
return saveConfig(ctx, objAPI, configFile, data)
}
// getLifecycleConfig - get lifecycle config for given bucket name.
func getLifecycleConfig(objAPI ObjectLayer, bucketName string) (*lifecycle.Lifecycle, error) {
// Construct path to lifecycle.xml for the given bucket.
configFile := path.Join(bucketConfigPrefix, bucketName, bucketLifecycleConfig)
configData, err := readConfig(context.Background(), objAPI, configFile)
if err != nil {
if err == errConfigNotFound {
err = BucketLifecycleNotFound{Bucket: bucketName}
}
return nil, err
}
return lifecycle.ParseLifecycleConfig(bytes.NewReader(configData))
}
func removeLifecycleConfig(ctx context.Context, objAPI ObjectLayer, bucketName string) error {
// Construct path to lifecycle.xml for the given bucket.
configFile := path.Join(bucketConfigPrefix, bucketName, bucketLifecycleConfig)
if err := objAPI.DeleteObject(ctx, minioMetaBucket, configFile); err != nil {
if _, ok := err.(ObjectNotFound); ok {
return BucketLifecycleNotFound{Bucket: bucketName}
}
return err
}
return nil
}
// NewLifecycleSys - creates new lifecycle system.
func NewLifecycleSys() *LifecycleSys {
return &LifecycleSys{
bucketLifecycleMap: make(map[string]lifecycle.Lifecycle),
}
}
// Init - initializes lifecycle system from lifecycle.xml of all buckets.
func (sys *LifecycleSys) Init(objAPI ObjectLayer) error {
if objAPI == nil {
return errServerNotInitialized
}
defer func() {
// Refresh LifecycleSys in background.
go func() {
ticker := time.NewTicker(globalRefreshBucketLifecycleInterval)
defer ticker.Stop()
for {
select {
case <-GlobalServiceDoneCh:
return
case <-ticker.C:
sys.refresh(objAPI)
}
}
}()
}()
doneCh := make(chan struct{})
defer close(doneCh)
// Initializing lifecycle needs a retry mechanism for
// the following reasons:
// - Read quorum is lost just after the initialization
// of the object layer.
for range newRetryTimerSimple(doneCh) {
// Load LifecycleSys once during boot.
if err := sys.refresh(objAPI); err != nil {
if err == errDiskNotFound ||
strings.Contains(err.Error(), InsufficientReadQuorum{}.Error()) ||
strings.Contains(err.Error(), InsufficientWriteQuorum{}.Error()) {
logger.Info("Waiting for lifecycle subsystem to be initialized..")
continue
}
return err
}
break
}
return nil
}
// Refresh LifecycleSys.
func (sys *LifecycleSys) refresh(objAPI ObjectLayer) error {
buckets, err := objAPI.ListBuckets(context.Background())
if err != nil {
logger.LogIf(context.Background(), err)
return err
}
sys.removeDeletedBuckets(buckets)
for _, bucket := range buckets {
config, err := objAPI.GetBucketLifecycle(context.Background(), bucket.Name)
if err != nil {
if _, ok := err.(BucketLifecycleNotFound); ok {
sys.Remove(bucket.Name)
}
continue
}
sys.Set(bucket.Name, *config)
}
return nil
}
// removeDeletedBuckets - to handle a corner case where we have cached the lifecycle for a deleted
// bucket. i.e if we miss a delete-bucket notification we should delete the corresponding
// bucket policy during sys.refresh()
func (sys *LifecycleSys) removeDeletedBuckets(bucketInfos []BucketInfo) {
buckets := set.NewStringSet()
for _, info := range bucketInfos {
buckets.Add(info.Name)
}
sys.Lock()
defer sys.Unlock()
for bucket := range sys.bucketLifecycleMap {
if !buckets.Contains(bucket) {
delete(sys.bucketLifecycleMap, bucket)
}
}
}
// Remove - removes policy for given bucket name.
func (sys *LifecycleSys) Remove(bucketName string) {
sys.Lock()
defer sys.Unlock()
delete(sys.bucketLifecycleMap, bucketName)
}
+1 -1
View File
@@ -21,7 +21,7 @@ import (
"sync"
"time"
"github.com/minio/dsync"
"github.com/minio/dsync/v2"
)
// lockRequesterInfo stores various info from the client for each lock that is requested.
+1 -1
View File
@@ -28,7 +28,7 @@ import (
"net/url"
"github.com/minio/dsync"
"github.com/minio/dsync/v2"
"github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/cmd/rest"
+1 -1
View File
@@ -19,7 +19,7 @@ package cmd
import (
"testing"
"github.com/minio/dsync"
"github.com/minio/dsync/v2"
xnet "github.com/minio/minio/pkg/net"
)
+1 -1
View File
@@ -25,7 +25,7 @@ import (
"time"
"github.com/gorilla/mux"
"github.com/minio/dsync"
"github.com/minio/dsync/v2"
"github.com/minio/minio/cmd/logger"
xnet "github.com/minio/minio/pkg/net"
)
+1 -1
View File
@@ -63,6 +63,6 @@ func AuditLog(w http.ResponseWriter, r *http.Request, api string, reqClaims map[
}
// Send audit logs only to http targets.
for _, t := range AuditTargets {
_ = t.Send(audit.ToEntry(w, r, api, statusCode, reqClaims))
_ = t.Send(audit.ToEntry(w, r, api, statusCode, reqClaims, globalDeploymentID))
}
}
+11 -1
View File
@@ -55,6 +55,8 @@ const (
var trimStrings []string
var globalDeploymentID string
// TimeFormat - logging time format.
const TimeFormat string = "15:04:05 MST 01/02/2006"
@@ -154,6 +156,11 @@ func uniqueEntries(paths []string) []string {
return m.ToSlice()
}
// SetDeploymentID -- Deployment Id from the main package is set here
func SetDeploymentID(deploymentID string) {
globalDeploymentID = deploymentID
}
// Init sets the trimStrings to possible GOPATHs
// and GOROOT directories. Also append github.com/minio/minio
// This is done to clean up the filename, when stack trace is
@@ -317,11 +324,14 @@ func logIf(ctx context.Context, err error) {
// Get the cause for the Error
message := err.Error()
if req.DeploymentID == "" {
req.DeploymentID = globalDeploymentID
}
entry := log.Entry{
DeploymentID: req.DeploymentID,
Level: ErrorLvl.String(),
RemoteHost: req.RemoteHost,
Host: req.Host,
RequestID: req.RequestID,
UserAgent: req.UserAgent,
Time: time.Now().UTC().Format(time.RFC3339Nano),
+5 -4
View File
@@ -22,6 +22,7 @@ import (
"time"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/pkg/handlers"
)
@@ -50,7 +51,7 @@ type Entry struct {
}
// ToEntry - constructs an audit entry object.
func ToEntry(w http.ResponseWriter, r *http.Request, api string, statusCode int, reqClaims map[string]interface{}) Entry {
func ToEntry(w http.ResponseWriter, r *http.Request, api string, statusCode int, reqClaims map[string]interface{}, deploymentID string) Entry {
vars := mux.Vars(r)
bucket := vars["bucket"]
object := vars["object"]
@@ -67,13 +68,13 @@ func ToEntry(w http.ResponseWriter, r *http.Request, api string, statusCode int,
for k, v := range w.Header() {
respHeader[k] = strings.Join(v, ",")
}
respHeader["Etag"] = strings.Trim(respHeader["Etag"], `"`)
respHeader[xhttp.ETag] = strings.Trim(respHeader[xhttp.ETag], `"`)
entry := Entry{
Version: Version,
DeploymentID: w.Header().Get("x-minio-deployment-id"),
DeploymentID: deploymentID,
RemoteHost: handlers.GetSourceIP(r),
RequestID: w.Header().Get("x-amz-request-id"),
RequestID: w.Header().Get(xhttp.AmzRequestID),
UserAgent: r.UserAgent(),
Time: time.Now().UTC().Format(time.RFC3339Nano),
ReqQuery: reqQuery,
+1
View File
@@ -43,6 +43,7 @@ type Entry struct {
Time string `json:"time"`
API *API `json:"api,omitempty"`
RemoteHost string `json:"remotehost,omitempty"`
Host string `json:"host,omitempty"`
RequestID string `json:"requestID,omitempty"`
UserAgent string `json:"userAgent,omitempty"`
Message string `json:"message,omitempty"`
+1
View File
@@ -36,6 +36,7 @@ type KeyVal struct {
// ReqInfo stores the request info.
type ReqInfo struct {
RemoteHost string // Client Host/IP
Host string // Node Host/IP
UserAgent string // User Agent
DeploymentID string // x-minio-deployment-id
RequestID string // x-amz-request-id
+7 -2
View File
@@ -89,6 +89,11 @@ func (c *Target) Send(e interface{}) error {
remoteHost = "\nRemoteHost: " + entry.RemoteHost
}
var host string
if entry.Host != "" {
host = "\nHost: " + entry.Host
}
var userAgent string
if entry.UserAgent != "" {
userAgent = "\nUserAgent: " + entry.UserAgent
@@ -99,8 +104,8 @@ func (c *Target) Send(e interface{}) error {
}
var msg = logger.ColorFgRed(logger.ColorBold(entry.Trace.Message))
var output = fmt.Sprintf("\n%s\n%s%s%s%s%s\nError: %s%s\n%s",
apiString, timeString, deploymentID, requestID, remoteHost, userAgent,
var output = fmt.Sprintf("\n%s\n%s%s%s%s%s%s\nError: %s%s\n%s",
apiString, timeString, deploymentID, requestID, remoteHost, host, userAgent,
msg, tagString, strings.Join(trace, "\n"))
fmt.Println(output)
+3 -2
View File
@@ -20,6 +20,7 @@ import (
"bytes"
"encoding/json"
"errors"
"net/http"
gohttp "net/http"
xhttp "github.com/minio/minio/cmd/http"
@@ -49,11 +50,11 @@ func (h *Target) startHTTPLogger() {
continue
}
req, err := gohttp.NewRequest("POST", h.endpoint, bytes.NewBuffer(logJSON))
req, err := gohttp.NewRequest(http.MethodPost, h.endpoint, bytes.NewBuffer(logJSON))
if err != nil {
continue
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set(xhttp.ContentType, "application/json")
resp, err := h.client.Do(req)
if err != nil {
+12 -11
View File
@@ -27,7 +27,7 @@ import (
"fmt"
"time"
"github.com/minio/dsync"
"github.com/minio/dsync/v2"
"github.com/minio/lsync"
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/cmd/logger"
@@ -127,7 +127,7 @@ type nsLockMap struct {
}
// Lock the namespace resource.
func (n *nsLockMap) lock(volume, path string, lockSource, opsID string, readLock bool, timeout time.Duration) (locked bool) {
func (n *nsLockMap) lock(ctx context.Context, volume, path string, lockSource, opsID string, readLock bool, timeout time.Duration) (locked bool) {
var nsLk *nsLock
n.lockMapMutex.Lock()
@@ -135,7 +135,7 @@ func (n *nsLockMap) lock(volume, path string, lockSource, opsID string, readLock
nsLk, found := n.lockMap[param]
if !found {
n.lockMap[param] = &nsLock{
LRWMutex: &lsync.LRWMutex{},
LRWMutex: lsync.NewLRWMutex(ctx),
ref: 1,
}
nsLk = n.lockMap[param]
@@ -199,7 +199,7 @@ func (n *nsLockMap) Lock(volume, path, opsID string, timeout time.Duration) (loc
readLock := false // This is a write lock.
lockSource := getSource() // Useful for debugging
return n.lock(volume, path, lockSource, opsID, readLock, timeout)
return n.lock(context.Background(), volume, path, lockSource, opsID, readLock, timeout)
}
// Unlock - unlocks any previously acquired write locks.
@@ -213,7 +213,7 @@ func (n *nsLockMap) RLock(volume, path, opsID string, timeout time.Duration) (lo
readLock := true
lockSource := getSource() // Useful for debugging
return n.lock(volume, path, lockSource, opsID, readLock, timeout)
return n.lock(context.Background(), volume, path, lockSource, opsID, readLock, timeout)
}
// RUnlock - unlocks any previously acquired read locks.
@@ -241,7 +241,7 @@ func (n *nsLockMap) ForceUnlock(volume, path string) {
// are blocking can now proceed as normal and any new locks will also
// participate normally.
if n.isDistXL { // For distributed mode, broadcast ForceUnlock message.
dsync.NewDRWMutex(pathJoin(volume, path), globalDsync).ForceUnlock()
dsync.NewDRWMutex(context.Background(), pathJoin(volume, path), globalDsync).ForceUnlock()
}
// Remove lock from the map.
@@ -291,6 +291,7 @@ func (di *distLockInstance) RUnlock() {
// localLockInstance - frontend/top-level interface for namespace locks.
type localLockInstance struct {
ctx context.Context
ns *nsLockMap
volume, path, opsID string
}
@@ -298,12 +299,12 @@ type localLockInstance struct {
// NewNSLock - returns a lock instance for a given volume and
// path. The returned lockInstance object encapsulates the nsLockMap,
// volume, path and operation ID.
func (n *nsLockMap) NewNSLock(volume, path string) RWLocker {
func (n *nsLockMap) NewNSLock(ctx context.Context, volume, path string) RWLocker {
opsID := mustGetUUID()
if n.isDistXL {
return &distLockInstance{dsync.NewDRWMutex(pathJoin(volume, path), globalDsync), volume, path, opsID}
return &distLockInstance{dsync.NewDRWMutex(ctx, pathJoin(volume, path), globalDsync), volume, path, opsID}
}
return &localLockInstance{n, volume, path, opsID}
return &localLockInstance{ctx, n, volume, path, opsID}
}
// Lock - block until write lock is taken or timeout has occurred.
@@ -311,7 +312,7 @@ func (li *localLockInstance) GetLock(timeout *dynamicTimeout) (timedOutErr error
lockSource := getSource()
start := UTCNow()
readLock := false
if !li.ns.lock(li.volume, li.path, lockSource, li.opsID, readLock, timeout.Timeout()) {
if !li.ns.lock(li.ctx, li.volume, li.path, lockSource, li.opsID, readLock, timeout.Timeout()) {
timeout.LogFailure()
return OperationTimedOut{Path: li.path}
}
@@ -330,7 +331,7 @@ func (li *localLockInstance) GetRLock(timeout *dynamicTimeout) (timedOutErr erro
lockSource := getSource()
start := UTCNow()
readLock := true
if !li.ns.lock(li.volume, li.path, lockSource, li.opsID, readLock, timeout.Timeout()) {
if !li.ns.lock(li.ctx, li.volume, li.path, lockSource, li.opsID, readLock, timeout.Timeout()) {
timeout.LogFailure()
return OperationTimedOut{Path: li.path}
}
+4 -3
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"testing"
"time"
)
@@ -31,7 +32,7 @@ func TestGetSource(t *testing.T) {
currentSource := func() string { return getSource() }
gotSource := currentSource()
// Hard coded line number, 32, in the "expectedSource" value
expectedSource := "[namespace-lock_test.go:32:TestGetSource()]"
expectedSource := "[namespace-lock_test.go:33:TestGetSource()]"
if gotSource != expectedSource {
t.Errorf("expected : %s, got : %s", expectedSource, gotSource)
}
@@ -204,7 +205,7 @@ func TestNamespaceForceUnlockTest(t *testing.T) {
isDistXL := false
initNSLock(isDistXL)
// Create lock.
lock := globalNSMutex.NewNSLock("bucket", "object")
lock := globalNSMutex.NewNSLock(context.Background(), "bucket", "object")
if lock.GetLock(newDynamicTimeout(60*time.Second, time.Second)) != nil {
t.Fatalf("Failed to get lock")
}
@@ -215,7 +216,7 @@ func TestNamespaceForceUnlockTest(t *testing.T) {
go func() {
// Try to claim lock again.
anotherLock := globalNSMutex.NewNSLock("bucket", "object")
anotherLock := globalNSMutex.NewNSLock(context.Background(), "bucket", "object")
if anotherLock.GetLock(newDynamicTimeout(60*time.Second, time.Second)) != nil {
t.Errorf("Failed to get lock")
return
+7
View File
@@ -195,3 +195,10 @@ func (d *naughtyDisk) ReadAll(volume string, path string) (buf []byte, err error
}
return d.disk.ReadAll(volume, path)
}
func (d *naughtyDisk) VerifyFile(volume, path string, empty bool, algo BitrotAlgorithm, sum []byte, shardSize int64) error {
if err := d.calcError(); err != nil {
return err
}
return d.disk.VerifyFile(volume, path, empty, algo, sum, shardSize)
}
+46 -3
View File
@@ -34,6 +34,7 @@ import (
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/lifecycle"
"github.com/minio/minio/pkg/madmin"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
@@ -498,6 +499,48 @@ func (sys *NotificationSys) RemoveBucketPolicy(ctx context.Context, bucketName s
}()
}
// SetBucketLifecycle - calls SetBucketLifecycle on all peers.
func (sys *NotificationSys) SetBucketLifecycle(ctx context.Context, bucketName string, bucketLifecycle *lifecycle.Lifecycle) {
go func() {
var wg sync.WaitGroup
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.SetBucketLifecycle(bucketName, bucketLifecycle); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(client)
}
wg.Wait()
}()
}
// RemoveBucketLifecycle - calls RemoveLifecycle on all peers.
func (sys *NotificationSys) RemoveBucketLifecycle(ctx context.Context, bucketName string) {
go func() {
var wg sync.WaitGroup
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.RemoveBucketLifecycle(bucketName); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(client)
}
wg.Wait()
}()
}
// PutBucketNotification - calls PutBucketNotification RPC call on all peers.
func (sys *NotificationSys) PutBucketNotification(ctx context.Context, bucketName string, rulesMap event.RulesMap) {
go func() {
@@ -602,7 +645,7 @@ func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLaye
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(ctx, minioMetaBucket, transactionConfigFile)
if err := objLock.GetRLock(globalOperationTimeout); err != nil {
return err
}
@@ -1058,7 +1101,7 @@ func SaveListener(objAPI ObjectLayer, bucketName string, eventNames []event.Name
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(ctx, minioMetaBucket, transactionConfigFile)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
@@ -1109,7 +1152,7 @@ func RemoveListener(objAPI ObjectLayer, bucketName string, targetID event.Target
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(ctx, minioMetaBucket, transactionConfigFile)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
+89
View File
@@ -21,6 +21,8 @@ import (
"path"
"sync"
"strings"
humanize "github.com/dustin/go-humanize"
"github.com/minio/minio/cmd/logger"
)
@@ -234,7 +236,94 @@ func removeListenerConfig(ctx context.Context, objAPI ObjectLayer, bucket string
return objAPI.DeleteObject(ctx, minioMetaBucket, lcPath)
}
func listObjectsNonSlash(ctx context.Context, obj ObjectLayer, bucket, prefix, marker, delimiter string, maxKeys int, tpool *TreeWalkPool, listDir ListDirFunc, getObjInfo func(context.Context, string, string) (ObjectInfo, error), getObjectInfoDirs ...func(context.Context, string, string) (ObjectInfo, error)) (loi ListObjectsInfo, err error) {
endWalkCh := make(chan struct{})
defer close(endWalkCh)
recursive := true
walkResultCh := startTreeWalk(ctx, bucket, prefix, "", recursive, listDir, endWalkCh)
var objInfos []ObjectInfo
var eof bool
var prevPrefix string
for {
if len(objInfos) == maxKeys {
break
}
result, ok := <-walkResultCh
if !ok {
eof = true
break
}
var objInfo ObjectInfo
var err error
index := strings.Index(strings.TrimPrefix(result.entry, prefix), delimiter)
if index == -1 {
objInfo, err = getObjInfo(ctx, bucket, result.entry)
if err != nil {
// Ignore errFileNotFound as the object might have got
// deleted in the interim period of listing and getObjectInfo(),
// ignore quorum error as it might be an entry from an outdated disk.
if IsErrIgnored(err, []error{
errFileNotFound,
errXLReadQuorum,
}...) {
continue
}
return loi, toObjectErr(err, bucket, prefix)
}
} else {
index = len(prefix) + index + len(delimiter)
currPrefix := result.entry[:index]
if currPrefix == prevPrefix {
continue
}
prevPrefix = currPrefix
objInfo = ObjectInfo{
Bucket: bucket,
Name: currPrefix,
IsDir: true,
}
}
if objInfo.Name <= marker {
continue
}
objInfos = append(objInfos, objInfo)
if result.end {
eof = true
break
}
}
result := ListObjectsInfo{}
for _, objInfo := range objInfos {
if objInfo.IsDir {
result.Prefixes = append(result.Prefixes, objInfo.Name)
continue
}
result.Objects = append(result.Objects, objInfo)
}
if !eof {
result.IsTruncated = true
if len(objInfos) > 0 {
result.NextMarker = objInfos[len(objInfos)-1].Name
}
}
return result, nil
}
func listObjects(ctx context.Context, obj ObjectLayer, bucket, prefix, marker, delimiter string, maxKeys int, tpool *TreeWalkPool, listDir ListDirFunc, getObjInfo func(context.Context, string, string) (ObjectInfo, error), getObjectInfoDirs ...func(context.Context, string, string) (ObjectInfo, error)) (loi ListObjectsInfo, err error) {
if delimiter != slashSeparator && delimiter != "" {
return listObjectsNonSlash(ctx, obj, bucket, prefix, marker, delimiter, maxKeys, tpool, listDir, getObjInfo, getObjectInfoDirs...)
}
if err := checkListObjsArgs(ctx, bucket, prefix, marker, delimiter, obj); err != nil {
return loi, err
}
+23
View File
@@ -254,6 +254,13 @@ func (e BucketPolicyNotFound) Error() string {
return "No bucket policy found for bucket: " + e.Bucket
}
// BucketLifecycleNotFound - no bucket lifecycle found.
type BucketLifecycleNotFound GenericError
func (e BucketLifecycleNotFound) Error() string {
return "No bucket life cycle found for bucket : " + e.Bucket
}
/// Bucket related errors.
// BucketNameInvalid - bucketname provided is invalid.
@@ -269,11 +276,27 @@ func (e BucketNameInvalid) Error() string {
// ObjectNameInvalid - object name provided is invalid.
type ObjectNameInvalid GenericError
// ObjectNameTooLong - object name too long.
type ObjectNameTooLong GenericError
// ObjectNamePrefixAsSlash - object name has a slash as prefix.
type ObjectNamePrefixAsSlash GenericError
// Return string an error formatted as the given text.
func (e ObjectNameInvalid) Error() string {
return "Object name invalid: " + e.Bucket + "#" + e.Object
}
// Return string an error formatted as the given text.
func (e ObjectNameTooLong) Error() string {
return "Object name too long: " + e.Bucket + "#" + e.Object
}
// Return string an error formatted as the given text.
func (e ObjectNamePrefixAsSlash) Error() string {
return "Object name contains forward slash as pefix: " + e.Bucket + "#" + e.Object
}
// AllAccessDisabled All access to this object has been disabled
type AllAccessDisabled GenericError
+8 -6
View File
@@ -166,18 +166,18 @@ func checkObjectArgs(ctx context.Context, bucket, object string, obj ObjectLayer
if err := checkBucketExist(ctx, bucket, obj); err != nil {
return err
}
if err := checkObjectNameForLengthAndSlash(bucket, object); err != nil {
return err
}
// Validates object name validity after bucket exists.
if !IsValidObjectName(object) {
logger.LogIf(ctx, ObjectNameInvalid{
Bucket: bucket,
Object: object,
})
return ObjectNameInvalid{
Bucket: bucket,
Object: object,
}
}
return nil
}
@@ -192,8 +192,10 @@ func checkPutObjectArgs(ctx context.Context, bucket, object string, obj ObjectLa
return err
}
if err := checkObjectNameForLengthAndSlash(bucket, object); err != nil {
return err
}
if len(object) == 0 ||
hasPrefix(object, slashSeparator) ||
(hasSuffix(object, slashSeparator) && size != 0) ||
!IsValidObjectPrefix(object) {
return ObjectNameInvalid{
+6
View File
@@ -22,6 +22,7 @@ import (
"net/http"
"github.com/minio/minio-go/v6/pkg/encrypt"
"github.com/minio/minio/pkg/lifecycle"
"github.com/minio/minio/pkg/madmin"
"github.com/minio/minio/pkg/policy"
)
@@ -107,4 +108,9 @@ type ObjectLayer interface {
// Compression support check.
IsCompressionSupported() bool
// Lifecycle operations
SetBucketLifecycle(context.Context, string, *lifecycle.Lifecycle) error
GetBucketLifecycle(context.Context, string) (*lifecycle.Lifecycle, error)
DeleteBucketLifecycle(context.Context, string) error
}
-4
View File
@@ -471,10 +471,6 @@ func testListObjects(obj ObjectLayer, instanceType string, t1 TestErrHandler) {
{"volatile-bucket-1", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "volatile-bucket-1"}, false},
{"volatile-bucket-2", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "volatile-bucket-2"}, false},
{"volatile-bucket-3", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "volatile-bucket-3"}, false},
// Valid, existing bucket, but sending invalid delimeter values (9-10).
// Empty string < "" > and forward slash < / > are the ony two valid arguments for delimeter.
{"test-bucket-list-object", "", "", "*", 0, ListObjectsInfo{}, fmt.Errorf("delimiter '%s' is not supported", "*"), false},
{"test-bucket-list-object", "", "", "-", 0, ListObjectsInfo{}, fmt.Errorf("delimiter '%s' is not supported", "-"), false},
// Testing for failure cases with both perfix and marker (11).
// The prefix and marker combination to be valid it should satisfy strings.HasPrefix(marker, prefix).
{"test-bucket-list-object", "asia", "europe-object", "", 0, ListObjectsInfo{}, fmt.Errorf("Invalid combination of marker '%s' and prefix '%s'", "europe-object", "asia"), false},
+37 -5
View File
@@ -36,6 +36,7 @@ import (
snappy "github.com/golang/snappy"
"github.com/minio/minio-go/v6/pkg/s3utils"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/hash"
@@ -135,7 +136,7 @@ func IsValidObjectName(object string) bool {
if len(object) == 0 {
return false
}
if hasSuffix(object, slashSeparator) || hasPrefix(object, slashSeparator) {
if hasSuffix(object, slashSeparator) {
return false
}
return IsValidObjectPrefix(object)
@@ -147,9 +148,6 @@ func IsValidObjectPrefix(object string) bool {
if hasBadPathComponent(object) {
return false
}
if len(object) > 1024 {
return false
}
if !utf8.ValidString(object) {
return false
}
@@ -160,6 +158,25 @@ func IsValidObjectPrefix(object string) bool {
return true
}
// checkObjectNameForLengthAndSlash -check for the validity of object name length and prefis as slash
func checkObjectNameForLengthAndSlash(bucket, object string) error {
// Check for the length of object name
if len(object) > 1024 {
return ObjectNameTooLong{
Bucket: bucket,
Object: object,
}
}
// Check for slash as prefix in object name
if hasPrefix(object, slashSeparator) {
return ObjectNamePrefixAsSlash{
Bucket: bucket,
Object: object,
}
}
return nil
}
// Slash separator.
const slashSeparator = "/"
@@ -346,7 +363,7 @@ func isCompressible(header http.Header, object string) bool {
// Eliminate the non-compressible objects.
func excludeForCompression(header http.Header, object string) bool {
objStr := object
contentType := header.Get("Content-Type")
contentType := header.Get(xhttp.ContentType)
if globalIsCompressionEnabled {
// We strictly disable compression for standard extensions/content-types (`compressed`).
if hasStringSuffixInSlice(objStr, standardExcludeCompressExtensions) || hasPattern(standardExcludeCompressContentTypes, contentType) {
@@ -793,3 +810,18 @@ func (cr *snappyCompressReader) Read(p []byte) (int, error) {
}
return n, err
}
// Returns error if the cancelCh has been closed (indicating that S3 client has disconnected)
type detectDisconnect struct {
io.ReadCloser
cancelCh <-chan struct{}
}
func (d *detectDisconnect) Read(p []byte) (int, error) {
select {
case <-d.cancelCh:
return 0, io.ErrUnexpectedEOF
default:
return d.ReadCloser.Read(p)
}
}
-1
View File
@@ -113,7 +113,6 @@ func TestIsValidObjectName(t *testing.T) {
// passing invalid object names.
{"", false},
{"a/b/c/", false},
{"/a/b/c", false},
{"../../etc", false},
{"../../", false},
{"/../../etc", false},
+15 -14
View File
@@ -23,6 +23,7 @@ import (
"time"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/handlers"
)
@@ -49,7 +50,7 @@ func checkCopyObjectPartPreconditions(ctx context.Context, w http.ResponseWriter
// x-amz-copy-source-if-none-match
func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Request, objInfo ObjectInfo, encETag string) bool {
// Return false for methods other than GET and HEAD.
if r.Method != "PUT" {
if r.Method != http.MethodPut {
return false
}
if encETag == "" {
@@ -68,15 +69,15 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
setCommonHeaders(w)
// set object-related metadata headers
w.Header().Set("Last-Modified", objInfo.ModTime.UTC().Format(http.TimeFormat))
w.Header().Set(xhttp.LastModified, objInfo.ModTime.UTC().Format(http.TimeFormat))
if objInfo.ETag != "" {
w.Header()["ETag"] = []string{"\"" + objInfo.ETag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + objInfo.ETag + "\""}
}
}
// x-amz-copy-source-if-modified-since: Return the object only if it has been modified
// since the specified time otherwise return 412 (precondition failed).
ifModifiedSinceHeader := r.Header.Get("x-amz-copy-source-if-modified-since")
ifModifiedSinceHeader := r.Header.Get(xhttp.AmzCopySourceIfModifiedSince)
if ifModifiedSinceHeader != "" {
if givenTime, err := time.Parse(http.TimeFormat, ifModifiedSinceHeader); err == nil {
if !ifModifiedSince(objInfo.ModTime, givenTime) {
@@ -90,7 +91,7 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
// x-amz-copy-source-if-unmodified-since : Return the object only if it has not been
// modified since the specified time, otherwise return a 412 (precondition failed).
ifUnmodifiedSinceHeader := r.Header.Get("x-amz-copy-source-if-unmodified-since")
ifUnmodifiedSinceHeader := r.Header.Get(xhttp.AmzCopySourceIfUnmodifiedSince)
if ifUnmodifiedSinceHeader != "" {
if givenTime, err := time.Parse(http.TimeFormat, ifUnmodifiedSinceHeader); err == nil {
if ifModifiedSince(objInfo.ModTime, givenTime) {
@@ -106,7 +107,7 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
// x-amz-copy-source-if-match : Return the object only if its entity tag (ETag) is the
// same as the one specified; otherwise return a 412 (precondition failed).
ifMatchETagHeader := r.Header.Get("x-amz-copy-source-if-match")
ifMatchETagHeader := r.Header.Get(xhttp.AmzCopySourceIfMatch)
if ifMatchETagHeader != "" {
etag := objInfo.ETag
if shouldDecryptEtag {
@@ -122,7 +123,7 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
// If-None-Match : Return the object only if its entity tag (ETag) is different from the
// one specified otherwise, return a 304 (not modified).
ifNoneMatchETagHeader := r.Header.Get("x-amz-copy-source-if-none-match")
ifNoneMatchETagHeader := r.Header.Get(xhttp.AmzCopySourceIfNoneMatch)
if ifNoneMatchETagHeader != "" {
etag := objInfo.ETag
if shouldDecryptEtag {
@@ -147,7 +148,7 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
// If-None-Match
func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Request, objInfo ObjectInfo) bool {
// Return false for methods other than GET and HEAD.
if r.Method != "GET" && r.Method != "HEAD" {
if r.Method != http.MethodGet && r.Method != http.MethodHead {
return false
}
// If the object doesn't have a modtime (IsZero), or the modtime
@@ -163,15 +164,15 @@ func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Requ
setCommonHeaders(w)
// set object-related metadata headers
w.Header().Set("Last-Modified", objInfo.ModTime.UTC().Format(http.TimeFormat))
w.Header().Set(xhttp.LastModified, objInfo.ModTime.UTC().Format(http.TimeFormat))
if objInfo.ETag != "" {
w.Header()["ETag"] = []string{"\"" + objInfo.ETag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + objInfo.ETag + "\""}
}
}
// If-Modified-Since : Return the object only if it has been modified since the specified time,
// otherwise return a 304 (not modified).
ifModifiedSinceHeader := r.Header.Get("If-Modified-Since")
ifModifiedSinceHeader := r.Header.Get(xhttp.IfModifiedSince)
if ifModifiedSinceHeader != "" {
if givenTime, err := time.Parse(http.TimeFormat, ifModifiedSinceHeader); err == nil {
if !ifModifiedSince(objInfo.ModTime, givenTime) {
@@ -185,7 +186,7 @@ func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Requ
// If-Unmodified-Since : Return the object only if it has not been modified since the specified
// time, otherwise return a 412 (precondition failed).
ifUnmodifiedSinceHeader := r.Header.Get("If-Unmodified-Since")
ifUnmodifiedSinceHeader := r.Header.Get(xhttp.IfUnmodifiedSince)
if ifUnmodifiedSinceHeader != "" {
if givenTime, err := time.Parse(http.TimeFormat, ifUnmodifiedSinceHeader); err == nil {
if ifModifiedSince(objInfo.ModTime, givenTime) {
@@ -199,7 +200,7 @@ func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Requ
// If-Match : Return the object only if its entity tag (ETag) is the same as the one specified;
// otherwise return a 412 (precondition failed).
ifMatchETagHeader := r.Header.Get("If-Match")
ifMatchETagHeader := r.Header.Get(xhttp.IfMatch)
if ifMatchETagHeader != "" {
if !isETagEqual(objInfo.ETag, ifMatchETagHeader) {
// If the object ETag does not match with the specified ETag.
@@ -211,7 +212,7 @@ func checkPreconditions(ctx context.Context, w http.ResponseWriter, r *http.Requ
// If-None-Match : Return the object only if its entity tag (ETag) is different from the
// one specified otherwise, return a 304 (not modified).
ifNoneMatchETagHeader := r.Header.Get("If-None-Match")
ifNoneMatchETagHeader := r.Header.Get(xhttp.IfNoneMatch)
if ifNoneMatchETagHeader != "" {
if isETagEqual(objInfo.ETag, ifNoneMatchETagHeader) {
// If the object ETag matches with the specified ETag.
+28 -23
View File
@@ -36,6 +36,7 @@ import (
miniogo "github.com/minio/minio-go/v6"
"github.com/minio/minio-go/v6/pkg/encrypt"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/event"
@@ -50,12 +51,12 @@ import (
// supportedHeadGetReqParams - supported request parameters for GET and HEAD presigned request.
var supportedHeadGetReqParams = map[string]string{
"response-expires": "Expires",
"response-content-type": "Content-Type",
"response-cache-control": "Cache-Control",
"response-content-encoding": "Content-Encoding",
"response-content-language": "Content-Language",
"response-content-disposition": "Content-Disposition",
"response-expires": xhttp.Expires,
"response-content-type": xhttp.ContentType,
"response-cache-control": xhttp.CacheControl,
"response-content-encoding": xhttp.ContentEncoding,
"response-content-language": xhttp.ContentLanguage,
"response-content-disposition": xhttp.ContentDisposition,
}
const (
@@ -165,8 +166,8 @@ func (api objectAPIHandlers) SelectObjectContentHandler(w http.ResponseWriter, r
BucketName: bucket,
Key: object,
Resource: r.URL.Path,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
})
writeResponse(w, serr.HTTPStatusCode(), encodedErrorResponse, mimeXML)
} else {
@@ -207,8 +208,8 @@ func (api objectAPIHandlers) SelectObjectContentHandler(w http.ResponseWriter, r
BucketName: bucket,
Key: object,
Resource: r.URL.Path,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
})
writeResponse(w, serr.HTTPStatusCode(), encodedErrorResponse, mimeXML)
} else {
@@ -650,7 +651,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
// TODO: Reject requests where body/payload is present, for now we don't even read it.
// Read escaped copy source path to check for parameters.
cpSrcPath := r.Header.Get("X-Amz-Copy-Source")
cpSrcPath := r.Header.Get(xhttp.AmzCopySource)
// Check https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html
// Regardless of whether you have enabled versioning, each object in your bucket
@@ -668,7 +669,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
// Note that url.Parse does the unescaping
cpSrcPath = u.Path
}
if vid := r.Header.Get("X-Amz-Copy-Source-Version-Id"); vid != "" {
if vid := r.Header.Get(xhttp.AmzCopySourceVersionID); vid != "" {
// Check if versionId header was added, if yes then check if
// its non "null" value, we should error out since we do not support
// any versions other than "null".
@@ -1064,6 +1065,9 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
bucket := vars["bucket"]
object := vars["object"]
// To detect if the client has disconnected.
r.Body = &detectDisconnect{r.Body, r.Context().Done()}
// X-Amz-Copy-Source shouldn't be set for this call.
if _, ok := r.Header["X-Amz-Copy-Source"]; ok {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidCopySource), r.URL, guessIsBrowserReq(r))
@@ -1268,7 +1272,7 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
} else if hasServerSideEncryptionHeader(r.Header) {
etag = getDecryptedETag(r.Header, objInfo, false)
}
w.Header()["ETag"] = []string{"\"" + etag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + etag + "\""}
if objectAPI.IsEncryptionSupported() {
if crypto.IsEncrypted(objInfo.UserDefined) {
@@ -1450,7 +1454,7 @@ func (api objectAPIHandlers) CopyObjectPartHandler(w http.ResponseWriter, r *htt
}
// Read escaped copy source path to check for parameters.
cpSrcPath := r.Header.Get("X-Amz-Copy-Source")
cpSrcPath := r.Header.Get(xhttp.AmzCopySource)
// Check https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html
// Regardless of whether you have enabled versioning, each object in your bucket
@@ -1468,7 +1472,7 @@ func (api objectAPIHandlers) CopyObjectPartHandler(w http.ResponseWriter, r *htt
// Note that url.Parse does the unescaping
cpSrcPath = u.Path
}
if vid := r.Header.Get("X-Amz-Copy-Source-Version-Id"); vid != "" {
if vid := r.Header.Get(xhttp.AmzCopySourceVersionID); vid != "" {
// Check if X-Amz-Copy-Source-Version-Id header was added, if yes then check if
// its non "null" value, we should error out since we do not support
// any versions other than "null".
@@ -1537,7 +1541,7 @@ func (api objectAPIHandlers) CopyObjectPartHandler(w http.ResponseWriter, r *htt
// Get request range.
var rs *HTTPRangeSpec
rangeHeader := r.Header.Get("x-amz-copy-source-range")
rangeHeader := r.Header.Get(xhttp.AmzCopySourceRange)
if rangeHeader != "" {
var parseRangeErr error
if rs, parseRangeErr = parseCopyPartRangeSpec(rangeHeader); parseRangeErr != nil {
@@ -1729,7 +1733,7 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
object := vars["object"]
// X-Amz-Copy-Source shouldn't be set for this call.
if _, ok := r.Header["X-Amz-Copy-Source"]; ok {
if _, ok := r.Header[xhttp.AmzCopySource]; ok {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidCopySource), r.URL, guessIsBrowserReq(r))
return
}
@@ -1747,7 +1751,7 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
rAuthType := getRequestAuthType(r)
// For auth type streaming signature, we need to gather a different content length.
if rAuthType == authTypeStreamingSigned {
if sizeStr, ok := r.Header["X-Amz-Decoded-Content-Length"]; ok {
if sizeStr, ok := r.Header[xhttp.AmzDecodedContentLength]; ok {
if sizeStr[0] == "" {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrMissingContentLength), r.URL, guessIsBrowserReq(r))
return
@@ -1949,7 +1953,7 @@ func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http
if isEncrypted {
etag = tryDecryptETag(objectEncryptionKey, partInfo.ETag, crypto.SSEC.IsRequested(r.Header))
}
w.Header()["ETag"] = []string{"\"" + etag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + etag + "\""}
writeSuccessResponseHeadersOnly(w)
}
@@ -2264,18 +2268,19 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
w.Header().Set(xhttp.RetryAfter, "120")
}
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path,
w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
w.Header().Get(xhttp.AmzRequestID), globalDeploymentID)
encodedErrorResponse, _ := xml.Marshal(errorResponse)
setCommonHeaders(w)
w.Header().Set("Content-Type", string(mimeXML))
w.Header().Set(xhttp.ContentType, string(mimeXML))
w.Write(encodedErrorResponse)
w.(http.Flusher).Flush()
}
w.Header().Set(xhttp.ContentType, "text/event-stream")
w = &whiteSpaceWriter{ResponseWriter: w, Flusher: w.(http.Flusher)}
completeDoneCh := sendWhiteSpace(ctx, w)
objInfo, err := completeMultiPartUpload(ctx, bucket, object, uploadID, completeParts, opts)
@@ -2307,7 +2312,7 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
}
// Set etag.
w.Header()["ETag"] = []string{"\"" + objInfo.ETag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + objInfo.ETag + "\""}
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
+3 -3
View File
@@ -156,11 +156,11 @@ func renameAll(srcFilePath, dstFilePath string) (err error) {
// Reliably retries os.RenameAll if for some reason os.RenameAll returns
// syscall.ENOENT (parent does not exist).
func reliableRename(srcFilePath, dstFilePath string) (err error) {
if err = reliableMkdirAll(path.Dir(dstFilePath), 0777); err != nil {
return err
}
i := 0
for {
if err = reliableMkdirAll(path.Dir(dstFilePath), 0777); err != nil {
return err
}
// After a successful parent directory create attempt a renameAll.
if err = os.Rename(srcFilePath, dstFilePath); err != nil {
// Retry only for the first retryable error.
+36 -3
View File
@@ -30,6 +30,7 @@ import (
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/cmd/rest"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/lifecycle"
"github.com/minio/minio/pkg/madmin"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
@@ -329,6 +330,37 @@ func (client *peerRESTClient) SetBucketPolicy(bucket string, bucketPolicy *polic
return nil
}
// RemoveBucketLifecycle - Remove bucket lifecycle configuration on the peer node
func (client *peerRESTClient) RemoveBucketLifecycle(bucket string) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
respBody, err := client.call(peerRESTMethodBucketLifecycleRemove, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// SetBucketLifecycle - Set bucket lifecycle configuration on the peer node
func (client *peerRESTClient) SetBucketLifecycle(bucket string, bucketLifecycle *lifecycle.Lifecycle) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(bucketLifecycle)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketLifecycleSet, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// PutBucketNotification - Put bucket notification on the peer node.
func (client *peerRESTClient) PutBucketNotification(bucket string, rulesMap event.RulesMap) error {
values := make(url.Values)
@@ -435,9 +467,10 @@ func (client *peerRESTClient) BackgroundHealStatus() (madmin.BgHealState, error)
return state, err
}
func (client *peerRESTClient) doTrace(traceCh chan interface{}, doneCh chan struct{}, trcAll bool) {
func (client *peerRESTClient) doTrace(traceCh chan interface{}, doneCh chan struct{}, trcAll, trcErr bool) {
values := make(url.Values)
values.Set(peerRESTTraceAll, strconv.FormatBool(trcAll))
values.Set(peerRESTTraceErr, strconv.FormatBool(trcErr))
// To cancel the REST request in case doneCh gets closed.
ctx, cancel := context.WithCancel(context.Background())
@@ -475,10 +508,10 @@ func (client *peerRESTClient) doTrace(traceCh chan interface{}, doneCh chan stru
}
// Trace - send http trace request to peer nodes
func (client *peerRESTClient) Trace(traceCh chan interface{}, doneCh chan struct{}, trcAll bool) {
func (client *peerRESTClient) Trace(traceCh chan interface{}, doneCh chan struct{}, trcAll, trcErr bool) {
go func() {
for {
client.doTrace(traceCh, doneCh, trcAll)
client.doTrace(traceCh, doneCh, trcAll, trcErr)
select {
case <-doneCh:
return
+3
View File
@@ -43,6 +43,8 @@ const (
peerRESTMethodTargetExists = "targetexists"
peerRESTMethodSendEvent = "sendevent"
peerRESTMethodTrace = "trace"
peerRESTMethodBucketLifecycleSet = "setbucketlifecycle"
peerRESTMethodBucketLifecycleRemove = "removebucketlifecycle"
)
const (
@@ -54,4 +56,5 @@ const (
peerRESTProfiler = "profiler"
peerRESTDryRun = "dry-run"
peerRESTTraceAll = "all"
peerRESTTraceErr = "err"
)
+56 -5
View File
@@ -28,8 +28,10 @@ import (
"time"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/lifecycle"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
trace "github.com/minio/minio/pkg/trace"
@@ -331,7 +333,7 @@ func (s *peerRESTServer) CPULoadInfoHandler(w http.ResponseWriter, r *http.Reque
}
ctx := newContext(r, w, "CPULoadInfo")
info := localEndpointsCPULoad(globalEndpoints)
info := localEndpointsCPULoad(globalEndpoints, r)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
@@ -345,7 +347,7 @@ func (s *peerRESTServer) DrivePerfInfoHandler(w http.ResponseWriter, r *http.Req
}
ctx := newContext(r, w, "DrivePerfInfo")
info := localEndpointsDrivePerf(globalEndpoints)
info := localEndpointsDrivePerf(globalEndpoints, r)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
@@ -358,7 +360,7 @@ func (s *peerRESTServer) MemUsageInfoHandler(w http.ResponseWriter, r *http.Requ
return
}
ctx := newContext(r, w, "MemUsageInfo")
info := localEndpointsMemUsage(globalEndpoints)
info := localEndpointsMemUsage(globalEndpoints, r)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
@@ -468,6 +470,47 @@ func (s *peerRESTServer) SetBucketPolicyHandler(w http.ResponseWriter, r *http.R
w.(http.Flusher).Flush()
}
// RemoveBucketLifecycleHandler - Remove bucket lifecycle.
func (s *peerRESTServer) RemoveBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
globalLifecycleSys.Remove(bucketName)
w.(http.Flusher).Flush()
}
// SetBucketLifecycleHandler - Set bucket lifecycle.
func (s *peerRESTServer) SetBucketLifecycleHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var lifecycleData lifecycle.Lifecycle
if r.ContentLength < 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lifecycleData)
if err != nil {
s.writeErrorResponse(w, err)
return
}
globalLifecycleSys.Set(bucketName, lifecycleData)
w.(http.Flusher).Flush()
}
type remoteTargetExistsResp struct {
Exists bool
}
@@ -674,17 +717,23 @@ func (s *peerRESTServer) TraceHandler(w http.ResponseWriter, r *http.Request) {
return
}
trcAll := r.URL.Query().Get(peerRESTTraceAll) == "true"
trcErr := r.URL.Query().Get(peerRESTTraceErr) == "true"
w.Header().Set("Connection", "close")
w.Header().Set(xhttp.Connection, "close")
w.WriteHeader(http.StatusOK)
w.(http.Flusher).Flush()
filter := func(entry interface{}) bool {
trcInfo := entry.(trace.Info)
if trcErr && isHTTPStatusOK(trcInfo.RespInfo.StatusCode) {
return false
}
if trcAll {
return true
}
trcInfo := entry.(trace.Info)
return !strings.HasPrefix(trcInfo.ReqInfo.Path, minioReservedBucketPath)
}
doneCh := make(chan struct{})
@@ -767,6 +816,8 @@ func registerPeerRESTHandlers(router *mux.Router) {
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketNotificationListen).HandlerFunc(httpTraceHdrs(server.ListenBucketNotificationHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodReloadFormat).HandlerFunc(httpTraceHdrs(server.ReloadFormatHandler)).Queries(restQueries(peerRESTDryRun)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketLifecycleSet).HandlerFunc(httpTraceHdrs(server.SetBucketLifecycleHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketLifecycleRemove).HandlerFunc(httpTraceHdrs(server.RemoveBucketLifecycleHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodTrace).HandlerFunc(server.TraceHandler)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBackgroundHealStatus).HandlerFunc(server.BackgroundHealStatusHandler)
+107 -1
View File
@@ -958,7 +958,7 @@ func (s *posix) ReadFile(volume, path string, offset int64, buffer []byte, verif
}
if !bytes.Equal(h.Sum(nil), verifier.sum) {
return 0, hashMismatchError{hex.EncodeToString(verifier.sum), hex.EncodeToString(h.Sum(nil))}
return 0, HashMismatchError{hex.EncodeToString(verifier.sum), hex.EncodeToString(h.Sum(nil))}
}
return int64(len(buffer)), nil
@@ -1525,3 +1525,109 @@ func (s *posix) RenameFile(srcVolume, srcPath, dstVolume, dstPath string) (err e
return nil
}
func (s *posix) VerifyFile(volume, path string, empty bool, algo BitrotAlgorithm, sum []byte, shardSize int64) (err error) {
defer func() {
if err == errFaultyDisk {
atomic.AddInt32(&s.ioErrCount, 1)
}
}()
if atomic.LoadInt32(&s.ioErrCount) > maxAllowedIOError {
return errFaultyDisk
}
if err = s.checkDiskFound(); err != nil {
return err
}
volumeDir, err := s.getVolDir(volume)
if err != nil {
return err
}
// Stat a volume entry.
_, err = os.Stat(volumeDir)
if err != nil {
if os.IsNotExist(err) {
return errVolumeNotFound
}
return err
}
// Validate effective path length before reading.
filePath := pathJoin(volumeDir, path)
if err = checkPathLength(filePath); err != nil {
return err
}
// Open the file for reading.
file, err := os.Open(filePath)
if err != nil {
switch {
case os.IsNotExist(err):
return errFileNotFound
case os.IsPermission(err):
return errFileAccessDenied
case isSysErrNotDir(err):
return errFileAccessDenied
case isSysErrIO(err):
return errFaultyDisk
default:
return err
}
}
// Close the file descriptor.
defer file.Close()
if algo != HighwayHash256S {
bufp := s.pool.Get().(*[]byte)
defer s.pool.Put(bufp)
h := algo.New()
if _, err = io.CopyBuffer(h, file, *bufp); err != nil {
return err
}
if !bytes.Equal(h.Sum(nil), sum) {
return HashMismatchError{hex.EncodeToString(sum), hex.EncodeToString(h.Sum(nil))}
}
return nil
}
buf := make([]byte, shardSize)
h := algo.New()
hashBuf := make([]byte, h.Size())
fi, err := file.Stat()
if err != nil {
return err
}
if empty && fi.Size() != 0 || !empty && fi.Size() == 0 {
return errFileUnexpectedSize
}
size := fi.Size()
for {
if size == 0 {
return nil
}
h.Reset()
n, err := file.Read(hashBuf)
if err != nil {
return err
}
size -= int64(n)
if size < int64(len(buf)) {
buf = buf[:size]
}
n, err = file.Read(buf)
if err != nil {
return err
}
size -= int64(n)
h.Write(buf)
if !bytes.Equal(h.Sum(nil), hashBuf) {
return HashMismatchError{hex.EncodeToString(hashBuf), hex.EncodeToString(h.Sum(nil))}
}
}
}
+94 -5
View File
@@ -1251,13 +1251,13 @@ var posixReadFileWithVerifyTests = []struct {
{file: "myobject", offset: 25, length: 74, algorithm: SHA256, expError: nil}, // 1
{file: "myobject", offset: 29, length: 70, algorithm: SHA256, expError: nil}, // 2
{file: "myobject", offset: 100, length: 0, algorithm: SHA256, expError: nil}, // 3
{file: "myobject", offset: 1, length: 120, algorithm: SHA256, expError: hashMismatchError{}}, // 4
{file: "myobject", offset: 1, length: 120, algorithm: SHA256, expError: HashMismatchError{}}, // 4
{file: "myobject", offset: 3, length: 1100, algorithm: SHA256, expError: nil}, // 5
{file: "myobject", offset: 2, length: 100, algorithm: SHA256, expError: hashMismatchError{}}, // 6
{file: "myobject", offset: 2, length: 100, algorithm: SHA256, expError: HashMismatchError{}}, // 6
{file: "myobject", offset: 1000, length: 1001, algorithm: SHA256, expError: nil}, // 7
{file: "myobject", offset: 0, length: 100, algorithm: BLAKE2b512, expError: hashMismatchError{}}, // 8
{file: "myobject", offset: 0, length: 100, algorithm: BLAKE2b512, expError: HashMismatchError{}}, // 8
{file: "myobject", offset: 25, length: 74, algorithm: BLAKE2b512, expError: nil}, // 9
{file: "myobject", offset: 29, length: 70, algorithm: BLAKE2b512, expError: hashMismatchError{}}, // 10
{file: "myobject", offset: 29, length: 70, algorithm: BLAKE2b512, expError: HashMismatchError{}}, // 10
{file: "myobject", offset: 100, length: 0, algorithm: BLAKE2b512, expError: nil}, // 11
{file: "myobject", offset: 1, length: 120, algorithm: BLAKE2b512, expError: nil}, // 12
{file: "myobject", offset: 3, length: 1100, algorithm: BLAKE2b512, expError: nil}, // 13
@@ -1296,7 +1296,7 @@ func TestPosixReadFileWithVerify(t *testing.T) {
if test.expError != nil {
expected := h.Sum(nil)
h.Write([]byte{0})
test.expError = hashMismatchError{hex.EncodeToString(h.Sum(nil)), hex.EncodeToString(expected)}
test.expError = HashMismatchError{hex.EncodeToString(h.Sum(nil)), hex.EncodeToString(expected)}
}
buffer := make([]byte, test.length)
@@ -1763,6 +1763,95 @@ func TestPosixStatFile(t *testing.T) {
}
}
// Test posix.VerifyFile()
func TestPosixVerifyFile(t *testing.T) {
// We test 4 cases:
// 1) Whole-file bitrot check on proper file
// 2) Whole-file bitrot check on corrupted file
// 3) Streaming bitrot check on proper file
// 4) Streaming bitrot check on corrupted file
// create posix test setup
posixStorage, path, err := newPosixTestSetup()
if err != nil {
t.Fatalf("Unable to create posix test setup, %s", err)
}
defer os.RemoveAll(path)
volName := "testvol"
fileName := "testfile"
if err := posixStorage.MakeVol(volName); err != nil {
t.Fatal(err)
}
// 1) Whole-file bitrot check on proper file
size := int64(4*1024*1024 + 100*1024) // 4.1 MB
data := make([]byte, size)
if _, err := rand.Read(data); err != nil {
t.Fatal(err)
}
algo := HighwayHash256
h := algo.New()
h.Write(data)
hashBytes := h.Sum(nil)
if err := posixStorage.WriteAll(volName, fileName, bytes.NewBuffer(data)); err != nil {
t.Fatal(err)
}
if err := posixStorage.VerifyFile(volName, fileName, false, algo, hashBytes, 0); err != nil {
t.Fatal(err)
}
// 2) Whole-file bitrot check on corrupted file
if err := posixStorage.AppendFile(volName, fileName, []byte("a")); err != nil {
t.Fatal(err)
}
if err := posixStorage.VerifyFile(volName, fileName, false, algo, hashBytes, 0); err == nil {
t.Fatal("expected to fail bitrot check")
}
if err := posixStorage.DeleteFile(volName, fileName); err != nil {
t.Fatal(err)
}
// 3) Streaming bitrot check on proper file
algo = HighwayHash256S
shardSize := int64(1024 * 1024)
shard := make([]byte, shardSize)
w := newStreamingBitrotWriter(posixStorage, volName, fileName, size, algo, shardSize)
reader := bytes.NewReader(data)
for {
// Using io.CopyBuffer instead of this loop will not work for us as io.CopyBuffer
// will use bytes.Buffer.ReadFrom() which will not do shardSize'ed writes causing error.
n, err := reader.Read(shard)
w.Write(shard[:n])
if err == nil {
continue
}
if err == io.EOF {
break
}
t.Fatal(err)
}
w.Close()
if err := posixStorage.VerifyFile(volName, fileName, false, algo, nil, shardSize); err != nil {
t.Fatal(err)
}
// 4) Streaming bitrot check on corrupted file
filePath := pathJoin(posixStorage.String(), volName, fileName)
f, err := os.OpenFile(filePath, os.O_WRONLY, 0644)
if err != nil {
t.Fatal(err)
}
if _, err := f.WriteString("a"); err != nil {
t.Fatal(err)
}
f.Close()
if err := posixStorage.VerifyFile(volName, fileName, false, algo, nil, shardSize); err == nil {
t.Fatal("expected to fail bitrot check")
}
}
// Checks for restrictions for min total disk space and inodes.
func TestCheckDiskTotalMin(t *testing.T) {
testCases := []struct {
+2 -1
View File
@@ -109,7 +109,7 @@ type PostPolicyForm struct {
}
}
// parsePostPolicyForm - Parse JSON policy string into typed POostPolicyForm structure.
// parsePostPolicyForm - Parse JSON policy string into typed PostPolicyForm structure.
func parsePostPolicyForm(policy string) (ppf PostPolicyForm, e error) {
// Convert po into interfaces and
// perform strict type conversion using reflection.
@@ -256,6 +256,7 @@ func checkPostPolicy(formValues http.Header, postPolicyForm PostPolicyForm) erro
return fmt.Errorf("Invalid according to Policy: Policy Condition failed")
}
// Check if current policy condition is satisfied
condPassed = checkPolicyCond(op, formValues.Get(formCanonicalName), v.Value)
if !condPassed {
return fmt.Errorf("Invalid according to Policy: Policy Condition failed")
}
+1 -1
View File
@@ -48,7 +48,7 @@ func registerDistXLRouters(router *mux.Router, endpoints EndpointList) {
// List of some generic handlers which are applied for all incoming requests.
var globalHandlers = []HandlerFunc{
// set x-amz-request-id, x-minio-deployment-id header.
// set x-amz-request-id header.
addCustomHeaders,
// set HTTP security headers such as Content-Security-Policy.
addSecurityHeaders,
+12 -1
View File
@@ -18,6 +18,7 @@ package cmd
import (
"context"
"encoding/gob"
"errors"
"fmt"
"net/http"
@@ -27,7 +28,7 @@ import (
"syscall"
"github.com/minio/cli"
"github.com/minio/dsync"
"github.com/minio/dsync/v2"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/certs"
@@ -36,6 +37,7 @@ import (
func init() {
logger.Init(GOPATH, GOROOT)
logger.RegisterUIError(fmtError)
gob.Register(HashMismatchError{})
}
// ServerFlags - server command specific flags
@@ -314,6 +316,7 @@ func serverMain(ctx *cli.Context) {
}()
newObject, err := newObjectLayer(globalEndpoints)
logger.SetDeploymentID(globalDeploymentID)
if err != nil {
// Stop watching for any certificate changes.
globalTLSCerts.Stop()
@@ -362,6 +365,14 @@ func serverMain(ctx *cli.Context) {
logger.Fatal(err, "Unable to initialize policy system")
}
// Create new lifecycle system.
globalLifecycleSys = NewLifecycleSys()
// Initialize lifecycle system.
if err = globalLifecycleSys.Init(newObject); err != nil {
logger.Fatal(err, "Unable to initialize lifecycle system")
}
// Create new notification system.
globalNotificationSys = NewNotificationSys(globalServerConfig, globalEndpoints)
+36 -5
View File
@@ -34,6 +34,7 @@ import (
"time"
humanize "github.com/dustin/go-humanize"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/pkg/policy"
)
@@ -1176,7 +1177,7 @@ func (s *TestSuiteCommon) TestPutObject(c *check) {
c.Assert(response.StatusCode, http.StatusOK)
c.Assert(response.ContentLength, int64(len([]byte("hello world"))))
var buffer2 bytes.Buffer
// retrive the contents of response body.
// retrieve the contents of response body.
n, err := io.Copy(&buffer2, response.Body)
c.Assert(err, nil)
c.Assert(n, int64(len([]byte("hello world"))))
@@ -1202,7 +1203,7 @@ func (s *TestSuiteCommon) TestPutObject(c *check) {
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
// The response Etag header should contain Md5sum of an empty string.
c.Assert(response.Header.Get("Etag"), "\""+emptyETag+"\"")
c.Assert(response.Header.Get(xhttp.ETag), "\""+emptyETag+"\"")
}
// TestListBuckets - Make request for listing of all buckets.
@@ -1353,7 +1354,37 @@ func (s *TestSuiteCommon) TestPutObjectLongName(c *check) {
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusOK)
// make long object name.
//make long object name.
longObjName = fmt.Sprintf("%0255d/%0255d/%0255d/%0255d/%0255d", 1, 1, 1, 1, 1)
if IsDocker() || IsKubernetes() {
longObjName = fmt.Sprintf("%0242d/%0242d/%0242d/%0242d/%0242d", 1, 1, 1, 1, 1)
}
// create new HTTP request to insert the object.
buffer = bytes.NewReader([]byte("hello world"))
request, err = newTestSignedRequest("PUT", getPutObjectURL(s.endPoint, bucketName, longObjName),
int64(buffer.Len()), buffer, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
// execute the HTTP request.
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusBadRequest)
verifyError(c, response, "KeyTooLongError", "Your key is too long", http.StatusBadRequest)
// make object name with prefix as slash
longObjName = fmt.Sprintf("/%0255d/%0255d", 1, 1)
buffer = bytes.NewReader([]byte("hello world"))
// create new HTTP request to insert the object.
request, err = newTestSignedRequest("PUT", getPutObjectURL(s.endPoint, bucketName, longObjName),
int64(buffer.Len()), buffer, s.accessKey, s.secretKey, s.signer)
c.Assert(err, nil)
// execute the HTTP request.
response, err = client.Do(request)
c.Assert(err, nil)
c.Assert(response.StatusCode, http.StatusBadRequest)
verifyError(c, response, "XMinioInvalidObjectName", "Object name contains a leading slash.", http.StatusBadRequest)
//make object name as unsuported
longObjName = fmt.Sprintf("%0256d", 1)
buffer = bytes.NewReader([]byte("hello world"))
request, err = newTestSignedRequest("PUT", getPutObjectURL(s.endPoint, bucketName, longObjName),
@@ -1430,7 +1461,7 @@ func (s *TestSuiteCommon) TestHeadOnObjectLastModified(c *check) {
// verify the status of the HTTP response.
c.Assert(response.StatusCode, http.StatusOK)
// retrive the info of last modification time of the object from the response header.
// retrieve the info of last modification time of the object from the response header.
lastModified := response.Header.Get("Last-Modified")
// Parse it into time.Time structure.
t, err := time.Parse(http.TimeFormat, lastModified)
@@ -2728,5 +2759,5 @@ func (s *TestSuiteCommon) TestObjectMultipart(c *check) {
parts = append(parts, part)
}
etag := getCompleteMultipartMD5(parts)
c.Assert(canonicalizeETag(response.Header.Get("Etag")), etag)
c.Assert(canonicalizeETag(response.Header.Get(xhttp.ETag)), etag)
}
+14 -12
View File
@@ -28,6 +28,8 @@ import (
"strconv"
"strings"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/pkg/auth"
)
@@ -68,13 +70,13 @@ var resourceList = []string{
func doesPolicySignatureV2Match(formValues http.Header) APIErrorCode {
cred := globalServerConfig.GetCredential()
accessKey := formValues.Get("AWSAccessKeyId")
accessKey := formValues.Get(xhttp.AmzAccessKeyID)
cred, _, s3Err := checkKeyValid(accessKey)
if s3Err != ErrNone {
return s3Err
}
policy := formValues.Get("Policy")
signature := formValues.Get("Signature")
signature := formValues.Get(xhttp.AmzSignatureV2)
if !compareSignatureV2(signature, calculateSignatureV2(policy, cred.SecretKey)) {
return ErrSignatureDoesNotMatch
}
@@ -131,11 +133,11 @@ func doesPresignV2SignatureMatch(r *http.Request) APIErrorCode {
return ErrInvalidQueryParams
}
switch keyval[0] {
case "AWSAccessKeyId":
case xhttp.AmzAccessKeyID:
accessKey = keyval[1]
case "Signature":
case xhttp.AmzSignatureV2:
gotSignature = keyval[1]
case "Expires":
case xhttp.Expires:
expires = keyval[1]
default:
filteredQueries = append(filteredQueries, query)
@@ -177,13 +179,13 @@ func doesPresignV2SignatureMatch(r *http.Request) APIErrorCode {
}
func getReqAccessKeyV2(r *http.Request) (auth.Credentials, bool, APIErrorCode) {
if accessKey := r.URL.Query().Get("AWSAccessKeyId"); accessKey != "" {
if accessKey := r.URL.Query().Get(xhttp.AmzAccessKeyID); accessKey != "" {
return checkKeyValid(accessKey)
}
// below is V2 Signed Auth header format, splitting on `space` (after the `AWS` string).
// Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature
authFields := strings.Split(r.Header.Get("Authorization"), " ")
authFields := strings.Split(r.Header.Get(xhttp.Authorization), " ")
if len(authFields) != 2 {
return auth.Credentials{}, false, ErrMissingFields
}
@@ -219,7 +221,7 @@ func getReqAccessKeyV2(r *http.Request) (auth.Credentials, bool, APIErrorCode) {
func validateV2AuthHeader(r *http.Request) (auth.Credentials, APIErrorCode) {
var cred auth.Credentials
v2Auth := r.Header.Get("Authorization")
v2Auth := r.Header.Get(xhttp.Authorization)
if v2Auth == "" {
return cred, ErrAuthHeaderEmpty
}
@@ -238,7 +240,7 @@ func validateV2AuthHeader(r *http.Request) (auth.Credentials, APIErrorCode) {
}
func doesSignV2Match(r *http.Request) APIErrorCode {
v2Auth := r.Header.Get("Authorization")
v2Auth := r.Header.Get(xhttp.Authorization)
cred, apiError := validateV2AuthHeader(r)
if apiError != ErrNone {
return apiError
@@ -380,7 +382,7 @@ func getStringToSignV2(method string, encodedResource, encodedQuery string, head
date := expires // Date is set to expires date for presign operations.
if date == "" {
// If expires date is empty then request header Date is used.
date = headers.Get("Date")
date = headers.Get(xhttp.Date)
}
// From the Amazon docs:
@@ -393,8 +395,8 @@ func getStringToSignV2(method string, encodedResource, encodedQuery string, head
// CanonicalizedResource;
stringToSign := strings.Join([]string{
method,
headers.Get("Content-MD5"),
headers.Get("Content-Type"),
headers.Get(xhttp.ContentMD5),
headers.Get(xhttp.ContentType),
date,
canonicalHeaders,
}, "\n")
+7 -6
View File
@@ -26,6 +26,7 @@ import (
"strconv"
"strings"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/sha256-simd"
@@ -44,12 +45,12 @@ func skipContentSha256Cksum(r *http.Request) bool {
)
if isRequestPresignedSignatureV4(r) {
v, ok = r.URL.Query()["X-Amz-Content-Sha256"]
v, ok = r.URL.Query()[xhttp.AmzContentSha256]
if !ok {
v, ok = r.Header["X-Amz-Content-Sha256"]
v, ok = r.Header[xhttp.AmzContentSha256]
}
} else {
v, ok = r.Header["X-Amz-Content-Sha256"]
v, ok = r.Header[xhttp.AmzContentSha256]
}
// If x-amz-content-sha256 is set and the value is not
@@ -81,15 +82,15 @@ func getContentSha256Cksum(r *http.Request, stype serviceType) string {
// X-Amz-Content-Sha256, if not set in presigned requests, checksum
// will default to 'UNSIGNED-PAYLOAD'.
defaultSha256Cksum = unsignedPayload
v, ok = r.URL.Query()["X-Amz-Content-Sha256"]
v, ok = r.URL.Query()[xhttp.AmzContentSha256]
if !ok {
v, ok = r.Header["X-Amz-Content-Sha256"]
v, ok = r.Header[xhttp.AmzContentSha256]
}
} else {
// X-Amz-Content-Sha256, if not set in signed requests, checksum
// will default to sha256([]byte("")).
defaultSha256Cksum = emptySHA256
v, ok = r.Header["X-Amz-Content-Sha256"]
v, ok = r.Header[xhttp.AmzContentSha256]
}
// We found 'X-Amz-Content-Sha256' return the captured value.
+20 -19
View File
@@ -36,6 +36,7 @@ import (
"time"
"github.com/minio/minio-go/v6/pkg/s3utils"
xhttp "github.com/minio/minio/cmd/http"
sha256 "github.com/minio/sha256-simd"
)
@@ -172,7 +173,7 @@ func doesPolicySignatureV4Match(formValues http.Header) APIErrorCode {
region := globalServerConfig.GetRegion()
// Parse credential tag.
credHeader, err := parseCredentialHeader("Credential="+formValues.Get("X-Amz-Credential"), region, serviceS3)
credHeader, err := parseCredentialHeader("Credential="+formValues.Get(xhttp.AmzCredential), region, serviceS3)
if err != ErrNone {
return ErrMissingFields
}
@@ -189,7 +190,7 @@ func doesPolicySignatureV4Match(formValues http.Header) APIErrorCode {
newSignature := getSignature(signingKey, formValues.Get("Policy"))
// Verify signature.
if !compareSignatureV4(newSignature, formValues.Get("X-Amz-Signature")) {
if !compareSignatureV4(newSignature, formValues.Get(xhttp.AmzSignature)) {
return ErrSignatureDoesNotMatch
}
@@ -223,11 +224,11 @@ func doesPresignedSignatureMatch(hashedPayload string, r *http.Request, region s
// Construct new query.
query := make(url.Values)
if req.URL.Query().Get("X-Amz-Content-Sha256") != "" {
query.Set("X-Amz-Content-Sha256", hashedPayload)
if req.URL.Query().Get(xhttp.AmzContentSha256) != "" {
query.Set(xhttp.AmzContentSha256, hashedPayload)
}
query.Set("X-Amz-Algorithm", signV4Algorithm)
query.Set(xhttp.AmzAlgorithm, signV4Algorithm)
// If the host which signed the request is slightly ahead in time (by less than globalMaxSkewTime) the
// request should still be allowed.
@@ -244,10 +245,10 @@ func doesPresignedSignatureMatch(hashedPayload string, r *http.Request, region s
expireSeconds := int(pSignValues.Expires / time.Second)
// Construct the query.
query.Set("X-Amz-Date", t.Format(iso8601Format))
query.Set("X-Amz-Expires", strconv.Itoa(expireSeconds))
query.Set("X-Amz-SignedHeaders", getSignedHeaders(extractedSignedHeaders))
query.Set("X-Amz-Credential", cred.AccessKey+"/"+pSignValues.Credential.getScope())
query.Set(xhttp.AmzDate, t.Format(iso8601Format))
query.Set(xhttp.AmzExpires, strconv.Itoa(expireSeconds))
query.Set(xhttp.AmzSignedHeaders, getSignedHeaders(extractedSignedHeaders))
query.Set(xhttp.AmzCredential, cred.AccessKey+"/"+pSignValues.Credential.getScope())
// Save other headers available in the request parameters.
for k, v := range req.URL.Query() {
@@ -273,24 +274,24 @@ func doesPresignedSignatureMatch(hashedPayload string, r *http.Request, region s
encodedQuery := query.Encode()
// Verify if date query is same.
if req.URL.Query().Get("X-Amz-Date") != query.Get("X-Amz-Date") {
if req.URL.Query().Get(xhttp.AmzDate) != query.Get(xhttp.AmzDate) {
return ErrSignatureDoesNotMatch
}
// Verify if expires query is same.
if req.URL.Query().Get("X-Amz-Expires") != query.Get("X-Amz-Expires") {
if req.URL.Query().Get(xhttp.AmzExpires) != query.Get(xhttp.AmzExpires) {
return ErrSignatureDoesNotMatch
}
// Verify if signed headers query is same.
if req.URL.Query().Get("X-Amz-SignedHeaders") != query.Get("X-Amz-SignedHeaders") {
if req.URL.Query().Get(xhttp.AmzSignedHeaders) != query.Get(xhttp.AmzSignedHeaders) {
return ErrSignatureDoesNotMatch
}
// Verify if credential query is same.
if req.URL.Query().Get("X-Amz-Credential") != query.Get("X-Amz-Credential") {
if req.URL.Query().Get(xhttp.AmzCredential) != query.Get(xhttp.AmzCredential) {
return ErrSignatureDoesNotMatch
}
// Verify if sha256 payload query is same.
if req.URL.Query().Get("X-Amz-Content-Sha256") != "" {
if req.URL.Query().Get("X-Amz-Content-Sha256") != query.Get("X-Amz-Content-Sha256") {
if req.URL.Query().Get(xhttp.AmzContentSha256) != "" {
if req.URL.Query().Get(xhttp.AmzContentSha256) != query.Get(xhttp.AmzContentSha256) {
return ErrContentSHA256Mismatch
}
}
@@ -311,7 +312,7 @@ func doesPresignedSignatureMatch(hashedPayload string, r *http.Request, region s
newSignature := getSignature(presignedSigningKey, presignedStringToSign)
// Verify signature.
if !compareSignatureV4(req.URL.Query().Get("X-Amz-Signature"), newSignature) {
if !compareSignatureV4(req.URL.Query().Get(xhttp.AmzSignature), newSignature) {
return ErrSignatureDoesNotMatch
}
return ErrNone
@@ -325,7 +326,7 @@ func doesSignatureMatch(hashedPayload string, r *http.Request, region string, st
req := *r
// Save authorization header.
v4Auth := req.Header.Get("Authorization")
v4Auth := req.Header.Get(xhttp.Authorization)
// Parse signature version '4' header.
signV4Values, err := parseSignV4(v4Auth, region, stype)
@@ -346,8 +347,8 @@ func doesSignatureMatch(hashedPayload string, r *http.Request, region string, st
// Extract date, if not present throw error.
var date string
if date = req.Header.Get(http.CanonicalHeaderKey("x-amz-date")); date == "" {
if date = r.Header.Get("Date"); date == "" {
if date = req.Header.Get(xhttp.AmzDate); date == "" {
if date = r.Header.Get(xhttp.Date); date == "" {
return ErrMissingDateHeader
}
}
+10 -8
View File
@@ -72,6 +72,9 @@ var errVolumeAccessDenied = errors.New("volume access denied")
// errFileAccessDenied - cannot access file, insufficient permissions.
var errFileAccessDenied = errors.New("file access denied")
// errFileUnexpectedSize - file has an unexpected size
var errFileUnexpectedSize = errors.New("file has unexpected size")
// errFileParentIsFile - cannot have overlapping objects, parent is already a file.
var errFileParentIsFile = errors.New("parent is a file")
@@ -91,18 +94,17 @@ var errLessData = errors.New("less data available than what was requested")
// errMoreData = returned when more data was sent by the caller than what it was supposed to.
var errMoreData = errors.New("more data was sent than what was advertised")
// hashMisMatchError - represents a bit-rot hash verification failure
// error.
type hashMismatchError struct {
expected string
computed string
// HashMismatchError represents a bit-rot hash verification failure error.
type HashMismatchError struct {
Expected string
Computed string
}
// error method for the hashMismatchError
func (h hashMismatchError) Error() string {
// Error method for the hashMismatchError
func (h HashMismatchError) Error() string {
return fmt.Sprintf(
"Bitrot verification mismatch - expected %v, received %v",
h.expected, h.computed)
h.Expected, h.Computed)
}
// Collection of basic errors.
+1
View File
@@ -52,6 +52,7 @@ type StorageAPI interface {
StatFile(volume string, path string) (file FileInfo, err error)
DeleteFile(volume string, path string) (err error)
DeleteFileBulk(volume string, paths []string) (errs []error, err error)
VerifyFile(volume, path string, empty bool, algo BitrotAlgorithm, sum []byte, shardSize int64) error
// Write all data, syncs the data to disk.
WriteAll(volume string, path string, reader io.Reader) (err error)
+38 -1
View File
@@ -17,6 +17,7 @@
package cmd
import (
"bufio"
"bytes"
"crypto/tls"
"io"
@@ -66,6 +67,8 @@ func toStorageErr(err error) error {
return io.EOF
case io.ErrUnexpectedEOF.Error():
return io.ErrUnexpectedEOF
case errFileUnexpectedSize.Error():
return errFileUnexpectedSize
case errUnexpected.Error():
return errUnexpected
case errDiskFull.Error():
@@ -105,7 +108,7 @@ func toStorageErr(err error) error {
fmt.Sscanf(err.Error(), "Bitrot verification mismatch - expected %s received %s", &expected, &received)
// Go's Sscanf %s scans "," that comes after the expected hash, hence remove it. Providing "," in the format string does not help.
expected = strings.TrimSuffix(expected, ",")
bitrotErr := hashMismatchError{expected, received}
bitrotErr := HashMismatchError{expected, received}
return bitrotErr
}
return err
@@ -433,6 +436,40 @@ func (client *storageRESTClient) getInstanceID() (err error) {
return nil
}
func (client *storageRESTClient) VerifyFile(volume, path string, empty bool, algo BitrotAlgorithm, sum []byte, shardSize int64) error {
values := make(url.Values)
values.Set(storageRESTVolume, volume)
values.Set(storageRESTFilePath, path)
values.Set(storageRESTBitrotAlgo, algo.String())
values.Set(storageRESTEmpty, strconv.FormatBool(empty))
values.Set(storageRESTLength, strconv.Itoa(int(shardSize)))
if len(sum) != 0 {
values.Set(storageRESTBitrotHash, hex.EncodeToString(sum))
}
respBody, err := client.call(storageRESTMethodVerifyFile, values, nil, -1)
defer http.DrainBody(respBody)
if err != nil {
return err
}
reader := bufio.NewReader(respBody)
for {
b, err := reader.ReadByte()
if err != nil {
return err
}
if b != ' ' {
reader.UnreadByte()
break
}
}
verifyResp := &VerifyFileResp{}
err = gob.NewDecoder(reader).Decode(verifyResp)
if err != nil {
return err
}
return toStorageErr(verifyResp.Err)
}
// Close - marks the client as closed.
func (client *storageRESTClient) Close() error {
client.connected = false
+3 -1
View File
@@ -16,7 +16,7 @@
package cmd
const storageRESTVersion = "v6"
const storageRESTVersion = "v8"
const storageRESTPath = minioReservedBucketPath + "/storage/" + storageRESTVersion + "/"
const (
@@ -38,6 +38,7 @@ const (
storageRESTMethodDeleteFile = "deletefile"
storageRESTMethodDeleteFileBulk = "deletefilebulk"
storageRESTMethodRenameFile = "renamefile"
storageRESTMethodVerifyFile = "verifyfile"
storageRESTMethodGetInstanceID = "getinstanceid"
)
@@ -51,6 +52,7 @@ const (
storageRESTDstPath = "destination-path"
storageRESTOffset = "offset"
storageRESTLength = "length"
storageRESTEmpty = "empty"
storageRESTCount = "count"
storageRESTMarkerPath = "marker"
storageRESTLeafFile = "leaf-file"
+72 -4
View File
@@ -29,6 +29,7 @@ import (
"time"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
)
@@ -114,7 +115,7 @@ func (s *storageRESTServer) GetInstanceID(w http.ResponseWriter, r *http.Request
s.writeErrorResponse(w, err)
return
}
w.Header().Set("Content-Length", strconv.Itoa(len(s.instanceID)))
w.Header().Set(xhttp.ContentLength, strconv.Itoa(len(s.instanceID)))
w.Write([]byte(s.instanceID))
w.(http.Flusher).Flush()
}
@@ -283,7 +284,7 @@ func (s *storageRESTServer) ReadAllHandler(w http.ResponseWriter, r *http.Reques
s.writeErrorResponse(w, err)
return
}
w.Header().Set("Content-Length", strconv.Itoa(len(buf)))
w.Header().Set(xhttp.ContentLength, strconv.Itoa(len(buf)))
w.Write(buf)
w.(http.Flusher).Flush()
}
@@ -327,7 +328,7 @@ func (s *storageRESTServer) ReadFileHandler(w http.ResponseWriter, r *http.Reque
s.writeErrorResponse(w, err)
return
}
w.Header().Set("Content-Length", strconv.Itoa(len(buf)))
w.Header().Set(xhttp.ContentLength, strconv.Itoa(len(buf)))
w.Write(buf)
w.(http.Flusher).Flush()
}
@@ -357,7 +358,7 @@ func (s *storageRESTServer) ReadFileStreamHandler(w http.ResponseWriter, r *http
return
}
defer rc.Close()
w.Header().Set("Content-Length", strconv.Itoa(length))
w.Header().Set(xhttp.ContentLength, strconv.Itoa(length))
io.Copy(w, rc)
w.(http.Flusher).Flush()
@@ -486,6 +487,71 @@ func (s *storageRESTServer) RenameFileHandler(w http.ResponseWriter, r *http.Req
}
}
// Send whitespace to the client to avoid timeouts as bitrot verification can take time on spinning/slow disks.
func sendWhiteSpaceVerifyFile(w http.ResponseWriter) <-chan struct{} {
doneCh := make(chan struct{})
go func() {
ticker := time.NewTicker(time.Second * 10)
for {
select {
case <-ticker.C:
w.Write([]byte(" "))
w.(http.Flusher).Flush()
case doneCh <- struct{}{}:
ticker.Stop()
return
}
}
}()
return doneCh
}
// VerifyFileResp - VerifyFile()'s response.
type VerifyFileResp struct {
Err error
}
// VerifyFile - Verify the file for bitrot errors.
func (s *storageRESTServer) VerifyFile(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
return
}
vars := mux.Vars(r)
volume := vars[storageRESTVolume]
filePath := vars[storageRESTFilePath]
empty, err := strconv.ParseBool(vars[storageRESTEmpty])
if err != nil {
s.writeErrorResponse(w, err)
return
}
shardSize, err := strconv.Atoi(vars[storageRESTLength])
if err != nil {
s.writeErrorResponse(w, err)
return
}
hashStr := vars[storageRESTBitrotHash]
var hash []byte
if hashStr != "" {
hash, err = hex.DecodeString(hashStr)
if err != nil {
s.writeErrorResponse(w, err)
return
}
}
algoStr := vars[storageRESTBitrotAlgo]
if algoStr == "" {
s.writeErrorResponse(w, errInvalidArgument)
return
}
algo := BitrotAlgorithmFromString(algoStr)
w.Header().Set(xhttp.ContentType, "text/event-stream")
doneCh := sendWhiteSpaceVerifyFile(w)
err = s.storage.VerifyFile(volume, filePath, empty, algo, hash, int64(shardSize))
<-doneCh
gob.NewEncoder(w).Encode(VerifyFileResp{err})
}
// registerStorageRPCRouter - register storage rpc router.
func registerStorageRESTHandlers(router *mux.Router, endpoints EndpointList) {
for _, endpoint := range endpoints {
@@ -533,6 +599,8 @@ func registerStorageRESTHandlers(router *mux.Router, endpoints EndpointList) {
subrouter.Methods(http.MethodPost).Path("/" + storageRESTMethodRenameFile).HandlerFunc(httpTraceHdrs(server.RenameFileHandler)).
Queries(restQueries(storageRESTSrcVolume, storageRESTSrcPath, storageRESTDstVolume, storageRESTDstPath)...)
subrouter.Methods(http.MethodPost).Path("/" + storageRESTMethodVerifyFile).HandlerFunc(httpTraceHdrs(server.VerifyFile)).
Queries(restQueries(storageRESTVolume, storageRESTFilePath, storageRESTBitrotAlgo, storageRESTLength)...)
subrouter.Methods(http.MethodPost).Path("/" + storageRESTMethodGetInstanceID).HandlerFunc(httpTraceAll(server.GetInstanceID))
}

Some files were not shown because too many files have changed in this diff Show More